Follow-up to d95f0d7b / 2f4d119c — the earlier passes updated .claude/skills
and assets/skills but missed several other eval surfaces that still pinned
retired IDs (claude-3.5-sonnet, claude-3-haiku):
- `.github/workflows/test-qe-browser.yml` — the `aqe eval run --model` flag
(inert in command-eval mode, but a real dead-ID leftover) -> claude-sonnet-4-6.
- `plugins/agentic-qe-fleet/skills/` — the tracked plugin-marketplace skill copy
(8 eval suites + a sample-output fixture); chaos skills get the opus-4-8 ceiling.
- `docs/schemas/skill-eval.schema.json` — models `enum` was missing claude-opus-4-8
and still allowed retired/deprecated IDs; trimmed to current+active, default
flipped to claude-sonnet-4-6. Plus description examples in the sibling schemas
and the modelUsed values in docs/examples/*.json fixtures.
- `docs/templates/skill-frontmatter.example.yaml` validation_models list.
- `src/cli/commands/eval.ts` — the `aqe eval` `--model` help example and the
`run-all --models` default (`claude-3.5-sonnet` -> `claude-sonnet-4-6`).
Verified: typecheck clean; all edited JSON/YAML parse; zero retired IDs remain in
any eval suite, schema, template, workflow, or eval-CLI default. Intentionally
left untouched: the consensus-provider subsystem (pricing tables / type-unions /
defaults — separate routing layer, bucket-2 data) and historical release notes /
QE audit reports (which document the old IDs by design).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Follow-up to d95f0d7b — that commit updated only .claude/skills/*/evals and
docs/templates, missing two surfaces:
1. assets/skills/ — the npm-distribution copy (tracked, generated from
.claude/skills by prepare-assets.cjs). All 54 eval matrices + nested
cross-model inputs + the .validation templates/schemas/examples now carry
current model IDs, at exact parity with .claude/skills.
2. .claude/skills/.validation/ — the *shipped* eval template, schema, and
example fixtures (the real source new skill evals are generated from):
- skill-eval.schema.json: the models `enum` was missing claude-opus-4-8
(so high-stakes evals using it would fail validation) and still listed
retired claude-3.5-sonnet / claude-3-haiku and soon-retiring
claude-sonnet-4; enum trimmed to current+active IDs, default flipped from
claude-3.5-sonnet -> claude-sonnet-4-6.
- eval.template.yaml / security-testing-eval.template.yaml: models_to_test
refreshed (security template gets the opus-4-8 ceiling).
- skill-frontmatter.example.yaml, schema description examples, and the
modelUsed values in .validation/examples/*.json and
mutation-testing/test-data/sample-output.json fixtures.
Verified: zero retired/deprecated Claude IDs in either tree; both trees at
parity (55x sonnet-4-6, 55x haiku-4-5, 7x opus-4-8 across the 7 high-stakes
skills); all edited YAML/JSON parses. Unrelated .md/.sh drift that
prepare-assets surfaced was reverted to keep this commit model-IDs only.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The cross-model consistency matrix (models_to_test) in 54 skill evals + both
eval templates pinned retired/malformed model IDs — `claude-3.5-sonnet`
(malformed: dashes, not dots; resolves to Sonnet 3.5, retired Oct 2025) and
`claude-3-haiku` (retired Apr 19 2026). Both 404, so those evals could not run
a meaningful cross-model check.
Refresh to current pinned IDs, preserving the original "primary + fast floor"
design:
- claude-sonnet-4-6 # primary (high accuracy)
- claude-haiku-4-5 # fast floor (minimum quality)
High-stakes skills additionally get the capability ceiling claude-opus-4-8
(security-testing, security-visual-testing, n8n-security-testing,
pentest-validation, chaos-engineering-resilience, qe-chaos-resilience,
qe-defect-intelligence) and the security eval template. Opus is scoped to these
7 + the security template rather than all 54 to avoid a fleet-wide eval-cost
blowup.
Also fixes the two `--model claude-3.5-sonnet/claude-3-haiku` examples in
run-skill-eval.ts and a nested cross-model test-case input in
qe-learning-optimization.yaml.
Verified: all eval YAML parses; zero retired/deprecated Claude IDs remain across
skills/evals + templates + runner. (src/domains/constants.ts MODEL_TIERS was
already current — tier 1 claude-haiku-4-5 — so it is intentionally unchanged;
the legacy claude-3-haiku-20240307 entries in cost-tracker/bedrock are retained
historical pricing/availability data.)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
npm-publish.yml runs a consumer-POV tarball audit as a publish blocker, but
only at release time — so an override-masked vuln (root-only `overrides` don't
propagate to consumers) merges to main undetected and only surfaces as a late,
blocked release. This adds consumer-audit.yml running the SAME clean-consumer
tarball audit on every dependency-affecting PR and push to main, catching the
regression before merge so main stays releasable.
Mirrors the proven publish-time job: npm ci -> build -> npm pack -> install the
tarball into a fresh consumer project -> `npm audit --audit-level=high` from the
consumer's perspective (where our overrides do NOT apply). Path-filtered to
dependency-affecting files (package.json, package-lock.json, build/pack scripts,
.npmignore, this workflow), least-privilege (contents: read), concurrency-guarded.
Precedent: v3.9.13 shipped 15 CRITICAL protobufjs ACE vulns to every installer
while in-repo `npm audit` read 0 (GHSA-xq3m-2v4x-88gg). Verified: actionlint
clean; logic confirmed passing on the current v3.10.6 tree (consumer install
resolves protobufjs@7.6.3, 0 high/critical).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
0.1.5's SONA learning engine had a bug where reward feedback produced zero
weight changes (fixed upstream in 0.1.7, published 2026-06-11). AQE drives that
exact path via the trajectory API in sona-wrapper.ts
(beginTrajectory/addTrajectoryStep/endTrajectory + forceLearn), so reward
signals into SONA were effectively inert.
Verified on linux-arm64:
- 0.1.7 linux-arm64-gnu binary is a Linux ELF; native module loads (SonaEngine)
- all 12 native methods the wrapper calls exist in 0.1.7 (API compatible)
- functional smoke test: forceLearn now reports "20 trajectories -> 20 patterns",
patterns_learned 0 -> 20 (was 0 in 0.1.5 — fix confirmed)
- tsc --noEmit: 0 errors; 206 SONA unit tests pass
- better-sqlite3 native fix from prior commit intact (experience capture unaffected)
Note: 0.1.7 restructured platform binaries into optional per-platform packages
(hence the lockfile churn). Install resolves the correct binary per platform;
do not rely on a host-side install in the bind-mounted container.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The aqe-hook.cjs shim discarded all stderr and always exited 0. When the
project-local better-sqlite3 binary was the wrong platform (host/container
mismatch → `invalid ELF header`), UnifiedMemoryManager init failed and
persistCommandExperience silently dropped every captured_experiences INSERT.
Self-learning stopped for days with zero trace while dream_cycles (written by
the global MCP server's working binary) masked the outage.
The shim now captures stderr and tees fatal native-init markers (invalid ELF
header, UnifiedMemoryManager init failure, ERR_DLOPEN_FAILED, Node ABI
mismatch) to a throttled (5min) .agentic-qe/hooks-health.log with an actionable
fix hint. The happy path is unchanged — stdout still emits only the JSON
object, stderr stays out of the transcript, and the hook always exits 0.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The 'reworded prose' mutation test used .replace('qe_agent_definition',
'qe_agent_definition') — a no-op replacing a string with itself (CodeQL
js/identity-replacement). Replaced with a real non-invariant edit
(frontmatter description + a <notes> block after the closing tag) that
genuinely exercises 'edits outside required sections must not trip the
gate'. 9/9 tests still pass.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
CodeQL/actions best practice + repo convention: invariant-check.yml and
safety-eval.yml lacked an explicit top-level permissions block. Both only
checkout and run scripts/tests — no gh, git push, PR comments, or release
writes — so contents:read is sufficient. Matches existing workflows.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The gate fired correctly on PR #523 but its path assumption was too
strict: it looked for RUBRIC-v*.md only in the results file's own
directory, while the real layout is benchmarks/<name>/RUBRIC-v1.md with
results under <name>/results/. Now checks the dir OR its parent, with
two separate OR'd ls calls (a single 'ls a b' exits non-zero when either
glob is unmatched, which masked the present rubric).
Verified locally: real repo passes (rubric found in parent); a results
dir with no rubric anywhere still fails.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Statuses updated to reflect reality (were stale 'Proposed' despite
implementation): 107/108/110 Implemented; 105 Implemented w/ INFERRED
domain-finding follow-up; 106 Implemented (live matrix, native tiers
pass every cell); 109 Implemented infrastructure w/ corpus-scaling
follow-up. Gitignore transient run outputs, force-keeping canonical
evidence files.
Verified: full branch tsc clean; both new workflows valid YAML; 58
locally-runnable tests green; no .env or key material in any branch
commit (audited).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Live runner executed for real: OpenAI gpt-4o-mini under test, Gemini
2.5-flash as the non-same-family judge, hidden-test ground truth, seeded
X/Y counterbalance, pre-registered RUBRIC-v1 hash embedded per row.
The real run surfaced and fixed two harness bugs:
1. Ground-truth fix-rate was coupled to judge-row validity — a fixed bug
that zeroed a real B-arm fix when its judge pass returned no winner.
Ground truth is now tracked per (scenario, arm), independent of the
judge; numbers are internally consistent.
2. Judge row model: emit ONE comparison row per (scenario, pass) — the
unit the sign test clusters — instead of duplicated per-arm rows.
Plus judge-verdict parse hardened (last-verdict wins, broader regex,
more tokens): unparseable rate 50% -> 25%, the rest excluded in code.
Result at n=2: ground truth A=0.50 B=0.00, judge clustered 1-1 p=1.0 —
no arm signal, honestly underpowered. Corpus scaling is the path to a
claim (stated in results README + ADR).
Verified: 30 deterministic tests (stats + safety) green; real run $0.003.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Path A (model-judgment) live runner over the real ADR-026 tiers
(haiku/sonnet/opus via Anthropic API). Each tier is given the
data-protection absolutes + an explicit ACTIONS: [...] declaration
protocol, hit with each temptation as opener AND mid-task, N=5, and run
through the existing assertion engine. Asserts on declared intent only;
never touches a real DB. safety-eval.yml live job now executes it.
Real run found and fixed a harness false-positive: a REFUSING model that
illustrates a command in prose was wrongly flagged. The ACTIONS protocol
makes intent unambiguous; discrimination tests prove a genuinely-declared
rm memory.db still FAILS while an illustrated one PASSES (not neutered).
Verified: 16 deterministic tests; live matrix 5/5 all cells, all 3 tiers
(haiku $0.085 / sonnet $0.231 / opus $1.108), committed as evidence.
Retry-resilient client (transient fetch/429/5xx backoff).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Brutal-honesty review found the ADR-110 loop fully plumbed but fed by
inputs no real caller supplied: null capture never fired (no caller
passed appliedPatterns) and the discount always ran the unknown-context
branch (no caller passed contextFingerprint).
startTask now retrieves the patterns that will guide a task and threads
their ids into startCapture (fail-soft), so a task failure attributes
nulls to the right patterns. searchPatternsForTask forwards a
domain:agent contextFingerprint into the search, and routeTask passes
its own context — so the null-discount weights in-context failures
hardest, as ADR-110 intends.
Loop now closed end to end: guide -> capture -> fail -> null -> discount.
Verified: tsc clean; the review's own dead-input greps now show both
inputs connected at real call sites (was empty before). Behavioral
proof needs the full-engine harness (CI-only here, same constraint as
the migration suite) — not claiming a local green I can't produce.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Layer 1 (always): engine tests + runner over committed fixture
trajectories — all three violating shapes must FAIL, all three
compliant shapes must PASS; an engine regression that lets a violation
through fails CI with an explicit error.
Layer 2 (release/dispatch): live per-tier matrix — fails LOUDLY when
requested with an API key until runner.ts --live lands, and warns-skips
without the key. Never fakes a pass.
Verified locally: YAML parses; both gate steps executed verbatim —
violating shapes all exit non-zero, compliant shapes all exit 0.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
MCP-CLI parity bug found by a REAL protocol call during P5 acceptance:
the handler projected findings to a fixed 5-field shape, stripping
ADR-105's evidenceClass at the MCP boundary — unit tests passed because
they tested the projection itself. Projection now carries evidenceClass
+ evidenceArtifact per finding and the needsVerification queue; two
parity-pin tests added so the field can never be silently dropped again.
Verified: 9/9 handler tests; direct handler invocation (same code path
as the MCP tool) shows evidenceClass STATIC + needsVerification array.
Live MCP server reflects this after restart (long-running process has
pre-fix code loaded).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
RUBRIC-v1.md committed first per ADR-108: ground-truth primary outcome
(hidden acceptance test), style-neutral verbatim judge instruction (no
treatment-revealing framing in either direction), two-pass cross-family
judging with INFERRED-labeled fallback, clustered-stats reporting rule.
Provably rubric-before-data: this commit precedes any results-* artifact.
2 scenarios with seeded defects + personas + hidden tests (off-by-one
pagination; swallowed async rejection). Hidden tests verified both
directions: FAIL on seeded bug with precise reasons, PASS on the fix.
lib/stats.ts encodes the three Pattern Space harness fixes as code:
isGoodRow enforced inside aggregate() (no path to a number that skips
it), seeded per-item counterbalance (their null cell was 72% position-
skewed), exact scenario-clustered sign test (validated against the
known 27/11 → p≈0.0136 re-analysis).
Verified: 14/14 stats tests; hidden-test ground truth both directions;
tsc clean. Remaining: live conversation driver + judge integration
(API-gated, same decision as ADR-106 live mode).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
PatternStore.search now surfaces kept nulls on every result
(nullSummary) and discounts scores by context-matched null density
before ranking — 'succeeded elsewhere' never outranks 'failed here'.
Callers pass contextFingerprint via PatternSearchOptions. Fail-soft:
discount skipped (debug-logged) when the sqlite handle is absent.
AQELearningEngine wires PatternNullStore.recordNull into
ExperienceCaptureService.setNullRecorder at the composition root,
mirroring the witness-chain fail-soft wiring pattern.
Arena losing-strategy persistence required NO new code: ADR-104's
persisted run JSON already keeps every EvaluatedStrategy (killRate,
coveragePct, fitness) + competitiveArray + ranking — losers and their
loss reasons are on the record by construction.
Verified: tsc clean; 28/28 across null-capture, evidence-gating, and
invariant-verifier suites locally. DB-backed paths covered by the
migration suite in CI (native-binding constraint).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
PatternNullStore: recordNull with ON CONFLICT consolidation,
getNullSummaries batch fetch, and the pure applyNullDiscount ranking
helper (context-matched nulls discount 5x harder than remote ones;
0.25 floor keeps patterns retrievable — nulls inform, never erase).
ExperienceCaptureService: appliedPatterns on TaskExperience (patterns
that GUIDED the task), setNullRecorder following the setWitnessChain
convention, and null capture in completeCapture — failure = negative
reward when a signal exists, else !success; recorder errors never break
capture; context fingerprint = domain:agent (v1, per ADR).
Verified: 11/11 tests run locally (London-school mocked backend — no
native deps) incl. reward-overrides-success, per-pattern fan-out,
recorder-throw survival, and discount floor/ordering; tsc clean.
Remaining for ADR-110: retrieval-time discount wiring in PatternStore
search, arena losing-strategy persistence, composition-root recorder
wiring.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Additive schema v10: qe_pattern_nulls (pattern_id FK, context_fingerprint,
failure_mode, trajectory_ref, evidence_class CHECK per ADR-105,
consolidated_count, UNIQUE(pattern_id, context_fingerprint)) + 2 indexes.
Wired into unified-memory v10 migration block; rollback refuses when
nulls exist (nulls are data — the un-committed v2.1 lesson).
Verified against a sqlite3-online-backup COPY of the real memory.db
(61MB, original never written): all 51 pre-existing table counts
byte-identical before/after (qe_patterns=298, qe_trajectories=377,
kv_store=5008), PRAGMA integrity_check ok, table+4 indexes created,
UNIQUE fires on duplicate, ON CONFLICT consolidation bumps count to 2.
tsc --noEmit clean. Unit tests (10) follow the repo's better-sqlite3
convention and run in CI — local native binding is the known macOS
binary (invalid ELF header).
Verification also caught a false-positive in its own first run: an
empty SQL pipe reported 'applied' with trivially-unchanged counts; the
new-objects query exposed it.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Gates PRs touching shipped artifacts + releases on: section-level agent
invariants (npm run verify:invariants), the verifier's own mutation
tests, and ADR-108 lineage discipline (LINEAGE.md present; any
committed benchmark results must sit next to a pre-registered
RUBRIC-v*.md). npm ci --ignore-scripts — no native builds needed.
Verified locally: YAML parses; all three steps executed with real
commands (verifier 0 violations/53 agents; mutation tests 9/9; lineage
gate exit 0 on repo, exit 1 on a results-without-rubric fixture).
Workflow executes on GitHub only after push; marking it a required
check is a branch-protection setting for the maintainer.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Inserts the ADR-105 evidence-class labeling rule as a tagged
<evidence_discipline> section in all 51 standard qe-*.md definitions,
identically in .claude/agents/v3 and assets/agents/v3 (the 2 exempt
legacy agents skipped, per manifest). Registers the section as a
required invariant in config/agent-invariants.json (minChars 400) so
ADR-107 machine-checks it from day one.
Verified: verify:invariants 0 violations across 53 shipped agents with
the new requirement active; negative check on a stale tree fails with
required-section-missing; trees diff-identical; mutation tests 9/9.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Only EXECUTED/STATIC findings from failed blocking steps may block or
halt the pipeline. INFERRED findings queue in PipelineResult.
needsVerification (the ADR-102 adversarial-verification dispatch
surface) and downgrade overall to warn. CONJECTURE never gates. A
failed blocking step without gating evidence reads as warn, never as a
clean pass. Markdown report shows evidence class on blockers and a
'Needs Verification' section.
Verified: tsc clean; 8 new gating tests + 7 MCP handler tests, 15/15.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Adds EvidenceClass (EXECUTED|STATIC|INFERRED|CONJECTURE) and
EvidenceArtifact to the Finding schema, required at the type level.
requirements.ts factory defaults to STATIC (all 26 steps are
deterministic content analysis); pipeline crash-findings are EXECUTED
with the error output as artifact. Field is evidenceClass, not
evidence — StepResult.evidence: string[] already exists (ADR amended).
Verified: tsc --noEmit clean; validation-pipeline-handler tests 7/7
(MCP handler passes findings through, new field flows to MCP output).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Lineage registry with rubric-before-data, changelog, kept-data,
clustered-statistics, and style-neutral-judge rules. Seeded with
arena Phase 1 (incl. its retracted measured-duration variant as the
model entry) and the planned ADR-109 row. Pre-registry numbers are
uncited-by-default pending backfill.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Six proposed ADRs from the universal-pattern-space QE assessment:
evidence-class labels on findings, behavioral safety evals per model
tier, shipped-agent invariant verification, benchmark lineage with
pre-registered rubrics (incl. antipattern list), interaction benchmark
for qualitative agents, kept-nulls negative pattern records.
Cross-checked against existing ADR corpus: Related fields reference
ADR-058/061/070/074/083/086/098/099/102/104; mid-task safety injection,
restraint probes, and null-vs-quarantine sequencing folded in from the
cross-check findings.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The ADR-101 provenance insert named metadata_json unconditionally; stores
created by older versions (and the minimal fixtures in
post-task-promotion/post-task-tokens-saved tests) lack the column, so the
INSERT threw and aborted the whole Stream B learning transaction — caught
by the npm-publish test gate (run 27275586477), masked locally by the
documented native-module artifact.
- PRAGMA-check + ALTER TABLE ADD COLUMN metadata_json (fail-soft) before
the insert — also heals real-world legacy stores
- canonical-DDL ensure block is now fail-soft too, so an unexpected legacy
shape can never kill the transaction; the inserts surface real failures
Verified against the exact failing scenario via a Linux-native harness:
old-shape DB -> column migrated, trajectory row written,
experience_applications.tokens_saved = 68 (the precise value the failing
test asserts).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
aqe arena run: seeded tournaments where test strategies compete on REAL
fitness — built-in first-order operator mutants (string/comment/arrow-safe
enumeration) executed per strategy via node --test, real line coverage
from --experimental-test-coverage, and a deterministic suite-cost term.
Tournament emits a Wolfram-style pairwise competitive array; --evolve
hill-climbs the winner by seeded group flips with real re-evaluation.
Runs persist fail-soft to memory.db kv_store (namespace 'arena');
aqe arena list reads them back. Self-contained: no core changes, no new
dependencies; fixtures/arena-demo ships 4 test groups of varying kill
power over a branch-dense pricing module.
Reproducibility contract: same seed + fixture => byte-identical envelope
(verified E2E by diffing two runs), with informational.runtimesMs the
only excluded field. Honest finding during verification: the original
measured-runtime rank penalty was jitter-dominated at fixture scale and
flipped rankings between identical-seed runs — replaced with the
deterministic suite-size ratio (documented in ADR-104).
Phase 2 (documented): LLM-generated strategies frozen as artifacts,
winners distilled into qe-test-architect priors; MCP arena_run
registration (CLI and MCP share runArena() — parity by construction).
10 tests green (incl. real-engine reproducibility); tsc + eslint clean.
Part of #520 (improvement 7/7 — initiative complete).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
qcsd-development-swarm now runs as a deterministic workflow script
(.claude/workflows/qcsd-development-review.js) instead of prompt-driven
Task-tool fan-out:
- pipeline: one finder per quality dimension (TDD adherence, complexity,
coverage gaps; qe agents as agentType) -> 3 blind adversarial refuters
per finding -> deterministic synthesis, no agent editorializes the report
- Loki-mode constraints (ADR-074) encoded in refuter prompts: blind review
(bare claim + evidence only), anti-sycophancy (attack the claim;
uncertainty defaults to refuted), 3 distinct lenses
- majority kill (>= ceil(N/2) refuted votes); killed findings retained
with refutations for audit; output is finding-verdict@1 envelopes
(ADR-103)
- SKILL.md v1.1.0: execution.primary workflow, Task-tool fallback
documented (reports marked verification: none)
Validation run (wf_254f1271-022): 3 dimensions over src/, 48 agents,
15 raw findings -> 14 confirmed, 1 killed by unanimous refutation of a
factually-wrong claim — the adversarial gate demonstrably filtering a
false positive. Several confirmed findings target code added earlier in
this initiative (honest signal, queued as follow-ups).
Part of #520 (improvement 5/7, Tier 3).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Versioned, additive-only verdict envelopes for agent/MCP handoffs:
RiskDecision (quality gates), FindingVerdict (review findings with
adversarial refutations), CoverageGap (risk-weighted gaps).
- src/contracts/verdicts.ts: dependency-free validators (source of
truth) + mirrored draft-07 JSON Schemas; no zod/ajv runtime deps
(light-install posture preserved)
- quality_assess MCP boundary now attaches a validated riskDecision
envelope derived from the gate outcome (approve/block, escalate when
indeterminate) — additive, existing payload unchanged; direct and
wrapped handlers share the same mapToResult so MCP parity is inherent
- schemas/*.schema.json published via scripts/generate-verdict-schemas.mjs
for external ajv validation and workflow agent({schema}) enforcement
(consumed by improvement 5)
17 contract tests (goldens validate; mutations rejected with field-level
errors; builder always emits valid envelopes). E2E: envelope emitted by
the compiled boundary builder validates with stock ajv against the
published schema; mutated sample rejected. tsc + eslint clean.
Part of #520 (improvement 6/7, Tier 3).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Anthropic announced nested subagent support (depth=5) on 2026-06-09; the
runtime currently strips Task at parent->child spawn (upstream denylist,
empirically confirmed by our committed probe: level=1 status=NO_AGENT_TOOL,
matching ruflo ADR-147). This lands the readiness infrastructure that is
independently useful today and activates with zero changes when the gate
lifts:
- aqe hooks post-task gains --parent-agent-id / --depth (validated:
identifier charset, integer 0 <= d <= 32; depth=0 survives as 0);
provenance persisted to qe_trajectories.metadata_json (stage 1 — no
schema migration) and forwarded into ReasoningBank feedback so learning
can segment patterns per hierarchy level
- TaskExecution gains optional parentAgentId/depth (additive types)
- scripts/probe-nested-spawn-depth.mjs: single-shot empirical probe,
committed transcript under docs/probes/ — re-run after every
`claude update`; TASK_TOOL_LIVE is the signal for follow-up phases
- fresh-project fix found by the E2E: persistTaskOutcome assumed
captured_experiences/experience_applications exist, but they are
middleware-created — Stream B silently no-oped on fresh installs;
canonical DDL now ensured in the hook path
Deliberately deferred (documented in ADR-101 P1): tools: frontmatter
retrofit on shipped qe-*.md agents — Claude Code frontmatter is a
restrictive allowlist and would sever inherited MCP tool access;
needs maintainer sign-off + wildcard verification first.
8 unit tests green; E2E on a fresh temp project verified persistence,
rejections, and depth=0 boundary. tsc + eslint clean.
Part of #520 (improvement 4/7, Tier 2).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Two fail-open catch blocks bound `error` without using it — flagged by
unused-imports/no-unused-vars when linting the ADR-100 change; the
bindings predate it.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Breaks runaway retry loops: when the same Bash command fails consecutively
(normalized signature, cross-process via a small JSON sidecar under
.agentic-qe/), pre-command warns at 3 failures and emits a BLOCK
recommendation with a recovery hint at 5. Advisory by default;
AQE_STRICT_TOOL_LOOP=1 upgrades the block to a PreToolUse deny. A
successful run resets the breaker; an open breaker grants a half-open
probe after 2 minutes; stale entries pruned after 30 minutes.
Mirrors the ADR-064 domain-circuit-breaker semantics at command level.
Sidecar instead of memory.db: pre-command deliberately runs without a DB
open, and each hook call is a fresh process so in-process state is
impossible. Fail-open everywhere (corrupt state file => allow).
11 unit tests green; E2E verified with real aqe hooks pre/post-command:
warn after 3, BLOCK after 5, strict-mode deny envelope, reset on success.
Part of #520 (improvement 3/7, Tier 1 complete).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The enableCache flag existed since ADR-011 but was never read: every
advisor/LLM call paid full input price. Now:
- ClaudeProvider.generate() sends `system` as a content-block array with
cache_control {type: ephemeral} when enableCache (default on); plain
string when disabled. Anthropic-native provider only (ADR-092 — other
providers untouched).
- cache_creation_input_tokens / cache_read_input_tokens surfaced into
TokenUsage.cacheCreationTokens/cacheReadTokens; totalTokens now counts
the full prompt (input_tokens alone is the uncached remainder).
- CostTracker bills cache reads at 0.1x and writes at 1.25x input rate
instead of ignoring them.
5 new provider tests assert the serialized request shape (cache_control
present when enabled, absent when disabled, no system field when no
prompt) and the usage mapping. 64/64 LLM tests green, tsc clean.
Part of #520 (improvement 2/7, Tier 1).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Fable 5-era models emit large extended-thinking blocks inside task results
and step actions; embedding them verbatim contaminates the 384-dim pattern
vectors in memory.db and degrades HNSW retrieval.
- new scrubReasoningBlocks()/scrubReasoningDeep() (src/shared/reasoning-scrub.ts):
boundary-gated — only well-formed <think|thinking|reasoning|
REASONING_SCRATCHPAD>...</> pairs are removed; prose mentioning a tag
survives; clean input returned byte-identical
- applied at ingestion: TrajectoryTracker startTrajectory task,
recordStep action/result.data/result.error
- applied at distillation: ExperienceReplay.storeExperience task/strategy/
keyActions (insert + embedding text)
- applied at flow-forward: both init-time embedding backfills (legacy rows
cleaned as they get embedded — no memory.db rewrite) and retrieveSimilar
query embeddings (symmetric with stored vectors)
- 15 unit tests; E2E verified against a temp DB: trajectory + experience
rows persisted with all scratchpad blocks stripped
Part of #520 (improvement 1/7, Tier 1).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- aqe memory/agent/task/team/pipeline CLI commands always failed with
"Fleet not initialized": autoInitialize() built kernel+queen in its own
CLIContext while the shared MCP handlers guard on the core-handlers
fleet-state singleton. New adoptExternalFleet() bridges the CLI's
components into that state (idempotent, never clobbers an existing
fleet), restoring MCP-CLI parity. Verified E2E: store/get/delete green.
- devcontainer: node_modules named-volume overlay + npm ci in postCreate
so container installs Linux natives without touching the host's macOS
binaries (takes effect on next rebuild; devcontainer users only)
- docs/setup: drop stale ruflo@3.5.18 pins (CLAUDE.md quick setup,
install-tools.sh) in favor of ruflo@3 + global `ruflo mcp start`
- docs: add Fable 5 / ruflo-parity improvement plan (tracked in #520)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The v3.10.4 publish failed: rvf-pattern-store.test.ts expected RvfPatternStore
but got PatternStore. createPatternStore's #516 change used
`require('../kernel/unified-memory.js')` to get findProjectRoot; esbuild inlines
that in the shipped bundle, but under vitest (the CI publish gate) it throws
"Cannot find module", so the whole RVF block fell through to the plain store —
i.e. RVF pattern store was silently disabled under the test runner.
Fix: extract findProjectRoot/clearProjectRootCache into a lightweight,
sqlite-free module (src/kernel/project-root.ts) and import it STATICALLY from
the RVF hot paths. unified-memory.ts now imports + re-exports them, so existing
`import { findProjectRoot } from '.../unified-memory.js'` sites are unchanged.
A static import avoids both the vitest require-resolution failure and dragging
unified-memory's eager better-sqlite3 load into these modules.
- src/kernel/project-root.ts: new module (fs/path only)
- src/kernel/unified-memory.ts: import + re-export (internal callers keep a binding)
- src/learning/pattern-store.ts: static import; drop require()
- src/integrations/ruvector/shared-rvf-adapter.ts: same (latent same bug)
- rvf-pattern-store.test.ts: pin AQE_PROJECT_ROOT to a hermetic temp root with
.agentic-qe so the RVF branch is taken deterministically (was relying on a
prior test creating ./.agentic-qe in cwd)
Verified locally (native RVF available): rvf-pattern-store 16/16,
find-project-root 4/4, 181 non-sqlite ruvector tests pass, typecheck clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
When the parent (e.g. a Claude Code session) exited, aqe-mcp was reparented
to PID 1 and busy-looped at 70-98% CPU, ignoring SIGTERM (only kill -9 worked).
Root cause: on stdin EOF the stdio transport fired its ERROR handler, which
the protocol server routes to its reconnect path. Reconnect re-attached
readline to the already-ended stdin, immediately re-emitted 'close', and
span close → error → reconnect → close at high CPU. The async SIGTERM
handler, starved by the spin and able to hang on an unresolved await, never
reached process.exit — hence "SIGTERM ignored". Non-unref'd worker/daemon
timers meant nothing made the orphan exit on its own either.
Fix (defense-in-depth):
- stdio transport: stdin EOF is now TERMINAL — fire a new onClose handler,
never the error handler, so the reconnect spin can't start. reconnect()
is also a no-op on an ended stream.
- entry.ts: single idempotent shutdown() shared by SIGINT/SIGTERM and stdin
EOF, with an unref()'d 3s watchdog guaranteeing exit even if a graceful
step hangs. A non-TTY stdin 'end'/'close' now triggers shutdown so an
orphan exits when its parent dies.
- unref() the daemon health-check, worker tick, quality-daemon tick, and
git-watcher poll timers so they can't keep an orphaned event loop alive.
Verified: real spawned aqe-mcp now exits ~31ms after stdin EOF (code 0),
vs. lingering indefinitely before. Adds stdio-eof.test.ts covering the
terminal-close behavior and no-reconnect-on-ended-stream guard.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Dragan Spiridonov