Stops unbounded growth of patterns.rvf / brain.rvf.
- #495: AbortSignal in HybridMemoryBackend.initialize() so a 5s
createHybridBackendWithTimeout abort actually cancels the work
instead of leaking a 14-min runaway init (~20 GB / 12 MB/s leak
observed in the field). UnifiedMemoryManager.initialize() also
accepts {signal} with throwIfAborted() at each await seam.
- Post-dream compaction in DreamScheduler — threshold-gated compact()
against patterns.rvf and brain.rvf so append-only growth has a
steady-state janitor.
- Boot-time size guard in getSharedRvfAdapter() for one-shot recovery
of installations already in the bad state.
- Bounded SQLite→RVF backfill (top-1000 by last_used_at) so
`: > patterns.rvf` for disk recovery doesn't trigger an unbounded
replay (the "regrew to 59 GB" pathology).
- RvfDualWriter.compact() for brain.rvf parity.
Three operator env knobs added (no rebuild needed):
- AQE_RVF_SIZE_GUARD_BYTES (default 256 MB)
- AQE_RVF_DEAD_RATIO_THRESHOLD (default 0.30)
- AQE_RVF_BACKFILL_LIMIT (default 1000)
29 new unit tests. Performance gate 15/15 pass. test:ci 4035 pass
(3 pre-existing failures, verified identical on main).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Six user-reported MCP fixes (see CHANGELOG and docs/releases/v3.9.30.md):
- #467 __dirname ESM polyfill across 7 source files
- #469 real all-MiniLM-L6-v2 embeddings (was random-vector mock)
- #470 NaN guard in coherence_collapse risk categorization
- #472 default test phases when MCP wrapper omits the list
- #473 wire SessionOperationCache into MCP tools/call dispatch
- #474 validate language and auto-derive framework in test_generate
Also fixes a pre-existing field-name typo in
tests/integrations/agentic-flow/e2e-integration.test.ts (.similarity →
.score) that was vacuously passing because the previous mock embedding
generator rarely cleared the threshold to exercise the assertion.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Hotfix for v3.9.28 — bridge payload shape (#482 round 3)
After v3.9.28 fixed the subscriber-loading order, handlers fired for
every bridge event but immediately short-circuited because
event.payload.experience was undefined. The bridge had been publishing a
flat payload { experienceId, domain, agent, task, ... } that didn't
match handleExperienceCaptured's destructure.
Fix: bridge now publishes the canonical nested
{ experience: TaskExperience-shaped, reward } shape that matches the
existing publisher in src/learning/experience-capture.ts:1043.
Also dropped the domain-specific fan-out events (test-execution.TestRunCompleted,
coverage-analysis.CoverageGapDetected, etc.) — their handlers expect
domain-event-specific fields (runId, passed, failed, gapId, riskScore)
that hooks don't capture, so publishing them with undefined fields
recorded degenerate experiences. Universal learning.ExperienceCaptured
is now the bridge's sole publication.
Integration test now uses requirements-validation domain (per Jordi's
suggestion) so the universal-event path is the sole thing being tested,
not masked by a fan-out side effect.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Hotfix for v3.9.27 — bridge subscriber wiring (#482)
The CapturedExperienceBridge shipped in v3.9.27 published events into a
kernel with no subscribers because lazyLoading: true left domain plugins
un-initialized when the bridge's first drain ran. Cursor advanced past
the rows; events were silently lost.
Fix: kernel.initialize() now eagerly loads all enabled domain plugins
before starting the bridge. CLI commands opt out via the new
enableExperienceBridge: false flag to keep their fast startup. Added
end-to-end integration test (real kernel + real plugin + real drain →
asserts kv_store has learning:experience:* key).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- AbortSignal threading bounds ReasoningBank.initialize() (#478)
prevents leaked init from runaway patterns.rvf writes
- CapturedExperienceBridge wires hook activity to domain plugins (#479)
via kernel-side drain of captured_experiences
- Removes dead v3-qe-bridge.sh, two stale planning docs, five orphan
workers in qe-workers.ts (subsumed by LearningConsolidationWorker)
Net: +430 / -2,090 lines
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Ten learning-pipeline fixes (#453, #454, #455, #456, #460, #461, #462,
#463, #464, #465) plus dup closures (#457-#459). Tightens the #465
sentinel sweep with the same task_json discriminator the LIMIT-1 close
uses, preserving the #451 ownership split (route sentinels owned by
post-route, pre-task sentinels by post-task). Updates the regression
suite to reflect the new sweep behavior.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(hooks): post-route Stop hook closes route sentinels 1:1 (#451)
The `route` hook fires on UserPromptSubmit and writes a sentinel row to
routing_outcomes with quality_score=-1, expecting `post-task` to close
it. But `post-task` only fires on PostToolUse ^(Task|Agent)$, which is
rare in direct-work sessions (Bash/Edit/Read without spawning sub-
agents). Sentinels accumulated indefinitely — Jordi reported 123+
unresolved rows per session. Stream D never converged.
Changes:
1. New `aqe hooks post-route` subcommand
(src/cli/commands/hooks-handlers/routing-hooks.ts)
Closes the most-recent unresolved route sentinel using a
`task_json NOT LIKE '%"taskId"%'` discriminator that isolates
route entries from pre-task entries. Quality formula collapses the
6-dim outcome formula for unknown duration:
0.325 + 0.25·success + 0.10·1.0
= 0.675 (success) / 0.425 (failure)
2. Inverse discriminator on updateRoutingOutcomeQuality
(src/cli/commands/hooks-handlers/hooks-dream-learning.ts)
Added `task_json LIKE '%"taskId"%'` to the WHERE clause so
post-task only ever closes pre-task sentinels. Without this,
post-task could grab a stale route sentinel by accident and
miscredit duration to a row that wasn't about a Task() call.
Symmetric counterpart to the post-route filter.
3. Stop-hook wiring in both init template paths
(src/init/phases/07-hooks.ts + src/init/init-wizard-hooks.ts)
Added `npx agentic-qe hooks post-route --success true --json`
to the Stop block. Timeout 5000ms, continueOnError: true, so a
failed post-route cannot block session teardown.
4. CLI help text (src/cli/commands/hooks.ts)
Added example for the new subcommand.
5. Regression test (tests/unit/cli/commands/post-route.test.ts)
4 tests using in-memory better-sqlite3:
- Closes most-recent route sentinel; leaves pre-task sentinels
and resolved rows untouched
- quality_score = 0.425 when --success false
- No-op when only pre-task sentinels exist (discriminator works)
- No-op on second invocation when only one sentinel was outstanding
Bug-introduce verification: stashed routing-hooks.ts, ran new test —
all 4 cases failed (Commander rejects unknown subcommand). Restored
fix, all 4 pass. Test distinguishes pre-fix and post-fix states.
Co-authored-by: Jordi Izquierdo <noreply@github.com>
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* chore(release): bump version to v3.9.25
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(hooks): post-task runs Stream B/D/F when --task-id is missing (#449)
The shipped PostToolUse hook command for Task/Agent matchers is
`npx agentic-qe hooks post-task --task-id "$TOOL_RESULT_agent_id" --success --json`
Claude Code does not populate $TOOL_RESULT_agent_id, so the flag arrives
empty and options.taskId is falsy on every real hook invocation. The
`if (options.taskId)` gate inside post-task then short-circuited the
entire Stream B/D/F learning chain:
- reasoningBank.recordOutcome — never called
- persistTaskOutcome — never called (no captured_experiences,
experience_applications, qe_trajectories, dream_insights.applied)
- updateRoutingOutcomeQuality — never called (routing_outcomes
sentinels accumulated unfilled at success=0/quality_score=-1)
- updateHookRouterQValue — never called (rl_q_values stayed empty
forever)
Same regression was fixed in v3.9.21 (patch 030) and silently came
back in v3.9.23.
Fix:
- Remove the `if (options.taskId)` gate in post-task. Use
`hook-${Date.now()}` fallback for the patternId / experience id
when the caller didn't supply one. The bridge lookup in
persistTaskOutcome keys by `ORDER BY created_at DESC LIMIT 1`
(not by taskId), so Stream F's Bellman Q-update still finds the
bridge payload it needs.
- Remove the parallel `&& options.taskId` clause from the pre-task
routing_outcomes sentinel write at line 195 — same root cause.
- Add `DO NOT REINTRODUCE` comment blocks at both sites pointing
at #449 and patch 030.
Test: tests/unit/cli/commands/task-hooks-no-taskid.test.ts mocks
hooks-shared and asserts persistTaskOutcome / updateHookRouterQValue
fire when --task-id is absent. Bug-introduce verification:
re-introducing the gate makes the no-taskId test fail with the
expected "called 1 times, but got 0 times".
Co-authored-by: Jordi Izquierdo <noreply@github.com>
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* chore(release): bump version to v3.9.24
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Fixes routing learning loop — bootstrap patterns now reach HNSW index
(#445). After this release, `patternCount` in routing hook output is
non-zero, existing installs auto-backfill `.agentic-qe/patterns.rvf` from
SQLite on next startup.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Bumps version + fleetVersion across package.json, plugin.json, and the
two skills-manifest.json copies. Adds CHANGELOG entry and release notes.
Includes one fix discovered during the release-prep e2e: stale-lock
recovery in shared-rvf-adapter's openOrCreateRvf ladder. The native
@ruvector/rvf-node binding leaves a `<file>.lock` artifact when a
process exits without calling close() — a behaviour that frequently
happens in CLI flows and that the previous always-create code path
masked. With open-or-create, those stale locks now surfaced as
`LockHeld` errors. The ladder now detects that, removes the stale
lock file, and retries open exactly once. E2E confirms RVF native
recovers cleanly across init → CLI invocation boundaries.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The shipped `aqe init` template wired the UserPromptSubmit hook to
`npx agentic-qe hooks route --task "$PROMPT" --json`, but Claude Code
never exposed `$PROMPT` as an env var — only stdin event JSON. The
shell expanded `"$PROMPT"` to `""`, so every prompt routed as empty
and `routing_outcomes.task_json` was persisted as `{"description":""}`,
silently breaking the routing learning loop.
- routing-hooks.ts: demote --task to optional, read stdin (500ms
timeout) and extract prompt/user_prompt/command/tool_input.{prompt,
description}; explicit error when neither source has content
- 07-hooks.ts and init-wizard-hooks.ts: drop --task "\$PROMPT", let
the CLI read stdin directly. isAqeHookEntry still matches both
shapes, so existing projects upgrade on `aqe init --upgrade`
- tests/unit/cli/commands/hooks-routing.test.ts: 10 unit tests
covering all event shapes, fallback ordering, malformed JSON, and
empty-field skip
Verified end-to-end: piped a Claude Code-shaped event into a fresh
project's configured route command and confirmed task_json captures
the real prompt instead of "". Backward-compat manual --task usage
preserved.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
ADR-091 (qe-browser) promoted to Implemented, trust tier 3:
- new aqe eval run --skill <skill> CLI executes skill eval YAMLs as real CommandRunner evals
- aqe init --auto on Linux ARM64 now prints an exact VIBIUM_BROWSER_PATH hint
- docs/guides/qe-browser-getting-started.md for new users
- .github/workflows/test-qe-browser.yml gates PRs with vitest + smoke + eval jobs
- qe-browser eval YAML reworked to use httpbin.org (no fixture server needed)
- Vibium pin comment corrected: ^26.3.18 is a major-line pin, not major.minor
No breaking changes. CLI surface additions only.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Security + supply-chain hardening closes five P0 release blockers from
the v3.9.13 QE audit. Shipped tarball drops 52% (19.9 MB → 9.6 MB),
15 critical runtime npm vulnerabilities are eliminated, and a
command-injection path in `aqe learning repair` is closed.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
These agents were updated in .claude/agents/v3/ for ADR-093 but the
corresponding files in assets/agents/v3/ (the directory shipped to
users via aqe init) still carried stale advisor model IDs
(claude-opus-4, claude-opus-4-6). Fresh installs would have received
the old model references.
Synced files:
qe-coverage-specialist, qe-fleet-commander, qe-pentest-validator,
qe-queen-coordinator, qe-risk-assessor, qe-root-cause-analyzer,
qe-security-auditor, qe-security-scanner, qe-test-architect
Verified in a fresh project: aqe init --auto now writes agent files
with claude-opus-4.7 / claude-opus-4-7 advisor models and the ADR-093
"effort: max" frontmatter for the three cyber-pinned agents.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Ships ADR-093 Opus 4.7 migration: Sonnet 4.6 becomes the fleet-wide
default, xhigh effort level is on everywhere, Opus 4.7 is the opt-in
escalation target via MultiModelExecutor, and cyber-sensitive agents
stay pinned to Sonnet 4.6 pending Cyber Verification Program approval.
Also fixes two stale integration-test assertions that still expected
the retiring claude-haiku-3-5 ID; both now assert the current
claude-haiku-4-5 per the model registry.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
New user projects created via aqe init will now receive the post-ADR-093
model set (Opus 4.7 / Sonnet 4.6 / Haiku 4.5) instead of retiring IDs.
* src/init/kiro-installer.ts: 4 hardcoded model fallbacks updated.
claude-sonnet-4 → claude-sonnet-4-6 (default + priority=critical),
claude-opus-4 → claude-opus-4-7 (opus-detected path),
claude-haiku-4 → claude-haiku-4-5 (haiku-detected path).
Affects: runtime agent transformation, default QE agent template
(writeDefaultQEAgent).
* src/init/init-wizard-steps.ts, src/init/phases/12-verification.ts:
generated aqe.yaml advisor block comment updated to
anthropic/claude-opus-4.7 so wizard-produced configs point to the
right advisor model out of the box.
* .claude/settings.json:188-189: claudeFlow.modelPreferences
default claude-opus-4-5-20251101 → claude-opus-4-7,
routing claude-3-5-haiku-20241022 → claude-haiku-4-5-20251001.
* Skill eval YAMLs (both .claude/ and assets/ copies):
api-testing-patterns, contract-testing, database-testing,
testability-scoring — models_to_test list updated from
claude-sonnet-4 / claude-3-haiku to claude-sonnet-4-6 / claude-haiku-4-5.
* Skill eval schemas (3 copies: .claude, assets, docs):
skill-eval.schema.json enum extended with claude-opus-4-7,
claude-sonnet-4-6, claude-haiku-4-5 as accepted values. Old
entries retained for backward compat with existing eval files.
* Sample outputs (3 copies): modelUsed field updated to
claude-sonnet-4-6.
* a11y-ally SKILL.md code example (both .claude and assets):
model: 'claude-sonnet-4-20250514' → 'claude-sonnet-4-6' (this was
caught by Commit 2's sweep; landed here for audit completeness).
Verification:
* 0 tsc errors
* 392/392 init module tests pass
* npm run build: CLI + MCP bundles rebuilt clean
Not re-running aqe init against a real fixture in this commit —
minimal+non-interactive mode doesn't materialize integration files
(kiro/opencode); full wizard-mode verification deferred until after
PR opens and a maintainer runs init smoke.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Multi-model routing layer with circuit breaking, PII redaction, and
domain-specific prompts for the Queen coordinator.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Previous safeRegex relied on type + length checks, but CodeQL's
js/regex-injection taint analysis still flagged the `new RegExp(pattern)`
call because the tainted value still reached the constructor. Length
alone is not a sanitizer.
Fix: layered defense with an explicit character allowlist that CodeQL
recognizes as a sanitizer:
1. Type check (must be string)
2. Length cap (MAX_REGEX_PATTERN_LENGTH = 1024)
3. REGEX_PATTERN_ALLOWLIST — rejects anything outside the set:
- ASCII alphanumerics and whitespace
- POSIX regex metacharacters (. * + ? ^ $ ( ) [ ] { } | \ -)
- Common punctuation used in real assertion patterns
- Forward slash (for path-like patterns)
4. Try/catch around RegExp constructor for syntax errors
Control characters, high-Unicode (emoji), and binary data are rejected
before they reach `new RegExp`. The allowlist is strict enough that
CodeQL's taint tracker treats it as a sanitizer and drops the alert.
3 new allowlist regression tests:
- Control chars (NUL) rejected with 'outside the allowlist'
- High-Unicode emoji rejected
- Common real-world patterns (email, path, HTTP status, etc.) accepted
Total: 109 qe-browser tests → 112 (was 103 before the CodeQL chain).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
CodeQL flagged the `new RegExp(check.pattern)` construction in
runConsoleCheck (assert.js:180) as a js/regex-injection vulnerability.
The pattern comes from the user-supplied --checks JSON, which CodeQL
correctly treats as a taint source even though it's the user's own
command-line input (defense in depth against ReDoS via pathological
patterns like `(a+)+$`).
Fix: new safeRegex() helper validates the pattern BEFORE constructing
the RegExp:
- Rejects non-string input
- Caps at MAX_REGEX_PATTERN_LENGTH (1024 chars) to prevent ReDoS
- Try/catches the RegExp constructor so invalid syntax returns an
error instead of crashing the whole assertion pass
Applied to runConsoleCheck's console_message_matches case. The
title_matches case at line 111 builds a RegExp in the browser-side
eval script; that's the browser's regex engine, not ours, so it's
outside CodeQL's taint tracking and doesn't need the guard.
6 new regression tests in qe-browser-assert.test.ts cover:
- safeRegex exports + MAX_REGEX_PATTERN_LENGTH
- normal input returns a working RegExp
- non-string input rejected
- invalid syntax caught without throwing
- over-length patterns rejected
- exactly-at-limit accepted
Test totals: 109 qe-browser tests (was 103), all passing.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The Tier 3 validator runs per-skill in a matrix job on every PR touching
that skill. The workflow does not install external runtime dependencies,
so marking vibium as a requiredTool caused the validator's Step 1
(tool existence check) to fail for qe-browser with:
[FAIL] Required tool MISSING: vibium
This matches the pattern used by a11y-ally, which lists playwright,
axe-core, npm, and python3 as optionalTools. The runtime tools are
verified via smoke-test.sh tc001-tc011 (when vibium is on PATH) and
by the missing-vibium contract in qe-browser-unavailable-e2e.test.ts
(when vibium is NOT on PATH) — not by the static validator.
The validator's schema check, JSON handling, library self-test,
and enum validations all still run unchanged. Only the `which vibium`
existence check moves from HARD FAIL to WARN.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Ships qe-browser fleet skill (Vibium engine, ADR-091) — a ~10MB Go binary
replacement for 300MB Playwright with typed assertions, batch execution,
visual-diff, prompt-injection scanning, and semantic intents. 11 skills
migrated to the new engine.
Total skills: 84 → 85. Tier 3 verified: 49 → 50. Manifest: 1.3.0 → 1.4.0.
fleetVersion bumped to 3.9.9 in skills-manifest.json.
See CHANGELOG.md and docs/releases/v3.9.9.md for details.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
qe-browser is a new Tier 3 fleet skill (added in commits cd4ea7e4
through 9c52823e on this branch, per ADR-091). This commit propagates
its existence to the user-facing documentation and manifests that
were still carrying pre-qe-browser counts.
Count changes (AQE skills only; platform skills unchanged):
- Total QE skills: 84 → 85
- Tier 3 verified: 48 → 49
- V3 Domain skills: 23 → 24
- README.md headline: "74 QE Skills" → "75 QE Skills"
- agentic-qe-intro.md: "74 total" → "75 total", "114 defs" → "115"
Files updated:
- README.md: bumped headline count, Tier 3 count in tier table,
"View all X skills" summary, and added a new "Browser Automation (1)"
category row pointing at ADR-091
- docs/agentic-qe-intro.md: bumped "Selected QE skills (74 total)" to 75,
added "Browser" category row with /qe-browser, bumped tree "114
skill definitions" to 115
- .claude/skills/README.md + assets mirror: bumped Summary counts,
V3 Domain Skills heading, and added qe-browser as a full description
entry in the V3 Domain Skills list with a link to ADR-091
- .claude/skills/skills-manifest.json + mirror: manifest version 1.3.0
→ 1.4.0, totalSkills 48 → 49, totalQESkills 80 → 81, added new
"browser-automation" category, bumped skillBreakdown counts,
updated notes to reference ADR-091
- .claude/skills/trust-tier-manifest.json + mirror: summary.tier3
49 → 50 (this file already counted one more than skills-manifest),
total 112 → 113, validationStatus.passing 49 → 50, added qe-browser
entry to skillsByTier.tier3 list with category "browser-automation"
Historical CHANGELOG entries (v3.9.x and earlier) intentionally left
at 84 — they describe past state.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Phase 5 UAT discovered the helper scripts surfaced missing-vibium as
`actual: "eval error: vibium binary not found..."` inside a `failed`
envelope. The Fallback Policy in SKILL.md said scripts MUST return
status:"skipped" with reason "browser-engine-unavailable" — that
contract was aspirational. F1 makes it real.
Contract changes:
- lib/vibium.js: ENOENT now throws typed VibiumUnavailableError (with
stable code:"BROWSER_ENGINE_UNAVAILABLE" for cross-module duck typing)
- New helper unavailableEnvelope() returns canonical skipped envelope
with top-level vibiumUnavailable:true and output.reason field
- New runOrSkip() wraps each helper's main() and converts the typed
error into the documented envelope on the way out
- New rethrowIfUnavailable() helper used inside per-script catch blocks
so missing-vibium bubbles past local "convert exception to fail()"
handlers — assert.runConsoleCheck/runNetworkCheck/runBrowserSideCheck,
batch's per-step catch, check-injection main, intent-score main,
visual-diff main all updated
- emit() now returns exit code 2 for status:"skipped" (was 1)
- Exit code contract documented: 0=success, 1=failed, 2=skipped
Downstream skills can now branch on result.vibiumUnavailable === true
or exit code === 2 instead of grepping the actual string for
"vibium binary not found".
Tests: 103 (was 86). New file qe-browser-unavailable-e2e.test.ts spawns
each of the 5 helpers under PATH=/tmp/fake-bin (node only, no vibium)
and asserts exit 2 + skipped envelope shape. vibium-lib test grew from
10 to 22 cases including VibiumUnavailableError, runOrSkip, exit codes.
Smoke test: 10/10 PASS (was 9/9). New tc011 builds a fake bin dir
inside the smoke test work dir and verifies the contract end-to-end
against the actual installed scripts.
SKILL.md updated:
- Output Contract section now documents all 3 statuses with shapes
and exit codes
- Fallback Policy replaced generic prose with concrete bash + Node
snippets that branch on the structured field
ADR-091 Phase 6 documents the change and verification.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
M1 visual-diff threshold: documented "max difference fraction" convention
matching Playwright maxDiffPixelRatio (was inconsistent in docs/code).
M2 fixture server bind: default to 127.0.0.1 instead of 0.0.0.0;
QE_BROWSER_FIXTURE_HOST=0.0.0.0 opt-in for explicit external access.
Codespaces auto-forwards 0.0.0.0 ports — this was an unintentional open
server every time the eval ran.
M3 path-traversal guard: replaced startsWith() with path.relative() +
'..'/absolute check. The old guard false-passed sibling-prefix dirs
(/foo vs /foo2) and was fragile on Windows mixed separators.
M4 ANSI escape injection: added sanitizeSnippet() that strips C0 controls
(0x00-0x1F except \t\n) and DEL (0x7F) from every check-injection
finding snippet. Prevents `cat findings.json` from triggering terminal
control sequences embedded in malicious page content.
M5 parseArgs --key=value: split on first '=' so --threshold=0.05 works
alongside --threshold 0.05. Only the first '=' is split so URL/base64
values survive intact.
M6 batch.js pre-validation: added validateAllSteps() that walks every
step's required fields BEFORE the first vibium call. Step 17 typo now
aborts before steps 1-16 execute their side effects.
M7 intent-score bare-x regex: anchored \bx\b in close_dialog scorer so
"fix", "exit", "extra", "sixteen" no longer match. Unicode × and ✕
unchanged.
M8 check-injection --exclude-selector: new flag drops subtrees from
the scan. Page text is read from a CLONE of <body> with matching
elements removed — live page unchanged. Lets docs sites scan their
own pages without self-flagging documentation about prompt injection.
M9 09-assets vibium catch: wrapped raw error with actionable recovery
guidance (`npm install -g vibium`, then re-run `aqe init`).
Hidden bug found and fixed during M2/M3 testing: fixtures/serve-skills.js
existed but had never been run because the repo root is ESM and the file
uses CommonJS require(). Added scoped fixtures/package.json with
"type": "commonjs" matching the existing scripts/package.json pattern.
Tests: 86 unit tests across 7 files (was 50). All pass.
Smoke test: 9/9 still passing against real Vibium v26.3.18.
Mirrored .claude/skills/qe-browser → assets/skills/qe-browser.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Add Platform support table to SKILL.md (mac-arm64/mac-x64/linux-x86_64
verified, Linux ARM64 needs documented chromium-symlink workaround)
- Add Headless mode section explaining why helpers auto-inject --headless
- Document the 7 gotchas Phase 3 found, in references/migration-from-playwright:
1. Vibium defaults to visible browser
2. screenshot -o ignores directory
3. screenshot --selector flag does not exist in v26.3.x
4. eval --stdin returns last expression value, not console.log
5. Linux ARM64 chromium-symlink workaround
6. visual-diff needs explicit viewport for determinism
7. npm install -g vibium can take 1-3 minutes
- Add .aqe/ to .gitignore so visual-baselines and helper local state
don't leak into commits
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
User pushed back on Phase 3 attempt 1 ("blocked on linux-arm64"). After
re-verifying upstream — Vibium 26.3.18 IS the latest on npm, Chrome for
Testing across Stable/Beta/Dev/Canary still ships only linux64 (x86_64),
mac-arm64, mac-x64, win32, win64 — installed Debian's native ARM64
chromium + chromium-driver via apt and symlinked them into Vibium's
cache. Then ran the smoke test for real and found four real bugs in
the qe-browser helper scripts that would have shipped broken.
Bugs found and fixed:
1. Eval contract was wrong. lib/vibium.js's vibiumEvalStdin sent
`console.log(...)` expecting Vibium to capture it. Vibium doesn't —
`eval` returns the LAST EXPRESSION's value as
`{"ok":true,"result":"<stringified>"}`. Fix: dropped console.log
wrappers from assert.js/intent-score.js/check-injection.js; added
unwrapEvalResult() in lib/vibium.js that parses the envelope and
JSON-decodes the result string back into our object.
2. lib/vibium.js did not inject --headless. Vibium defaults to visible
browser, which fails in headless containers with "Missing X server
or $DISPLAY". Fix: lib/vibium.js prepends --headless unless
QE_BROWSER_HEADED=1 is set in the env.
3. `vibium screenshot -o /abs/path/foo.png` ignores the directory and
saves to `~/Pictures/Vibium/foo.png`. Verified live. Fix:
visual-diff.js passes just the basename, reads from Vibium's actual
output dir, copies to the requested path, unlinks the source.
4. `vibium screenshot --selector` flag does NOT exist in v26.3.x.
Verified via `vibium screenshot --help`. Fix: visual-diff.js throws
a clear error with remediation guidance if a caller passes
--selector. (B3's "fall back to eval-clip" comment is now removed.)
Plus a real environment-flakiness fix: smoke-test.sh sets
`vibium viewport 1280 720` before tc006/tc007 visual-diff captures
because httpbin.org/html renders at non-deterministic dimensions
(765x672 vs 780x654 observed) when no viewport is set, causing
spurious pixel-diff failures.
Verbatim smoke test result on this codespace:
Smoke testing against vibium v26.3.18
PASS tc001 url_contains on httpbin form
PASS tc002 selector_visible h1 on httpbin /html
PASS tc003 failing assertion exits 1
PASS tc004 batch 3-step happy path
PASS tc005 batch stops on first failure
PASS tc006 visual-diff baseline created
PASS tc007 visual-diff second run matches
PASS tc008 check-injection clean page
PASS tc010 intent-score submit_form on httpbin form
PASS: 9 FAIL: 0 SKIPPED: 0
Unit tests still pass: 50/50 (one test had to be updated — it asserted
`buildScript` includes 'console.log', which is now correctly NOT in the
script after the eval-contract fix).
Phase 3 verification COMPLETE per ADR-091. The qe-browser helpers work
end-to-end against real Vibium + real Chromium + pinned httpbin.org
fixtures. The Linux ARM64 chromium-symlink workaround is documented in
ADR-091; user-facing docs in references/migration-from-playwright.md
still need updating (marked as Phase 4 follow-up).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
H1: detectVibium reads stderr too, extracts semver
- Many Go CLIs (including some Vibium versions) write --version to stderr.
detectVibium used to read only stdout and silently report 'unknown',
polluting the init log with "vibium unknown (already installed)".
- Fix: read both streams, prefer stdout, extract bare semver via regex
so output is consistent regardless of where the binary printed it.
- Tests: 5 new regression tests covering stderr fallback, stdout
preference, verbose-format extraction, prerelease tags, fallback to
first whitespace token.
- Note: result.version is now bare semver (26.3.18) not v-prefixed.
Two pre-existing tests updated to match the new contract.
H2: instructions_in_html_comment dead-code regex
- TreeWalker(SHOW_COMMENT).textContent yields the inner text without
the <!-- --> delimiters, so the pattern that requires the literal
`<!--` prefix could never fire. Eval tc009 was passing only by
accident (ignore_previous_instructions fired on the same fixture).
- Fix: re-wrap each comment node in `<!-- ... -->` before adding to
the hidden bucket. Pattern can now actually match.
- Tests: 2 new regression tests asserting wrapped form fires AND
unwrapped form does NOT fire (the bug condition).
H4: e2e-flow-verifier frontmatter regression — INVALID FINDING
- Verified against `git show main:` — the skill always had minimal
frontmatter on main (just name + description + user-invocable: true)
and has no evals/, schemas/, or scripts/ directories. It is a
user-invocable runbook-style skill, not trust-tier-3 validated.
No metadata to restore. Marked closed-as-invalid.
H6: log-before-spawn for the 3-minute Vibium install
- spawnSync(npm install -g vibium) blocks the event loop for up to
3 minutes on a cold cache while Chrome for Testing downloads. The
user saw zero output and would Ctrl-C thinking init hung.
- Fix: pre-flight detect call (5s timeout) short-circuits when vibium
is already installed (the common case, no banner). When the install
is actually about to run, log a clear "this can take 1-3 minutes,
downloads Chrome for Testing lazily" message BEFORE the spawn.
H7: scoped scripts/package.json survives npm pack — VERIFIED
- `npm pack --dry-run` confirms both
.claude/skills/qe-browser/scripts/package.json (225B) AND
assets/skills/qe-browser/scripts/package.json (225B)
are included in the published tarball. The CommonJS override that
keeps the helper scripts loadable from an ESM project root will
survive `npm install` after publication. No fix needed.
H8: eval yaml uses unsupported runner fields
- The structured assertion fields I invented (exit_code, json_fields,
severity_at_least, candidate_count_at_least) are NOT understood by
src/validation/parallel-eval-runner.ts which expects must_contain
keyword matching. The yaml as written is unrunnable.
- Honest fix: relabel evals/qe-browser.yaml as `status: design-spec`
with a HONESTY NOTE explaining it's a specification, not a runnable
eval. Two paths to make it runnable documented (extend runner OR
rewrite as must_contain).
- Parallel artifact: scripts/smoke-test.sh — a runnable bash smoke
test against the same pinned httpbin.org fixtures. THIS is the
script that gates PR-reopen per ADR-091 Phase 3.
Bonus from B2/H5 already in Phase 1:
- assert.js now uses ?? instead of || for the `expected` field
Verification:
- 50/50 qe-browser tests pass (was 36 in initial PR; +14 regressions)
- 392/392 init tests pass (was 387; +5 from H1)
- npm run build: clean
- npm pack --dry-run: scoped package.json confirmed in tarball
- eslint on all touched source files: clean
- bash -n smoke-test.sh: syntax OK
Phase 3 (real Vibium smoke test) still gating PR reopen per ADR-091.
Branch intentionally not pushed.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Three correctness blockers identified by qe-devils-advocate review of the
initial qe-browser implementation (PR #420, closed without merge):
B1: intent-score.js String.replace special-char corruption
- SCORER_JS.replace('__SCOPE__', JSON.stringify(scope)) interpreted $&,
$`, $', $1-$9 in the replacement string. Scopes like
div[data-x="$amount"] silently corrupted the generated script.
- Fix: split('__TOKEN__').join(value) for literal substitution.
- Tests: 4 new regression tests covering $&, $', $1, intent placeholder.
B2: assert.js fail-OPEN on console/network telemetry errors
- runConsoleCheck/runNetworkCheck used to return ok:true when
`vibium console --json` failed for any reason. no_console_errors
silently reported pass when telemetry was unavailable.
- Fix: new unavailable() sentinel returns ok:false + unavailable:true.
runCheck surfaces it via a new `unavailable` field. Summary
distinguishes "assertion failed" from "telemetry unavailable".
- Bonus (H5): expected-field uses ?? not || so element_count {count: 0}
reports `expected: 0` instead of `null`.
- Tests: 2 new regression tests on the sentinel + element_count zero.
B3: visual-diff.js dead fallback comment
- Source comment promised "fall back to eval-clip if not present" but
no fallback existed. Direct violation of feedback_no_unverified_failure_modes.
- Fix: comment removed; specific error handler added for unsupported
--selector flag with clear remediation message. No fake fallback —
silently substituting full-page would produce wrong baselines.
Plus: ADR-091 documents the Vibium-as-engine decision with full
rejected-alternatives section (dev-browser, gsd-browser, Playwright MCP,
status quo, build-from-scratch), four implementation phases, and a risks
table tracking the 11 devil's-advocate findings with status. Phase 3
(real Vibium install + smoke test) is explicitly gating PR reopen.
Verification:
- 420/420 tests pass (387 init + 33 qe-browser, 6 new regression tests)
- npm run build: clean
- eslint on touched files: clean
- All 6 helper scripts pass node --check
Branch is intentionally not pushed and PR is intentionally not reopened
until Phase 2 (H1, H2, H4, H6, H7, H8) and Phase 3 (real Vibium smoke
test) are complete.
Refs: docs/implementation/adrs/ADR-091-qe-browser-skill-vibium-engine.md
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Introduces a new AQE-owned browser automation skill layered over Vibium
(WebDriver BiDi, single 10MB Go binary) and migrates 11 browser-using QE
skills to use it instead of raw Playwright/Puppeteer.
qe-browser ships five helper scripts that add QE primitives Vibium doesn't
cover on its own: typed assertions (16 check kinds), batch execution,
pixel-diff against stored baselines, prompt-injection scanning, and
semantic intent scoring ported from gsd-browser (MIT/Apache). Each script
emits a structured JSON envelope matching schemas/output.json so hooks
and CI gates can parse results without stdout grep.
Vibium installs automatically during `aqe init` via a new graceful
browser-engine installer that shells out to `npm install -g vibium` and
reports `already-installed`, `installed`, `install-failed`, or
`npm-unavailable` without ever failing init.
Skills migrated to reference qe-browser:
a11y-ally, e2e-flow-verifier, qe-visual-accessibility,
security-visual-testing, visual-testing-advanced, testability-scoring,
compatibility-testing, accessibility-testing, localization-testing,
observability-testing-patterns, enterprise-integration-testing.
Ships 36 new unit tests (browser-engine-installer, assert, intent-score,
check-injection) with dependency-injected spawners to avoid touching real
processes. All 387 existing init tests still pass.
Eval harness uses pinned public fixtures (httpbin.org, Vibium docs at a
pinned tag) plus a local static server that wraps this repo's own skill
docs, per the "no synthetic fixtures" rule.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Integrate tree-sitter WASM as the default parser for 5 languages, replacing
regex-only parsing (~97-98%) with AST-based parsing (~100% accuracy).
- Add TreeSitterWASMParser implementing ILanguageParser for 5 languages
- Bundle only needed WASM grammars in assets/grammars/ (8.5MB vs 50MB)
- Add FallbackParser with 3-retry logic before permanent regex degradation
- Add per-call parser instances to prevent concurrency races across await
- Add cyclomatic complexity computation from AST branching nodes
- Add property extraction from class/struct AST nodes
- Add AQE_PARSER_REGEX_ONLY=1 env var for opt-out to regex
- Replace require() with ESM-compatible dynamic import() in registry
- Mark web-tree-sitter as external in CLI/MCP esbuild configs
- Amend ADR-076 with decision rationale and alternatives evaluated
Alternatives evaluated and rejected:
- @ast-grep/napi: npm package only ships JS/TS/CSS/HTML, others need native .so
- node-tree-sitter: requires native C compilation via node-gyp
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Based on Anthropic's "Lessons from Building Claude Code: How We Use Skills" article.
Transforms QE skills from flat knowledge cards into folder-based systems with
gotchas, progressive disclosure, composition, config, hooks, and run history.
Changes across all 4 phases:
- Add Gotchas sections to 30 skills with battle-tested failure data from Nagual/memory.db
- Rewrite all 84 skill descriptions as "Use when..." trigger conditions
- Strip 892 lines of textbook knowledge from 15 skills (26% reduction)
- Create 10 reference/template files across 7 skills (OWASP, k6, mutation operators, etc.)
- Add config.json with _setupPrompt to 7 skills
- Add run-history.json with write instructions to 5 metric-producing skills
- Create 5 on-demand hook skills with executable scripts (strict-tdd, no-skip, coverage-guard, freeze-tests, security-watch)
- Create 5 new category-filling skills (test-failure-investigator, coverage-drop-investigator, e2e-flow-verifier, test-metrics-dashboard, skill-stats)
- Add Skill Composition cross-references to 10 key skills
- Remove 3 redundant/obsolete skills (qe-contract-testing → contract-testing, qe-security-compliance → security-testing, aqe-v2-v3-migration)
- Add DEPRECATED_SKILLS cleanup to skills-installer.ts for --upgrade
- Update all manifests (trust-tier, TRUST-TIERS.md, README.md, CLAUDE.md)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Dragan Spiridonov