Evolve the QE framework from PACT (Proactive, Autonomous, Collaborative,
Targeted) to PACTS by adding a fifth principle: Structured — governance,
observability, and explainability of agent behavior. We measure confidence,
not trust (trust is a human feeling; confidence can be mathematically
explained). Especially important in regulated industries. Inspired by DORA's
research on AI-assisted delivery and platform-engineering quality measurement.
Anti-pattern: bolting agentic AI onto chaos.
Scope:
- Source skills (.claude/skills): holistic-testing-pact + agentic-quality-
engineering SKILL.md, agentic-quality-engineering schema (added optional
`structured` dimension, kept slug keys), README, skills-manifest.json
- Regenerated derived artifacts: assets/skills (via prepare-assets equivalent
copy) and the two affected .opencode yaml (via generate-opencode-skills)
- Hand-maintained trees updated consistently: .kiro skills, docs (catalog,
glossary, intro), plugin/marketplace manifests, qe-learning-coordinator
Stability:
- Kept skill id/dir `holistic-testing-pact` (and `.kiro` `qe-holistic-testing-pact`)
unchanged — referenced across other repos. Only titles/descriptions/content
changed to PACTS.
Attribution: PACT originated with Reuven Cohen (Agentics Foundation); adapted
to QE by Dragan Spiridonov. PACTS adds Structured.
Co-Authored-By: claude-flow & Agentic QE fleet
Free-tier local test generation (opt-in) — cheap-first on local/free models
with a repair loop, auto-escalation to paid tiers, and self-learning routing
feedback. Plus #541 dependency fixes (ruvllm optional deps + hono security).
Version synced across all product manifests: package.json, fleetVersion in
both .claude/ and assets/ skills-manifests, and .claude-plugin/plugin.json
(the latter two had drifted at 3.10.4).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Self-audit of the free-tier lane found two gaps:
- Multi-file: the free-tier path generated for sourceFiles[0] only, silently
skipping the rest AND the paid path. Now it engages only for single-file
requests; multi-file falls back to the normal path so every file is covered.
(+1 regression test.)
- Exports/docs mismatch: the guide + README showed
`import ... from 'agentic-qe/routing/free-tier'`, but package.json had no
./routing export — the import would fail for npm consumers. Added the
./routing/free-tier subpath export.
Verified: project `tsc --noEmit` clean; 236 routing/coordinator/plugin tests +
18 routing-feedback tests green; no sibling breakage from the generic tracker.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Completes the cheap-first economics: local → repair → escalate the hard tail.
- coordinator: build a claudeRunner from the existing HybridRouter (tier →
complexity hint) and run the free-tier path with escalate:true. Falls back to
local-only automatically when no router is wired. patternsUsed now records the
winning tier.
- plugin (kernel layer): onInitialize constructs a live RoutingFeedbackCollector
(EMA calibration + auto-escalation, memory-only fallback) when the free tier is
opted in, and injects it as the coordinator's routingFeedback sink — so each
cheap-vs-escalated outcome drives the calibrator/escalation tracker/confidence
(lifts the stuck-at-40% routing metric). Built only when opted in.
- tests: +2 (escalation via router, D9 sink invocation); 113 green; existing
coordinator (47) and plugin (38) suites unchanged. Type-safe (MessageRole /
TaskComplexity / RoutingConfig fields all confirmed).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Apply the free-tier escalation work to the test-generation coordinator as an
OPT-IN, default-off capability (Ruv's cheap-first economics). Zero impact unless
enabled via `enableFreeTier` config or `AQE_FREE_TIER=1`.
- coordinator: guarded `tryFreeTierGeneration()` tries the free local model first
(reads source, prompts, verifies test+assertion), falls through to the
unchanged paid path on any miss. 47 existing coordinator tests still green.
New optional `routingFeedback?` ctor param wires D9.
- D8 repair loop: executor gains same-tier repair (`repairAttempts`, verifier
feedback fed back) + `escalate:false` repair-only mode. Coordinator path runs
local-only + repair, no paid escalation yet.
- D9 sink: createRoutingFeedbackSink() maps executor outcomes onto the existing
RoutingFeedbackCollector (calibrator + escalation + confidence); cheap wins
raise the cheap tier's confidence, lifting the stuck-at-40% routing metric.
- docs: user guide docs/guides/free-tier-local-models.md (local/cloud Ollama,
OpenRouter, OpenAI-compatible) linked from README.
- tests: +11 new (executor repair, sink, coordinator opt-in); 111 green total;
free-tier modules strict-tsc clean.
Model: qwen3:8b default. Keys read from named env vars, never stored.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Turns the dormant free-tier ladder into a running loop implementing Ruv's
SWE-bench economics for a QE task: run on the cheap local tier, verify with an
objective oracle, escalate this task up the ladder on failure, and record the
start-tier verdict so the base tier adapts across tasks.
- executor.ts: decoupled by injection — free tiers via freeTierChat, Claude
tiers via an injected ClaudeTierRunner (no Anthropic SDK dep; coordinator
delegates to its existing HybridRouter). Local-only mode when no runner.
onOutcome sink = the D9 seam into routing-feedback.
- tests: 8 new (53/53 green across free-tier + tracker), strict-tsc clean.
- prototype/d7-wire-proof.mjs: LIVE — qwen3:8b generated a correct vitest test
that passed the objective verifier in 8.65s ($0, no escalation); an
impossible verify climbed local->haiku->sonnet->opus and failed at the top.
Model: qwen3:8b (most productive worker in D0; 5.2GB fits the 8GB story).
Additive only — no shipped coordinator hot-path edited. Adoption is a one-call
snippet documented in docs/metaharness/06 (behind a flag; needs sign-off).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Ruv's SWE-bench Round-3 economics for AQE: cheap-local handles the bulk,
escalate only failures up the existing Claude tiers. Maps onto AQE's
existing AutoEscalationTracker, which previously bottomed out at haiku (paid).
- escalation/auto-escalation-tracker.ts: made generic over the tier-name type
with a configurable tierOrder (default resolved in ctor — DEFAULT_ESCALATION_
CONFIG unchanged, so the exact-match test stays green). 26/26 original tests
pass; zero behaviour change for existing callers.
- routing/free-tier/: configurable provider layer — local-ollama / cloud-ollama
/ openrouter / openai-compatible. Keys read from a named ENV VAR, never
stored. freeTierChat() is OpenAI-compatible + reasoning-model aware + never
throws (transport error = tier failure → escalate). defaultFreeTierLadder()
= local -> haiku -> sonnet -> opus.
- tests: 19 new (45/45 green with the tracker suite), strict-tsc clean.
- prototype/d7-proof.mjs: live proof vs M5 host Ollama — free local tier
answers ($0), escalates local->opus on failures, de-escalates back to local.
Dormant by design: not yet wired into a coordinator (next: D7-wire + D8 repair
loop + D9 feed outcomes into routing-feedback).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Cross-pollination with Ruv's MetaHarness Darwin Mode: evolve the harness
around a frozen cheap local model, graded by AQE's objective QE metrics.
- docs/metaharness/06-...: GOAP action lane + ADR-style decision (→ ADR-111)
- A1: real mcp-scan snapshot of AQE's MCP surface (.harness/mcp-policy.json
mirror) → 2 HIGH (default-deny OFF, shell granted), 6 MED, 2 LOW
- D0: local-mutator spike benchmark on host M5 Ollama (qwen3:8b/30b > gemma4
as mutator; mock+LLM gives no fitness signal — documented)
- D1: src/integrations/darwin — QE→Darwin ScoreCard adapter folding ADR-104
arena metrics (kill-rate/coverage/cost) + ADR-072 promotion-gate mirror
(15/15 tests green, strict-tsc clean, zero coupling to @metaharness/darwin)
- D6: prototyped + proved a `customScore` hook in Darwin's evolve loop
(patch in prototype/, Ruv's tree left pristine). QE-native scoring now
DIFFERENTIATES variants: winner +0.1230 over baseline, 3 promotions
(vs flat 0.765 from the mock substrate)
Two upstream PRs to ruvnet/agent-harness-generator are staged but unsent:
the customScore hook + --ruvllm-timeout/max-tokens CLI flags.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Dependabot's hono 4.12.23→4.12.25 bump (#541) regenerated the lockfile
and pruned the 5 optional @ruvector/ruvllm-* native platform packages
(darwin-arm64/x64, linux-arm64/x64-gnu, win32-x64-msvc), which breaks
`npm ci` (EUSAGE) for consumers — the same class of failure as #540.
This lockfile = main's lockfile with ONLY hono bumped to 4.12.25
(registry-verified integrity), optional platform deps preserved.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The v3.10.9 publish failed: the "Integration tests on tag SHA" job runs
test:integration:fast (which now includes the #528/#535 MCP integration tests
that drive the BUILT bundles), but the job never downloaded the dist artifact
from the build job. With CI=true the tests' guard refuses to silently skip, so
they hard-failed on a missing bundle. Add the download-artifact step (like the
init-gate and consumer-audit jobs already do) so the bundles are present.
No source/package change — re-releasing v3.10.9 (never published to npm).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
MCP server reliability + database-free completeness (community issues from
@nagoodman): in-process `agentic-qe mcp` (#528), memory_delete stale-cache fix
(#535), db-free propagation to all editor MCP configs (#533), and the opt-in
--no-claude exclusive install (#532). Plus the js-yaml dev bump (#540) and the
devil's-advocate follow-up fixes (routing/merge/startup + CI guard).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The #528 and #535 integration tests (tests/integration/mcp, gated on
describe.skipIf(!BUILT)) never executed in CI — a false green:
- the mcp-integration-tests job ran `npm ci` but no `npm run build`, so the
dist bundles were absent and both tests silently skipped;
- the step was `continue-on-error: true`, so even a real failure passed;
- neither test was in the release gate's test:integration:fast list.
Fixes:
- mcp-tools-test.yml: add `npm run build` before test:mcp:integration and drop
continue-on-error so the job is a real gate (the exit-124 hang tolerance
stays for the known vitest-hang flake).
- both tests: skip only for LOCAL unbuilt checkouts (`!BUILT && !process.env.CI`)
— in CI a missing build now fails loudly instead of skipping. Verified:
CI=1 with the bundle removed fails the suite instead of reporting green.
- package.json: add both tests to test:integration:fast so they also run on the
release gate (npm-publish.yml), which builds first.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Devil's-advocate review of today's branch surfaced three real gaps:
C1 (init-handler.ts): bare `aqe init --no-claude` (no --with-<platform>, no
--no-database) fell through to runStandardInit and installed the FULL Claude
surface it was meant to suppress — `noClaude` was only honored inside the
modular path. Add `options.claude === false` to the modular routing guard.
Verified: `aqe init --no-claude --minimal` now writes no .claude/, .mcp.json,
or CLAUDE.md (and the empty-install warning fires).
C2 (continuedev-installer.ts): the mergeExistingYamlConfig branch (existing
`mcpServers:` + --overwrite) appended a hardcoded AQE_MEMORY_PATH entry,
ignoring memoryBackend — re-introducing the #533 db leak in db-free installs
over an existing Continue config. Honor memoryBackend on this path too. Adds a
regression test.
H3 (entry.ts): main() ran fire-and-forget, so a startup rejection that escaped
its internal try hit the unhandledRejection handler (which intentionally keeps
the server alive at runtime) → on the in-process `agentic-qe mcp` path the CLI
wrapper would hang on its never-resolving promise with a zero exit. main() now
`.catch`es and exits non-zero on a failed startup. Re-verified #528/#535
integration tests still green (normal startup unaffected).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
#533 — database-free mode now propagates AQE_MEMORY_BACKEND=memory to ALL
generated MCP-server configs, not just opencode.json. Previously `aqe init
--no-database` left .mcp.json (Claude Code) and every editor config pointed at
.agentic-qe/memory.db, so launching the server via any non-OpenCode client
recreated the db at runtime and defeated the db-free install.
- platform-config-generator.ts: generateMcpConfig(platformId, { memoryBackend })
emits AQE_MEMORY_BACKEND=memory and drops AQE_MEMORY_PATH for JSON/TOML/YAML;
covers the 8 shared-generator installers (cursor/cline/continuedev/kilocode/
roocode/windsurf/codex/copilot).
- 08-mcp.ts (.mcp.json) and kiro-installer.ts: same db-free env; learning/workers
not forced on in db-free mode (they persist to SQLite).
- 09-assets.ts threads memoryBackend into every platform installer.
Default (persistent) output is byte-for-byte unchanged.
#532 — new opt-in `--no-claude` suppresses the default Claude Code surface
(.claude/ skills+agents, .mcp.json, CLAUDE.md, governance, hooks) so
`--with-<platform>` flags become the only install targets — e.g. a clean
OpenCode-only install. Gated via shouldRun/run on phases 07-hooks, 08-mcp,
09-assets (core assets), 11-claude-md, 13-governance. Default install
(no flag) is unchanged; warns if --no-claude is passed with no platform.
Verified by real `aqe init` runs in temp dirs:
- `--no-database --with-codex --with-cursor`: .mcp.json/.cursor/.codex all emit
AQE_MEMORY_BACKEND=memory; no memory.db created.
- `--no-claude --with-opencode --no-database`: .opencode present; .claude/,
.mcp.json, CLAUDE.md all absent.
- default `--with-cursor`: full surface + persistent env intact (no regression).
Adds db-free + no-claude cases to platform-config-generator and mcp-phase tests;
all 406 init unit tests pass.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
memory_delete returned {deleted:true} but an immediate memory_retrieve still
returned the value (with an identical timestamp). Root cause was NOT the delete
— it works. memory_retrieve is isConcurrencySafe, so the Issue-#473 session
cache stored its result; memory_delete never invalidated it, so the follow-up
read served the stale cached "found" entry (the identical timestamp was the
cache fingerprint). Confirmed by reproduction: with AQE_SESSION_CACHE=off the
post-delete read already returned found:false.
Fix: mutating tools now invalidate their domain's cached reads. Adds
SessionOperationCache.invalidateDomain() (clears in-memory + persisted rows);
the protocol server calls it after any successful tool whose name ends in a
mutation verb (_store/_delete/_set/_share/_update/_remove/_clear/_promote/
_cleanup). Generalizes to other write/read pairs (e.g. cross_phase_store).
Verified end-to-end through the protocol server with the cache ENABLED:
store -> retrieve(found) -> delete -> retrieve(found:false). Adds
tests/integration/mcp/memory-delete-cache-invalidation.test.ts.
Note: the other findings in #535 (cross_phase_stats counter, GOAP
maxSteps/plan-lookup/activeAgents, test_generate_enhanced quality,
qe_coherence_consensus text) are separate root causes and remain open for
dedicated follow-up per the issue's own "split into its own issue" note.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Records the 2026-06-17 evidence: ADR-038 (DRACO ceiling) is now Accepted
upstream → the A3 gate leans G-ABORT; MetaHarness v0.1.15 ships mcp-scan --json
+ scorecard, shrinking A1 to "consume, not build"; and two new standalone
candidate actions lifted from ADR-150 (A9 graceful-degradation, A10 KRR router).
Cross-links the ADR-150 spawnSync bug class to AQE #528.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The `mcp` subcommand spawned `node dist/mcp/bundle.js` with stdio:'inherit'.
With a piped stdin (any programmatic MCP client), the grandchild received
`stdin-eof` right after answering `initialize` and the Issue #513 stdin-EOF
guard shut it down before serving `tools/list` — so `tools/list` never
returned. The dedicated `aqe-mcp` bin (which runs the bundle directly) worked.
Fix: dynamic-import the MCP entry so it runs in-process on the command's real
stdio, exactly like `aqe-mcp`. entry.ts kicks off `main()` fire-and-forget, so
the action then awaits a never-resolving promise to keep the CLI wrapper from
exiting out from under the server; entry.ts owns shutdown.
Verified with the exact issue reproduction (piped stdin + EOF): `agentic-qe mcp`
now answers initialize AND tools/list (86 tools), matching the direct bundle.
Adds tests/integration/mcp/mcp-cli-inprocess.test.ts as a regression guard.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Supersedes #536. Dependabot's regenerated lockfile dropped the
@ruvector/ruvllm-* optional platform packages, making `npm ci` fail
with EUSAGE (lockfile out of sync). Regenerated with a targeted
`npm update vite --package-lock-only` on Linux, which keeps the
optional platform entries intact.
vite 8.0.9 -> 8.0.16 (transitive dev dependency; routine bug-fix bump).
Lockfile-only change; `npm ci` sync check passes; `npm audit --omit=dev`
reports 0 vulnerabilities.
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
The earlier lean boot skipped fleet auto-init, experience capture, CrossPhaseHooks,
ReasoningBank and workers in memory mode. That left the server functionally
degraded — aqe_health reported unhealthy/memory.connected:false/loadedDomains:0,
and read-only fleet/team calls failed with 'Fleet not initialized'. Now that all
persistence layers are gated to run in-memory (unified memory, RVF, hypergraph,
sessions, results), those subsystems write nothing, so boot the SAME as normal
mode. Verified: aqe_health => healthy/connected/14 domains, full tool surface
works, and still ZERO files under .agentic-qe/. Refs #534
Smoke testing showed fleet_init still wrote patterns.rvf and agent_spawn wrote
branches/ in database-free mode — the per-site gates missed the getSharedRvfAdapter
pattern-store path and the agent memory-branch path. Gate the single chokepoint
instead: isRvfNativeAvailable() reports false when AQE_MEMORY_BACKEND=memory, so
every RVF consumer (pattern store, dual-writer, agent memory branches) takes its
existing no-native-RVF degraded path and writes nothing. Verified: fleet_init +
agent_spawn x2 create zero files; default mode still creates RVF/db. Refs #534
Without --auto, `aqe init` fell through to the legacy runStandardInit path,
which ignores --with-opencode (no assets installed) and --no-database/--memory
memory (boots the full kernel and writes .agentic-qe/memory.db). Route to
runModularInit whenever a platform flag or database-free mode is requested, so
those flags take effect regardless of --auto. Refs #529, #530
When AQE_MEMORY_BACKEND=memory the MCP server now uses InMemoryBackend and
boots lean (skips token persistence, experience-capture, ReasoningBank
prewarm, workers, QualityDaemon and fleet auto-init), writing nothing
under .agentic-qe/. Also route console.log/info/debug to stderr so stray
subsystem diagnostics never corrupt the JSON-RPC stream on stdout, which
made strict MCP clients fail with 'Failed to get tools'. Refs #530, #527
Add an opt-in in-memory backend mode wired through the modular
orchestrator. Skips the database, learning, code-intelligence and workers
phases; verification adapts (no DB to verify/write); the kernel no longer
creates .agentic-qe/ in memory mode. Default persistent behavior is
unchanged. Refs #530
OpenCode could not load AQE's .opencode/agents/*.yaml schema, so 0 agents
and 0 skills appeared. Convert at install time to OpenCode-native
.opencode/agent/*.md and .opencode/skills/<name>/SKILL.md (with per-agent
permission frontmatter); drop the incompatible tool wrappers. The
generated opencode.json now invokes the 'aqe-mcp' bin directly to avoid
the 'agentic-qe mcp' double-spawn stdin bug. Refs #529, #528
handleInitialize() returned a fixed protocolVersion regardless of the
client's request, which strict MCP clients (OpenCode, official SDK) can
reject. Echo the client's requested version when supported, else fall
back to the latest. Refs #527
Beta-test feedback for ruvnet/agent-harness-generator (MetaHarness):
- 00-04: executive summary, quality, product, technical, and six-hats
cross-pollination analyses.
- qcsd-development/: full QCSD Development Swarm pass (6 specialized QE
agents) — TDD, complexity, coverage, security, mutation, defect prediction,
plus executive summary and learning-persistence record. Gate: HOLD (2 HIGH
security findings in the provenance/secret-handling code).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Follow-up to d95f0d7b / 2f4d119c — the earlier passes updated .claude/skills
and assets/skills but missed several other eval surfaces that still pinned
retired IDs (claude-3.5-sonnet, claude-3-haiku):
- `.github/workflows/test-qe-browser.yml` — the `aqe eval run --model` flag
(inert in command-eval mode, but a real dead-ID leftover) -> claude-sonnet-4-6.
- `plugins/agentic-qe-fleet/skills/` — the tracked plugin-marketplace skill copy
(8 eval suites + a sample-output fixture); chaos skills get the opus-4-8 ceiling.
- `docs/schemas/skill-eval.schema.json` — models `enum` was missing claude-opus-4-8
and still allowed retired/deprecated IDs; trimmed to current+active, default
flipped to claude-sonnet-4-6. Plus description examples in the sibling schemas
and the modelUsed values in docs/examples/*.json fixtures.
- `docs/templates/skill-frontmatter.example.yaml` validation_models list.
- `src/cli/commands/eval.ts` — the `aqe eval` `--model` help example and the
`run-all --models` default (`claude-3.5-sonnet` -> `claude-sonnet-4-6`).
Verified: typecheck clean; all edited JSON/YAML parse; zero retired IDs remain in
any eval suite, schema, template, workflow, or eval-CLI default. Intentionally
left untouched: the consensus-provider subsystem (pricing tables / type-unions /
defaults — separate routing layer, bucket-2 data) and historical release notes /
QE audit reports (which document the old IDs by design).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Follow-up to d95f0d7b — that commit updated only .claude/skills/*/evals and
docs/templates, missing two surfaces:
1. assets/skills/ — the npm-distribution copy (tracked, generated from
.claude/skills by prepare-assets.cjs). All 54 eval matrices + nested
cross-model inputs + the .validation templates/schemas/examples now carry
current model IDs, at exact parity with .claude/skills.
2. .claude/skills/.validation/ — the *shipped* eval template, schema, and
example fixtures (the real source new skill evals are generated from):
- skill-eval.schema.json: the models `enum` was missing claude-opus-4-8
(so high-stakes evals using it would fail validation) and still listed
retired claude-3.5-sonnet / claude-3-haiku and soon-retiring
claude-sonnet-4; enum trimmed to current+active IDs, default flipped from
claude-3.5-sonnet -> claude-sonnet-4-6.
- eval.template.yaml / security-testing-eval.template.yaml: models_to_test
refreshed (security template gets the opus-4-8 ceiling).
- skill-frontmatter.example.yaml, schema description examples, and the
modelUsed values in .validation/examples/*.json and
mutation-testing/test-data/sample-output.json fixtures.
Verified: zero retired/deprecated Claude IDs in either tree; both trees at
parity (55x sonnet-4-6, 55x haiku-4-5, 7x opus-4-8 across the 7 high-stakes
skills); all edited YAML/JSON parses. Unrelated .md/.sh drift that
prepare-assets surfaced was reverted to keep this commit model-IDs only.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The cross-model consistency matrix (models_to_test) in 54 skill evals + both
eval templates pinned retired/malformed model IDs — `claude-3.5-sonnet`
(malformed: dashes, not dots; resolves to Sonnet 3.5, retired Oct 2025) and
`claude-3-haiku` (retired Apr 19 2026). Both 404, so those evals could not run
a meaningful cross-model check.
Refresh to current pinned IDs, preserving the original "primary + fast floor"
design:
- claude-sonnet-4-6 # primary (high accuracy)
- claude-haiku-4-5 # fast floor (minimum quality)
High-stakes skills additionally get the capability ceiling claude-opus-4-8
(security-testing, security-visual-testing, n8n-security-testing,
pentest-validation, chaos-engineering-resilience, qe-chaos-resilience,
qe-defect-intelligence) and the security eval template. Opus is scoped to these
7 + the security template rather than all 54 to avoid a fleet-wide eval-cost
blowup.
Also fixes the two `--model claude-3.5-sonnet/claude-3-haiku` examples in
run-skill-eval.ts and a nested cross-model test-case input in
qe-learning-optimization.yaml.
Verified: all eval YAML parses; zero retired/deprecated Claude IDs remain across
skills/evals + templates + runner. (src/domains/constants.ts MODEL_TIERS was
already current — tier 1 claude-haiku-4-5 — so it is intentionally unchanged;
the legacy claude-3-haiku-20240307 entries in cost-tracker/bedrock are retained
historical pricing/availability data.)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
npm-publish.yml runs a consumer-POV tarball audit as a publish blocker, but
only at release time — so an override-masked vuln (root-only `overrides` don't
propagate to consumers) merges to main undetected and only surfaces as a late,
blocked release. This adds consumer-audit.yml running the SAME clean-consumer
tarball audit on every dependency-affecting PR and push to main, catching the
regression before merge so main stays releasable.
Mirrors the proven publish-time job: npm ci -> build -> npm pack -> install the
tarball into a fresh consumer project -> `npm audit --audit-level=high` from the
consumer's perspective (where our overrides do NOT apply). Path-filtered to
dependency-affecting files (package.json, package-lock.json, build/pack scripts,
.npmignore, this workflow), least-privilege (contents: read), concurrency-guarded.
Precedent: v3.9.13 shipped 15 CRITICAL protobufjs ACE vulns to every installer
while in-repo `npm audit` read 0 (GHSA-xq3m-2v4x-88gg). Verified: actionlint
clean; logic confirmed passing on the current v3.10.6 tree (consumer install
resolves protobufjs@7.6.3, 0 high/critical).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Dragan Spiridonov