The cross-model consistency matrix (models_to_test) in 54 skill evals + both
eval templates pinned retired/malformed model IDs — `claude-3.5-sonnet`
(malformed: dashes, not dots; resolves to Sonnet 3.5, retired Oct 2025) and
`claude-3-haiku` (retired Apr 19 2026). Both 404, so those evals could not run
a meaningful cross-model check.
Refresh to current pinned IDs, preserving the original "primary + fast floor"
design:
- claude-sonnet-4-6 # primary (high accuracy)
- claude-haiku-4-5 # fast floor (minimum quality)
High-stakes skills additionally get the capability ceiling claude-opus-4-8
(security-testing, security-visual-testing, n8n-security-testing,
pentest-validation, chaos-engineering-resilience, qe-chaos-resilience,
qe-defect-intelligence) and the security eval template. Opus is scoped to these
7 + the security template rather than all 54 to avoid a fleet-wide eval-cost
blowup.
Also fixes the two `--model claude-3.5-sonnet/claude-3-haiku` examples in
run-skill-eval.ts and a nested cross-model test-case input in
qe-learning-optimization.yaml.
Verified: all eval YAML parses; zero retired/deprecated Claude IDs remain across
skills/evals + templates + runner. (src/domains/constants.ts MODEL_TIERS was
already current — tier 1 claude-haiku-4-5 — so it is intentionally unchanged;
the legacy claude-3-haiku-20240307 entries in cost-tracker/bedrock are retained
historical pricing/availability data.)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Dragan Spiridonov
d95f0d7b18