Follow-up to d95f0d7b / 2f4d119c — the earlier passes updated .claude/skills
and assets/skills but missed several other eval surfaces that still pinned
retired IDs (claude-3.5-sonnet, claude-3-haiku):
- `.github/workflows/test-qe-browser.yml` — the `aqe eval run --model` flag
(inert in command-eval mode, but a real dead-ID leftover) -> claude-sonnet-4-6.
- `plugins/agentic-qe-fleet/skills/` — the tracked plugin-marketplace skill copy
(8 eval suites + a sample-output fixture); chaos skills get the opus-4-8 ceiling.
- `docs/schemas/skill-eval.schema.json` — models `enum` was missing claude-opus-4-8
and still allowed retired/deprecated IDs; trimmed to current+active, default
flipped to claude-sonnet-4-6. Plus description examples in the sibling schemas
and the modelUsed values in docs/examples/*.json fixtures.
- `docs/templates/skill-frontmatter.example.yaml` validation_models list.
- `src/cli/commands/eval.ts` — the `aqe eval` `--model` help example and the
`run-all --models` default (`claude-3.5-sonnet` -> `claude-sonnet-4-6`).
Verified: typecheck clean; all edited JSON/YAML parse; zero retired IDs remain in
any eval suite, schema, template, workflow, or eval-CLI default. Intentionally
left untouched: the consensus-provider subsystem (pricing tables / type-unions /
defaults — separate routing layer, bucket-2 data) and historical release notes /
QE audit reports (which document the old IDs by design).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(v3.4.1): bundle missing dependencies and test timeouts
- Fixed issue #219: MCP bundle failing due to missing packages
- Changed build scripts from --packages=external to selective externalization
- Pure JS deps (fast-json-patch, jose, uuid) now bundled inline
- Native modules properly externalized (better-sqlite3, hnswlib-node, etc.)
- CommonJS modules with dynamic requires externalized (typescript, fast-glob, yaml, commander, cli-progress, ora)
- Bundle size reduced from ~15MB to ~5MB
- Added timeouts to handleQualityAssess tests to prevent CI failures
- Updated version to 3.4.1
Closes#219
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* feat(v3.4.2): add skill validation system with trust tiers (ADR-056)
Implement 4-layer skill validation system for deterministic outputs:
- Tier 3 (Verified): 46 skills with full eval test suites
- Tier 2 (Validated): 7 skills with executable validators
- Tier 1 (Structured): 5 skills with JSON output schemas
- Tier 0 (Advisory): 5 skills with SKILL.md guidance only
Key additions:
- JSON Schema validation templates for skill outputs
- Executable validator scripts for correctness verification
- Evaluation test framework with multi-model support
- CLI commands: aqe skill report/summary/compare, aqe eval run/status
- ReasoningBank integration for validation pattern learning
- GitHub Actions workflow for CI skill validation
- Trust tier frontmatter in all 63 skill SKILL.md files
- Comprehensive documentation and user guides
Infrastructure:
- .claude/skills/.validation/ - Templates and frameworks
- .claude/skills/*/schemas/ - Output schemas per skill
- .claude/skills/*/scripts/ - Validator scripts per skill
- .claude/skills/*/evals/ - Test cases per skill
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* fix(ci): trust tier-based skill validation
- Tier 3 (Verified): Blocks PR on validation failure
- Tier 2 (Validated): Warns only, doesn't block
- Tier 1 (Structured): Warns only, doesn't block
- Tier 0 (Advisory): Skipped entirely
Reads trust_tier from SKILL.md frontmatter to determine
validation requirements. Only 46 Tier 3 skills block merge.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* fix(validators): remove local keyword from self-test blocks
Shell script fix: 'local' can only be used inside functions, not in
if/else blocks. Both accessibility-testing and compatibility-testing
validators had 'local' declarations in their self-test sections that
caused errors when run outside of a function context.
- Remove local test_file from accessibility-testing self-test
- Remove local test_file, browser_count, tier1_pass from compatibility-testing self-test
- Sync fixes to v3/assets/skills/
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* fix(validators): add --self-test support to n8n skill validators
The 5 n8n skill validators were failing CI because they didn't handle
the --self-test flag properly. Added self-test handling that:
- Checks for required tools (jq)
- Verifies schema file exists and is valid JSON
- Returns exit 0 on success
Fixed validators:
- n8n-expression-testing
- n8n-integration-testing-patterns
- n8n-security-testing
- n8n-trigger-testing-strategies
- n8n-workflow-testing-fundamentals
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
Dragan Spiridonov