Commit Graph

3 Commits

Author SHA1 Message Date
Dragan Spiridonov 7f01310556 fix(adr-093): aqe init + skill evals + settings routing use new models
New user projects created via aqe init will now receive the post-ADR-093
model set (Opus 4.7 / Sonnet 4.6 / Haiku 4.5) instead of retiring IDs.

* src/init/kiro-installer.ts: 4 hardcoded model fallbacks updated.
  claude-sonnet-4 → claude-sonnet-4-6 (default + priority=critical),
  claude-opus-4 → claude-opus-4-7 (opus-detected path),
  claude-haiku-4 → claude-haiku-4-5 (haiku-detected path).
  Affects: runtime agent transformation, default QE agent template
  (writeDefaultQEAgent).
* src/init/init-wizard-steps.ts, src/init/phases/12-verification.ts:
  generated aqe.yaml advisor block comment updated to
  anthropic/claude-opus-4.7 so wizard-produced configs point to the
  right advisor model out of the box.
* .claude/settings.json:188-189: claudeFlow.modelPreferences
  default claude-opus-4-5-20251101 → claude-opus-4-7,
  routing claude-3-5-haiku-20241022 → claude-haiku-4-5-20251001.
* Skill eval YAMLs (both .claude/ and assets/ copies):
  api-testing-patterns, contract-testing, database-testing,
  testability-scoring — models_to_test list updated from
  claude-sonnet-4 / claude-3-haiku to claude-sonnet-4-6 / claude-haiku-4-5.
* Skill eval schemas (3 copies: .claude, assets, docs):
  skill-eval.schema.json enum extended with claude-opus-4-7,
  claude-sonnet-4-6, claude-haiku-4-5 as accepted values. Old
  entries retained for backward compat with existing eval files.
* Sample outputs (3 copies): modelUsed field updated to
  claude-sonnet-4-6.
* a11y-ally SKILL.md code example (both .claude and assets):
  model: 'claude-sonnet-4-20250514' → 'claude-sonnet-4-6' (this was
  caught by Commit 2's sweep; landed here for audit completeness).

Verification:
* 0 tsc errors
* 392/392 init module tests pass
* npm run build: CLI + MCP bundles rebuilt clean

Not re-running aqe init against a real fixture in this commit —
minimal+non-interactive mode doesn't materialize integration files
(kiro/opencode); full wizard-mode verification deferred until after
PR opens and a maintainer runs init smoke.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-17 13:40:14 +00:00
Profa 18e15e843c feat(validation): add trigger optimizer, version comparator, and skill intent classification
Implements three improvements from Claude Blog skill-creator analysis:
- Trigger Optimizer: analyzes skill descriptions/tags for false positive/negative activation risks
- Version Comparator: A/B testing between skill versions with Cohen's d effect size
- Skill Intent: classify skills as capability_uplift, encoded_preference, or hybrid

Includes 52 unit tests (23 + 29), schema sync across all 3 copies, and ADR-056 update.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-07 11:01:05 +00:00
Dragan Spiridonov 99511738c1 feat(v3.4.2): add skill validation system with trust tiers (ADR-056) (#221)
* fix(v3.4.1): bundle missing dependencies and test timeouts

- Fixed issue #219: MCP bundle failing due to missing packages
- Changed build scripts from --packages=external to selective externalization
- Pure JS deps (fast-json-patch, jose, uuid) now bundled inline
- Native modules properly externalized (better-sqlite3, hnswlib-node, etc.)
- CommonJS modules with dynamic requires externalized (typescript, fast-glob, yaml, commander, cli-progress, ora)
- Bundle size reduced from ~15MB to ~5MB
- Added timeouts to handleQualityAssess tests to prevent CI failures
- Updated version to 3.4.1

Closes #219

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* feat(v3.4.2): add skill validation system with trust tiers (ADR-056)

Implement 4-layer skill validation system for deterministic outputs:
- Tier 3 (Verified): 46 skills with full eval test suites
- Tier 2 (Validated): 7 skills with executable validators
- Tier 1 (Structured): 5 skills with JSON output schemas
- Tier 0 (Advisory): 5 skills with SKILL.md guidance only

Key additions:
- JSON Schema validation templates for skill outputs
- Executable validator scripts for correctness verification
- Evaluation test framework with multi-model support
- CLI commands: aqe skill report/summary/compare, aqe eval run/status
- ReasoningBank integration for validation pattern learning
- GitHub Actions workflow for CI skill validation
- Trust tier frontmatter in all 63 skill SKILL.md files
- Comprehensive documentation and user guides

Infrastructure:
- .claude/skills/.validation/ - Templates and frameworks
- .claude/skills/*/schemas/ - Output schemas per skill
- .claude/skills/*/scripts/ - Validator scripts per skill
- .claude/skills/*/evals/ - Test cases per skill

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* fix(ci): trust tier-based skill validation

- Tier 3 (Verified): Blocks PR on validation failure
- Tier 2 (Validated): Warns only, doesn't block
- Tier 1 (Structured): Warns only, doesn't block
- Tier 0 (Advisory): Skipped entirely

Reads trust_tier from SKILL.md frontmatter to determine
validation requirements. Only 46 Tier 3 skills block merge.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* fix(validators): remove local keyword from self-test blocks

Shell script fix: 'local' can only be used inside functions, not in
if/else blocks. Both accessibility-testing and compatibility-testing
validators had 'local' declarations in their self-test sections that
caused errors when run outside of a function context.

- Remove local test_file from accessibility-testing self-test
- Remove local test_file, browser_count, tier1_pass from compatibility-testing self-test
- Sync fixes to v3/assets/skills/

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* fix(validators): add --self-test support to n8n skill validators

The 5 n8n skill validators were failing CI because they didn't handle
the --self-test flag properly. Added self-test handling that:
- Checks for required tools (jq)
- Verifies schema file exists and is valid JSON
- Returns exit 0 on success

Fixed validators:
- n8n-expression-testing
- n8n-integration-testing-patterns
- n8n-security-testing
- n8n-trigger-testing-strategies
- n8n-workflow-testing-fundamentals

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
2026-02-02 12:51:51 +01:00