 Bo
|
092b99dad9
|
feat(skills): add skill-eval.sh ms-backed quality gate (ag-yzoz #skill-eval) (#726)
## What
Adds `scripts/skill-eval.sh` — a thin gate that runs one skill's
`SKILL.md` through Jeff Emanuel's `ms` (meta_skill) `lint` + `validate`.
The wrapper decides the gate purely on whether a **blocking** `rule_id`
appears (ms's own severity/exit is not trusted):
- Blocking: `no-secrets`, `no-injection`, `safe-paths`,
`required-metadata`, `no-cycle`, `valid-version` → non-zero exit.
- Everything else → annotated only (`::warning::`), never fails.
- `ms` not on PATH (or `MS_BIN` missing) → **loud `::error::` hard-fail
(exit 3)**, never skip-and-pass — a silent skip would recreate the
no-evaluation gap this gate closes.
Planted `skills/_fixtures/{bad,good}-skill/SKILL.md` fixtures +
`tests/scripts/skill-eval.bats` (10 tests) exercise the
bad/good/ms-absent contracts.
## Fixture scoping
`skills/_fixtures/` is non-skill scaffolding (planted fixtures). The
skill-enumerating gates/generators are taught to skip `skills/_*` so the
fixtures never inflate the skill count (stays 82) or fail registry /
schema / wiring / runtime / count gates. Touched enumerators:
`generate-registry`, `validate-skill-schema`, `sync-skill-counts`,
`validate-skill-count`, `generate-skill-catalog`,
`generate-context-map`, `validate-manifests`, `check-wiring-closure`,
`validate-headless-runtime-skills`, `lib/sku_catalog.py`,
`check-alias-collisions`, `run-all`, `test-runtime-opencode-smoke`,
`release-smoke-test`, `smoke-test`, `validate-skill`.
Does **not** touch `validate.yml` — wiring skill-eval.sh into CI is a
separate bead.
Epic: ag-czzf
## Verification
- `bats tests/scripts/skill-eval.bats` → 10/10 pass (ms 0.1.2 on PATH)
- `shellcheck scripts/skill-eval.sh` → clean
- `check-registry-drift`, `validate-skill-schema`, `sync-skill-counts`,
`run-all`, opencode-smoke → all PASS with fixtures present, skill count
= 82 (fixtures excluded)
Closes-scenario: ag-yzoz#skill-eval
Bounded-context: BC2-Validation
Evidence: scripts/skill-eval.sh
---------
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
|
2026-06-04 16:38:57 -04:00 |
|