092b99dad9
feat(skills): add skill-eval.sh ms-backed quality gate (ag-yzoz #skill-eval) ( #726 )
...
## What
Adds `scripts/skill-eval.sh` — a thin gate that runs one skill's
`SKILL.md` through Jeff Emanuel's `ms` (meta_skill) `lint` + `validate`.
The wrapper decides the gate purely on whether a **blocking** `rule_id`
appears (ms's own severity/exit is not trusted):
- Blocking: `no-secrets`, `no-injection`, `safe-paths`,
`required-metadata`, `no-cycle`, `valid-version` → non-zero exit.
- Everything else → annotated only (`::warning::`), never fails.
- `ms` not on PATH (or `MS_BIN` missing) → **loud `::error::` hard-fail
(exit 3)**, never skip-and-pass — a silent skip would recreate the
no-evaluation gap this gate closes.
Planted `skills/_fixtures/{bad,good}-skill/SKILL.md` fixtures +
`tests/scripts/skill-eval.bats` (10 tests) exercise the
bad/good/ms-absent contracts.
## Fixture scoping
`skills/_fixtures/` is non-skill scaffolding (planted fixtures). The
skill-enumerating gates/generators are taught to skip `skills/_*` so the
fixtures never inflate the skill count (stays 82) or fail registry /
schema / wiring / runtime / count gates. Touched enumerators:
`generate-registry`, `validate-skill-schema`, `sync-skill-counts`,
`validate-skill-count`, `generate-skill-catalog`,
`generate-context-map`, `validate-manifests`, `check-wiring-closure`,
`validate-headless-runtime-skills`, `lib/sku_catalog.py`,
`check-alias-collisions`, `run-all`, `test-runtime-opencode-smoke`,
`release-smoke-test`, `smoke-test`, `validate-skill`.
Does **not** touch `validate.yml` — wiring skill-eval.sh into CI is a
separate bead.
Epic: ag-czzf
## Verification
- `bats tests/scripts/skill-eval.bats` → 10/10 pass (ms 0.1.2 on PATH)
- `shellcheck scripts/skill-eval.sh` → clean
- `check-registry-drift`, `validate-skill-schema`, `sync-skill-counts`,
`run-all`, opencode-smoke → all PASS with fixtures present, skill count
= 82 (fixtures excluded)
Closes-scenario: ag-yzoz#skill-eval
Bounded-context: BC2-Validation
Evidence: scripts/skill-eval.sh
---------
Co-authored-by: Cursor Agent <cursoragent@cursor.com >
2026-06-04 16:38:57 -04:00
edcc654876
fix(validate-skill-schema): detect yq variant to support Python kislyuk yq
...
CI installs Mike Farah's Go yq (supports `-o json`), but many Linux distros
ship the kislyuk Python yq (jq wrapper that emits JSON by default). The
previous `yq -o json '.'` call aborted on the Python variant and printed
"invalid YAML frontmatter" for every skill, masking real schema errors.
Detect the variant once via `yq --help` and route through a shared
`yq_to_json` helper so both variants produce identical JSON input to the
jsonschema/jq validators.
Effect: `scripts/validate-skill-schema.sh` now reports 69/69 PASS on hosts
with the Python yq instead of 0/69 FAIL; CI behavior on the Go yq is
unchanged.
2026-04-23 07:08:00 +00:00
6ba1a76f52
fix(ci): fix skill schema, runtime format, and CLI snippet validation
...
- Add output_contract, cross-vendor, knowledge to valid schema keys/tiers
- Fix codex lint regex to exclude $-prefixed slash-commands
- Fix codex forge/post-mortem/scenario skills: /cmd → $/cmd prefix
- Fix scenario SKILL.md: replace non-existent --source/--scope flags
with valid --status variants
2026-04-06 21:51:08 -04:00
Bo