mirror of
https://github.com/boshu2/agentops.git
synced 2026-09-14 15:08:13 +08:00
Bo
092b99dad9
## What
Adds `scripts/skill-eval.sh` — a thin gate that runs one skill's
`SKILL.md` through Jeff Emanuel's `ms` (meta_skill) `lint` + `validate`.
The wrapper decides the gate purely on whether a **blocking** `rule_id`
appears (ms's own severity/exit is not trusted):
- Blocking: `no-secrets`, `no-injection`, `safe-paths`,
`required-metadata`, `no-cycle`, `valid-version` → non-zero exit.
- Everything else → annotated only (`::warning::`), never fails.
- `ms` not on PATH (or `MS_BIN` missing) → **loud `::error::` hard-fail
(exit 3)**, never skip-and-pass — a silent skip would recreate the
no-evaluation gap this gate closes.
Planted `skills/_fixtures/{bad,good}-skill/SKILL.md` fixtures +
`tests/scripts/skill-eval.bats` (10 tests) exercise the
bad/good/ms-absent contracts.
## Fixture scoping
`skills/_fixtures/` is non-skill scaffolding (planted fixtures). The
skill-enumerating gates/generators are taught to skip `skills/_*` so the
fixtures never inflate the skill count (stays 82) or fail registry /
schema / wiring / runtime / count gates. Touched enumerators:
`generate-registry`, `validate-skill-schema`, `sync-skill-counts`,
`validate-skill-count`, `generate-skill-catalog`,
`generate-context-map`, `validate-manifests`, `check-wiring-closure`,
`validate-headless-runtime-skills`, `lib/sku_catalog.py`,
`check-alias-collisions`, `run-all`, `test-runtime-opencode-smoke`,
`release-smoke-test`, `smoke-test`, `validate-skill`.
Does **not** touch `validate.yml` — wiring skill-eval.sh into CI is a
separate bead.
Epic: ag-czzf
## Verification
- `bats tests/scripts/skill-eval.bats` → 10/10 pass (ms 0.1.2 on PATH)
- `shellcheck scripts/skill-eval.sh` → clean
- `check-registry-drift`, `validate-skill-schema`, `sync-skill-counts`,
`run-all`, opencode-smoke → all PASS with fixtures present, skill count
= 82 (fixtures excluded)
Closes-scenario: ag-yzoz#skill-eval
Bounded-context: BC2-Validation
Evidence: scripts/skill-eval.sh
---------
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
61 lines
2.1 KiB
Bash
Executable File
61 lines
2.1 KiB
Bash
Executable File
#!/usr/bin/env bash
|
|
# Validates artifact-to-reference wiring across the project.
|
|
# Exit 0 = all wired, exit 1 = orphans found.
|
|
# Used as a GOALS.yaml meta-goal.
|
|
set -euo pipefail
|
|
|
|
ERRORS=0
|
|
|
|
# 1. Every scripts/check-*.sh must be referenced somewhere (goals, CI, tests, or other scripts)
|
|
for script in scripts/check-*.sh; do
|
|
[ -f "$script" ] || continue
|
|
base=$(basename "$script")
|
|
if ! grep -rq "$base" GOALS.md GOALS.yaml .github/workflows/ tests/ scripts/ 2>/dev/null; then
|
|
echo "UNWIRED SCRIPT: $base not referenced in goals, CI, tests, or scripts"
|
|
ERRORS=$((ERRORS + 1))
|
|
fi
|
|
done
|
|
|
|
# 2. Every skill directory should appear in SKILL-TIERS.md
|
|
if [ -f skills/SKILL-TIERS.md ]; then
|
|
for skill_dir in skills/*/; do
|
|
[ -d "$skill_dir" ] || continue
|
|
skill_name=$(basename "$skill_dir")
|
|
# Leading-underscore dirs (e.g. skills/_fixtures/) are non-skill scaffolding
|
|
# (planted test fixtures); they are intentionally absent from SKILL-TIERS.md.
|
|
case "$skill_name" in _*) continue ;; esac
|
|
if ! grep -q "$skill_name" skills/SKILL-TIERS.md 2>/dev/null; then
|
|
echo "UNWIRED SKILL: $skill_name not in SKILL-TIERS.md"
|
|
ERRORS=$((ERRORS + 1))
|
|
fi
|
|
done
|
|
fi
|
|
|
|
# 3. Every lib/scripts/*.sh should be referenced by at least one SKILL.md, script, or test
|
|
for lib_script in lib/scripts/*.sh; do
|
|
[ -f "$lib_script" ] || continue
|
|
base=$(basename "$lib_script")
|
|
if ! grep -rq "$base" skills/*/SKILL.md scripts/ tests/ 2>/dev/null; then
|
|
echo "ORPHANED LIB SCRIPT: $base not referenced by any skill, script, or test"
|
|
ERRORS=$((ERRORS + 1))
|
|
fi
|
|
done
|
|
|
|
# 4. Every lib/*.sh helper should be referenced by the CLI, a script, a skill, or a test
|
|
for helper_script in lib/*.sh; do
|
|
[ -f "$helper_script" ] || continue
|
|
base=$(basename "$helper_script")
|
|
if ! grep -rq "$base" cli/ scripts/ skills/ tests/ 2>/dev/null; then
|
|
echo "ORPHANED LIB HELPER: $base not referenced by the CLI, scripts, skills, or tests"
|
|
ERRORS=$((ERRORS + 1))
|
|
fi
|
|
done
|
|
|
|
if [ "$ERRORS" -eq 0 ]; then
|
|
echo "All wiring checks passed"
|
|
exit 0
|
|
fi
|
|
|
|
echo "Found $ERRORS wiring issue(s)"
|
|
exit 1
|