Commit Graph

4 Commits

Author SHA1 Message Date
Bo 65a986c691 fix(skill-eval): constrain backtick safe-paths exemption to known file extensions (ag-eatf follow-up) (#801)
## Why (quorum remediation)
#767 (via an autonomous resolver, **merged without quorum**) broadened
the ag-eatf safe-paths false-positive filter to strip **any**
backtick-wrapped `../path`. Side effect: a backtick-wrapped
`../../../../etc/passwd` in a SKILL.md would be treated as a safe
doc-reference and **bypass the safe-paths gate**.

Cross-model quorum on the as-merged change: **Codex = REVISE**
("unacceptable bypass… constrain to repo-internal + approved extensions,
else revert"). (agy/Gemini was unreachable this round — timeouts.)
Tightening a gate back toward its documented intent is the safe
direction, so this lands the constrained version.

## Fix
Backtick strip now requires a **known repo-file extension**:
`md|markdown|mdx|txt|rst|json|yaml|toml|sh|go|py|ts|rs`. Legit
inline-code refs (``../../schemas/x.json``, ``../scripts/y.sh``) stay
exempt; an extension-less traversal (``../../../../etc/passwd``) no
longer matches → still **BLOCKS**.

## Evidence
`bats tests/scripts/skill-eval.bats` → 14/14 (2 new:
backtick-`/etc/passwd` blocks; backtick-`.json` ref passes; the
bare-traversal-blocks test still green).

Closes-scenario: ag-eatf#constrain-backtick-exemption
Bounded-context: BC2-Validation
Evidence: scripts/skill-eval.sh
2026-06-06 12:39:15 -04:00
Bo bdd62816ea docs(skills): add trigger markers to judgment-tier skills (ag-okfu #judgment-wave) (#767)
## What

Wave 1 of **ag-okfu** — adds an explicit `Use when:` discovery trigger
clause to the 8 **judgment-tier** skills: `council`, `design`,
`post-mortem`, `pre-mortem`, `red-team`, `review`, `validate`, `vibe`.

`scan_descriptions.py` flagged 73/81 skills with no trigger marker — a
silent skill-selection risk since runtime discovery is pure LLM
reasoning over the `description` field. The judgment tier ships first
because mis-selection there (e.g. grabbing `review` when you needed
`validate`) is the most costly.

## Why this slice

The bead says *"wave by metadata.tier."* This is one coherent wave = one
PR with a single rollback semantic. Remaining tiers (execution 22,
knowledge 8, meta 8, session 6, background 5, product 5, contribute 4,
cross-vendor 3, library 3, orchestration 1) ship as follow-on waves.

## Evidence

- Corpus missing-trigger count: **73 → 65** (the 8 judgment skills now
pass).
- `python3 skills/skill-builder/scripts/scan_descriptions.py skills
--strict` — all 8 judgment skills report `has_trigger=True`.
- `bash scripts/generate-registry.sh --check` → `OK: registry.json is up
to date` (regenerated, descriptions embedded).
- `markdownlint` clean on all 8 changed SKILL.md files.

## Notes

- Does **not** close ag-okfu — the `--strict` gate stays red until all
tiers are done. Advances the bead; unblocks nothing yet (ag-cx7d ratchet
waits on full corpus clean).

Bounded-context: BC1-Corpus
Evidence: skill-domain-map golden gate
2026-06-06 11:33:18 -04:00
Bo fec947be0d fix(skill-eval): filter safe-paths false-positives on markdown doc-link ../ (ag-eatf #safe-paths-doclink-filter) (#796)
## What

`scripts/skill-eval.sh` treated ms's `safe-paths` rule as blocking on
**every** `../`. That rule is a blunt regex, and `../` relative markdown
doc-links are the repo-wide SKILL.md convention (47+ skills, e.g.
`[x](../other/SKILL.md)`, `../../docs/...md`). SKILL.md is
documentation, not executed code, so those are false positives — they
redded `skill-eval` → `summary` on **every skills-touching PR** (blocked
#776; gated #759/#764/#767 in this session's drain).

## Fix

Added a file-level filter: `safe-paths` stays **blocking only when a
`../` survives stripping (1) markdown inline-link targets `](...)` and
(2) relative doc-path tokens (`*.md/.markdown/.mdx/.txt/.rst`)** — i.e.
a real, non-doc `../`. Otherwise the findings downgrade to advisory
annotations (announced, not silenced).

Net effect: **purely removes false positives; real protection is
preserved.** A genuine traversal (e.g. `../../../../etc/passwd` in a
description) still blocks — verified. No regression to the other
blocking rules.

## Evidence

- `bats tests/scripts/skill-eval.bats` → 12/12 pass (10 prior + 2 new
ag-eatf cases: doc-links downgrade→PASS; real non-doc `../`→BLOCK).
- Manual: `agent-native` and `domain` skills (previously red on
safe-paths) now PASS; a mixed real-threat fixture BLOCKs.

Closes-scenario: ag-eatf#safe-paths-doclink-filter
Bounded-context: BC2-Validation
Evidence: scripts/skill-eval.sh
2026-06-06 09:25:51 -04:00
Bo 092b99dad9 feat(skills): add skill-eval.sh ms-backed quality gate (ag-yzoz #skill-eval) (#726)
## What

Adds `scripts/skill-eval.sh` — a thin gate that runs one skill's
`SKILL.md` through Jeff Emanuel's `ms` (meta_skill) `lint` + `validate`.
The wrapper decides the gate purely on whether a **blocking** `rule_id`
appears (ms's own severity/exit is not trusted):

- Blocking: `no-secrets`, `no-injection`, `safe-paths`,
`required-metadata`, `no-cycle`, `valid-version` → non-zero exit.
- Everything else → annotated only (`::warning::`), never fails.
- `ms` not on PATH (or `MS_BIN` missing) → **loud `::error::` hard-fail
(exit 3)**, never skip-and-pass — a silent skip would recreate the
no-evaluation gap this gate closes.

Planted `skills/_fixtures/{bad,good}-skill/SKILL.md` fixtures +
`tests/scripts/skill-eval.bats` (10 tests) exercise the
bad/good/ms-absent contracts.

## Fixture scoping

`skills/_fixtures/` is non-skill scaffolding (planted fixtures). The
skill-enumerating gates/generators are taught to skip `skills/_*` so the
fixtures never inflate the skill count (stays 82) or fail registry /
schema / wiring / runtime / count gates. Touched enumerators:
`generate-registry`, `validate-skill-schema`, `sync-skill-counts`,
`validate-skill-count`, `generate-skill-catalog`,
`generate-context-map`, `validate-manifests`, `check-wiring-closure`,
`validate-headless-runtime-skills`, `lib/sku_catalog.py`,
`check-alias-collisions`, `run-all`, `test-runtime-opencode-smoke`,
`release-smoke-test`, `smoke-test`, `validate-skill`.

Does **not** touch `validate.yml` — wiring skill-eval.sh into CI is a
separate bead.

Epic: ag-czzf

## Verification

- `bats tests/scripts/skill-eval.bats` → 10/10 pass (ms 0.1.2 on PATH)
- `shellcheck scripts/skill-eval.sh` → clean
- `check-registry-drift`, `validate-skill-schema`, `sync-skill-counts`,
`run-all`, opencode-smoke → all PASS with fixtures present, skill count
= 82 (fixtures excluded)

Closes-scenario: ag-yzoz#skill-eval
Bounded-context: BC2-Validation
Evidence: scripts/skill-eval.sh

---------

Co-authored-by: Cursor Agent <cursoragent@cursor.com>
2026-06-04 16:38:57 -04:00