Commit Graph

8 Commits

Author SHA1 Message Date
Bo 092b99dad9 feat(skills): add skill-eval.sh ms-backed quality gate (ag-yzoz #skill-eval) (#726)
## What

Adds `scripts/skill-eval.sh` — a thin gate that runs one skill's
`SKILL.md` through Jeff Emanuel's `ms` (meta_skill) `lint` + `validate`.
The wrapper decides the gate purely on whether a **blocking** `rule_id`
appears (ms's own severity/exit is not trusted):

- Blocking: `no-secrets`, `no-injection`, `safe-paths`,
`required-metadata`, `no-cycle`, `valid-version` → non-zero exit.
- Everything else → annotated only (`::warning::`), never fails.
- `ms` not on PATH (or `MS_BIN` missing) → **loud `::error::` hard-fail
(exit 3)**, never skip-and-pass — a silent skip would recreate the
no-evaluation gap this gate closes.

Planted `skills/_fixtures/{bad,good}-skill/SKILL.md` fixtures +
`tests/scripts/skill-eval.bats` (10 tests) exercise the
bad/good/ms-absent contracts.

## Fixture scoping

`skills/_fixtures/` is non-skill scaffolding (planted fixtures). The
skill-enumerating gates/generators are taught to skip `skills/_*` so the
fixtures never inflate the skill count (stays 82) or fail registry /
schema / wiring / runtime / count gates. Touched enumerators:
`generate-registry`, `validate-skill-schema`, `sync-skill-counts`,
`validate-skill-count`, `generate-skill-catalog`,
`generate-context-map`, `validate-manifests`, `check-wiring-closure`,
`validate-headless-runtime-skills`, `lib/sku_catalog.py`,
`check-alias-collisions`, `run-all`, `test-runtime-opencode-smoke`,
`release-smoke-test`, `smoke-test`, `validate-skill`.

Does **not** touch `validate.yml` — wiring skill-eval.sh into CI is a
separate bead.

Epic: ag-czzf

## Verification

- `bats tests/scripts/skill-eval.bats` → 10/10 pass (ms 0.1.2 on PATH)
- `shellcheck scripts/skill-eval.sh` → clean
- `check-registry-drift`, `validate-skill-schema`, `sync-skill-counts`,
`run-all`, opencode-smoke → all PASS with fixtures present, skill count
= 82 (fixtures excluded)

Closes-scenario: ag-yzoz#skill-eval
Bounded-context: BC2-Validation
Evidence: scripts/skill-eval.sh

---------

Co-authored-by: Cursor Agent <cursoragent@cursor.com>
2026-06-04 16:38:57 -04:00
Bo e90d6d79f9 feat(registry): SKU capability catalog (schema v2) — generated skill×command×BC join (ag-cbm) (#525)
## What

Extends `registry.json` into a **generated SKU capability catalog**
(schema_version 2) — the flagship 3.0 inventory feature. It is a 4th
DERIVED projection (alongside DDD-vocab / Hex-frontmatter /
Gherkin-acceptance), a JOIN of existing sources — **never
hand-authored** (repo bans hand-edited inventory maps). Oracle-validated
design: `.agents/discovery/2026-05-24-capability-inventory-and-gaps.md`.

## Catalog

214 SKU entries: **81 skills + 66 cli-commands + 64 gates + 3
reference-impls**. Each entry: `sku` (stable id), `name`, `type`,
`bounded_context`, `hex_role`, `tier`, `purpose`, `status` (active |
deprecated | planned | alias-of:&lt;sku&gt;), `disposition`, `consumes`,
`produces`, `drives_commands`, `driven_by_skills`.

`registry.json` schema v2 is a strict **superset** of v1 — existing
`summary`/`surfaces`/`cadence_recommendations` consumers (the Go `ao
registry` command) keep working; new
`capabilities`/`capability_summary`/`cli_top_level_commands` blocks are
added.

## The join key — `drives_commands`

Derived from each skill **BODY** (SKILL.md + `references/*`), resolved
against the live cobra tree (`scripts/lib/sku_extract.py`). Stale refs
that resolve to nothing (e.g. the removed `ao schedule`) never become
edges. `driven_by_skills` is the cmd→skill reverse. This closes oracle
gap #1 (no skill↔command linkage anywhere).

## New gate — `validate-sku-catalog-drift`

Three required checks: **(a) drift** (regenerate + diff), **(b) linkage
integrity** (every `drives_commands` edge resolves to a real `ao`
command — closes gaps #1/#2/#3), **(c) coverage** (every BC + every
operating-loop move has an active skill; every BC has a cli-command).
Wired into `validate.yml` + the verdict aggregator.

## Stale refs fixed

`ao schedule` in `skills/crank` + `skills/rpi` best-practices and
`agentopsd` in `skills/swarm` shared-checkout-discipline → reframed to
the hookless / out-of-session (cron-driven `ao rpi`/`ao evolve` via Gas
City) model. Disjoint from PR #524 (which fixed the hooks/inject refs);
rebased on it — `validate-skill-cli-snippets.sh` is now 0.

## Retired

The bogus "163 cli_commands" count (Go-file count, oracle gap #6) → **66
real top-level cobra nodes** from the live tree.

## Gates (all green locally)

registry-check · validate-sku-catalog-drift (3/3) ·
validate-context-map-drift · skill-domain-map-golden ·
bounded-contexts-drift · codex-parity-drift · regen-codex-hashes --check
· contract-compatibility (sku-catalog.md catalogued) · doc-release ·
validate-skill-cli-snippets (0) · skill-frontmatter · heal --strict ·
markdownlint (0) · shellcheck (0 errors) · mkdocs --strict · `go test
./cmd/ao` (9207) · new bats suite (5/5).

Closes-scenario: ag-cbm#sku-catalog
Bounded-context: BC1-Corpus
Evidence: .agents/discovery/2026-05-24-capability-inventory-and-gaps.md
2026-05-24 20:47:04 -04:00
Boden Fuller 9e981d5907 fix(ci): close practice provenance validator gaps 2026-05-12 19:09:20 -04:00
Boden Fuller fb5813d180 fix(ci): registry non-determinism + pre-push goals-validate fallback
Two real bugs surfaced by cycle 49 CI (now diagnosable thanks to
`--print-output-on-failure` from c49):

1. **registry-check** was stale on CI because `generate-registry.sh`
   walked `evals/` via filesystem `find`, which picks up untracked
   files like `evals/workbench/scorecard-latest.json` (written by
   local eval runs, never committed). CI's clean checkout produces a
   different count, so `--check` diverges. This is the same soc-k47k
   pattern already applied to `.agents/` — extending it to `evals/`:
   walk `git ls-files 'evals/*/*.json'` filtered to immediate
   children. After fix: workbench shows 1 tracked eval
   (`baseline-scorecard`), CI and local agree.

2. **bats-tests** (CI failure on cycle 48 + 49) was caused by
   `pre-push-gate.sh`'s goals-validate fallback. In full mode (no
   `--fast`), when `cli/bin/ao` isn't pre-built, the gate falls back
   to `bash -c 'cd cli && go build -o /tmp/ao-goals-val ./cmd/ao &&
   /tmp/ao-goals-val goals validate --json'`. In the FAKE_REPO setup
   of pre-push-gate.bats, cli/ exists with only a go.mod stub — no
   cli/cmd/ao package. `go build` failed degenerately and the
   `/tmp/ao-goals-val` execution emitted "No such file or
   directory", surfacing as `FAIL goals validate` → gate BLOCKED →
   bats `[ status -eq 0 ]` failed. Adding a pre-check for
   `cli/cmd/ao/` directory skips gracefully with a clear reason.

Out of scope:
- go-build coverage failure (cmd/ao 75.8% < 76% floor) — separate
  issue; needs a real test or threshold ratchet adjustment.
- soc-l4yt canary flake — recurring intermittent.
- security-toolchain-gate (advisory, exit 3 with 0 missing tools) —
  pre-existing.

Cites: 2026-05-07-ci-push-gate-toil-pattern.md learning (local/CI
drift in non-deterministic walks burns 3-5x push cycles per PR).

Validation:
- bats tests/scripts/pre-push-gate.bats — 51 ok, 0 failed.
- bats `passes when no Go changes` + `clears GIT env` — pass.
- bash scripts/generate-registry.sh --check — OK.
- jq '[.surfaces.evals[] | select(.suite=="workbench")]' — 1 eval
  tracked (no scorecard-latest), matches CI's view.
2026-05-12 09:55:58 -04:00
Boden Fuller 5fb34c3dc5 fix(ci): refresh registry generation 2026-05-09 17:09:20 -04:00
Bo 4cf012b16e CI toil-reduction sweep: registry determinism + codex auto-refresh + local CI dry-run (3 beads) (#261)
* feat(ci): toil-reduction sweep — registry determinism + codex auto-refresh + local CI dry-run (soc-k47k, soc-7qq9, soc-ws40)

Three compounding fixes filed during the 2026-05-07 retrospective on
CI/push-gate toil. See `.agents/learnings/2026-05-07-ci-push-gate-toil-pattern.md`
for the data: 9 push iterations across 4 PRs, ~45min wall-clock + thousands
of orchestrator tokens spent diagnosing identical-looking failures with
different root causes.

soc-k47k — registry-check determinism:
- scripts/generate-registry.sh build_knowledge_stores() now walks
  `git ls-files .agents/` instead of filesystem `find`. CI's clean
  checkout and a session-built local checkout produce identical output
  because both consult the git index (deterministic).
- Removed the `[[ -d .agents ]] || return` early-exit so behavior is
  uniform regardless of working-tree state.
- Eliminates the dominant single source of failed-push retries this
  session (5 iterations across PRs #255 and #260 alone).

soc-7qq9 — PostToolUse codex auto-refresh:
- New hooks/postedit-codex-refresh.sh fires after Edit/Write of any
  skills/<name>/SKILL.md, references/, scripts/, or schemas/ file.
  Auto-runs `scripts/refresh-codex-artifacts.sh --scope head` so codex
  hashes stay current; operator never sees the parity warning.
- Registered in hooks/hooks.json (matchers Edit + Write, 10s timeout).
- Embedded copy synced via `make sync-hooks`.
- Disable via AGENTOPS_HOOKS_DISABLED=1 or
  AGENTOPS_CODEX_AUTOREFRESH_DISABLED=1.
- Best-effort: failures log to stderr but always exit 0.

soc-ws40 — local CI-equivalent dry-run:
- New scripts/test-ci-deterministic-gates.sh runs the 4 deterministic
  CI gates that have historically surprised PRs at push time: registry-
  check, skill-lint, heal-skill --strict, codex artifact metadata.
- Non-fail-fast: reports all failures together so you can fix them in
  one diagnostic round instead of N push iterations.
- Currently runs in <30s on a clean tree (vs. ~5min for one CI canary
  cycle).

Together these compound: with #1 + #2 + #3 wired, the CI surprises that
drove this session's toil should be catchable locally before push.

go test ./... → 11858 passed in 50 packages.
bash scripts/test-ci-deterministic-gates.sh → 4/4 pass.

* test(hooks): add coverage for postedit-codex-refresh

Required by scripts/check-test-fixture-parity.sh — every hooks/*.sh must
have at least one test reference. 4 smoke tests:
- non-skill path → silent exit 0
- AGENTOPS_HOOKS_DISABLED=1 short-circuits
- AGENTOPS_CODEX_AUTOREFRESH_DISABLED=1 short-circuits
- skill path outside any git repo → no-op

* fix(eval): bump hook-regression-suite Total count 222→226 for postedit-codex-refresh tests
2026-05-07 19:42:29 -04:00
Boden Fuller 8a2cd35ba3 fix: remove unused variable in generate-registry.sh (shellcheck SC2034) 2026-05-04 20:58:13 -04:00
Boden Fuller d10bc65332 feat: add unified registry generator and registry.json
Walks all repo surfaces (71 skills, 41 hooks, 46 .agents/ stores,
11 daemon job types, 61 evals, 143 CLI commands) and produces a
single queryable manifest with cadence recommendations.

Supports --check mode for CI staleness detection and --stdout for
piping. Tier assignments parsed from SKILL-TIERS.md with known-tier
validation to avoid dependency-table pollution.
2026-05-04 20:57:29 -04:00