Commit Graph

18 Commits

Author SHA1 Message Date
Bo 092b99dad9 feat(skills): add skill-eval.sh ms-backed quality gate (ag-yzoz #skill-eval) (#726)
## What

Adds `scripts/skill-eval.sh` — a thin gate that runs one skill's
`SKILL.md` through Jeff Emanuel's `ms` (meta_skill) `lint` + `validate`.
The wrapper decides the gate purely on whether a **blocking** `rule_id`
appears (ms's own severity/exit is not trusted):

- Blocking: `no-secrets`, `no-injection`, `safe-paths`,
`required-metadata`, `no-cycle`, `valid-version` → non-zero exit.
- Everything else → annotated only (`::warning::`), never fails.
- `ms` not on PATH (or `MS_BIN` missing) → **loud `::error::` hard-fail
(exit 3)**, never skip-and-pass — a silent skip would recreate the
no-evaluation gap this gate closes.

Planted `skills/_fixtures/{bad,good}-skill/SKILL.md` fixtures +
`tests/scripts/skill-eval.bats` (10 tests) exercise the
bad/good/ms-absent contracts.

## Fixture scoping

`skills/_fixtures/` is non-skill scaffolding (planted fixtures). The
skill-enumerating gates/generators are taught to skip `skills/_*` so the
fixtures never inflate the skill count (stays 82) or fail registry /
schema / wiring / runtime / count gates. Touched enumerators:
`generate-registry`, `validate-skill-schema`, `sync-skill-counts`,
`validate-skill-count`, `generate-skill-catalog`,
`generate-context-map`, `validate-manifests`, `check-wiring-closure`,
`validate-headless-runtime-skills`, `lib/sku_catalog.py`,
`check-alias-collisions`, `run-all`, `test-runtime-opencode-smoke`,
`release-smoke-test`, `smoke-test`, `validate-skill`.

Does **not** touch `validate.yml` — wiring skill-eval.sh into CI is a
separate bead.

Epic: ag-czzf

## Verification

- `bats tests/scripts/skill-eval.bats` → 10/10 pass (ms 0.1.2 on PATH)
- `shellcheck scripts/skill-eval.sh` → clean
- `check-registry-drift`, `validate-skill-schema`, `sync-skill-counts`,
`run-all`, opencode-smoke → all PASS with fixtures present, skill count
= 82 (fixtures excluded)

Closes-scenario: ag-yzoz#skill-eval
Bounded-context: BC2-Validation
Evidence: scripts/skill-eval.sh

---------

Co-authored-by: Cursor Agent <cursoragent@cursor.com>
2026-06-04 16:38:57 -04:00
Bo a1730829e7 chore(release): v3.0.1 — plugin version bump + Nightly Static Validation fix (ag-s3z) (#539)
Two release-engineering surfaces the per-PR validate.yml gate never
exercised:

1. **Plugin updatability.** The 3.0.0 version label never moved across
the 15 commits that landed the skill consolidation + doc reconciliation,
so existing installs got "already at latest version" from `claude plugin
update` and stayed stale. Bump 3.0.0 → 3.0.1 (plugin.json,
marketplace.json metadata+plugin, install-ao.sh) + CHANGELOG 3.0.1 entry
(root + docs, byte-identical).

2. **Nightly Static Validation (red on main).** The hooks rip (#511)
deleted `scripts/validate-hooks-doc-parity.sh` but Nightly still ran it;
`tests/docs/validate-skill-count.sh` greps PRODUCT.md for hook-era
phrasing the 3.0 restructure removed → 2 MISSING_PATTERN fails
(warn-tolerated in validate.yml, fail-closed in Nightly). Removed the
dead step; repointed product_total to the four-layers line; retired
product_layer_total.

Verified locally: validate-skill-count.sh + validate-doc-release.sh pass
(total=75); version-consistency = 3.0.1 across all 3 manifest fields.

Follow-ups: ag-728 (PRODUCT.md daemon/hook prose), ag-x0d (3 golangci
quality findings).

Closes-scenario: ag-s3z#v3-0-1-release-readiness
Bounded-context: BC5-Runtime
Evidence: tests/docs/validate-skill-count.sh
2026-05-25 08:54:37 -04:00
Bo b4c6aec361 feat(skills)!: 3.0 consolidation — merge 4, cut 2, resolve 6 keep-flags (81→75) (#533)
## AgentOps 3.0 skill consolidation — reconcile-and-tighten

Resolves all 12 skills flagged `merge-review`/`cut-review` in
`docs/contracts/skill-dispositions.yaml` per the operator-approved
dispositions. **Zero capability loss** — every merged skill's content
lives in its target as a mode/section.

### MERGE (4) — content folded into target, source removed
| Source | Target | How |
|---|---|---|
| expert-council | council | `--mode=debate` (alias triggers documented
in council) |
| oss-docs | doc | `--mode=oss` (refs + `audit-oss-docs.sh` copied into
`doc/`) |
| readme | doc | `--mode=readme` (`readme-craft.md`; `bootstrap`
consumer repointed readme→doc) |
| pr-retro | post-mortem | `--scope=pr` (`pr-scope.md` + gh PR-outcome
mining) |

### CUT (2) — genuinely redundant, no live consumer
- **pr-plan** — folds into `/plan` + `/pr-research`; pr-* chain
repointed.
- **update** — install-script concern; refresh note added to
`using-agentops` + install one-liner.

### KEEP — resolved 6 flags (rationale in `skill-dispositions.yaml`)
`validate` + `curate` (designed-future canonical validator/miner; epic
`soc-cp7pv` GO/REVERT is a separate decision) · `security` (release-gate
driver, distinct from security-suite) · `converter` (packaging adapter
skill-builder consumes) · `knowledge-activation` (outer-loop
operationalization) · `grafana-platform-dashboard` (marketplace
specialization).

### Result
**81 → 75 skills** (65 user-facing + 10 internal). Trunk regenerated
once: skill-counts, skill-catalog, skill-domain-map, context-map,
registry, sku-catalog, codex hashes, embedded-sync. Evals repointed to
absorbed locations.

### Gates verified locally (all green)
lint-skills · heal.sh --strict · skill-schema · skill-frontmatter ·
skill-dependency-check · scenario-test-linkage ·
validate-context-map-drift · check-registry-drift · generate-registry
--check · skill-domain-map golden · validate-sku-catalog-drift ·
bounded-contexts-drift · ci-policy-parity · codex
parity/override-coverage/manifest/generated-artifacts/skill-parity/runtime-sections/backbone/lifecycle
· regen-codex-hashes --check · embedded-sync · validate-skill-count ·
markdownlint · shellcheck (--severity=error) · `cd cli && go build && go
vet && go test ./cmd/ao` (9126 passed).

Closes-scenario: ag-cons#consolidation
Bounded-context: BC1-Corpus
Evidence: .agents/discovery/2026-05-24-skill-consolidation-plan.md
2026-05-24 23:17:27 -04:00
Boden Fuller 6ce5df6f45 fix(ci): clear 4 post-merge CI failures on DDD/hexagonal PR
doc-release-gate: exclude generated cli/embedded/ mirror from link validation
  (its relative links resolve at source depth, not mirror depth) + allowlist
  3 doctrine links to the gitignored .agents/research/ note
registry-check: regenerate registry.json (77 skills) stale after the 127-commit merge
plugin-load-test + skill-schema: add hexagonal_role/consumes/produces/context_rel
  to skill-frontmatter.v1 schema — v2 dropped v1's other props so it is not a
  drop-in superset; extending v1 keeps full validation. 77/77 -> 77/77 pass.

Verified locally: validate-links 0 broken, validate-skill-schema 77/77,
validate-manifests exit 0, doc-release-gate PASS.
2026-05-15 16:20:23 -04:00
Boden Fuller ed89b956db fix: allowlist broken reference links from skill consolidation
The skill consolidation (12→4) brought reference files with
aspirational links to non-existent targets (PATTERNS.md,
TEMPLATE.md, LANGUAGES.md, etc.). Add to broken-links-allowlist
and remove the direct broken links where possible.
2026-05-04 23:06:47 -04:00
Bo f0cc98e8ab feat(validate): add skill-frontmatter-json Go helper as PyYAML fallback
Adds cli/cmd/skill-frontmatter-json, a standalone Go binary that parses
SKILL.md YAML frontmatter and emits JSON. scripts/validate-manifests.sh
now builds and uses it when python3 lacks PyYAML, removing the hard
PyYAML dependency on validation hosts.

Also simplifies tests/skills/lint-skills.sh to use jq for schema parsing
and rewrites tests/docs/validate-links.sh link extraction with awk so it
skips fenced and inline code blocks.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-27 08:32:03 -04:00
Boden Fuller 2156c627ca test(runtime): add runtime proof gates 2026-04-23 23:54:40 -04:00
Boden Fuller 2385dbd4fb fix(ci): resolve 7 CI failures from cc6750c4
## doc-release-gate (link validation)
- tests/docs/validate-links.sh: fix canonical-path assignment bug where
  inline `|| fallback` never fires if the outer $() assignment itself
  does not fail. Rewrite as explicit if-else so mkdocs-generated paths
  (docs/skills/*.md, docs/cli/*.md) resolve correctly in CI environments
  where those directories don't exist on disk.

## skill-lint + skill-integrity + go-build (TestSkillContract_ReferencesLinkedInSKILLMD)
- Add strict-delegation-contract.md link to skills/shared/SKILL.md and
  skills-codex/shared/SKILL.md Reference Documents sections (heal.sh
  --strict requires every references/*.md to be linked in its SKILL.md).
- Extend tests/skills/lint-skills.sh regex to support ../shared/references/
  paths in addition to skills/<name>/references/ — matches heal.sh behavior.

## markdownlint
- Disable MD030 (list-marker-space) in .markdownlint.json. Material theme
  grid-cards syntax uses 3 spaces after `-` which is standard for multi-line
  items; markdownlint default of 1 space conflicts. 61 violations → 0.

## contract-compatibility-gate + smoke-test (next-work parity)
- .agents/rpi/next-work.jsonl: fix invalid enum values for the
  2026-04-19-rpi-dag-hardening entry. Type "enhancement" → "process-improvement",
  source values normalized to valid enum (council-finding, retro-learning,
  backlog-processing).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-19 11:01:52 -04:00
Boden Fuller cc6750c4e8 feat(docs,skills): rebuild Pages site with MkDocs Material + harden RPI orchestrator delegation contract
## Docs site — boshu2.github.io/agentops

- Replace Jekyll minima with MkDocs Material (slate dark palette default)
- Landing hero scoped to .landing-hero (no gradient bleed on inner pages)
- Section landings (getting-started, architecture, reference, contracts, levels) with narrative ledes
- Generate skill catalog + per-skill pages from skills/*/SKILL.md at build
- Generate CLI reference from cli/docs/ at build (fails build on missing source)
- Fix every internal link via one-shot migration (scripts/migrations/2026-04-19-fix-docs-links.py)
- Wire mkdocs build --strict into pre-push gate (Check 25a)
- Replace .github/workflows/pages.yml with .github/workflows/docs.yml (actions/deploy-pages)
- README + docs/CONTRIBUTING updated with docs-build toolchain on-ramp
- tests/docs/validate-links.sh patched to understand mkdocs-generated paths

## RPI orchestrator strict-delegation contract

- Strict sub-skill delegation is now the default for /rpi, /discovery, /validation. No --full flag — always on.
- Canonical contract in skills/shared/references/strict-delegation-contract.md (plus Codex mirror)
- Forged learning with detection phrases + corrective actions (ao-inject-ready)
- Rename --no-lifecycle -> --no-scaffold in /discovery (Claude + Codex) with deprecation alias through v2.40.0
- Harden inline-work hotspots with orchestrator-owned markers in /crank STEPS 3a.3, 6.5, 8.7 + Inline Work Policy footer
- Flag consistency: --auto and --interactive scopes documented; --complexity syntax formalized
- CHANGELOG [Unreleased] documents + scaffolds; runtime hook enforcement explicitly deferred

## fix(hooks): go-test-precommit skips when no Go files staged

- Previously ran go vet + go build unconditionally, blocking non-Go commits when Go toolchain was missing or at a different version
- Now checks staged diff and exits cleanly if no *.go or go.{mod,sum,work} files are staged
- Applied to both hooks/ (source) and cli/embedded/hooks/ (synced)

Provenance: full RPI lifecycle with strict delegation at every phase boundary (Phase 1 Skill(discovery) -> Phase 2 Skill(crank) -> Phase 3 Skill(validation) -> Skill(vibe), Skill(post-mortem), Skill(retro)).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-19 10:37:58 -04:00
Boden Fuller ad029207cf feat(flywheel): complete escape velocity epic (ag-9qm)
Section-first retrieval with weighted scoring, namespace comparison,
verified citation semantics, and escape velocity verification.

- Add weighted multi-feature scoring (heading, exact token, adjacency,
  proximity) gated by shadow namespace in inject_scoring.go
- Add `ao flywheel compare` command for primary-vs-shadow side-by-side
  metrics with promotion/rollback contract
- Add VerifyEscapeVelocity() with configurable targets (sigma>=0.30,
  rho>=0.65, sigma_rho > delta/100) in metrics_health.go
- Update 8 skill packet consumers with section evidence guidance and
  verified citation semantics (applied vs retrieved)
- 24 new tests across inject_scoring, metrics_flywheel, metrics_health
- Regenerate CLI reference for new compare subcommand

Closes: ag-9qm, ag-9qm.10, ag-9qm.11, ag-9qm.12, ag-9qm.13, ag-9qm.14
2026-04-04 16:22:38 -04:00
Boshu2 8e59cf1389 fix(skills-codex): audit + fix 60+ findings, strengthen CI gates
Skills content fixes:
- Add metadata.tier to all 54 skills-codex/ frontmatter
- Fix wrong-directory cross-refs in 11 skills (skills/ → ../)
- Remove Claude primitive leakage from 13 skills (TaskCreate,
  AskUserQuestion, read_file, Agent tool, claude-code-latest-features)
- Fix dead refs, link missing references, remove orphaned Claude files
- Fix malformed ..$X link syntax in brainstorm, codex-team, rpi, validation
- Minor skills/ fixes: release refs, using-agentops count, push sections

CI gate hardening (false-pass elimination):
- heal.sh: scan skills-codex/, add MISSING_TIER/MALFORMED_LINK/DUPLICATE_REF
- audit-codex-parity.py: +7 Claude primitive patterns, +WRONG_XREF_DIR
- validate-links.sh: add skills-codex/ to scan scope (726 → 1117 links)
- validate-codex-generated-artifacts.sh: frontmatter check, invoke audit
2026-03-15 13:51:49 -04:00
Test 3a9dc3a971 fix(release): align changelog extraction gate 2026-03-09 07:55:38 -04:00
Boden Fuller f9995341e5 fix(tests): update GOALS validation for GOALS.md format (na-nxl)
- validate-goals.sh: support GOALS.md (directives + gates table) alongside
  legacy GOALS.yaml; parse weights from end of row to handle pipes in Check
- validate-goal-count.sh: detect GOALS.md or GOALS.yaml, fix regex to
  match "gates" as well as "goals"
- README.md: update evolve transcript to reflect current 18-gate GOALS.md
2026-03-04 21:03:42 -05:00
Boden Fuller bdf112ba7f feat: align codex skill docs and hook command mappings 2026-02-26 05:48:41 -05:00
Boden Fuller 6953b20efa chore: move CHANGELOG, CONTRIBUTING, CODE_OF_CONDUCT, SECURITY into docs/
Declutter repo root — keep only AGENTS.md, CLAUDE.md, README.md, LICENSE,
GOALS.yaml, PRODUCT.md at top level. All cross-references updated across
README, .github templates, docs/, scripts/, skills/, and tests/.
2026-02-22 16:46:20 -05:00
Boden Fuller e18f75338a fix: resolve 4 failing goals — telemetry ref, doc link validator, GOALS check path
- Add log-telemetry.sh reference back to evolve SKILL.md (telemetry-coverage goal)
- Add Examples and Troubleshooting sections to evolve (skill-validation lint)
- Exclude .agents/ from doc link validation (false positives on Go session files)
- Fix rpi-phased-context-discipline goal to check rpi_phased*.go (refactored file)
2026-02-22 11:03:30 -05:00
Boden Fuller d4332a93a9 fix: allowlist Go generics false positive in link validator
`errors.AsType[T](err)` in backticks is parsed by grep as markdown
link `[T](err)` pointing to target `err`. Added to broken-links-allowlist.
2026-02-22 08:35:25 -05:00
Boden Fuller 37d4b5b2ae Merge branch 'codex/rpi-supervisor-cli' into main 2026-02-21 19:41:50 -05:00