mirror of
https://github.com/boshu2/agentops.git
synced 2026-09-14 15:08:13 +08:00
Bo
10f0277bdb
## Legible membrane, Train 2: what a stranger meets Provenance: the 2026-09-02 field audit of this repo against mattpocock/skills, compound-engineering, and the jsm corpus, findings F5 through F9. This train is sized by a consumer inventory built with `rg` on the tip before any lane was written; the promoted-set directory move the audit proposed is deferred because that inventory shows skill-builder backing two blocking gates, swarm pinned by the cathedral gate and a routing golden, using-gc required by Go code, and `ao skills link` unable to install a second root. That inventory is the plan for a later train. **What changes.** - **Archival sweep by consumer disposition.** 172 audit snapshots, 29 pawl receipts, the `evals/workbench` and `evals/membrane` trees with their two bats consumers, four stray scratch docs, four retired eval contracts, and nine caller-less `scripts/check-*.sh` are deleted; git history is the archive. Every machine list that referenced them is pruned (evidence-grounding baseline, preamble grandfather, broken-links allowlist, `.gitattributes`, `.gitignore`, two eval fixtures, the workflow-coverage deferred list). `docs/audits/manifests/` and `.agents/ao/config.yaml` survive because they have live readers. About 48,000 lines. - **Three skills retired.** `goals` (alias of fitness), `shared` (tombstone), and `scope` (folded into plan step 3 as five write-scope checks). Consumers edited; the probe denominator exclusion for goals pruned; Codex package and golden count pins updated. - **Negative routing** on research, codebase-recon, reverse-engineer, premortem, one-way-door, and council, all within the 180-char budget, with a teardown golden (`rq-08`). One wording was changed after the router's prefix stemming showed "repository teardown" leaking into the wrong skill. - **Every promoted skill answers "It's working if"** with observable tells in backticks, and carries a paste-ready `## Prompt` with a concrete subject. Two fictional `ao` subcommands a draft prompt named were caught by the body-ref validator and replaced with real commands. - **Doctrine diet on the core five.** rpi, plan, implement, validate, and anti-ceremony drop from about 5,100 words to 3,600 (bodies from 4,700 to 3,150) by moving the shared ownership boundary, dated incidents, and mechanics tables into step-loaded references (`skills/rpi/references/boundaries.md`, `skills/validate/references/mechanics.md`, `skills/plan/references/ground-truth-routing.md`). Every cathedral canary and every skill validator grep survives unchanged. - **ADR-0018** records the goals, shared, and scope retirement; the cathedral gate tombstone and the routing goldens cite it instead of ADR-0017. - **Router and twins.** `ao skills find` holds a description's "Not for X; that is <sibling>." sentence out of its haystack, so premortem no longer ranks first for "is this live decision reversible" (golden `rq-10` pins the reciprocal of `rq-02`); a penalty variant was tried and reverted because it suppressed skills the caller named outright. A declared trigger phrase of two or more words quoted whole in the query now earns the name weight once, so "check this change" lands on validate rather than on reality-check's name token; a live-catalog test pins seven such queries. Single-quoted YAML descriptions unescape `''`. The Codex catalog keeps the exclusion sentence, and a closing `>` no longer turns `<run-id>/codebase-recon.json` into an invocation. - **Residue the judges found.** handoff, learn, and status open a `## Contract` heading after their tells; validate's prompt names its helper at `skills/validate/scripts/validate.py`; the explicit-skill prompt catalog names only live skills (five stale prompts replaced by nine, floor 20 restored, TESTING.md names the suite); the corpus-delta receipt binds the runner's path and SHA-256 and labels a `live_agent` claim as an unverified caller declaration; the probe README and ledger describe the 12-skill denominator; SKILL-API counts 30 of 54. **Evidence on the tip.** Regen check clean; full gate green with a HEAD-built binary; CI's bats command green; Go build/vet/test green; lint clean; security gate quick PASS; docs-build warnings did not rise. Fresh validation by Fable 5.1 (caller-elected) and a cross-family read by Codex, both recorded in the PR thread. --------- Co-authored-by: Bo <bofuller55@gmail.com>
1.2 KiB
1.2 KiB
Testing
Use the narrowest deterministic check that proves the changed behavior during an experiment. Run the repository's ordinary full deterministic suite once on the complete candidate.
Core contract checks
python3 -m unittest discover -s skills/validate/tests
./scripts/check-cathedral-cut-conformance.py
python3 scripts/generate-skill-mesh.py --check
cd cli && go test ./...
The Validate helper is also probed in a temporary non-Git directory with no
ao on PATH. Fake git, ao, tracker, push, and delivery executables ensure
identity and verdict helpers do not acquire hidden substrate dependencies.
ao gate check is an ordinary test runner. Its exit status is not a semantic
verdict and does not authorize delivery.
Explicit skill requests
tests/explicit-skill-requests/ holds one prompt per shipped skill that a
caller invokes by name (prompts/<skill>.txt). run-all.sh feeds each prompt
to a fresh session and asserts the named skill is what fires, before any tool
call. The static half runs without a model:
evals/agentops-core/fixtures/explicit-skill-prompts-smoke.sh proves every
prompt names a skill that exists in both skills/ and skills-codex/, so a
retired skill cannot linger as a live prompt.