Three hundred lines of regex over prose, and the only evidence they caught
anything was a set of breakages applied by hand and reverted. A green run
could not be told apart from a check that matched nothing.
Fifteen tests copy the real repository to a temporary directory, break one
thing, and assert that the validator names it: a dangling section number,
a rule id in a list, a strategy letter, a renamed named-rule heading, a
case rule with a bad id, a bad quotation, or a missing heading, a sibling
reference path, an orphan, the line limit, the description limit, and a
mismatched name. The first test asserts the repository as committed
passes, and one checks that code fences are skipped.
validate.yml runs them after the validator and now also triggers on any
change under .github/scripts.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The validator only resolved `references/<file>.md` paths written in
SKILL.md. Reference files point at each other in 28 places, and none of
those were checked, so deleting or renaming a reference would break the
SKILL.md route loudly and the sibling routes silently.
Every document is now scanned for reference paths. The frontmatter check
also enforces the Agent Skills limits of 64 characters for `name` and 1024
for `description`; the description is at 906 and the body at 497 of 500
lines, and both are now printed so the headroom is visible on every run.
The validator is reshaped into an exported validateRepository(root) that
returns the problems it found, with the CLI entry point guarded behind an
argv check, so a test can run it against a mutated copy of the repository.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Section and rule numbers are how these documents address each other, and
nothing noticed when a renumbering or a renamed heading left one dangling.
evals/README.md said as much about the `rule` field: keep it in sync by
hand. That is a job for a script.
The validator now resolves `Section N`, `Rule N-M`, `Step N`, `Strategy X`,
`Snapshot N`, `Part N`, and `Question N` against the headings and bold
labels of the skill package, and checks each `;`-separated fragment of a
case's `rule` the same way. A named rule cited from more than one file must
be an anchor somewhere, so renaming it breaks the build instead of the
reader.
The existing documents and all 19 cases pass unchanged.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Nothing in this repository checked the thing the skill exists for. The
validator confirms the documents are well formed, and the install job
confirms the package installs, but no check asks whether an agent reading
these rules places code correctly. That gap is why several contradictions in
this branch survived until someone read the files side by side.
Add `evals/cases.json`: ten placement questions with the expected answer, the
regression each one guards, and the file and rule that decides it. They cover
the rules this branch changed, including both halves of the business-logic
test, since an agent that over-applies Rule 4-5 fails the negative case while
passing the positive one.
Grading needs a model, so CI does not grade. What CI can do is stop the cases
from rotting: every `source` must resolve, every case needs all six fields,
ids must be unique, and the file must parse. A case citing the reference file
that the earlier split removed is caught by this, which is the failure mode
worth guarding.
`evals/README.md` documents the manual procedure and when a case earns its
place. Add `evals/**` to the workflow path filters so edits there run CI.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The validator enforced two things: the SKILL.md body line count, and that
every reference path named in SKILL.md resolves. Three failures it could not
see are all silent ones, where the package still installs and nothing looks
wrong until an agent is already using it.
- A frontmatter `name` that disagrees with the directory name. The installed
skill is addressed by its directory, so the mismatch routes to nothing.
- A missing or empty `description`. That field is the only text an agent
reads when deciding whether to activate the skill.
- A `references/*.md` that SKILL.md never mentions. It ships with the package
but no routing rule in Conditional References can reach it, so it costs
install size and gets no reads. This one guards the split in the previous
commits, where a renamed reference could have left the old file behind.
Each check was confirmed to fire by breaking it deliberately and to stay
quiet on the current tree.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Gaic4o