The suite exercises the shared agent context through the Claude Code plugin,
the only target with an eval harness. Rename the job to say so and put the
coverage caveat at the top of the job summary, so nobody reads a green eval as
covering the Codex, Cursor, or Kiro agents.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
agent-context/evals/ holds seven `claude plugin eval` cases with graders and
mocks for both Mintlify MCP servers. Five cases test skill knowledge (docs.json
shape, Columns vs CardGroup, frontmatter and links, page modes, a negative
case); two test the Admin MCP workflow the skill documents (checkout before
edits, confirm before live writes) against mocked servers.
The new eval-claude-plugin job in agent-context-ci.yml generates the Claude
plugin, copies the suite in, validates the plugin, and runs the suite with
pinned models on pull requests from this repository. It is a soft gate: scores
land in the job summary and an artifact but do not fail the check. It needs an
ANTHROPIC_API_KEY secret and reports when one is missing. Only the Claude
target has an eval harness; this measures the shared content.
Measured before this change, against a no-plugin baseline: Columns +0.53,
page modes +0.25, keywords frontmatter roughly +0.67; docs.json +0.06 because
the model already knows it.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Ethan Palm