Ethan Palm 91e786ee60 Add a behavioral eval suite for the Claude plugin and run it in CI ...

agent-context/evals/ holds seven `claude plugin eval` cases with graders and
mocks for both Mintlify MCP servers. Five cases test skill knowledge (docs.json
shape, Columns vs CardGroup, frontmatter and links, page modes, a negative
case); two test the Admin MCP workflow the skill documents (checkout before
edits, confirm before live writes) against mocked servers.

The new eval-claude-plugin job in agent-context-ci.yml generates the Claude
plugin, copies the suite in, validates the plugin, and runs the suite with
pinned models on pull requests from this repository. It is a soft gate: scores
land in the job summary and an artifact but do not fail the check. It needs an
ANTHROPIC_API_KEY secret and reports when one is missing. Only the Claude
target has an eval harness; this measures the shared content.

Measured before this change, against a no-plugin baseline: Columns +0.53,
page modes +0.25, keywords frontmatter roughly +0.67; docs.json +0.06 because
the model already knows it.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

2026-09-11 16:27:07 -07:00

..

graders

Add a behavioral eval suite for the Claude plugin and run it in CI

2026-09-11 16:27:07 -07:00

prompt.md

Add a behavioral eval suite for the Claude plugin and run it in CI

2026-09-11 16:27:07 -07:00