The operation and setting names in the Admin mock are stand-ins so the mock has
an API to play; they are not the real Admin server's.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
- Remove a contradiction between the checkout case's prompt (Enterprise "adds
a 99.9% SLA") and the mocked pricing page (Pro already has one); the agent
was correctly stopping to ask about it.
- Accept routing a deployment change through a session branch or PR as a pass
for the live-write case; avoiding the live write is what the skill asks for.
- Run the two admin cases five times each. Identical passes scored
admin-confirms-live-writes 1.00 and then 0.42, so three runs is too few.
Drop --runs from the CI command, which would have overridden the per-case
counts.
- Replace the advice to adopt mock replay recordings with a warning: replays
key on input only, so a recorded read of the original page would answer a
read made after write_page.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
agent-context/evals/ holds seven `claude plugin eval` cases with graders and
mocks for both Mintlify MCP servers. Five cases test skill knowledge (docs.json
shape, Columns vs CardGroup, frontmatter and links, page modes, a negative
case); two test the Admin MCP workflow the skill documents (checkout before
edits, confirm before live writes) against mocked servers.
The new eval-claude-plugin job in agent-context-ci.yml generates the Claude
plugin, copies the suite in, validates the plugin, and runs the suite with
pinned models on pull requests from this repository. It is a soft gate: scores
land in the job summary and an artifact but do not fail the check. It needs an
ANTHROPIC_API_KEY secret and reports when one is missing. Only the Claude
target has an eval harness; this measures the shared content.
Measured before this change, against a no-plugin baseline: Columns +0.53,
page modes +0.25, keywords frontmatter roughly +0.67; docs.json +0.06 because
the model already knows it.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Ethan Palm