The score table was only visible in the job summary. Post it as one comment
per PR, found by a marker and updated in place on every run, with a link to
the run for the full HTML report. The job gets pull-requests: write for this;
the rest of the workflow keeps contents: read.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The suite exercises the shared agent context through the Claude Code plugin,
the only target with an eval harness. Rename the job to say so and put the
coverage caveat at the top of the job summary, so nobody reads a green eval as
covering the Codex, Cursor, or Kiro agents.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
- Remove a contradiction between the checkout case's prompt (Enterprise "adds
a 99.9% SLA") and the mocked pricing page (Pro already has one); the agent
was correctly stopping to ask about it.
- Accept routing a deployment change through a session branch or PR as a pass
for the live-write case; avoiding the live write is what the skill asks for.
- Run the two admin cases five times each. Identical passes scored
admin-confirms-live-writes 1.00 and then 0.42, so three runs is too few.
Drop --runs from the CI command, which would have overridden the per-case
counts.
- Replace the advice to adopt mock replay recordings with a warning: replays
key on input only, so a recorded read of the original page would answer a
read made after write_page.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
agent-context/evals/ holds seven `claude plugin eval` cases with graders and
mocks for both Mintlify MCP servers. Five cases test skill knowledge (docs.json
shape, Columns vs CardGroup, frontmatter and links, page modes, a negative
case); two test the Admin MCP workflow the skill documents (checkout before
edits, confirm before live writes) against mocked servers.
The new eval-claude-plugin job in agent-context-ci.yml generates the Claude
plugin, copies the suite in, validates the plugin, and runs the suite with
pinned models on pull requests from this repository. It is a soft gate: scores
land in the job summary and an artifact but do not fail the check. It needs an
ANTHROPIC_API_KEY secret and reports when one is missing. Only the Claude
target has an eval harness; this measures the shared content.
Measured before this change, against a no-plugin baseline: Columns +0.53,
page modes +0.25, keywords frontmatter roughly +0.67; docs.json +0.06 because
the model already knows it.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The previous commit generated plugin.json at the repository root, which
`claude plugin validate` accepts but `claude plugin eval` ignores: the plugin
still resolved by directory name with no version. Only
.claude-plugin/plugin.json sets the name and version.
Adds an optional per-target pluginManifestFile (plugin.json or
.claude-plugin/plugin.json), used by both the build and the repository sync,
and points the claude target and the sync workflow at the new path. Kiro is
unchanged.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Ethan Palm