The score table was only visible in the job summary. Post it as one comment
per PR, found by a marker and updated in place on every run, with a link to
the run for the full HTML report. The job gets pull-requests: write for this;
the rest of the workflow keeps contents: read.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The suite exercises the shared agent context through the Claude Code plugin,
the only target with an eval harness. Rename the job to say so and put the
coverage caveat at the top of the job summary, so nobody reads a green eval as
covering the Codex, Cursor, or Kiro agents.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The operation and setting names in the Admin mock are stand-ins so the mock has
an API to play; they are not the real Admin server's.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
- Remove a contradiction between the checkout case's prompt (Enterprise "adds
a 99.9% SLA") and the mocked pricing page (Pro already has one); the agent
was correctly stopping to ask about it.
- Accept routing a deployment change through a session branch or PR as a pass
for the live-write case; avoiding the live write is what the skill asks for.
- Run the two admin cases five times each. Identical passes scored
admin-confirms-live-writes 1.00 and then 0.42, so three runs is too few.
Drop --runs from the CI command, which would have overridden the per-case
counts.
- Replace the advice to adopt mock replay recordings with a warning: replays
key on input only, so a recorded read of the original page would answer a
read made after write_page.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
agent-context/evals/ holds seven `claude plugin eval` cases with graders and
mocks for both Mintlify MCP servers. Five cases test skill knowledge (docs.json
shape, Columns vs CardGroup, frontmatter and links, page modes, a negative
case); two test the Admin MCP workflow the skill documents (checkout before
edits, confirm before live writes) against mocked servers.
The new eval-claude-plugin job in agent-context-ci.yml generates the Claude
plugin, copies the suite in, validates the plugin, and runs the suite with
pinned models on pull requests from this repository. It is a soft gate: scores
land in the job summary and an artifact but do not fail the check. It needs an
ANTHROPIC_API_KEY secret and reports when one is missing. Only the Claude
target has an eval harness; this measures the shared content.
Measured before this change, against a no-plugin baseline: Columns +0.53,
page modes +0.25, keywords frontmatter roughly +0.67; docs.json +0.06 because
the model already knows it.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The previous commit generated plugin.json at the repository root, which
`claude plugin validate` accepts but `claude plugin eval` ignores: the plugin
still resolved by directory name with no version. Only
.claude-plugin/plugin.json sets the name and version.
Adds an optional per-target pluginManifestFile (plugin.json or
.claude-plugin/plugin.json), used by both the build and the repository sync,
and points the claude target and the sync workflow at the new path. Kiro is
unchanged.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The claude target only shipped .claude-plugin/marketplace.json, so Claude
Code resolved the plugin by directory name. A plugin.json pins the name to
"mintlify" regardless of where the repo is checked out, which the eval
suite's MCP tool-name graders depend on. Uses the existing pluginManifest
mechanism from the kiro target; `claude plugin validate` accepts the output.
Also ignores agent-context/evals/results/, written by every eval run.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* docs: sync Mintlify skill with new component, OpenAPI file uploads, and playground JS API
* docs: split long sentence in runtime server variables section
---------
Co-authored-by: mintlify[bot] <109931778+mintlify[bot]@users.noreply.github.com>
* docs: fix ParamField signature and fence nesting in components skill
ParamField examples used a dot-prefixed path prop (path="query.limit")
that contradicts the documented signature, where the prop name is the
parameter location and its value is the parameter name. CodeGroup,
RequestExample/ResponseExample, and View examples nested fenced code
inside a 3-backtick outer fence, which CommonMark closes early; switch
to 4-backtick outer fences like the existing Mermaid example.
Fixes#6897
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* docs: fix CodeGroup fence in Mintlify skill
* docs: refresh contextual menu identifiers in Mintlify skill
* docs: correct page mode theme support in Mintlify skill
* docs: refresh CLI coverage in Mintlify skill
* docs: complete API language list in Mintlify skill
* docs: document SDK reference navigation in Mintlify skill
---------
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
Ethan Palm