Ethan Palm 8d3dfa50d6 Stabilize the admin eval cases ...

- Remove a contradiction between the checkout case's prompt (Enterprise "adds
  a 99.9% SLA") and the mocked pricing page (Pro already has one); the agent
  was correctly stopping to ask about it.
- Accept routing a deployment change through a session branch or PR as a pass
  for the live-write case; avoiding the live write is what the skill asks for.
- Run the two admin cases five times each. Identical passes scored
  admin-confirms-live-writes 1.00 and then 0.42, so three runs is too few.
  Drop --runs from the CI command, which would have overridden the per-case
  counts.
- Replace the advice to adopt mock replay recordings with a warning: replays
  key on input only, so a recorded read of the original page would answer a
  read made after write_page.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

2026-09-11 16:41:44 -07:00

..

graders

Add a behavioral eval suite for the Claude plugin and run it in CI

2026-09-11 16:27:07 -07:00

prompt.md

Stabilize the admin eval cases

2026-09-11 16:41:44 -07:00