After WAITING_URL is detected, wait 5s and check authsome list — if the
provider is already connected, resume the session without interrupting the
user. Also marks evals 4-6 optional and bumps skill version to 0.2.0.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Allows evals to specify an expected agent-initiated interrupt (e.g. the
agent asks the user a clarifying question) and a canned reply to auto-send
when that interrupt is detected, enabling the runner to continue the session
without human input.
- Parse script now emits AGENT_INTERRUPTED=true when the last assistant
message ends with '?'
- Runner step d gains Case 1: if expected_interrupt is set and
AGENT_INTERRUPTED=true, auto-resume with next_turn_instruction
- Eval 6 (ClickUp) wired up with both new fields
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Remove evals/run_evals.py (replaced by inline orchestration in
run-evals.md), bump version to 0.3.2, update lockfile, and add
authsome skill command file.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Replace the Agent-tool / self-reported transcript approach with a proper
subprocess invocation of `claude --verbose --output-format stream-json`,
which produces a structured event stream including tool calls and results.
A new `_parse_stream_json` helper converts the event stream into a readable
transcript for the LLM judge.
Also adds `--dangerously-skip-permissions` so the eval agent runs without
interactive approval prompts, and updates the /run-evals skill to:
- Run the script directly for all agent types (no Agent tool detour)
- Fix state mismatches inline via authsome commands instead of asking the user
- Only pause for user help when human action is genuinely required
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Replace the old setup guide (wipe state, init from scratch) with a
lighter flow: sync latest CLI, verify the existing identity is healthy,
create a fresh profile for the session, switch to it, confirm clean
state with authsome list.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Move Agentic Installation (was id 2) to last position (id 7) and mark
it optional — it invalidates stored identities so it should run after
all scenario tests.
Add --profile and --run-dir flags to run_evals.py so the /run-evals
command can orchestrate evals one at a time, managing profile lifecycle
and accumulating results into a single run directory. Remove hardcoded
input() prompts — Claude handles the state-check conversation natively.
Add .claude/commands/run-evals.md: the /run-evals slash command. Claude
reads authsome state, asks the user to confirm or fix it, then calls
the Python script per eval with --profile and --run-dir.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Create a fresh authsome profile at the start of each eval run so tests
don't pollute the user's real credentials. Tear it down (delete key
files, restore previous profile) at the end via try/finally.
Before each eval, capture `authsome list` output and show it in the
banner so the user can verify state matches the expected environment.
Always prompt for Enter before starting — a single consistent gate
that also handles human-in-the-loop evals.
Include authsome_state in grading.json and render it as a collapsible
"authsome state at start" section in the HTML report.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
claude -p no longer works without an API key (billing change). Switch
grading to `hermes chat -Q` which uses the user's configured provider.
Add a preflight check that smoke-tests hermes before running any evals.
Strip the session_id line hermes prepends in quiet mode before parsing
the JSON verdict.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Switch grading from anthropic SDK to `claude -p` subprocess to avoid
a hard dependency. Truncate transcript to last 4000 chars to stay within
CLI arg limits. Grade partial transcripts even when rate limit is hit
(emit WARNING instead of skipping).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Ankit Ranjan