First live run of the claude agent (all prior verification used codex)
failed instantly: skillgrade containers run as root, and Claude Code
refuses --dangerously-skip-permissions as root unless IS_SANDBOX=1 marks
the environment as a sandbox, which a skillgrade container is. A claude
wrapper in docker.setup sets it, mirroring the codex wrapper. Without this
every claude run fails in ~1s, including the canonical
pnpm eval:skill:setup flow.
Auth is whatever the forwarded .env provides: ANTHROPIC_API_KEY (Console)
or CLAUDE_CODE_OAUTH_TOKEN from claude setup-token (subscription billing).
Verified live with a subscription token: PASS 1.00, 8/8 deterministic, 61s
(roughly 5x faster than the codex trials).
Installs the codex CLI in the eval image unconditionally, working around
skillgrade 0.1.5 only baking the YAML-default agent into the Dockerfile
(which made the --agent CLI override fail with command-not-found). With
both CLIs present, an OpenAI-only contributor runs the eval against the
canonical claude-default config via:
echo "OPENAI_API_KEY=sk-..." > skill-evals/copilotkit-setup/.env
pnpm exec skillgrade --trials=1 --agent=codex --grader=deterministic
The --grader filter skips the anthropic-keyed rubric; skillgrade
renormalizes reward over the graders that run, so deterministic-only still
scores 0..1 against the same threshold. Verified live: PASS 1.00 with the
key sourced from .env and the YAML untouched.
npx copilotkit skills install runs the skills CLI against skills/, which
copies each skill directory into user projects wholesale (its only excludes
are metadata.json, .git, __pycache__). That would ship eval.yaml and the
vite-react fixture to every user, and the fixture's package.json would get
indexed by GitHub's dependency graph in their repos. The harness now lives
in skill-evals/copilotkit-setup/, pointed back at the skill under test via
skillgrade's skill: field; skills/copilotkit-setup/ contains only shippable
content.
Also instructs eval agents not to start dev servers: a live trial timed out
at 600s because codex booted vite and the Express backend to verify its
(complete) solution, and codex exec never returned. Verified end-to-end
from the new location: PASS 1.00 (8/8 deterministic + rubric, 324s).
Austin Merrick