mirror of
https://github.com/getsentry/sentry-for-ai.git
synced 2026-09-14 14:28:22 +08:00
Evan Purkhiser
8016554e33
Add a per-skill eval system for the skills-next tree, built on waza. Each eval lives next to the skill it tests (skills-next/skills/<skill>/evals/<name>/) and shares one Docker-isolated harness under skills-next/evals/harness/. run-docker.sh builds the shipped skill tree with a real plugin builder (plugin-src/<builder>/build.sh, default claude) against CONTENT_ROOT=skills-next, stages a minimal context, and runs the agent in a sandbox where the host repo is never mounted. Skills reach the agent via waza's skill_directories config (injected at runtime by run.sh) — not ~/.copilot/skills, which waza ignores. The agent runs under OpenRouter gpt-5.5 (no Copilot auth needed); MCP is deferred. The first eval (sentry-get-started/first-error) carries three tasks: the full first-error flow (new-node-express, pending MCP wiring) and two account-branch tasks that pass today without MCP — no-account-signup and has-account-needs-mcp — which validate orient -> detect-not-connected -> correct branch -> don't-fabricate. Also exclude per-skill evals/ from copy_skills so eval material (tasks, fixtures, grader prompts) never ships in a built plugin or leaks to the agent under test.