Johannes Daxböck 5402298cfc feat(evals): Add eval infrastructure for SDK install skills ...

Add vitest-evals based eval system for testing SDK install skills
against realistic fixture projects with real Claude API calls.

Includes sandboxed runner with file tools (no shell/network), LLM judge
for soft assertions, auto-discovered scenarios, and CI workflow gated
on eval/skill path changes or run-evals label.

Initial scenario: sentry-ruby-sdk rails-basic (Rails 8, Puma, SQLite).
Assertions validated against real telemetry using a file-writing transport.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

2026-06-15 13:25:08 -07:00

45 KiB

Raw Permalink Blame History

View Raw