Files
getsentry__sentry-for-ai/evals/vitest.config.ts
Johannes Daxböck 5402298cfc feat(evals): Add eval infrastructure for SDK install skills
Add vitest-evals based eval system for testing SDK install skills
against realistic fixture projects with real Claude API calls.

Includes sandboxed runner with file tools (no shell/network), LLM judge
for soft assertions, auto-discovered scenarios, and CI workflow gated
on eval/skill path changes or run-evals label.

Initial scenario: sentry-ruby-sdk rails-basic (Rails 8, Puma, SQLite).
Assertions validated against real telemetry using a file-writing transport.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-06-15 13:25:08 -07:00

20 lines
528 B
TypeScript

import { defineConfig } from "vitest/config";
const jsonOutputFile = process.env["VITEST_EVALS_JSON"];
export default defineConfig({
test: {
include: ["src/**/*.eval.ts"],
exclude: ["**/node_modules/**", "**/dist/**"],
setupFiles: ["./src/setup.ts"],
includeTaskLocation: true,
reporters: [
["vitest-evals/reporter", { toolDetails: false }],
...(jsonOutputFile ? ([["json"]] as const) : []),
],
outputFile: {
...(jsonOutputFile ? { json: jsonOutputFile } : {}),
},
},
});