Tests that the agent detects an existing OpenTelemetry setup and
chooses the OTLP integration path — config.otlp.enabled instead of
config.traces_sample_rate, plus sentry-opentelemetry gem.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Tests that the agent detects an existing Rollbar setup, removes it,
and replaces it with Sentry including migrating Rollbar API calls in
application code to Sentry equivalents.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Prefix config keys with `config.` to ensure they appear as actual
config lines rather than in comments. Keep soft assertions to one
per scenario for semantic detection only. Relax sinatra file placement
to allow extracted initializer files.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Adds a scenario that validates the agent detects Sidekiq in a Rails 8
project and recommends sentry-sidekiq alongside sentry-rails. Verified
with a custom file transport that all three telemetry types (event,
transaction, log) flow correctly.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Tests the non-Rails code path: sentry-ruby only, Rack middleware
in config.ru, no Rails initializer. Validated with file-writing
transport (event + transaction envelopes confirmed).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add vitest-evals based eval system for testing SDK install skills
against realistic fixture projects with real Claude API calls.
Includes sandboxed runner with file tools (no shell/network), LLM judge
for soft assertions, auto-discovered scenarios, and CI workflow gated
on eval/skill path changes or run-evals label.
Initial scenario: sentry-ruby-sdk rails-basic (Rails 8, Puma, SQLite).
Assertions validated against real telemetry using a file-writing transport.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Johannes Daxböck