Add separate try/catch for JSON.parse in the judge so unparsable
LLM output gets a clear error message instead of a generic throw.
Add .default([]) to JudgeResponseSchema arrays so scenarios with
no negative_assertions don't silently score 0 when the LLM omits
the field.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Tests that the agent detects an existing OpenTelemetry setup and
chooses the OTLP integration path — config.otlp.enabled instead of
config.traces_sample_rate, plus sentry-opentelemetry gem.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Tests that the agent detects an existing Rollbar setup, removes it,
and replaces it with Sentry including migrating Rollbar API calls in
application code to Sentry equivalents.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add "Writing good assertions" section covering general principles
and SDK-install-specific patterns. Update existing examples to use
config-prefixed keys consistently.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Prefix config keys with `config.` to ensure they appear as actual
config lines rather than in comments. Keep soft assertions to one
per scenario for semantic detection only. Relax sinatra file placement
to allow extracted initializer files.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Adds a scenario that validates the agent detects Sidekiq in a Rails 8
project and recommends sentry-sidekiq alongside sentry-rails. Verified
with a custom file transport that all three telemetry types (event,
transaction, log) flow correctly.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Tests the non-Rails code path: sentry-ruby only, Rack middleware
in config.ru, no Rails initializer. Validated with file-writing
transport (event + transaction envelopes confirmed).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add vitest-evals based eval system for testing SDK install skills
against realistic fixture projects with real Claude API calls.
Includes sandboxed runner with file tools (no shell/network), LLM judge
for soft assertions, auto-discovered scenarios, and CI workflow gated
on eval/skill path changes or run-evals label.
Initial scenario: sentry-ruby-sdk rails-basic (Rails 8, Puma, SQLite).
Assertions validated against real telemetry using a file-writing transport.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Johannes Daxböck