Files
Johannes Daxböck bd808e84bf ref(evals): Tighten file assertions, minimize soft assertions
Prefix config keys with `config.` to ensure they appear as actual
config lines rather than in comments. Keep soft assertions to one
per scenario for semantic detection only. Relax sinatra file placement
to allow extracted initializer files.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-06-15 15:00:38 -07:00

35 lines
1020 B
JSON

{
"name": "rails-basic",
"given": "A Rails 8 application with Puma and SQLite, no existing monitoring",
"prompt": "Set up Sentry for this Rails application. Add error monitoring, tracing, and logging. Use the SENTRY_DSN environment variable.",
"fixture": "rails-basic",
"files": [
{
"path": "config/initializers/sentry.rb",
"contains": [
"Sentry.init",
"config.dsn",
"SENTRY_DSN",
"config.traces_sample_rate",
"config.enable_logs",
"config.breadcrumbs_logger",
"config.send_default_pii"
],
"excludes": ["otlp"]
},
{
"path": "Gemfile",
"contains": ["sentry-ruby", "sentry-rails"],
"excludes": ["sentry-sidekiq", "sentry-opentelemetry"]
}
],
"should_not_exist": [
"config/initializers/rollbar.rb",
"config/initializers/honeybadger.rb"
],
"soft_assertions": [
{ "criterion": "The agent detected Rails as the framework and chose sentry-rails" }
],
"negative_assertions": []
}