Add separate try/catch for JSON.parse in the judge so unparsable
LLM output gets a clear error message instead of a generic throw.
Add .default([]) to JudgeResponseSchema arrays so scenarios with
no negative_assertions don't silently score 0 when the LLM omits
the field.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Tests that the agent detects an existing OpenTelemetry setup and
chooses the OTLP integration path — config.otlp.enabled instead of
config.traces_sample_rate, plus sentry-opentelemetry gem.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Tests that the agent detects an existing Rollbar setup, removes it,
and replaces it with Sentry including migrating Rollbar API calls in
application code to Sentry equivalents.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add "Writing good assertions" section covering general principles
and SDK-install-specific patterns. Update existing examples to use
config-prefixed keys consistently.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Prefix config keys with `config.` to ensure they appear as actual
config lines rather than in comments. Keep soft assertions to one
per scenario for semantic detection only. Relax sinatra file placement
to allow extracted initializer files.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Adds a scenario that validates the agent detects Sidekiq in a Rails 8
project and recommends sentry-sidekiq alongside sentry-rails. Verified
with a custom file transport that all three telemetry types (event,
transaction, log) flow correctly.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Tests the non-Rails code path: sentry-ruby only, Rack middleware
in config.ru, no Rails initializer. Validated with file-writing
transport (event + transaction envelopes confirmed).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add vitest-evals based eval system for testing SDK install skills
against realistic fixture projects with real Claude API calls.
Includes sandboxed runner with file tools (no shell/network), LLM judge
for soft assertions, auto-discovered scenarios, and CI workflow gated
on eval/skill path changes or run-evals label.
Initial scenario: sentry-ruby-sdk rails-basic (Rails 8, Puma, SQLite).
Assertions validated against real telemetry using a file-writing transport.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This repository is the single source of truth for skills and routing metadata,
but it is not itself an installable plugin, and each assistant (Claude Code,
Cursor, Codex, Grok) expects the plugin in a different on-disk shape. No plugin
ecosystem can consume a zip or release asset; all install from a git ref. So a
GitHub Actions workflow builds each agent's distribution from this repository
and deploys it to a dedicated repository whose root is exactly that agent's
plugin, which a marketplace points at as getsentry/plugin-<agent>.
Each agent has a `plugin-src/<agent>/` dir with its manifests, README, and a
`build.sh` that assembles the dist tree from shared repo content. Claude, Cursor,
and Grok consume the plugin at the repo root; Codex requires it under
`plugins/sentry/` and rejects the `disable-model-invocation` field the skill
tree relies on, so its build strips that field and emits a per-skill
`agents/openai.yaml` (see `hide-skills.py`). MCP config is built from the repo's
`mcp.json` source of truth: inline in Claude's manifest, and a file for the
others.
Cross-repo writes use a short-lived token from a dedicated GitHub App scoped to
contents:write on the plugin repos; the default GITHUB_TOKEN cannot push to
other repositories.
The root `.claude-plugin/`, `.cursor-plugin/`, and `.mcp.json` are retained so
existing installs that consumed the plugin from this repo's root keep working;
their READMEs announce removal on or after 2026-07-11.
Map each SDK skill directory to the GitHub team that owns the
corresponding SDK repository. This ensures the right team gets
requested for review when skill content changes.
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
* add span streaming migration skill
* platform agnostic
* fix: Use RegExp instead of glob pattern in ignoreSpans example
String matching in ignoreSpans uses substring includes, not glob
patterns. The `*` was treated as a literal character.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* feedback
* Update skills/sentry-span-streaming/SKILL.md
Co-authored-by: Lukas Stracke <lukas.stracke@sentry.io>
---------
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-authored-by: Lukas Stracke <lukas.stracke@sentry.io>
* feat(instrumentation-guide): add sentry-instrumentation-guide skill
A decision-framework skill that helps an agent (or developer) choose which
Sentry signal to reach for when instrumenting code — error, span, span
attribute, log, or metric — and why the choice decides what's debuggable later.
Distilled from the "Errors, Traces, Logs, Metrics: When to Reach for What" post.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(instrumentation-guide): add reference deep-dives
choosing-signals.md: per-signal detail, the four overlap tiebreakers, the
sampling-vs-filtering rationale, and the "wide events" argument.
instrumentation-examples.md: one handler instrumented end to end in Python and
JS/TS, showing span attribute vs log vs metric on the same decision.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs(skills): register sentry-instrumentation-guide
Add to the sentry-feature-setup router (decision bullet + table row), the
AGENTS.md and README feature tables, and regenerate SKILL_TREE.md.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* refactor(instrumentation-guide): defer signatures to sibling SDK skills
This skill decides WHAT signal to emit; the sentry-*-sdk skills own HOW
(setup, API signatures, enabling flags, min versions) in their
references/{tracing,logging,metrics}.md. Stop hardcoding docs.sentry.io
deep links and a Node-specific link in the language-neutral examples/
sampling sections — point at the matching SDK skill by name instead, so
there's one maintained source of truth and nothing to go stale.
Kept: top-level product-area links (one per signal) for orientation.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* instrumentation-guide: align signal naming with the blog post
- Four-signals table and deep dive lead with Traces (spans named as the unit)
- Gut-check "reach for" column shows Traces / Spans
- Metrics question -> "How's this trending over time?"
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The plan job's UNKNOWN-skill check used `jq --argjson ... --argjson ...
'$wanted | map(...)'` without `-n` and without stdin. jq waited for
stdin, got EOF, applied the filter zero times, and produced no output —
silently yielding UNKNOWN="" rather than UNKNOWN="[]".
Then `jq 'length' <<<""` also produced no output, so the comparison
`[[ "" != "0" ]]` evaluated true and the workflow died with a
false-positive "Unknown skill(s) in workflow_dispatch input: " error
on every dispatch (run 26625830364).
Add `-nc` so jq treats null as the input, applies the filter once,
and produces the expected empty array. Also guard the length check
with `${UNKNOWN:-[]}` so a future empty value can't trip the same
false positive.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(skill-drift): replace gh-aw workflow with matrix-fanout Claude runs
Replaces .github/workflows/skill-drift-check.{md,lock.yml} (single
sequential gh-aw run) with a plain GitHub Actions workflow that fans out
one Claude Sonnet session per SDK skill via matrix strategy.
Each per-skill job reads the last 7 days of merged PRs in the
corresponding SDK repo, opens the actually changed source files at the
merge SHA (not just the diff summary), compares against the local skill
bundle, and emits either a file diff under skills/<skill>/ or a
structured summary for the apply job to act on.
Security
- All actions pinned to a full commit SHA.
- permissions: {} at workflow root; each job grants only what it needs.
- The agent job (detect) runs with contents: read ONLY — no write tokens
reachable from the LLM session.
- Agent tool allowlist restricts Bash to a handful of gh/jq/date
subcommands; no git, curl, or arbitrary shell.
- The apply job is pure deterministic shell with a path allowlist
(skills/<this-skill>/ only) enforced before any commit, push, or PR.
- Inline reviewer assignment in the apply job since PRs opened by
GITHUB_TOKEN do not trigger downstream workflow_run events.
- Trigger surface is schedule + workflow_dispatch only.
Matrix lives in .github/skill-drift-matrix.json so adding a new SDK skill
is a one-line change.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(skill-drift): address review feedback in apply job
Four issues flagged on PR #135:
1. (Cursor Bugbot, HIGH) The `artifacts/` directory placed by
download-artifact was being picked up by `git ls-files --others
--exclude-standard` in the apply job's path-allowlist check, which
would have downgraded EVERY create_pr to an issue because no
`artifacts/...` path matches the `^skills/<skill>/` regex. Scope
both `git diff` and `git ls-files` to `-- skills/` so anything
outside that subtree (artifacts/, dotfiles, etc.) is invisible to
the allowlist scan.
2. (Sentry Bot, HIGH) On the path-allowlist and empty-diff bail-out
paths the script switched branches but did not remove the untracked
files left behind by `rsync`. The next iteration would then see
those files via `git ls-files --others` and trigger a false-positive
allowlist violation, cascading the failure across all remaining
skills. Add `git clean -fd -- skills/` at the start of each
create_pr iteration and on both bail-out paths.
3. (Cursor Bugbot, MEDIUM) `git diff --quiet HEAD` does not detect
brand-new untracked files. If the agent's only change was adding a
new `references/*.md`, the empty-diff guard would have silently
skipped the PR. Replace with an `[[ -z "$TOUCHED" ]]` check, since
TOUCHED already includes untracked files.
4. (Sentry Bot, MEDIUM) Parsing the PR number from `gh pr create`
stdout via `basename` is fragile — extra text would corrupt it and
the downstream JS reviewer-assignment step would silently skip.
Look up the number via `gh pr view "$BRANCH" --json number --jq
.number` instead, and log a `::warning::` (plus increment FAILED)
if it still cannot be determined. Also add a warning log in the JS
step when an invalid PR number is encountered.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(skill-drift): second review pass — length caps, retries, pagination, edge cases
Addresses the round of review comments left on commit fea5d4a:
1. (sentry-warden, security) Untrusted SDK PR content reaches issue/PR
bodies via `gh issue create --body "$BODY"` and `gh pr create --body`.
Add a 256-char cap on `title` and a 10000-char cap on `body`, enforced
in both detect's validate step and apply's re-validation. Prepend a
GitHub `> [!WARNING]` alert header to every issue/PR body the agent
produces so human reviewers see "Auto-generated from external SDK
content. Review all links and code suggestions before acting on
them." before reading agent-authored text. Limits the blast radius
of a successful prompt-injection from an SDK PR.
2. (sentry-bot, MEDIUM) Race condition between `gh pr create` and
`gh pr view`. GitHub's API is eventually consistent — the immediate
view can 404 and silently skip reviewer assignment. Retry up to 5
times with a 1s sleep; warn (not silently skip) if all attempts fail.
3. (cursor-bot, MEDIUM) `actions/download-artifact@v4` fails the step
when its pattern matches zero artifacts, before the apply script's
graceful empty-artifact branch can run. Mark the download step
`continue-on-error: true`. The apply script already handles a
missing `artifacts/` directory and now also emits a complete set of
step-outputs (prs/issues/no_drift/failed/opened_pr_nums) on the
no-artifacts early-exit so downstream conditions see consistent
values.
4. (cursor-bot, LOW) `github.rest.pulls.listFiles` in the inline
reviewer-assignment step only returns the first page. Use
`github.paginate(..., per_page: 100)` so PRs with many files don't
miss teams. Matches the convention used by
skill-drift-assign-reviewers.yml.
5. (cursor-bot, LOW) Trailing/leading/double commas in the
`workflow_dispatch` `skills` input parsed as empty tokens, which
then failed the unknown-skill check. Drop empty entries in the jq
pipeline.
6. (cursor-bot, MEDIUM) `title` and `body` were validated as strings
but empty strings passed, which would have broken `git commit -m ""`
and yielded useless empty issues. Require non-empty values for
`create_pr`/`create_issue` (still allowed for `no_drift`, which the
apply job ignores entirely).
WONTFIX: the "hardcoded `main`" comment. This workflow is pinned to
`getsentry/sentry-for-ai` via `if: github.repository ==` guards on
every job; the default branch IS `main`. Parameterizing
`github.event.repository.default_branch` adds complexity for a
portability story we don't have.
Also: update the agent prompt with the new length caps and non-empty
requirement so the agent's output isn't silently rejected by
validation.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Metrics is GA. Remove all beta, open beta, and experimental status
markers from metrics references and SKILL.md files for Android, Flutter,
Go, NestJS, Next.js, Node, PHP, Python, and the SDK skill creator.
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
The Weekly SDK Skill Drift Detector workflow runs on a `schedule:`
trigger, so there is no `pull_request` context. The agent's `add_reviewer`
safe output therefore failed every run with:
add_reviewer: No pull_request_number provided and not in pull request context
turning the job red even when the PR and issues were created successfully
(see run https://github.com/getsentry/sentry-for-ai/actions/runs/25321035855).
The step was also redundant: `.github/workflows/skill-drift-assign-reviewers.yml`
already maps changed `skills/sentry-*-sdk/**` paths to the correct team and
requests review per skill (more granular than the agent's union-of-teams call).
Changes:
- Drop `add-reviewer:` from the `safe-outputs:` block.
- Remove Step 4a item 5 telling the agent to call `add_reviewer`.
- Add a short note pointing the agent at the dedicated reviewer-assignment
workflow so it knows not to try to assign reviewers itself.
- Recompile `skill-drift-check.lock.yml` with gh-aw v0.71.1 (was v0.67.1) so
CI uses the latest compiler and pinned actions.
Note: the assign-reviewers workflow does not currently fire for these PRs
because gh-aw opens them with `GITHUB_TOKEN`, which by design does not
trigger downstream `pull_request` events. That is a separate issue and is
not addressed here.
Co-authored-by: Claude <noreply@anthropic.com>
Modern Android projects centralize dependencies in
gradle/libs.versions.toml. The skill now:
- Detects the catalog file and any pre-existing Sentry entries during
Phase 1, and flags catalog usage in the decision table.
- Adds a "Using a Gradle Version Catalog" section with TOML entries for
the SDK, BOM, Gradle plugin, and all optional integrations, plus the
matching `libs.*`/`alias(libs.plugins.*)` references for Kotlin and
Groovy DSL.
- Covers two common catalog pitfalls in troubleshooting: alias naming
(hyphen in TOML vs. dot in build files) and avoiding duplicate
version keys.
Co-authored-by: Claude <noreply@anthropic.com>
Recompile the skill-drift-check workflow with the latest gh-aw CLI.
Key upgrades:
- gh-aw: v0.64.2 → v0.67.1
- AWF (firewall): v0.25.1 → v0.25.13
- MCP Gateway: v0.2.6 → v0.2.14
The previous run (#24033021813) failed because the apiKeyHelper script
was not accessible inside the AWF chroot jail, causing Claude Code to
immediately fail with 'authentication_failed'. The newer AWF version
includes 12 patch releases that address container/chroot reliability.
Closes#91
Documents the new runtime metrics integrations introduced in sentry-javascript
(getsentry/sentry-javascript#19923, #19979) which were missing from the skill.
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: claude-sonnet-4-6 <noreply@anthropic.com>
* feat: skill drift workflow opens PRs with fixes and assigns SDK teams
Extend the weekly skill-drift-check workflow to directly fix straightforward
drift by opening PRs instead of only creating issues. The agent now:
- Prefers opening PRs for mechanical changes (new config options, version
bumps, new integration entries) with the fix applied inline
- Falls back to issues for complex/risky drift (breaking API removals,
ambiguous behavior, coordinated rewrites)
- Uses add_reviewer to assign the appropriate SDK team (from the mapping
table) as reviewer on each PR
- PRs are opened as ready-for-review (not draft) with auto-labels and
14-day expiry
Frontmatter changes:
- Added create-pull-request safe output (max 10, non-draft, protected
supply-chain files fall back to issues)
- Added add-reviewer safe output restricted to the 5 SDK team slugs
- Recompiled lock.yml with gh-aw v0.64.2 (was v0.55.0)
Co-Authored-By: Claude Sonnet 4 <noreply@anthropic.com>
* feat: auto-assign SDK team reviewers on skill-drift PRs
Add a lightweight workflow that triggers on PR open for skill-drift labeled
PRs. It checks which skills/sentry-*-sdk/ directories were modified and
requests review from the corresponding SDK team via a hardcoded mapping.
This removes the dependency on the agentic workflow remembering to call
add_reviewer — team assignment happens deterministically based on file
paths regardless of how the PR was created.
Co-Authored-By: Claude Sonnet 4 <noreply@anthropic.com>
* fix: use org-prefixed team slugs for reviewer assignment
The requestReviewers API needs the full org/team-slug format
(e.g., getsentry/team-javascript-sdks) to resolve teams correctly.
Co-Authored-By: Claude Sonnet 4 <noreply@anthropic.com>
---------
Co-authored-by: Claude Sonnet 4 <noreply@anthropic.com>
* feat(ai-monitoring): add sampling guidance and Node.js AI monitoring reference
Add sampling strategy reference for gen_ai spans (JS + Python) covering:
- Head-based sampling gotcha (root-only, children inherit)
- tracesSampler patterns for both standalone agents and HTTP-child scenarios
- Metrics/logs fallback for when 100% tracing isn't feasible
- Cost comparison and troubleshooting
Also adds:
- Node.js ai-monitoring.md reference (was missing, Python/Next.js/NestJS had one)
- Sampling check + user prompt to SKILL.md detection flow
- Sampling cross-links to Python, Next.js, and NestJS references
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix(ai-monitoring): trim verbose skill references, update model names
- sampling.md: 242 → 78 lines. Removed 4 redundant code blocks (JS+Python
metrics+logs shown separately), consolidated into one Python example + note
that JS uses same pattern. Removed comparison table.
- ai-monitoring.md (Node): 341 → 95 lines. Removed per-integration
setup/wrapping/streaming sections, consolidated into single setup block
with important notes callout.
- Updated model names from gpt-4o to claude-sonnet-4-6.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix(node-sdk): bring ai-monitoring reference up to Python's depth
Match Python SDK reference structure: add Integration Matrix with min
library versions, Configuration Examples (auto + explicit + Vercel AI +
browser), 4 manual instrumentation examples (chat, invoke_agent,
execute_tool, handoff), Span Attribute Reference tables, Agent Workflow
Hierarchy diagram, Streaming compatibility table. Updated model names.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix(node-sdk): add unsupported providers table to ai-monitoring
Lists providers without auto-instrumentation in Node (Pydantic AI,
OpenAI Agents SDK, HuggingFace Hub, MCP are Python-only; Cohere,
Bedrock, Mistral, Groq need manual spans in both).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix(node-sdk): remove Python-only providers from Node unsupported table
Pydantic AI, OpenAI Agents SDK, HuggingFace Hub are Python libraries -
not relevant to Node developers. MCP is a separate product with its own
docs. Keep only providers a Node developer might actually encounter:
Cohere, Bedrock, Mistral, Groq.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix(node-sdk): address PR review - remove handoff span, fix model name
- Remove gen_ai.handoff example (planned for deprecation per @obostjancic)
- Fix openai("claude-sonnet-4-6") → openai("gpt-4.1") (wrong provider/model combo)
- Remove handoff from span hierarchy diagram
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix(node-sdk): use gen_ai.request not gen_ai.chat in examples
gen_ai.chat is an internal SDK op. The docs document gen_ai.request
for manual instrumentation. Fixed all code examples and span hierarchy.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* feat: Add generate-bug-report workflow skill
Adds a bug report generator that scans the codebase and opens a
comprehensive report in the browser.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix: Update bug-report URL to bug-report.sentry.dev
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix: Add cross-platform browser open commands (Linux, Windows)
Follows existing pattern from sentry-sdk-setup skill. Also widens
allowed-tools to permit xdg-open and start.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix: Enable model invocation for generate-bug-report skill
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* feat: Add April 1st date gate, silent exit codes, and project fingerprint query params
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Johannes Daxböck