Stand up the approach for giving evals a Sentry MCP without talking to real
Sentry: run the real @sentry/mcp-server over stdio and intercept its
*.sentry.io API traffic with mitmproxy, answering from a Python addon. The
agent exercises the real MCP tool surface; only the Sentry REST API beneath it
is mocked — and that addon is where each task will seed responses and state.
Routing is a transparent iptables REDIRECT of 443/80 to mitmproxy (excluding
its own uid); the mitmproxy CA is trusted both system-wide and by Node via
NODE_EXTRA_CA_CERTS. A fake SENTRY_ACCESS_TOKEN makes the server believe it's
connected so it issues real API calls. Requires --cap-add=NET_ADMIN.
Verified end to end: curl, a Node fetch, and the real sentry-mcp server calling
find_organizations are all intercepted. Standalone POC under harness/mcp-proxy/;
not yet wired into run.sh. Next: a complete correctly-shaped default addon plus
per-task seeding.
Add a per-skill eval system for the skills-next tree, built on waza. Each
eval lives next to the skill it tests
(skills-next/skills/<skill>/evals/<name>/) and shares one Docker-isolated
harness under skills-next/evals/harness/.
run-docker.sh builds the shipped skill tree with a real plugin builder
(plugin-src/<builder>/build.sh, default claude) against CONTENT_ROOT=skills-next,
stages a minimal context, and runs the agent in a sandbox where the host repo
is never mounted. Skills reach the agent via waza's skill_directories config
(injected at runtime by run.sh) — not ~/.copilot/skills, which waza ignores.
The agent runs under OpenRouter gpt-5.5 (no Copilot auth needed); MCP is
deferred.
The first eval (sentry-get-started/first-error) carries three tasks: the full
first-error flow (new-node-express, pending MCP wiring) and two account-branch
tasks that pass today without MCP — no-account-signup and
has-account-needs-mcp — which validate orient -> detect-not-connected ->
correct branch -> don't-fabricate.
Also exclude per-skill evals/ from copy_skills so eval material (tasks,
fixtures, grader prompts) never ships in a built plugin or leaks to the agent
under test.
Task-shaped skills will draw their reference material (per-platform SDK
guides, per-signal concepts) from a single maintained library rather
than duplicating it. Each shipped skill must still be self-contained, so
the build copies a skill's declared references into it.
Add a CONTENT_ROOT env var (default: repo root) so the same build steps
can assemble an alternate skill tree, e.g. CONTENT_ROOT=skills-next.
hydrate-references.py walks a skills directory: for every skill with a
references.yml it brace-expands and globs the `needs:` patterns against
the library, copies the matches into the skill's references/ preserving
subpaths, and strips the manifest so it never ships. It runs via `uv run
--script` with inline PyYAML and braceexpand dependencies. build-common.sh
invokes it once per build, and only when some skill declares references,
so existing skills are unaffected and the common case spawns no
interpreter.
Factor the steps shared across all four agent builders into
build-common.sh (content-root resolution, skill copy+hydrate, commands,
SKILL_TREE), sourced by each plugin-src/<agent>/build.sh.
Begin consolidating the 19 separate per-platform SDK skills into a single
sentry-sdk skill that carries a bundled reference tree per SDK under
references/[sdk]/ instead of routing to standalone skills.
This commit scaffolds the two foundational files:
- SKILL.md: a brief overview of what Sentry can capture (error monitoring,
tracing, logging, profiling, metrics, crons, session replay, user feedback,
AI monitoring), the SDK catalog mapping each platform to
references/[sdk]/index.md, and the platform-detection and quick-lookup tables
carried over from the sentry-sdk-setup router. It is framed as the single HOW
layer that both first-error setup and add-a-signal land on, with a distinct
full-setup scope for when the user wants sensible defaults.
- STRUCTURE.md: an internal contributor contract for the references/[sdk]/ tree
-- the canonical signal set and filenames, the WHAT-vs-HOW boundary (strategy
stays in the concept skills), the index.md section format, and the per-signal
file format.
No SDK references are migrated yet and no existing skills are removed; this is
the scaffold the migration builds on.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Several SDK skills embedded full source-map / debug-symbol tutorials
that duplicate the improve-setup skills: the generic auth-token creation
walkthrough ("go to settings/auth-tokens, create a token with these
scopes"), CI pipeline YAML (GitHub Actions), env/.gitignore boilerplate,
and EAS secret orchestration. The get-started design doc is explicit
that the SDK skill should carry the platform-specific note and point to
sentry-source-maps for the generic setup, not reimplement it.
Keep the genuinely SDK-coupled mechanics in each skill (withSentryConfig
authToken, the Vite/webpack plugin config, MSBuild upload properties,
the sentry-cli build step, React Native's per-platform symbol upload
table) and the verification notes, then replace the generic steps with a
one-line prerequisite and a cross-link to sentry-source-maps (and
sentry-releases for .NET's release properties).
Also folds React Native's Source Maps & Debug Symbols and EAS Build
Hooks sections under Configuration Reference, restoring its canonical
top-level skeleton.
Expand the consolidated WHY/DO/HOW/USE skill set: per-signal concept
references, the full per-platform sentry-sdk reference tree, setup
references (instrumenting, verify, hardening, monitors, alerts), and the
workflow intent references. Checkpoint before reworking the skill
architecture.
feat(skills): add /sentry-get-started and restructure into five routers
Reorganize the skill tree around a single guided entry point. Previously the
three routers (sdk-setup / workflow / feature-setup) assumed the user already
knew their intent, there was no orientation layer, and SDK setup never
confirmed an event actually reached Sentry.
Add a `/sentry-get-started` command that orients the user from a cheap probe
(MCP auth, repo already using Sentry, has a project) and routes into the right
skill -- sending brand-new users straight into first-error setup and showing an
existing user the relevant menu.
Move to five visible routers so every menu item maps 1:1 to a router the user
can invoke directly later (get-started is just a first-run concierge, never the
sole path to anything): sdk-setup, add-signal, improve-setup, workflow,
monitors. Retire feature-setup and redistribute its skills.
Separate WHAT/WHY from HOW: new platform-agnostic concept skills (tracing,
logging, metrics, crons, profiling, session-replay, user-feedback) cover
strategy and best practices, while the SDK skill remains the source of truth
for platform mechanics and is entered scoped to a capability.
Add sentry-verify-instrumentation, a shared hidden utility that closes the loop
-- trigger a real event and confirm it landed via the MCP -- so no SDK skill
reimplements verification.
Several new improve-setup (source-maps, releases, data-scrubbing,
reduce-volume, enable-ai-review) and monitors (uptime, metric-alerts,
dashboards) leaves are intentionally stubs to flesh out in follow-up passes.
Extend build-skill-tree.sh for the new categories/sections, regenerate
SKILL_TREE.md, and update AGENTS.md. All four plugin builds validate.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The SDK skills had drifted into two conventions for the same section:
15 of 19 used the heading "Configuration Reference" while 4 used the
short "Config Reference", and 14 of 19 placed it before "Verification"
while the same 4 placed it after. Align the four outliers (cloudflare,
dotnet, nextjs, node) with the majority so every SDK skill reads
Guide -> Configuration Reference -> Verification -> Cross-Link.
Pure reorder and rename; no content changed.
The mobile and native SDK skills had accreted extra top-level sections
over time (Environment Variables, Production Settings, Ecosystem
Integrations, Integration Reference, Expo Config Plugin Reference,
Default Auto-Enabled Integrations, SvelteKit File Summary), breaking the
canonical Phase 1-4 + Configuration Reference + Verification skeleton
that every SDK skill is meant to follow.
Relocate them without changing any content:
- Config-reference-grade material (env vars, default integrations,
production hardening, file summaries) folds into Configuration
Reference as ### subsections, matching the backend SDK skills.
- Longer standalone deep-dives (Android integrations, Flutter ecosystem
packages, RN Expo config plugin) move to references/ files, pointed to
from the Phase 3 feature table.
The React Native source-map and EAS Build sections are intentionally
left in place; a follow-up trims those. Pure relocation, no prose
changed.
Installing the Sentry plugin was a one-way door — there was no built-in
way to take it back out. Add a `remove` subcommand that mirrors install:
it reuses the detect and select steps, then uninstalls the plugin from
each chosen agent (claude plugin uninstall, codex plugin remove, grok
plugin uninstall, and deleting the Cursor checkout).
Removal only takes out our own plugin. Any marketplace we registered is
left in place — Claude's is Anthropic's shared official one, so
deregistering it could pull it out from under other plugins — and the
install-time cleanup of conflicting plugins does not run.
The UI is factored into a shared runFlow driven by a FlowMode descriptor
so install and remove share detection, selection, concurrency, and
output streaming while differing only in their tagline, prompt copy,
per-agent action, and closing line. The remove flow leads with a static
"Neuralyzing your agents of Sentry." header (no shimmer).
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Ships the updated installer README (install command + demo) to the npm
package page. craft's "simple" changelog policy requires the entry to exist
before prepare.
Aligns Claude's plugin.json with Cursor, Codex, and Grok which all
include ?utm_source=plugin in their distributed MCP configs, so
Claude-originated traffic is attributed under app.utm_source:plugin.
Documents the two telemetry surfaces for sentry-for-ai:
- Primary: sentry/mcp-server spans filtered by app.utm_source:plugin
(Cursor, Codex, Grok attributed; Claude gap noted)
- Secondary: installer CLI errors in project 4511570959335425
Includes query recipes, investigation pivots, domains, configuration,
and attribute notes following the sentry-mcp/junior TELEMETRY.md pattern.
Co-authored-by: sentry-junior[bot] <264270552+sentry-junior[bot]@users.noreply.github.com>
The preReleaseCommand failed in craft's container with "pnpm: command not
found" -- the image ships npm but not pnpm. Switch to `npm pkg set version`,
a plain package.json edit with no install, lockfile, or git-check side
effects.
Add a tagline under the banner and rework the closing line so the run
opens and closes on the same beat ("teaching your agents to speak
Sentry" -> "your agents now speak Sentry") instead of a flat restart
hint.
On a color terminal the tagline animates: a bright band shimmers across
"speak Sentry" while the install runs, then settles to a static line.
It is painted as a header by a custom listr renderer (ShimmerRenderer)
so the step list renders normally beneath it; off a TTY (CI, pipes,
NO_COLOR) it stays a plain static line. The shimmer effect lives in its
own module, text-shimmer.ts.
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
craft's default version bump targets the repo-root package.json, which is
private and unversioned. The published package lives in packages/installer,
so its version stayed at 0.1.0 and npm rejected the 0.1.1 publish as a
duplicate. Wire a preReleaseCommand that bumps packages/installer instead.
When an install streams through an output sink, runStreaming piped the
command's stderr to the live view but never captured it, so a failure
fell through to runInstallCommand's generic "exit code N" message. The
interactive UI always passes a sink, so Listr failure text was almost
always that generic message instead of the agent's actual error.
Tee stderr: keep piping it to the sink for the live view, and also
buffer it into the result so the thrown error carries the real text.
Show what each agent is actually doing instead of a static spinner. An
optional output sink threads from the install task down through
install/update/cleanup to the shell runner; the real system spawns the
command and pipes its stdout/stderr into the task's stream, so progress
appears live under each agent's row. Detection probes still buffer and
parse as before — only the mutating commands stream.
Theme the renderer to match the Sentry banner: diamond task icons with a
small-to-large pulsing diamond spinner, streamed output grayed (per line,
so blank lines stay empty and removeEmptyLines can drop them) and kept on
screen via persistentOutput, an arrow prompt cursor with the gap moved
onto the checkbox icons to preserve alignment, and a blue hollow-diamond
prompt prefix. Also drop the Cursor restart note now that the closing
line covers it.
The install subtasks run under a parent "Installing plugins" task, which
collapses its children once it completes — folding away each agent's
resolved command and cleanup note. Set collapseSubtasks: false so the
per-agent rows and their output stay visible after the group finishes.
The closing line said "Restart your AI tools to load the Sentry plugin",
which is vague when only some of the detected agents were installed.
List the agents that actually received the plugin instead, e.g. "Restart
Claude Code and Codex to use Sentry with AI". Blocked or failed agents
are excluded since there is nothing to restart for them.
Switch every harness's installed-probe from substring-grepping plain
`plugin list` output to parsing the `--json` variant and matching on the
plugin's real identity. Claude and Codex match a marketplace-qualified
id; Grok matches our repo source with no marketplace, since the source
alone cannot tell a direct repo install from one made through the "xAI
Official" marketplace.
Add an optional `cleanup` step that runs before install to remove a
conflicting or legacy Sentry plugin that would otherwise shadow ours:
Codex removes the curated `sentry@openai-curated`, and Grok uninstalls a
marketplace-installed sentry so our direct-repo install resolves.
`cleanup` returns a short description of what it removed, which the
installer surfaces as persistent task output so the removal is visible
rather than silent.
Split the single `install` method into explicit `install` and `update`
so the orchestrator picks based on what detection already found, instead
of each harness re-probing its own listing.
Add the Craft release pipeline so packages/installer can be published to
npm as @sentry/ai.
- .craft.yml configures a single npm target (public scoped access) with a
tarball-name guard. There is deliberately no github target, so a release
creates no git tags and no GitHub releases -- nothing repo-wide implying
the skills/plugins are versioned by it.
- The release branch is scoped to `release-installer/<version>` via
releaseBranchPrefix, so it reads as a package release. craft creates it
during prepare and deletes it after publish, so it is transient.
- changelogPolicy is `simple` (matching sentry-javascript) and the
changelog lives with the package at packages/installer/CHANGELOG.md. The
entry is written by hand before releasing; `auto` would sweep this repo's
skill-drift and plugin churn into the installer changelog.
- release.yml runs `craft prepare` via the Sentry release bot App token.
It must be an App token rather than GITHUB_TOKEN so the pushed release
branch triggers release-build.yml, which packs the tarball and uploads
it as the SHA-named artifact Craft's GitHub provider expects.
- The package gains `files: ["dist", "CHANGELOG.md"]`, publishConfig.access
public, and an explicit LICENSE so the published tarball is complete
without relying on pnpm copying the root LICENSE.
Publishing runs from getsentry/publish after a team lead approves the
release issue; this repo only prepares the release and builds the artifact.
Add a pnpm workspace with a `packages/installer` package (`@sentry/ai`)
that provides a single CLI for installing the Sentry plugin into any
supported AI coding assistant: Claude Code, Codex, Cursor, and Grok.
The CLI (built on citty) exposes an `install` subcommand. By default it
shows an interactive selector pre-selecting the detected agents;
`--no-interactive`/`--yes` skips the prompt and installs every detected
agent — used for CI and unattended runs.
Each agent is a self-contained harness implementing a common strategy
interface — detect, isInstalled, canInstall, install — so adding a new
agent is just another module. System access (shell, fs, platform,
homedir) is injected through a SystemDeps seam, keeping every harness
unit-testable without side effects.
Behavior:
- Already-installed agents update in place rather than erroring or
no-opping: Cursor pulls, Grok/Claude use their update commands, Codex
re-adds (idempotent; it has no update command).
- Claude refreshes its marketplace index before installing, since a fresh
CLI ships a stale index that cannot resolve the plugin.
- canInstall gates prerequisites (Cursor needs git) and reports a clear
skip instead of a cryptic failure.
- One agent failing does not abort the rest, and the CLI exits non-zero
if any selected agent failed or was blocked.
- Cross-platform: binary detection uses where/which per OS, Cursor is
located via its per-OS install path, and git paths are quoted for
Windows spaces.
Distribution:
- rolldown bundles the npm package (deps external).
- fossilize builds standalone Node SEA binaries; --hole-punch drops
non-English ICU data to cut the download ~30%.
CI:
- A smoke-test matrix (Linux, macOS, Windows) installs the real agent
CLIs and runs the installer non-interactively, asserting the plugin
lands for every detected agent.
Instrumented with the Sentry Node SDK.
The hardcoded skill-to-team mapping in this workflow is now handled by
CODEOWNERS, which gives GitHub native reviewer assignment without a
separate Actions job.
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
* [skill-drift] update(sentry-react-sdk): document dataCollection option (v10.55.0)
Automated drift-fix run.
Co-Authored-By: Claude (claude-sonnet-4-5) <noreply@anthropic.com>
* update(sentry-react-sdk): simplify dataCollection snippets
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* update(sentry-react-sdk): align dataCollection docs with defaults
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* update(sentry-react-sdk): correct SDK version to 10.57.0
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
---------
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Claude (claude-sonnet-4-5) <noreply@anthropic.com>
Co-authored-by: Michael Hoffmann <michi@sentry.io>
* [skill-drift] update(sentry-react-router-framework-sdk): migrate sendDefaultPii to dataCollection API
Automated drift-fix run.
Co-Authored-By: Claude (claude-sonnet-4-5) <noreply@anthropic.com>
* update(sentry-react-router-framework-sdk): simplify dataCollection snippets
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* update(sentry-react-router-framework-sdk): align dataCollection docs with defaults
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
---------
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Claude (claude-sonnet-4-5) <noreply@anthropic.com>
Co-authored-by: Michael Hoffmann <michi@sentry.io>
* [skill-drift] fix(sentry-svelte-sdk): replace deprecated sendDefaultPii with dataCollection
Automated drift-fix run.
Co-Authored-By: Claude (claude-sonnet-4-5) <noreply@anthropic.com>
* fix(sentry-svelte-sdk): simplify dataCollection snippets
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix(sentry-svelte-sdk): align dataCollection docs with defaults
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix(sentry-svelte-sdk): correct SDK version to 10.57.0
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
---------
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Claude (claude-sonnet-4-5) <noreply@anthropic.com>
Co-authored-by: Michael Hoffmann <michi@sentry.io>
The xAI plugin marketplace scanner (generate-plugin-index.py) resolves
plugin manifests from MANIFEST_PATHS = ['.grok-plugin/plugin.json',
'.claude-plugin/plugin.json']. With the manifest at the repo root it was
silently ignored — components were still discovered via default paths, but
any manifest-level overrides (e.g. logo) would not have been picked up.
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
- Link each supported agent in the intro to its distribution repo.
- Note that the skill library is also browsable at skills.sentry.dev and
installable via the skills.sh installer.
* [skill-drift] fix(sentry-cocoa-sdk): app start limit + SentryObjC docs
Automated drift-fix run.
Co-Authored-By: Claude (claude-sonnet-4-5) <noreply@anthropic.com>
* fix(sentry-cocoa-sdk): address PR feedback on app start limit and SentryObjC docs
- Fix inconsistent 5s vs 180s app start threshold in troubleshooting table
- Remove experimental tag from SentryObjC (stable since 9.17.0)
- Bump SentryObjC min version to 9.17.1 for debug symbol support
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
---------
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Claude (claude-sonnet-4-5) <noreply@anthropic.com>
Co-authored-by: Philip Niedertscheider <phil.niedertscheider@sentry.io>
Evan Purkhiser