Add a per-skill eval system for the skills-next tree, built on waza. Each
eval lives next to the skill it tests
(skills-next/skills/<skill>/evals/<name>/) and shares one Docker-isolated
harness under skills-next/evals/harness/.
run-docker.sh builds the shipped skill tree with a real plugin builder
(plugin-src/<builder>/build.sh, default claude) against CONTENT_ROOT=skills-next,
stages a minimal context, and runs the agent in a sandbox where the host repo
is never mounted. Skills reach the agent via waza's skill_directories config
(injected at runtime by run.sh) — not ~/.copilot/skills, which waza ignores.
The agent runs under OpenRouter gpt-5.5 (no Copilot auth needed); MCP is
deferred.
The first eval (sentry-get-started/first-error) carries three tasks: the full
first-error flow (new-node-express, pending MCP wiring) and two account-branch
tasks that pass today without MCP — no-account-signup and
has-account-needs-mcp — which validate orient -> detect-not-connected ->
correct branch -> don't-fabricate.
Also exclude per-skill evals/ from copy_skills so eval material (tasks,
fixtures, grader prompts) never ships in a built plugin or leaks to the agent
under test.
Task-shaped skills will draw their reference material (per-platform SDK
guides, per-signal concepts) from a single maintained library rather
than duplicating it. Each shipped skill must still be self-contained, so
the build copies a skill's declared references into it.
Add a CONTENT_ROOT env var (default: repo root) so the same build steps
can assemble an alternate skill tree, e.g. CONTENT_ROOT=skills-next.
hydrate-references.py walks a skills directory: for every skill with a
references.yml it brace-expands and globs the `needs:` patterns against
the library, copies the matches into the skill's references/ preserving
subpaths, and strips the manifest so it never ships. It runs via `uv run
--script` with inline PyYAML and braceexpand dependencies. build-common.sh
invokes it once per build, and only when some skill declares references,
so existing skills are unaffected and the common case spawns no
interpreter.
Factor the steps shared across all four agent builders into
build-common.sh (content-root resolution, skill copy+hydrate, commands,
SKILL_TREE), sourced by each plugin-src/<agent>/build.sh.
feat(skills): add /sentry-get-started and restructure into five routers
Reorganize the skill tree around a single guided entry point. Previously the
three routers (sdk-setup / workflow / feature-setup) assumed the user already
knew their intent, there was no orientation layer, and SDK setup never
confirmed an event actually reached Sentry.
Add a `/sentry-get-started` command that orients the user from a cheap probe
(MCP auth, repo already using Sentry, has a project) and routes into the right
skill -- sending brand-new users straight into first-error setup and showing an
existing user the relevant menu.
Move to five visible routers so every menu item maps 1:1 to a router the user
can invoke directly later (get-started is just a first-run concierge, never the
sole path to anything): sdk-setup, add-signal, improve-setup, workflow,
monitors. Retire feature-setup and redistribute its skills.
Separate WHAT/WHY from HOW: new platform-agnostic concept skills (tracing,
logging, metrics, crons, profiling, session-replay, user-feedback) cover
strategy and best practices, while the SDK skill remains the source of truth
for platform mechanics and is entered scoped to a capability.
Add sentry-verify-instrumentation, a shared hidden utility that closes the loop
-- trigger a real event and confirm it landed via the MCP -- so no SDK skill
reimplements verification.
Several new improve-setup (source-maps, releases, data-scrubbing,
reduce-volume, enable-ai-review) and monitors (uptime, metric-alerts,
dashboards) leaves are intentionally stubs to flesh out in follow-up passes.
Extend build-skill-tree.sh for the new categories/sections, regenerate
SKILL_TREE.md, and update AGENTS.md. All four plugin builds validate.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The preReleaseCommand failed in craft's container with "pnpm: command not
found" -- the image ships npm but not pnpm. Switch to `npm pkg set version`,
a plain package.json edit with no install, lockfile, or git-check side
effects.
craft's default version bump targets the repo-root package.json, which is
private and unversioned. The published package lives in packages/installer,
so its version stayed at 0.1.0 and npm rejected the 0.1.1 publish as a
duplicate. Wire a preReleaseCommand that bumps packages/installer instead.
* docs(flutter-sdk): add profiling, logging, metrics, session-replay references
Add four reference files for the Flutter SDK skill covering features that
require precise documentation of platform constraints and limitations:
- profiling.md: iOS/macOS only, Alpha status, SDK ≥7.12.0. Documents the
profilesSampleRate/tracesSampleRate relationship, simulator vs. real device
caveats, Dart symbol upload requirements, and troubleshooting table.
- logging.md: Two paths — direct Sentry.logger API (SDK ≥9.5.0) and the
sentry_logging integration bridging the Dart logging package. Documents
LoggingIntegration configuration params (minBreadcrumbLevel, minEventLevel,
minSentryLogLevel), stack trace gotchas with Logger.root, scope-level
attributes via withScope, beforeSendLog filtering, and trace correlation.
- metrics.md: Open Beta, SDK ≥9.11.0 for trace-connected metrics. Honest about
the missing 'set' type, 2 KB per-metric key limit, and no per-metric sampling.
Covers all three supported types (counter/distribution/gauge) with unit
enums, tag constraints, and practical Dart examples.
- session-replay.md: iOS/Android only (not Web/macOS/Linux/Windows), SDK ≥9.0.0,
~1 fps screenshot-based capture. Documents options.replay.* and
options.privacy.* APIs, SentryWidget requirement, third-party widget masking
requirement, session lifecycle (30s background timeout, 60 min max),
performance overhead benchmarks, and full troubleshooting table.
All files follow the established React Native SDK reference format with a
minimum SDK version header, TOC, real Dart/Flutter code examples, honest
limitation sections, and troubleshooting tables.
Co-Authored-By: Claude Sonnet 4.6 (Anthropic)
* docs(flutter-sdk): add error-monitoring and tracing reference files
Complete the Flutter SDK skill reference library by adding the two
remaining deep-dive files referenced in the main SKILL.md but missing
from the initial commit.
error-monitoring.md:
- Documents all auto-captured error sources (FlutterError.onError,
PlatformDispatcher, Android NDK, iOS/macOS Mach exceptions)
- captureException / captureMessage / captureFeedback APIs with examples
- Scope management: configureScope, withScope, scope sync to native
- Breadcrumb capture and filtering via beforeBreadcrumb
- Event filtering with beforeSend (incl. honest note that it doesn't
intercept native crashes)
- Isolate error capture pattern (addSentryErrorListener)
- Attachments: screenshot, view hierarchy, file upload
- Android/iOS native crash options (tombstone, NDK sync, ANR, watchdog)
- Release health session tracking options
- Troubleshooting table
tracing.md:
- Setup: tracesSampleRate vs tracesSampler with sampling precedence order
- Navigation: SentryNavigatorObserver for MaterialApp, GoRouter, Auto Route
with honest limitations (GoRouter tabs, Auto Route nested nav)
- TTID (always on) vs TTFD (opt-in) with both setup patterns
- User interaction tracing: Key requirement, idle timeout, PII opt-in
- HTTP tracing: SentryHttpClient and sentry_dio with distributed tracing
config (tracePropagationTargets, propagateTraceparent)
- App start instrumentation: iOS/Android only, add-to-app caveat, disable path
- Slow/frozen frames: SentryWidgetsFlutterBinding workaround
- Custom instrumentation: transactions, nested spans, span data, dynamic sampling
- All four DB integrations: sqflite, Drift, Hive, Isar with setup snippets
- Asset bundle instrumentation
- Production config pattern with dynamic sampler
- Troubleshooting table including gRPC Windows issue
Co-Authored-By: Claude Sonnet 4.5 (Anthropic)
* docs(agents): add sentry-flutter-sdk to SDK skills table
* fix(flutter-sdk): address review findings — API consistency and version accuracy
- Fix enableLogs version: ≥9.0.0 → ≥9.5.0 in SKILL.md (matches logging.md)
- Replace fabricated Sentry.startSpan() with verified Sentry.startTransaction()
in logging.md and metrics.md trace correlation examples
- Fix Sentry.withScope() → Sentry.configureScope() in logging.md (correct Dart API)
- Add flutter-sdk keywords to build-skill-tree.sh Quick Lookup mapping
- Regenerate SKILL_TREE.md with proper keywords
* fix(flutter-sdk): remove fabricated cardinality claims from metrics reference
The 2 KB per-metric key limit is real, but the cardinality warnings,
'silently dropped' claims, and high-cardinality troubleshooting entry
were fabricated with no basis in Sentry docs or SDK source.
* fix(flutter-sdk): correct enableLogs version in Quick Reference section
Fix remaining ≥9.0.0 → ≥9.5.0 for enableLogs in the Quick Reference
code snippet. The earlier fix only caught the Phase 3 init block.
The biggest reliability improvement came from changing the user's
prompt from 'read and follow: URL' to 'use curl to download, read
and follow: URL'. This bypasses WebFetch summarization entirely.
Make this discoverable:
- SKILL_TREE.md now opens with the exact prompt to copy-paste
- Entry points table shows /sdks, /workflows, /features shortcuts
- 'Why curl?' explanation for context
- Router skills shortened to use curl-first examples
- Individual skill fetching section moved below the entry points
Co-Authored-By: Claude Sonnet 4 (Anthropic)
WebFetch and similar agent fetch tools often summarize long markdown
files instead of returning the full content. Sentry skill files are
10-20 KB of detailed configuration instructions — summaries lose
critical details like exact code snippets and config options.
Observed in a Claude Code session where WebFetch returned a 2,500-char
summary of an 18,000-char skill file twice before the agent figured out
to fall back to curl. Adding a one-line hint in the 'How to Fetch'
section saves agents from discovering this the hard way.
Co-Authored-By: Claude Sonnet 4 (Anthropic)
Rewrite the generated SKILL_TREE.md to help AI agents find and fetch
skills on the first try instead of guessing URLs, and to stop them from
rushing into action without understanding what the user wants.
Problem: agents read the SKILL_TREE, identify the platform they need,
then guess URL patterns (/nextjs, /sdk-setup/nextjs, etc.) instead of
following the markdown links — burning 4+ requests before finding the
correct path. They also skip the 'Start Here' section entirely and
immediately start installing packages and creating config files without
ever asking the user what they actually want.
Changes to build-skill-tree.sh:
- Replace 'How It Works' header with directive 'How to Fetch Skills'
section that puts the URL pattern front-and-center with an explicit
example
- Rewrite 'Start Here' as 'Start Here — Read This Before Doing
Anything' with explicit instructions to ask the user first, wait for
their answer, then fetch the matching skill. Tells agents not to
assume, not to skip, and not to improvise.
- Add a Path column to all skill tables showing the exact fetchable
path as plain-text code (e.g. `sentry-nextjs-sdk/SKILL.md`)
- Reorder table columns: description first (what agents match on),
then skill name, then path
- Add a Quick Lookup section at the bottom mapping common keywords
(nextjs, django, rails, etc.) to their skill paths — the grab-and-go
for agents that skim rather than parse tables
Co-Authored-By: Claude Sonnet 4 (Anthropic)
* feat(skill-tree): transform SKILL_TREE.md into agent-facing entry point
Rewrite SKILL_TREE.md from an internal plugin reference into the root
entry point for any AI agent accessing Sentry's skill library. When an
agent reads this file (via local install or https://skills.sentry.gg),
it now knows to introduce itself as Sentry's AI assistant, present the
user with options (SDK setup, debugging, features, code review, upgrade),
and dynamically load the matching skill.
Key changes:
- Agent persona and interactive greeting flow in 'Start Here' section
- Relative paths for all skill links (works on GitHub raw, web UI, and
local filesystem; proxy can serve them too)
- Base URL documented for skills.sentry.gg URL construction
- Platform Detection Priority section with disambiguation rules
- Dropped redundant 'Path' column from all tables
- Removed Internal section (contributor-only, not user-facing)
- Updated all 3 router skills to reference skills.sentry.gg URLs
instead of raw GitHub URLs in Dynamic Skill Fetching sections
- Updated build-skill-tree.sh to generate the new format
* chore: remove dead SKILLS_INTERNAL array
Addresses Bugbot review — the array was populated but never read
after the Internal section was removed from the generated output.
Internal skills are still validated, just not rendered.
* docs: reference skills.sentry.gg proxy in SKILL_TREE.md
Add a short paragraph explaining the proxy is available for clean URL
access, link to its source repo, and note that both path styles work.
Also fix the example URL to use the canonical form without the
skills/ prefix.
Use ${ARRAY[@]+"${ARRAY[@]}"} pattern for all array expansions that
could be empty (category arrays, errors array, function parameters).
Bash 3.2 with set -u treats "${EMPTY[@]}" as an unbound variable error,
while bash 4+ handles it gracefully. The +"..." syntax expands to nothing
when the array is unset/empty, avoiding the crash.
Also fix AGENTS.md claiming CI only runs on skills/** changes — the
validate-skill-tree job runs on all PRs (it's fast enough not to need
path filtering).
Adds scripts/build-skill-tree.sh — a bash-only script that scans
all skills/*/SKILL.md files, regenerates SKILL_TREE.md from live
frontmatter data, and validates the entire skill hierarchy.
What the script does:
1. Scan — finds all SKILL.md files via find, parses YAML frontmatter
(name, description, category, parent, role, disable-model-invocation)
using awk. No external dependencies beyond standard Unix tools.
2. Categorize — groups skills by category field, identifies routers
by role: router.
3. Generate — rebuilds SKILL_TREE.md with Quick Navigation table,
per-category skill tables (SDK Setup, Workflow, Feature Setup,
Internal), and short column values extracted from descriptions.
4. Validate — checks every rule without stopping on first failure:
- Non-router/non-internal skills must have category, parent, and
disable-model-invocation: true
- Internal skills must have disable-model-invocation: true
- Every skill's parent must exist and have role: router
- Each skill must appear in its router's SKILL.md
- All breadcrumb .md links must resolve to real files
- Warns on unknown categories
5. Stale check — diffs generated vs existing SKILL_TREE.md; writes
file unless --check is passed (check-only mode for CI).
Uses a temp-directory pattern for per-skill data storage instead of
bash 4 associative arrays, keeping it compatible with macOS bash 3.2.
Also updates SKILL_TREE.md to match the script's generated output
(column values are now extracted from descriptions via sed, ordering
follows find | sort alphabetical).
Co-Authored-By: Claude Sonnet 4.5 (Anthropic)
Update documentation to reflect the full skill catalog ported from
sentry-agent-skills. Skills are now organized into four categories:
- SDK bundles (9): full platform setup wizards with reference docs
- Setup skills (2): AI monitoring, OTel exporter
- Workflow skills (4): code review, PR review, fix issues, create alerts
- Authoring skills (1): SDK skill creator
Also adds scripts/generate-skill-list.sh from sentry-agent-skills for
generating skill lists from SKILL.md frontmatter.
Co-Authored-By: Claude (Anthropic)
Evan Purkhiser