mirror of
https://github.com/workos/skills.git
synced 2026-09-14 20:07:08 +08:00
main
13 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Garen Torikian
|
e5b8d82641 |
Add WorkOS MCP setup and recovery guidance (#39)
* feat(workos): add MCP setup recovery guidance
* fix(workos): scope MCP recovery guardrail and add MCP eval coverage
Narrow the MCP-recovery guardrail so it only applies when the user wants
MCP or a configured server is broken, instead of diverting intentional
CLI workspace-management requests away from the supported CLI fallback.
Add eval cases covering direct Codex MCP recovery and a management
request with MCP intentionally absent, so regressions in the new
first-match MCP route surface as negative deltas.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* refactor(workos): defer MCP specifics to docs
* test(workos): add MCP setup and scope-selection eval case
The MCP route covered recovery (mcp-codex-startup-interrupted) and the
no-MCP management fallback (mcp-management-cli-no-mcp), but nothing
exercised first-time setup or the user-global vs per-repo scope decision
that references/workos-mcp.md guards.
Expectations are pinned to the shipped CLI rather than memory: workos/cli
src/commands/mcp.ts exposes install|remove|status, and src/lib/mcp-clients.ts
installs the Claude Code server at --scope user because the management MCP is
account-level.
Anti-patterns and hallucinations are drawn from the failure both arms
actually produced on a live run: a fabricated stdio server
({"command": "workos"}) written into a Claude Desktop config path. The case
currently scores ~77/76 (delta +1) — it detects the failure, but the skill
does not yet prevent it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* fix(workos): restore load-bearing MCP anchors in the reference
|
||
|
Nick Nisi
|
2f57863afe |
feat: add FGA, Pipes, Feature Flags, and Radar references + evals (#19)
* feat: add FGA, Pipes, Feature Flags, and Radar reference files and evals
Close coverage gaps for four WorkOS products that had no reference files
or routing in the skill router.
New reference files (references/*.md):
- workos-fga.md — Fine-Grained Authorization (new API, not legacy warrants)
- workos-pipes.md — Pipes / Connected Apps (OAuth integrations)
- workos-feature-flags.md — Feature Flags (access token claims)
- workos-radar.md — Radar (bot/fraud detection)
New eval cases (scripts/eval/cases/*.yaml):
- fga.yaml (5 cases), pipes.yaml (4 cases),
feature-flags.yaml (4 cases), radar.yaml (4 cases)
- Total cases: 42 → 60
Skill router updates (SKILL.md):
- Added 4 rows to Features routing table
- Updated frontmatter description to trigger on new products
- Expanded routing decision tree with new product slugs
- Added FGA vs RBAC disambiguation
Eval results (2-sample, Sonnet):
- Pipes: +21% avg delta (strongest new product)
- FGA: +11% avg delta
- Radar: +10% avg delta (after blocklist case fix)
- Feature Flags: ~+1% avg delta (model already knows most of this)
- No regressions in existing products
* fix(workos): correct feature-flags and pipes reference inaccuracies
Feature flags: clarify that the SDK exposes management APIs
(addFlagTarget, removeFlagTarget, enableFeatureFlag, disableFeatureFlag),
not just runtime evaluation via token claims. The previous gotcha and
matching eval case incorrectly treated all targeting as Dashboard-only,
which would have rejected correct API-based solutions.
Pipes: distinguish the Node SDK response shape (camelCase accessToken
on a discriminated union) from the raw REST payload (snake_case
access_token). Update pipes-node-get-token flow steps to branch on the
active boolean instead of probing fields that only exist on one branch.
Verified against workos-node source for both modules.
* fix(eval): tighten feature-flags-node-refresh antipatterns
Previous antipatterns ("poll a feature flags API", "invalidate cache")
were too vague to catch the actual wrong approach: implementing a
custom in-memory cache with TTL instead of refreshing the session
token. Tightened to four explicit antipatterns that match the failure
mode observed in the without-skill transcript.
* fix(workos): correct Next.js helper, add namespace gotchas for FGA and Radar
Three quality issues caught while reviewing reference files against eval
transcripts and SDK source:
1. feature-flags: AuthKit Next.js exposes flags via `withAuth()`, not
`getUser()`. Soften the loadSealedSession prohibition — the README
doesn't actually forbid it, just doesn't surface it. Updated the
feature-flags-nextjs-check eval case to match.
2. fga: Add explicit gotcha that SDK methods live under
`workos.authorization.*`, NOT `workos.fga.*`. Eval transcripts show
the model inventing `workos.fga.assignRole` despite the product being
called FGA — verified workos-node has no `fga` directory under src/.
3. radar: Add explicit gotcha that there is NO `workos.radar.*`
namespace at all. Verified workos-node has no radar directory.
The standalone Radar API is direct-HTTP only.
|
||
|
Nick Nisi
|
dc9dfb093c |
fix(workos): tighten auth guidance and risky eval reruns (#25)
* fix(workos): include required --org flag in portal generate-link examples The "Not in the CLI" table in workos-management.md listed `workos portal generate-link --intent=sso` and `workos portal generate-link --intent=dsync` as the correct way to reach the Admin Portal for connection creation. Both omit the required --org flag and fail before opening the Portal. This is the same hallucination shape the PR is trying to prevent agents from producing — caught in review. Add --org=<org_id> to both rows, matching the Quick Reference entry at the top of the file. * test(workos): tighten no-CLI assertion in connection-create evals The first assertion in evals 8 and 9 was titled "States connection creation is NOT in the CLI" but its needles included Admin Portal and WorkOS Dashboard. An answer that only said "Use the Admin Portal" without ever stating CLI support is absent passed the assertion. Since the second assertion in each eval already checks for the Admin Portal/Dashboard destination, the first was both redundant and weakened. Remove destination terms from the first assertion so it genuinely checks for the explicit no-CLI signal. * test(workos): improve risky eval coverage Add targeted recipes for SSO, AuthKit, and RBAC cases that showed negative or noisy eval deltas. Tighten brittle flow-step expectations where the previous wording rewarded incidental prose ordering instead of the intended behavior. Add multi-case eval filtering and an eval:risk helper so the latest triage cases can be rerun with more samples and saved outputs. * Fix explicit triage handling in risk reruns * fix: format:check script incorrectly using prettier * chore: format files for oxfmt check |
||
|
Nick Nisi
|
c66e0d04c6 |
feat(workos): add terminology reference for docs URL lookups (#21)
* feat(workos): add terminology reference for docs URL lookups
New `references/workos-terms.md` maps WorkOS-specific terms
(Sign-in endpoint, initiate_login_uri, Redirect URI, Organization ID,
Admin Portal intent, JWKS, Sealed session, WORKOS_COOKIE_PASSWORD, etc.)
to canonical docs URLs. 10 seed rows, all URLs verified via WebFetch.
Router gains **Rule 0** for lookup-shaped queries ("what is X",
"docs URL for X") with explicit anti-triggers against setup phrasing
("set up Vault", "enable Admin Portal") so feature-setup requests
still route to Rule 3.
SKILL.md tightening:
- Frontmatter description shortened to ~485 chars, terminology-first
- How-to-Use rewritten as imperative MUST, not advisory
- llms.txt prohibition scoped to Rule 0 (Rules 7/8 still use it
for vague/no-match cases)
- "suggest open a PR" rather than "add a row" (agents can't persist
edits across sessions reliably)
New eval cases (`scripts/eval/cases/terminology.yaml`):
- terms-initiate-login-uri
- terms-sign-in-endpoint
Fixes the failure mode where agents fetched llms.txt blind on
"what is X" queries instead of using curated references.
* test(workos): add skill-creator eval harness + prompt/assertion spec
Adds `plugins/workos/skills/workos/evals/evals.json` — four realistic
terminology lookup prompts with assertion-level grading (content-contains-any/all/none
checks including explicit anti-pattern URLs the agent should NOT invent).
Complements the existing substring-based `scripts/eval/cases/terminology.yaml`:
the assertion format catches failure modes that flowStep substring matching
misses (e.g., "does the answer mention 'Dashboard' AND 'Redirects'",
"does it avoid inventing `/docs/initiate-login-uri`").
Workspace outputs under plugins/**/*-workspace/ are gitignored — iteration
results are ephemeral, only the spec is committed.
Iteration 1 result (4 evals × 2 arms, graded via grade.py):
With skill: 100% (17/17 assertions)
Without skill: 77.5% (13/17)
Delta: +22.5%
* chore: gitignore skill-creator eval workspaces
|
||
|
Nick Nisi
|
21853e2cd5 |
fix: harden Next.js AuthKit skill against Server Component cookie violations (#13)
* fix: harden Next.js AuthKit skill against Server Component cookie violations
- Rewrite Step 8 UI Integration with explicit NavAuth client component
pattern using useAuth() + refreshAuth({ ensureSignedIn: true })
- Add useAuth import from @workos-inc/authkit-nextjs/components
- Warn against raw getAuthorizationUrl() usage (returns object, not string)
- Add error recovery sections for [object Object] and OAuth state mismatch
- Add verification checks for unsafe auth URL patterns
- Add anti-pattern checks to authkit-nextjs-protected eval case
- Add authkit-nextjs-nav-auth-pkce regression eval case
- Add scorer regression test (broken: 59, fixed: 98)
Addresses Alexander Southgate's friction log: getSignInUrl() in Server
Components, [object Object] redirect, and PKCE state mismatch.
* chore: formatting
|
||
|
Nick Nisi
|
f8a36b452a |
feat: consolidate all skills into single-source references (#6)
* feat: consolidate all skills into references with single router entry point
Move all 17 CLI skill files (AuthKit frameworks, backend SDKs, management)
into references/ as the single source of truth. Delete 6 AuthKit skill
directories — CLI canonical versions replace them entirely.
- Add 17 new reference files from CLI (authkit-*, node, python, dotnet,
go, ruby, php, php-laravel, kotlin, elixir, sveltekit, management)
- Delete workos-authkit-{base,nextjs,react,react-router,tanstack-start,
vanilla-js} skill directories
- Update router (workos/SKILL.md) with progressive disclosure for all
references: AuthKit frameworks, backend SDKs, language detection,
management
- Add ESM package entry point (plugins/workos/index.ts) exporting
getReferencePath, getSkillsDir, getSkillPath helpers
- Add exports field to package.json
- Simplify eval runner — all skills now load from references/
- Remove dead HAND_CRAFTED_SKILLS constant
Only 2 registered skills remain: workos (router) and workos-widgets.
Everything else is a reference file dispatched by the router.
* chore: remove AGENTS.md
* docs: add Do/Don't and PR checklist to CLAUDE.md
* docs: rewrite README for consolidated architecture and npm usage
* feat: add async getReference/getSkill helpers to package exports
* chore: add release-please with version sync for plugin.json
Configure release-please to bump versions in package.json,
plugin.json, and marketplace.json together. Uses extra-files
with jsonpath to keep all three in sync on each release.
Also syncs plugin.json and marketplace.json to 0.1.0 to match
the current package.json version.
|
||
|
Nick Nisi
|
059fc5deda |
Replace generation pipeline with lean topic files (doc URLs + gotchas) (#5)
* feat: add lean topic files for skills simplification (phase 1) Replace generated 7-11KB guides with ~30-line topic files containing doc URLs + extracted gotchas. Evals showed the real value comes from "fetch docs first" directives and encoded traps, not the scaffolding. 22 topic files created alongside existing files (phase 2 will delete old files and rename .topic.md → .md). * feat: simplify skills — delete pipeline, replace guides with lean topic files (phase 2) Delete the generation pipeline (scripts/generate.ts, scripts/lib/), all guide files, summary files, API ref stubs, and feedback files. Rename .topic.md → .md. Update router to point to new topic files. Update CLAUDE.md and package.json. 113 files changed: +553 / -10,739 lines. The 7-11KB generated guides are replaced by ~30-line topic files containing doc URLs + gotchas. The eval framework showed these two things drive all the value. Inline HAND_CRAFTED_SKILLS in eval runner (was importing from deleted scripts/lib/config.ts). All 182 tests pass. * fix: add implementation-level gotchas to directory-sync topic Eval regression (-6.5% vs old pipeline avg) traced to missing webhook handler pattern details. Added: verify→200→async ordering, Ruby raw_post trap, and upsert idempotency pattern. |
||
|
Nick Nisi
|
874f809808 | chore: formatting | ||
|
Nick Nisi
|
a66de91677 |
feat: add transcript diff view and calibration metrics
Transcript diff (pnpm eval:diff) shows with/without outputs with ANSI-highlighted signals — green for matched methods, red for hallucinations, yellow for anti-patterns. Calibration (pnpm eval:calibrate) compares scorer ship/no-ship decisions against human labels with 80% agreement gate integrated into --fail-on-regression (requires 10+ labels to activate). |
||
|
Nick Nisi
|
5004296b5b |
feat: add eval transcript tooling — triage report, review labels, save-all-samples
Triage report surfaces top 10 risky cases per run (negative delta, high σ, hallucination regression). Review labels store enables append-only human judgments in JSONL for scorer calibration. --save-all-samples persists all N sample outputs for high-variance case debugging. |
||
|
Nick Nisi
|
b420a3b5d1 |
feat: add --samples=N for eval variance measurement and improve scorer accuracy
Add multi-sample support to the eval framework: each case runs N times with mean/stddev reported per case and per product. This distinguishes real skill regressions from LLM non-determinism (e.g., authkit-nextjs-15-fix confirmed at +34% δ with σ=0.5%). Also adds negation-aware hallucination counting, transcript persistence for manual review, and improved flow step matching with minimum co-occurrence thresholds. |
||
|
Nick Nisi
|
60969251e9 | chore: add oxfmt/oxlint and apply formatting + lint fixes | ||
|
Nick Nisi
|
beee54aa35 |
feat: eval framework for measuring skill effectiveness (#3)
* refactor: port runtime from Bun to Node (tsx + vitest)
Replace Bun-specific APIs (Bun.file, Bun.write, Bun.sleep) with Node
equivalents (node:fs/promises, node:timers/promises). Swap test runner
from bun:test to vitest. Use tsx for TypeScript execution without a
build step.
* docs: use pnpm in CLAUDE.md command examples
* feat: add eval harness for measuring skill effectiveness
A/B testing framework that runs identical prompts with and without
skill content, scoring outputs against expected signals (methods,
env vars, imports, flow steps, hallucinations). Answers whether
skills actually improve Claude's code generation quality.
- Deterministic scorer (100-point scale, 30 unit tests)
- Content-addressed response cache (SHA-256 keyed)
- Console table + JSON report output
- CLI: bun run eval [--product, --case, --dry-run, --no-cache]
* feat: add eval test cases, fix scorer false positives, tune generation pipeline
Eval harness phase 2: 12 YAML test cases across 5 products (SSO,
AuthKit/Next.js, DSync, Audit Logs, RBAC) with concurrent execution.
Pipeline fixes based on eval findings (SSO was -5% delta):
- Scorer: negation-aware anti-pattern matching (isNegated, 30-char lookback)
- Refiner: hard 150-line cap, implementation-level decision trees, bash verification
- Refiner: MAX_TOKENS 4096 for feature guides (was 8192)
- Template: cap scaffold extraction at 30 lines, skip non-actionable headings
- Quality gate: reverse size scoring (reward ≤4KB), bash/decision-tree bonuses
Results: SSO delta moved from -5% to +1%, no skills with negative delta.
* refactor: regenerate all skills with leaner pipeline
Full regeneration with updated refiner prompts (150-line cap,
implementation-level decision trees, bash verification commands)
and trimmed template scaffolds.
Net -1,769 lines across 64 skills. Guide sizes reduced toward
3-6KB range. 62/64 passing quality gate (same 2 persistent
standalone-sso-api semantic failures as before).
* fix: eval scorer false positives for anti-patterns and flow ordering
- Add isInEnvBlock() guard to skip anti-pattern matches inside .env
blocks and placeholder values (fixes sk_test false positive)
- Replace single-keyword indexOf in scoreFlowOrder() with proximity-
based window search to prevent keyword collisions across sections
- Fix dead Related Skills cross-refs in 3 migration guides: point
workos-organizations → workos-api-organization, remove non-existent
workos-user-management refs
* chore: format codebase and refactor eval runner output
- Run prettier across all files
- Refactor eval runner to print results after batch completion
- Add HAND_CRAFTED_GUIDES protection to generate pipeline
* feat: add 5 SSO eval cases targeting skill-specific knowledge gaps
New cases test scenarios where LLMs fail without the skill guide:
- error-handling: error field check order, code expiry (+22% delta)
- domain-routing: domainHint param for email→IdP routing (+30%)
- social-login: provider param for Google OAuth via WorkOS (+30%)
- state-csrf: state="" edge case for IdP-initiated (+25%)
- python-basic: cross-language snake_case SDK coverage (+25%)
SSO eval: 3 cases → 8, avg delta 0% → +17%
* fix: correct domainHint misuse in SSO guide and eval cases
domainHint is a supplementary UX param (pre-fills domain field),
NOT a connection selector. The three mutually exclusive selectors
are: connection, organization, provider.
SSO guide changes:
- Add /docs/sso/guide and API reference to doc fetch list
- Rewrite decision tree: organization is the primary selector,
app must map email domain to org_id itself
- Clarify domain_hint and login_hint as optional UX params
- Fix trap warning and error recovery references
Eval case changes:
- sso-node-domain-routing: expect organization param instead of
domainHint, update flow steps and anti-patterns accordingly
SSO eval: 8 cases, avg delta +12% (down from +17% which included
false positive from rewarding wrong domainHint usage)
* fix: correct RBAC eval cases with verified expected signals
Old cases had wrong expected methods (getUser, singular
getOrganizationMembership) causing methodAccuracy=0 on both arms,
producing artificial 0% delta.
Fixed cases:
- check-permission: no SDK method (reads from session/JWT), expect
permissions param and session-based flow
- role-assignment: listOrganizationMemberships + update (verified
against WorkOS API reference)
New cases targeting RBAC-specific knowledge:
- permission-vs-slug: permission checks vs role slug checks
- idp-override: IdP group mapping silently reverts API assignments
- org-vs-env-roles: organization-level vs environment-level role scoping
Result: RBAC composite 69% → 96%, delta +1%. Low delta is genuine —
Sonnet 4.5 already knows RBAC well (100% on methods, params, env
vars, anti-patterns on both arms). Skill value is marginal for this
domain.
* feat: add dsync and audit-logs eval cases for broader coverage
Directory Sync (2 → 6 cases, +6% avg delta):
- directory-deleted: dsync.deleted cascade trap (0% delta — LLM knows)
- inactive-users: state:inactive vs user.deleted (0% delta — LLM knows)
- user-identity: email vs directory_user_id for matching (+4%)
- group-membership: group events and webhook verification (+7%)
Audit Logs (2 → 5 cases, +4% avg delta):
- event-naming: group.object.action convention (+5%)
- system-events: actor.type=system for automated actions (0%)
- admin-portal: self-service SIEM config via Admin Portal (+3%)
Both products confirmed as low-delta domains — LLMs already handle
these well. Skill value concentrated in webhook patterns (dsync +29%)
and Dashboard-specific config (audit-logs log-streams +10%).
* feat: add importAccuracy scoring, invocation-aware method matching, and fix error categories
- Add importAccuracy as new ScoreCard dimension (weight 10, redistributed
from methods 25→20 and flow 25→20)
- Add methodRatioFound with 4-pass matching: full invocation → last-segment
invocation → full substring → last-segment substring
- Fix wrong_params error category to check params instead of methods
- Add missing_method error category for missing expected methods
- Consolidate system prompt construction in runner.ts, remove redundant
prompt building from api.ts
- Add 12 new tests (62 total, up from 50)
* feat: add Python and Ruby eval cases for all 5 products
12 new cases (6 Python + 6 Ruby) covering SSO, RBAC, Directory Sync,
Audit Logs, and AuthKit. Tests SDK-specific method names, imports, and
conventions per language. Total eval cases: 39 (up from 27).
* feat: add reporting distributions, language breakdown, CLI flags, and regression gates
- Add --lang flag to filter cases by language (python, ruby, node)
- Add --report flag to control output format (table, json, both)
- Add --fail-on-regression flag with hard gates: no negative product
delta, hallucination reduction >= 50%
- Add median/p80/min/max delta to ProductSummary
- Add per-language breakdown to report output
- Add error reduction table (both-arm error categorization)
- 20 new reporter tests (201 total tests)
* feat: add CI wrapper script, skill content hash, and eval runbook
- Add scripts/eval-ci.sh for CI environments (env validation, gates enabled)
- Add skillHash to EvalReport JSON for cache provenance (SHA-256 of
loaded skill file contents)
- Add eval framework runbook to CLAUDE.md with commands, interpretation,
and troubleshooting
* fix: correct Ruby SDK method names and SSO state anti-pattern false positive
- Fix WorkOS::SSO.get_authorization_url → authorization_url (Ruby SDK
doesn't use get_ prefix) in 2 SSO Ruby cases
- Fix WorkOS::UserManagement.get_authorization_url → authorization_url
in authkit-ruby case
- Tighten SSO state-csrf anti-patterns to avoid matching conditional
advice ("skip state verification when state is empty string")
* fix: anti-pattern index mismatch, unbounded 429 retry, and JSDoc errors
- Fix negationAwareRatioFound index drift: search lowerOutput first
(aligned indexes) before falling back to normalized match. Prevents
false positive anti-pattern detection after camelCase content.
- Cap 429 retries at 3 with linear backoff (was unbounded recursion)
- Fix JSDoc: "Three-pass" → "Four-pass", "60 chars" → "30 chars"
- Remove unused RATE_LIMIT_DELAY_MS constant
* feat: add 3 Ruby SSO edge cases and fix state-csrf flow step volatility
- Add sso-ruby-idp-initiated: IdP-initiated empty state handling
- Add sso-ruby-error-handling: invalid_grant and consent denied
- Add sso-ruby-domain-routing: email domain to organization routing
- Remove volatile "skip state verification" flow step from state-csrf
case — was causing -18% swings due to output structure variance when
skill elaborates on the IdP-initiated exception
* fix: align state-csrf case expectations with prompt intent and fix Ruby method in guide
- Remove getAuthorizationUrl from state-csrf expected methods — prompt
asks about callback CSRF handling, not auth URL generation. Was
penalizing focused expert answers (-20% delta).
- Simplify flow steps to match callback-focused prompt
- Fix Ruby Quick Checklist: get_authorization_url → authorization_url
* feat: add Ruby webhook example to dsync guide and relax Ruby method matching
- Add Rails webhook handler example to directory-sync guide with
WorkOS::Webhooks.verify_event and signature error handling
- Accept construct_event as alternative method name in dsync-ruby case
- Fix anti-pattern wording: "no signature verification" → "skip
signature verification"
* fix: replace Bun-specific APIs with Node equivalents
- Replace Bun.CryptoHasher with crypto.createHash in cache.ts
- Replace Bun.file/Bun.write with fs/promises readFile/writeFile
- Replace bun:test imports with vitest in eval test files
|