* fix(workos): include required --org flag in portal generate-link examples
The "Not in the CLI" table in workos-management.md listed
`workos portal generate-link --intent=sso` and
`workos portal generate-link --intent=dsync` as the correct way to
reach the Admin Portal for connection creation. Both omit the
required --org flag and fail before opening the Portal. This is the
same hallucination shape the PR is trying to prevent agents from
producing — caught in review.
Add --org=<org_id> to both rows, matching the Quick Reference
entry at the top of the file.
* test(workos): tighten no-CLI assertion in connection-create evals
The first assertion in evals 8 and 9 was titled "States connection
creation is NOT in the CLI" but its needles included Admin Portal
and WorkOS Dashboard. An answer that only said "Use the Admin
Portal" without ever stating CLI support is absent passed the
assertion. Since the second assertion in each eval already checks
for the Admin Portal/Dashboard destination, the first was both
redundant and weakened.
Remove destination terms from the first assertion so it genuinely
checks for the explicit no-CLI signal.
* test(workos): improve risky eval coverage
Add targeted recipes for SSO, AuthKit, and RBAC cases that showed negative or noisy eval deltas. Tighten brittle flow-step expectations where the previous wording rewarded incidental prose ordering instead of the intended behavior.
Add multi-case eval filtering and an eval:risk helper so the latest triage cases can be rerun with more samples and saved outputs.
* Fix explicit triage handling in risk reruns
* fix: format:check script incorrectly using prettier
* chore: format files for oxfmt check
* feat: add lean topic files for skills simplification (phase 1)
Replace generated 7-11KB guides with ~30-line topic files containing
doc URLs + extracted gotchas. Evals showed the real value comes from
"fetch docs first" directives and encoded traps, not the scaffolding.
22 topic files created alongside existing files (phase 2 will delete
old files and rename .topic.md → .md).
* feat: simplify skills — delete pipeline, replace guides with lean topic files (phase 2)
Delete the generation pipeline (scripts/generate.ts, scripts/lib/),
all guide files, summary files, API ref stubs, and feedback files.
Rename .topic.md → .md. Update router to point to new topic files.
Update CLAUDE.md and package.json.
113 files changed: +553 / -10,739 lines. The 7-11KB generated guides
are replaced by ~30-line topic files containing doc URLs + gotchas.
The eval framework showed these two things drive all the value.
Inline HAND_CRAFTED_SKILLS in eval runner (was importing from deleted
scripts/lib/config.ts). All 182 tests pass.
* fix: add implementation-level gotchas to directory-sync topic
Eval regression (-6.5% vs old pipeline avg) traced to missing
webhook handler pattern details. Added: verify→200→async ordering,
Ruby raw_post trap, and upsert idempotency pattern.
* Add claude skill to create WorkOS widgets
* Add guidelines about styling existing ui components and where to implement the widget
* Optimize some skills
* Add list of endpoints to fetching api docs
* fix: resolve widget skill conflicts and route to hand-crafted skill
- Delete hallucinated generated widget files (fabricated workos.widgets.mount() API)
- Route widget requests via Skill tool instead of generated references
- Add workos-widgets to HAND_CRAFTED_SKILLS, skip widgets in SECTION_CONFIG
- Move OpenAPI spec from always-loaded to on-demand in SKILL.md
- Add error response documentation to fetching-apis.md
- Add strip-openapi-spec.ts script and strip boilerplate (3375 → 2774 lines)
- Fix paths.spec.ts and splitter.spec.ts for new config
* feat: add on-demand OpenAPI spec query script
Replace always-loaded/stripped spec with a query script that extracts
just the endpoints and resolved schemas for a specific widget.
- Add references/scripts/query-spec.ts (--widget, --path, --list modes)
- Revert spec to original unstripped version
- Update SKILL.md to reference query script instead of raw spec
- Remove strip-openapi-spec.ts (no longer needed)
* feat: add query-spec tests and fix resolution in arrays
- Add 36 tests for query-spec (resolveRef, resolveSchema, extractEndpoints,
formatEndpoint, groupPathsByWidget, WIDGET_PREFIXES, integration)
- Fix resolveSchema to recursively resolve $refs inside arrays (oneOf, anyOf)
- Revert OpenAPI spec to original unstripped version
- Remove strip-openapi-spec.ts
* feat: bundle query-spec as self-contained .cjs for portability
- Add esbuild to bundle query-spec.ts → query-spec.cjs (includes yaml dep)
- Update SKILL.md to use 'node references/scripts/query-spec.cjs' (no tsx needed)
- Add build:query-spec script to package.json
- Fix __dirname resolution to support both ESM and CJS contexts
* chore: add .oxlintrc and ignore built file in it and oxfmt
* fix: point fetching-apis.md references to query script, remove redundant SKILL.md mention
* chore: remove unused strip-openapi-spec.ts
Superseded by the on-demand query-spec approach.
* fix: address final review findings
- token-strategies.md: point elevated access to fetching-apis.md markers
instead of raw spec, use explicit verify endpoint path
- fetching-apis.md: update objective line, hardcode base URL fallback
to https://api.workos.com instead of referencing spec
- query-spec.ts: add SKILL.md slug aliases (user-management,
user-profile, admin-portal-sso-connection, admin-portal-domain-verification)
- query-spec.ts: update source comments to reference .cjs bundle
- Rebuild query-spec.cjs
---------
Co-authored-by: Lucas Motta <mail@lucasmotta.com>
Transcript diff (pnpm eval:diff) shows with/without outputs with
ANSI-highlighted signals — green for matched methods, red for
hallucinations, yellow for anti-patterns.
Calibration (pnpm eval:calibrate) compares scorer ship/no-ship
decisions against human labels with 80% agreement gate integrated
into --fail-on-regression (requires 10+ labels to activate).
Triage report surfaces top 10 risky cases per run (negative delta,
high σ, hallucination regression). Review labels store enables
append-only human judgments in JSONL for scorer calibration.
--save-all-samples persists all N sample outputs for high-variance
case debugging.
Add multi-sample support to the eval framework: each case runs N times
with mean/stddev reported per case and per product. This distinguishes
real skill regressions from LLM non-determinism (e.g., authkit-nextjs-15-fix
confirmed at +34% δ with σ=0.5%).
Also adds negation-aware hallucination counting, transcript persistence
for manual review, and improved flow step matching with minimum
co-occurrence thresholds.
Separate marketplace root from plugin root per Claude Code docs so only
the plugin directory gets cached on install, not the entire repo.
- Move skills/ → plugins/workos/skills/ (plugin root)
- Add plugins/workos/.claude-plugin/plugin.json (plugin manifest)
- Simplify marketplace.json to single plugin entry with source: "./plugins/workos"
- Remove per-skill plugin.json files (auto-discovery via skills/ directory)
- Remove obsolete scripts (build-plugin-manifests, sync-marketplace)
- Update all pipeline write paths, feedback loader, refiner, and tests
- Strip frontmatter from reference files
Reference .md files with YAML frontmatter may interfere with plugin
skill discovery. Strip frontmatter from all reference files (they're
loaded via Read, not by the plugin system). Remove em dashes from
plugin descriptions. Update quality gate and templates to match.
Separates the SKILL.md router (entry point) from bundled reference
files. All file references now use `references/{name}.md` paths —
portable across any agent platform, no hardcoded install paths.
- Move 67 skill files from skills/workos/ to skills/workos/references/
- Update generator write paths and content references
- Update guide pointers, quality gate regexes, feedback loader
- Update all tests for new directory structure
* fix: correct dsync rule to acknowledge Events API polling
The dsync-webhooks-mandatory rule was enforcing the incorrect claim
that "webhooks are mandatory and polling is not supported." WorkOS
supports both webhooks (recommended) and the Events API for directory
sync events. Inverts the rule and updates skill content accordingly.
* feat: add feedback mechanism, content taxonomy, and quality gate evolution (Phase 1)
Replace rules-based refiner injection with markdown feedback files and content
taxonomy. Feedback files (`.feedback.md`) let domain experts provide corrections
in plain markdown. Quality gate now penalizes behavioral assertions without doc
deferrals and supports LLM semantic checks during --refine.
* feat: add directory sync feedback file for refiner guidance
Backfills the dsync domain corrections (Events API support, webhook
non-mandatory) that were previously injected via .rules.yml into the
new .feedback.md format consumed by the refiner.
* refactor: replace rules system with feedback files, promote semantic check (Phase 2)
Migrate all 5 .rules.yml files to .feedback.md format. Remove rules.ts,
rules.spec.ts, and SkillRule/RuleViolation types entirely. Semantic check
failures are now hard blocks in the quality gate during --refine runs.
* chore: run the new generation
* feat: parallelize refinement, fix --refine-only scoping
- Refine skills with concurrency 5 (~6x faster)
- Scope write phase to targeted skill when --refine-only is set,
preventing accidental overwrite of other refined skills
- Scope quality gate to targeted skill when --refine-only is set,
preventing false semantic check failures on unrefined scaffolds
- Strengthen standalone SSO API migration feedback
* feat: progressive disclosure — split skills into summary + guide
Each generated skill now produces two files:
- summary (.md, 1-2KB): when to use, key concepts, guide pointer
- guide (.guide.md, 8-16KB): implementation steps, verification, errors
Reduces context usage 60-80% for exploratory workflows. Agents load
summaries during exploration, guides only when implementing.
Adds summary-specific refiner prompt for key concepts extraction,
dual scoring in quality gate, and SUMMARY_VALIDATION size constraints.
* fix: add legacy scoring for single-file skills, bump summary limit to 5KB
Router and integration router have frontmatter but no type field — they
were being scored by scoreGuide() which penalizes frontmatter. Added
scoreLegacy() for untyped single-file skills. Bumped summary size limit
from 3KB to 5KB since the refiner consistently produces 3-5KB summaries.
* chore: regenerate skills with progressive disclosure (summary + guide)
32 summaries (2-5KB) + 32 guides (6-20KB) + router + integration router.
Summaries contain When to Use, Key Concepts, guide pointer.
Guides contain implementation steps, verification, error recovery.
* refactor: tune prompts — summaries as routing docs, guides as doc supplements
Summary refiner: strict routing-document framing. Hard rules against
verification commands, SDK methods, behavioral claims, decision trees,
and trap warnings in summaries. Target 500-1000 bytes. Key Concepts
renamed to Key Vocabulary (entity names + ID prefixes only, max 10).
Guide refiner: reframed as "value beyond docs." Don't restate what docs
say — add decision trees, trap warnings, verification commands, and
error recovery that docs don't provide. Target 80-150 lines. No SDK
method names (vary by language).
* chore: regenerate skills with lean routing summaries
Summaries now 1-1.7KB (down from 3-5KB). Key Vocabulary limited to
entity names and ID prefixes. No verification commands, SDK methods,
behavioral claims, or decision trees in summaries. Guides are leaner
too — 6-12KB focused on value beyond docs.
* fix: preserve Documentation section in summary refinement
The summary refiner prompt listed 4 sections but omitted Documentation,
causing the LLM to strip doc URLs during refinement. All summaries lost
15 quality gate points. Now lists 5 sections with Documentation as #2
and reinforces PRESERVE in the user prompt. Also removed dashboard
navigation paths from allowed vocabulary.
* chore: regenerate skills with doc URLs preserved in summaries
Summary scores jump from 85 to 95-100 now that Documentation section
is preserved through refinement. 60/64 pass quality gate.
* fix: tiered doc URLs, broader verification, smarter semantic check
Quality gate:
- 2 doc URLs now scores 15pts (was 10) for thin API domains
- Verification section with bash blocks accepted (not just checkboxes)
- Semantic check: feedback emphasis overrides taxonomy, agent-directed
instructions (STOP, BLOCKING, MUST PASS) aren't behavioral claims
Refiner:
- Remove env vars (WORKOS_API_KEY etc) from allowed Key Vocabulary
- Exclude generic protocol concepts and behavioral claims from vocab
* chore: regenerate skills with cleaner vocab and fixed rubric
63/64 pass. Summaries have no env vars in vocabulary. API ref guides
now score 85+ with tiered doc URLs. Descope passes semantic check.
* feat: skill quality V2 — code examples, API ref stubs, leaner summaries, path tests
- Refiner: allow ONE language-agnostic SDK code example (10-25 lines) per feature guide
- Template: add renderApiRefStub() for deterministic ~500B API ref guide stubs
- Generator: route workos-api-* specs to stubs, skip refinement for them
- Quality gate: scoreApiRefStub() + code example bonus (+5pts) + summary scoring rebalance
- Summaries: drop Documentation section (lives in guide), target 400-600B, max 5 vocab
- Config: SUMMARY_VALIDATION.maxSize 5KB→2KB
- Tests: path resolution integration tests (guide pointers, router refs, Related Skills)
* chore: regenerate skills with V2 improvements
- Feature guides now include language-agnostic SDK code examples
- API ref guides replaced with lightweight stubs (0.5-1.3KB)
- Summaries trimmed (no Documentation section, max 5 vocab)
- 62/64 passing quality gate (standalone-sso + cognito semantic persist)
* fix: strengthen feedback for cognito + standalone-sso, re-refine
- cognito: clarified hash limitation is Cognito-side, WorkOS CAN import
hashes. Fixed WORKOS_COOKIE_PASSWORD scope (all server-side AuthKit SDKs,
not just Next.js). Now passing semantic check (90 + 80).
- standalone-sso: promoted terminology rule to Corrections, clarified
WORKOS_COOKIE_PASSWORD scope. Semantic check still strict (accepted).
- 63/64 passing quality gate.
* chore: formatting
* docs: update README, CLAUDE, and AGENTS for progressive disclosure + V2
- README: replace stale commands, add progressive disclosure and feedback
system sections, remove defunct domain rules and batch refine references
- CLAUDE: add summary/guide/stub/feedback file types, update conventions
- AGENTS: add two-step loading pattern, update file layout and skill patterns
* chore: fixes
* chore: delete .cursorrules
Hidden docs now live alongside the router that references them:
skills/workos/SKILL.md (router)
skills/workos/workos-sso.md (hidden doc)
skills/workos/workos-sso.rules.yml (domain rules)
Top-level skills/ is now clean: 7 directories only (router + 6 AuthKit).
Pipeline, rules loader, and manifest builder updated for new paths.
The entry-point skill is now just `workos` — cleaner name for the
master dispatcher. Updated all references in generator, refiner,
refine-batch, tests, README, AGENTS.md, and .cursorrules.
Phase 2 — Domain rules:
- Add per-skill .rules.yml with must_contain/must_not_contain regex checks
- Graduated enforcement (warn by default, promoted: true = hard gate)
- Quality gate evaluates rules and reports violations
- Rules injected into refiner prompts so LLM knows constraints
- First rule: workos-directory-sync webhooks are mandatory
- 14 new tests for rule loading, evaluation, and severity logic
Phase 3 — Source attribution:
- All three refiner prompts now instruct LLM to only claim what's in docs
- Cite doc URLs for non-obvious claims, never hallucinate SDK methods
Phase 1 of the WorkOS skills generator. Sets up the repo as a publishable
npm package compatible with skills.sh, copies 6 hand-crafted AuthKit
framework skills from the CLI repo, and builds the foundation for skill
generation from llms-full.txt.
- npm package: @workos-inc/skills with skills/ in files array
- Fetcher: downloads llms.txt/llms-full.txt with retry + local cache
- Parser: extracts 24-section tree from ## Name {#anchor} boundaries
- Validator: format guards that fail loudly on doc structure changes
- 18 passing tests (bun test)
Nick Nisi