2 Commits

Author	SHA1	Message	Date
Nick Nisi	60969251e9	chore: add oxfmt/oxlint and apply formatting + lint fixes	2026-02-26 11:49:05 -06:00
Nick Nisi	beee54aa35	feat: eval framework for measuring skill effectiveness (#3) ... * refactor: port runtime from Bun to Node (tsx + vitest) Replace Bun-specific APIs (Bun.file, Bun.write, Bun.sleep) with Node equivalents (node:fs/promises, node:timers/promises). Swap test runner from bun:test to vitest. Use tsx for TypeScript execution without a build step. * docs: use pnpm in CLAUDE.md command examples * feat: add eval harness for measuring skill effectiveness A/B testing framework that runs identical prompts with and without skill content, scoring outputs against expected signals (methods, env vars, imports, flow steps, hallucinations). Answers whether skills actually improve Claude's code generation quality. - Deterministic scorer (100-point scale, 30 unit tests) - Content-addressed response cache (SHA-256 keyed) - Console table + JSON report output - CLI: bun run eval [--product, --case, --dry-run, --no-cache] * feat: add eval test cases, fix scorer false positives, tune generation pipeline Eval harness phase 2: 12 YAML test cases across 5 products (SSO, AuthKit/Next.js, DSync, Audit Logs, RBAC) with concurrent execution. Pipeline fixes based on eval findings (SSO was -5% delta): - Scorer: negation-aware anti-pattern matching (isNegated, 30-char lookback) - Refiner: hard 150-line cap, implementation-level decision trees, bash verification - Refiner: MAX_TOKENS 4096 for feature guides (was 8192) - Template: cap scaffold extraction at 30 lines, skip non-actionable headings - Quality gate: reverse size scoring (reward ≤4KB), bash/decision-tree bonuses Results: SSO delta moved from -5% to +1%, no skills with negative delta. * refactor: regenerate all skills with leaner pipeline Full regeneration with updated refiner prompts (150-line cap, implementation-level decision trees, bash verification commands) and trimmed template scaffolds. Net -1,769 lines across 64 skills. Guide sizes reduced toward 3-6KB range. 62/64 passing quality gate (same 2 persistent standalone-sso-api semantic failures as before). * fix: eval scorer false positives for anti-patterns and flow ordering - Add isInEnvBlock() guard to skip anti-pattern matches inside .env blocks and placeholder values (fixes sk_test false positive) - Replace single-keyword indexOf in scoreFlowOrder() with proximity- based window search to prevent keyword collisions across sections - Fix dead Related Skills cross-refs in 3 migration guides: point workos-organizations → workos-api-organization, remove non-existent workos-user-management refs * chore: format codebase and refactor eval runner output - Run prettier across all files - Refactor eval runner to print results after batch completion - Add HAND_CRAFTED_GUIDES protection to generate pipeline * feat: add 5 SSO eval cases targeting skill-specific knowledge gaps New cases test scenarios where LLMs fail without the skill guide: - error-handling: error field check order, code expiry (+22% delta) - domain-routing: domainHint param for email→IdP routing (+30%) - social-login: provider param for Google OAuth via WorkOS (+30%) - state-csrf: state="" edge case for IdP-initiated (+25%) - python-basic: cross-language snake_case SDK coverage (+25%) SSO eval: 3 cases → 8, avg delta 0% → +17% * fix: correct domainHint misuse in SSO guide and eval cases domainHint is a supplementary UX param (pre-fills domain field), NOT a connection selector. The three mutually exclusive selectors are: connection, organization, provider. SSO guide changes: - Add /docs/sso/guide and API reference to doc fetch list - Rewrite decision tree: organization is the primary selector, app must map email domain to org_id itself - Clarify domain_hint and login_hint as optional UX params - Fix trap warning and error recovery references Eval case changes: - sso-node-domain-routing: expect organization param instead of domainHint, update flow steps and anti-patterns accordingly SSO eval: 8 cases, avg delta +12% (down from +17% which included false positive from rewarding wrong domainHint usage) * fix: correct RBAC eval cases with verified expected signals Old cases had wrong expected methods (getUser, singular getOrganizationMembership) causing methodAccuracy=0 on both arms, producing artificial 0% delta. Fixed cases: - check-permission: no SDK method (reads from session/JWT), expect permissions param and session-based flow - role-assignment: listOrganizationMemberships + update (verified against WorkOS API reference) New cases targeting RBAC-specific knowledge: - permission-vs-slug: permission checks vs role slug checks - idp-override: IdP group mapping silently reverts API assignments - org-vs-env-roles: organization-level vs environment-level role scoping Result: RBAC composite 69% → 96%, delta +1%. Low delta is genuine — Sonnet 4.5 already knows RBAC well (100% on methods, params, env vars, anti-patterns on both arms). Skill value is marginal for this domain. * feat: add dsync and audit-logs eval cases for broader coverage Directory Sync (2 → 6 cases, +6% avg delta): - directory-deleted: dsync.deleted cascade trap (0% delta — LLM knows) - inactive-users: state:inactive vs user.deleted (0% delta — LLM knows) - user-identity: email vs directory_user_id for matching (+4%) - group-membership: group events and webhook verification (+7%) Audit Logs (2 → 5 cases, +4% avg delta): - event-naming: group.object.action convention (+5%) - system-events: actor.type=system for automated actions (0%) - admin-portal: self-service SIEM config via Admin Portal (+3%) Both products confirmed as low-delta domains — LLMs already handle these well. Skill value concentrated in webhook patterns (dsync +29%) and Dashboard-specific config (audit-logs log-streams +10%). * feat: add importAccuracy scoring, invocation-aware method matching, and fix error categories - Add importAccuracy as new ScoreCard dimension (weight 10, redistributed from methods 25→20 and flow 25→20) - Add methodRatioFound with 4-pass matching: full invocation → last-segment invocation → full substring → last-segment substring - Fix wrong_params error category to check params instead of methods - Add missing_method error category for missing expected methods - Consolidate system prompt construction in runner.ts, remove redundant prompt building from api.ts - Add 12 new tests (62 total, up from 50) * feat: add Python and Ruby eval cases for all 5 products 12 new cases (6 Python + 6 Ruby) covering SSO, RBAC, Directory Sync, Audit Logs, and AuthKit. Tests SDK-specific method names, imports, and conventions per language. Total eval cases: 39 (up from 27). * feat: add reporting distributions, language breakdown, CLI flags, and regression gates - Add --lang flag to filter cases by language (python, ruby, node) - Add --report flag to control output format (table, json, both) - Add --fail-on-regression flag with hard gates: no negative product delta, hallucination reduction >= 50% - Add median/p80/min/max delta to ProductSummary - Add per-language breakdown to report output - Add error reduction table (both-arm error categorization) - 20 new reporter tests (201 total tests) * feat: add CI wrapper script, skill content hash, and eval runbook - Add scripts/eval-ci.sh for CI environments (env validation, gates enabled) - Add skillHash to EvalReport JSON for cache provenance (SHA-256 of loaded skill file contents) - Add eval framework runbook to CLAUDE.md with commands, interpretation, and troubleshooting * fix: correct Ruby SDK method names and SSO state anti-pattern false positive - Fix WorkOS::SSO.get_authorization_url → authorization_url (Ruby SDK doesn't use get_ prefix) in 2 SSO Ruby cases - Fix WorkOS::UserManagement.get_authorization_url → authorization_url in authkit-ruby case - Tighten SSO state-csrf anti-patterns to avoid matching conditional advice ("skip state verification when state is empty string") * fix: anti-pattern index mismatch, unbounded 429 retry, and JSDoc errors - Fix negationAwareRatioFound index drift: search lowerOutput first (aligned indexes) before falling back to normalized match. Prevents false positive anti-pattern detection after camelCase content. - Cap 429 retries at 3 with linear backoff (was unbounded recursion) - Fix JSDoc: "Three-pass" → "Four-pass", "60 chars" → "30 chars" - Remove unused RATE_LIMIT_DELAY_MS constant * feat: add 3 Ruby SSO edge cases and fix state-csrf flow step volatility - Add sso-ruby-idp-initiated: IdP-initiated empty state handling - Add sso-ruby-error-handling: invalid_grant and consent denied - Add sso-ruby-domain-routing: email domain to organization routing - Remove volatile "skip state verification" flow step from state-csrf case — was causing -18% swings due to output structure variance when skill elaborates on the IdP-initiated exception * fix: align state-csrf case expectations with prompt intent and fix Ruby method in guide - Remove getAuthorizationUrl from state-csrf expected methods — prompt asks about callback CSRF handling, not auth URL generation. Was penalizing focused expert answers (-20% delta). - Simplify flow steps to match callback-focused prompt - Fix Ruby Quick Checklist: get_authorization_url → authorization_url * feat: add Ruby webhook example to dsync guide and relax Ruby method matching - Add Rails webhook handler example to directory-sync guide with WorkOS::Webhooks.verify_event and signature error handling - Accept construct_event as alternative method name in dsync-ruby case - Fix anti-pattern wording: "no signature verification" → "skip signature verification" * fix: replace Bun-specific APIs with Node equivalents - Replace Bun.CryptoHasher with crypto.createHash in cache.ts - Replace Bun.file/Bun.write with fs/promises readFile/writeFile - Replace bun:test imports with vitest in eval test files	2026-02-26 10:22:53 -06:00