Commit Graph

87 Commits

Author SHA1 Message Date
melkeydev 6205e61949 removing bash 2026-04-09 12:28:14 -07:00
melkeydev fbbead58af changing telemetry gating 2026-04-09 11:23:16 -07:00
MelkeyDev 92bc3dcf70 Merge branch 'main' into fix/yaml-repo-syntax-error 2026-04-06 11:05:10 -07:00
melkeydev 2475570761 Fixing yaml syntax error 2026-04-06 09:14:22 -07:00
melkeydev 3df45a85e9 adding a kill switch for telemetry 2026-04-06 08:37:28 -07:00
melkeydev 35bf7ef55f making telemetry fully opt-out 2026-04-06 08:13:43 -07:00
melkeydev 396181d588 PR comments 2026-04-02 09:39:08 -07:00
melkeydev 8449a2a561 reduce-vercel-injection-at-start 2026-04-01 15:57:07 -07:00
melkeydev ee0df43c53 Adding proper gate for lexical injection 2026-04-01 10:41:36 -07:00
melkeydev ab572d3e68 fixing ci tests 2026-03-30 09:46:47 -07:00
melkeydev eab4da0aba fixing readme and CI 2026-03-22 15:51:45 -07:00
John Lindquist ed7c3eb8ad fix(skill-frontmatter): warn on broken startup skills
Normalize fragile SKILL.md frontmatter in ncc and next-forge so the custom
frontmatter parser keeps their regex and nested prompt signal data intact.
Add a session-start summary log for broken skill frontmatter and keep
telemetry env export non-fatal when the env file is missing.

Verified: bun test tests/skill-map-frontmatter.test.ts tests/session-start-profiler.test.ts

Swarm-Agent: codex-skill-reliability
2026-03-11 19:01:59 -06:00
John Lindquist b7e9b1b053 feat: merge upstream telemetry + Cursor compat, fix tests and dedup migration; bump to 0.17.0 2026-03-11 16:12:55 -06:00
John Lindquist aba9c59158 test(pretooluse): sync temp hook fixtures with runtime deps
Temp hook fixtures had a stale hardcoded module list, so debug and empty-skill tests
were failing before their assertions ran once pretooluse imported telemetry.
This centralizes fixture copying and keeps overrides available for patched
patterns tests.

Verified: bun test tests/pretooluse-skill-inject.test.ts -t "returns \\{\\} when skills directory is empty|SKILLMAP_EMPTY issue emitted when skills directory has no SKILL\.md files|SKILLMD_PARSE_FAIL issue emitted for malformed YAML frontmatter|SKILLMD_PARSE_FAIL not emitted when debug is off" (4 tests pass)
Attempted: bun test tests/pretooluse-skill-inject.test.ts (7 unrelated failures remain in dedup/validation expectations)
Swarm-Agent: codex-pretooluse
2026-03-11 15:27:50 -06:00
John Lindquist 8e24d0694a feat: add chainTo skill chaining, prompt signals, validation upgrades, and dedup reset on context clear
Add a chainTo frontmatter field to skills that triggers follow-up skill injection
when PostToolUse file contents match regex patterns. Add upgradeToSkill/upgradeWhy
fields to validation rules so validate errors can recommend loading a specific skill.
Register posttooluse-bash-chain.mjs in hooks.json. Add ChainToRule interface to
patterns.mts and skill-map-frontmatter.mts, with parseChainToRules() parser. Add
duplicate-key detection to the YAML parser. Reset dedup claim dir and session file
on clear/compact events in session-start-seen-skills so skills re-inject into fresh
context windows.

Add chainTo rules to: agent-browser-verify, agent-browser, ai-elements, ai-gateway,
ai-generation-persistence, ai-sdk, auth, bootstrap, chat-sdk, cms, cron-jobs,
deployments-cicd, email, env-vars, geist, investigation-mode, json-render,
marketplace, micro, ncc, next-forge, nextjs, observability, payments,
react-best-practices, routing-middleware, runtime-cache, satori, shadcn,
sign-in-with-vercel, swr, turbopack, turborepo, v0-dev, vercel-agent, vercel-api,
vercel-cli, vercel-firewall, vercel-flags, vercel-functions, vercel-queues,
vercel-sandbox, vercel-storage, verification, workflow. Add upgradeToSkill to
ai-elements and ai-sdk validate rules. Expand ai-sdk validate messages with
Run Skill() hints. Update nextjs, vercel-storage, runtime-cache, workflow, turborepo
skill bodies.

Add new skills: geistdocs (Geist design system docs), zzz-test-meta-name-mask
(test fixture). Add skills/_chain-audit.md chain coverage audit doc.

Delete .claude-plugin/marketplace.json, .claude-plugin/plugin.json (deprecated),
skills/edge-runtime/SKILL.md (consolidated into vercel-functions).

Add tests: posttooluse-chain.test.ts (4699 lines, chain injection e2e),
ai-sdk-companion.test.ts (181 lines). Expand build-skill-map.test.ts (+335 lines),
validate-rules.test.ts (+936 lines), session-start-seen-skills.test.ts (+74 lines),
skill-map-frontmatter.test.ts (+50 lines), verification-skill.test.ts (+20 lines).

Update build-manifest.ts to emit chainTo rules and upgradeToSkill fields. Rebuild
generated/skill-manifest.json, generated/skill-catalog.md, generated/build-from-skills.manifest.json.
Rebuild all compiled hooks/*.mjs. Update CLAUDE.md lexical prompt default to on.
Update vercel.md ecosystem graph, docs, and cli-reference.
2026-03-11 15:18:35 -06:00
John Lindquist 089b5aefa4 fix(benchmark-sandbox): extend sandbox before snapshot
Move the timeout extension ahead of the snapshot call so long builds do not
expire before the restore point is created.
Also double the sandbox creation lifetime and only apply a post-snapshot
extension when keep-alive is enabled.

Verified: bun test tests/run-eval-build-polling.test.ts
Swarm-Agent: codex-snapshot-timing
2026-03-11 15:18:19 -06:00
John Lindquist a80ac3581c fix(session-start-profiler): store profiler state in session files
Claude session-start-profiler no longer requires CLAUDE_ENV_FILE to persist likely skills or greenfield state.
Those values now come from the hook input session ID and are written to session files,
while Claude env exports remain only for the other profiler vars.

Verified: bun run build:hooks
Verified: bun test tests/session-start-profiler.test.ts
Swarm-Agent: codex-profiler-env-cleanup
2026-03-11 15:18:19 -06:00
John Lindquist 7566bed4d4 fix(session-start): noop Claude seen-skills init
Claude session-start no longer appends VERCEL_PLUGIN_SEEN_SKILLS to
CLAUDE_ENV_FILE now that claim-dir/session initialization handles dedup.
Cursor still emits the empty env var payload on stdout.

Verified: bun run build:hooks
Verified: bun test tests/session-start-seen-skills.test.ts
Verified: bun test tests/session-timeline-subagent.test.ts
Verified: bun test hooks/session-hooks-platform-compat.test.ts
Swarm-Agent: codex-seen-skills-init-cleanup
2026-03-11 15:18:19 -06:00
John Lindquist 8e7c146c73 fix(prompt-submit): drop seen-skills env appends
UserPromptSubmit no longer appends VERCEL_PLUGIN_SEEN_SKILLS into CLAUDE_ENV_FILE.
Session claims/files remain the persistence path, while Cursor keeps its existing output-env fallback when no env file is present.

Verified: bun run build:hooks
Verified: bun test tests/user-prompt-submit.test.ts -t "cursor payload returns flat output with continue and env patch"
Verified: bun test tests/user-prompt-submit.test.ts -t "does not append seen skills to CLAUDE_ENV_FILE when available"
Swarm-Agent: codex-prompt-submit-env-cleanup
2026-03-11 15:18:19 -06:00
John Lindquist 95176e59dc fix(pretooluse): move hook counters off Claude env file
PreToolUse now persists TSX review and dev-server verify counters via session files,
and it no longer appends runtime env exports to CLAUDE_ENV_FILE for Claude.
This keeps seen-skills dedup on the claim/session-file path and avoids shell-init
side effects during hook execution.

Verified: bun run build:hooks
Verified: bun test tests/pretooluse-skill-inject.test.ts -t "file-based dedup persists across invocations with same session_id"
Verified: bun test tests/pretooluse-skill-inject.test.ts -t "empty VERCEL_PLUGIN_SEEN_SKILLS env var dedups across invocations"
Verified: custom node smoke test (tsx-migration-check) — CLAUDE_ENV_FILE unchanged, tsx-edit-count reset in session file, seen-skills persisted
Verified: custom node smoke test (dev-migration-check) — CLAUDE_ENV_FILE unchanged, dev-verify-count persisted in session file
Swarm-Agent: codex-pretooluse-env-cleanup
2026-03-11 15:18:18 -06:00
John Lindquist 91fd7a9da1 fix(run-eval): fire-and-forget deploy polling
Deploy now mirrors the build phase fire-and-forget pattern so long Claude deploy sessions keep running past the 300s sandbox API timeout.
The build snapshot is also taken before the build score call so the restore point exists before any extra analysis work.

Verified: bun test tests/run-eval-build-polling.test.ts
Verified: deploy phase now reads /tmp/claude-deploy.log after polling to extract the vercel.app URL
Swarm-Agent: codex-deploy-fireforget
2026-03-11 15:18:18 -06:00
John Lindquist 14e951e1df test(pretooluse-skill-inject): align budget expectations
Update the remaining budget-related assertions in pretooluse-skill-inject tests to match the current hook output and skillInjection metadata.\n\nVerified: bun test tests/pretooluse-skill-inject.test.ts (279 pass, 0 fail)\nHow to test: bun test tests/pretooluse-skill-inject.test.ts\nSwarm-Agent: codex-budget-fix
2026-03-11 15:18:18 -06:00
John Lindquist a78ac86798 fix(hooks): add Cursor support for session hooks
Session start hooks now detect Cursor payloads from stdin and emit Cursor-compatible JSON while keeping Claude Code env-file behavior intact.
Session end cleanup now falls back to conversation_id so Cursor sessions clean up the same temp artifacts.

Verified: bun test hooks/session-hooks-platform-compat.test.ts tests/session-start-seen-skills.test.ts tests/session-end-cleanup.test.ts
How to test: run the same bun test command, or pipe Cursor-style JSON into the three hook scripts under hooks/*.mjs.
Swarm-Agent: codex-session-start-v2
2026-03-11 15:17:31 -06:00
John Lindquist 1142c51906 fix(user-prompt-submit): support cursor prompt hook
Normalize UserPromptSubmit input fields across Claude Code and Cursor,
including conversation/session IDs, cwd roots, and prompt/message
fallbacks. Cursor now returns flat beforeSubmitPrompt output with
continue:true and emits seen-skill env patches when no CLAUDE_ENV_FILE is
available, while Claude continues using hookSpecificOutput and appends
seen-skill exports to CLAUDE_ENV_FILE when present.

Verified: bun test hooks/user-prompt-submit-skill-inject.test.ts tests/user-prompt-submit.test.ts
Verified: cursor beforeSubmitPrompt payload returns continue:true + env patch in tests/user-prompt-submit.test.ts
Swarm-Agent: codex-prompt-hook
2026-03-11 15:17:02 -06:00
John Lindquist 875a9e7d20 test(upgrade-rules): cover skill upgrade validate fixtures
Add focused fixture tests for each current upgradeToSkill validate rule.
Each case asserts the positive trigger and the skipIfFileContains path so future SKILL.md changes do not silently break upgrade guidance.

Verified: bun test tests/upgrade-rules.test.ts
How to test: bun test tests/upgrade-rules.test.ts
Swarm-Agent: codex-test-fixtures
2026-03-11 15:16:21 -06:00
John Lindquist 994a5431a8 fix(posttooluse-validate): dedupe skill upgrade instructions
formatOutput now emits a single skill-upgrade instruction block per target skill, adds the requested blank line and reason text, and marks hard upgrades as REQUIRED so agents get clearer follow-up actions.

Verified: bun run build:hooks
Verified: bun test tests/posttooluse-validate.test.ts
Verified: bun run typecheck
Swarm-Agent: codex-hook-chain
2026-03-11 15:16:21 -06:00
John Lindquist 1aae11eab4 feat(benchmark-sandbox): add eval observability telemetry
Add per-phase timing telemetry to run-eval results and report output.
Expand artifact extraction into readable observability files with an archive manifest.

Verified: bun test tests/run-eval-build-polling.test.ts
Verified: bunx tsc --noEmit --module esnext --moduleResolution bundler --target es2022 --lib es2022 --types node .claude/skills/benchmark-sandbox/run-eval.ts
Swarm-Agent: codex-observability-v2
2026-03-11 15:16:21 -06:00
John Lindquist 7a124e6561 Add sitemap field to skill metadata: parse metadata.sitemap in skill-map-frontmatter.mts, add sitemap to SkillConfig interface and KNOWN_KEYS, render sitemap URLs in buildDocsBlock() alongside doc links, include sitemap in generated manifest, and add sitemap URLs to 35 SKILL.md files for domains that have sitemap.xml. Also fix 15 broken doc URLs across 14 skills (vercel.com/docs path changes, anthropic docs, stripe, descope, flags-sdk, ai-sdk). Bump to 0.16.1. 2026-03-10 14:31:40 -06:00
John Lindquist d5b5ef47f7 Replace skill body injection with Skill tool invocation instructions
Instead of injecting the full SKILL.md body as additionalContext, inject
"You must run the Skill(<name>) tool." — a more conventional way of
telling the agent to use the Skill tool for context loading.
2026-03-10 12:46:11 -06:00
John Lindquist 905ffba7fa feat: add lexical stemmer, prompt signal scoring, docs, and plugin metadata
- Add stemmer and shared contractions modules for lexical prompt matching
- Enhance lexical index and prompt patterns with stemming support
- Add promptSignals metadata to all 43 skill frontmatter files
- Add comprehensive documentation site (docs/)
- Add .claude-plugin marketplace and plugin metadata
- Add benchmark scenarios script
- Update skill manifest with prompt signal data
- Add lexical-index and stemmer tests, expand prompt-patterns tests
2026-03-10 03:24:38 -06:00
John Lindquist 319bf1eb65 feat(hooks): add lexical prompt fallback scoring
Add scorePromptWithLexical as an additive wrapper around the existing exact prompt matcher.
It preserves current matching behavior, then falls back to lexical index hits when the exact score stays below threshold.

Verified: bun test tests/prompt-patterns-lexical.test.ts
Verified: tsc -p hooks/tsconfig.json --noEmit
Swarm-Agent: codex-prompt-patterns
2026-03-09 19:07:07 -06:00
John Lindquist e9ee9f2966 feat(hooks): add unified skill ranker
Add a shared rankSkills utility that combines path, command, import, profiler, prompt, lexical, and priority signals into a single sortable score with per-signal breakdowns.

Add a focused Bun regression test covering weighted scoring, ordering, and default field behavior for sparse candidates.

Verified: bun test tests/unified-ranker.test.ts
How to test: bun test tests/unified-ranker.test.ts
Swarm-Agent: codex-unified-ranker
2026-03-09 19:04:01 -06:00
John Lindquist b8b233995c feat(skills): broaden trigger patterns for 7 under-injected skills based on eval findings; add next-forge skill; bump to 0.15.0
Eval analysis of 9 real sessions showed 10 skills never triggering despite being
requested. Root causes: pathPatterns too narrow (agents write to lib/email-template.tsx
not emails/), promptSignals containing regex instead of plain text (vercel-sandbox),
and missing promptSignals entirely (v0-dev, vercel-firewall).

Skills updated: email, vercel-queues, edge-runtime, vercel-firewall, chat-sdk,
v0-dev, vercel-sandbox. New skill: next-forge (bootstrap detection).
2026-03-09 17:56:23 -06:00
John Lindquist 0c49abc547 fix(skills): quote YAML validate patterns ending with colon to fix js-yaml parse errors; bump to 0.14.1 2026-03-09 16:38:11 -06:00
John Lindquist 1e0fc2467c feat(hooks): wire subagent launch bootstrap
Connect SubagentStart to pending launch records so prompt-signal matches
can be merged into likely skills before context assembly.
SessionEnd now cleans hashed pending-launch temp directories and the
hook registry observes Agent tool launches.

Verified: node --input-type=module -e "import { build } from 'tsup'; await build({ entry: ['hooks/src/subagent-start-bootstrap.mts','hooks/src/session-end-cleanup.mts'], format: ['esm'], outDir: 'hooks', outExtension: () => ({ js: '.mjs' }), bundle: false, splitting: false, sourcemap: false, dts: false, clean: false, target: 'node20' });"
Verified: bun test tests/subagent-start-context.test.ts tests/session-end-cleanup.test.ts
Swarm-Agent: codex-wiring
2026-03-09 16:30:02 -06:00
John Lindquist 1dfa612df7 feat(hooks): observe agent subagent spawns
Add a PreToolUse observer hook that records Agent tool launches as pending subagent spawn metadata for downstream bootstrap logic.
It preserves the existing no-mutation contract by always returning {} and now integrates with the committed subagent-state append API.

Verified: bun test tests/pretooluse-subagent-spawn-observe.test.ts
Verified: ./node_modules/.bin/tsc -p hooks/tsconfig.json --noEmit
How to test: run bun test tests/pretooluse-subagent-spawn-observe.test.ts
Swarm-Agent: codex-observer
2026-03-09 16:28:50 -06:00
John Lindquist 2f8111da6d feat(subagent-state): add pending launch state helpers
Add a shared hooks library for pending subagent launch records and per-agent JSON state. The module filters stale launch records, claims the oldest matching launch under a file lock, and writes agent state with atomic rename semantics.

Verified: bun test tests/subagent-state.test.ts
Verified: node --input-type=module -e "import('./hooks/subagent-state.mjs').then((mod) => { if (typeof mod.listPendingLaunches !== 'function' || typeof mod.claimPendingLaunch !== 'function' || typeof mod.appendPendingLaunch !== 'function' || typeof mod.readAgentState !== 'function' || typeof mod.writeAgentState !== 'function') { process.exit(1); } })"
Swarm-Agent: codex-subagent-state
2026-03-09 16:25:50 -06:00
John Lindquist 5d548e5be8 fix(security): address audit findings — path traversal, brace expansion, injection escaping, profiler hardening; bump to 0.12.0
- Validate sessionId in dedupClaimDirPath() to prevent path traversal (HIGH)
- Add brace expansion pre-pass in globToRegex for {ts,js,mjs} patterns (HIGH)
- Escape --> sequences in HTML comment metadata blocks (MEDIUM)
- Harden checkVercelCli/checkAgentBrowser with 3s timeouts, suggest latest version (MEDIUM)
- Replace silent catch {} blocks with debug logging (LOW)
- Add semver-aware version comparison (LOW)
- Update golden snapshots and fix invocationId sharing
2026-03-09 14:18:43 -06:00
John Lindquist 830e49c69b fix(profiler): harden CLI availability checks
Make the session-start profiler resolve binaries from PATH safely before
invoking them, cap the version-check subprocesses at 3 seconds, and
avoid crashing when npm or agent-browser is missing.

Also expand the outdated Vercel CLI guidance to include the pnpm global
upgrade path and cover the new skip/timeout behavior in profiler tests.

Verified: bun test tests/session-start-profiler.test.ts
Swarm-Agent: codex-profiler-harden-split-2-v2
2026-03-09 14:06:35 -06:00
John Lindquist ee21fef192 fix(hooks): log swallowed profiler errors
Replace empty catch blocks in hook-env and session-start-profiler with\nstructured debug logging using the shared hook logger.\nAlso make the Vercel CLI update check compare numeric version\nsegments so 1.9.0 correctly sorts below 1.10.0.\n\nVerified: bun test tests/session-start-profiler.test.ts\nHow to test: bun test tests/session-start-profiler.test.ts\nSwarm-Agent: codex-profiler-harden-split-1
2026-03-09 13:55:54 -06:00
John Lindquist f035d9afb3 fix(inject): strip YAML frontmatter from skill bodies before injection; bump to 0.11.1
Skill frontmatter (pathPatterns, bashPatterns, promptSignals, etc.)
was being injected alongside the skill body, wasting token budget on
metadata only useful for hook matching. Now uses extractFrontmatter()
to emit only the markdown body.
2026-03-09 13:43:45 -06:00
John Lindquist 0930f2aa66 fix(patterns): expand brace globs in globToRegex
Brace groups like {ts,js,mjs} were being escaped literally, which prevented extension-list path patterns from matching. The glob parser now expands balanced brace groups into recursive regex alternations while preserving literal braces when no alternation is present.

Verified: bun test tests/patterns.test.ts
Verified: bun test tests/fuzz-glob.test.ts
Verified: bun test tests/hook-sync.test.ts
Verified: bun test tests/pretooluse-skill-inject.test.ts -t "matches src/middleware\.(mjs|mts) to routing-middleware skill"
Swarm-Agent: codex-brace-expand
2026-03-09 13:39:41 -06:00
John Lindquist ad5859e2c7 fix(hook-env): harden dedup temp paths
Hash invalid session IDs before constructing dedup temp paths so crafted stdin values cannot smuggle traversal segments into recursive claim cleanup. Shared temp-path resolution now verifies the resolved target stays under tmpdir, and seen-skills tests cover both stable safe IDs and hashed invalid IDs.

Verified: bun test tests/session-start-seen-skills.test.ts
How to test: bun test tests/session-start-seen-skills.test.ts
Swarm-Agent: codex-path-traversal
2026-03-09 13:39:32 -06:00
John Lindquist 7ac8edfb8f feat(skills): aggressive prompt signals for workflow/sandbox, "check the docs" directives across 8 skills; bump to 0.11.0
- workflow: add ~60 new promptSignal phrases for reliability language
  (survive page reload, fault-tolerant, retry on failure, session
  persistence, reconnect, durable chat/agent), human-in-the-loop
  patterns (approval, pause until, wait for), and pipeline vocabulary.
  Add ~25 new allOf pairs and 12 new anyOf terms.

- vercel-sandbox: add promptSignals from scratch (had none). 33 phrases
  covering isolated execution, sandbox environments, code safety,
  FFmpeg/media processing, code playgrounds, and tutor patterns.
  27 allOf pairs, 8 anyOf terms, noneOf excludes iframe/codesandbox.

- Add "CRITICAL — your training data is outdated" blockquote at the top
  of 8 fast-moving skill bodies (workflow, ai-sdk, vercel-sandbox,
  ai-gateway, chat-sdk, vercel-flags, vercel-queues, ai-elements)
  directing agents to fetch docs before writing code.

Eval results that motivated these changes:
- content-pipeline: 9/10 (workflow triggered, correct WDK patterns)
- customer-support: 3/10 → workflow never injected (now scores 50)
- code-sandbox-tutor: 1/10 → sandbox never injected (now scores 80)
2026-03-09 13:16:16 -06:00
John Lindquist 300488ebf3 fix(workflow): add explicit --no-src-dir guidance for create-next-app, update golden fixtures; bump to 0.10.1 2026-03-09 12:55:59 -06:00
John Lindquist 5ce557a33e feat(skills): add ai-generation-persistence and verification skills; bump to 0.10.0
New ai-generation-persistence skill (priority 6) injects guidance for treating
AI generations as first-class persistent resources — unique IDs, addressable
URLs, database/Blob storage, cost tracking, and generate-then-redirect UX
patterns. Triggers on AI SDK imports and broad prompt signals.

New verification skill added. Updated ecosystem graph, catalog, manifest,
fixtures, and snapshots.
2026-03-09 12:23:19 -06:00
John Lindquist ff965cc377 fix(skills): remove styled-jsx, regenerate catalog, add investigation-mode skill, build-from-skills templates; bump to 0.9.10 2026-03-09 11:38:08 -06:00
John Lindquist a43cb60a64 fix(skills): stronger outdated model warnings with "your knowledge is outdated" language, make AI Elements mandatory for AI text rendering; bump to 0.9.8 2026-03-09 07:59:56 -06:00
John Lindquist 62f6718d31 feat(skills): remove streamdown skill, expand ai-elements, update profiler and pretool hook, update tests; bump to 0.9.0 2026-03-08 19:50:24 -06:00
John Lindquist 2d53cbe2bf feat(hooks): bump skill cap from 3 to 5, add shadcn create-next-app nudge; bump to 0.8.0 2026-03-08 17:26:11 -06:00