Instead of injecting full SKILL.md bodies as additionalContext, hooks now
inject "You must run the Skill(<name>) tool." — leveraging the conventional
Skill tool mechanism for context loading.
Instead of injecting the full SKILL.md body as additionalContext, inject
"You must run the Skill(<name>) tool." — a more conventional way of
telling the agent to use the Skill tool for context loading.
Source extraction: After each eval, tar the project source (excluding node_modules, .next, .git) from the sandbox via sandbox.readFile(), save as source.tar.gz in the results directory. Include extraction instructions in the markdown report with commands to untar and run locally.
Deploy fix: The VERCEL_TOKEN env var (vca_* session token) doesn't support vercel deploy. Fix by unsetting VERCEL_TOKEN before running vercel link + vercel deploy so the CLI falls back to ~/.local/share/com.vercel.cli/auth.json which has proper auth. Also run vercel link --yes --scope vercel-labs --project <name> before deploy.
Hoist runId to module-level let so it's accessible from runScenario for archive paths. Add sourcePath field to ScenarioResult.
Verified: source.tar.gz extracted successfully (17KB for ai-writing-assistant with shadcn components, API routes, app structure).
Rewrite run-eval.ts with a two-phase eval flow:
- Phase 1: Claude Code builds a Next.js app (existing)
- Phase 2: A follow-up Claude Code session uses agent-browser to walk through 3 user stories per app, fixing issues until all pass
New scenarios (replacing the old 5): pomodoro-timer, color-palette-gen, markdown-previewer, weather-dashboard, quiz-builder. Each has 3 concrete user stories like "As a user, I can type markdown and see it rendered as a heading in the preview".
Install agent-browser globally alongside claude-code and vercel in each sandbox. After build completes and dev server starts on port 3000, build a verification prompt from the user stories that instructs Claude to use agent-browser open/snapshot/click/fill/screenshot commands. Parse VERIFICATION_RESULTS from output to track pass/fail per story.
Add --skip-verify flag to skip Phase 2. Summary table now shows Build/Skills/Files/Verify columns. Verification details section shows per-story pass/fail with checkmarks.
Proven: markdown-previewer 3/3 passed, quiz-builder 3/3 passed (6/6 total).
Usage: bun run .claude/skills/benchmark-sandbox/run-eval.ts --keep-alive --keep-hours 8 --concurrency 5
When --keep-alive is passed: after each Claude Code session times out (5-min Hobby cap), start `npx next dev --turbopack` in the background, call `sandbox.extendTimeout()` to keep it alive for --keep-hours (default 8), and print the public https://sb-XXXXX.vercel.run URL. The process blocks at the end so sandboxes stay alive for overnight checks.
Also add ports: [3000] to Sandbox.create() so domain(3000) returns a public URL at creation time. Update the 20s poll loop to include port 3000 HTTP status via curl. Only stop sandboxes in the finally block when --keep-alive is NOT set.
Usage: bun run .claude/skills/benchmark-sandbox/run-eval.ts --keep-alive --keep-hours 8 --concurrency 5
- Add stemmer and shared contractions modules for lexical prompt matching
- Enhance lexical index and prompt patterns with stemming support
- Add promptSignals metadata to all 43 skill frontmatter files
- Add comprehensive documentation site (docs/)
- Add .claude-plugin marketplace and plugin metadata
- Add benchmark scenarios script
- Update skill manifest with prompt signal data
- Add lexical-index and stemmer tests, expand prompt-patterns tests
Proven working eval system that runs Claude Code sessions inside Vercel
Sandboxes with the vercel-plugin installed. Key capabilities:
- 5 parallel sandboxes with unique public URLs (ports: [3000])
- Fresh sandbox per scenario (no snapshots — npm globals don't persist)
- Plugin uploaded via writeFiles() + installed via npx add-plugin
- Claude Code with --dangerously-skip-permissions --debug
- 20s progress polling (skills, files, port 3000 status)
- Skill coverage analysis (expected vs actual)
Critical environment findings documented in SKILL.md:
- Home dir: /home/vercel-sandbox (not /home/user)
- SDK: @vercel/sandbox@1.8.0 with ports: [3000] for public URLs
- Hobby tier caps at 5 min regardless of timeout param
- add-plugin works because claude is in sh PATH after npm -g install
Add scorePromptWithLexical as an additive wrapper around the existing exact prompt matcher.
It preserves current matching behavior, then falls back to lexical index hits when the exact score stays below threshold.
Verified: bun test tests/prompt-patterns-lexical.test.ts
Verified: tsc -p hooks/tsconfig.json --noEmit
Swarm-Agent: codex-prompt-patterns
Add a MiniSearch-backed lexical index for retrieval frontmatter with synonym and contraction expansion so hooks can rank skills from short natural-language queries.
Verified: bun test hooks/lexical-index.test.ts
Verified: ./node_modules/.bin/tsc --noEmit --module NodeNext --moduleResolution NodeNext --target ES2022 --strict --skipLibCheck --types node hooks/src/lexical-index.mts
Swarm-Agent: codex-lexical-index
Add a shared rankSkills utility that combines path, command, import, profiler, prompt, lexical, and priority signals into a single sortable score with per-signal breakdowns.
Add a focused Bun regression test covering weighted scoring, ordering, and default field behavior for sparse candidates.
Verified: bun test tests/unified-ranker.test.ts
How to test: bun test tests/unified-ranker.test.ts
Swarm-Agent: codex-unified-ranker
Eval analysis of 9 real sessions showed 10 skills never triggering despite being
requested. Root causes: pathPatterns too narrow (agents write to lib/email-template.tsx
not emails/), promptSignals containing regex instead of plain text (vercel-sandbox),
and missing promptSignals entirely (v0-dev, vercel-firewall).
Skills updated: email, vercel-queues, edge-runtime, vercel-firewall, chat-sdk,
v0-dev, vercel-sandbox. New skill: next-forge (bootstrap detection).
Update skill frontmatter to reflect the file paths and prompt language agents
actually use for email templates, workflow-backed queue code, edge runtime
middleware entrypoints, and firewall configuration prompts.
Verified: bun test tests/skill-map-frontmatter.test.ts
Verified: bun --eval "import { buildSkillMap } from './hooks/src/skill-map-frontmatter.mts'; import { compileSkillPatterns, matchImportWithReason, matchPathWithReason } from './hooks/src/patterns.mts'; import { compilePromptSignals, matchPromptWithReason, normalizePromptText } from './hooks/src/prompt-patterns.mts'; const skillMap = buildSkillMap('./skills').skills; const compiled = compileSkillPatterns(skillMap); const bySkill = (name) => { const entry = compiled.find((item) => item.skill === name); if (!entry) throw new Error('missing skill: ' + name); return entry; }; if (!matchPathWithReason('lib/email-template.tsx', bySkill('email').compiledPaths)) throw new Error('email pathPatterns did not match lib/email-template.tsx'); const queueEntry = bySkill('vercel-queues'); if (!matchPathWithReason('app/api/workflows/process/route.ts', queueEntry.compiledPaths)) throw new Error('vercel-queues pathPatterns did not match app/api/workflows/process/route.ts'); if (!matchImportWithReason(\"import { workflow } from '@vercel/workflow'\", queueEntry.compiledImports)) throw new Error('vercel-queues importPatterns did not match @vercel/workflow import'); const edgeSignals = compilePromptSignals(skillMap['edge-runtime'].promptSignals); if (!matchPathWithReason('middleware.ts', bySkill('edge-runtime').compiledPaths)) throw new Error('edge-runtime pathPatterns did not match middleware.ts'); if (!matchPromptWithReason(normalizePromptText('I need an edge function that should run at the edge'), edgeSignals).matched) throw new Error('edge-runtime promptSignals did not match edge prompt'); const firewallSignals = compilePromptSignals(skillMap['vercel-firewall'].promptSignals); if (!matchPromptWithReason(normalizePromptText('Add rate limiting and WAF protection to this app'), firewallSignals).matched) throw new Error('vercel-firewall promptSignals did not match firewall prompt'); console.log('skill trigger verification passed');"\nSwarm-Agent: codex-patterns-infra
Add a PreToolUse observer hook that records Agent tool launches as pending subagent spawn metadata for downstream bootstrap logic.
It preserves the existing no-mutation contract by always returning {} and now integrates with the committed subagent-state append API.
Verified: bun test tests/pretooluse-subagent-spawn-observe.test.ts
Verified: ./node_modules/.bin/tsc -p hooks/tsconfig.json --noEmit
How to test: run bun test tests/pretooluse-subagent-spawn-observe.test.ts
Swarm-Agent: codex-observer
- SubagentStart bootstrap hook injects project context (likely skills, summaries) into spawned subagents
with budget scaling by agent type (minimal for Explore/Plan, standard for general-purpose)
- SubagentStop sync hook writes agent metadata to a session-scoped JSONL ledger for observability
- SessionEnd cleanup extended to remove subagent ledger files
- Updated ai-elements/nextjs skills, benchmark-agents and eval skill definitions
Hook modules were creating separate logger instances, so a single PreToolUse run could emit multiple invocationIds once hook-env catch logging fired. Reusing one process-scoped invocationId keeps all lines from one hook invocation correlated.
This also demotes internal trigger diagnostics back to debug so summary mode stays limited to complete and issue events, matching the logger contract and tests.
Verified: bun test tests/pretooluse-skill-inject.test.ts (278 tests pass)
Verified: bun test tests/logger.test.ts (12 tests pass)
Verified: bun test tests/hook-sync.test.ts -t "logger .mts/.mjs sync|pretooluse-skill-inject .mts/.mjs sync" (6 tests pass)
Swarm-Agent: codex-invocation-id-fix
Make the session-start profiler resolve binaries from PATH safely before
invoking them, cap the version-check subprocesses at 3 seconds, and
avoid crashing when npm or agent-browser is missing.
Also expand the outdated Vercel CLI guidance to include the pnpm global
upgrade path and cover the new skip/timeout behavior in profiler tests.
Verified: bun test tests/session-start-profiler.test.ts
Swarm-Agent: codex-profiler-harden-split-2-v2
Replace empty catch blocks in hook-env and session-start-profiler with\nstructured debug logging using the shared hook logger.\nAlso make the Vercel CLI update check compare numeric version\nsegments so 1.9.0 correctly sorts below 1.10.0.\n\nVerified: bun test tests/session-start-profiler.test.ts\nHow to test: bun test tests/session-start-profiler.test.ts\nSwarm-Agent: codex-profiler-harden-split-1
Skill frontmatter (pathPatterns, bashPatterns, promptSignals, etc.)
was being injected alongside the skill body, wasting token budget on
metadata only useful for hook matching. Now uses extractFrontmatter()
to emit only the markdown body.
The PreToolUse hook restores path regexes from generated/skill-manifest.json when a v2 manifest is present, so the brace-expansion fix also needs regenerated pathRegexSources. This refresh updates the affected extension-list patterns from literal brace matches to alternations.
Verified: bun run build:manifest
Verified: bun test tests/pretooluse-skill-inject.test.ts -t "matches src/middleware\.(mjs|mts) to routing-middleware skill"
Swarm-Agent: codex-brace-expand
Brace groups like {ts,js,mjs} were being escaped literally, which prevented extension-list path patterns from matching. The glob parser now expands balanced brace groups into recursive regex alternations while preserving literal braces when no alternation is present.
Verified: bun test tests/patterns.test.ts
Verified: bun test tests/fuzz-glob.test.ts
Verified: bun test tests/hook-sync.test.ts
Verified: bun test tests/pretooluse-skill-inject.test.ts -t "matches src/middleware\.(mjs|mts) to routing-middleware skill"
Swarm-Agent: codex-brace-expand
Hash invalid session IDs before constructing dedup temp paths so crafted stdin values cannot smuggle traversal segments into recursive claim cleanup. Shared temp-path resolution now verifies the resolved target stays under tmpdir, and seen-skills tests cover both stable safe IDs and hashed invalid IDs.
Verified: bun test tests/session-start-seen-skills.test.ts
How to test: bun test tests/session-start-seen-skills.test.ts
Swarm-Agent: codex-path-traversal
New ai-generation-persistence skill (priority 6) injects guidance for treating
AI generations as first-class persistent resources — unique IDs, addressable
URLs, database/Blob storage, cost tracking, and generate-then-redirect UX
patterns. Triggers on AI SDK imports and broad prompt signals.
New verification skill added. Updated ecosystem graph, catalog, manifest,
fixtures, and snapshots.
Plain "provider/model" strings (e.g., model: "openai/gpt-5.4") route
through AI Gateway automatically — the gateway() wrapper is optional and
only needed for providerOptions.gateway (routing, failover, tags). Updated
vercel.md, ai-sdk, and ai-gateway skills to match official Vercel docs.
melkeydev