* Reduce unnecessary CI runtime
* Fix shared E2E artifact extraction path
* Stabilize getWorkflowPort timeout test on Windows
* Preserve UI unit coverage on CI fast path
* ci: extract wait-for-vercel-project to vercel/wait-for-deployment-action
The action's logic was duplicated between this repo and
vercel/workflow-server, which is annoying to keep in sync. Move it to
a standalone repository so both can consume the same pinned build.
Changes:
- Delete .github/actions/wait-for-vercel-project entirely.
- Replace all five `uses: ./.github/actions/wait-for-vercel-project`
references with `uses: vercel/wait-for-deployment-action@<sha>` in:
benchmarks.yml, dispatch-front-workflow-release-pr.yml,
docs-checks.yml, tarballs-checks.yml, tests.yml
- All `with:` inputs (project-slug, environment, timeout,
check-interval, github-token) are unchanged — the new action's
input contract is backwards-compatible.
The new action is ESM-only, targets Node 24, ships a ~12KB bundle
(down from ~830KB in the old in-repo version) by dropping
@actions/core and its transitive undici dependency, and is
unit-tested. See https://github.com/vercel/wait-for-deployment-action.
* ci: bump wait-for-deployment-action to fix/status-context-auto for verification
Repinning to vercel/wait-for-deployment-action#fix/status-context-auto
(SHA 04d46ef) which fixes the broken 'opt-out' heuristic that made
status-context resolution silently disabled for every consumer.
Reproduced in this repo's E2E logs:
Looking for GitHub deployment in environment "Preview – example-workflow"
Deployment ID resolution disabled (status-context is empty)
Deployment ready: https://example-workflow-...labs.vercel.dev
Run E2E Tests: VERCEL_DEPLOYMENT_ID= <-- empty
Will repin to the post-merge main SHA once CI is green.
* ci: bump wait-for-deployment-action pin to merged main SHA
Repinning from the fix/status-context-auto branch (04d46ef) to the
post-merge main SHA (0e2b0c5, vercel/wait-for-deployment-action#4).
The deployment-id resolution fix verified against the prior fix-branch
pin (E2E tests now read VERCEL_DEPLOYMENT_ID=dpl_... correctly across
the matrix; only flaky/unrelated Vercel deployment failures remain).
* ci: grant statuses:read alongside deployments:read
The wait-for-deployment-action also reads the 'Vercel – <slug>'
combined commit status to resolve the dpl_xxx ID. The official
permissions table lists statuses:read for
GET /repos/{owner}/{repo}/commits/{ref}/status.
Major-version refs like `@v2`/`@v5` resolve to mutable refs on the
upstream repos — sometimes a tag, sometimes a branch (e.g. marocchino
keeps `v1`/`v2`/`v3` as branches), and dawidd6 force-pushes the bare
`v6` tag forward outside of releases. A compromised maintainer account
could push new code that our CI picks up on the next run with
GITHUB_TOKEN (or, for changesets/action, NPM_TOKEN) in hand.
Pin all third-party `uses:` references to full commit SHAs with a
trailing version comment so the upstream release is still visible to
reviewers. Dependabot/Renovate can keep these fresh going forward.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The repo's enterprise `~ALL` required-signatures ruleset rejects the
unsigned commits produced by `peaceiris/actions-gh-pages@v4`, breaking
the benchmark and E2E result publishing jobs on `main`. Replace those
steps with a shared composite action that uses the GraphQL
`createCommitOnBranch` mutation — same pattern already used by
`backport.yml` — so commits are signed automatically by GitHub and
satisfy the rule.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* ci: refactor wait-for-vercel-project to use GitHub Deployments API
Replaces the Vercel SDK / Vercel API token-based implementation with one
that resolves the deployment URL via the GitHub Deployments API:
- Find the GitHub Deployment for (target SHA, environment) where
environment matches the Vercel-app-created "Preview \u2013 <slug>" or
"Production \u2013 <slug>" naming pattern.
- Wait for the latest deployment status to be `success` (or `inactive`
when Vercel skips a duplicate build, in which case its environment_url
still points at the live deployment).
- Probe the URL to confirm the edge can route to it (any non-5xx
response counts as live, including 401/403 from Deployment Protection
and 404/405 from the app). Manual redirect handling treats redirects
to vercel.com as "still building".
- Resolve the dpl_xxx deployment ID from the matching commit status
(Vercel posts `Vercel \u2013 <slug>` statuses where target_url's last
path segment is the inspector ID == deployment ID without the prefix).
Inputs change: project-slug + bypass-secret + github-token (with
GITHUB_TOKEN default) replace team-id + project-id + vercel-token.
Removes the @vercel/sdk dependency, shrinking the bundled dist from
5.4MB to 829KB. The VERCEL_DOCS_TOKEN secret is no longer referenced
anywhere in the repo and can be deleted from GH after this lands.
* ci(wait-for-vercel-project): drop URL probe and bypass-secret input
The GitHub Deployment status transitions to `success` only after the
Vercel app finishes building and routing is live, so an extra HTTP
liveness probe of the deployment URL was redundant. Removing it lets
us also drop the bypass-secret input \u2014 protected deployments don't
need a workaround anymore because we never make the request.
Reduces the action surface area and eliminates a runtime fetch.
* ci(wait-for-vercel-project): address PR review
- Fail loudly when the dpl_xxx deployment ID can't be resolved instead
of returning an empty string. Consumers wire this into
VERCEL_DEPLOYMENT_ID, which world-target uses to pick between the
vercel and local worlds (packages/utils/src/world-target.ts), so an
empty value would silently flip execution mode.
- Pass the GitHub App token to wait-for-vercel-project in the dispatch
release workflow. The job sets `permissions: contents: read`, which
blocks the default GITHUB_TOKEN from reading the Deployments API.
The App token (already generated for workflow,front) has the
necessary scopes.
* ci: fix VERCEL_WORKFLOW_SERVER_* ternary so main actually unsets them
In GitHub Actions expressions, '' is falsy, so the original
`cond && '' || secrets.X` pattern always fell through to the secret
regardless of branch. The result was that pushes to main were sending
preview workflow-server values to production, causing 'invalid_url'
errors on `x-vercel-workflow-api-url` across all e2e jobs.
Flip the condition so the secret sits in the truthy branch and ||
correctly selects '' on main.
* ci: skip pnpm cache in matrix-generation jobs
The Get Test Matrix and Get Community Worlds Matrix jobs only run a
small Node script to emit a JSON matrix; they never run `pnpm install`.
With `cache: 'pnpm'` set on actions/setup-node, the post-job cache save
step fails with 'Path Validation Error' because the pnpm store path was
never created, marking the whole job as failed.
Add a cache-pnpm input to setup-workflow-dev (default true) and opt out
in the two matrix-generation jobs.
* CI script improvements
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* address codex feedback: harden remaining workflows
- add allowlist regex in prepare-workbench-path to block path traversal
- move matrix/input values to env vars across e2e-vercel-prod,
benchmarks (local/postgres/vercel), and the reusable community-world
workflows
- validate app-name/world-id/world-package inputs in the reusable
community-world workflows
- pipe getCommunityWorldsMatrix script output through jq -c to prevent
\$GITHUB_OUTPUT injection
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
* ci: upgrade pnpm/action-setup to v6 and read version from package.json
Removes hardcoded pnpm version (10.14.0) from all workflows and instead
reads the version from the packageManager field in package.json, so CI
stays in sync with the version used locally.
* ci: update setup-workflow-dev composite action to use pnpm/action-setup@v6
Also removes the pnpm-version input since the action now reads the
version from package.json#packageManager.
* ci: downgrade pnpm/action-setup to v5
v6 installs pnpm 11 RC/beta, which has a regression
(pnpm/pnpm#11264, pnpm/action-setup#225/#227/#228) that causes
'ERR_PNPM_BROKEN_LOCKFILE: expected a single document in the stream'
when the project's packageManager pins a 10.x pnpm version. v5 is the
latest stable release before v6 and supports reading the version from
package.json#packageManager.
* Add new github workflow
* Enable pull_request trigger for community worlds workflow
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
* Add community worlds manifest and generation scripts
- Add community-worlds.json manifest as single source of truth
- Add scripts/generate-community-worlds-workflow.mjs to generate CI workflow
- Add scripts/generate-community-worlds-docs.mjs to generate docs section
- Update aggregate-benchmarks.js to load community worlds dynamically
- Add pnpm generate:community-worlds script
- Update docs/deploying/world/index.mdx with community worlds
The manifest-based approach allows:
- E2E tests to be auto-generated from the manifest
- Benchmark aggregation to include community worlds
- Docs to stay in sync with tested worlds
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
* Fix YAML syntax error - quote strings starting with @
The @ symbol has special meaning in YAML, so package names like
@workflow-worlds/turso need to be quoted.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
* Fix Redis health-cmd quoting, remove unpublished starter world
- Quote health-cmd when it contains spaces (fixes Docker arg parsing)
- Remove @workflow-worlds/starter as it's not published to npm
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
* Add benchmarks and summary job for community worlds
- Add build job to share artifacts between benchmark jobs
- Add benchmark jobs for Turso, MongoDB, and Redis worlds
- Update summary job to show both E2E and benchmark status matrix
- Add left border/indent to sidebar child items for visual hierarchy
- Update workflow generator to support benchmark generation
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
* Reuse build artifacts for E2E tests
E2E jobs now depend on the shared build job and download
artifacts instead of rebuilding packages from scratch.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
* Add Worlds Ecosystem dashboard to docs
- Create worlds-manifest.json with official and community worlds
- Add aggregate-worlds-data.mjs script for processing E2E and benchmark results
- Create WorldsDashboard, WorldCard, and BenchmarkChart components
- Add /docs/worlds page showing compatibility status and performance
- Include sample data for development
The dashboard shows:
- E2E test progress per world (pass/fail/skip counts)
- Benchmark performance comparison across all worlds
- Filter by official vs community worlds
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
* Fix CI to output JSON test results and add Jazz world
- Update workflow generator to output JSON test results from vitest
- Upload E2E results as artifacts for parsing in summary job
- Summary job now shows actual pass/fail/skip counts per world
- Add Jazz world to worlds-manifest.json (requires external credentials)
- Add update-worlds-status.yml workflow to auto-update dashboard data
- Update TypeScript types to support null lastRun and metrics
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
* Refactor community worlds to use reusable workflows
Instead of a generated workflow file, integrate community world testing
directly into tests.yml and benchmarks.yml using reusable workflows.
- Add reusable workflows for E2E tests: e2e-community-world.yml (no services),
e2e-community-world-mongodb.yml, e2e-community-world-redis.yml
- Add reusable workflows for benchmarks: benchmark-community-world.yml,
benchmark-community-world-mongodb.yml, benchmark-community-world-redis.yml
- Update tests.yml to call reusable workflows for Turso, MongoDB, Redis
- Update benchmarks.yml to include community world benchmarks in summary
- Delete generated community-worlds.yml and generator script
This approach:
- Inherits proper Rust/SWC setup from the main workflows
- Keeps all CI in the established patterns
- Makes adding new community worlds straightforward
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
* Fix benchmark timing file naming for community worlds
Add WORKFLOW_BENCH_BACKEND env var support to bench.bench.ts so community
world benchmarks generate timing files with the correct backend suffix
(e.g., bench-timings-nextjs-turbopack-turso.json instead of -local.json).
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
* Add @workflow-worlds/starter to community worlds test matrix
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
* Unify worlds manifest and add dynamic GitHub API fetching
- Merge community-worlds.json into worlds-manifest.json with type field
- Add server-side data fetching from GitHub API for worlds dashboard
- Remove static worlds-status.json, fetch CI artifacts dynamically
- Update all references to use unified manifest format
- Remove obsolete community-worlds.yml and update-worlds-status.yml workflows
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
* Skip community worlds for non-nextjs-turbopack in benchmark summary
Community worlds only run against nextjs-turbopack, so hide the
"missing" rows for Express and Nitro frameworks in the benchmark
comparison tables.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
* Fix stream benchmark detection to check for actual TTFB data
The previous check `!== null` incorrectly returned true for undefined,
causing all benchmarks to show TTFB columns. Now explicitly checks
for a number type.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
* Hide Worlds Ecosystem page from sidebar
The page is still accessible via direct link at /docs/worlds but
won't appear in the navigation until it's been further iterated on.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
* Fix review comments: trailing newline and division by zero
- Add trailing newline when replacing Community Worlds section in docs
- Fix division by zero in WorldCard benchmark calculation when metrics is empty
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
* Consolidate community world workflows with service-type parameter
- Create setup-workflow-dev composite action for common setup steps
- Add service-type input to benchmark-community-world.yml and e2e-community-world.yml
- Use conditional job execution (if: inputs.service-type == 'mongodb') to handle different services
- Update benchmarks.yml and tests.yml to pass service-type parameter
- Delete redundant workflow files:
- benchmark-community-world-mongodb.yml
- benchmark-community-world-redis.yml
- e2e-community-world-mongodb.yml
- e2e-community-world-redis.yml
Reduces workflow files from 11 to 7 and eliminates ~500 lines of duplicated YAML.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
* Generate community world test matrix from worlds-manifest.json
Replace hardcoded community world jobs with dynamic matrix generation using
scripts/create-community-worlds-matrix.mjs. This allows adding/removing
community worlds by editing the manifest instead of multiple workflow files.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
* Apply suggestion from @vercel[bot]
Co-authored-by: vercel[bot] <35613825+vercel[bot]@users.noreply.github.com>
* Add Samples column and separate local/production benchmarks
- Add Samples column to all benchmark tables showing iteration count
- Separate benchmark results into Local Development and Production sections
- Add explanatory context for each section (localhost vs Vercel deployment)
- Add GitHub action step summaries to e2e community world tests
- Create aggregate-e2e-results.js script for parsing vitest JSON output
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
* Remove obsolete generate-community-worlds-docs script
The Worlds Ecosystem page now fetches from worlds-manifest.json at runtime,
making this script unnecessary. The npm script also referenced a non-existent
workflow generator script.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
* Maximize composite action usage and reorganize benchmark output
- Update setup-workflow-dev composite action with optional Rust, install-dependencies, and install-args inputs
- Update tests.yml to use composite action in unit, e2e-vercel-prod, getTestMatrix, e2e-local-*, and getCommunityWorldsMatrix jobs
- Update benchmarks.yml to use composite action in build, benchmark-local, benchmark-postgres, benchmark-vercel, and getCommunityWorldsMatrix jobs
- Reorganize benchmark output to group by benchmark test with local/production tables within each benchmark
- Remove invalid $schema reference from worlds-manifest.json
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
* Add beads stealth mode stuff (for personal claude memory - will remove stealth if people want)
* Add E2E test results PR comment summary
- Add pr-comment-start job to create/update PR comment when tests start
- Add artifact uploads to all e2e test jobs (vercel-prod, local-dev, local-prod, local-postgres, windows)
- Update e2e-community-world.yml with consistent artifact naming (e2e-community-*)
- Add summary job to aggregate all e2e results and update PR comment
- Extend aggregate-e2e-results.js with --mode aggregate for multi-job PR summary
- Group results by category (Vercel Production, Local Development, etc.)
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
* Add step summaries to all e2e test jobs
Add "Generate E2E summary" step to each individual e2e job:
- e2e-vercel-prod
- e2e-local-dev
- e2e-local-prod
- e2e-local-postgres
- e2e-windows
Each job now outputs pass/fail/skip counts to GITHUB_STEP_SUMMARY.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
* Consolidate community world workflows to single job
Replace 3 mutually exclusive jobs (e2e/e2e-mongodb/e2e-redis) with a single
job that starts services via docker run when needed. This eliminates the
skipped job entries that appear in the GitHub Actions UI.
- Use conditional docker run steps instead of services: block
- Add health check loops to wait for service readiness
- Add cleanup step to stop containers
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
* Fix extractWorldId to handle community world artifact naming
Add handling for `e2e-results-community-{world}` pattern so community
world test results are properly extracted (e.g., `e2e-results-community-turso`
now correctly extracts `turso` instead of `community-turso`).
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
---------
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: vercel[bot] <35613825+vercel[bot]@users.noreply.github.com>
* Add stream benchmarks and cnealup for benchmark code
* 10m delay for local world race
* potential fix for first bute"
* Improve sticky comment stuff
* local world: ignore controller close errors
* show missing data in benchmark comment
* nitrpicks
* Improve leaderboard
* Add benchmark comparisons against main
* changeset
Nathan Colosimo