skills/vercel__workflow
1
0
Fork 0
mirror of https://github.com/vercel/workflow.git synced 2026-09-14 19:59:43 +08:00
Code Issues Packages Projects Releases Wiki Activity
Files
codex/stream-tail-cursor
vercel__workflow/.github/workflows/tests.yml
T
Shalabh Chaturvedi dc68611fbf Default the events transport to WebSockets (#3702) 
* Default the events transport to WebSockets

WORKFLOW_EVENTS_TRANSPORT=http is the opt-out. Only that exact value
disables it, so a typo'd or empty value fails toward the default rather
than quietly pinning a deployment to HTTP.

The prerequisite the gate named for defaulting on is met:
postEventFrameOverWs opens a client span per frame. What is still missing
is Vercel's outgoing-requests view, which reads instrumented fetch calls
rather than spans and so cannot show a transport that issues no request.

Co-Authored-By: opencode <opencode@vercel.com>

Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com>

* docs: WORKFLOW_EVENTS_TRANSPORT defaults to ws

Three places still documented http as the default. Each now states the
opt-out is the exact value http, rather than leaving 'default: ws' to
imply that anything non-ws disables it — the asymmetry is deliberate in
the code and is the part a reader would otherwise get wrong.

Also drops 'Experimental' from the Vercel World page: a setting that is
on for everyone by default is not opt-in experimental, whatever else it
is.

Co-Authored-By: opencode <opencode@vercel.com>

Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com>

* Fix the gate's own unit tests for the flipped default

Five tests in ws-transport.test.ts still encoded the opt-in semantics.
Three were the isWsEventsTransportEnabled table itself; the other two
(openWsChannel 'does nothing when the gate is off', and the channel
release equivalent) relied on the suite's ambient unset environment
meaning 'off', which it no longer does. Both now set http explicitly.

Two tests in ws-transport-spans.test.ts asserted HTTP-side span
behaviour the same way. The write one would have kept passing by
falling through resolveWsTransport's null rather than because the gate
was off - passing for the wrong reason, which is what this file exists
to catch.

Also makes the opt-out case-insensitive and trimmed. The gate is
deliberately asymmetric - unrecognized values take the default - but
that asymmetry should not extend to swallowing HTTP or ' http '.
Whoever reaches for the escape hatch is plausibly mid-incident, and
silently ignoring their opt-out over a capital letter is the same class
of silent-wrong-transport bug this flip is meant to stop shipping.

554 tests pass in packages/world-vercel.

Co-Authored-By: opencode <opencode@vercel.com>

Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com>

* ci: add a required forced-HTTP e2e lane (#3703)

Flipping the default makes e2e-vercel-prod a WebSocket lane: it sets no
WORKFLOW_EVENTS_TRANSPORT, and unset now means ws. Nothing in the file
would exercise the HTTP events transport against a real deployment any
more, so this is not additive coverage — it replaces coverage the flip
silently removed.

Unconditional and required rather than label-gated like the WS lane.
HTTP is now the fallback, and the fallback is silent: resolveWsTransport
returning null costs a write nothing and logs nothing, which is the
shape of the durabench bug this stack came out of.

Two apps rather than the WS lane's four, since every row is a real
vercel deploy charged to every PR. nextjs-turbopack is the only fixture
emitting OTEL spans, so it is the one that can show which transport
actually ran; express covers the non-Next server path.

Also corrects the WS lane's docblock, which claimed every other job
exercises HTTP only. That stopped being true one commit ago.

Co-authored-by: vercel[bot] <35613825+vercel[bot]@users.noreply.github.com>

Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com>

* Fail loudly when step_completed falls back to HTTP under a strict flag

The WS e2e lane asserts that the transport is harmless, not that it is
used: an event written over HTTP produces the same run outcome as one
written over the socket, so the lane stayed green through the entire
period the transport was silently demoted.

WORKFLOW_INTERNAL_EVENTS_TRANSPORT_STRICT turns that one case into a
failed run, and the WS lane now sets it.

Scoped to step_completed alone, because most fallback is legitimate:
run_created is written outside any invocation that opens a channel;
run_started routinely lands before the channel is registered (34% HTTP
on a healthy deployment); step_created and wait_created mostly fold into
events.createBatch, which is not wired to the socket; and a write after
the invocation released its claim falls back by design. step_completed
is issued after a step body has run, and was 100% ws across every
WS-enabled deployment measured on two SDK versions.

The flag reads as off unless the value is exactly 1 or true - the
opposite asymmetry from the transport gate, which treats an unrecognized
value as on. That gate risks a deployment sitting quietly on the wrong
transport; this one fails runs, and should not be acquired by a typo.

Co-Authored-By: opencode <opencode@vercel.com>

Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com>

* ci: run the WS transport lane on every PR

It was opt-in behind ws-transport-test because four real vercel deploys
were too much to charge an unrelated PR for a transport that was off by
default. Flipping the default expires that reasoning from both ends: the
cost is no longer for someone else's feature, and this is now the only
lane that asserts the socket carried the events. e2e-vercel-prod
inherits the new default but checks nothing, so behind a label the
average PR would move every deployment onto WebSockets with nothing
verifying they were used.

Drops WS_REQUIRED from the gate along with it. That existed only to let
the lane be legitimately skipped on an unlabelled PR; with no label the
lane is required unconditionally, like e2e-vercel-prod and the HTTP
lane, and the skipped case is now a failure rather than a warning.

Gate script extracted and run against the cases that matter: ws skipped
fails on a standard PR, ws skipped fails under workflow-server-test, and
all-green passes.

Co-Authored-By: opencode <opencode@vercel.com>

Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com>

* ci: widen the HTTP transport lane to six server shapes

Before the flip, HTTP was the default and all 28 e2e-vercel-prod
lane-runs covered it. After the flip they cover WebSockets instead, and
this lane is the entirety of the HTTP coverage - two apps was too thin
for a transport that is still supported.

Six, not the full 14, because every row is a real vercel deploy charged
to every PR. Chosen by server shape rather than count: example
(baseline), nextjs-turbopack (Next, and the only fixture emitting OTEL
spans), vite (Vite SSR), express (Node req/res), nitro (h3, also covers
nuxt) and hono (fetch-API Request/Response, a different mount shape from
express). The rest duplicate a shape already covered; python is left out
because it has no conformance gate and needs routes this suite does not
serve.

The first four match the WS lane's matrix on purpose, so the same
fixture runs on both transports and a failure on one can be read against
the other.

Project ids and slugs are copied from e2e-vercel-prod and verified equal
to it; both lanes already use the same team and token.

Co-Authored-By: opencode <opencode@vercel.com>

Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com>

---------

Co-authored-by: vercel[bot] <35613825+vercel[bot]@users.noreply.github.com>
2026-08-21 17:17:04 -07:00

1793 lines
79 KiB
YAML
Raw Permalink Blame History

name: Tests
on:
push:
branches:
- main
tags:
- "!*"
pull_request:
types:
- opened
- reopened
- synchronize
concurrency:
# Unique group for this workflow and branch
group: ${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: ${{ github.ref != 'refs/heads/main' }}
jobs:
ci-scope:
name: Detect CI Scope
runs-on: ubuntu-latest
outputs:
fast-path: ${{ steps.scope.outputs.runtime-fast-path }}
validation-fast-path: ${{ steps.scope.outputs.validation-fast-path }}
steps:
- name: Checkout Repo
uses: actions/checkout@v4
with:
ref: ${{ github.event.pull_request.head.sha || github.sha }}
- name: Classify changed files
id: scope
uses: ./.github/actions/detect-ci-scope
with:
github-token: ${{ secrets.GITHUB_TOKEN }}
# Phase 0: Update PR comment to show tests are running
pr-comment-start:
name: Create PR Comment
runs-on: ubuntu-latest
needs: ci-scope
if: github.event_name == 'pull_request' && needs.ci-scope.outputs.fast-path != 'true'
timeout-minutes: 2
steps:
- name: Find existing test comment
uses: peter-evans/find-comment@3eae4d37986fb5a8592848f6a574fdf654e61f9e # v3.1.0
id: find-comment
with:
issue-number: ${{ github.event.pull_request.number }}
comment-author: 'github-actions[bot]'
body-includes: '<!-- e2e-test-results -->'
- name: Get existing comment body
if: steps.find-comment.outputs.comment-id != ''
id: get-comment
uses: actions/github-script@v7
env:
STARTED_AT: ${{ github.run_started_at }}
with:
script: |
const fs = require('fs');
const comment = await github.rest.issues.getComment({
owner: context.repo.owner,
repo: context.repo.repo,
comment_id: ${{ steps.find-comment.outputs.comment-id }}
});
const body = comment.data.body;
// Check if there are actual results (tables)
if (body.includes('|') && body.includes('Passed')) {
// Extract results section (everything after header)
const resultsSection = body
.replace(/<!-- e2e-test-results -->\n## 🧪 E2E Test Results\n\n> ⚠️ \*\*Results below are stale\*\*[^\n]*\n\n/g, '')
.replace(/<!-- e2e-test-results -->\n## 🧪 E2E Test Results\n\n/g, '')
.replace(/⏳ \*\*Tests are running\.\.\.\*\*\n\n---\n_Started at:[^_]*_\n\n---\n\n/g, '')
.replace(/⏳ \*\*Tests are running\.\.\.\*\*\n\n---\n_Started at:[^_]*_/g, '')
.trim();
if (resultsSection && resultsSection.includes('|')) {
// Write the full stale-banner message to disk and pass the
// path to the sticky-pull-request-comment action below.
// Inlining the previous results via `message:` blew past
// ARG_MAX once the matrix doubled (snapshot + replay).
const startedAt = process.env.STARTED_AT;
const message =
'<!-- e2e-test-results -->\n' +
'## 🧪 E2E Test Results\n\n' +
'> ⚠️ **Results below are stale** and not from the latest commit. This comment will be updated when CI completes on the latest run.\n\n' +
'⏳ **Tests are running...**\n\n' +
'---\n' +
`_Started at: ${startedAt}_\n\n` +
'---\n\n' +
resultsSection +
'\n';
const path = `${process.env.RUNNER_TEMP}/stale-comment.md`;
fs.writeFileSync(path, message);
core.setOutput('has-results', 'true');
core.setOutput('stale-comment-path', path);
} else {
core.setOutput('has-results', 'false');
}
} else {
core.setOutput('has-results', 'false');
}
- name: Create new test comment
if: steps.find-comment.outputs.comment-id == ''
uses: marocchino/sticky-pull-request-comment@773744901bac0e8cbb5a0dc842800d45e9b2b405 # v2.9.4
with:
header: e2e-test-results
message: |
<!-- e2e-test-results -->
## 🧪 E2E Test Results
⏳ **Tests are running...**
This comment will be updated with the results when the tests complete.
---
_Started at: ${{ github.run_started_at }}_
- name: Update existing test comment with stale warning
if: steps.find-comment.outputs.comment-id != '' && steps.get-comment.outputs.has-results == 'true'
uses: marocchino/sticky-pull-request-comment@773744901bac0e8cbb5a0dc842800d45e9b2b405 # v2.9.4
with:
header: e2e-test-results
path: ${{ steps.get-comment.outputs.stale-comment-path }}
- name: Update existing test comment without results
if: steps.find-comment.outputs.comment-id != '' && steps.get-comment.outputs.has-results != 'true'
uses: marocchino/sticky-pull-request-comment@773744901bac0e8cbb5a0dc842800d45e9b2b405 # v2.9.4
with:
header: e2e-test-results
message: |
<!-- e2e-test-results -->
## 🧪 E2E Test Results
⏳ **Tests are running...**
This comment will be updated with the results when the tests complete.
---
_Started at: ${{ github.run_started_at }}_
unit:
name: Unit Tests (${{ matrix.os }})
runs-on: ${{ matrix.os }}
timeout-minutes: 30
needs: ci-scope
if: ${{ needs.ci-scope.outputs.validation-fast-path != 'true' && !contains(github.event.pull_request.labels.*.name, 'workflow-server-test') }}
strategy:
fail-fast: false
matrix:
os: [ubuntu-latest, windows-latest]
env:
TURBO_TOKEN: ${{ secrets.TURBO_TOKEN }}
TURBO_TEAM: ${{ vars.TURBO_TEAM }}
steps:
- name: Checkout Repo
uses: actions/checkout@v4
with:
ref: ${{ github.event.pull_request.head.sha || github.sha }}
- name: Setup environment
uses: ./.github/actions/setup-workflow-dev
with:
setup-rust: 'true'
install-args: '--ignore-scripts'
build-packages: 'false'
- name: Run Unit Tests
run: pnpm test --filter='!./docs' --filter='!./workbench/vitest'
serializable-revivers:
name: Serializable Reviver Compatibility
runs-on: ubuntu-latest
timeout-minutes: 10
needs: ci-scope
if: ${{ needs.ci-scope.outputs.validation-fast-path != 'true' && !contains(github.event.pull_request.labels.*.name, 'workflow-server-test') }}
env:
TURBO_TOKEN: ${{ secrets.TURBO_TOKEN }}
TURBO_TEAM: ${{ vars.TURBO_TEAM }}
steps:
- name: Checkout Repo
uses: actions/checkout@v4
with:
ref: ${{ github.event.pull_request.head.sha || github.sha }}
- name: Setup environment
uses: ./.github/actions/setup-workflow-dev
with:
install-args: '--ignore-scripts'
build-packages: 'false'
- name: Build reviver packages
run: pnpm turbo run build --filter='@workflow/cli' --filter='@workflow/web-shared'
- name: Check web-shared and CLI revivers
run: pnpm vitest run packages/web-shared/test/serializable-revivers.test.ts
build-error-messages:
name: Node.js Module Build Errors Test
runs-on: ubuntu-latest
timeout-minutes: 30
needs: ci-scope
if: ${{ needs.ci-scope.outputs.fast-path != 'true' && !contains(github.event.pull_request.labels.*.name, 'workflow-server-test') }}
env:
TURBO_TOKEN: ${{ secrets.TURBO_TOKEN }}
TURBO_TEAM: ${{ vars.TURBO_TEAM }}
steps:
- name: Checkout Repo
uses: actions/checkout@v4
with:
ref: ${{ github.event.pull_request.head.sha || github.sha }}
- name: Setup pnpm
uses: pnpm/action-setup@fc06bc1257f339d1d5d8b3a19a8cae5388b55320 # v5.0.0
- name: Setup Node.js 22.x
uses: actions/setup-node@v4
with:
node-version: 22.x
cache: "pnpm"
- name: Install Dependencies
run: pnpm install --frozen-lockfile
- name: Run Initial Build
run: pnpm turbo run build --filter='!./workbench/*'
- name: Generate Workflows Registry
run: node workbench/scripts/generate-workflows-registry.js workbench/nextjs-turbopack/workflows workbench/nextjs-turbopack/_workflows.ts
- name: Run Build Error Tests
run: pnpm vitest run packages/core/e2e/build-errors.test.ts
env:
APP_NAME: "nextjs-turbopack"
# Regression test for the o2flow beta.26 incident: hook.resume() from a
# plain API route must resolve the workflow world inside an isolated
# (lambda-like) route bundle. Runs in this job because it also performs
# a production Turbopack build of the nextjs-turbopack workbench.
- name: Run Route Bundle Isolation Tests
run: pnpm vitest run packages/core/e2e/route-bundle-isolation.test.ts
env:
APP_NAME: "nextjs-turbopack"
vitest-plugin:
name: Vitest Plugin Tests (${{ matrix.vm }})
runs-on: ubuntu-latest
needs: ci-scope
if: ${{ needs.ci-scope.outputs.fast-path != 'true' }}
strategy:
fail-fast: false
matrix:
# Workflow VM engines: node:vm (default) and the opt-in QuickJS
# WASM engine (WORKFLOW_VM=quickjs).
vm: [node, quickjs]
env:
TURBO_TOKEN: ${{ secrets.TURBO_TOKEN }}
TURBO_TEAM: ${{ vars.TURBO_TEAM }}
steps:
- name: Checkout Repo
uses: actions/checkout@v4
- name: Setup environment
uses: ./.github/actions/setup-workflow-dev
with:
setup-rust: 'true'
install-args: '--ignore-scripts'
build-packages: 'false'
- name: Build packages
run: pnpm turbo run build --filter='!./workbench/*'
- name: Run Vitest Plugin Tests
run: pnpm test
working-directory: workbench/vitest
env:
WORKFLOW_VM: ${{ matrix.vm }}
e2e-package-build:
name: Build Shared E2E Packages
runs-on: ubuntu-latest
needs: ci-scope
if: ${{ needs.ci-scope.outputs.fast-path != 'true' && !contains(github.event.pull_request.labels.*.name, 'workflow-server-test') }}
timeout-minutes: 30
env:
TURBO_TOKEN: ${{ secrets.TURBO_TOKEN }}
TURBO_TEAM: ${{ vars.TURBO_TEAM }}
steps:
- name: Checkout Repo
uses: actions/checkout@v4
with:
ref: ${{ github.event.pull_request.head.sha || github.sha }}
- name: Setup environment and build packages
uses: ./.github/actions/setup-workflow-dev
with:
build-packages: 'true'
- name: Upload package build artifacts
uses: actions/upload-artifact@v4
with:
name: e2e-package-build-artifacts
# Keep this list aligned with package build outputs in turbo.json.
# upload-artifact strips the common packages/ root; consumers restore into packages/.
path: |
packages/*/dist
packages/*/.well-known
packages/*/src/version.ts
packages/core/src/runtime/vm-serde-bundle.generated.ts
packages/core/src/runtime/quickjs-assets.generated.ts
packages/swc-plugin-workflow/swc_plugin_workflow.wasm
packages/swc-plugin-workflow/build-hash.json
include-hidden-files: true
retention-days: 1
e2e-vercel-prod:
name: E2E Vercel Prod Tests (${{ matrix.app.name }} - ${{ matrix.vm }})
runs-on: ubuntu-latest
timeout-minutes: 30
needs: ci-scope
if: ${{ needs.ci-scope.outputs.fast-path != 'true' }}
permissions:
id-token: write
contents: read
deployments: read
statuses: read
strategy:
fail-fast: false
matrix:
# Workflow VM engines: node:vm (default) and the opt-in QuickJS
# WASM engine. The env var is set on the e2e test runner, which is
# the client that starts runs against the deployed app — start()
# stamps executionContext.workflowVm so the deployed handler
# executes each run on the requested engine.
vm: [node, quickjs]
app:
- name: "example"
project-id: "prj_xWq20Dd860HHAfzMjK2Mb6TPVxMa"
project-slug: "example-workflow"
- name: "nextjs-turbopack"
project-id: "prj_yjkM7UdHliv8bfxZ1sMJQf1pMpdi"
project-slug: "example-nextjs-workflow-turbopack"
- name: "nextjs-webpack"
project-id: "prj_avRPBF3eWjh6iDNQgmhH4VOg27h0"
project-slug: "example-nextjs-workflow-webpack"
- name: "nitro"
project-id: "prj_e7DZirYdLrQKXNrlxg7KmA6ABx8r"
project-slug: "workbench-nitro-workflow"
- name: "vite"
project-id: "prj_uLIcNZNDmETulAvj5h0IcDHi5432"
project-slug: "workbench-vite-workflow"
- name: "nuxt"
project-id: "prj_oTgiz3SGX2fpZuM6E0P38Ts8de6d"
project-slug: "workbench-nuxt-workflow"
- name: "sveltekit"
project-id: "prj_MqnBLm71ceXGSnm3Fs8i8gBnI23G"
project-slug: "workbench-sveltekit-workflow"
- name: "hono"
project-id: "prj_p0GIEsfl53L7IwVbosPvi9rPSOYW"
project-slug: "workbench-hono-workflow"
- name: "express"
project-id: "prj_cCZjpBy92VRbKHHbarDMhOHtkuIr"
project-slug: "workbench-express-workflow"
- name: "fastify"
project-id: "prj_5Yap0VDQ633v998iqQ3L3aQ25Cck"
project-slug: "workbench-fastify-workflow"
- name: "nest"
project-id: "prj_AV3YrzooDhMlvfmZybqeugHRk48Y"
project-slug: "workbench-nestjs-workflow"
- name: "astro"
project-id: "prj_YDAXj3K8LM0hgejuIMhioz2yLgTI"
project-slug: "workbench-astro-workflow"
- name: "tanstack-start"
project-id: "prj_643jeVugTMq5ivsOFQHcbLG1qcnu"
project-slug: "workbench-tanstack-start-workflow"
- name: "python"
project-id: "prj_MtdPMACTDJnukz8LzWM65Fob0IVU"
project-slug: "workbench-python-workflow"
exclude:
# `vm` selects the JS engine the workflow body runs on. There is no
# Python meaning to that axis — the Python app has one runtime — so
# only the `node` half of the pair is a real configuration.
#
# GitHub Actions matches an `exclude` entry by deep equality on every
# key it names, so the whole `app` object has to be repeated verbatim.
# Keep it in sync with the entry above.
- vm: quickjs
app:
name: "python"
project-id: "prj_MtdPMACTDJnukz8LzWM65Fob0IVU"
project-slug: "workbench-python-workflow"
env:
TURBO_TOKEN: ${{ secrets.TURBO_TOKEN }}
TURBO_TEAM: ${{ vars.TURBO_TEAM }}
WORKFLOW_PUBLIC_MANIFEST: '1'
WORKFLOW_RETURN_VALUE_POLL_INTERVAL_MS: '5000'
steps:
- name: Checkout Repo
uses: actions/checkout@v4
with:
ref: ${{ github.event.pull_request.head.sha || github.sha }}
- name: Setup environment
uses: ./.github/actions/setup-workflow-dev
with:
build-packages: 'false'
- name: Build CLI
run: pnpm turbo run build --filter='@workflow/cli'
# `changeset-release/*` PRs (the changesets "Version Packages" PR) are
# deliberately not deployed: the branch is force-pushed and can point at
# main's HEAD SHA, and Vercel keeps one commit status per project per
# SHA, so a same-SHA preview deployment overwrites the production one
# that the step below reads the deployment ID from. Every project's
# vercel.json therefore sets `git.deploymentEnabled` false for that
# branch, which leaves these PRs with no deployment of their own to wait
# for. Their content is main plus a version-bump commit, so they run
# against the production deployment main already produced instead.
- name: Waiting for the Vercel deployment
id: waitForDeployment
if: ${{ !startsWith(github.head_ref, 'changeset-release/') }}
uses: vercel/wait-for-deployment-action@0e2b0c5c5cce31f1648108aeec56467187aca037
with:
project-slug: ${{ matrix.app.project-slug }}
timeout: 1000
check-interval: 15
environment: ${{ github.ref == 'refs/heads/main' && 'production' || 'preview' }}
# The base SHA, not the PR head: that is the main commit whose
# production deployment carries the code under test. Waiting on it also
# means a production build still in flight is waited out instead of
# testing whatever happens to be aliased to production. The commit-status
# ID this action reads is unambiguous for main SHAs once this repo stops
# deploying `changeset-release/main` (see above) — that branch was the
# only one that ever deployed a commit main also deployed.
- name: Waiting for the Vercel deployment (changeset-release PR)
id: prodDeployment
if: ${{ startsWith(github.head_ref, 'changeset-release/') }}
uses: vercel/wait-for-deployment-action@0e2b0c5c5cce31f1648108aeec56467187aca037
with:
project-slug: ${{ matrix.app.project-slug }}
timeout: 1000
check-interval: 15
environment: production
sha: ${{ github.event.pull_request.base.sha }}
- name: Record E2E start time
id: e2eStart
run: echo "ms=$(($(date +%s) * 1000))" >> "$GITHUB_OUTPUT"
- name: Run E2E Tests
run: pnpm run test:e2e --reporter=verbose --reporter=json --reporter=./packages/core/e2e/github-reporter.ts "--outputFile=e2e-vercel-prod-$APP_NAME-$WORKFLOW_VM.json"
env:
NODE_OPTIONS: "--enable-source-maps"
DEPLOYMENT_URL: ${{ steps.waitForDeployment.outputs.deployment-url || steps.prodDeployment.outputs.deployment-url }}
VERCEL_DEPLOYMENT_ID: ${{ steps.waitForDeployment.outputs.deployment-id || steps.prodDeployment.outputs.deployment-id }}
APP_NAME: ${{ matrix.app.name }}
WORKFLOW_VM: ${{ matrix.vm }}
# changeset-release PRs test main's production deployment, so they
# must be treated as a production run everywhere downstream.
WORKFLOW_VERCEL_ENV: ${{ (github.ref == 'refs/heads/main' || startsWith(github.head_ref, 'changeset-release/')) && 'production' || 'preview' }}
WORKFLOW_VERCEL_AUTH_TOKEN: ${{ secrets.VERCEL_LABS_TOKEN }}
WORKFLOW_VERCEL_TEAM: "team_nO2mCG4W8IxPIeKoSsqwAxxB"
WORKFLOW_VERCEL_PROJECT: ${{ matrix.app.project-id }}
WORKFLOW_VERCEL_PROJECT_SLUG: ${{ matrix.app.project-slug }}
# The trusted-sources OIDC token (x-vercel-trusted-oidc-idp-token)
# is minted on demand inside `scripts/trusted-sources-headers.mjs`
# via the runner's ACTIONS_ID_TOKEN_REQUEST_URL/_TOKEN env vars
# (exposed when `permissions: id-token: write` is set on the job),
# and re-minted before its 5-minute expiry. This lets long e2e
# suites keep a fresh token throughout the run instead of using
# a single pre-minted token that would expire mid-suite.
# See: https://vercel.com/docs/deployment-protection/methods-to-bypass-deployment-protection/trusted-sources
# Point PRs at the protected workflow-server preview; unset on main
# so production runs hit the public vercel-workflow.com URL. Also
# unset for changeset-release PRs: they test a production deployment
# that is wired to the production workflow-server, and the harness
# has to read run state from the same server the app writes it to.
VERCEL_WORKFLOW_SERVER_URL: ${{ github.ref != 'refs/heads/main' && !startsWith(github.head_ref, 'changeset-release/') && secrets.VERCEL_WORKFLOW_SERVER_URL || '' }}
- name: Capture runtime logs on failure
if: failure()
env:
APP_NAME: ${{ matrix.app.name }}
WORKFLOW_VERCEL_AUTH_TOKEN: ${{ secrets.VERCEL_LABS_TOKEN }}
WORKFLOW_VERCEL_TEAM: "team_nO2mCG4W8IxPIeKoSsqwAxxB"
WORKFLOW_VERCEL_PROJECT: ${{ matrix.app.project-id }}
WORKFLOW_VERCEL_ENV: ${{ (github.ref == 'refs/heads/main' || startsWith(github.head_ref, 'changeset-release/')) && 'production' || 'preview' }}
VERCEL_DEPLOYMENT_ID: ${{ steps.waitForDeployment.outputs.deployment-id || steps.prodDeployment.outputs.deployment-id }}
E2E_START_MS: ${{ steps.e2eStart.outputs.ms }}
run: node .github/scripts/fetch-e2e-runtime-logs.mjs
- name: Generate E2E summary
if: always()
env:
APP_NAME: ${{ matrix.app.name }}
WORKFLOW_VM: ${{ matrix.vm }}
run: node .github/scripts/aggregate-e2e-results.js . --job-name "E2E Vercel Prod ($APP_NAME - $WORKFLOW_VM)" >> $GITHUB_STEP_SUMMARY || true
- name: Upload E2E results
if: always()
uses: actions/upload-artifact@v4
with:
name: e2e-results-vercel-prod-${{ matrix.app.name }}-${{ matrix.vm }}
path: |
e2e-vercel-prod-${{ matrix.app.name }}-${{ matrix.vm }}.json
e2e-metadata-${{ matrix.app.name }}-vercel.json
e2e-failures-${{ matrix.app.name }}-vercel.json
e2e-flaky-${{ matrix.app.name }}-vercel.json
e2e-infra-${{ matrix.app.name }}-vercel.json
e2e-diagnostics-${{ matrix.app.name }}-vercel.json
e2e-runtime-logs-${{ matrix.app.name }}-vercel.json
retention-days: 7
if-no-files-found: ignore
e2e-vercel-multi-region:
# Vercel-specific multi-region routing suite (start()'s `region`
# option). Deliberately separate from e2e-vercel-prod: that job runs
# the shared suite as a matrix across all worlds/frameworks, while
# this functionality only exists for @workflow/world-vercel and only
# the nextjs-turbopack workbench is deployed multi-region
# (workbench/nextjs-turbopack/vercel.json pins iad1+sfo1+fra1).
name: E2E Vercel Multi-Region Tests (nextjs-turbopack)
runs-on: ubuntu-latest
# 19 regions: the implicit-route cases pay one cold start each,
# sequentially.
timeout-minutes: 35
needs: ci-scope
if: ${{ needs.ci-scope.outputs.fast-path != 'true' }}
permissions:
id-token: write
contents: read
deployments: read
statuses: read
env:
TURBO_TOKEN: ${{ secrets.TURBO_TOKEN }}
TURBO_TEAM: ${{ vars.TURBO_TEAM }}
WORKFLOW_PUBLIC_MANIFEST: '1'
WORKFLOW_RETURN_VALUE_POLL_INTERVAL_MS: '5000'
steps:
- name: Checkout Repo
uses: actions/checkout@v4
with:
ref: ${{ github.event.pull_request.head.sha || github.sha }}
- name: Setup environment
uses: ./.github/actions/setup-workflow-dev
with:
build-packages: 'false'
- name: Build CLI
run: pnpm turbo run build --filter='@workflow/cli'
# See the note on e2e-vercel-prod: `changeset-release/*` PRs have no
# deployment of their own and test main's production deployment.
- name: Waiting for the Vercel deployment
id: waitForDeployment
if: ${{ !startsWith(github.head_ref, 'changeset-release/') }}
uses: vercel/wait-for-deployment-action@0e2b0c5c5cce31f1648108aeec56467187aca037
with:
project-slug: "example-nextjs-workflow-turbopack"
timeout: 1000
check-interval: 15
environment: ${{ github.ref == 'refs/heads/main' && 'production' || 'preview' }}
- name: Waiting for the Vercel deployment (changeset-release PR)
id: prodDeployment
if: ${{ startsWith(github.head_ref, 'changeset-release/') }}
uses: vercel/wait-for-deployment-action@0e2b0c5c5cce31f1648108aeec56467187aca037
with:
project-slug: "example-nextjs-workflow-turbopack"
timeout: 1000
check-interval: 15
environment: production
sha: ${{ github.event.pull_request.base.sha }}
- name: Run Multi-Region E2E Tests
run: pnpm vitest run packages/core/e2e/e2e-region.test.ts --reporter=verbose --reporter=json --reporter=./packages/core/e2e/github-reporter.ts "--outputFile=e2e-vercel-multi-region-$APP_NAME.json"
env:
NODE_OPTIONS: "--enable-source-maps"
DEPLOYMENT_URL: ${{ steps.waitForDeployment.outputs.deployment-url || steps.prodDeployment.outputs.deployment-url }}
VERCEL_DEPLOYMENT_ID: ${{ steps.waitForDeployment.outputs.deployment-id || steps.prodDeployment.outputs.deployment-id }}
APP_NAME: "nextjs-turbopack"
# changeset-release PRs test main's production deployment, so they
# must be treated as a production run everywhere downstream.
WORKFLOW_VERCEL_ENV: ${{ (github.ref == 'refs/heads/main' || startsWith(github.head_ref, 'changeset-release/')) && 'production' || 'preview' }}
WORKFLOW_VERCEL_AUTH_TOKEN: ${{ secrets.VERCEL_LABS_TOKEN }}
WORKFLOW_VERCEL_TEAM: "team_nO2mCG4W8IxPIeKoSsqwAxxB"
WORKFLOW_VERCEL_PROJECT: "prj_yjkM7UdHliv8bfxZ1sMJQf1pMpdi"
WORKFLOW_VERCEL_PROJECT_SLUG: "example-nextjs-workflow-turbopack"
# See the note on e2e-vercel-prod: PRs point at the protected
# workflow-server preview; unset on main for production.
VERCEL_WORKFLOW_SERVER_URL: ${{ github.ref != 'refs/heads/main' && !startsWith(github.head_ref, 'changeset-release/') && secrets.VERCEL_WORKFLOW_SERVER_URL || '' }}
- name: Capture runtime logs on failure
if: failure()
env:
APP_NAME: "nextjs-turbopack"
WORKFLOW_VERCEL_AUTH_TOKEN: ${{ secrets.VERCEL_LABS_TOKEN }}
WORKFLOW_VERCEL_TEAM: "team_nO2mCG4W8IxPIeKoSsqwAxxB"
WORKFLOW_VERCEL_PROJECT: "prj_yjkM7UdHliv8bfxZ1sMJQf1pMpdi"
WORKFLOW_VERCEL_ENV: ${{ (github.ref == 'refs/heads/main' || startsWith(github.head_ref, 'changeset-release/')) && 'production' || 'preview' }}
VERCEL_DEPLOYMENT_ID: ${{ steps.waitForDeployment.outputs.deployment-id || steps.prodDeployment.outputs.deployment-id }}
run: node .github/scripts/fetch-e2e-runtime-logs.mjs
- name: Upload E2E results
if: always()
uses: actions/upload-artifact@v4
with:
name: e2e-results-vercel-multi-region-nextjs-turbopack
path: |
e2e-vercel-multi-region-nextjs-turbopack.json
e2e-metadata-nextjs-turbopack-vercel.json
e2e-failures-nextjs-turbopack-vercel.json
e2e-flaky-nextjs-turbopack-vercel.json
e2e-infra-nextjs-turbopack-vercel.json
e2e-diagnostics-nextjs-turbopack-vercel.json
e2e-runtime-logs-nextjs-turbopack-vercel.json
retention-days: 7
if-no-files-found: ignore
e2e-vercel-ws-transport:
# Dedicated coverage for world-vercel's WebSocket events transport
# (WORKFLOW_EVENTS_TRANSPORT=ws — see events-v4.ts's
# isWsEventsTransportEnabled). This lane predates the default flip,
# when it was the only WS coverage in the file; it is kept because it
# still pins the transport explicitly, which every other Vercel lane
# now only gets by inheriting the default. See
# e2e-vercel-http-transport below for the other side.
#
# Mirrors e2e-vercel-prod's shape (same apps subset, same full e2e
# suite) but deploys itself via a plain `vercel deploy` (no
# `--prebuilt`) instead of waiting for the GitHub App's passive
# deployment — that's the only way to get a *second*, independent
# deployment of a project we don't own the Environment Variables UI
# for, with its own env var, without creating a whole new project.
#
# Deliberately NOT `--prebuilt`: an earlier version of this job built
# locally on the runner via `vercel build --prebuilt` and hit a string
# of environment-specific bugs that don't reproduce on a real
# deployment (missing `serverExternalPackages` tracing for
# `@workflow/world-vercel`, nitro preset auto-detection picking
# `node-server` instead of `vercel`, needing the whole monorepo
# pre-built on this runner). Plain `vercel deploy` uploads source and
# builds remotely on Vercel's own infra — the exact same pipeline the
# GitHub App uses for e2e-vercel-prod's deployments — so none of that
# applies here.
# Runs on every PR. It was opt-in behind `ws-transport-test` when four real
# `vercel deploy`s were too much to charge an unrelated PR for a transport
# that was off by default. That reasoning expired with the default: every
# deployment now uses WebSockets, so this stopped being a cost charged for
# someone else's feature and became the only check that the feature works.
name: E2E Vercel WS Transport Test (${{ matrix.app.name }})
runs-on: ubuntu-latest
timeout-minutes: 30
needs: ci-scope
# Mirrors e2e-vercel-prod and the HTTP lane: this is the only lane that
# *asserts* the socket carried the events, since e2e-vercel-prod inherits
# the default but checks nothing.
if: ${{ needs.ci-scope.outputs.fast-path != 'true' }}
permissions:
id-token: write
contents: read
strategy:
fail-fast: false
matrix:
app:
- name: "example"
project-id: "prj_xWq20Dd860HHAfzMjK2Mb6TPVxMa"
project-slug: "example-workflow"
- name: "nextjs-turbopack"
project-id: "prj_yjkM7UdHliv8bfxZ1sMJQf1pMpdi"
project-slug: "example-nextjs-workflow-turbopack"
- name: "express"
project-id: "prj_cCZjpBy92VRbKHHbarDMhOHtkuIr"
project-slug: "workbench-express-workflow"
# Vite resolves `ws`'s absent optional accelerators to a stub
# instead of failing the require, so a build check can't see the
# breakage and only a masked frame at or above 48 bytes trips it
# (see WORKFLOW_OPTIONAL_WS_NATIVE_MODULES). Every CBOR event
# frame clears that, so this lane is the coverage for it.
- name: "vite"
project-id: "prj_uLIcNZNDmETulAvj5h0IcDHi5432"
project-slug: "workbench-vite-workflow"
env:
TURBO_TOKEN: ${{ secrets.TURBO_TOKEN }}
TURBO_TEAM: ${{ vars.TURBO_TEAM }}
WORKFLOW_PUBLIC_MANIFEST: '1'
WS_TEAM_ID: "team_nO2mCG4W8IxPIeKoSsqwAxxB"
steps:
- name: Checkout Repo
uses: actions/checkout@v4
with:
ref: ${{ github.event.pull_request.head.sha || github.sha }}
- name: Setup environment
uses: ./.github/actions/setup-workflow-dev
with:
build-packages: 'false'
- name: Build CLI
run: pnpm turbo run build --filter='@workflow/cli'
# Pinned and installed once rather than `npx vercel@<version>` per
# command — repeated `npx` resolution showed inconsistent behavior
# (different Node version in the CLI's own banner, a doubled project
# path) across otherwise-identical earlier CI attempts.
- name: Install vercel CLI
run: npm install -g vercel@56.2.0
- name: Deploy a dedicated WS-transport preview
id: wsDeploy
env:
WS_VERCEL_TOKEN: ${{ secrets.VERCEL_LABS_TOKEN }}
run: |
set -euo pipefail
# `vercel deploy` (source-upload mode, no --prebuilt) re-applies
# the linked project's own Root Directory setting internally —
# unlike `vercel build`, it expects to run from the repo root and
# narrow down itself, the same way the GitHub App's own deploy
# does. Passing `--cwd=<app dir>` here double-joins that Root
# Directory underneath itself ("workbench/x/workbench/x does not
# exist"). Link and deploy from the repo root — no --cwd at all —
# and let the project's own Root Directory setting do the
# narrowing, matching how every passively-triggered deployment in
# this file already works.
vercel pull --yes --environment=preview \
--project="${{ matrix.app.project-id }}" \
--scope="$WS_TEAM_ID" --token="$WS_VERCEL_TOKEN"
# `vercel deploy` writes the new deployment's URL to stdout and
# progress/logs to stderr, specifically so it composes as
# `url=$(vercel deploy ...)` in scripts — do not redirect stderr
# into this capture. `--target=preview` is explicit here even
# though it's the default without `--prod` — these are shared,
# production-adjacent projects, and this shouldn't ever silently
# start deploying to their production aliases if a future edit
# changes flags around this call.
# Strict mode turns a silent HTTP demotion into a failed run, so this
# lane actually asserts the socket carried traffic instead of only
# asserting the feature was harmless. Scoped to `step_completed` —
# see STRICT_WS_EVENT_TYPES in events-v4.ts for why the other event
# types fall back legitimately. Never set on e2e-vercel-prod.
URL=$(vercel deploy --yes --target=preview \
-e WORKFLOW_EVENTS_TRANSPORT=ws \
-e WORKFLOW_INTERNAL_EVENTS_TRANSPORT_STRICT=1 \
--scope="$WS_TEAM_ID" --token="$WS_VERCEL_TOKEN")
echo "url=$URL" >> "$GITHUB_OUTPUT"
# Best-effort: only used by the runtime-logs capture on failure,
# which already skips gracefully if this doesn't parse.
DEPLOYMENT_ID=$(vercel inspect "$URL" --format=json \
--scope="$WS_TEAM_ID" --token="$WS_VERCEL_TOKEN" 2>/dev/null \
| node -e "let d='';process.stdin.on('data',c=>d+=c).on('end',()=>{try{console.log(JSON.parse(d).id||'')}catch{}})" || true)
echo "deploymentId=$DEPLOYMENT_ID" >> "$GITHUB_OUTPUT"
- name: Record E2E start time
id: e2eStart
run: echo "ms=$(($(date +%s) * 1000))" >> "$GITHUB_OUTPUT"
- name: Run E2E Tests
run: pnpm run test:e2e --reporter=verbose --reporter=json --reporter=./packages/core/e2e/github-reporter.ts "--outputFile=e2e-vercel-ws-transport-$APP_NAME.json"
env:
NODE_OPTIONS: "--enable-source-maps"
DEPLOYMENT_URL: ${{ steps.wsDeploy.outputs.url }}
VERCEL_DEPLOYMENT_ID: ${{ steps.wsDeploy.outputs.deploymentId }}
APP_NAME: ${{ matrix.app.name }}
WORKFLOW_VERCEL_ENV: "preview"
WORKFLOW_VERCEL_AUTH_TOKEN: ${{ secrets.VERCEL_LABS_TOKEN }}
WORKFLOW_VERCEL_TEAM: ${{ env.WS_TEAM_ID }}
WORKFLOW_VERCEL_PROJECT: ${{ matrix.app.project-id }}
WORKFLOW_VERCEL_PROJECT_SLUG: ${{ matrix.app.project-slug }}
VERCEL_WORKFLOW_SERVER_URL: ${{ github.ref != 'refs/heads/main' && secrets.VERCEL_WORKFLOW_SERVER_URL || '' }}
- name: Capture runtime logs on failure
if: failure()
env:
APP_NAME: ${{ matrix.app.name }}
WORKFLOW_VERCEL_AUTH_TOKEN: ${{ secrets.VERCEL_LABS_TOKEN }}
WORKFLOW_VERCEL_TEAM: ${{ env.WS_TEAM_ID }}
WORKFLOW_VERCEL_PROJECT: ${{ matrix.app.project-id }}
WORKFLOW_VERCEL_ENV: "preview"
VERCEL_DEPLOYMENT_ID: ${{ steps.wsDeploy.outputs.deploymentId }}
E2E_START_MS: ${{ steps.e2eStart.outputs.ms }}
run: node .github/scripts/fetch-e2e-runtime-logs.mjs
- name: Generate E2E summary
if: always()
env:
APP_NAME: ${{ matrix.app.name }}
run: node .github/scripts/aggregate-e2e-results.js . --job-name "E2E Vercel WS Transport ($APP_NAME)" >> $GITHUB_STEP_SUMMARY || true
- name: Upload E2E results
if: always()
uses: actions/upload-artifact@v4
with:
name: e2e-results-vercel-ws-transport-${{ matrix.app.name }}
path: |
e2e-vercel-ws-transport-${{ matrix.app.name }}.json
e2e-metadata-${{ matrix.app.name }}-vercel.json
e2e-failures-${{ matrix.app.name }}-vercel.json
e2e-flaky-${{ matrix.app.name }}-vercel.json
e2e-infra-${{ matrix.app.name }}-vercel.json
e2e-diagnostics-${{ matrix.app.name }}-vercel.json
e2e-runtime-logs-${{ matrix.app.name }}-vercel.json
retention-days: 7
if-no-files-found: ignore
e2e-vercel-http-transport:
# The counterpart to e2e-vercel-ws-transport, and the reason it exists
# is the default flip: with WORKFLOW_EVENTS_TRANSPORT defaulting to
# `ws`, e2e-vercel-prod — which sets no transport env var at all — is
# now a WebSocket lane. Nothing in this file would exercise the HTTP
# events transport against a real Vercel deployment any more. This is
# not additive coverage; it replaces coverage that the flip silently
# took away.
#
# HTTP is now the fallback path, and the fallback is *silent*:
# resolveWsTransport returning null costs a write nothing and logs
# nothing. A path that is both unexercised and quiet is the exact
# shape of the durabench bug this stack came out of, so this lane is
# unconditional and required rather than label-gated like the WS one.
#
# Six apps, chosen by server shape rather than by count. Before the flip
# HTTP was the default, so all 28 e2e-vercel-prod lane-runs covered it;
# after the flip they cover WebSockets instead and this lane is the whole
# of the HTTP coverage. Each row is a real `vercel deploy` charged to
# every PR, so the full 14 is too expensive, but two was too thin for a
# transport that is still supported.
#
# The shapes, one fixture each:
# example - baseline, no framework server
# nextjs-turbopack - Next; also the only fixture emitting OTEL spans,
# so the only one that can show which transport ran
# vite - Vite SSR
# express - Node req/res server
# nitro - Nitro/h3, which also covers nuxt
# hono - fetch-API Request/Response server, a genuinely
# different mount shape from express
#
# Omitted because they duplicate a shape already here: nextjs-webpack
# (same runtime as turbopack, different bundler), nuxt (nitro), fastify
# and nest (both express-shaped), astro and tanstack-start (adapter
# variants). `python` is omitted because it has no conformance gate and
# needs routes this suite does not serve - see the e2e-python job.
#
# The first four deliberately match the WS lane's matrix, so the same
# fixture is exercised on both transports and a failure on one can be
# compared against the other.
#
# Deploys itself with `vercel deploy` rather than `--prebuilt` for the
# same reasons documented on e2e-vercel-ws-transport above.
name: E2E Vercel HTTP Transport Test (${{ matrix.app.name }})
runs-on: ubuntu-latest
timeout-minutes: 30
needs: ci-scope
# Mirrors e2e-vercel-prod's condition exactly, so this lane runs in
# every case that one does, including under `workflow-server-test`.
if: ${{ needs.ci-scope.outputs.fast-path != 'true' }}
permissions:
id-token: write
contents: read
strategy:
fail-fast: false
matrix:
app:
- name: "example"
project-id: "prj_xWq20Dd860HHAfzMjK2Mb6TPVxMa"
project-slug: "example-workflow"
- name: "nextjs-turbopack"
project-id: "prj_yjkM7UdHliv8bfxZ1sMJQf1pMpdi"
project-slug: "example-nextjs-workflow-turbopack"
- name: "vite"
project-id: "prj_uLIcNZNDmETulAvj5h0IcDHi5432"
project-slug: "workbench-vite-workflow"
- name: "express"
project-id: "prj_cCZjpBy92VRbKHHbarDMhOHtkuIr"
project-slug: "workbench-express-workflow"
- name: "nitro"
project-id: "prj_e7DZirYdLrQKXNrlxg7KmA6ABx8r"
project-slug: "workbench-nitro-workflow"
- name: "hono"
project-id: "prj_p0GIEsfl53L7IwVbosPvi9rPSOYW"
project-slug: "workbench-hono-workflow"
env:
TURBO_TOKEN: ${{ secrets.TURBO_TOKEN }}
TURBO_TEAM: ${{ vars.TURBO_TEAM }}
WORKFLOW_PUBLIC_MANIFEST: '1'
WS_TEAM_ID: "team_nO2mCG4W8IxPIeKoSsqwAxxB"
steps:
- name: Checkout Repo
uses: actions/checkout@v4
with:
ref: ${{ github.event.pull_request.head.sha || github.sha }}
- name: Setup environment
uses: ./.github/actions/setup-workflow-dev
with:
build-packages: 'false'
- name: Build CLI
run: pnpm turbo run build --filter='@workflow/cli'
- name: Install vercel CLI
run: npm install -g vercel@56.2.0
- name: Deploy a dedicated HTTP-transport preview
id: httpDeploy
env:
WS_VERCEL_TOKEN: ${{ secrets.VERCEL_LABS_TOKEN }}
run: |
set -euo pipefail
vercel pull --yes --environment=preview \
--project="${{ matrix.app.project-id }}" \
--scope="$WS_TEAM_ID" --token="$WS_VERCEL_TOKEN"
# `http` is the one value that opts out; the gate compares against
# exactly this string, so a typo here would silently test the
# default instead of the fallback and this lane would quietly
# become a duplicate of e2e-vercel-prod.
URL=$(vercel deploy --yes --target=preview \
-e WORKFLOW_EVENTS_TRANSPORT=http \
--scope="$WS_TEAM_ID" --token="$WS_VERCEL_TOKEN")
echo "url=$URL" >> "$GITHUB_OUTPUT"
DEPLOYMENT_ID=$(vercel inspect "$URL" --format=json \
--scope="$WS_TEAM_ID" --token="$WS_VERCEL_TOKEN" 2>/dev/null \
| node -e "let d='';process.stdin.on('data',c=>d+=c).on('end',()=>{try{console.log(JSON.parse(d).id||'')}catch{}})" || true)
echo "deploymentId=$DEPLOYMENT_ID" >> "$GITHUB_OUTPUT"
- name: Record E2E start time
id: e2eStart
run: echo "ms=$(($(date +%s) * 1000))" >> "$GITHUB_OUTPUT"
- name: Run E2E Tests
run: pnpm run test:e2e --reporter=verbose --reporter=json --reporter=./packages/core/e2e/github-reporter.ts "--outputFile=e2e-vercel-http-transport-$APP_NAME.json"
env:
NODE_OPTIONS: "--enable-source-maps"
DEPLOYMENT_URL: ${{ steps.httpDeploy.outputs.url }}
VERCEL_DEPLOYMENT_ID: ${{ steps.httpDeploy.outputs.deploymentId }}
APP_NAME: ${{ matrix.app.name }}
WORKFLOW_VERCEL_ENV: "preview"
WORKFLOW_VERCEL_AUTH_TOKEN: ${{ secrets.VERCEL_LABS_TOKEN }}
WORKFLOW_VERCEL_TEAM: ${{ env.WS_TEAM_ID }}
WORKFLOW_VERCEL_PROJECT: ${{ matrix.app.project-id }}
WORKFLOW_VERCEL_PROJECT_SLUG: ${{ matrix.app.project-slug }}
VERCEL_WORKFLOW_SERVER_URL: ${{ github.ref != 'refs/heads/main' && secrets.VERCEL_WORKFLOW_SERVER_URL || '' }}
- name: Capture runtime logs on failure
if: failure()
env:
APP_NAME: ${{ matrix.app.name }}
WORKFLOW_VERCEL_AUTH_TOKEN: ${{ secrets.VERCEL_LABS_TOKEN }}
WORKFLOW_VERCEL_TEAM: ${{ env.WS_TEAM_ID }}
WORKFLOW_VERCEL_PROJECT: ${{ matrix.app.project-id }}
WORKFLOW_VERCEL_ENV: "preview"
VERCEL_DEPLOYMENT_ID: ${{ steps.httpDeploy.outputs.deploymentId }}
E2E_START_MS: ${{ steps.e2eStart.outputs.ms }}
run: node .github/scripts/fetch-e2e-runtime-logs.mjs
- name: Generate E2E summary
if: always()
env:
APP_NAME: ${{ matrix.app.name }}
run: node .github/scripts/aggregate-e2e-results.js . --job-name "E2E Vercel HTTP Transport ($APP_NAME)" >> $GITHUB_STEP_SUMMARY || true
- name: Upload E2E results
if: always()
uses: actions/upload-artifact@v4
with:
name: e2e-results-vercel-http-transport-${{ matrix.app.name }}
path: |
e2e-vercel-http-transport-${{ matrix.app.name }}.json
e2e-metadata-${{ matrix.app.name }}-vercel.json
e2e-failures-${{ matrix.app.name }}-vercel.json
e2e-flaky-${{ matrix.app.name }}-vercel.json
e2e-infra-${{ matrix.app.name }}-vercel.json
e2e-diagnostics-${{ matrix.app.name }}-vercel.json
e2e-runtime-logs-${{ matrix.app.name }}-vercel.json
retention-days: 7
if-no-files-found: ignore
getTestMatrix:
name: Get Test Matrix
runs-on: ubuntu-latest
needs: ci-scope
if: ${{ needs.ci-scope.outputs.fast-path != 'true' && !contains(github.event.pull_request.labels.*.name, 'workflow-server-test') }}
outputs:
matrix: ${{ steps.set-matrix.outputs.matrix }}
steps:
- name: Checkout Repo
uses: actions/checkout@v4
with:
ref: ${{ github.event.pull_request.head.sha || github.sha }}
- name: Setup environment
uses: ./.github/actions/setup-workflow-dev
with:
install-dependencies: 'false'
build-packages: 'false'
cache-pnpm: 'false'
- id: set-matrix
run: |
MATRIX_JSON="$(node ./scripts/create-test-matrix.mjs | jq -c .)"
echo "matrix=$MATRIX_JSON" >> "$GITHUB_OUTPUT"
e2e-local-dev:
name: E2E Local Dev Tests (${{ matrix.app.name }} - ${{ matrix.app.runLabel }})
runs-on: ubuntu-latest
timeout-minutes: 30
if: ${{ needs.ci-scope.outputs.fast-path != 'true' && !contains(github.event.pull_request.labels.*.name, 'workflow-server-test') }}
needs: [ci-scope, getTestMatrix, e2e-package-build]
strategy:
fail-fast: false
matrix: ${{fromJson(needs.getTestMatrix.outputs.matrix)}}
env:
TURBO_TOKEN: ${{ secrets.TURBO_TOKEN }}
TURBO_TEAM: ${{ vars.TURBO_TEAM }}
WORKFLOW_PUBLIC_MANIFEST: '1'
steps:
- name: Checkout Repo
uses: actions/checkout@v4
with:
ref: ${{ github.event.pull_request.head.sha || github.sha }}
- name: Setup environment
uses: ./.github/actions/setup-workflow-dev
with:
install-dependencies: 'false'
build-packages: 'false'
- name: Setup canary
if: ${{ matrix.app.canary }}
env:
APP_NAME: ${{ matrix.app.name }}
run: |
cat packages/next/package.json | jq '.dependencies.next|="16.3.1-canary.17"' > packages/next/package.json.new && mv packages/next/package.json.new packages/next/package.json
cat "workbench/$APP_NAME/package.json" | jq '.dependencies.next|="16.3.1-canary.17"' > "workbench/$APP_NAME/package.json.new" && mv "workbench/$APP_NAME/package.json.new" "workbench/$APP_NAME/package.json"
pnpm install --no-frozen-lockfile
- name: Install Dependencies
run: pnpm install --frozen-lockfile
- name: Download shared package builds
if: ${{ !matrix.app.canary }}
uses: actions/download-artifact@v4
with:
name: e2e-package-build-artifacts
path: packages
- name: Run Initial Build for Next.js canary
if: ${{ matrix.app.canary }}
run: pnpm turbo run build --filter='!./workbench/*'
- name: Prepare workbench path
id: prepare-workbench
uses: ./.github/actions/prepare-workbench-path
with:
app-name: ${{ matrix.app.name }}
- name: Run E2E Tests
run: |
export DEV_SERVER_LOG_PATH="$GITHUB_WORKSPACE/dev-server-${{ matrix.app.name }}-${{ matrix.app.artifactSuffix }}.log"
rm -f "$DEV_SERVER_LOG_PATH"
(cd "$WORKBENCH_APP_PATH" && pnpm dev 2>&1 | tee "$DEV_SERVER_LOG_PATH") &
echo "starting tests in 10 seconds" && sleep 10
pnpm vitest run packages/core/e2e/dev.test.ts; sleep 10
pnpm run test:e2e --reporter=verbose --reporter=json --reporter=./packages/core/e2e/github-reporter.ts --outputFile=e2e-local-dev-${{ matrix.app.name }}-${{ matrix.app.artifactSuffix }}.json
env:
NODE_OPTIONS: "--enable-source-maps"
APP_NAME: ${{ matrix.app.name }}
WORKBENCH_APP_PATH: ${{ steps.prepare-workbench.outputs.workbench_app_path }}
DEPLOYMENT_URL: "http://localhost:${{ matrix.app.name == 'sveltekit' && '5173' || (matrix.app.name == 'astro' && '4321' || '3000') }}"
DEV_TEST_CONFIG: ${{ toJSON(matrix.app) }}
WORKFLOW_DEV_HMR_LOGS: "1"
NEXT_CANARY: ${{ matrix.app.canary && '1' || '' }}
WORKFLOW_VM: ${{ matrix.app.vm || '' }}
- name: Generate E2E summary
if: always()
run: node .github/scripts/aggregate-e2e-results.js . --job-name "E2E Local Dev (${{ matrix.app.name }} - ${{ matrix.app.runLabel }})" >> $GITHUB_STEP_SUMMARY || true
- name: Upload E2E results
if: always()
uses: actions/upload-artifact@v4
with:
name: e2e-results-local-dev-${{ matrix.app.name }}-${{ matrix.app.artifactSuffix }}
path: |
e2e-local-dev-${{ matrix.app.name }}-${{ matrix.app.artifactSuffix }}.json
e2e-flaky-${{ matrix.app.name }}-local.json
e2e-infra-${{ matrix.app.name }}-local.json
retention-days: 7
if-no-files-found: ignore
e2e-local-prod:
name: E2E Local Prod Tests (${{ matrix.app.name }} - ${{ matrix.app.runLabel }})
runs-on: ubuntu-latest
timeout-minutes: 30
if: ${{ needs.ci-scope.outputs.fast-path != 'true' && !contains(github.event.pull_request.labels.*.name, 'workflow-server-test') }}
needs: [ci-scope, getTestMatrix, e2e-package-build]
strategy:
fail-fast: false
matrix: ${{fromJson(needs.getTestMatrix.outputs.matrix)}}
env:
TURBO_TOKEN: ${{ secrets.TURBO_TOKEN }}
TURBO_TEAM: ${{ vars.TURBO_TEAM }}
WORKFLOW_PUBLIC_MANIFEST: '1'
steps:
- name: Checkout Repo
uses: actions/checkout@v4
with:
ref: ${{ github.event.pull_request.head.sha || github.sha }}
- name: Setup environment
uses: ./.github/actions/setup-workflow-dev
with:
install-dependencies: 'false'
build-packages: 'false'
- name: Setup canary
if: ${{ matrix.app.canary }}
env:
APP_NAME: ${{ matrix.app.name }}
run: |
cat packages/next/package.json | jq '.dependencies.next|="16.3.1-canary.17"' > packages/next/package.json.new && mv packages/next/package.json.new packages/next/package.json
cat "workbench/$APP_NAME/package.json" | jq '.dependencies.next|="16.3.1-canary.17"' > "workbench/$APP_NAME/package.json.new" && mv "workbench/$APP_NAME/package.json.new" "workbench/$APP_NAME/package.json"
pnpm install --no-frozen-lockfile
- name: Install Dependencies
run: pnpm install --frozen-lockfile
- name: Download shared package builds
if: ${{ !matrix.app.canary }}
uses: actions/download-artifact@v4
with:
name: e2e-package-build-artifacts
path: packages
- name: Run Initial Build for Next.js canary
if: ${{ matrix.app.canary }}
run: pnpm turbo run build --filter='!./workbench/*'
- name: Prepare workbench path
id: prepare-workbench
uses: ./.github/actions/prepare-workbench-path
with:
app-name: ${{ matrix.app.name }}
- name: Run Build Tests
run: pnpm vitest run packages/core/e2e/local-build.test.ts
env:
APP_NAME: ${{ matrix.app.name }}
WORKBENCH_APP_PATH: ${{ steps.prepare-workbench.outputs.workbench_app_path }}
# Asserts the shape of the manifest the step above just built. It needs
# its own invocation rather than another file in that one: local-build
# is what runs `pnpm build`, and vitest would run both files
# concurrently, against a manifest that does not exist yet.
- name: Run Manifest Tests
run: pnpm vitest run packages/core/e2e/manifest.test.ts
env:
APP_NAME: ${{ matrix.app.name }}
WORKBENCH_APP_PATH: ${{ steps.prepare-workbench.outputs.workbench_app_path }}
- name: Run E2E Tests
run: |
export PROD_SERVER_LOG_PATH="$GITHUB_WORKSPACE/prod-server-${{ matrix.app.name }}-${{ matrix.app.artifactSuffix }}.log"
(cd "$WORKBENCH_APP_PATH" && pnpm start 2>&1 | tee "$PROD_SERVER_LOG_PATH") &
echo "starting tests in 10 seconds" && sleep 10
pnpm run test:e2e --reporter=verbose --reporter=json --reporter=./packages/core/e2e/github-reporter.ts --outputFile=e2e-local-prod-${{ matrix.app.name }}-${{ matrix.app.artifactSuffix }}.json
env:
NODE_OPTIONS: "--enable-source-maps"
APP_NAME: ${{ matrix.app.name }}
WORKBENCH_APP_PATH: ${{ steps.prepare-workbench.outputs.workbench_app_path }}
DEPLOYMENT_URL: "http://localhost:${{ matrix.app.name == 'sveltekit' && '4173' || (matrix.app.name == 'astro' && '4321' || '3000') }}"
NEXT_CANARY: ${{ matrix.app.canary && '1' || '' }}
WORKFLOW_VM: ${{ matrix.app.vm || '' }}
- name: Generate E2E summary
if: always()
run: node .github/scripts/aggregate-e2e-results.js . --job-name "E2E Local Prod (${{ matrix.app.name }} - ${{ matrix.app.runLabel }})" >> $GITHUB_STEP_SUMMARY || true
- name: Upload E2E results
if: always()
uses: actions/upload-artifact@v4
with:
name: e2e-results-local-prod-${{ matrix.app.name }}-${{ matrix.app.artifactSuffix }}
path: |
e2e-local-prod-${{ matrix.app.name }}-${{ matrix.app.artifactSuffix }}.json
e2e-flaky-${{ matrix.app.name }}-local.json
e2e-infra-${{ matrix.app.name }}-local.json
retention-days: 7
if-no-files-found: ignore
e2e-local-postgres:
name: E2E Local Postgres Tests (${{ matrix.app.name }} - ${{ matrix.app.runLabel }})
runs-on: ubuntu-latest
timeout-minutes: 30
if: ${{ needs.ci-scope.outputs.fast-path != 'true' && !contains(github.event.pull_request.labels.*.name, 'workflow-server-test') }}
needs: [ci-scope, getTestMatrix, e2e-package-build]
strategy:
fail-fast: false
matrix: ${{fromJson(needs.getTestMatrix.outputs.matrix)}}
services:
postgres:
image: postgres:18-alpine
env:
POSTGRES_USER: world
POSTGRES_PASSWORD: world
POSTGRES_DB: world
ports:
- 5432:5432
options: >-
--health-cmd pg_isready
--health-interval 10s
--health-timeout 5s
--health-retries 5
env:
TURBO_TOKEN: ${{ secrets.TURBO_TOKEN }}
TURBO_TEAM: ${{ vars.TURBO_TEAM }}
WORKFLOW_PUBLIC_MANIFEST: '1'
WORKFLOW_TARGET_WORLD: "@workflow/world-postgres"
WORKFLOW_POSTGRES_URL: "postgres://world:world@localhost:5432/world"
steps:
- name: Checkout Repo
uses: actions/checkout@v4
with:
ref: ${{ github.event.pull_request.head.sha || github.sha }}
- name: Setup environment
uses: ./.github/actions/setup-workflow-dev
with:
install-dependencies: 'false'
build-packages: 'false'
- name: Setup canary
if: ${{ matrix.app.canary }}
env:
APP_NAME: ${{ matrix.app.name }}
run: |
cat packages/next/package.json | jq '.dependencies.next|="16.3.1-canary.17"' > packages/next/package.json.new && mv packages/next/package.json.new packages/next/package.json
cat "workbench/$APP_NAME/package.json" | jq '.dependencies.next|="16.3.1-canary.17"' > "workbench/$APP_NAME/package.json.new" && mv "workbench/$APP_NAME/package.json.new" "workbench/$APP_NAME/package.json"
pnpm install --no-frozen-lockfile
- name: Install Dependencies
run: pnpm install --frozen-lockfile
- name: Download shared package builds
if: ${{ !matrix.app.canary }}
uses: actions/download-artifact@v4
with:
name: e2e-package-build-artifacts
path: packages
- name: Run Initial Build for Next.js canary
if: ${{ matrix.app.canary }}
run: pnpm turbo run build --filter='!./workbench/*'
- name: Setup PostgreSQL Database
run: ./packages/world-postgres/bin/setup.js
- name: Prepare workbench path
id: prepare-workbench
uses: ./.github/actions/prepare-workbench-path
with:
app-name: ${{ matrix.app.name }}
- name: Run Build Tests
run: pnpm vitest run packages/core/e2e/local-build.test.ts
env:
APP_NAME: ${{ matrix.app.name }}
WORKBENCH_APP_PATH: ${{ steps.prepare-workbench.outputs.workbench_app_path }}
- name: Run E2E Tests
run: |
export PROD_SERVER_LOG_PATH="$GITHUB_WORKSPACE/prod-server-postgres-${{ matrix.app.name }}-${{ matrix.app.artifactSuffix }}.log"
(cd "$WORKBENCH_APP_PATH" && pnpm start 2>&1 | tee "$PROD_SERVER_LOG_PATH") &
echo "starting tests in 10 seconds" && sleep 10
pnpm run test:e2e --reporter=verbose --reporter=json --reporter=./packages/core/e2e/github-reporter.ts --outputFile=e2e-local-postgres-${{ matrix.app.name }}-${{ matrix.app.artifactSuffix }}.json
env:
NODE_OPTIONS: "--enable-source-maps"
APP_NAME: ${{ matrix.app.name }}
WORKBENCH_APP_PATH: ${{ steps.prepare-workbench.outputs.workbench_app_path }}
DEPLOYMENT_URL: "http://localhost:${{ matrix.app.name == 'sveltekit' && '4173' || (matrix.app.name == 'astro' && '4321' || '3000') }}"
NEXT_CANARY: ${{ matrix.app.canary && '1' || '' }}
WORKFLOW_VM: ${{ matrix.app.vm || '' }}
- name: Generate E2E summary
if: always()
run: node .github/scripts/aggregate-e2e-results.js . --job-name "E2E Local Postgres (${{ matrix.app.name }} - ${{ matrix.app.runLabel }})" >> $GITHUB_STEP_SUMMARY || true
- name: Upload E2E results
if: always()
uses: actions/upload-artifact@v4
with:
name: e2e-results-local-postgres-${{ matrix.app.name }}-${{ matrix.app.artifactSuffix }}
path: |
e2e-local-postgres-${{ matrix.app.name }}-${{ matrix.app.artifactSuffix }}.json
e2e-flaky-${{ matrix.app.name }}-local.json
e2e-infra-${{ matrix.app.name }}-local.json
retention-days: 7
if-no-files-found: ignore
# Cross-language conformance: the same TypeScript e2e suite, driven against the
# Python workbench app instead of a JavaScript one. Deliberately its own job
# rather than a `getTestMatrix` row — every step in the two local lanes is JS
# toolchain (`pnpm dev` / `pnpm start`, the canary `next` patch, `dev.test.ts`
# and `local-build.test.ts`, and the per-framework `DEV_TEST_CONFIG` of
# compiler-generated paths), and none of it has a Python counterpart. The
# `e2e-vercel-prod` lane, whose rows only need a URL, does carry `python` as an
# ordinary matrix row.
#
# What this lane protects is the gate, not the fixtures it happens to run: a
# test that stops skipping correctly, or a fixture declared in
# `e2e-conformance.json` that the app stops registering, fails here and nowhere
# else.
e2e-python:
name: E2E Python Conformance
runs-on: ubuntu-latest
timeout-minutes: 20
if: ${{ needs.ci-scope.outputs.fast-path != 'true' && !contains(github.event.pull_request.labels.*.name, 'workflow-server-test') }}
needs: [ci-scope, e2e-package-build]
env:
TURBO_TOKEN: ${{ secrets.TURBO_TOKEN }}
TURBO_TEAM: ${{ vars.TURBO_TEAM }}
WORKFLOW_PUBLIC_MANIFEST: '1'
steps:
- name: Checkout Repo
uses: actions/checkout@v4
with:
ref: ${{ github.event.pull_request.head.sha || github.sha }}
- name: Setup environment
uses: ./.github/actions/setup-workflow-dev
with:
build-packages: 'false'
# The Python app needs nothing from `packages/`, but the driver does: the
# vitest suite imports `@workflow/core` and shells out to the `workflow`
# CLI to inspect runs.
- name: Download shared package builds
uses: actions/download-artifact@v4
with:
name: e2e-package-build-artifacts
path: packages
# Pinned rather than floating to latest: `uv sync --locked` below is only
# a meaningful gate if the tool reading the lock is a known quantity.
# Pinning here rather than via `[tool.uv] required-version` is deliberate
# — that key would also bind the uv that `@vercel/python` bundles on the
# build image, which we do not control.
- name: Setup uv
uses: astral-sh/setup-uv@c771a70e6277c0a99b617c7a806ffedaca235ff9 # v9.0.0
with:
version: '0.12.2'
enable-cache: true
cache-dependency-glob: workbench/python/uv.lock
# `--locked` rather than `--frozen`: a `pyproject.toml` edit that was never
# relocked should fail the job, not be silently ignored.
- name: Install Python dependencies
working-directory: workbench/python
run: uv sync --locked
- name: Run E2E Tests
run: |
APP_LOG="$GITHUB_WORKSPACE/python-app.log"
rm -f "$APP_LOG"
(cd workbench/python && uv run uvicorn app:app --port 3000 > "$APP_LOG" 2>&1) &
for _ in $(seq 1 30); do
if curl -sf http://localhost:3000/.well-known/workflow/v1/manifest.json -o /dev/null; then
echo "app is up"
break
fi
sleep 1
done
curl -sf http://localhost:3000/.well-known/workflow/v1/manifest.json | head -c 2000
echo
# Only `e2e.test.ts` — `e2e-agent.test.ts` (which `pnpm test:e2e` also
# runs) has no conformance gate and needs routes this app does not serve.
pnpm vitest run packages/core/e2e/e2e.test.ts \
--reporter=verbose --reporter=json \
--reporter=./packages/core/e2e/github-reporter.ts \
--outputFile=e2e-conformance-python.json
env:
NODE_OPTIONS: "--enable-source-maps"
APP_NAME: "python"
WORKBENCH_APP_PATH: ${{ github.workspace }}/workbench/python
DEPLOYMENT_URL: "http://localhost:3000"
- name: Print Python app logs
if: always()
run: cat "$GITHUB_WORKSPACE/python-app.log" || true
- name: Generate E2E summary
if: always()
run: node .github/scripts/aggregate-e2e-results.js . --job-name "E2E Python Conformance" >> $GITHUB_STEP_SUMMARY || true
- name: Upload E2E results
if: always()
uses: actions/upload-artifact@v4
with:
name: e2e-results-conformance-python
path: |
e2e-conformance-python.json
e2e-flaky-python-local.json
e2e-infra-python-local.json
e2e-diagnostics-python-local.json
retention-days: 7
if-no-files-found: ignore
e2e-windows:
name: E2E Windows Tests (${{ matrix.vm }})
runs-on: windows-latest
timeout-minutes: 30
needs: ci-scope
if: ${{ needs.ci-scope.outputs.fast-path != 'true' && !contains(github.event.pull_request.labels.*.name, 'workflow-server-test') }}
strategy:
fail-fast: false
matrix:
# Workflow VM engines: node:vm (default) and the opt-in QuickJS
# WASM engine (WORKFLOW_VM=quickjs).
vm: [node, quickjs]
env:
TURBO_TOKEN: ${{ secrets.TURBO_TOKEN }}
TURBO_TEAM: ${{ vars.TURBO_TEAM }}
WORKFLOW_PUBLIC_MANIFEST: '1'
steps:
- name: Checkout Repo
uses: actions/checkout@v4
with:
ref: ${{ github.event.pull_request.head.sha || github.sha }}
- name: Setup Rust
uses: actions-rust-lang/setup-rust-toolchain@46268bd060767258de96ed93c1251119784f2ab6 # v1.16.1
with:
toolchain: stable
target: wasm32-unknown-unknown
- name: Setup pnpm
uses: pnpm/action-setup@fc06bc1257f339d1d5d8b3a19a8cae5388b55320 # v5.0.0
- name: Setup Node.js 22.x
uses: actions/setup-node@v4
with:
node-version: 22.x
cache: "pnpm"
- name: Install Dependencies
run: pnpm install --frozen-lockfile --ignore-scripts
- name: Run Initial Build
run: pnpm turbo run build --filter='!./workbench/*' --filter='!./docs'
- name: Run E2E Tests (Next.js)
run: |
cd workbench/nextjs-turbopack
$logFile = "$env:GITHUB_WORKSPACE/nextjs-server.log"
$env:DEV_SERVER_LOG_PATH = $logFile
# `$using:` only resolves PowerShell variables, not env vars, so
# copy MATRIX_VM into a session variable before Start-Job.
$matrixVm = $env:MATRIX_VM
$job = Start-Job -ScriptBlock { Set-Location $using:PWD; $env:WORKFLOW_VM = $using:matrixVm; pnpm dev *>&1 | Tee-Object -FilePath $using:logFile }
Start-Sleep -Seconds 15
cd ../..
# Fail-fast guards: if the Next.js dev server gets into a corrupted
# state during HMR (a recurring Turbopack-on-Windows issue where
# `packages/core/dist/*.js` becomes "MODULE_UNPARSABLE" and every
# subsequent request returns 500), the e2e suite would otherwise burn
# the full 30-minute job window polling stuck workflow runs before
# finally being cancelled. Below we (1) bail out if `dev.test.ts`
# fails, since that's the first thing to start hitting 500s, and
# (2) probe the server before the long e2e run to catch any other
# silent breakage.
#
# Get-DevServerStatus returns the HTTP status code of GET /api/chat,
# or 0 if the request never produced a response. /api/chat is a
# POST-only endpoint, so a healthy dev server responds with 405
# Method Not Allowed; we only treat 5xx (and no-response) as
# unhealthy.
function Get-DevServerStatus {
try {
$resp = Invoke-WebRequest -Uri "http://localhost:3000/api/chat" -UseBasicParsing -TimeoutSec 15 -ErrorAction Stop
return [int]$resp.StatusCode
} catch {
if ($_.Exception.Response) {
try { return [int]$_.Exception.Response.StatusCode } catch { return 0 }
}
return 0
}
}
$devExit = 0
pnpm vitest run packages/core/e2e/dev.test.ts
$devExit = $LASTEXITCODE
if ($devExit -ne 0) {
Write-Host "::warning title=dev.test.ts failed::dev.test.ts exited with code $devExit. Skipping the remainder of the e2e suite to avoid waiting for the 30-minute job timeout."
Stop-Job $job -ErrorAction SilentlyContinue
exit $devExit
}
# Sanity-check the server before kicking off the long e2e suite. This
# catches the case where Turbopack quietly went sideways (e.g. lost
# track of files in `packages/core/dist` after HMR on Windows) and is
# returning 500 for every request. Without this check the e2e suite
# would burn the full 30-minute job window polling stuck workflow
# runs before getting cancelled.
$status = Get-DevServerStatus
Write-Host "[health-check:pre-e2e] GET /api/chat -> $status"
if ($status -eq 0 -or $status -ge 500) {
Write-Host "::error title=Next.js dev server unhealthy::Dev server returned $status before e2e tests. Aborting to avoid the 30-minute job timeout. See the 'Print Next.js server logs' step for the underlying Turbopack error."
Stop-Job $job -ErrorAction SilentlyContinue
exit 1
}
pnpm run test:e2e --reporter=verbose --reporter=json --reporter=./packages/core/e2e/github-reporter.ts --outputFile=e2e-windows-nextjs-turbopack-$env:MATRIX_VM.json
$e2eExit = $LASTEXITCODE
Stop-Job $job -ErrorAction SilentlyContinue
exit $e2eExit
shell: powershell
env:
NODE_OPTIONS: "--enable-source-maps"
APP_NAME: "nextjs-turbopack"
DEPLOYMENT_URL: "http://localhost:3000"
# Use a real workflow file for the Windows HMR test. Most of this
# workbench's workflows are symlinks into workbench/example, and
# Windows/Turbopack can rebuild from the symlink path while still
# serving the previous module contents from the realpath cache.
DEV_TEST_CONFIG: '{"generatedStepRegistrationPath":"app/.well-known/workflow/v1/flow/__step_registrations.js","generatedWorkflowPath":"app/.well-known/workflow/v1/flow/route.js","apiFilePath":"app/api/chat/route.ts","apiFileImportPath":"../../..","port":3000,"testWorkflowFile":"96_many_steps.ts"}'
DEV_SERVER_LOG_PATH: "${{ github.workspace }}/nextjs-server.log"
WORKFLOW_DEV_HMR_LOGS: "1"
WORKFLOW_VM: ${{ matrix.vm }}
MATRIX_VM: ${{ matrix.vm }}
- name: Print Next.js server logs
if: always()
shell: powershell
run: |
$logFile = "$env:GITHUB_WORKSPACE/nextjs-server.log"
if (Test-Path $logFile) {
Write-Host "=== Next.js Server Logs ===" -ForegroundColor Cyan
Get-Content $logFile
Write-Host "=== End of Server Logs ===" -ForegroundColor Cyan
} else {
Write-Host "No server log file found at $logFile" -ForegroundColor Yellow
}
- name: Generate E2E summary
if: always()
shell: bash
env:
MATRIX_VM: ${{ matrix.vm }}
run: node .github/scripts/aggregate-e2e-results.js . --job-name "E2E Windows (nextjs-turbopack - $MATRIX_VM)" >> $GITHUB_STEP_SUMMARY || true
- name: Upload E2E results
if: always()
uses: actions/upload-artifact@v4
with:
name: e2e-results-windows-nextjs-turbopack-${{ matrix.vm }}
path: |
e2e-windows-nextjs-turbopack-${{ matrix.vm }}.json
e2e-flaky-nextjs-turbopack-local.json
e2e-infra-nextjs-turbopack-local.json
retention-days: 7
if-no-files-found: ignore
- name: Upload Next.js server logs
if: always()
uses: actions/upload-artifact@v4
with:
name: nextjs-server-logs-windows-${{ matrix.vm }}
path: nextjs-server.log
retention-days: 7
if-no-files-found: ignore
# Community World E2E Tests (dynamically generated from worlds-manifest.json)
# Disabled on main: community worlds target spec v2 and do not yet support the
# CBOR queue transport used by world-vercel (see #1627 / #1658). They continue
# to run on `stable`, where spec v2 is still the primary compat target. Once
# the community worlds (mizzle-dev/workflow-worlds) ship CBOR support we can
# re-enable these jobs here.
getCommunityWorldsMatrix:
name: Get Community Worlds Matrix
runs-on: ubuntu-latest
if: false
outputs:
matrix: ${{ steps.set-matrix.outputs.matrix }}
steps:
- uses: actions/checkout@v4
with:
ref: ${{ github.event.pull_request.head.sha || github.sha }}
- name: Setup environment
uses: ./.github/actions/setup-workflow-dev
with:
install-dependencies: 'false'
build-packages: 'false'
cache-pnpm: 'false'
- id: set-matrix
run: |
MATRIX_JSON="$(node ./scripts/create-community-worlds-matrix.mjs | jq -c .)"
echo "matrix=$MATRIX_JSON" >> "$GITHUB_OUTPUT"
e2e-community:
name: E2E Community World (${{ matrix.world.name }})
if: false
needs: getCommunityWorldsMatrix
strategy:
fail-fast: false
matrix: ${{ fromJson(needs.getCommunityWorldsMatrix.outputs.matrix) }}
uses: ./.github/workflows/e2e-community-world.yml
with:
world-id: ${{ matrix.world.id }}
world-name: ${{ matrix.world.name }}
world-package: ${{ matrix.world.package }}
package-version: ${{ matrix.world.version }}
service-type: ${{ matrix.world.service-type }}
env-vars: ${{ matrix.world.env-vars }}
services: ${{ matrix.world.services }}
secrets: inherit
# Final job: Aggregate all E2E results and update PR comment
summary:
name: E2E Summary
runs-on: ubuntu-latest
needs: [ci-scope, e2e-vercel-prod, e2e-vercel-ws-transport, e2e-vercel-http-transport, e2e-local-dev, e2e-local-prod, e2e-local-postgres, e2e-python, e2e-windows]
if: always() && !cancelled() && needs.ci-scope.outputs.fast-path != 'true'
timeout-minutes: 10
steps:
- uses: actions/checkout@v4
with:
ref: ${{ github.event.pull_request.head.sha || github.sha }}
- name: Download all E2E artifacts
uses: actions/download-artifact@v4
with:
pattern: e2e-results-*
path: e2e-results
merge-multiple: true
- name: List downloaded files
run: find e2e-results -type f -name "*.json" | sort || echo "No files found"
- name: Aggregate E2E results
id: aggregate
run: |
node .github/scripts/aggregate-e2e-results.js e2e-results --mode aggregate --run-url "${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}" | tee e2e-summary.md >> $GITHUB_STEP_SUMMARY
- name: Update PR comment with results
if: github.event_name == 'pull_request'
uses: marocchino/sticky-pull-request-comment@773744901bac0e8cbb5a0dc842800d45e9b2b405 # v2.9.4
with:
header: e2e-test-results
path: e2e-summary.md
# Final required check: passes only when unit + all E2E jobs succeed.
# Community worlds are intentionally excluded — they are disabled on main.
e2e-required-check:
name: E2E Required Check
runs-on: ubuntu-latest
needs: [ci-scope, unit, e2e-package-build, e2e-vercel-prod, e2e-vercel-ws-transport, e2e-vercel-http-transport, e2e-local-dev, e2e-local-prod, e2e-local-postgres, e2e-python, e2e-windows]
if: always()
timeout-minutes: 5
steps:
- name: Validate E2E job statuses
env:
UNIT_STATUS: ${{ needs.unit.result }}
BUILD_STATUS: ${{ needs.e2e-package-build.result }}
VERCEL_STATUS: ${{ needs.e2e-vercel-prod.result }}
VERCEL_WS_STATUS: ${{ needs.e2e-vercel-ws-transport.result }}
VERCEL_HTTP_STATUS: ${{ needs.e2e-vercel-http-transport.result }}
LOCAL_DEV_STATUS: ${{ needs.e2e-local-dev.result }}
LOCAL_PROD_STATUS: ${{ needs.e2e-local-prod.result }}
POSTGRES_STATUS: ${{ needs.e2e-local-postgres.result }}
PYTHON_STATUS: ${{ needs.e2e-python.result }}
WINDOWS_STATUS: ${{ needs.e2e-windows.result }}
FAST_PATH: ${{ needs.ci-scope.outputs.fast-path }}
VALIDATION_FAST_PATH: ${{ needs.ci-scope.outputs.validation-fast-path }}
HAS_LABEL: ${{ contains(github.event.pull_request.labels.*.name, 'workflow-server-test') }}
run: |
FAILED_JOBS=()
if [[ "$FAST_PATH" == "true" ]]; then
if [[ "$VALIDATION_FAST_PATH" != "true" && "$HAS_LABEL" != "true" && "$UNIT_STATUS" != "success" ]]; then
echo "UI-only fast path still requires unit tests; unit status was $UNIT_STATUS."
exit 1
fi
echo "Docs/UI-only change detected - no runtime E2E lanes are required."
exit 0
fi
# When workflow-server-test label is present, only check vercel-prod (others should be skipped)
# Otherwise, all jobs must succeed
if [[ "$HAS_LABEL" == "true" ]]; then
echo "workflow-server-test label detected - only checking required jobs"
[[ "$VERCEL_STATUS" == "success" ]] || FAILED_JOBS+=("e2e-vercel-prod ($VERCEL_STATUS)")
# This label is itself opt-in for the WS lane (see that job's `if`),
# so it runs here and is required.
[[ "$VERCEL_WS_STATUS" == "success" ]] || FAILED_JOBS+=("e2e-vercel-ws-transport ($VERCEL_WS_STATUS)")
# Unconditional, exactly like e2e-vercel-prod above: the HTTP
# lane has no opt-in label, so there is no case in which it is
# legitimately skipped while the rest of this branch runs.
[[ "$VERCEL_HTTP_STATUS" == "success" ]] || FAILED_JOBS+=("e2e-vercel-http-transport ($VERCEL_HTTP_STATUS)")
# Verify other jobs were actually skipped
[[ "$UNIT_STATUS" == "skipped" ]] || echo "Warning: unit was not skipped ($UNIT_STATUS)"
[[ "$LOCAL_DEV_STATUS" == "skipped" ]] || echo "Warning: e2e-local-dev was not skipped ($LOCAL_DEV_STATUS)"
[[ "$LOCAL_PROD_STATUS" == "skipped" ]] || echo "Warning: e2e-local-prod was not skipped ($LOCAL_PROD_STATUS)"
[[ "$POSTGRES_STATUS" == "skipped" ]] || echo "Warning: e2e-local-postgres was not skipped ($POSTGRES_STATUS)"
[[ "$PYTHON_STATUS" == "skipped" ]] || echo "Warning: e2e-python was not skipped ($PYTHON_STATUS)"
[[ "$WINDOWS_STATUS" == "skipped" ]] || echo "Warning: e2e-windows was not skipped ($WINDOWS_STATUS)"
else
echo "Standard PR - checking all jobs"
[[ "$UNIT_STATUS" == "success" ]] || FAILED_JOBS+=("unit ($UNIT_STATUS)")
[[ "$BUILD_STATUS" == "success" ]] || FAILED_JOBS+=("e2e-package-build ($BUILD_STATUS)")
[[ "$VERCEL_STATUS" == "success" ]] || FAILED_JOBS+=("e2e-vercel-prod ($VERCEL_STATUS)")
[[ "$VERCEL_WS_STATUS" == "success" ]] || FAILED_JOBS+=("e2e-vercel-ws-transport ($VERCEL_WS_STATUS)")
[[ "$VERCEL_HTTP_STATUS" == "success" ]] || FAILED_JOBS+=("e2e-vercel-http-transport ($VERCEL_HTTP_STATUS)")
[[ "$LOCAL_DEV_STATUS" == "success" ]] || FAILED_JOBS+=("e2e-local-dev ($LOCAL_DEV_STATUS)")
[[ "$LOCAL_PROD_STATUS" == "success" ]] || FAILED_JOBS+=("e2e-local-prod ($LOCAL_PROD_STATUS)")
[[ "$POSTGRES_STATUS" == "success" ]] || FAILED_JOBS+=("e2e-local-postgres ($POSTGRES_STATUS)")
[[ "$PYTHON_STATUS" == "success" ]] || FAILED_JOBS+=("e2e-python ($PYTHON_STATUS)")
[[ "$WINDOWS_STATUS" == "success" ]] || FAILED_JOBS+=("e2e-windows ($WINDOWS_STATUS)")
fi
if [[ ${#FAILED_JOBS[@]} -gt 0 ]]; then
echo "The following E2E jobs did not succeed:"
printf '%s\n' "${FAILED_JOBS[@]}"
exit 1
fi
echo "All required E2E jobs succeeded."
# Publish E2E results to GitHub Pages (main branch only)
publish-results:
name: Publish E2E Results
runs-on: ubuntu-latest
needs: [summary]
# Use always() because summary uses always() - without it this job gets skipped when any E2E test fails
if: always() && !cancelled() && github.event_name == 'push' && github.ref == 'refs/heads/main'
timeout-minutes: 5
permissions:
contents: write
steps:
- uses: actions/checkout@v4
with:
ref: ${{ github.event.pull_request.head.sha || github.sha }}
- name: Download all E2E artifacts
uses: actions/download-artifact@v4
with:
pattern: e2e-results-*
path: e2e-results
merge-multiple: true
- name: Generate docs data
run: |
mkdir -p docs-data
node .github/scripts/generate-docs-data.js \
--type e2e \
--results-dir e2e-results \
--output docs-data/e2e-results.json \
--commit "${{ github.sha }}" \
--branch "${{ github.ref_name }}"
- name: Deploy to GitHub Pages
uses: ./.github/actions/publish-to-gh-pages
with:
github-token: ${{ secrets.GITHUB_TOKEN }}
source-dir: ./docs-data
destination-dir: ci
Reference in New Issue View Git Blame Copy Permalink