mirror of
https://github.com/vercel/workflow.git
synced 2026-09-14 19:59:43 +08:00
test/server-pr-952
41 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Alex Langenfeld
|
080c592567 |
test: persist E2E flake history (#4120)
## Summary & Motivation Job conclusions hide tests that pass on retry, and the evidence only lived in overwritten PR comments and 7-day artifacts, so recurring flakes couldn't be ranked against how often they ran. - retry sidecars are now named after the Vitest report they came from, so lanes, VMs, and worlds no longer overwrite each other when artifacts merge — and the aggregate comment can attribute a flake to an exact lane - `generate-e2e-flake-history.js` publishes a bounded 30-run history to gh-pages: one series per (lane, app, world, vm, platform) × test, carrying both an executed count and a passed-on-retry count so a rate has a denominator - the reporter always writes the sidecar, which is what lets a clean run be distinguished from a lane that reported no retry telemetry at all ## Test Plan Unit tests added for dimension parsing, denominators, schema validation, and the 30-run window; verified every current report filename maps to explicit dimensions, and a one-run history built from a full artifact set came out around 100 KiB. |
||
|
Peter Wielander
|
8a91d18d0d | [core] Add the wake-loop scenario to the event log race repro (#4017) | ||
|
Peter Wielander
|
c8bcde53d0 | [ci] Track the /flow route bundle size against main (#3739) | ||
|
Peter Wielander
|
9760f640bb | [e2e] Change race repro hook poke to soft-degrade instead of hard-stop at budget (#3561) | ||
|
Peter Wielander
|
c431cc18fd | [e2e] Add blocked-branch scenario to the event-log race repro (#3696) | ||
|
Alex Langenfeld
|
5fca3bb589 |
bench: consolidate stream throughput columns (#3585)
Signed-off-by: Alex Langenfeld <alex.langenfeld@vercel.com> |
||
|
Alex Langenfeld
|
3837ca2dad |
[core] Warm cold e2e targets before the suite starts runs (#3590)
A target answers HTTP well before its first run is picked up promptly, for two reasons with one shape: a fresh Vercel deployment's queue consumer takes a while to start delivering, and a local dev server pays its first flow-route compile on the first queue delivery. The run-pickup watchdog's telemetry shows the cost - stalls concentrated on the suite's first test (addTenWorkflow), waitedMs pegged at the full 15s pickup budget, timestamps right at suite start; the sidecar backends identify local-dev lanes as a dominant source alongside fresh Vercel deployments. Each stall burns pickup budget inside a test, drowns the infra telemetry in cold-start noise, and leaves the first tests one stalled replacement away from failing. warmDeployment() runs in the suite's beforeAll: it starts throwaway probe runs, abandoning (best-effort cancelling) any still pending after the pickup budget, until one is picked up or a total budget (WORKFLOW_E2E_WARMUP_BUDGET_MS, default 120s) is spent. A warmup that needed abandoned probes is recorded as a single cold-start-warmup infra event - one per suite instead of per-test run-pickup-stall noise - and an exhausted budget proceeds anyway: the per-test watchdog still guards every start, and test failures carry run diagnostics a thrown warmup would not. Signed-off-by: Alex Langenfeld <alex.langenfeld@vercel.com> |
||
|
Alex Langenfeld
|
234d3dd7b8 |
[core] Replace never-picked-up e2e runs instead of failing whole tests (#3560)
## Summary & Motivation The e2e `start()` wrappers now poll the new run until it leaves `pending`. A run still pending after `WORKFLOW_E2E_PICKUP_BUDGET_MS` (default 15s) has executed no workflow code, so it is abandoned and replaced in place and the test continues — one replacement, with the CI-level retry still the backstop if that one stalls too. Each replacement is recorded to an `e2e-infra-*.json` sidecar that every e2e job uploads, and the aggregation script renders it as an "Infra Events" section in the step summary and PR comment, so clustered timestamps read as a backend blip rather than as unrelated flaky tests. ## Test Plan Unit tests cover the pickup watchdog; a local nextjs-turbopack run exercised both the clean path and, with a forced 1ms budget, the replacement path end to end, and the aggregation script was smoke-tested against synthetic sidecars in both modes. Signed-off-by: Alex Langenfeld <alex.langenfeld@vercel.com> |
||
|
Fantix King
|
09b299a03e |
[e2e] Add Python e2e Test (#3369)
Turns `packages/core/e2e/e2e.test.ts` into a cross-language conformance suite and adds `workbench/python` as its first non-JavaScript subject. --------- Co-authored-by: Claude Opus 5 <noreply@anthropic.com> |
||
|
Alex Langenfeld
|
af91cc2582 |
bench: per-chunk stream latency (CRTT/CDV) and replay-driven stream scenarios (#3393)
## Summary & Motivation - **CRTT (chunk round-trip time)** — per-chunk write→read latency for a paced stream, aggregated inside the reader step on the deployment (one clock domain) into a fixed log-bin histogram plus index buckets and mean-RTT profiles over stream progress and chunk size. Fills the gap between SL (first chunk only) and SO (whole-stream throughput), where a mid-stream delivery regression was invisible. It is deliberately a *round*-trip name: the future production one-way write→read metric is CTT, with its own skew caveats. - **CDV (chunk delay variation)** — inter-arrival gap minus inter-write gap per seq-adjacent pair, so each gap subtracts same-clock stamps and the stat stays skew-free and measurable in production later. Reported as each run's max positive value, since a 1-in-300 delivery stall dilutes out of pooled percentiles. - **Replay scenarios** — two real captured cadences (eve envelope protocol via gpt-5.6-sol; raw gateway SSE via gpt-5.4-nano) replayed through the same rig on an absolute open-loop schedule, so the workload is measured rather than invented; the 2x speed multiplier is the only chosen number, and matches how real fast-tier models behave (same chunk sizes, compressed time). Each capture carries a semantic sha256 over canonical `(offsetMs, bytes)` tuples so durabench's independent copy can be checked for drift. - **Streams table** — stream scenarios render in their own table with writer/reader sustained rates, CRTT percentiles, and median worst stall. No pass/fail targets yet: numbers and vs-main deltas only. - **SL/SO report rows retired** — CRTT's seq-0 slice reproduces SL and its aggregate reproduces SO's signal at ~100x the samples; write slip stays as artifact-only data, the only guard for producer stalls that neither CRTT nor CDV can see. ## Test Plan - [x] Unit tests for the bucketing/merge/CDV helpers and the renderer; the full benchmarks job ran green against real preview deployments, and the first Streams numbers separated workload strain (eve 2x: read 173 < write 181 c/s, CRTT p75 1278ms) from the transport floor (the paced control and the 1x reality row both clean). --------- Signed-off-by: Alex Langenfeld <alex.langenfeld@vercel.com> |
||
|
Peter Wielander
|
f5591aa278 | [e2e] Fix event-log-race-repro for local/postgres (#3558) | ||
|
Peter Wielander
|
0c5a6495bc | [ci] Report all three event-log-race-repro lanes in one small PR comment (#3556) | ||
|
Alex Langenfeld
|
c041d3d231 |
[ci] Retry e2e tests once in CI, keeping retried tests visible (#3530)
* [ci] Retry e2e tests once in CI, keeping retried tests visible Over the last 10 days ~93 Tests runs were manually re-run until green, some taking 6 attempts: the e2e suites drive real deployments, and a single test losing a timing race fails a whole 20+ minute matrix job. A CI-only vitest retry (retry: 1) absorbs those single-test races. beforeEach/afterEach hooks run per attempt, so suites with file-restore hooks (dev.test.ts) retry cleanly. Retries must not hide real races, so a retried-then-passed test stays visible everywhere a failure would have been: the github-reporter emits ::warning annotations and an e2e-flaky-*.json sidecar, every e2e job uploads it, and aggregate-e2e-results.js renders a 'Flaky E2E Tests (passed on retry)' section in both the per-job step summary and the PR comment, with per-app occurrence counts. Harnesses whose failures are themselves the signal pin retry: 0: event-log-race-repro (a pass runs the full configured budget) and benchmarks (a regression should not be papered over by a luckier second sample). Local runs keep retry at 0 so races reproduce while debugging. Signed-off-by: Alex Langenfeld <alex.langenfeld@vercel.com> * Add changeset Signed-off-by: Alex Langenfeld <alex.langenfeld@vercel.com> --------- Signed-off-by: Alex Langenfeld <alex.langenfeld@vercel.com> |
||
|
Peter Wielander
|
dc85865718 | [core] Drop pre-slot event ID support and preconditionGuard capability (#3519) | ||
|
Peter Wielander
|
0b7c9671ee | [bench] Add a Promise.all fan-out scenario with Fan-out TTFS/TTLS rows (#3522) | ||
|
Shalabh Chaturvedi
|
264ddff67b |
Add WebSocket transport for step-execution event writes (opt-in) (#3084)
* sdk side for workflow server websockets Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * hardcoded workflow server Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * debug info Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * more debug Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * remove unnecessary debug Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * default on websockets, and override url Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * fix for missing funcs Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * make websockets opt outo Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * enable ws again Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * [revert later] reduce test to single test, test both http and ws at the same time Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * empty Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * Run full suite with and without ws Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * empty Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * improve e2e test Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * minimize tests Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * fallback to http when proxy present Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * default to websockets, remove matrix Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * remove smoke test Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * update to new protocol Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * adjust for new protocol (runid in path) Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * fix ws transport error Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * blank Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * fix ws dep Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * fix ws external: only accelerators, not ws itself Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * add dedicated WS-transport e2e job; flip WS default back to opt-in Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * build all packages before local vercel build (needs workflow/nitro on disk) Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * force NITRO_PRESET=vercel for the local vercel build step Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * install vercel CLI once instead of npx-ing it per command Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * add changeset for WS events transport Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * Harden the WS events transport and gate its e2e jobs Follow-ups from review of the WS transport. CI: `e2e-vercel-ws-transport` was wired into the `summary` job but not into `e2e-required-check`, so all three WS jobs could fail while the required check stayed green. Added to both branches of the status validation — including the `workflow-server-test` label branch, where the job runs under the same gating as `e2e-vercel-prod`. Transport: - `reqId` and the pending-reply map are now per connection rather than per transport. The protocol defines `reqId` as a per-connection counter, so a reconnected socket restarts at 1; with one shared map that collided with the previous socket's still-registered waiters. It also makes the superseded-socket guard structural instead of something the close path has to remember. - Post-open socket errors are no longer silent. The only `'error'` listener closed over the connect promise's `reject`, already settled once `'open'` fired, so every broken pipe / 1009 / protocol fault was swallowed and its requests hung with no per-request timeout to save them. Now logged, and the connection is torn down. - An unexpected close reconnects eagerly instead of waiting for the next write, since a socket breaking mid-run means more writes are coming. Bounded by exponential backoff, an attempt cap that falls back to lazy reconnect, a bail-out when a newer socket is already live, and an `unref()`ed timer so a backoff window can't delay handler exit. - `ws.send()` failures reject their request. `send()` doesn't throw on a non-OPEN socket — it reports through a callback we weren't passing — so the request just sat in `pending` forever. - The reserved `reqId: -1` malformed-frame reply and undecodable frames are logged loudly instead of dropped. - Auth headers resolve once per socket via a thunk, not once per event. The bearer only rides the upgrade, so the old code awaited `getVercelOidcToken()` on every write and discarded all but the first. Re-resolving on reconnect also means a new socket gets a fresh token. Adapter: a reply with no numeric status now fails closed. Defaulting to 200 reported a write as applied whenever the client met a frame it didn't understand — and the protocol is explicitly designed to grow new response variants. Tests: 24 new unit tests over the paths the e2e suite can't reach on demand (send failure mid-flight, error after open, late close from a superseded socket, reconnect backoff and give-up, sentinel/undecodable frame logging, one-token-per-socket) plus the adapter's fail-closed and typed-error mapping. Co-Authored-By: Shalabh Chaturvedi <shalabhc@users.noreply.github.com> Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * ci: re-trigger to confirm the prior e2e failures were flake No code change. The 5 failures on 0bb21e7 clustered in a ~20s window across HTTP-path jobs (example/nuxt on the same test, sveltekit on a timeout) and one WS job (sleepingWorkflow's clock-skew assertion), which points at the environment rather than the transport changes. Re-running to confirm. Co-Authored-By: Shalabh Chaturvedi <shalabhc@users.noreply.github.com> Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * Add WS wire-contract conformance tests and pin the transport gate Three gaps in the existing coverage. **The HTTP path was already covered** — `events-v4.test.ts` has 22 tests, including five directly on `createWorkflowRunEventV4` over HTTP (alias URL, frame meta contents, response decoding, skipPreload/stateUpdatedAt forwarding). Those run with the gate unset, so they do confirm the two-branch refactor didn't disturb HTTP. No new tests needed there. **But nothing pinned the gate itself.** Every HTTP assertion stays green if the default flips to WS, because the transports are built to be indistinguishable at the result layer — and an earlier revision of this branch did flip the default deliberately, for benchmarking. Added tests for `isWsEventsTransportEnabled()` across values, and one that drives a real HTTP request through a MockAgent while asserting the WS transport is never constructed. **Nothing verified the bytes.** `ws-transport.test.ts` replies with whatever the test hands it, which proves the client's lifecycle but not that its frames are what workflow-server accepts. That's the drift the spec doc exists to prevent, and it already happened once: event meta flat on the frame where the server wanted it nested under `event`, with both sides' tests passing. `ws-protocol-conformance.test.ts` pairs the real client stack (through `createWorkflowRunEventV4`) with a fixture mirroring the server route's per-message handling: decode one frame, validate against a local copy of `WsRequestFrameSchema`, dispatch, encode the reply the way `replyMeta` does. `experimental_upgradeWebSocket` needs a real Vercel runtime, so the socket is faked — everything above it is genuine. Covers: the frame shape the server accepts (and that `reqId`/`type`/ `runId` don't leak into the event meta), payload passthrough, exactly one frame per message, 409 → the same typed error HTTP raises, fail-closed on an unknown reply variant, and reqId correlation across concurrent writes. Plus golden byte fixtures, since the schema copy is the one thing here that can silently drift. This is the "golden-frame interop test" the server spec lists as an open gap; the matching half still needs to land in workflow-server. Verified the conformance suite is not vacuous: flattening the client's frame meta fails 5 of its tests. Co-Authored-By: Shalabh Chaturvedi <shalabhc@users.noreply.github.com> Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * match the HTTP RetryAgent's transient-failure policy on WS HTTP event writes go through an undici RetryAgent (RETRY_AGENT_OPTIONS): 5xx and transient connection errors are retried in-process, honoring Retry-After. The WS path never touches undici, so it shipped with no transient-failure handling at all — a single 503 or a mid-write reset surfaced straight to the step runtime and cost a whole step retry where HTTP would have absorbed it in milliseconds. That gap is invisible in a passing test run: writes still succeed, they just cost far more. So copy the policy rather than reinvent it — [500, 502, 503, 504] plus transport failures, undici's default backoff, Retry-After honored, and 429 deliberately excluded for the same reason RETRY_AGENT_OPTIONS excludes it (a firewall challenge this client cannot solve, which in-process retries only amplify). Adds WsTransportError so retryability is a typed property of the failure rather than something the adapter infers by string-matching. Splits resolveWsTransport()/wsReplyStatus() out of postEventFrameOverWs so the retry loop stays readable. The existing "fails closed on an error frame" test used status 500, which is now absorbed by the retry — switched to 403 so it keeps testing fail-closed rather than accidentally testing no-retry. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Co-Authored-By: Shalabh Chaturvedi <7066873+shalabhc@users.noreply.github.com> Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * lazy-load `ws` so the default HTTP path never evaluates it events-v4.ts imports ws-transport.js unconditionally — the transport gate is a runtime branch, not a build-time one — so a top-level `import { WebSocket } from 'ws'` put `ws` and its optional native accelerators on the module-init path of every deployment, including the overwhelming majority that never opt in and never open a socket. Defer it to the first connect, memoized as a promise so concurrent first connects share one import. WebSocket.OPEN becomes an inlined constant so the readyState check doesn't pull the module in just to read it off the constructor. This does NOT remove the need for the bufferutil/utf-8-validate externals this branch also adds: webpack and Rollup both statically follow a dynamic import(), so the build-time story is unchanged. What it buys is that a deployment which never enables the transport never *evaluates* `ws`, so a mis-bundled accelerator can't break it. The test lives in its own file because vitest caches a vi.mock factory result for the life of the module registry — once any test in a file has connected, the factory never runs again and the counter can't distinguish "loaded lazily" from "loaded at import". Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Co-Authored-By: Shalabh Chaturvedi <7066873+shalabhc@users.noreply.github.com> Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * release idle WS transports instead of renewing them forever The transports map was never pruned and WsEventsTransport had no way to close. Combined with eager reconnect that made a connection immortal by construction: the server drains at its own maxDuration and closes, the client immediately reopens, and the server pins a fresh invocation — for a run that finished long ago. A warm container ended up holding a live socket, and a live server invocation, for every runId it had ever served. workflow-server#683 already lists "one invocation stays resident per run rather than per write" as a known gap; this made it "per run, forever". Add close() plus a 60s idle release. There is no "run complete" signal to hang teardown off — the events adapter is a stateless per-write call — so idleness is the available proxy. 60s sits well below the server's ~680s drain deadline, so the client releases rather than the server reclaiming, and well above the gap between steps of an active run. scheduleReconnect() now bails when closed: close() closes the socket, which fires the same close handler an unexpected drop would, and without the guard the transport would instantly reconnect what it just released. request() revives an idle-closed transport rather than failing the write, re-registering itself only if nothing newer has claimed the map slot. Eviction therefore costs one handshake, not an error. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Co-Authored-By: Shalabh Chaturvedi <7066873+shalabhc@users.noreply.github.com> Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * refresh the bearer on an auth_expiry drain workflow-server#683 tags a drain frame with why it is closing: max_duration means the socket aged out and a plain reconnect is right, auth_expiry means the *bearer* ran out and reconnecting with the same one just earns a 401. This client logged the drain and ignored the reason, so against #683 an auth_expiry drain would burn all five reconnect attempts against a token the server had already rejected, then give up. Parse the reason (absent reads as max_duration, so this stays correct against the currently-deployed server) and thread forceRefresh through the getHeaders thunk, which triggers @vercel/oidc's refresh path via a wide expirationBufferMs. Worth being precise about when that can actually help. getVercelOidcToken resolves getContext().headers['x-vercel-oidc-token'] ?? env, and refreshToken() only writes the env var — the request-context header wins. So inside a deployed function there is genuinely no fresher token mid-invocation and the refresh is a no-op; outside one (CLI, local dev, a long-lived server) it works. That makes the guard the load-bearing half: if the re-resolved bearer is byte-identical, decline to reconnect, say so, and wait for the next write — which usually arrives on a new invocation carrying a new token. That failure is marked non-retryable so the retry loop doesn't spin on it either. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Co-Authored-By: Shalabh Chaturvedi <7066873+shalabhc@users.noreply.github.com> Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * document the WS path's instrumentation gap The HTTP branch goes through fetchV4 -> instrumentedFetch, which is not just a fetch wrapper: it opens the OTEL CLIENT span, injects trace context, sets the cache-bust header, emits the DEBUG logs, and routes through the global fetch that Vercel's observability "outgoing requests" view instruments. The comment on fetchV4 records why that matters — bypassing it via undici.request() is exactly what once made v4 event traffic disappear from the log viewer. The WS branch bypasses all of it. With the flag on, per-event writes have no client span, propagate no trace context to workflow-server, and don't appear in the outgoing-requests view; the server's own transport-tagged request metrics are the only remaining signal. That's acceptable for an opt-in POC behind a flag and unacceptable as a default, so write it down where someone deciding to flip the default will read it: instrumenting the transport is a prerequisite for that, not a follow-up nicety. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Co-Authored-By: Shalabh Chaturvedi <7066873+shalabhc@users.noreply.github.com> Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * ship the ws-accelerator externals instead of documenting a workaround `bufferutil` and `utf-8-validate` are optional native accelerators for `ws`, and neither is installed by default. Every bundler has to be told to leave them alone, for two different reasons: Rollup/Vite/Nitro fail the build outright (`Could not resolve "bufferutil" imported by "ws"`), while webpack bundles the JS wrapper without its native `.node` binding and throws `bufferUtil.mask is not a function` at runtime. The webpack half shipped in `@workflow/next`. The Rollup half only existed in `workbench/vite` and `workbench/tanstack-start` as `nitro.rollupConfig.external` — app configs, not shipped code. So a real user of `@workflow/vite`, `@workflow/nitro`, `@workflow/nuxt`, `@workflow/sveltekit` or `@workflow/astro` hit the same build failure the workbench had already worked around, and had to rediscover the fix. Fix it where it propagates: `workflowTransformPlugin` in `@workflow/rollup`, which all of those integrations already install. It is already the home of exactly this pattern for the optional `@opentelemetry/api` peer, so this sits next to its closest precedent. Note the treatment is deliberately the inverse of the OTEL one, which is externalized only when it *can't* be resolved. The OTEL API must load for tracing to work, so a self-contained output has to bundle it when present. These accelerators must specifically NOT load — they are a performance nicety with a correct try/catch fallback in `ws` — so unconditional external is both simpler and safer than risking a half-bundled native module. The two workbench configs drop their local copies, which is what proves the shipped fix actually works rather than being masked by them. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Co-Authored-By: Shalabh Chaturvedi <7066873+shalabhc@users.noreply.github.com> Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * one retry policy for both transports, and no unanswerable waiters Two review findings on the WS events transport. **Retry belongs to `event-retry.ts`, not the adapter.** The WS path had its own retry loop, justified as mirroring undici's `RetryAgent`. That justification was wrong: `RetryHandler` defaults `methods` to GET/HEAD/ OPTIONS/PUT/DELETE/TRACE and nothing overrides it, so the `RetryAgent` never retried an event POST on either transport — which is precisely why `event-retry.ts` exists. Worse, that loop sat *inside* `withEventPostRetry`, so it defeated a compile-checked safety gate: `EVENT_RETRY_ELIGIBILITY` marks `step_started`, `step_retrying` and `hook_received` non-retryable (a replayed `step_started` double-increments `attempt`), and those frames were re-sent up to five times before the gate ever saw a failure. For eligible types the two loops multiplied: 3 outer attempts x 6 inner, with an inner backoff reaching 30s against an outer base deliberately set to 100ms. `postEventFrameOverWs` now makes one attempt and translates failures into the vocabulary that policy already speaks — a transport failure becomes a `WorkflowWorldError` with `code: 'TRANSPORT'`, exactly as `utils.ts` does for a failed `fetch`, and `isRetryableEventPostError` gains one clause keyed on that code. `WsTransportError` loses its `retryable` flag; its only consumer was the deleted loop. Two deliberate consequences. The code-keyed clause broadens HTTP in-process retry to `UND_ERR_CONNECT`, `UND_ERR_CLOSED` and `EAI_AGAIN`, which were in utils.ts's transient set but missing from event-retry.ts's — two hand-maintained lists collapsed into one semantic code. And the stale-token case (drain for auth expiry, refresh yields the same bearer) now gets two in-process attempts that cannot succeed, ~300ms before it falls through to queue redelivery; that is cheaper than keeping a WS-specific policy alive for one call site. `TIMEOUT` is deliberately not in the clause: utils.ts maps a caller-supplied `AbortError` onto it, and a cancelled write must not be re-issued. A status-less reply also stops being a bare `Error` — as one it failed `WorkflowWorldError.is()` and surfaced a protocol version skew as a USER_ERROR. It is now `code: 'PARSE_ERROR'`, the same code utils.ts uses for an unreadable HTTP body, and for the same reason: the write may or may not have landed. **No waiter is left unanswerable.** An undecodable frame, the server's malformed-frame sentinel (`reqId: -1`) and a non-numeric `reqId` were logged and dropped. None can be correlated by construction, so the request that provoked them stayed in `pending` with nothing in existence able to settle it — freed only by the server's own drain (~680s from connect), typically past the invocation's `maxDuration`. Each now fails the connection: every waiter learns why, and the socket is replaced. A reply for an id nobody is waiting on stays log-and-drop, deliberately — that request already settled, so nothing is orphaned, and failing the socket would punish healthy in-flight writes. A per-request deadline backs that up for whatever is left, including a server that accepts a frame and never answers it. Same knob as the HTTP path (`WORKFLOW_REQUEST_TIMEOUT_MS`, 60s), whose doc comment already describes this exact hang-to-SIGTERM pathology. One existing idle-teardown test needed the deadline raised: the idle window and the default deadline are both 60s, so a request could not outlive the former without also outliving the latter. The test is about `inFlight > 0` suppressing the teardown, so it now sets the deadline out of the way. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * Open the ws socket when the invocation starts, not on its first write Lazily connecting bills the whole handshake — an upgrade round-trip plus the OIDC token mint that rides it — to whichever event a fresh invocation writes first. When that is a `step_started` issued as the step body is already running, the event's server-recorded timestamp lands later than the work it describes: the step looks shorter than it was. That is the shape of the e2e timing failure on this branch, where a 9s step measured 6.5s from `getStepMetadata().stepStartedAt`. The queue handler is the earliest point that knows the run id, and a message delivered for a run means writes are coming, so `warmWsEventsTransport` starts the handshake there. By the first write it is done or in flight, and the write just uses it. Nothing about it is load-bearing: - It doesn't await, and can't fail the handler. A warm that fails logs and stops — a never-opened first connect is precisely the case `connect`'s close handler already declines to retry, so no backoff loop starts for a run that may never write. The first real write connects as it would have anyway, carrying the shared retry policy. - No-op unless `WORKFLOW_EVENTS_TRANSPORT=ws`, and no-op for the api-workflow proxy World, which can't serve an upgrade at all — the same fallback the write path takes. - Warming arms the idle timer as if a request had settled, so an invocation that warms and never writes (a health probe carrying the run id it is about to create) releases its socket on the usual 60s rather than stranding it. The socket is not `unref`'d, so a stranded one would hold this process and a server invocation open. Also closes a race that warming makes reachable: `close()` can only drop the connection it can see, so a release landing mid-handshake left the socket to install itself afterwards onto a transport already evicted from the cache, which nothing would then ever close. The `open` handler now declines to adopt a socket whose transport was released while it was connecting. This was already reachable via the eager reconnect path, just much harder to hit. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * changeset: just the env var Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * inline the ws-accelerator predicate at its only call site Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * refactor(world-vercel): trim ws-transport comments Comments were 47% of the file. Cut the historical narration, the restatements of adjacent code, and the repeated rationale (the `unref` reasoning appeared four times, per-connection reqId three), keeping the non-obvious facts: `ws.send()` reports failure via callback instead of throwing, reqId is per-connection so `pending` must be too, the unknown-reqId case is deliberately non-fatal, the auth_expiry same-token bail-out, and why the idle timeout exists at all. No code changes. Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * own transport selection in the transport module `events-v4.ts` was assembling the WS transport itself: reading the opt-in flag, resolving the URL, deciding which Worlds can use a socket, minting the per-connection header thunk, and holding the two once-per-process log latches. None of that is about turning an event into a frame, which is what the rest of that file does. Move it next to the socket it configures — `events-v4.ts` now consumes one seam (`resolveWsTransport`) plus the gate, and `queue.ts` gets `warmWsEventsTransport` from the module that owns the warm. `headersToRecord` now lives in `http-core.ts` because both callers need it and neither may import the other: `events-v4` already depends on the transport, so the reverse edge would be a cycle. Test fallout, and the reason the move is worth it: `events-v4-ws.test.ts` mocked `getWsEventsTransport` to observe the resolve step, which no longer intercepts anything now that the call is intra-module — an ESM mock replaces a module's exports, not its own call sites. That mock's tests were only ever about selection, so they move to `ws-transport.test.ts`, where the real selection code runs against the existing fake-socket harness instead of a stub. `resetWsEventsTransportsForTest` clears the log latches so the once-per-process assertions don't depend on test order. What stays behind mocks `resolveWsTransport` and covers what that file is actually for: reply frame in, `Response`-shaped result out — including the null-resolve fallback to HTTP, which nothing covered before. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * import `ws` statically The lazy `import('ws')` was there to keep the package off the module-init path of deployments that never opt in — `events-v4.ts` imports this module unconditionally, since the transport gate is a runtime branch. Measured, that buys ~17ms: `require('ws')` is 16.5-18.0ms cold, 13 modules, and neither `bufferutil` nor `utf-8-validate` loads (optional peers, absent by default). Bundle size is identical either way — webpack and Rollup both statically follow a dynamic `import()`, which is why the externals in `@workflow/builders` are unaffected by this change. For 17ms it cost a memoized promise, an inlined `WS_READY_STATE_OPEN` (so a readyState check wouldn't force the module to load just to read a constant off the constructor), and a whole test file — `ws-transport-lazy.test.ts` had to live alone, because vitest caches a `vi.mock` factory result for the lifetime of a module registry, so only a file that connects exactly once can observe the laziness at all. It also skewed the thing this branch exists to measure. The import lands inside the first connect, so on a warm container it is billed to whichever event write opens the socket, inflating the timestamp of the step it labels — the same distortion the queue pre-warm was added to remove. Also drops `WS_READY_STATE_OPEN` in favour of `WebSocket.OPEN`, now that reading it is free. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * tighten the comments on the ws transport Comments only — no code changes in this commit. Cuts ~150 lines of prose across the WS additions. The rule applied: keep the design factors a future reader needs (why the connection is scoped to a run, why a bad reply takes the socket down, why the accelerators are externalized unconditionally, why `TIMEOUT` is excluded from the `TRANSPORT` classification) and drop the narrative of how the code got here — which revision did what, what an earlier attempt got wrong, what was measured on the way. That history lives in the PR and the git log, where it doesn't have to be re-read on every visit to the file. Biggest reductions: the retry essay above `postEventFrameOverWs` (30 lines to 11), the flag's OTEL-gap note (34 to 13), the OIDC refresh explainer (26 to 14), the accelerator rationale in `@workflow/builders` (26 to 14), and the conformance suite's header (28 to 17). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * inject W3C trace context on the ws upgrade Frames carry no headers, so the upgrade is the only place this transport can propagate context; the server parents a run's event spans to whichever invocation opened the socket. Covered in trace-propagation.test.ts, both with and without an active span. Splits the opt-in gate into an import-free ws-transport-enabled.ts so callers can answer it without loading this module (used by the next commit). Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * load the ws transport module only when it is enabled Both call sites read the gate from the import-free module and dynamically import ws-transport.js behind a true result, so a deployment on the HTTP default never pays ws's ~17ms of module init. The queue pre-warm absorbs it for one that opted in, keeping it off the first event write. Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * document WORKFLOW_EVENTS_TRANSPORT as experimental Names the instrumentation gap (no client span per write) and the proxy path where the variable is ignored. Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * correct why the ws accelerators are externalized No bundler fails the build on the unresolvable require — verified against Rollup 4.62. webpack half-bundles the native module and Vite substitutes a stub that makes the require succeed; both leave bufferUtil.mask undefined and throw only once a frame reaches the native masker at 48 bytes, which every CBOR event frame does. Same claim was repeated in the rollup plugin and its test. Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * trim the WORKFLOW_EVENTS_TRANSPORT docs to user level Mirrors the other Vercel World env vars: same facts on both pages, each in its page's format. The instrumentation and socket-lifetime detail belongs in the code, not in a user-facing reference. Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * cover the vite bundler in the ws transport lane Vite substitutes a stub for ws's absent native accelerators rather than failing the require, so nothing catches it until a masked frame reaches 48 bytes — and this job's three existing lanes are esbuild, turbopack and nitro. Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * claim only what is measured about rollup and the ws accelerators The rationale asserted plain Rollup was "safe by accident" via a mechanism only ever observed in a minimal repro. Nitro traces and externalizes `ws` in a production build, so the bundled path is not reached there at all. Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * Give the events socket an explicit lifetime instead of an idle timer `openWsChannel` / `closeWsChannel` bracket one invocation of the flow route, and are the only calls anywhere that create a channel. Writes ask `resolveWsTransport` whether one is open — a lookup now, never a create — and take pooled HTTP when it says no. That removes the reason the idle timeout existed. A lazily-created socket has no owner, so a timer was the only thing able to end it, and the socket is not `unref`'d: the process could not exit, and a server invocation stayed pinned, for the full window past the last write. It also settles `run_created`. The trigger path opens no channel, so a lone write no longer pays for a handshake it cannot amortize — `start()` runs in an arbitrary request handler with no boundary the SDK can see. Refcounted rather than a flag: inline step executions ride the flow topic on per-step topics, so a run's steps can be concurrent invocations in one instance sharing the channel, and the first to finish must not cut the others short. A failed connect closes the channel so the invocation's writes fall back to HTTP instead of each paying its own doomed handshake. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * Name the one reply header the WS path does not map The server copies six headers into an `event_ack`'s meta and this record maps five. The sixth, `X-API-Deprecated`, is inert today — the v4 route's middleware chain has no deprecation middleware to set it — but the record is the only header source a WS reply has, so an unmapped key is gone rather than merely unread, which is not true of the `Response` the HTTP path returns. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * docs: note that WORKFLOW_EVENTS_TRANSPORT=ws is ignored on the proxy path The api-workflow proxy is an HTTP-only REST gateway and does not forward a WebSocket upgrade, so a World configured with projectConfig keeps writing events over HTTP regardless of the setting. Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * ci: gate the ws-transport e2e lanes on a label Three real `vercel deploy`s per run is too much to charge every unrelated PR in the repo for a transport that is off by default. PRs opt in with `ws-transport-test` (or `workflow-server-test`, which already exists to test the half of this the protocol lives in); main keeps the signal on every commit. The required aggregate has to allow the lane to be skipped in that case, so its status is asserted only when the lane was actually supposed to run. Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * chore: regenerate pnpm-lock against current main main resolved `ws` to 8.20.0 as a transitive peer; this branch adds it as a direct dependency of world-vercel and floats it forward, which rewrites every `openai@x(ws@y)` peer key in the lockfile. Merging main textually combined the two, leaving those keys pointing at a `ws` entry the merged file no longer had — `--frozen-lockfile` then failed with ERR_PNPM_LOCKFILE_MISSING_DEPENDENCY on the PR's merge ref. Regenerated from main's lockfile so ours is a minimal delta on top of it. Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * fix(world-vercel): align ws on the version main already resolves The lockfile broke on the PR's merge ref, not on this branch's head: main resolves ws@8.20.0 as a transitive peer, and a `^8.21.1` direct dep here floated it forward, rewriting all 73 `(ws@8.20.0)` peer keys. Git merged the two lockfiles without a conflict but left main-side keys pointing at a ws entry the merged file no longer had, so `--frozen-lockfile` failed with ERR_PNPM_LOCKFILE_MISSING_DEPENDENCY. `^8.20.0` resolves to the copy main already has, so the lockfile delta is the two importer entries instead of a repo-wide rewrite that re-breaks every time main moves. Also keeps one ws in the store rather than two. Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * Bind the channel release to the instance it claimed closeWsChannel resolved the transport by URL, but the refcount lives on the instance. A channel is evicted from the map as soon as it closes — a refused upgrade does that on the connect path — so the next opener for the same run registers a different instance under the same URL, and the first invocation's close then decremented that one instead. It dropped a socket a live invocation was still writing over, and for the event types EVENT_RETRY_ELIGIBILITY marks non-retryable there is no second attempt to carry the in-flight write over HTTP. openWsChannel now returns an idempotent release closed over the transport it incremented, and queue.ts holds that instead of re-resolving the run. The close awaits the open's own promise, so it also can no longer land ahead of the claim it releases. Also names the scope of the connect-failure de-opt: it covers the handshake only, so a channel that connects and then fails every write keeps taking the WS path. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * Decode a transport result, not a Response Main extracted the v4 POST decode into a helper typed `Response` while this branch narrowed the POST result to `FrameResponseLike`, because the WS branch synthesizes its result rather than holding a real `Response`. The two merge without a textual conflict and then fail to typecheck. Widen the helper: it reads only the two members `FrameResponseLike` declares, and a `Response` still satisfies them, so the HTTP call sites are unchanged. Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * Re-run CI Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * Re-run CI Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * Reconcile the WS transport with main's v4 POST rework main moved the materialized POST result off the `x-wf-*` response headers and onto a typed CBOR body, and added a second response shape: two callers now POST with `Accept: application/vnd.workflow.v4-frames` and read back a sentinel-terminated sequence of frames. A frame stream has no representation in a protocol that pairs one reply frame with one request frame, so the WS switch moves off the shared poster and onto `createWorkflowRunEventV4` alone — the materialized write, which is the hot per-step path this branch exists to shorten. `run_started` and the `hook_received` preload stay on HTTP. `decodeCreateEventResponse` takes `FrameResponseLike` rather than `Response` because the WS branch has none to hand over; a real `Response` satisfies the interface, so the HTTP callers are unchanged. The ids now come out of the CBOR body, so `replyMetaToHeaderRecord` no longer maps any `x-wf-*` name — only the two headers `errorFromV4Response` reads. Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * Re-run CI Resample the WS-arm sleepingWorkflow failure: it has now recurred on a second axis (nextjs-turbopack, 7709ms; previously vite, 7570ms), so the arm needs more samples before the skew can be called WS-specific or repo-wide flake. Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * Re-run CI Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * Re-run CI Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * blank * blank --------- Co-authored-by: vercel[bot] <35613825+vercel[bot]@users.noreply.github.com> |
||
|
Nathan Rajlich
|
32ac8e73fd |
Fix Biome lint violations and add Biome CI check (#3222)
* Fix Biome lint violations and add Biome CI check Biome was not configured to respect .gitignore, so ~92% of the 13,355 reported diagnostics came from gitignored build artifacts. Enable VCS integration (useIgnoreFile), apply safe auto-fixes across the repo, fix the remaining mechanical errors by hand, downgrade judgment-call a11y / dangerouslySetInnerHTML rules to warnings, and add a 'biome ci' job to the Lint workflow so violations block PRs going forward. * Use an empty changeset (no behavior change, no release needed) |
||
|
Shalabh Chaturvedi
|
8bda7cef79 |
[benchmarks] Split STSO by inline vs queue-hop steps, add distribution diffs vs main (#3213)
* Split STSO by inline vs queue-hop steps, add distribution diffs vs main
The sequential-steps benchmark's STSO metric mixed two unrelated
phenomena: gaps between steps running back-to-back in the same warm
process, and gaps across an invocation boundary (queue dispatch, client
reinit, event-log replay), which cost ~10x more. The old step-index
windows (1-20 / 101-120 / 1001-1020) sampled 19 gaps each and captured
neither cleanly: whether a boundary happened to land inside a window
moved that window's P99 by hundreds of percent, which is most of the
run-to-run variance the benchmark comment was reporting.
The workflow now tags each step with whether it was the first step body
executed in its process ('queue-hop') or a later one in the same warm
process ('inline') via a process-global, so the split is ground truth
rather than inferred from step index or trace timestamps. STSO is
reported as two rows over *every* gap in the run instead of three
sampled windows. No targets on the new rows — the old ones described the
index-bucketed grouping.
computeStats now keeps the full sorted sample array alongside the
percentiles, and the comment renders a histogram + cumulative-time diff
against `main` under the table, one per STSO kind. Percentiles alone
hide how many samples moved and by how much, which is exactly where the
variance lives. Inline rows use a fixed 50ms bin width (the adaptive
width is coarse enough to hide structure inside that cluster); queue-hop
rows keep the adaptive width. Negative gaps (clock skew between two step
bodies' clocks) get their own bucket rather than being counted with the
slow tail.
Raw samples are stripped from the comment's embedded data block — ~1000
per run would exceed GitHub's comment size limit within a couple of
history entries — so the histogram renders for the current run only,
while collapsed history keeps its tables. Until this lands on `main` no
baseline has raw samples, so the section renders this run's distribution
as a single series.
Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com>
* Clarify what stripping raw samples from the data block does not affect
Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com>
* Drop the bucket tables; fold counts and deltas into the histogram bars
Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com>
* Collapse the STSO distribution section into a dropdown
Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com>
* Fix footer assertion after the dropdown wording change
Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com>
---------
Co-authored-by: vercel[bot] <35613825+vercel[bot]@users.noreply.github.com>
|
||
|
Peter Wielander
|
62c01d94b0 | [e2e] Report partial results when the event-log race repro is cut short (#3148) | ||
|
Peter Wielander
|
04e5ec9873 | [e2e] Rebuild the event-log corruption repro around step-count divergence (#3147) | ||
|
Peter Wielander
|
599250771d | [benchmarks/ci] SO payload variants + restructured E2E Test Results comment (#3080) | ||
|
Peter Wielander
|
604aecb021 | [benchmarks] Add SO (stream overhead) scenario and polish test result comment (#3077) | ||
|
Peter Wielander
|
0bc22c8e9b | [ci] Benchmark comment: Best column + best/p75/p99 deltas (drop Avg/P10) (#3005) | ||
|
Peter Wielander
|
d53b055a2b | [ci] Run benchmarks in-deployment to avoid proxy overhead (#2967) | ||
|
Peter Wielander
|
8977666479 | [ci] Benchmark comment: show avg-latency deltas vs main (#2842) | ||
|
Peter Wielander
|
da4e0995b0 | [ci] Overhaul performance benchmarks: focused metrics + sticky PR comment (#2820) | ||
|
Pranay Prakash
|
cb181392b9 |
feat(cli): print run deep links with --url, fix dashboard route (#2467)
Add a `--url` flag to `inspect`/`web` that prints a run's observability dashboard deep link to stdout and exits — no browser, no local server — so scripts and agents can share a link instead of opening a UI. Fix the Vercel dashboard URL to the current `…/workflows/runs/<id>?environment=<env>` route (drop the legacy `/observability` segment) and respect `--env`. Apply the same route fix to the e2e helpers, CI aggregation scripts, and the nextjs-turbopack workbench. Document deep-linking in the workflow skill and observability docs. Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
Karthik Kalyan
|
67dcb0e355 | Prevent peer dependency-only major bumps (#2437) | ||
|
Peter Wielander
|
c000462502 | Capture Vercel runtime logs when e2e Vercel Prod lanes fail (#2356) | ||
|
Peter Wielander
|
4e8a9657c9 | Fix e2e failure reporting under vitest 4 and preserve fetch error causes (#2355) | ||
|
Peter Wielander
|
ae3c833acd | [e2e] Improve error labeling in event-log-race-repro CI job (#2190) | ||
|
Peter Wielander
|
625fab46c8 |
[e2e] Add event-log-race-repro label for triggering CI stress-test (#2159)
|
||
|
JJ Kasper
|
00a011dee4 |
Add stable Next.js eager and lazy test coverage (#1747)
* Add stable Next.js eager and lazy test coverage * Address PR review feedback * Fix eager Next step route builds * Fix eager Next manifest refreshes * Fix eager Next e2e stack assertions * Externalize native step bundle bindings * Lazy load Vercel world runtime * Fix Next dev step sourcemap assertions * Consolidate eager build changesets * Fix Vercel world tracing in Next deployments * Externalize Vercel world in Next builds * Fix webpack tracing for Vercel world deps * Fix eager workflow route bundling * Rely on Next server externals |
||
|
Pranay Prakash
|
cd4abd80fe |
test: improve e2e test failure diagnostics (#1426)
* test: improve e2e test failure diagnostics with run context and GitHub annotations When e2e tests fail, automatically dump workflow run diagnostics (status, input/output, error details, event timeline, dashboard link) to the CI logs. Emit GitHub Actions annotations that surface on PR file diffs. Fix collectedRunIds which was declared but never populated, enabling observability links in the PR comment. Enrich the aggregation script to include run IDs and dashboard URLs for failed tests. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: increase diagnostics hook timeout and fix flaky Vercel Prod tests - Increase onTestFailed hook timeout to 30s (default was 10s) so diagnostics can fetch run data even after slow test timeouts - parallelSleepWorkflow: increase elapsed threshold from 10s to 25s to accommodate Vercel cold start latency - webhookWorkflow: increase hook polling deadline from 30s to 60s and test timeout from 60s to 120s for slow Vercel webhook registration - readableStreamWorkflow: stop reading once expected content is received instead of waiting for stream close (which can hang on Vercel), and increase test timeout to 120s Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: emit ::error annotations via process.stdout.write to bypass vitest ANSI prefix Vitest's console interceptor prepends ANSI escape codes to console.log output, which prevents GitHub Actions from parsing ::error workflow commands. Use process.stdout.write() directly to ensure clean output. Also enhance the custom reporter to emit annotations in onFinished (which runs after vitest output is complete) as a reliable fallback, and enrich failure data from the diagnostics sidecar. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: only show observability links for vercel-prod test failures Community world and local tests don't run on Vercel's backend, so dashboard links are meaningless for those categories. Previously, test name collisions across sidecar files could cause community test failures to show Vercel dashboard URLs from vercel-prod runs. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: link annotations to test files instead of symlinked workflow sources The workflow source files in workbench/ are symlinks that GitHub can't resolve, causing annotations to show raw paths like #L0 instead of linking to code. Now: - utils.ts: omit file= from onTestFailed annotations (just show title) - github-reporter.ts: use the actual test file path (e.g. packages/core/e2e/e2e.test.ts) which GitHub can resolve Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com> |
||
|
Pranay Prakash
|
8a146a5bb1 |
docs: revamp World documentation pages (#763)
* docs: revamp World documentation pages with new structure and design
- Add dedicated world detail pages at /worlds/[id] with MDX-driven content
- Create WorldDetailHero, WorldDetailToc, WorldTestingPerformance components
- Add benchmark history charts and test summary links
- Rewrite local-world, postgres-world, and vercel-world MDX with cleaner structure
- Use h3 headings for configuration options instead of tables/accordions
- Add WorldDataProvider context for passing world data to MDX components
- Add example field to worlds-manifest for linking to example repos
- Update worlds index page design
* fix: update worlds page meta title and description
* feat(docs): improve globe backdrop responsiveness and positioning
- Globe now scales with container width up to 1200px max
- Increased opacity from 20% to 30% for better visibility
- Positioned globe 30% down to show north pole behind header
- Globe component now auto-sizes based on container width
* fix(docs): add light mode support for globe backdrop
* feat: improve world page metadata and add dynamic OG images
- Update title format to '{Name} World | Workflow DevKit'
- Update worlds index page title to 'Worlds | Workflow DevKit'
- Improve world descriptions in worlds-manifest.json for better SEO
- Add dynamic OG image generation at /worlds/[id]/og
- Add openGraph and twitter metadata to world detail pages
* fix: use opengraph-image.tsx file convention for dynamic OG images
- Replace route-based og/route.tsx with opengraph-image.tsx file convention
- Remove runtime = 'edge' to allow generateStaticParams for static generation
- Simplify page.tsx metadata (Next.js auto-detects opengraph-image.tsx)
* docs: improve worlds page UI and restructure deploying section
- Fix globe visibility in light/dark mode
- Fix tooltip positioning on mobile breakpoints
- Remove redundant Documentation button for community worlds
- Add missing pages to Foundations index (streaming, serialization, idempotency)
- Restructure deploying section with new index and building-a-world guide
- Restore original MDX content for official worlds (local, vercel, postgres)
- Auto-expand sidebar folders when child page is active
* docs: simplify worlds page footer sections
* docs: move last updated section under worlds grid
* fix(docs): fix broken links and skip typecheck for interface definitions
- Fix /docs/deploying/world link to /docs/deploying/building-a-world
- Fix /docs/deploying#what-are-worlds invalid anchor to /docs/deploying
- Add @skip-typecheck markers to interface definition code blocks
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* fix(docs): address PR review feedback
- benchmark-history/route.ts: Add error handling for JSON parsing,
distinguish 404s from actual errors
- Globe.tsx: Fix hydration mismatch by deferring theme-dependent
rendering until component is mounted
- sidebar.tsx: Improve path check robustness for active child detection
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
|
||
|
Pranay Prakash
|
ad7d14e6e1 | Fix asset upload in CI | ||
|
Pranay Prakash
|
ee4fff6814 |
benchmarking: make steps simulate real work (+ misc improvements) (#565)
* perf: add 5s delay to benchmark steps to simulate real work * perf: add realistic workloads to benchmark steps Add realistic workloads to benchmark step functions: - doWork() - 1 second delay to simulate real computation - stressTestStep() - 1 second delay to simulate real computation - genBenchStream() - generates ~5KB of data in 50 chunks - transformStream() - uppercases stream content (renamed from doubleNumbers) Add "Slurp Time" metric to stream benchmarks measuring time from first byte to complete stream consumption, complementing TTFB. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * fix(e2e): increase dev test timeouts for Windows Windows file watching and rebuilding is slower than macOS/Linux, causing the 10s timeouts to fail. Increased to 30s to accommodate. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * fix(ci): treat community world failures as warnings, not errors Community world tests/benchmarks are now non-blocking: - Removed from has_failures check in both tests.yml and benchmarks.yml - Added separate has_warnings output for community failures - Split PR comment notices: ❌ for failures, ⚠️ for community warnings 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * fix: apply PR review suggestions - Fix chunk size calculation: chunkSize - 11 for ~100 bytes per chunk - Remove redundant metadata check (always truthy) 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
Pranay Prakash
|
54ba1888cd |
fix: compare benchmarks against PR base branch instead of main (#560)
* fix: compare benchmarks against PR base branch instead of main - Use github.event.pull_request.base.ref instead of hardcoded main - Remove search_artifacts: true to ensure most recent baseline is used - For stacked PRs, this compares against the parent PR's baseline 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * fix: group failed e2e tests by category and app in summary Instead of listing each failed test as a separate item, group them by: 1. Category (world): e.g., "Community Worlds", "Vercel Production" 2. App (framework): e.g., "mongodb", "turso", "nextjs-turbopack" This makes the summary much more readable when there are many failures. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * fix: ensure local E2E tests always produce JSON output - Add 'fastify' to app detection list in aggregate-e2e-results.js - Change && to ; so e2e tests run even if dev.test.ts fails - This ensures local-dev, local-prod, and local-postgres categories appear in the E2E summary comment 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * fix: ensure local E2E tests always produce JSON output - Add 'fastify' to app detection list in aggregate-e2e-results.js - Change && to ; so e2e tests run even if dev.test.ts fails - This ensures local-dev, local-prod, and local-postgres categories appear in the E2E summary comment 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * feat: publish CI results to GitHub Pages for docs - Add generate-docs-data.js script to create JSON summaries from CI artifacts - Add publish-results job to tests.yml and benchmarks.yml workflows - Update docs/lib/worlds-data.ts to fetch from GitHub Pages URLs - Results published to https://vercel.github.io/workflow/ci/ This allows the docs worlds page to display actual test/benchmark results without requiring a GITHUB_TOKEN. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * fix: correct outputFile path for local E2E test artifacts The --outputFile path was using ../../ which placed files outside the repo because pnpm run test:e2e executes from workspace root, not from the cd'd workbench directory. This prevented local-dev, local-prod, and local-postgres test results from being uploaded as artifacts. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * fix: show green checkmark for skipped tests instead of warning Skipped tests are intentional and shouldn't show as warnings in the E2E test summary comments. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * feat: use collapsible sections in benchmark PR comment Wrap each benchmark, stream benchmarks section, and summary tables in <details> toggles to make the PR comment more compact and readable. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * feat: add Vercel observability links to benchmark PR comments - Store runId in benchmark timing data - Add project-slug to Vercel benchmark matrix - Pass WORKFLOW_VERCEL_PROJECT_SLUG env var to benchmarks - Store Vercel metadata (teamSlug, projectSlug, environment) in timing files - Generate observability deep links for each Vercel world benchmark - Show observability links below Production (Vercel) tables 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * fix: use correct Vercel project slugs for observability links - nextjs-turbopack → example-nextjs-workflow-turbopack - nitro-v3 → workbench-nitro-workflow 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
Pranay Prakash
|
5dd15452cd |
Test and benchmark community worlds against e2e tests (#482)
* Add new github workflow * Enable pull_request trigger for community worlds workflow 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * Add community worlds manifest and generation scripts - Add community-worlds.json manifest as single source of truth - Add scripts/generate-community-worlds-workflow.mjs to generate CI workflow - Add scripts/generate-community-worlds-docs.mjs to generate docs section - Update aggregate-benchmarks.js to load community worlds dynamically - Add pnpm generate:community-worlds script - Update docs/deploying/world/index.mdx with community worlds The manifest-based approach allows: - E2E tests to be auto-generated from the manifest - Benchmark aggregation to include community worlds - Docs to stay in sync with tested worlds 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * Fix YAML syntax error - quote strings starting with @ The @ symbol has special meaning in YAML, so package names like @workflow-worlds/turso need to be quoted. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * Fix Redis health-cmd quoting, remove unpublished starter world - Quote health-cmd when it contains spaces (fixes Docker arg parsing) - Remove @workflow-worlds/starter as it's not published to npm 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * Add benchmarks and summary job for community worlds - Add build job to share artifacts between benchmark jobs - Add benchmark jobs for Turso, MongoDB, and Redis worlds - Update summary job to show both E2E and benchmark status matrix - Add left border/indent to sidebar child items for visual hierarchy - Update workflow generator to support benchmark generation 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * Reuse build artifacts for E2E tests E2E jobs now depend on the shared build job and download artifacts instead of rebuilding packages from scratch. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * Add Worlds Ecosystem dashboard to docs - Create worlds-manifest.json with official and community worlds - Add aggregate-worlds-data.mjs script for processing E2E and benchmark results - Create WorldsDashboard, WorldCard, and BenchmarkChart components - Add /docs/worlds page showing compatibility status and performance - Include sample data for development The dashboard shows: - E2E test progress per world (pass/fail/skip counts) - Benchmark performance comparison across all worlds - Filter by official vs community worlds 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * Fix CI to output JSON test results and add Jazz world - Update workflow generator to output JSON test results from vitest - Upload E2E results as artifacts for parsing in summary job - Summary job now shows actual pass/fail/skip counts per world - Add Jazz world to worlds-manifest.json (requires external credentials) - Add update-worlds-status.yml workflow to auto-update dashboard data - Update TypeScript types to support null lastRun and metrics 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * Refactor community worlds to use reusable workflows Instead of a generated workflow file, integrate community world testing directly into tests.yml and benchmarks.yml using reusable workflows. - Add reusable workflows for E2E tests: e2e-community-world.yml (no services), e2e-community-world-mongodb.yml, e2e-community-world-redis.yml - Add reusable workflows for benchmarks: benchmark-community-world.yml, benchmark-community-world-mongodb.yml, benchmark-community-world-redis.yml - Update tests.yml to call reusable workflows for Turso, MongoDB, Redis - Update benchmarks.yml to include community world benchmarks in summary - Delete generated community-worlds.yml and generator script This approach: - Inherits proper Rust/SWC setup from the main workflows - Keeps all CI in the established patterns - Makes adding new community worlds straightforward 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * Fix benchmark timing file naming for community worlds Add WORKFLOW_BENCH_BACKEND env var support to bench.bench.ts so community world benchmarks generate timing files with the correct backend suffix (e.g., bench-timings-nextjs-turbopack-turso.json instead of -local.json). 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * Add @workflow-worlds/starter to community worlds test matrix 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * Unify worlds manifest and add dynamic GitHub API fetching - Merge community-worlds.json into worlds-manifest.json with type field - Add server-side data fetching from GitHub API for worlds dashboard - Remove static worlds-status.json, fetch CI artifacts dynamically - Update all references to use unified manifest format - Remove obsolete community-worlds.yml and update-worlds-status.yml workflows 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * Skip community worlds for non-nextjs-turbopack in benchmark summary Community worlds only run against nextjs-turbopack, so hide the "missing" rows for Express and Nitro frameworks in the benchmark comparison tables. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * Fix stream benchmark detection to check for actual TTFB data The previous check `!== null` incorrectly returned true for undefined, causing all benchmarks to show TTFB columns. Now explicitly checks for a number type. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * Hide Worlds Ecosystem page from sidebar The page is still accessible via direct link at /docs/worlds but won't appear in the navigation until it's been further iterated on. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * Fix review comments: trailing newline and division by zero - Add trailing newline when replacing Community Worlds section in docs - Fix division by zero in WorldCard benchmark calculation when metrics is empty 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * Consolidate community world workflows with service-type parameter - Create setup-workflow-dev composite action for common setup steps - Add service-type input to benchmark-community-world.yml and e2e-community-world.yml - Use conditional job execution (if: inputs.service-type == 'mongodb') to handle different services - Update benchmarks.yml and tests.yml to pass service-type parameter - Delete redundant workflow files: - benchmark-community-world-mongodb.yml - benchmark-community-world-redis.yml - e2e-community-world-mongodb.yml - e2e-community-world-redis.yml Reduces workflow files from 11 to 7 and eliminates ~500 lines of duplicated YAML. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * Generate community world test matrix from worlds-manifest.json Replace hardcoded community world jobs with dynamic matrix generation using scripts/create-community-worlds-matrix.mjs. This allows adding/removing community worlds by editing the manifest instead of multiple workflow files. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * Apply suggestion from @vercel[bot] Co-authored-by: vercel[bot] <35613825+vercel[bot]@users.noreply.github.com> * Add Samples column and separate local/production benchmarks - Add Samples column to all benchmark tables showing iteration count - Separate benchmark results into Local Development and Production sections - Add explanatory context for each section (localhost vs Vercel deployment) - Add GitHub action step summaries to e2e community world tests - Create aggregate-e2e-results.js script for parsing vitest JSON output 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * Remove obsolete generate-community-worlds-docs script The Worlds Ecosystem page now fetches from worlds-manifest.json at runtime, making this script unnecessary. The npm script also referenced a non-existent workflow generator script. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * Maximize composite action usage and reorganize benchmark output - Update setup-workflow-dev composite action with optional Rust, install-dependencies, and install-args inputs - Update tests.yml to use composite action in unit, e2e-vercel-prod, getTestMatrix, e2e-local-*, and getCommunityWorldsMatrix jobs - Update benchmarks.yml to use composite action in build, benchmark-local, benchmark-postgres, benchmark-vercel, and getCommunityWorldsMatrix jobs - Reorganize benchmark output to group by benchmark test with local/production tables within each benchmark - Remove invalid $schema reference from worlds-manifest.json 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * Add beads stealth mode stuff (for personal claude memory - will remove stealth if people want) * Add E2E test results PR comment summary - Add pr-comment-start job to create/update PR comment when tests start - Add artifact uploads to all e2e test jobs (vercel-prod, local-dev, local-prod, local-postgres, windows) - Update e2e-community-world.yml with consistent artifact naming (e2e-community-*) - Add summary job to aggregate all e2e results and update PR comment - Extend aggregate-e2e-results.js with --mode aggregate for multi-job PR summary - Group results by category (Vercel Production, Local Development, etc.) 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * Add step summaries to all e2e test jobs Add "Generate E2E summary" step to each individual e2e job: - e2e-vercel-prod - e2e-local-dev - e2e-local-prod - e2e-local-postgres - e2e-windows Each job now outputs pass/fail/skip counts to GITHUB_STEP_SUMMARY. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * Consolidate community world workflows to single job Replace 3 mutually exclusive jobs (e2e/e2e-mongodb/e2e-redis) with a single job that starts services via docker run when needed. This eliminates the skipped job entries that appear in the GitHub Actions UI. - Use conditional docker run steps instead of services: block - Add health check loops to wait for service readiness - Add cleanup step to stop containers 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * Fix extractWorldId to handle community world artifact naming Add handling for `e2e-results-community-{world}` pattern so community world test results are properly extracted (e.g., `e2e-results-community-turso` now correctly extracts `turso` instead of `community-turso`). 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> --------- Co-authored-by: Claude <noreply@anthropic.com> Co-authored-by: vercel[bot] <35613825+vercel[bot]@users.noreply.github.com> |
||
|
Pranay Prakash
|
6e8e828252 |
Add stream benchmarks and slightly improve the benchmarking code (#470)
* Add stream benchmarks and cnealup for benchmark code * 10m delay for local world race * potential fix for first bute" * Improve sticky comment stuff * local world: ignore controller close errors * show missing data in benchmark comment * nitrpicks * Improve leaderboard * Add benchmark comparisons against main * changeset |
||
|
Pranay Prakash
|
a8f48c5a08 | add benchmarking (#460) |