mirror of
https://github.com/vercel/workflow.git
synced 2026-09-14 19:59:43 +08:00
pgp/dispatch-skip-queue-owned
238 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Nathan Colosimo
|
c29200fac5 |
docs(ai): clean up WorkflowAgent docs and examples (#3891)
Co-authored-by: Peter Wielander <peter.wielander@vercel.com> |
||
|
Alex Langenfeld
|
d864efb07b |
test: tighten CI health signals (#4107)
## Summary & Motivation - Manifest coverage now declares an entry for every matrix app, uses real Vitest skips instead of silent early returns, and fails when a targeted app's manifest is missing, unknown, or unparseable. - Retries are scoped to deployment e2e runs (`DEPLOYMENT_URL` set), so a flaky unit or integration test can no longer be hidden by a second attempt. - The stop-workflow cookbook parks on a sleep between iterations, giving the hook an observable barrier to race instead of a fixed delay, and the AbortController hook test waits on queue state rather than a 10ms timer. - The world-postgres direct-storage fixture drives its run to a terminal state so the conformance worker doesn't recover and replay an unregistered workflow. - Generated e2e result sidecars are ignored and the committed copies removed; they're CI artifacts, not fixtures. ## Test Plan Existing coverage runs in CI. With retries disabled: the cookbook agent suite passed 8/8, the two stop-workflow tests passed 10/10, and the AbortController hook replay test passed 25/25 under `CI=1`. The manifest suite skips 52 apps explicitly when nothing is built, and fails on unknown or missing targeted apps. The Docker-backed Postgres spec could not run locally (Testcontainers found no container runtime); `@workflow/world-postgres` typechecks. |
||
|
Pranay Prakash
|
0b1216ebe2 | (chore) Update Next.js to 16.3.4 in the workbench apps and @workflow/next (#4026) | ||
|
Peter Wielander
|
45a3072948 | [core] Fix the python e2e conformance suite after the retention merge (#4022) | ||
|
Peter Wielander
|
8a91d18d0d | [core] Add the wake-loop scenario to the event log race repro (#4017) | ||
|
Nathan Rajlich
|
b30ed49187 | [swc-playground] Update to Next.js v16.3.4 (#4018) | ||
|
Peter Wielander
|
61fb1f93bd | [core] Add a retention option to start() (#3787) | ||
|
Fantix King
|
9ffd37d551 | [e2e] Cover Python cancellable steps (#3930) | ||
|
Fantix King
|
144b6d7601 |
[e2e] Expand Python conformance coverage (#3495)
Also fixes the issue that specVersion 7 broke Python e2e test. |
||
|
Nathan Colosimo
|
5c4eef0a97 |
chore: upgrade to pnpm 11.24.0 (#3901)
* chore: upgrade to pnpm 12 * fix: support pnpm 12 in CI * fix: enable pnpm 12 on Vercel * refactor: simplify pnpm 12 setup * refactor: target pnpm 11.24.0 * refactor: let pnpm setup own CI installs * refactor: limit workspace Node versions * fix: complete pnpm 11 CI migration |
||
|
Peter Wielander
|
d9e0777eb8 | [core] Never write hook_received eagerly on the lazy resume path (#3794) | ||
|
Nathan Rajlich
|
e1e64e3de3 |
docs: apply Vercel technical writing standards (#3704)
* docs: apply Vercel technical writing standards Audit the complete documentation corpus, package READMEs, skills, and source TSDoc/comments against the vercel-technical-writing skill and style-rules.md. Normalize sentence-case headings without changing published anchors, remove prose em dashes and filler wording, improve active voice and self-contained phrasing, standardize product/brand capitalization, American English, list punctuation, units, and code fence languages, and preserve exact runtime strings/table placeholders. All executable code is unchanged. Modified skills have their metadata versions bumped. * docs: extend writing audit to repository Markdown Apply the same technical-writing rules to design documents, compiler specifications, workbench guides, package changelogs, and the remaining tracked Markdown outside the deployed docs corpus. Preserve historical meaning, commands, output literals, table placeholders, and heading anchors. * docs: exclude generated package changelogs from audit |
||
|
Peter Wielander
|
c431cc18fd | [e2e] Add blocked-branch scenario to the event-log race repro (#3696) | ||
|
Nathan Rajlich
|
5b5a926f88 |
fix(core): make step-argument serialization failures catchable in workflow code (#3675)
* fix(core): make step-argument serialization failures catchable in workflow code
A step whose arguments fail to serialize is now finalized by the
suspension handler as step_created + step_failed (mirroring a step-body
failure) instead of rejecting the whole suspension. The next replay —
forced in-process, since no step message is dispatched for the failed
step — rejects the step's promise with the SerializationError, so a
try/catch around the step call observes it. Uncaught, the error
propagates out of the workflow body and fails the run as a fatal
USER_ERROR immediately, instead of redelivering the orchestrator
message until max deliveries (49/48) as reported in production on v4.
* Serialize the step_failed error with the VM global; one-sentence changeset
Addresses review feedback: dehydrateStepError in
finalizeUnserializableStep now receives suspension.globalThis like every
other dehydration in this file. Error detection is realm-independent, so
the host-created SerializationError serializes identically, but VM-realm
values guest code threw into the cause chain are now detected by the
realm-sensitive reducers.
* Address review: QuickJS engine support, deferred-batch join, drain gate, placeholder marker, telemetry, docs
- QuickJS: dumpPendingOps now catches a step input's serialization
failure per-op, reframes it as a SerializationError with the same
framed message as dehydrateStepArguments, and surfaces it on the
pending op instead of failing the whole collection. The entrypoint's
dispatchPendingOps finalizes such steps as step_created (placeholder
input) + step_failed, excludes them from inline claims and queue
publishes, marks them handled, and raises the requeue signal so the
failure is observed even when the feed lags — mirroring the node:vm
engine, so both engines agree: catchable in workflow code, USER_ERROR
with the framed message when uncaught. Both step-argument e2e tests
now pass on WORKFLOW_VM=quickjs.
- runtime.ts: the failed-step replay path now joins
suspensionResult.deferredBatchWork before continuing, so a trailing
chunk commit or step-message publish rejection propagates instead of
being swallowed after ack; committed inline claims are documented as
deliberately handed to owned recovery.
- Terminal drain: finalization is gated on a stepDispatch target. The
drain caller has no replay to observe a finalization, so a completed
run no longer gains failed-step rows for an unawaited unserializable
step — the rethrown error is swallowed by the drain's catch,
preserving its pre-existing behavior.
- The placeholder input now carries a marker string ('[input
unavailable: step argument serialization failed]', shared via
runtime/unserializable-step.ts) so inspect/o11y don't render the
failed step as a genuine zero-argument call.
- New workflow.steps.failed_serialization span attribute on the
suspension span, so occurrence is measurable without log search.
- Docs: v5 serialization-failed error page documents where each
boundary's failure surfaces (catchable step failure vs run failure)
and the no-retry USER_ERROR semantics; foundations/errors-and-retries
gains a Serialization Failures section with the try/catch shape.
* Guard the finalization crash window; self-contained docs samples
- A crash or transient failure between finalization's two durable
writes leaves a lone placeholder step_created, and redelivery then
dispatches the step through normal crash recovery — previously
running user code with the placeholder arguments. The placeholder
now carries a structural flag on the input triple's top level (which
user code never controls, so no false positives), and the step
executor checks it after hydration: instead of running the body, it
throws the intended fatal SerializationError, completing the
interrupted finalization as step_failed. Applies to both engines
(they share the placeholder and the executor).
- Regression tests: executor fails a placeholder-input step without
running the body (and doesn't trip on a genuine argument equal to
the display marker); handleSuspension rejects for redelivery when
step_failed can't be written after step_created landed, leaving the
recoverable placeholder behind; mixed bad-step + large fan-out
returns the failure set alongside still-pending deferredBatchWork
whose rejection surfaces — the contract the runtime's failed-step
join (added previously) relies on.
- Docs: the two new code samples are now self-contained so the docs
code-sample typecheck passes.
|
||
|
Alex Langenfeld
|
8789f4529b |
[e2e] Host the abort-fetch slow endpoint inside the step (#3618)
The abort-fetch tests cancelled an in-flight fetch against external slow endpoints (postman-echo, httpbin /delay/10, tried in order). Those upstreams 5xx and return early from GH Actions runners often enough to be a recurring flake class - the tests were measuring the public internet instead of abort propagation - and heavier suite load (e.g. re-enabling e2e concurrency, #2083) makes both upstreams flake at once. fetchWithSignal now hosts its own slow endpoint: an in-process node:http server on a loopback ephemeral port that holds each response open for ~30s. The subject is unchanged - a real in-flight HTTP fetch cancelled mid-flight - with no external dependency. The 30s hold keeps regression detection honest: broken abort propagation surfaces as natural completion (ok: true) within the tests' 60s budgets. A per-workbench /api/delay route was rejected earlier because it would only exist on whichever workbench it was added to; the in-step server travels with the workflow fixture to every app. Signed-off-by: Alex Langenfeld <alex.langenfeld@vercel.com> |
||
|
Shalabh Chaturvedi
|
ec709ba533 |
[world-sim] Remove mint-ordered log (#3544)
Since we're moving to server side serialized ids, we dont need to simulate the prior model. ## Summary - remove the mint-ordered simulation mode and reservation API - make commit-time log positions and lagging-prefix reads the only simulator behavior - simplify CI, scenarios, tests, and documentation to the single log model ## Testing - node --check on changed TypeScript files - pnpm --filter @workflow/world-sim test (blocked: node_modules is absent; vitest unavailable) --------- Co-authored-by: vercel[bot] <35613825+vercel[bot]@users.noreply.github.com> |
||
|
Peter Wielander
|
1321570464 | [docs] Document duplicate-event handling, and describe webhook token generation accurately (#3497) | ||
|
Fantix King
|
09b299a03e |
[e2e] Add Python e2e Test (#3369)
Turns `packages/core/e2e/e2e.test.ts` into a cross-language conformance suite and adds `workbench/python` as its first non-JavaScript subject. --------- Co-authored-by: Claude Opus 5 <noreply@anthropic.com> |
||
|
Alex Langenfeld
|
af91cc2582 |
bench: per-chunk stream latency (CRTT/CDV) and replay-driven stream scenarios (#3393)
## Summary & Motivation - **CRTT (chunk round-trip time)** — per-chunk write→read latency for a paced stream, aggregated inside the reader step on the deployment (one clock domain) into a fixed log-bin histogram plus index buckets and mean-RTT profiles over stream progress and chunk size. Fills the gap between SL (first chunk only) and SO (whole-stream throughput), where a mid-stream delivery regression was invisible. It is deliberately a *round*-trip name: the future production one-way write→read metric is CTT, with its own skew caveats. - **CDV (chunk delay variation)** — inter-arrival gap minus inter-write gap per seq-adjacent pair, so each gap subtracts same-clock stamps and the stat stays skew-free and measurable in production later. Reported as each run's max positive value, since a 1-in-300 delivery stall dilutes out of pooled percentiles. - **Replay scenarios** — two real captured cadences (eve envelope protocol via gpt-5.6-sol; raw gateway SSE via gpt-5.4-nano) replayed through the same rig on an absolute open-loop schedule, so the workload is measured rather than invented; the 2x speed multiplier is the only chosen number, and matches how real fast-tier models behave (same chunk sizes, compressed time). Each capture carries a semantic sha256 over canonical `(offsetMs, bytes)` tuples so durabench's independent copy can be checked for drift. - **Streams table** — stream scenarios render in their own table with writer/reader sustained rates, CRTT percentiles, and median worst stall. No pass/fail targets yet: numbers and vs-main deltas only. - **SL/SO report rows retired** — CRTT's seq-0 slice reproduces SL and its aggregate reproduces SO's signal at ~100x the samples; write slip stays as artifact-only data, the only guard for producer stalls that neither CRTT nor CDV can see. ## Test Plan - [x] Unit tests for the bucketing/merge/CDV helpers and the renderer; the full benchmarks job ran green against real preview deployments, and the first Streams numbers separated workload strain (eve 2x: read 173 < write 181 c/s, CRTT p75 1278ms) from the transport floor (the paced control and the 1x reality row both clean). --------- Signed-off-by: Alex Langenfeld <alex.langenfeld@vercel.com> |
||
|
Peter Wielander
|
dc85865718 | [core] Drop pre-slot event ID support and preconditionGuard capability (#3519) | ||
|
Peter Wielander
|
0b7c9671ee | [bench] Add a Promise.all fan-out scenario with Fan-out TTFS/TTLS rows (#3522) | ||
|
Peter Wielander
|
b589460ce8 | [core] Report the replay position on every event write (#3479) | ||
|
Peter Wielander
|
6786db9953 | World-side incrementing event ID (specVersion 6) (#3389) | ||
|
Shalabh Chaturvedi
|
b7591fbf21 |
sim-world - deterministic scenario testing for race conditions (#3328)
* Add world-sim: a deterministic simulation World for concurrency scenarios `@workflow/world-sim` is an in-process World implementation whose point is that nothing in it races. Every world call is a stoppable point with `before`/`positioned`/`after` phases, every call is attributed to a writer (orchestrator, a named step body, or an out-of-band external client), and the clock is virtual — a thirty-day sleep costs no wall time. A scenario script says "stop this writer here, do that, let it go", so an interleaving that a real deployment leaves to chance becomes something you can name. The model follows workflow-server where it matters: - Event ids are minted at the handler boundary, not at storage append. DynamoDB does not generate ids, so the server does, in the request handler — and that id is the log's sort key. This is what makes "earlier log position, later commit" expressible, and it is the hazard the fence scenarios are about. - The out-of-band write marker is keyed on the event's own ULID time and is forward-only, over `hook_received`, `step_completed` and `step_failed`. - Both halves of the staleness fence are modelled behind flags: the watermark (`preconditionGuard`) that clients send today, and the count (`countGuard`) that they do not. The count is synthesized on the caller's behalf and keyed by run rather than by writer, since an orchestrator and its inline step bodies are one process sharing one loaded log. `workbench/sim-world` is the scenario book — 39 of them, each a workflow plus a script. Several are pinned corruptions rather than passing assertions: they record what the runtime does today, so that a change in behaviour shows up as a diff. The doc-29/30/31 trio is the argument for the count guard, one flag apart: (B, A, C) corrupts under the watermark alone, is fenced once the count is on, and (B, C, A) corrupts with both on. That last one suspends mid-run on purpose. The fence is a conditional append evaluated inside the storage write, so there is no checked-but-uncommitted moment to slip past; the window nothing can close is the quiescent gap between deliveries, where the run makes no writes and so meets no checks. Both packages are private and unpublished, so no changeset. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * Add the new sim-world workspaces to the lockfile Kept separate from the source commit because it is not a clean diff. The two new importers are the part that belongs to this branch; the rest is re-resolution churn from running `pnpm install` at a later date than whoever last touched the file — `latest` specifiers like docs' `radix-ui` move on their own. Drop or regenerate this commit if the churn is unwelcome; the source commit stands on its own. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * Make a consistency violation fail the scenario that tripped it `expect.violations` let a scenario declare the corruption it reproduced and pass for reproducing it. That is a green suite describing a broken system, and it goes red on the day someone *fixes* the bug — backwards, and the opposite of what a test is for. The field is gone. A scenario now states the outcome the run should have reached, which for a corruption means the branch its own durable log implies, and stays red until the runtime gets there. Any violation fails the scenario. The six reproductions read as ordinary failures now: expected output "afterSlow:doc-26", got "afterFast:doc-26" Six scenarios are therefore red, and `run.ts` exits non-zero. The count is the signal: seven is a regression, five means something was fixed and a scenario is ready to retire. Five of the six have known fixes — four predate the count guard, and doc-29 goes green the moment a client sends `stateEventCount`. doc-31 has none, which is the point of it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * Document the world-sim implementation in DESIGN.md Folds the working notes into a checked-in design doc: module map, the runtime model the simulator has to match, the interception model and its three call phases, the determinism machinery, the store's guards and fault injection, the writer vocabulary, the termination budgets, both consistency checkers, and current test status. Links it from both READMEs, and updates the workbench's doc-31 note now that the append-tail fence it needs exists as a proposal. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * Say which fix closes each red scenario, and which are demonstrated The fix column read "predates the count guard" on three rows, which scans as "the count guard is involved" when it meant the opposite. It was also wrong on one: `two racing STEPS, no hook anywhere` cannot be fixed by anything predating the count guard, since the row below it is that same fault with `preconditionGuard` on and failing identically. The column now names the specific change, and a second column separates a fix that is argued for from one that is shown — a passing scenario that is the red one with the fix armed, same tempo, one flag apart. Two of the six have that; three name a fix with no paired scenario yet; doc-31 has none. Also states the thing the column could imply but does not mean: `countGuard` requires `stateEventCount`, which no client sends, so three of the five identified fixes are dark in production. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * Give scenarios stable ids, refer to events by log position, add colour Four changes to make a rendered scenario something you can cite and read. Scenario ids. Every scenario gets a hyphenated `id` next to its prose name — `in-flight-after-decision`, `stale-read-step-count-fork`. The name is a sentence and will be reworded; the id is what a commit message, a bug report or `pnpm sim <id>` refers to. The runner matches the id first and falls back to the name, so existing invocations keep working, and the table of the six red scenarios in DESIGN.md now cites ids. One reference scheme. Events were referred to three ways at once: the trace printed raw correlation ULIDs, violation messages carried `evnt_<ulid>` event ids, and the docs spoke of "position 7". Now there is only log position. `#12` is the twelfth event in the durable log sorted the way `events.list` sorts it, `@7` is the resource created at position 7, and ids inside violation messages are rewritten on the way out. That numbering is deliberately in *log* order while the trace prints in *commit* order, which makes the subject of the red scenarios visible on the page: a run whose log disagrees with the order its writers committed in shows positions counting backwards. Colour, by event family, and only when the destination is a terminal. Off under `NO_COLOR` or `--no-color`, forced on with `--color`. With colour off the output is the same plain ASCII, so it stays usable as a golden file. The workflows under test are now one file. Nothing imported across them and scenarios read them alongside the tempo that steers them, so five modules were a two-file hop for no benefit. Unit tests 61/61. Scenario book unchanged at 33 passed, 6 failed. Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * Add an append-only log world, and split the book into one file per scenario Three things the book could not do before. `appendOnlyLog` gives an event its log position at *commit* instead of when its handler minted one. That is the single change that makes a stale read impossible — the log can be behind, never wrong — so playing the same 39 scenarios with and without it is how you tell which of the six reds the change would actually close. All six: 33 pass / 6 violations mint-ordered, 39 pass / 0 violations append-only. It bundles two effects that have to move together: an overtaken write re-mints at the tail, and a withheld read returns a prefix rather than a hole, which is why `onStaleRead` now carries `{eventId, hidden, truncated}` and the trace distinguishes a lagging read from a stale one. `preconditionGuard` on `RunScenarioOptions` forces the fence off across the book, asking whether anything relies on it. Violations 6 -> 8 mint-ordered, so it is load-bearing there; 0 -> 0 append-only, so it is dead weight once positions are assigned at commit. Both flags are tri-state: `undefined` leaves it to the spec, which is not the same as `false`. The scenario book was a 1020-line file; it is now 39 files and an index that only decides reading order. Same 39 ids. This is the whole answer to "how do I add a scenario" — copy the file next door — and it is why the README can be short. Three `in-flight-*` scenarios are reworded so that no expectation is restated per world: a scenario is one sequence of movements, the only thing a world changes is what a read returns, and what catches the fault in both is the invariant that a run's log must replay back into that run. Also: `loadFlowHandler` moves out of `build.ts` into `load.ts`, so playing scenarios no longer drags SWC and esbuild into the module graph; the CLI gains `--report-only`, `--summary-file` and `--detail-file` for CI, with the default still exiting non-zero; and the workbench's `test` script points at `--report-only` so a recursive `pnpm -r test` does not go red for the six. Docs are split by task: the workbench README is how to add a scenario, and the package README plus DESIGN.md are how to change the simulator. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * Publish the scenario book from CI, without blocking a merge Plays the book on every pull request, once per log world, and posts both summaries as one sticky comment. It never gates. Six scenarios are red on purpose — each is a reproduction of a corruption the runtime can still produce, stating the outcome its own durable log implies and staying red until the runtime gets there — so a lane that failed on them would be red on every PR and read as broken rather than as informative. What it publishes is the pair of counts, and the thing to look at is whether they still say 33/6 and 39/0. A seventh red is a regression; five means something got fixed and a scenario is ready to retire. The append-only column is the measurement the pair exists for: it says which of the six would close if positions were assigned at commit. Non-blocking at the job level rather than only on the two sim steps, so that a failed install or a PR comment the token cannot write does not turn this into a red X either. The steps themselves keep their real exit codes, so the run still says which world was clean. `--title` is new on the CLI: two summaries land in one comment, and two headings reading "world-sim scenario book" would leave the chips line as the only way to tell them apart. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * Make the Sim World comment four lines until you open it Two collapsed folds, one per world, each with its count and a green or orange dot on the visible line; the table is behind them. Nothing else above the fold. The failures list is gone. Six scenarios are red on purpose, so a comment that led with them led with the part that was not news, and grew a wall of text on exactly the PRs that changed nothing. The count is the signal — and anyone who wants the names can open the table, which has always had them. `renderMarkdownSummary` now renders no heading of its own, since it is built to be stacked under one; the workflow supplies the heading and the one-line description. `mdCell`, `clip` and `dedupe` existed only for the failures table and go with it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * docs(sim-world): drop the six-red list, add an API reference The "six red scenarios" section was a second copy of something `pnpm sim` already prints exactly, and the copy that goes stale. What replaces it says how to read a red — an open bug stating the outcome its own durable log implies, green when fixed rather than when seen again — and points at DESIGN.md for the part that is not re-derivable from a run. The reference has three tables. Writers: what a concurrent thread of execution is here, and the four ids one can have. Movements: the instruction that moves a writer to a place and holds it, described against the world boundary, the position in the event log, and the commit to storage — three moments, which is why there are three stops rather than one. Withholdings: the two ways to change what a reader sees without holding anybody. Terminology unified on hold/held (the word `Held` and the trace already use) and on "assigned a position in the event log" / "committed to storage", which also fixes the `runToEventProduced` docstring: the event has crossed the world boundary there, so "submitted" was the misleading half. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * docs(sim-world): move the API reference into the package, uniform vocabulary The reference documents `@workflow/world-sim`, so it belongs beside the API and not in the workbench that happens to be its first caller. It replaces the old `## Writers` section, which said most of the same things in a different order and a different vocabulary. Three nouns, one meaning each. A *writer* is a thread of execution — the table lists all three kinds with the handle that names each and the events each one writes. An *advance* moves one writer to a named place and holds it; the three `runTo*` stops are the three moments a write has (crossed the world boundary, assigned a position in the event log, committed to storage), which is why there are three rather than one, and the table says which way a concurrent commit sorts at each. A *withholding* hides something from readers without holding anybody, which is what `withholdNextEvent` and `beginHookDelivery` have in common and why the latter is not an advance. "Movements" is gone in favour of "advances", which the code and the older prose already used. "Held" replaces "stopped"/"paused" throughout, matching `Held` and `isHeld()`. The workbench README loses the duplicate tables and keeps the four rules that bite on a first scenario, so it is a guide to adding one and links out for the rest. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * docs(world-sim): rename "Two logs" to "World behaviors", one-line motivations The section described two logs, but there are four world behaviors a scenario can pick between: the two log orderings and the two guards. Renamed and regrouped so each one is described by what it does rather than by which of the current reds it closes. Measured counts and "production" / "what happens today" framing are out of this README throughout — they belong to a run of the book, and a doc that carries them is stale the first time a scenario changes colour. The workbench README and the CI lane still print them, where they read as measurements. Every section's motivation is one line; the file-top motivation is unchanged. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * docs(world-sim): Usage shows both halves — the workflow and the script The example was a script with no workflow beside it, which hid the fact that you write both. Replaced with the smallest pair that has something to control: two steps in flight at once, and a script that decides which of them reaches the log first. The scenario in it was run before it was written down — `prepare` held before it takes a position, `finalize` committed into the earlier slot, #6 then #7 — so the positions the prose cites are the ones the trace prints. Link to the workbench moved to the end of the section, where it reads as "where to go next" rather than as an aside mid-example. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * docs(world-sim): say what "arm" meant instead of using the word The READMEs told a reader to arm a wait without ever saying that calling an advance and awaiting it are two separate things. That is the whole mechanism — `runTo` registers its watch synchronously when called, and the returned promise only reports arrival — so it is stated plainly once in Advances and the jargon is dropped everywhere it stood in for the explanation. "Armed" survives only where it means a guard is switched on, which is a different word doing a different job. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * docs(world-sim): two symmetric watches in the Usage example The example started one watch and awaited the other inline, which is exactly the shape a reader cannot generalise from — it looks like the two calls do different things. Both are now started, then both awaited, so the sentence above the block and the code below it say the same thing. Re-run before committing: same log, finalize at #6 and prepare at #7. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * docs(world-sim): neutral names in the Usage example `prepare` / `finalize` / `held` / `committed` made a reader decode four domain-ish names to see a mechanism that has nothing to do with any of them. stepA and stepB, watchA and watchB: the only thing left to notice is that the two watches stop at different points, which is the whole lesson. This detaches the example from the workbench's `parallelStepsWorkflow`, so it was verified against a throwaway copy of the workflow rather than assumed — same shape, stepB at #6 and stepA at #7, replay ok. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * refactor(world-sim): one event fold, two call phases, a smaller entry Three reductions to the same end — less surface to keep consistent — with the scenario book as the check that none of them changed behaviour. **One fold.** `create` validated *and* applied an event in a 17-case switch; `foldSeededEvent` applied it again in 15 cases without validating. Two copies of the event -> entity state machine that only a replay compares, and any divergence between them makes a replay disagree with the run it is checking for a reason that is not the runtime's fault. Both paths now end in one total `applyEvent`: the write path validates first and calls it, `seedFromLog` calls it with no validation at all, because those events were accepted once already and re-litigating them would reject legitimate history. store.ts loses 175 lines. Characterization tests for the seeded path went in first and were confirmed to bite before the refactor started. **Two phases.** `CallPhase` had a `positioned` phase between `before` and `after`, and `runToPositionMinted` / `runToCall` to park on it. Nothing in the book used any of the three: the mint/commit gap they existed for is reached through `beginHookDelivery`, which owns both halves explicitly and does not block the writer that made the write. Removing the phase also removes an `await` from the interception path, so this was measured rather than reasoned about — all four book runs are unchanged. **A smaller entry.** `index.ts` re-exported the construction kit — `createSimWorld`, `createSimStore`, `driveQueue`, `verifyReplay`, `checkInvariants`, the clock — which nothing outside the package imports and which made every one of their signatures a compatibility promise. The entry is now the scenario surface; extenders import from the module. `InFlightWrite` joins it, having been missing though `beginHookDelivery` returns it. Unchanged, and the point of saying so: 33/6/6 mint-ordered, 39/0/0 append-only, 8 violations with `--no-fence`, 0 with both. 70 unit tests green (up from 67), tsc and biome clean in both workspaces. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * fix(world-sim): follow the new event-result contract Rebase fixups for four core replay commits, no behaviour change of our own. `EventResult` is now a union — a populated page (`events` + `cursor` + `hasMore`) or all three absent — so the three fields cannot be assembled from three independent variables the type has no way to see agree. They travel as one `deltaPage` object, spread as a whole or not at all. The spread has to be conditional rather than optional: `...maybe` widens each field to `T | undefined`, which is neither arm. `processExitTriggersQueueRedelivery` is gone from the `World` interface — #3385 stopped exiting on an exhausted replay budget, so there is nothing left to tell it not to. The book is unmoved across the rebase, which is the thing worth reporting: 33/6/6 mint-ordered, 39/0/0 append-only, 8 violations with `--no-fence`, 0 with both — and the same six ids red, not a swap that nets to the same count. 70 unit tests green. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * feat(world-sim): a scenario for the unclaimed-payload delivery race Reproduces vercel/workflow#3406 as a pair of scenarios. A hook payload nobody reads registers an unarmed delivery barrier, and a step result is allowed to skip it — otherwise the workflow stalls until the barrier registry idles. The skip is transitive, so the step also skips an armed `wait_completed` merely parked behind that payload. Both branches then draw their next `step_created` id in the order the log does not record. The replay invariant cannot see this: live and replay run the same code, make the same mistake, and agree. What is checkable is the log disagreeing with itself — `wait_completed` is committed first, yet the step branch draws the earlier id. Verified red on the current runtime and green with #3406's diff applied. `claimed-payload-under-fork` is the control: same steps, same tempo, same log order through the step result, one branch awaiting the hook so the payload lands claimed. It passes in both cases. Getting there needed one new primitive. The delivery loop is serial, so a held inline step body stops the loop inside its own delivery and no timer can fire — every interleaving where a `wait_completed` lands while a step result is outstanding was unreachable. `sim.deliverQueued(select?)` takes a message out of the pending set and delivers it from the script, concurrently with the hold; `takeById` removes it first, so the loop can never pick up the same message. The book is now 41 scenarios: 34/7 with 6 violations mint-ordered, 40/1 with 0 violations append-only. The six violation reds and the --no-fence 6 -> 8 / 0 -> 0 measurements are unchanged. The new red is the first that stays red in both worlds, because no log position is wrong in it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * fix(world-sim): address review on the simulation world Fifteen findings from review, and one measurement that moved because of them. Correctness: - tempo: `abort()` stored the raw `reject` rather than the wrapper that also disposes the watch, so a point reached after teardown blocked a world call on a promise nobody could release. The deadline is one-shot and already spent by then, so that is a hang. Route abort through the disposing path and make the action a no-op once aborted. - world: `externalDepth` was a plain counter, so a step body writing concurrently with a scripted `deliverHook` was attributed to the script — invisible to any armed `runTo`, and `ext` in the trace. Measured: one misattribution book-wide. Scope it with `AsyncLocalStorage` instead. - writers: `runToEventCommitted` matched a *rejected* create, so in fence scenarios a script could wake believing a write was durable when it had 412'd. Record `failed` on the observed point and require `failed: false`. - clock/ids: reject fractional `advanceBy`, and floor in `ulid()` — a fractional millisecond silently minted `undefinedundefined…` ids that failed much later at `z.ulid()`. - streams: literal NUL bytes as key separators, `limit: 0` paging forever, and a garbled cursor slicing the array away as `NaN`. - scenario: register the handler under the exported `WORKFLOW_QUEUE_PREFIX`. The count guard, which is the substantive one. Since #3145 `@workflow/core` sends `stateEventCount` on every replay-context create and the server's count guard defaults on, so the sim's "no client sends the count" claim was stale in six places. `countGuard` now follows the fence, the sim prefers the runtime's own count over its reconstruction, and the two scenarios whose subject is isolating the watermark half say `countGuard: false` explicitly. The scoreboard does not move under the production-shaped default, which is itself the answer to the review's question. `log.monotonic-order` could never fire: its only caller fed it the sorted array. It now takes commit order, supplied only by a world that promises the two agree — under a mint-ordered log an out-of-order commit is the premise the scenario injected, not a defect. Docs: DESIGN §5 gains the two ways the sim's guards are stronger than production's (FIFO-vs-mint-order pruning, and a fence that is exact in-process where production's is region-local and fails open); §9's scoreboard is regenerated and its "no fix armed anywhere real" claim corrected; §10 gains the parallel hook-resume path, which no sim delivery takes. `step-vs-step-fork` and its twin now say who the withheld reader is in production. `unclaimed-payload-under-fork` is green after the rebase — #3406 fixed the delivery-barrier ordering — so the book is back to six reds and stays there. Counts refreshed everywhere they appear: 35/6/6 mint-ordered, 41/0/0 append-only. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * fix(world-sim): make the README's code samples type-check `packages/docs-typecheck` globs `packages/*/README.md`, so the new README went into the Docs Code Samples job and five of its `ts` blocks failed. Four were genuine excerpts with an unbound `sim` or `spec`; one was a match- object *sketch* containing a `…`, so not TypeScript at all — that one loses its `ts` tag, joining the eleven other untagged blocks in the file. The rest name what they use. That needed two `paths` entries in the checker: `@workflow/world-sim` was unmapped, and an unresolved import is deliberately tolerated there (`isExpectedMissingModule`), so every identifier in those samples was `any` — they would have gone green while checking nothing. Mapped, they are checked against the real declarations: seeding `id: 12345`, `readyAtMsTYPO`, `dirsTYPO` and `runToEventCommittedTYPO` is caught against `ScenarioSpec`, `PendingMessageView`, `SimBuildOptions` and `Writer`. The lead teaser keeps its four unadorned lines and takes a `@skip-typecheck` marker instead; the same calls appear in full, checked form further down. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> --------- Co-authored-by: vercel[bot] <35613825+vercel[bot]@users.noreply.github.com> |
||
|
Peter Wielander
|
fbebf7104d | [core] Keep step results ordered behind waits parked on unread hook payloads (#3406) | ||
|
Shalabh Chaturvedi
|
264ddff67b |
Add WebSocket transport for step-execution event writes (opt-in) (#3084)
* sdk side for workflow server websockets Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * hardcoded workflow server Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * debug info Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * more debug Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * remove unnecessary debug Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * default on websockets, and override url Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * fix for missing funcs Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * make websockets opt outo Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * enable ws again Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * [revert later] reduce test to single test, test both http and ws at the same time Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * empty Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * Run full suite with and without ws Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * empty Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * improve e2e test Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * minimize tests Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * fallback to http when proxy present Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * default to websockets, remove matrix Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * remove smoke test Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * update to new protocol Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * adjust for new protocol (runid in path) Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * fix ws transport error Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * blank Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * fix ws dep Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * fix ws external: only accelerators, not ws itself Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * add dedicated WS-transport e2e job; flip WS default back to opt-in Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * build all packages before local vercel build (needs workflow/nitro on disk) Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * force NITRO_PRESET=vercel for the local vercel build step Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * install vercel CLI once instead of npx-ing it per command Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * add changeset for WS events transport Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * Harden the WS events transport and gate its e2e jobs Follow-ups from review of the WS transport. CI: `e2e-vercel-ws-transport` was wired into the `summary` job but not into `e2e-required-check`, so all three WS jobs could fail while the required check stayed green. Added to both branches of the status validation — including the `workflow-server-test` label branch, where the job runs under the same gating as `e2e-vercel-prod`. Transport: - `reqId` and the pending-reply map are now per connection rather than per transport. The protocol defines `reqId` as a per-connection counter, so a reconnected socket restarts at 1; with one shared map that collided with the previous socket's still-registered waiters. It also makes the superseded-socket guard structural instead of something the close path has to remember. - Post-open socket errors are no longer silent. The only `'error'` listener closed over the connect promise's `reject`, already settled once `'open'` fired, so every broken pipe / 1009 / protocol fault was swallowed and its requests hung with no per-request timeout to save them. Now logged, and the connection is torn down. - An unexpected close reconnects eagerly instead of waiting for the next write, since a socket breaking mid-run means more writes are coming. Bounded by exponential backoff, an attempt cap that falls back to lazy reconnect, a bail-out when a newer socket is already live, and an `unref()`ed timer so a backoff window can't delay handler exit. - `ws.send()` failures reject their request. `send()` doesn't throw on a non-OPEN socket — it reports through a callback we weren't passing — so the request just sat in `pending` forever. - The reserved `reqId: -1` malformed-frame reply and undecodable frames are logged loudly instead of dropped. - Auth headers resolve once per socket via a thunk, not once per event. The bearer only rides the upgrade, so the old code awaited `getVercelOidcToken()` on every write and discarded all but the first. Re-resolving on reconnect also means a new socket gets a fresh token. Adapter: a reply with no numeric status now fails closed. Defaulting to 200 reported a write as applied whenever the client met a frame it didn't understand — and the protocol is explicitly designed to grow new response variants. Tests: 24 new unit tests over the paths the e2e suite can't reach on demand (send failure mid-flight, error after open, late close from a superseded socket, reconnect backoff and give-up, sentinel/undecodable frame logging, one-token-per-socket) plus the adapter's fail-closed and typed-error mapping. Co-Authored-By: Shalabh Chaturvedi <shalabhc@users.noreply.github.com> Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * ci: re-trigger to confirm the prior e2e failures were flake No code change. The 5 failures on 0bb21e7 clustered in a ~20s window across HTTP-path jobs (example/nuxt on the same test, sveltekit on a timeout) and one WS job (sleepingWorkflow's clock-skew assertion), which points at the environment rather than the transport changes. Re-running to confirm. Co-Authored-By: Shalabh Chaturvedi <shalabhc@users.noreply.github.com> Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * Add WS wire-contract conformance tests and pin the transport gate Three gaps in the existing coverage. **The HTTP path was already covered** — `events-v4.test.ts` has 22 tests, including five directly on `createWorkflowRunEventV4` over HTTP (alias URL, frame meta contents, response decoding, skipPreload/stateUpdatedAt forwarding). Those run with the gate unset, so they do confirm the two-branch refactor didn't disturb HTTP. No new tests needed there. **But nothing pinned the gate itself.** Every HTTP assertion stays green if the default flips to WS, because the transports are built to be indistinguishable at the result layer — and an earlier revision of this branch did flip the default deliberately, for benchmarking. Added tests for `isWsEventsTransportEnabled()` across values, and one that drives a real HTTP request through a MockAgent while asserting the WS transport is never constructed. **Nothing verified the bytes.** `ws-transport.test.ts` replies with whatever the test hands it, which proves the client's lifecycle but not that its frames are what workflow-server accepts. That's the drift the spec doc exists to prevent, and it already happened once: event meta flat on the frame where the server wanted it nested under `event`, with both sides' tests passing. `ws-protocol-conformance.test.ts` pairs the real client stack (through `createWorkflowRunEventV4`) with a fixture mirroring the server route's per-message handling: decode one frame, validate against a local copy of `WsRequestFrameSchema`, dispatch, encode the reply the way `replyMeta` does. `experimental_upgradeWebSocket` needs a real Vercel runtime, so the socket is faked — everything above it is genuine. Covers: the frame shape the server accepts (and that `reqId`/`type`/ `runId` don't leak into the event meta), payload passthrough, exactly one frame per message, 409 → the same typed error HTTP raises, fail-closed on an unknown reply variant, and reqId correlation across concurrent writes. Plus golden byte fixtures, since the schema copy is the one thing here that can silently drift. This is the "golden-frame interop test" the server spec lists as an open gap; the matching half still needs to land in workflow-server. Verified the conformance suite is not vacuous: flattening the client's frame meta fails 5 of its tests. Co-Authored-By: Shalabh Chaturvedi <shalabhc@users.noreply.github.com> Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * match the HTTP RetryAgent's transient-failure policy on WS HTTP event writes go through an undici RetryAgent (RETRY_AGENT_OPTIONS): 5xx and transient connection errors are retried in-process, honoring Retry-After. The WS path never touches undici, so it shipped with no transient-failure handling at all — a single 503 or a mid-write reset surfaced straight to the step runtime and cost a whole step retry where HTTP would have absorbed it in milliseconds. That gap is invisible in a passing test run: writes still succeed, they just cost far more. So copy the policy rather than reinvent it — [500, 502, 503, 504] plus transport failures, undici's default backoff, Retry-After honored, and 429 deliberately excluded for the same reason RETRY_AGENT_OPTIONS excludes it (a firewall challenge this client cannot solve, which in-process retries only amplify). Adds WsTransportError so retryability is a typed property of the failure rather than something the adapter infers by string-matching. Splits resolveWsTransport()/wsReplyStatus() out of postEventFrameOverWs so the retry loop stays readable. The existing "fails closed on an error frame" test used status 500, which is now absorbed by the retry — switched to 403 so it keeps testing fail-closed rather than accidentally testing no-retry. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Co-Authored-By: Shalabh Chaturvedi <7066873+shalabhc@users.noreply.github.com> Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * lazy-load `ws` so the default HTTP path never evaluates it events-v4.ts imports ws-transport.js unconditionally — the transport gate is a runtime branch, not a build-time one — so a top-level `import { WebSocket } from 'ws'` put `ws` and its optional native accelerators on the module-init path of every deployment, including the overwhelming majority that never opt in and never open a socket. Defer it to the first connect, memoized as a promise so concurrent first connects share one import. WebSocket.OPEN becomes an inlined constant so the readyState check doesn't pull the module in just to read it off the constructor. This does NOT remove the need for the bufferutil/utf-8-validate externals this branch also adds: webpack and Rollup both statically follow a dynamic import(), so the build-time story is unchanged. What it buys is that a deployment which never enables the transport never *evaluates* `ws`, so a mis-bundled accelerator can't break it. The test lives in its own file because vitest caches a vi.mock factory result for the life of the module registry — once any test in a file has connected, the factory never runs again and the counter can't distinguish "loaded lazily" from "loaded at import". Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Co-Authored-By: Shalabh Chaturvedi <7066873+shalabhc@users.noreply.github.com> Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * release idle WS transports instead of renewing them forever The transports map was never pruned and WsEventsTransport had no way to close. Combined with eager reconnect that made a connection immortal by construction: the server drains at its own maxDuration and closes, the client immediately reopens, and the server pins a fresh invocation — for a run that finished long ago. A warm container ended up holding a live socket, and a live server invocation, for every runId it had ever served. workflow-server#683 already lists "one invocation stays resident per run rather than per write" as a known gap; this made it "per run, forever". Add close() plus a 60s idle release. There is no "run complete" signal to hang teardown off — the events adapter is a stateless per-write call — so idleness is the available proxy. 60s sits well below the server's ~680s drain deadline, so the client releases rather than the server reclaiming, and well above the gap between steps of an active run. scheduleReconnect() now bails when closed: close() closes the socket, which fires the same close handler an unexpected drop would, and without the guard the transport would instantly reconnect what it just released. request() revives an idle-closed transport rather than failing the write, re-registering itself only if nothing newer has claimed the map slot. Eviction therefore costs one handshake, not an error. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Co-Authored-By: Shalabh Chaturvedi <7066873+shalabhc@users.noreply.github.com> Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * refresh the bearer on an auth_expiry drain workflow-server#683 tags a drain frame with why it is closing: max_duration means the socket aged out and a plain reconnect is right, auth_expiry means the *bearer* ran out and reconnecting with the same one just earns a 401. This client logged the drain and ignored the reason, so against #683 an auth_expiry drain would burn all five reconnect attempts against a token the server had already rejected, then give up. Parse the reason (absent reads as max_duration, so this stays correct against the currently-deployed server) and thread forceRefresh through the getHeaders thunk, which triggers @vercel/oidc's refresh path via a wide expirationBufferMs. Worth being precise about when that can actually help. getVercelOidcToken resolves getContext().headers['x-vercel-oidc-token'] ?? env, and refreshToken() only writes the env var — the request-context header wins. So inside a deployed function there is genuinely no fresher token mid-invocation and the refresh is a no-op; outside one (CLI, local dev, a long-lived server) it works. That makes the guard the load-bearing half: if the re-resolved bearer is byte-identical, decline to reconnect, say so, and wait for the next write — which usually arrives on a new invocation carrying a new token. That failure is marked non-retryable so the retry loop doesn't spin on it either. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Co-Authored-By: Shalabh Chaturvedi <7066873+shalabhc@users.noreply.github.com> Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * document the WS path's instrumentation gap The HTTP branch goes through fetchV4 -> instrumentedFetch, which is not just a fetch wrapper: it opens the OTEL CLIENT span, injects trace context, sets the cache-bust header, emits the DEBUG logs, and routes through the global fetch that Vercel's observability "outgoing requests" view instruments. The comment on fetchV4 records why that matters — bypassing it via undici.request() is exactly what once made v4 event traffic disappear from the log viewer. The WS branch bypasses all of it. With the flag on, per-event writes have no client span, propagate no trace context to workflow-server, and don't appear in the outgoing-requests view; the server's own transport-tagged request metrics are the only remaining signal. That's acceptable for an opt-in POC behind a flag and unacceptable as a default, so write it down where someone deciding to flip the default will read it: instrumenting the transport is a prerequisite for that, not a follow-up nicety. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Co-Authored-By: Shalabh Chaturvedi <7066873+shalabhc@users.noreply.github.com> Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * ship the ws-accelerator externals instead of documenting a workaround `bufferutil` and `utf-8-validate` are optional native accelerators for `ws`, and neither is installed by default. Every bundler has to be told to leave them alone, for two different reasons: Rollup/Vite/Nitro fail the build outright (`Could not resolve "bufferutil" imported by "ws"`), while webpack bundles the JS wrapper without its native `.node` binding and throws `bufferUtil.mask is not a function` at runtime. The webpack half shipped in `@workflow/next`. The Rollup half only existed in `workbench/vite` and `workbench/tanstack-start` as `nitro.rollupConfig.external` — app configs, not shipped code. So a real user of `@workflow/vite`, `@workflow/nitro`, `@workflow/nuxt`, `@workflow/sveltekit` or `@workflow/astro` hit the same build failure the workbench had already worked around, and had to rediscover the fix. Fix it where it propagates: `workflowTransformPlugin` in `@workflow/rollup`, which all of those integrations already install. It is already the home of exactly this pattern for the optional `@opentelemetry/api` peer, so this sits next to its closest precedent. Note the treatment is deliberately the inverse of the OTEL one, which is externalized only when it *can't* be resolved. The OTEL API must load for tracing to work, so a self-contained output has to bundle it when present. These accelerators must specifically NOT load — they are a performance nicety with a correct try/catch fallback in `ws` — so unconditional external is both simpler and safer than risking a half-bundled native module. The two workbench configs drop their local copies, which is what proves the shipped fix actually works rather than being masked by them. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Co-Authored-By: Shalabh Chaturvedi <7066873+shalabhc@users.noreply.github.com> Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * one retry policy for both transports, and no unanswerable waiters Two review findings on the WS events transport. **Retry belongs to `event-retry.ts`, not the adapter.** The WS path had its own retry loop, justified as mirroring undici's `RetryAgent`. That justification was wrong: `RetryHandler` defaults `methods` to GET/HEAD/ OPTIONS/PUT/DELETE/TRACE and nothing overrides it, so the `RetryAgent` never retried an event POST on either transport — which is precisely why `event-retry.ts` exists. Worse, that loop sat *inside* `withEventPostRetry`, so it defeated a compile-checked safety gate: `EVENT_RETRY_ELIGIBILITY` marks `step_started`, `step_retrying` and `hook_received` non-retryable (a replayed `step_started` double-increments `attempt`), and those frames were re-sent up to five times before the gate ever saw a failure. For eligible types the two loops multiplied: 3 outer attempts x 6 inner, with an inner backoff reaching 30s against an outer base deliberately set to 100ms. `postEventFrameOverWs` now makes one attempt and translates failures into the vocabulary that policy already speaks — a transport failure becomes a `WorkflowWorldError` with `code: 'TRANSPORT'`, exactly as `utils.ts` does for a failed `fetch`, and `isRetryableEventPostError` gains one clause keyed on that code. `WsTransportError` loses its `retryable` flag; its only consumer was the deleted loop. Two deliberate consequences. The code-keyed clause broadens HTTP in-process retry to `UND_ERR_CONNECT`, `UND_ERR_CLOSED` and `EAI_AGAIN`, which were in utils.ts's transient set but missing from event-retry.ts's — two hand-maintained lists collapsed into one semantic code. And the stale-token case (drain for auth expiry, refresh yields the same bearer) now gets two in-process attempts that cannot succeed, ~300ms before it falls through to queue redelivery; that is cheaper than keeping a WS-specific policy alive for one call site. `TIMEOUT` is deliberately not in the clause: utils.ts maps a caller-supplied `AbortError` onto it, and a cancelled write must not be re-issued. A status-less reply also stops being a bare `Error` — as one it failed `WorkflowWorldError.is()` and surfaced a protocol version skew as a USER_ERROR. It is now `code: 'PARSE_ERROR'`, the same code utils.ts uses for an unreadable HTTP body, and for the same reason: the write may or may not have landed. **No waiter is left unanswerable.** An undecodable frame, the server's malformed-frame sentinel (`reqId: -1`) and a non-numeric `reqId` were logged and dropped. None can be correlated by construction, so the request that provoked them stayed in `pending` with nothing in existence able to settle it — freed only by the server's own drain (~680s from connect), typically past the invocation's `maxDuration`. Each now fails the connection: every waiter learns why, and the socket is replaced. A reply for an id nobody is waiting on stays log-and-drop, deliberately — that request already settled, so nothing is orphaned, and failing the socket would punish healthy in-flight writes. A per-request deadline backs that up for whatever is left, including a server that accepts a frame and never answers it. Same knob as the HTTP path (`WORKFLOW_REQUEST_TIMEOUT_MS`, 60s), whose doc comment already describes this exact hang-to-SIGTERM pathology. One existing idle-teardown test needed the deadline raised: the idle window and the default deadline are both 60s, so a request could not outlive the former without also outliving the latter. The test is about `inFlight > 0` suppressing the teardown, so it now sets the deadline out of the way. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * Open the ws socket when the invocation starts, not on its first write Lazily connecting bills the whole handshake — an upgrade round-trip plus the OIDC token mint that rides it — to whichever event a fresh invocation writes first. When that is a `step_started` issued as the step body is already running, the event's server-recorded timestamp lands later than the work it describes: the step looks shorter than it was. That is the shape of the e2e timing failure on this branch, where a 9s step measured 6.5s from `getStepMetadata().stepStartedAt`. The queue handler is the earliest point that knows the run id, and a message delivered for a run means writes are coming, so `warmWsEventsTransport` starts the handshake there. By the first write it is done or in flight, and the write just uses it. Nothing about it is load-bearing: - It doesn't await, and can't fail the handler. A warm that fails logs and stops — a never-opened first connect is precisely the case `connect`'s close handler already declines to retry, so no backoff loop starts for a run that may never write. The first real write connects as it would have anyway, carrying the shared retry policy. - No-op unless `WORKFLOW_EVENTS_TRANSPORT=ws`, and no-op for the api-workflow proxy World, which can't serve an upgrade at all — the same fallback the write path takes. - Warming arms the idle timer as if a request had settled, so an invocation that warms and never writes (a health probe carrying the run id it is about to create) releases its socket on the usual 60s rather than stranding it. The socket is not `unref`'d, so a stranded one would hold this process and a server invocation open. Also closes a race that warming makes reachable: `close()` can only drop the connection it can see, so a release landing mid-handshake left the socket to install itself afterwards onto a transport already evicted from the cache, which nothing would then ever close. The `open` handler now declines to adopt a socket whose transport was released while it was connecting. This was already reachable via the eager reconnect path, just much harder to hit. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * changeset: just the env var Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * inline the ws-accelerator predicate at its only call site Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * refactor(world-vercel): trim ws-transport comments Comments were 47% of the file. Cut the historical narration, the restatements of adjacent code, and the repeated rationale (the `unref` reasoning appeared four times, per-connection reqId three), keeping the non-obvious facts: `ws.send()` reports failure via callback instead of throwing, reqId is per-connection so `pending` must be too, the unknown-reqId case is deliberately non-fatal, the auth_expiry same-token bail-out, and why the idle timeout exists at all. No code changes. Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * own transport selection in the transport module `events-v4.ts` was assembling the WS transport itself: reading the opt-in flag, resolving the URL, deciding which Worlds can use a socket, minting the per-connection header thunk, and holding the two once-per-process log latches. None of that is about turning an event into a frame, which is what the rest of that file does. Move it next to the socket it configures — `events-v4.ts` now consumes one seam (`resolveWsTransport`) plus the gate, and `queue.ts` gets `warmWsEventsTransport` from the module that owns the warm. `headersToRecord` now lives in `http-core.ts` because both callers need it and neither may import the other: `events-v4` already depends on the transport, so the reverse edge would be a cycle. Test fallout, and the reason the move is worth it: `events-v4-ws.test.ts` mocked `getWsEventsTransport` to observe the resolve step, which no longer intercepts anything now that the call is intra-module — an ESM mock replaces a module's exports, not its own call sites. That mock's tests were only ever about selection, so they move to `ws-transport.test.ts`, where the real selection code runs against the existing fake-socket harness instead of a stub. `resetWsEventsTransportsForTest` clears the log latches so the once-per-process assertions don't depend on test order. What stays behind mocks `resolveWsTransport` and covers what that file is actually for: reply frame in, `Response`-shaped result out — including the null-resolve fallback to HTTP, which nothing covered before. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * import `ws` statically The lazy `import('ws')` was there to keep the package off the module-init path of deployments that never opt in — `events-v4.ts` imports this module unconditionally, since the transport gate is a runtime branch. Measured, that buys ~17ms: `require('ws')` is 16.5-18.0ms cold, 13 modules, and neither `bufferutil` nor `utf-8-validate` loads (optional peers, absent by default). Bundle size is identical either way — webpack and Rollup both statically follow a dynamic `import()`, which is why the externals in `@workflow/builders` are unaffected by this change. For 17ms it cost a memoized promise, an inlined `WS_READY_STATE_OPEN` (so a readyState check wouldn't force the module to load just to read a constant off the constructor), and a whole test file — `ws-transport-lazy.test.ts` had to live alone, because vitest caches a `vi.mock` factory result for the lifetime of a module registry, so only a file that connects exactly once can observe the laziness at all. It also skewed the thing this branch exists to measure. The import lands inside the first connect, so on a warm container it is billed to whichever event write opens the socket, inflating the timestamp of the step it labels — the same distortion the queue pre-warm was added to remove. Also drops `WS_READY_STATE_OPEN` in favour of `WebSocket.OPEN`, now that reading it is free. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * tighten the comments on the ws transport Comments only — no code changes in this commit. Cuts ~150 lines of prose across the WS additions. The rule applied: keep the design factors a future reader needs (why the connection is scoped to a run, why a bad reply takes the socket down, why the accelerators are externalized unconditionally, why `TIMEOUT` is excluded from the `TRANSPORT` classification) and drop the narrative of how the code got here — which revision did what, what an earlier attempt got wrong, what was measured on the way. That history lives in the PR and the git log, where it doesn't have to be re-read on every visit to the file. Biggest reductions: the retry essay above `postEventFrameOverWs` (30 lines to 11), the flag's OTEL-gap note (34 to 13), the OIDC refresh explainer (26 to 14), the accelerator rationale in `@workflow/builders` (26 to 14), and the conformance suite's header (28 to 17). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * inject W3C trace context on the ws upgrade Frames carry no headers, so the upgrade is the only place this transport can propagate context; the server parents a run's event spans to whichever invocation opened the socket. Covered in trace-propagation.test.ts, both with and without an active span. Splits the opt-in gate into an import-free ws-transport-enabled.ts so callers can answer it without loading this module (used by the next commit). Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * load the ws transport module only when it is enabled Both call sites read the gate from the import-free module and dynamically import ws-transport.js behind a true result, so a deployment on the HTTP default never pays ws's ~17ms of module init. The queue pre-warm absorbs it for one that opted in, keeping it off the first event write. Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * document WORKFLOW_EVENTS_TRANSPORT as experimental Names the instrumentation gap (no client span per write) and the proxy path where the variable is ignored. Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * correct why the ws accelerators are externalized No bundler fails the build on the unresolvable require — verified against Rollup 4.62. webpack half-bundles the native module and Vite substitutes a stub that makes the require succeed; both leave bufferUtil.mask undefined and throw only once a frame reaches the native masker at 48 bytes, which every CBOR event frame does. Same claim was repeated in the rollup plugin and its test. Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * trim the WORKFLOW_EVENTS_TRANSPORT docs to user level Mirrors the other Vercel World env vars: same facts on both pages, each in its page's format. The instrumentation and socket-lifetime detail belongs in the code, not in a user-facing reference. Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * cover the vite bundler in the ws transport lane Vite substitutes a stub for ws's absent native accelerators rather than failing the require, so nothing catches it until a masked frame reaches 48 bytes — and this job's three existing lanes are esbuild, turbopack and nitro. Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * claim only what is measured about rollup and the ws accelerators The rationale asserted plain Rollup was "safe by accident" via a mechanism only ever observed in a minimal repro. Nitro traces and externalizes `ws` in a production build, so the bundled path is not reached there at all. Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * Give the events socket an explicit lifetime instead of an idle timer `openWsChannel` / `closeWsChannel` bracket one invocation of the flow route, and are the only calls anywhere that create a channel. Writes ask `resolveWsTransport` whether one is open — a lookup now, never a create — and take pooled HTTP when it says no. That removes the reason the idle timeout existed. A lazily-created socket has no owner, so a timer was the only thing able to end it, and the socket is not `unref`'d: the process could not exit, and a server invocation stayed pinned, for the full window past the last write. It also settles `run_created`. The trigger path opens no channel, so a lone write no longer pays for a handshake it cannot amortize — `start()` runs in an arbitrary request handler with no boundary the SDK can see. Refcounted rather than a flag: inline step executions ride the flow topic on per-step topics, so a run's steps can be concurrent invocations in one instance sharing the channel, and the first to finish must not cut the others short. A failed connect closes the channel so the invocation's writes fall back to HTTP instead of each paying its own doomed handshake. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * Name the one reply header the WS path does not map The server copies six headers into an `event_ack`'s meta and this record maps five. The sixth, `X-API-Deprecated`, is inert today — the v4 route's middleware chain has no deprecation middleware to set it — but the record is the only header source a WS reply has, so an unmapped key is gone rather than merely unread, which is not true of the `Response` the HTTP path returns. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * docs: note that WORKFLOW_EVENTS_TRANSPORT=ws is ignored on the proxy path The api-workflow proxy is an HTTP-only REST gateway and does not forward a WebSocket upgrade, so a World configured with projectConfig keeps writing events over HTTP regardless of the setting. Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * ci: gate the ws-transport e2e lanes on a label Three real `vercel deploy`s per run is too much to charge every unrelated PR in the repo for a transport that is off by default. PRs opt in with `ws-transport-test` (or `workflow-server-test`, which already exists to test the half of this the protocol lives in); main keeps the signal on every commit. The required aggregate has to allow the lane to be skipped in that case, so its status is asserted only when the lane was actually supposed to run. Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * chore: regenerate pnpm-lock against current main main resolved `ws` to 8.20.0 as a transitive peer; this branch adds it as a direct dependency of world-vercel and floats it forward, which rewrites every `openai@x(ws@y)` peer key in the lockfile. Merging main textually combined the two, leaving those keys pointing at a `ws` entry the merged file no longer had — `--frozen-lockfile` then failed with ERR_PNPM_LOCKFILE_MISSING_DEPENDENCY on the PR's merge ref. Regenerated from main's lockfile so ours is a minimal delta on top of it. Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * fix(world-vercel): align ws on the version main already resolves The lockfile broke on the PR's merge ref, not on this branch's head: main resolves ws@8.20.0 as a transitive peer, and a `^8.21.1` direct dep here floated it forward, rewriting all 73 `(ws@8.20.0)` peer keys. Git merged the two lockfiles without a conflict but left main-side keys pointing at a ws entry the merged file no longer had, so `--frozen-lockfile` failed with ERR_PNPM_LOCKFILE_MISSING_DEPENDENCY. `^8.20.0` resolves to the copy main already has, so the lockfile delta is the two importer entries instead of a repo-wide rewrite that re-breaks every time main moves. Also keeps one ws in the store rather than two. Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * Bind the channel release to the instance it claimed closeWsChannel resolved the transport by URL, but the refcount lives on the instance. A channel is evicted from the map as soon as it closes — a refused upgrade does that on the connect path — so the next opener for the same run registers a different instance under the same URL, and the first invocation's close then decremented that one instead. It dropped a socket a live invocation was still writing over, and for the event types EVENT_RETRY_ELIGIBILITY marks non-retryable there is no second attempt to carry the in-flight write over HTTP. openWsChannel now returns an idempotent release closed over the transport it incremented, and queue.ts holds that instead of re-resolving the run. The close awaits the open's own promise, so it also can no longer land ahead of the claim it releases. Also names the scope of the connect-failure de-opt: it covers the handshake only, so a channel that connects and then fails every write keeps taking the WS path. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * Decode a transport result, not a Response Main extracted the v4 POST decode into a helper typed `Response` while this branch narrowed the POST result to `FrameResponseLike`, because the WS branch synthesizes its result rather than holding a real `Response`. The two merge without a textual conflict and then fail to typecheck. Widen the helper: it reads only the two members `FrameResponseLike` declares, and a `Response` still satisfies them, so the HTTP call sites are unchanged. Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * Re-run CI Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * Re-run CI Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * Reconcile the WS transport with main's v4 POST rework main moved the materialized POST result off the `x-wf-*` response headers and onto a typed CBOR body, and added a second response shape: two callers now POST with `Accept: application/vnd.workflow.v4-frames` and read back a sentinel-terminated sequence of frames. A frame stream has no representation in a protocol that pairs one reply frame with one request frame, so the WS switch moves off the shared poster and onto `createWorkflowRunEventV4` alone — the materialized write, which is the hot per-step path this branch exists to shorten. `run_started` and the `hook_received` preload stay on HTTP. `decodeCreateEventResponse` takes `FrameResponseLike` rather than `Response` because the WS branch has none to hand over; a real `Response` satisfies the interface, so the HTTP callers are unchanged. The ids now come out of the CBOR body, so `replyMetaToHeaderRecord` no longer maps any `x-wf-*` name — only the two headers `errorFromV4Response` reads. Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * Re-run CI Resample the WS-arm sleepingWorkflow failure: it has now recurred on a second axis (nextjs-turbopack, 7709ms; previously vite, 7570ms), so the arm needs more samples before the skew can be called WS-specific or repo-wide flake. Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * Re-run CI Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * Re-run CI Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * blank * blank --------- Co-authored-by: vercel[bot] <35613825+vercel[bot]@users.noreply.github.com> |
||
|
Nathan Colosimo
|
e6f1b6f548 |
feat(world-local): support Hook minimum retention (#2866)
* feat(core): add hook token retention contract * refactor(core): constrain hook retention options * fix(core): preserve boolean hook visibility options * revert(core): preserve HookOptions interface * docs(core): clarify retained conflict ownership * docs(core): retain newest-wins conflict pattern * docs(core): simplify hook retention guidance * docs(core): explain retained token cleanup * docs(core): simplify idempotency guidance * docs(core): clarify retained token results * refactor(core): rename hook token expiration option * chore(core): name hook expiration changeset * docs(core): simplify Hook expiration language * docs(core): clarify Hook expiration deadline * docs(core): remove Hook deadline caveat * refactor(core): align Hook expiration field names * docs(core): narrow Hook expiration documentation * docs(core): clarify hook expiration availability * Update packages/core/src/workflow/hook.ts Co-authored-by: Peter Wielander <mittgfu@gmail.com> Signed-off-by: Nathan Colosimo <110621881+NathanColosimo@users.noreply.github.com> * docs(core): clarify Hook token expiration behavior * docs(core): explain active Hook expiration behavior * feat(world): advertise hook ttl capability * fix(core): validate hook ttl capability after main merge * refactor(core): rename hook expiry to minimum retention * docs: keep hook retention guidance on v5 * docs: define retained run availability * fix(core): validate Hook retention at creation * feat(core): define retained Hook lookup semantics * refactor(core): simplify hook retention checks * feat(world-local): support Hook token expiration * fix(world-local): make hook recovery atomic * refactor(world-local): align Hook minimum retention * fix(world-local): preserve Hook creation order * fix(world-local): expose retained Hooks consistently * refactor(world-local): simplify retained hook storage * fix(world-local): allow stale lock recovery * refactor(world-local): simplify hook retention storage Signed-off-by: Nathan Colosimo <110621881+NathanColosimo@users.noreply.github.com> * fix(world-local): serialize expired hook token handoff Signed-off-by: Nathan Colosimo <110621881+NathanColosimo@users.noreply.github.com> * fix(world-local): preserve hook creation order Signed-off-by: Nathan Colosimo <110621881+NathanColosimo@users.noreply.github.com> * refactor(world-local): clarify hook availability cleanup * docs: note Local World Hook retention support * fix(world-local): harden hook retention persistence * fix(web-shared): render hook retention deadline * fix(world-postgres): exclude unsupported hook retention * feat(world-local): enforce Hook retention limit * docs(world-local): clarify retention limit error * docs(world): clarify Hook retention deadline * docs(hooks): link retention configuration --------- Signed-off-by: Nathan Colosimo <110621881+NathanColosimo@users.noreply.github.com> Co-authored-by: Peter Wielander <mittgfu@gmail.com> |
||
|
Peter Wielander
|
cb77725960 | [core] Derive correlation ids from per-kind sequences (opt-in) (#3301) | ||
|
Nathan Colosimo
|
5d591d2886 |
perf(core): retain workflow VM across inline steps (primitives-gated) (#3046)
* perf(core): retain workflow VM across inline steps Combines the retained-session architecture from #2984 with the env kill switch and loop-level single-VM test from #2966. - executeWorkflow with discriminated request/result types and a WorkflowSession state machine (running/suspended/failed/replay/completed) - EventsConsumer.append: only newly durable events feed the live VM - WORKFLOW_RETAINED_VM=0 kill switch (default on) - retained-vm-loop.test.ts: proves one VM per run and byte-identical output vs the from-scratch replay path * refactor(core): simplify retained-session control flow - executeWorkflow overloads: a fresh replay request can no longer return { type: 'replay' }, deleting the runtime invariant throw and runWorkflow's dead branch - isSameSuspensionBoundary reduced to the steps-array comparison (all suspension counts are derived from steps in the constructor) - runtime loop initializes workflowResult with a ternary * fix(core): decline retention for VMs that ran host-timed async work crypto.subtle.digest is the only sandbox API whose promise resolves on host timing rather than from the event log, so a workflow racing it against a step can advance while suspended and diverge from what replay reconstructs. A sticky usedHostAsync bit on the VM context makes canRetainWorkflowSession fall back to ordinary replay for such VMs; a quiescent step-only VM remains a pure function of the consumed event prefix and stays retainable. * fix(core): track all host-timed async VM APIs for retention Atomics.waitAsync (a wall-clock timer via SharedArrayBuffer) and the async WebAssembly compilation entry points resolve on host timing just like crypto.subtle.digest. Wrap every such intrinsic in createContext so usedHostAsync covers the complete set; dynamic import() settles within a microtask and cannot advance a suspended VM. * feat(core): compute crypto.subtle.digest synchronously in the sandbox node:crypto createHash produces byte-identical values to WebCrypto and settles the digest promise on a deterministic microtask instead of host threadpool timing. A digest can therefore never advance a suspended workflow, so digest-using VMs stay retainable; only Atomics.waitAsync and async WebAssembly compilation remain host-timed. createHash is stable and undeprecated on Node 18-26 (DEP0179 only removed the direct Hash constructor). * fix(core): remove WeakRef and FinalizationRegistry from the sandbox GC observation depends on host GC timing that neither replay nor a retained VM can reconstruct from the event log. WeakMap/WeakSet stay available (they do not expose GC state). * fix(core): enforce the BufferSource contract in the sandbox digest Reject non-BufferSource digest input with TypeError like WebCrypto does, via the native ArrayBuffer.prototype.byteLength brand check (works across vm realms). Previously a plain number was treated as a Uint8Array length, turning a small input into a giant allocation. * fix(core): demote retention when suspension serialization draws randomness handleSuspension dehydrates step arguments with the live VM, and that serialization can execute user code (getters, WORKFLOW_SERIALIZE hooks). Randomness drawn there would desync the retained VM's future correlation IDs from what a fresh replay regenerates. Count every draw from the seeded stream at its single source in createContext and fall back to ordinary replay if handleSuspension consumed any. * refactor(core): make VM quiescence unconditional, cut tracking machinery Delete Atomics.waitAsync and the async WebAssembly entry points from the sandbox instead of tracking their use — with digest synchronous and GC intrinsics removed, no sandbox API settles a promise on host timing, so a suspended VM provably cannot advance. This deletes the trackHostAsync wrapper, the usedHostAsync bit and session method, the runtime gate clause, the session 'failed' state (unreachable), and the background-progress test scenarios (impossible by construction). * refactor(core): gate retention on passively cloneable step inputs Replace the RNG draw-counter demotion with prevention: when a session is a retention candidate, new step inputs take a passive descriptor walk (never invoking getters; proxies, accessors, functions, custom classes, and platform wrappers decline) and safe values are structuredClone'd into the host realm before dehydration, so serialization never executes workflow-owned code against a retained VM. Unsafe inputs serialize the old way and the session falls back to ordinary replay. * fix(core): harden the passive step-input walker - require enumerable on array index descriptors: structuredClone drops non-enumerable indices that devalue persists - read workflow globals and constructor prototypes via own-property descriptors only, so validation can never execute workflow-owned accessors on redefined globals * fix(core): guard proxied constructors in the passive-input walker constructorPrototype reads both realms' constructors via own-property descriptors only and refuses proxies before any descriptor read, so a proxied redefined global can never observe validation. * fix(core): preserve retention gate after rebase * fix(core): all-or-nothing clone batches; reject SAB views in digest - A mixed step batch (one unsafe sibling input) now serializes every input through the ordinary VM path: a clone snapshotted before an unsafe sibling's serialization runs its getters could otherwise durably capture stale sibling state. - crypto.subtle.digest rejects SharedArrayBuffer-backed views with TypeError, matching WebCrypto's BufferSource contract. * fix(core): narrow the fast path to prototype-independent types devalue serializes Map/Set through the realm's iterator protocol and Date/RegExp/typed arrays through prototype getters, all of which workflow code can mutate — so their serialization is not provably passive and their bytes could differ between retained and cold modes. The fast path now accepts only primitives, plain objects, and plain arrays, which devalue traverses exclusively via own-property reads. Slot-bearing exotics decline even with a swapped prototype. The sandbox digest now reads view metadata (buffer/byteOffset/ byteLength) through captured intrinsic getters, so own properties shadowing them cannot change which bytes are hashed or bypass the SharedArrayBuffer rejection. * fix(core): freeze serialization-consulted sandbox intrinsics instanceof dispatch (Symbol.hasInstance via the constructor, Function.prototype, and Object.prototype), the class reducer's value.constructor walk, and devalue's Object/Array traversal all consult intrinsics workflow code could redefine — legally and deterministically — which would make the durable step input depend on WORKFLOW_RETAINED_VM (spoofed values serialize as e.g. Maps on the cold path but as plain clones on the retained path). Freeze Object/Array/Function (constructors and prototypes), the VM collection constructors, and every reducer-referenced global binding (absent ones pinned to undefined) right before the workflow bundle evaluates, so the retained-input equivalence holds by construction. Host-realm constructor escapes (e.g. TextEncoder.constructor) remain out of the determinism contract: code scheduling host timers was never deterministic under ordinary replay either; documented on canRetainWorkflowSession. * fix(core): freeze every non-shared serialization constructor Typed-array constructors (and their shared %TypedArray% parent), the Date wrapper, and the session-local AbortController/AbortSignal/ Request/Response bindings were pinned but not frozen, so workflow code could still add Symbol.hasInstance statics that diverge reducer dispatch between the retained clone (host constructors) and ordinary VM serialization. Freeze every binding value that is not the shared host intrinsic; shared host objects are dispatched identically by both paths, so mutations there cannot cause mode divergence. * fix(core): build retained clones in a pristine realm Replace structuredClone with an explicit deep copy into an SDK-private realm: clones previously inherited host prototypes, which workflow code can reach (e.g. via structuredClone's return values) and vandalize with Symbol.toStringTag or constructor overrides, shifting devalue's classification of the clone relative to the ordinary VM path. The pristine realm is unreachable by any user code, and the explicit copy serializes exactly what devalue traverses (own indices, own enumerable string props). Arrays also now decline own constructor properties, which the class reducer reads even when non-enumerable. * fix(core): verify host dispatch pristineness before retained cloning Host intrinsics are shared with the whole process and cannot be frozen, but workflow code can reach them (structuredClone results, exposed host classes) and install Symbol.hasInstance predicates that distinguish the original from its clone — or WORKFLOW_SERIALIZE statics on host Object/Array that the class reducer reads for host-prototype originals (hydrated step results). prepareRetainedStepInput now verifies, via own-descriptor reads only, that every host dispatch point is pristine and declines retention before any clone exists — so a spoofed predicate can never observe or capture a pristine-realm object. * fix(core): reject symbol properties from retained step inputs Reducers dispatch on symbol tags (e.g. the workflow abort-signal markers) that are non-enumerable and dropped by the pristine-realm copy, so a tagged object would serialize as an abort descriptor on the cold path but as plain data on the retained path. * fix(core): retained inputs accept only own enumerable data properties Hidden own keys of any kind — non-enumerable properties, accessors, symbols — can be observed by serialization dispatch (reducer probes like .signal, thenable checks, the class reducer) while the pristine clone drops them. With no hidden own keys, every probe on an accepted object resolves deterministically through validated data or pristine prototypes. * fix(core): freeze binding prototype chains for hasInstance lookup Symbol.hasInstance dispatch walks the constructor's prototype chain, so the frozen Date wrapper still exposed the unfrozen original VM Date it delegates statics to. Freeze each non-shared binding's full chain (stopping at host Function/Object prototypes) and verify host Object.prototype carries no added hasInstance on the detection side. * refactor(core): single-path retained serialization via pinned members (v2) Serialize step inputs for retained boundaries through the one ordinary pipeline (original value, workflow global) instead of cloning into a pristine realm and serializing under the host global. With a single serialization event shared by every mode, durable bytes structurally cannot depend on WORKFLOW_RETAINED_VM; the only property retention needs is that serialization executes no workflow code, established by: - the passive walker (descriptor-only, unchanged in spirit), now also accepting Map/Set/Date/typed arrays/ArrayBuffer — the common built-in step arguments — via prototype-identity checks - vm/serialization-pins.ts: the 10 prototype members serialization executes for those built-ins (measured empirically), captured at context creation and identity-verified at each retained boundary; the 'touches only pinned members' test instruments every member and locks the list against serde drift - host-realm instances (hydrated step results) accepted without member verification: host members run host code, which cannot touch retained VM state Deletes the pristine clone realm, the host-dispatch pristineness checks, and the batch clone bookkeeping. * refactor(core): freeze built-in prototypes instead of pinning members (v3) Review found the pin approach's structural hole: the class reducer READS value.constructor through Map.prototype — a data property when pristine (so member instrumentation never listed it), but executable the moment workflow code redefines it as a getter. Pinning what serialization executes misses what it reads. Freeze the accepted built-ins' prototypes wholesale (Map/Set/Date + iterator prototypes, %TypedArray% + subclass prototypes, ArrayBuffer): reads and executes are both immutable, and a patch attempt now throws loudly at the patch site instead of silently degrading. Deletes vm/serialization-pins.ts; the walker requires Object.isFrozen on the realm prototype (also covering realms where the freeze never ran). Also restores the host-dispatch pristineness check the v2 cut lost: workflow code can reach shared host constructors (exposed classes, structuredClone results) and plant workflow-realm Symbol.hasInstance hooks or WORKFLOW_SERIALIZE statics that reducers would execute during retained serialization. Host-realm built-in instances decline for the same reason; host-realm plain data (hydrated results) stays retainable. * fix(core): harden the passivity checker's own execution surface - Capture Map/Set forEach and the %TypedArray% buffer getter as module- load primordials: the checker previously invoked live host methods that workflow code can reach (structuredClone(new Map()).constructor) and replace with delegating workflow-realm closures. - Typed arrays must have one of the realm's real frozen subclass prototypes by identity — 'frozen and chains to %TypedArray%' admitted manufactured frozen hostile prototypes with delegating buffer getters. * fix(core): checker uses module-load primordials; verify inherited serializer statics - The walker resolved Object.getOwnPropertyDescriptor, Reflect.ownKeys, Array.isArray, Number/String helpers, and Object.getPrototypeOf/isFrozen from live host globals workflow code can reach and replace; all are now module-load captures, so the checker can never execute a planted delegate. - The class reducer reads cls[WORKFLOW_SERIALIZE]/cls.classId as inherited Gets, so isHostDispatchPristine now also verifies host Function.prototype and Object.prototype carry no serializer statics. Generic replacement of shared host statics (Object.keys, Array.from, …) via realm escape remains the documented host-reachability boundary, tracked by the realm-local intrinsics follow-up. * fix(core): stale-suspension generation token; cover BigInt toString - Suspension signals capture ctx.suspensionGeneration when scheduled and no-op if the session resumed past that boundary. The harmful interleaving was already unreachable (queue items are deleted on consume, completion writes state synchronously, nextTick precedes timers) — the token turns those ordering facts into an explicit invariant. - The BigInt reducer calls .toString() on primitives from host code, which resolves on host BigInt.prototype: its identity joins the host dispatch check, and the VM BigInt.prototype is frozen besides. * feat(core): deterministic sandbox hardening - crypto.subtle.digest computes synchronously via node:crypto: byte-identical values, promise settles on a deterministic microtask, full BufferSource validation (internal-slot view reads, SAB rejection) - Atomics.waitAsync (a wall-clock timer), async WebAssembly compilation, WeakRef, and FinalizationRegistry are removed from the sandbox — wall clock and GC observation are unreplayable; sync WebAssembly constructors remain - freezeSerializationIntrinsics pins the universal dispatch surfaces: Object.prototype/Array.prototype/Function.prototype are frozen (every missed property read and hasInstance lookup terminates there) and serialization-referenced global bindings are non-writable. Value-type prototypes and constructor statics stay patchable so polyfills (Temporal's Date.prototype.toTemporalInstant, core-js Set.prototype .union / Object.groupBy) keep working — the retained-input gate verifies the members serialization executes per boundary instead. Groundwork for retained-VM replay (#2990). * feat(core): retain the workflow VM across inline steps (primitive args) Keeps the suspended workflow VM, its events consumer, and the paused async stack alive across inline step executions within one invocation. Each loop iteration appends only the newly written events instead of replaying the entire event log in a fresh VM, so step-to-step overhead stays flat as runs grow. - WorkflowSession wraps executeWorkflow: suspended sessions expose resume(events) which appends to the retained EventsConsumer and lets the parked run() continuation settle; any divergence (unexpected suspension shape, consumer error) demotes to full replay permanently - Retention is gated per boundary: only suspensions whose queued step inputs are all primitives (null/undefined/boolean/number/string) are retainable, because serializing primitives executes no workflow code; a follow-up widens this to plain data and standard built-ins - Suspensions with hooks, waits, or attributes always fall back - A suspension generation token invalidates stale timer callbacks from an abandoned suspension so they cannot advance a resumed VM - WORKFLOW_RETAINED_VM=0 kill switch; telemetry records workflow.execution.mode = replay | retained Part 2 of the retained-VM stack (#2990); requires the determinism hardening in part 1. * chore: retrigger vercel deployments * Drop serialization intrinsic freezing from the sandbox The retained-VM passivity design moved from pinning/verifying the sandbox surfaces serialization dispatches on to injecting hardened operations into devalue itself (with taint-based de-opt), so freezing Object/Array/Function prototypes and pinning global bindings is no longer needed. Keep only the determinism hardening (sync digest, removal of wall-clock/GC-observing APIs). * Document and lock in why async crypto.subtle methods cannot break quiescence The remaining async subtle methods reject immediately through the crypto proxy (brand check — the receiver is not a real SubtleCrypto), so they can never mint a host-timing promise. Narrow the quiescence comment to what the code actually enforces and add a test so the unreachability is not silently "fixed" later. * simplify sandbox hardening: lean digest input conversion, async digest, explicit subtle throwers * simplify retention: single decision site in suspension catch, steps-only allow-list gate, drop prepareForRetention param * mark sandbox API removals as a major change * simplify retention further: one staleness mechanism (generation bump on suspend), whole predicate in canRetainWorkflowSession, lazy hook/wait scan, prewarm on resume path * simplify session API and tests: replace executeWorkflow overloads with replayWorkflow/resumeWorkflow, drop low-value events-consumer tests, compact session and retained-loop tests * add parallel-batch retention test (sibling signal absorption) and document the unguarded-signaler invariant * simplify workflow.ts types: 5 named types (WorkflowResult/WorkflowResumeResult), async resume(), rename runtime local to retainedSession * add retention-interleaving e2e (retained/demoted/wait/hook boundaries), drop session telemetry test * discard the retained session on every in-process 412 restart Review finding (both panel reviewers): restartReplayInProcess — added on main by #3145 while this branch was in flight — reset the cached log but not the parked VM session. Any stale-snapshot continue then resumed a session belonging to the discarded log: after a run_completed 412 the completed session's resume() throws and the run is durably failed despite having completed; after a suspension-create 412 the session is resumed without ever passing the retention decision, bypassing both the WORKFLOW_RETAINED_VM kill switch and the step-input gate. A restart now always falls back to a fresh replay. Regression test injects a 412 on run_completed and proves fresh-replay completion (red without the fix). * review round 2: set suspensionGeneration in typed test harness contexts; correct the open-hook/wait scan comment (this suspension's writes are not merged into the cached log — non-step suspensions never reach the scan) * simplify pass: reuse once() from @workflow/utils for the open-hook/wait memo; drop optional-chaining that contradicted the surrounding guards --------- Co-authored-by: Peter Wielander <mittgfu@gmail.com> |
||
|
Shalabh Chaturvedi
|
ba2cddc861 |
[benchmarks] Log the run id and Datadog trace for each sequential-steps run (#3248)
* [benchmarks] Link the run id and Datadog trace under the STSO histograms The STSO distribution section added in #3213 shows the shape of the sequential-steps run but not which run produced it, so investigating an odd-looking bucket meant hunting for the run by deployment id and time window. Capture the identity alongside the samples (the mechanism prototyped on the WIP variance branch, #3107): `/api/bench` returns the trace id of the span @vercel/otel opened for the trigger request, the runner threads it through the sequential iteration and records `sequentialRuns` in the result file, and the renderer prints one line under the histograms with the run id + Datadog trace link for this run and for the `main` run it is diffed against. Every part is optional — a deployment predating the route change yields a bare run id, and a `main` baseline predating this yields only this run's side — so the section degrades instead of breaking on mixed-vintage artifacts. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Co-Authored-By: Shalabh Chaturvedi <7066873+shalabhc@users.noreply.github.com> Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * Log the run/trace links instead of rendering them into the comment The run id and Datadog trace are debugging aids, not part of the benchmark's reported result, so they belong in the job's own output rather than in the PR comment body. Logging them where the runs are produced also makes them available in two cases the comment could never cover: a local `pnpm bench`, and a job that fails before the comment step runs. This drops the comment-rendering side entirely -- `renderSequentialRunLinks`, the `baselineSequentialRuns` baseline plumbing in `annotateWithBaseline`, and the `sequentialRuns` field on the result artifact, which existed only to carry the data to the renderer. The `main`-baseline side of the link goes away with it: which run produced the baseline histogram is only knowable at comment-render time, where the two artifacts are matched. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * Say what the trigger trace actually contains under linked mode The route comment claimed the trigger request's span "propagates into the workflow's own spans". That only holds under WORKFLOW_TRACE_MODE=continuous. Nothing in the workbench or benchmarks.yml sets the mode, so the benchmark deployment runs the default `linked` (packages/core/src/telemetry.ts), where each workflow/step invocation is its own trace root and the trigger's trace carries `workflow.start` plus span links out to those roots. The logged link is still the right entry point -- one hop through the links, which Datadog renders -- but the comment should describe that, so nobody opening a trigger-only trace while debugging a histogram concludes the run produced no spans. Raised by @TooTallNate in review of #3248. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Co-Authored-By: Shalabh Chaturvedi <7066873+shalabhc@users.noreply.github.com> Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * Log a Datadog span search alongside the trigger trace link Under the default linked trace mode the trigger's trace holds only `workflow.start` plus span links, so opening it lands one hop away from the spans an STSO investigation needs. Log an APM search on `@workflow.run.id:<runId>` next to it, which goes straight to the run's execution spans. Both links are logged rather than one replacing the other: the search depends on `workflow.run.id` being an indexed span tag in the org, and the permalink works regardless. Suggested by @TooTallNate in review of #3248. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Co-Authored-By: Shalabh Chaturvedi <7066873+shalabhc@users.noreply.github.com> Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> --------- Co-authored-by: vercel[bot] <35613825+vercel[bot]@users.noreply.github.com> |
||
|
christopherkindl
|
3c7875ad73 |
[docs] upgrade @vercel/geistdocs to 1.19.0 (#3170)
* chore: upgrade @vercel/geistdocs to 1.17.1
Picks up the new footer (Footer no longer takes a config prop), the
heading font-weight change to 450, and the tightened navbar OSS-menu
marks. Also switches the site's own navbar logo from the vendored
geistcn LogoWorkflow fallback to the package's LogoWorkflowSdk, using
its new tuned default height instead of a hardcoded 15.
Fixes a resulting regression: navbarOssProducts entries lacked an `id`,
so resolveOssProducts' `product.id !== activeProduct` filter evaluated
to `undefined !== undefined` (false) for every entry and emptied the
OSS flyout. Added stable `id`/`label` values to each entry.
* refactor: use default navbarOssProducts list instead of a custom override
The manual navbarOssProducts array (with local logo imports/heights) is
no longer needed now that the package's DEFAULT_OSS_PRODUCTS list
already includes all these SDKs with proper id/label/section values.
navbarActiveProduct: 'workflow-sdk' now handles self-exclusion instead.
* fix: use bg-background-200 for docs surfaces to match the template
The docs, cookbook, and v5 route layouts plus the shared DocsLayout
container hardcoded bg-background-100 (pure white), so /docs/* pages
rendered on a lighter surface than the rest of the site. Switch them to
bg-background-200, matching the geistdocs template's page background.
* style: adopt package text-heading-* utilities for homepage headings
The marketing homepage headings hardcoded font-semibold (weight 600)
plus manual responsive sizes/tracking, so they rendered heavier than
the docs headings that now use Geist's 450 heading weight. Swap each
display heading to the package's text-heading-* utilities, which bundle
the 450 weight, line-height, and tracking, mapped across breakpoints to
the nearest design-system size. Inline label/emphasis spans keep their
own weight.
* style: adopt package text-heading-* utilities for worlds headings
Extends the homepage heading change to the /worlds section: the world
listing, detail, compare, and building-a-world pages plus their
components hardcoded font-semibold display headings. Swap each to the
package's text-heading-* utilities (450 weight + line-height +
tracking), mapped across breakpoints to the nearest design-system size.
Mono stat numbers, per-benchmark item labels, and the dialog title keep
their own weight.
* style: remove the bordered grid framing from the homepage
The homepage sections were wrapped in a grid divide-y border-y sm:border-x
container, drawing side borders and divider lines between every section.
Drop that framing so the sections flow with whitespace separation.
* style: remove vertical column dividers from homepage sections
Drop the divide-x column dividers still drawn inside the use-cases
(3-col), feature-grid (2-col), and templates sections, so no vertical
lines remain after the section-grid removal. Section padding keeps the
columns visually separated.
* style: make the homepage "Get started" CTA button rounded-full
* style: use text-heading-* for the feature-grid paragraph text
The two 2-col feature blurbs ("Deep integration with AI SDK.",
"Durable agents by default.") hardcoded their size/leading/tracking
plus font-medium/font-semibold weights. Those manual sizes already
equal text-heading-20/24, so swap to text-heading-20 lg:text-heading-24
— same sizes, but the Geist 450 heading weight (lead drops 600 -> 500
via the utility's [&>strong] rule). The lead stays gray-1000 for
emphasis; body stays gray-900.
* style: fade out the run-anywhere provider logos at the left/right edges
Add linear-gradient masks to the flanking cloud-provider logo groups in
the "Run anywhere, no lock-in" viz so they fade to transparent toward
the outer edges, leaving the centered code block untouched.
* style: widen the right-edge fade on the Vercel dashboard viz
The "Workflow SDK on Vercel" dashboard is offset off the right edge, so
the existing to_left black_10% mask fell off-screen and the visible
right edge hard-clipped. Widen it to black_40% so the dashboard fades
out gradually at the visible right edge.
* style: add spacing between the Vercel, use-cases, and templates sections
Wrap the UseCases and Templates sections with a top margin so there's
clear separation between "Workflow SDK on Vercel", "Build anything with
AI Agents", and "Get started" now that the section dividers are gone.
* style: widen the homepage layout from 1080px to 1200px
* style: align use-cases code block and templates cards with the Vercel section
Switch the "Build anything with" and "Get started" sections from
grid-cols-3 / [1fr_2fr] to [1fr_1.5fr], matching the "Workflow SDK on
Vercel" section above so their code block and cards share the same
right-hand column. The wider text column also lets "Build anything with"
sit on one line. Normalize both to outer padding + column gap so the
code block and cards line up exactly.
* style: extend use-cases/templates content to the right layout edge
Drop the right padding at md+ (md:pr-0) so the code block and template
cards reach the same right edge as the "Workflow SDK on Vercel"
dashboard above, which bleeds to the container edge. Mobile keeps its
padding.
* style: remove the divider between the two feature cards
Drop divide-y/lg:divide-y-0 from the feature grid so no border shows
between "Deep integration with AI SDK" and "Durable agents by default".
* refactor: position homepage sections on a shared 12-col grid
Replace the ad-hoc [1fr_1.5fr] + md:pr-0 + lg:pl-* positioning on the
Vercel, use-cases, and templates sections with a shared grid-cols-12
layout (text col-span-5, visual col-span-7), matching the vercel.com
marketing grid convention. The Vercel dashboard becomes a proper grid
cell instead of an absolutely-offset right-bleed, so all three
sections' visuals align by the grid columns with no magic values.
* style: align homepage width with the navbar content
Widen the homepage container from max-w-[1200px] to the site's
max-w-[1448px] (matching the navbar/footer) and reduce the section
gutters from sm:px-12 to sm:px-6, so section content lines up with the
navbar's content edges (right edge flush at the same column as the
navbar and footer). Also convert the "Reliability-as-code" section to
the shared grid-cols-12 layout (col-span-5 text / col-span-7 code
example), replacing its lg:grid-cols-[330px_1fr] magic values.
* refactor: handle homepage horizontal padding at the root container
Move the mobile/desktop gutter (px-4 sm:px-6) onto the homepage root
container and remove the horizontal padding from every section
component. Section content still aligns with the navbar/footer content
edges, but the gutter is now defined once instead of repeated per
section. Inner-element padding (tab buttons, visual internals) is
unchanged.
* style: left-align content sections on mobile + fix run-anywhere/o11y viz
- Left-align the centered content sections on mobile only (FeatureCardWide,
TweetWall heading, Frameworks, Run-anywhere heading/buttons), restoring
their centered layout at sm and up.
- Make the "Inspect every run" timeline span edge-to-edge by shifting its
gantt from a 14-col grid (content in cols 2-13) to a flush 12-col grid.
- Constrain the run-anywhere viz cluster to the code block width so the
provider cards (AWS/Docker/etc.) overlap behind the code block again.
* style: anchor run-anywhere provider cards to overlap the code block
Position the flanking provider-card groups relative to the centered
code block (right/left calc(50%+140px)) instead of the section edges,
so the cards sit behind and overlap the code block regardless of the
section width.
* style: make the reliability-as-code example fill its column to the right edge
Drop max-w-3xl mx-auto from the workflow/non-workflow code examples so
they fill the col-span-7 cell, aligning the code block's right edge with
the layout's right content edge (matching the tabs and other sections).
* style: split feature-card copy into a title + description
Break the AI SDK / durable-agents feature blurbs into a heading and a
separate muted description with a gap (matching the other sections)
instead of one inline paragraph, and drop the trailing periods from the
feature titles so they read as headings.
* update
* update
* style: give the tweet cards a bg-background-100 surface
* fix(swc-playground): pin monaco-editor to 0.55.1
The lockfile refresh resolved the unpinned `monaco-editor: "latest"` from
0.55.1 to 0.56.0, breaking the workflow-swc-playground Turbopack build.
0.56.0 rewrote its exports map to reroot subpaths under `esm/vs/`
("./*": "./esm/vs/*.js"). monaco-vim@0.4.4 deep-imports
`monaco-editor/esm/vs/editor/editor.api` and
`.../common/commands/shiftCommand`, which now map to
`esm/vs/esm/vs/...` — a path that does not exist. Under 0.55.1
("./*": "./*") both specifiers resolve to real files.
monaco-vim 0.4.4 is the latest published release, so pinning
monaco-editor is the only available fix.
* update
---------
Signed-off-by: christopherkindl <53372002+christopherkindl@users.noreply.github.com>
|
||
|
Pranay Prakash
|
11dc036854 |
ci: stop deploying changeset-release/main, run its e2e against production (#3243)
* ci: stop deploying changeset-release/main, run its e2e against production The changesets action force-pushes `changeset-release/main`, and it can point at exactly main's HEAD SHA. Vercel keeps one commit status per project per SHA, so when both a production deployment (from main) and a preview deployment (from changeset-release/main) are built for the same commit, whichever finishes last owns the status. On 2026-07-30 the preview finished last, so `vercel/wait-for-deployment-action` — which reads the deployment ID out of that status — handed production e2e runs a preview deployment ID and forked runs across environments. Disable git deployments for that branch in every Vercel project rooted in this repo, and give the changeset PR's Vercel e2e lanes a deployment to test that actually exists: main's production deployment for the PR's base SHA, resolved by SHA so a mid-flight production build is waited out rather than silently replaced by an older one. Signed-off-by: Pranay Prakash <pranay.gp@gmail.com> * ci: resolve changeset-release e2e deployments with the wait action, tokenless Per review: with changeset-release/main no longer deployed, main SHAs can never again be deployed to a second environment of these projects, so the per-SHA commit status the action reads is unambiguous for exactly this lane. Reuse vercel/wait-for-deployment-action with environment: production and sha pinned to the PR base SHA instead of the Vercel-API polling script, drop the script and its VERCEL_TOKEN usage, and inherit the action's inactive/skipped-build handling. Signed-off-by: Pranay Prakash <pranay.gp@gmail.com> --------- Signed-off-by: Pranay Prakash <pranay.gp@gmail.com> |
||
|
Nathan Rajlich
|
32ac8e73fd |
Fix Biome lint violations and add Biome CI check (#3222)
* Fix Biome lint violations and add Biome CI check Biome was not configured to respect .gitignore, so ~92% of the 13,355 reported diagnostics came from gitignored build artifacts. Enable VCS integration (useIgnoreFile), apply safe auto-fixes across the repo, fix the remaining mechanical errors by hand, downgrade judgment-call a11y / dangerouslySetInnerHTML rules to warnings, and add a 'biome ci' job to the Lint workflow so violations block PRs going forward. * Use an empty changeset (no behavior change, no release needed) |
||
|
Shalabh Chaturvedi
|
8bda7cef79 |
[benchmarks] Split STSO by inline vs queue-hop steps, add distribution diffs vs main (#3213)
* Split STSO by inline vs queue-hop steps, add distribution diffs vs main
The sequential-steps benchmark's STSO metric mixed two unrelated
phenomena: gaps between steps running back-to-back in the same warm
process, and gaps across an invocation boundary (queue dispatch, client
reinit, event-log replay), which cost ~10x more. The old step-index
windows (1-20 / 101-120 / 1001-1020) sampled 19 gaps each and captured
neither cleanly: whether a boundary happened to land inside a window
moved that window's P99 by hundreds of percent, which is most of the
run-to-run variance the benchmark comment was reporting.
The workflow now tags each step with whether it was the first step body
executed in its process ('queue-hop') or a later one in the same warm
process ('inline') via a process-global, so the split is ground truth
rather than inferred from step index or trace timestamps. STSO is
reported as two rows over *every* gap in the run instead of three
sampled windows. No targets on the new rows — the old ones described the
index-bucketed grouping.
computeStats now keeps the full sorted sample array alongside the
percentiles, and the comment renders a histogram + cumulative-time diff
against `main` under the table, one per STSO kind. Percentiles alone
hide how many samples moved and by how much, which is exactly where the
variance lives. Inline rows use a fixed 50ms bin width (the adaptive
width is coarse enough to hide structure inside that cluster); queue-hop
rows keep the adaptive width. Negative gaps (clock skew between two step
bodies' clocks) get their own bucket rather than being counted with the
slow tail.
Raw samples are stripped from the comment's embedded data block — ~1000
per run would exceed GitHub's comment size limit within a couple of
history entries — so the histogram renders for the current run only,
while collapsed history keeps its tables. Until this lands on `main` no
baseline has raw samples, so the section renders this run's distribution
as a single series.
Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com>
* Clarify what stripping raw samples from the data block does not affect
Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com>
* Drop the bucket tables; fold counts and deltas into the histogram bars
Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com>
* Collapse the STSO distribution section into a dropdown
Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com>
* Fix footer assertion after the dropdown wording change
Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com>
---------
Co-authored-by: vercel[bot] <35613825+vercel[bot]@users.noreply.github.com>
|
||
|
Peter Wielander
|
a09d00135b | Revert "Statically inject workflow world target" (#2752) (#3142) | ||
|
Nathan Colosimo
|
fba26fd9bf | Correct step registration documentation (#3129) | ||
|
Peter Wielander
|
04e5ec9873 | [e2e] Rebuild the event-log corruption repro around step-count divergence (#3147) | ||
|
Nathan Colosimo
|
62d570ed4b | Remove retired v1 step route plumbing (#3061) | ||
|
Nathan Rajlich
|
706b6c41a7 | fix: upgrade postcss to >=8.5.18 to address GHSA-r28c-9q8g-f849 (#3102) | ||
|
Peter Wielander
|
599250771d | [benchmarks/ci] SO payload variants + restructured E2E Test Results comment (#3080) | ||
|
Peter Wielander
|
604aecb021 | [benchmarks] Add SO (stream overhead) scenario and polish test result comment (#3077) | ||
|
Nathan Rajlich
|
f11e9fe56f | fix: upgrade next to 16.2.11 to address CVE-2026-64641 (#3071) | ||
|
Nathan Rajlich
|
9216556bf5 |
fix: upgrade postcss to >=8.5.12 to address CVE-2026-45623 (#3067)
* fix: upgrade postcss to >=8.5.12 to address CVE-2026-45623 * fix: override transitive postcss <8.5.12 to patched version |
||
|
Pranay Prakash
|
9a2770ab34 |
test: regression coverage for hook.resume() from isolated route bundles (o2flow beta.26 incident) (#3001)
* test: regression coverage for hook.resume() from isolated route bundles (o2flow beta.26 incident) Reproduces the o2flow v5 upgrade failure (workflow@5.0.0-beta.26, fixed by #2752 in beta.28): a plain API route importing defineHook() from the root `workflow` entry and calling .resume() failed with Turbopack's "Cannot find module as expression is too dynamic" stub, because the world registration was tree-shaken out of the route bundle and getWorldLazy()'s dynamic-import fallback got stubbed. The bug only manifests when a route bundle loads in isolation (a Vercel lambda): local `next dev`/`next start` evaluates next.config.ts, whose workflow/next import chain registers the world process-wide and masks it — which is why no existing server-driven suite caught it. - route-bundle-isolation.test.ts: production Turbopack build of the nextjs-turbopack workbench, then loads ONLY the compiled route bundle in a bare Node subprocess (cold-lambda simulation) and invokes its POST handler. Fails with the exact incident error on regressed code; passes on main. Wired into the build-error-messages CI job. - e2e: plainModuleDoneHook round-trip through a plain API route on the two Next workbenches (deployed matrix covers real lambda isolation). - Workbench fixtures mirroring o2flow: a directive-less defineHook module shared by a workflow (create) and a plain route (resume). The webpack workbench gets a real route file because `next dev` (webpack) does not serve directory-symlinked app routes. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Pranay Prakash <pranay.gp@gmail.com> * test: authenticate plain hook resume request * test: address review — marker-based harness output parsing, changeset summary - route-bundle-isolation: prefix the harness result line with a unique marker and locate it explicitly instead of JSON.parse()ing the last stdout line, so stray logging from the route bundle or the world can't break parsing; failures now include the full subprocess stdout. - changeset: add a human-readable summary to the (release-less) changeset. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Pranay Prakash <pranay.gp@gmail.com> --------- Signed-off-by: Pranay Prakash <pranay.gp@gmail.com> Co-authored-by: Claude Fable 5 <noreply@anthropic.com> Co-authored-by: Karthik Kalyanaraman <karthik.kalyanaraman@vercel.com> Co-authored-by: Karthik Kalyan <105607645+karthikscale3@users.noreply.github.com> |
||
|
Peter Wielander
|
96719d8220 | [ci] Enable NestJS e2e-vercel-prod and add to docs as "experimental" (#3011) | ||
|
Peter Wielander
|
542138dc0b | [nest] Fix NestJS Vercel build output (#2988) | ||
|
Peter Wielander
|
d53b055a2b | [ci] Run benchmarks in-deployment to avoid proxy overhead (#2967) | ||
|
Nathan Rajlich
|
9da2d76260 |
[core][world][world-vercel] Add World.createRunId() and region-aware queue routing (#1981)
* [world-vercel] Add /run-id sub-export with tagged ULID encode/decode Encodes a tag bit, 5-bit version, and 6-bit Vercel region ID into a ULID-shaped string used for workflow run IDs. Tagged values remain valid 26-char Crockford-Base32 ULIDs so they still sort and round-trip through any system that accepts ULIDs. * [world-vercel] Add string-value assertions to run-id tests Add exact-string expectations for encoded outputs at known inputs, covering the default region/version pair, numeric region IDs, version overrides, boundary values (all-zero, all-max), the dirty-input overwrite case, and the lexicographic-order checks. Also adds an explicit byte-array expectation for the canonical ULID-spec example string and an additional first-char-range coverage test for isTagged. * [world-vercel] Remove internal-repo reference from regions doc comment * [world-vercel] Address PR review feedback on run-id sub-export - isTaggedString now fully validates the input as a 26-char Crockford Base32 ULID (delegating to ulidToBytes) instead of only inspecting the first character. This fixes false positives on inputs like '4UUUU...' that have a valid tag-bit position but invalid chars later in the string. - isTagged() now accepts `unknown` to match its documented behavior of safely rejecting non-string inputs without requiring callers to cast. - Introduce `RegionKey` for the full set of keys including 'unknown', and narrow `RegionCode` to `Exclude<RegionKey, 'unknown'>` so the return type of `lookupRegion` and the `DecodedRunId.region` field accurately reflect that 'unknown' is never produced. Updates `encode` to reject 'unknown' as a region code string at runtime (callers wanting the unknown sentinel should pass numeric 0). * [world] [core] [world-vercel] Add World.createRunId() and region-aware queue routing - @workflow/world: add optional createRunId(input?) to the World interface so worlds can mint run IDs with embedded metadata, and add an optional 'region' field to QueueOptions for per-message routing hints. - @workflow/core: start() now delegates run ID generation to world.createRunId() when defined (falling back to a monotonic ULID otherwise), and accepts a new 'runIdInput' option that is forwarded verbatim to createRunId. When runIdInput.region is a string, it is also threaded onto the queue options so the initial workflow message is dispatched to the matching region. - @workflow/world-vercel: implement createRunId() to mint region-tagged ULIDs, preferring an explicit runIdInput.region and falling back to the VERCEL_REGION env var. The queue now resolves its destination region from (in order): an explicit opts.region, the region embedded in the payload's tagged run ID, the VERCEL_REGION env var, and finally a hardcoded 'iad1' default. This replaces the previous unconditional 'iad1' region passed to the @vercel/queue client. Monotonicity within a process is preserved by tracking the last emitted run ID and bumping the bit immediately above the 11-bit metadata window when a same-ms collision would otherwise occur, then re-stamping the requested region/version on top so metadata remains stable. * [core] [world] [world-vercel] Pass full StartOptions to World.createRunId Drop the dedicated 'runIdInput' field on StartOptions and forward the entire options bag to world.createRunId() instead. This keeps the public API surface smaller and lets each World pick the fields it recognises (e.g. world-vercel reads 'region'). The top-level 'region' option remains on StartOptionsBase and is also threaded onto the queue's per-call region opt when set. * Address review feedback: doc fixes and deterministic same-ms tests - Document the final iad1 fallback in QueueOptions.region (world) - Correct the World.createRunId doc: start() always passes an object - Fix the clientOptions comment: the handler client omits region and relies on SDK auto-detection + the ce-vqsregion header for acks - Fix a misleading QueueClient-construction comment in queue.test.ts - Freeze time in the same-ms monotonicity test so it deterministically exercises the intended path, and add a test covering the bump-above-metadata fallback when the region changes mid-millisecond Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * test: keep workflow-server override rewrite-compatible Export WORKFLOW_SERVER_URL_OVERRIDE while keeping the one-line const shape that workflow-server's cross-repo e2e test automation rewrites. Update world-vercel tests to import that exported value for mock origins and URL expectations instead of duplicating the temporary preview URL. * fix(world): clear region tag bit before ULID timestamp validation Region-tagged run IDs set the high bit of the ULID timestamp byte. The shared world timestamp validator used raw decodeTime(), so current tagged run IDs appeared thousands of years in the future and were rejected before reaching workflow-server. Clear the tag bit before decoding, matching the workflow-server behavior, and cover tagged IDs in tests. * fix(world-vercel): validate tagged runId timestamps via run-id decode Keep @workflow/world's ULID helpers generic; they should not know about world-vercel's region-tagged run ID layout. Instead, world-vercel decodes its tagged runId to the original ULID before using the shared timestamp validator for run_created events. Add a world-vercel regression test that a current sfo1-tagged runId passes validation. * fix(world-vercel): default run ID region to iad1 instead of unknown When neither an explicit region option nor VERCEL_REGION is available, createRunId minted a tagged ULID with the unknown (0) region sentinel, producing the tagged: true, region: null state. The server already resolves unknown/untagged runs to DEFAULT_VERCEL_REGION (iad1), so mint a concrete iad1 tag instead, keeping every run ID self-describing and routable. * test(e2e): use verbose reporter + per-test start heartbeat The default vitest reporter buffers per-file output, so a stalling e2e test produces no output until its timeout — making CI look like a silent 30-minute hang. Switch the e2e CI invocations to the verbose reporter (prints each test result as it completes) and emit a '[e2e] ▶ start:' heartbeat to stdout at the start of every test (bypassing vitest's console buffering) so a stuck test is immediately identifiable in the live CI log. * test(world-vercel): point WORKFLOW_SERVER_URL_OVERRIDE at combined 527+529 preview Temporarily target the workflow-server combined-527-529-preview deployment, which bundles platform-directed multi-region routing (vercel/workflow-server#527, incl. the iad1 hook pin) and durable stream state (vercel/workflow-server#529), so e2e can validate the full multi-region path end-to-end. Revert to empty on main. * fix(core): region-tag the health-check correlationId The health-check response is delivered over a Redis stream whose name (and synthetic run ID) embed the correlationId. Under platform-directed routing the responding endpoint and the polling reader can be served from different physical regions; Redis is physical-region-local, so the correlationId must carry the region for both sides to resolve the same backend. Generate the correlationId via world.createRunId() (a region-tagged ULID) when the world provides it, falling back to a plain ULID for worlds that don't tag IDs (e.g. local, single-region). The synthetic wrun_hc_<id> run ID then carries the region; workflow-server's region middleware decodes it. * Address review feedback: validate region overrides, reset server override - queue: validate opts.region and VERCEL_REGION against the known region table before routing, ignoring unrecognised codes so a bad override can't clobber the payload-derived region (Copilot) - add isKnownRegionCode() runtime guard to run-id/regions - reset WORKFLOW_SERVER_URL_OVERRIDE to '' (must be empty on main) - fold the within-PR iad1-default changeset into the main world-vercel changeset and delete it (review) - start.test: declare specVersion on createRunId mock worlds now that the merged world-compatibility check requires it - cover the new region-validation fall-through paths in queue.test Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test(world-vercel): point WORKFLOW_SERVER_URL_OVERRIDE at wave-1 multi-region preview BRANCH-ONLY — revert the override to '' before merge (lint enforces). Points this PR's e2e/benchmark runs at the wave-1 multi-region workflow-server preview (vercel/workflow-server#590: iad1+sfo1+fra1 serving, staging data backends) so region-tagged runs are validated against real multi-region serving end-to-end. Also makes the unit-test mock origins in events-v4.test.ts and trace-propagation.test.ts override-aware (same pattern the rest of the file and utils.test.ts already use), so the suite passes whether or not the override is set — these two files were the only spots hardcoding https://vercel-workflow.com. * test(e2e): Vercel multi-region suite for start()'s region option Adds a dedicated e2e suite validating @workflow/world-vercel region routing end to end, run as its own CI job (e2e-vercel-multi-region) against the nextjs-turbopack workbench only — deliberately separate from e2e.test.ts, which runs as a matrix across all worlds/frameworks where Vercel-specific multi-region behavior doesn't apply. - workbench/nextjs-turbopack/vercel.json: deploy to iad1+sfo1+fra1 so region-routed flow messages have a function to land on in each region. - workflows/99_e2e.ts: regionProbeWorkflow returns the VERCEL_REGION observed by both the workflow and a step, so tests can assert the run EXECUTED in the intended region (not just that it was tagged). - packages/core/e2e/e2e-region.test.ts: per-region cases assert 1) start(..., { region }) mints a region-tagged run ID (decoded via @workflow/world-vercel/run-id), 2) the workflow + step both observed VERCEL_REGION === region, 3) the server reports the run completed; plus a concurrent all-regions case guarding against cross-region misrouting under simultaneous multi-region traffic. Skips on local deployments. - .github/workflows/tests.yml: new e2e-vercel-multi-region job mirroring e2e-vercel-prod's env/deployment-wait, running only the new suite. * test(e2e): start region probes in-function; fix getWorld await The first multi-region CI run surfaced two issues: 1. sfo1/fra1-tagged runs executed in iad1. The suite started runs from the external test process, which uses the api.vercel.com token proxy — and the proxy's queues path forwards every send to the region-less VQS host (the world's proxy-mode resolveBaseUrl ignores the region argument, and the proxy's x-vercel-vqs-api-url escape hatch only allowlists vqs-server-*.vercel.sh preview hosts). Production traffic publishes IN-FUNCTION (direct regional queue routing), so the suite now triggers start() through a new workbench route (/api/e2e-region-start) and rehydrates the run with getRun() — testing the path production actually takes. Proxy-mode regional queue routing is a known gap to address separately in api-workflow. 2. TypeError on world.runs.get: getWorld() is async and was called without await. * test(e2e): cover explicit and implicit region starts in the multi-region suite With regional VQS routing now working through the api.vercel.com proxy (vercel/api#79056 + #2789 + this branch's per-send region resolution), the suite covers both start configurations, asserting the same three properties for each (region-tagged run ID, execution in the intended region via VERCEL_REGION echoed in the return value, server-side completion): 1. EXPLICIT: start(..., { region }) called directly in the vitest runner — publishes through the token proxy, per-send region carried by x-vercel-queue-region. Restores the direct-start shape the suite had originally, plus the concurrent all-regions case. 2. IMPLICIT: dedicated per-region workbench routes (/api/e2e-region-implicit/{iad1,sfo1,fra1}), each pinned to a single region via a per-function 'regions' entry in the workbench vercel.json, calling start() with NO region option — createRunId derives the tag from the minting function's VERCEL_REGION. The test also asserts the route reported executing in its pinned region, so the implicit-tagging assertion can't pass vacuously. Replaces the interim /api/e2e-region-start route (explicit region via request body), which existed to work around the pre-#79056 proxy gap. * Revert WORKFLOW_SERVER_URL_OVERRIDE to '' — wave-1 multi-region serving is in production workflow-server#590 (iad1+sfo1+fra1 serving) merged and deployed to production and the e2e backend, so this branch's e2e/benchmark runs no longer need to target the wave-1 preview. Restores the empty override the No Test Overrides lint job enforces for merge. The override-aware unit-test origins (events-v4/trace-propagation) stay — they are correct under any override value. * test(e2e): cross-region stream visibility (iad1 writer, sfo1 reader) Regression coverage for a backend bug that made cross-region stream reads report zero chunks on IN-PROGRESS streams (completed streams were unaffected), which forced the multi-region serving rollback. The new case exercises exactly that geometry: - crossRegionStreamWorkflow (99_e2e.ts) writes N chunks to the default output stream, then holds the stream OPEN for 45s before closing — the in-progress window is the point, since completed streams are the easy case. - The e2e starts it with region iad1, waits (same-region, via the api.vercel.com proxy) until all chunks are written, asserts the run is still 'running', then reads through a new sfo1-pinned workbench route (/api/e2e-stream-read/sfo1) that returns getTailIndex() plus its VERCEL_REGION. The reader's region served none of the stream's writes, so the reported chunk count must come from the backend's cross-region stream metadata. The test fails loudly if the route isn't actually executing in sfo1. Also bumps the explicit-region test timeout to 120s: the first case in the file absorbs every cold start at once (fresh workbench instances in up to three regions plus a cold backend preview) and was observed just over the 60s default. BRANCH-ONLY (revert before merge, lint enforces): WORKFLOW_SERVER_URL_OVERRIDE points at a multi-region backend preview that includes the fix, so this validates cross-region stream visibility end-to-end before multi-region serving is re-enabled. * test(e2e): extend multi-region suite to all 19 provisioned regions Points the suite at an all-regions backend preview and widens coverage from the wave-1 trio to every provisioned region: - Explicit path: a single concurrent all-regions case starts one tagged run per region (one shared cold-start window instead of 19 sequential ones) and aggregates per-region failures so a single region's breakage reports alongside the full picture. The trio keeps its detailed per-region cases and the 9-way concurrent-isolation case. - Implicit path: workbench gains a region-pinned /api/e2e-region-implicit/<region> route per provisioned region (19 total, shared handler), the workbench itself now deploys to all of them, and the test.each covers the full set with per-case timeouts for regional cold starts. - Multi-region CI job timeout 20m -> 35m for the sequential implicit cases. BRANCH-ONLY (revert before merge, lint enforces): WORKFLOW_SERVER_URL_OVERRIDE now targets the all-regions backend preview instead of the previous (stale, since-merged) fix preview. * test(e2e): tolerate geo-adjacent execution of queue callbacks The first all-regions run surfaced a subtle execution-locality behavior: queue delivery is guaranteed to the tagged region's dataplane and the delivery callback egresses from that region, but the consumer invocation's execution region is chosen by where that callback enters Vercel's edge — and adjacent regions can geo-resolve to each other's functions. Observed live: kix1-tagged runs (callback egressing from Osaka) deterministically executing in hnd1/Tokyo on both the explicit and implicit paths, with tagging, data placement, and completion all still strictly kix1. expectRunInRegion now asserts execution lands in the tagged region OR one of its geographic neighbors (EXECUTION_ADJACENCY), while run-ID tagging and server-side completion remain strictly the requested region. Gross misrouting (e.g. kix1 -> iad1) still fails. * Revert WORKFLOW_SERVER_URL_OVERRIDE to '' — all-regions serving is in production The all-regions workflow-server rollout is deployed and serving production traffic from every Vercel region, so this branch's e2e no longer needs to target a branch preview. Restores the empty override the No Test Overrides lint enforces for merge. With this the PR is complete: region-tagged run IDs, region-aware queue routing, and the multi-region e2e suite (explicit + implicit + all-regions + cross-region streams) all validate against the production-default backends. * docs: fix three stale comments flagged in review - start.ts: StartOptionsBase.region fallback is iad1, not the unknown sentinel (createRunId always mints a concrete routable region) - queue.ts: example used a nonexistent start({ runIdInput }) API; the real option is start({ region }) - events.ts: decode() clears only the tag bit (top bit of the 48-bit timestamp field) — it does not restore the original untagged ULID; reword to say what actually matters for timestamp validation * test(e2e): cover hook resolve/resume for runs owned by non-iad1 regions Hooks are resolved by opaque token, which carries no region hint, so lookup and resume must work regardless of which region owns the run's data. Exercises the full follow-up-message path on sfo1- and fra1-tagged runs: create inside the workflow, resolve by token from the test process, resume twice sequentially, and assert payload order and completion. Regression coverage for the failure mode where the first message to a hook-driven app on a non-iad1 run worked but every follow-up failed with 'Hook not found'. --------- Co-authored-by: Peter Wielander <peter.wielander@vercel.com> Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |
||
|
Peter Wielander
|
0b956f65cb | Rename experimental_setAttributes to setAttributes (#2882) | ||
|
Peter Wielander
|
25b1509e19 | [rollup] Externalize optional @opentelemetry/api peer (only when absent) so framework builds don't fail (#1947) |