Review follow-ups on #4100:
- The queue-owned row no longer sends a backstop per running step per
replay. The dispatch pass folds every queue-owned running step into ONE
delayed run continuation, due when the latest bare start's lease expires
and keyed `${runId}:queue-backstop:${latestStartedAt}`. The invocation
remembers the epochs it armed and skips the send on later passes over an
unchanged log, so three replay passes over one running step (or one pass
over N running steps) cost a single publish; concurrent invocations
collapse onto one pending wake server-side via the shared key. A coarser
bucket key would either be deduped against an in-flight wake that fires
before a newer step's lease expires, or need a delay past the queue's
per-message cap, so the epoch is the key. A wake on a finished run is the
ordinary already-terminal exit.
- `isQueueOwnedRunning` additionally requires the run's specVersion to be
at or above SPEC_VERSION_SUPPORTS_SLOT_IDENTITY (6), the first spec
version every ownership-stamping runtime mints (#2848 shipped under spec
5). A bare start on an older or unknown run is not proof of a queue
delivery, since the replay contract tolerates a legacy unstamped inline
start, so such runs keep the immediate re-enqueue.
- `workflow.queue_ownership.backstop_wakes_armed` is now 0/1 per pass.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The pending-step dispatch pass re-enqueued every pending step that is not
inline-owned on every replay. That is needed for a step that is created
but never started (step_created proves nothing about whether its message
was sent), but not for a step whose bare step_started is already in the
log with no terminal event: a queue delivery is executing its body and has
not acked its message, and a queue that redelivers unacked messages will
redeliver it if that consumer dies. On such Worlds the immediate re-send
was duplicate traffic, one send per pending step per replay of a fan-out.
Add `capabilities.queueRedeliversUnacked` to @workflow/world, declare it in
@workflow/world-vercel, and have the dispatch pass arm the same delayed
backstop wake an inline-owned step gets (lease remainder, epoch-scoped
key) for such steps instead of the immediate step enqueue. Once the lease
is spent the backstop falls through to the immediate enqueue, so a wrong
guess costs at most one lease. `WORKFLOW_QUEUE_OWNED_BACKSTOP=0` restores
the previous behaviour; the count is reported on the span as
`workflow.queue_ownership.backstop_wakes_armed`.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
On a fan-out that runs some steps inline, the orchestrator invocation
publishes the queued siblings' step-execution messages, runs its inline
steps, falls back into the replay loop, reloads the log, and its
pending-step dispatch pass publishes every one of those messages again
(measured 19-28 re-sends per pass on a 32-branch fan-out, ~400 ms after
the originals). The queue dedupes them by idempotency key, but the sends
still cost round-trips on the shared connection pool and hold the
invocation open past its useful work.
Track, per delivery, the correlation ids this invocation has already
published a step message for (the suspension handler's resilient
publishes plus the dispatch pass's own immediate enqueues) and skip the
immediate re-enqueue for those on later passes, unless a step_retrying
has been observed since (a new schedule). The set is invocation-scoped
and never derived from the log: a step_created does not prove the message
was ever sent, so a different delivery still re-enqueues unconditionally.
Reported as a debug log and the `workflow.dispatch.republish_skipped`
span attribute. The QuickJS engine already keeps the equivalent
invocation-scoped `queuedStepIds` set, so it is unchanged.
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
The only thing a World does with `eventCount` is bump-and-report: when the
write lands above the position named, it reads the events in between and
returns them so the writer can merge them without a second round-trip. The
replay loop and the suspension handler merge that page into their loaded
log. The step executor has no log to merge into, so it took the page's
highest position and discarded the rest.
In production that discarded read fell on a third of all `step_started`
writes (10.8M of 13.4M skipped-slot report reads per day were on executor
event types), each a strongly consistent DynamoDB query on the run
partition with resolved refs, on the response path. This removes the
executor's `knownSlot` / `observeSlot` machinery, the `slotSnapshot`
executor param, and the `batchCommittedSlotCeiling` the suspension handler
computed only to seed it. The loop's and the suspension handler's own
snapshots are unchanged; they consume their reports.
The World contract already describes omitting the count for a caller with
no loaded log to be stale against; the executor now matches it.
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
* perf(world-vercel): batch a fan-out's step-execution queue publishes
A `Promise.all` fan-out dispatched one queue message per branch. Those
publishes ride the shared default undici agent (8 connections, HTTP/1.1,
`pipelining: 1` — see `getQueueDispatcher`), and `handleSuspension` is
awaited in full before the first inline step body runs, so an N-branch
fan-out paid ~N/8 serialized round trips straight onto time-to-first-step.
The `step_created` writes were already batched and HTTP/2-multiplexed; the
publishes were the remaining per-branch round trip.
Adds an optional `Queue.queueBatch`, implemented on `@vercel/queue`'s
`experimental_sendBatch` (0.5.1), and uses it for the batched fan-out fold's
publishes. Each commit chunk now publishes in one request instead of up to
32.
`queueBatch` reports per-entry outcomes rather than throwing, because a
batch can partially fail. `queueMessages` in core keeps the previous
all-or-nothing behavior for this call site: it rejects if any entry failed,
so the delivery is redelivered and republishes the set, deduped by the
per-step `idempotencyKey` the caller already passed. Worlds without
`queueBatch` fall back to concurrent single sends.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* fix(core): reject a short queueBatch result set instead of reading it as success
`queueMessages` only inspected `error`, so a World whose `queueBatch`
returned fewer results than it was given messages reported success for the
whole batch. The omitted entries were never published and nothing raised:
`handleSuspension` resolved, the delivery was acked, and those steps were
never dispatched, so the run stalls with no error recorded anywhere.
Reproduced at 64 branches against a World returning half its results: 32 of
63 steps silently lost.
world-vercel guards this internally and `@vercel/queue` length-checks its
own response, so it was not reachable through the world added here. It is
reachable through the interface `building-a-world` opens to third-party
worlds, which is where the check belongs. Documented on the interface and
in the guide alongside it.
Also notes that the batch grouping degenerates to one request per message
under WORKFLOW_SEQUENTIAL_REPLAYS=1 (per-step physical topics are one of
the routing dimensions groups split on), and corrects the comment claiming
the error's `retryable` flag is consumed downstream: nothing reads it yet.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* fix(world-vercel): carry trace context on each batched queue message
`experimental_sendBatch` injects the active trace context into the multipart
REQUEST headers, and the per-part headers it builds never see it. VQS stores
headers per message and re-emits a stored `traceparent` at delivery as
`x-vercel-queue-traceparent`, which is what lets a consumer attach a span link
back to its producer, so a batched message arrived with no producer context
and its `vqs.process` span got no link. `send()` is unaffected: for a single
message the request headers ARE that message's headers.
At 64 branches that was 63 of 64 step dispatches losing the transport-level
producer link. The run's own step tracing was never affected: that carrier
travels in the message payload (`WorkflowInvokePayload.traceCarrier`), which
is what the consumer builds its trace context from, not a header.
Injects the active context into each entry's headers in `queueBatch` — last,
so it wins over caller-supplied `opts.headers` exactly as the SDK's own
injection does — and honors VERCEL_QUEUE_TRACE_PROPAGATION so that kill
switch still covers both paths. `getTraceContextHeaders()` is factored out of
`injectTraceContextIntoHeaders` so the two share one source.
Verified on the wire against a stub VQS speaking the real batch endpoint:
`traceparent` carrying the producer's traceId/spanId lands on all 64
multipart parts through the real SDK, with the per-message idempotency keys
still alongside it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Karthik Kalyanaraman <karthik.kalyanaraman@vercel.com>
## Summary & Motivation
- Manifest coverage now declares an entry for every matrix app, uses real Vitest skips instead of silent early returns, and fails when a targeted app's manifest is missing, unknown, or unparseable.
- Retries are scoped to deployment e2e runs (`DEPLOYMENT_URL` set), so a flaky unit or integration test can no longer be hidden by a second attempt.
- The stop-workflow cookbook parks on a sleep between iterations, giving the hook an observable barrier to race instead of a fixed delay, and the AbortController hook test waits on queue state rather than a 10ms timer.
- The world-postgres direct-storage fixture drives its run to a terminal state so the conformance worker doesn't recover and replay an unregistered workflow.
- Generated e2e result sidecars are ignored and the committed copies removed; they're CI artifacts, not fixtures.
## Test Plan
Existing coverage runs in CI. With retries disabled: the cookbook agent suite passed 8/8, the two stop-workflow tests passed 10/10, and the AbortController hook replay test passed 25/25 under `CI=1`. The manifest suite skips 52 apps explicitly when nothing is built, and fails on unknown or missing targeted apps. The Docker-backed Postgres spec could not run locally (Testcontainers found no container runtime); `@workflow/world-postgres` typechecks.
## Summary & Motivation
Keeps WebSocket setup off the first stream group, then starts the background upgrade when the second HTTP request is dispatched. Later groups continue over HTTP without waiting until the socket is OPEN, when the serialized writer switches transports at a confirmed request boundary. One-group streams never create a socket.
An ambiguous HTTP outcome poisons the writer and retires any provisional socket before it can carry a frame. Initial upgrades have a dedicated 10-second background timeout; the existing 250ms bound remains scoped to reconnects after an established socket closes. WS write spans include chunk sequence and count for direct takeover analysis.
## Test Plan
Tests cover dispatch overlap, continued HTTP writes while connecting, one-group streams, close during background connection, ambiguous HTTP outcomes, timeout and late OPEN behavior, and trace propagation. The world-vercel suite and typecheck pass locally.
## Summary & Motivation
While the first socket is still connecting, complete groups go over HTTP instead of parking on the handshake; the socket takes over once it opens. An HTTP-first write that fails poisons the writer rather than falling back, since its outcome may be unknown and replaying it over WS could duplicate a group.
## Test Plan
Tests added for transport switching, ordering against close, and the poisoned-writer path; the world-vercel suite passes locally.
## Summary & Motivation
### Situation
- Workflow stream writers are expected to call `releaseLock()` when a step finishes writing so another step can acquire the stream.
- The runtime observes that release and drains the server sink, but `step_completed` currently races the overall stream operation against 500ms.
- A slow PUT can therefore continue under `waitUntil` while the next step starts and reads a stale tail.
- Release can also happen while native `writer.write()` promises remain unsettled, leaving frames upstream of the server sink when a naive drain runs.
- Writers intentionally kept locked must remain non-blocking so producer and consumer steps can overlap.
### Fix
- Treat a writer released before step return as an implicit durable handoff boundary.
- At step end, acquire the unlocked stream with a temporary writer and enqueue an internal checkpoint behind all writes queued by the released writer.
- Once the checkpoint crosses serialization, wait for those frames to reach the server sink and drain the group-commit PUT before `step_completed`.
- If the writer remains locked, do not wait for durability; preserve the existing 500ms inline-loop heuristic and background `waitUntil` lifecycle.
- Drain failures or the 30-second safety timeout fail/retry the step. Client disconnect errors remain non-fatal.
## Test Plan
- Covers released and held locks, release with unsettled writes, delayed first writer acquisition, forwarded writable arguments, drain timeout/failure, and multiple streams.
- `pnpm --filter @workflow/core build`
- `pnpm --filter @workflow/core typecheck`
- `pnpm --filter @workflow/core test` — 2,405 passed, 3 expected failures, 1 skipped
## Summary & Motivation
Adds three bounded trace attributes so a sampled trace says how a `step_started` claim was made: `workflow.step_start.strategy` on the step span (`awaited` / `optimistic` / `batch_preclaimed`, set before the write so a losing claim keeps it after its 409 reconciles to `skipped`), and `workflow.step_start.mode` plus `workflow.step_start.owner_stamped` on the world-vercel write spans across the http, batch, and ws paths. Purely additive telemetry — no change to execution behavior.
## Test Plan
Test added covering a stamped lazy claim's attributes on the per-write span; existing coverage runs in CI. Local run of 81 focused core and world-vercel tests passed; full core typecheck is blocked by unrelated workspace resolution issues.
## Summary & Motivation
Tags the first write of a session, and the first write on each reconnect, with phase timings on the existing `workflow.stream.write` span — token/config resolution, connect, wait-for-open, send, and ack round trip — so cold WebSocket write latency can be attributed to a client-side phase before changing first-chunk transport behavior. Later writes keep the two attributes they had, since the per-phase clocks only mean anything while a connection is being established.
## Test Plan
Unit tests added; existing world-vercel suite and typecheck pass.
## Summary & Motivation
Implements the client half of `workflow-stream-ws/v1` behind the existing default-off `WORKFLOW_STREAMS_TRANSPORT=ws` gate, populating the `createWriteSession` seam only when opted in.
- Writes and closes are serialized over one socket per writer lifetime; groups above the v1 per-request chunk cap are split without resetting writer-local sequence.
- Any failure before the upgrade is accepted (declined upgrade, proxy, load error, a dispatch that beats the handshake) falls back to HTTP for the rest of the writer's life.
- Once a write is on the socket, a missing or uncorrelatable reply poisons the session rather than replaying over HTTP, since a duplicate append cannot be ruled out.
- Idle clean closes reconnect with the same writer id, capped at three attempts so a draining server can't hot-loop.
- The handshake gets a `workflow.stream.ws.connect` span and each frame a synthesized `http POST` span, so per-event tracing survives the non-`fetch` transport.
## Test Plan
New unit tests cover the lifecycle, fallback, and poisoning paths; 598 `@workflow/world-vercel` tests plus package build/typecheck and workspace lint/format pass. Root build/typecheck couldn't run locally (missing Rust toolchain for the unrelated `@workflow/swc-plugin`).
* [core] Make `hook.metadata` a lazy Promise getter
Hydrating a hook's metadata is a decrypting READ: it needs the owning
run's payload keys, and resolving those costs a run fetch plus a
`run-key` API round trip (~350ms). `getHookByToken()` did that work
eagerly on every lookup that found a metadata-bearing hook, so callers
that only wanted `runId`/`token` — and hook resumption, which never
reads metadata at all — paid for it anyway.
`metadata` is now a getter returning a memoized Promise, the same shape
as `run.returnValue`. The lookup is one read again; hydration and the
key resolution behind it happen on first access, or never.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Pranay Prakash <1797812+pranaygp@users.noreply.github.com>
* docs: surface the lazy hook.metadata change in What's new, the migration skill, and the resumeHook reference
Adds the breaking-change row to the v5 What's new page and puts that page
in the sidebar as the first visible entry (the /v5/docs redirect to
getting-started is unchanged). Teaches the migrating-workflow-v4-to-v5
skill the `await hook.metadata` rewrite and bumps its version. Points the
resumeHook reference at HookWithLazyMetadata, and notes on the World
storage page that world.hooks.getByToken() returns raw serialized
metadata.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* [core] Export the lazy-metadata hook type as `Hook` from `workflow/api`
`getHookByToken()` and `resumeHook()` return `Hook`, not a separate
`HookWithLazyMetadata`: one public hook type whose `metadata` is a lazy
Promise, mirroring `Run` for runs. The World-level record from
`@workflow/world` is unchanged and is referenced as `WorldHook` inside the
runtime.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* [core] Define the lazy `metadata` getter in place; tighten changeset and docs wording
Review feedback: the hook record a World returns is a fresh object per
lookup and the eager path mutated it anyway, so define the getter on it
directly instead of copying it with Object.create(). The changeset is one
sentence, and the docs describe hydration as extra network round trips
rather than decryption, since not every World encrypts.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
---------
Co-authored-by: vercel[bot] <35613825+vercel[bot]@users.noreply.github.com>
Co-authored-by: Pranay Prakash <1797812+pranaygp@users.noreply.github.com>
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
## Summary & Motivation
Gives one in-memory stream writer a stable identity and its own sequence space, so a transport can preserve chunk ordering across a mid-stream HTTP/WebSocket transition. `Streamer.streams.createWriteSession` is optional — Worlds that don't implement it keep using `write`/`writeMulti`/`close` unchanged.
Abort disposes the session rather than closing it, since a producer failure is transport cleanup, not stream completion.
## Test Plan
Tests added, plus the full `@workflow/world-vercel` suite and package builds/typecheck pass. Root build/typecheck is blocked locally by a missing Rust toolchain for the unrelated `@workflow/swc-plugin`.
* fix(world-vercel): honor WORKFLOW_NODE_HTTP on the queue transport
getQueueDispatcher was the one dispatcher getter that ignored the flag. The
reasoning was that `undefined` cannot move the queue client onto node:http
(QueueClient takes a dispatcher and no fetch override), so returning it would
only drop this package's pool tuning and fall back to undici's global agent.
That misses what the flag is actually for. The deployments that need it are the
ones where the undici copy *this package bundles* is unusable, and `undefined`
does move the request off that copy: global fetch dispatches on the runtime's
own undici instead. On such a deployment every other request survives while the
queue client keeps dispatching through the broken copy, and an
acknowledgeMessage that never resolves means the message is redelivered for as
long as the platform keeps killing the invocation holding it.
Losing pool tuning is the correct trade under a flag whose premise is that the
bundled undici is not usable here. An explicit config.dispatcher still wins.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* docs: correct the queue-send note for WORKFLOW_NODE_HTTP
The queue client is a partial exception to the flag, not a full one: it cannot
move to node:http, but it does honor the flag by dispatching through the
runtime's own copy of the HTTP client library instead of the copy the World
bundles. That distinction is the whole point when the bundled copy is what does
not work.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
## Summary & Motivation
`WORKFLOW_STREAMS_TRANSPORT=ws` advertises client support for `workflow-stream-ws/v1` on stream writes; anything else keeps HTTP. It's a capability signal only — the server decides each upgrade, so there's no version or tenant-policy heuristic on the client side.
## Test Plan
Unit tests for the gate's accepted values; typecheck, build, and lint pass locally.
## Summary & Motivation
Lets a long-lived run own a shared stream while independent runs append through `run.writable` or `run.getWritable()` using only the owner's run ID. The handle seals to the owner's public key when it has one, so it grants append access without read capability, and carries the existing forwarding symbols so passing it through `start()` and into a step keeps the owner's identity.
## Test Plan
Tests added
## Summary & Motivation
Defines the `workflow-stream-ws/v1` frame schemas and encoder in `@workflow/world-vercel`, versioned independently of the REST API and workflow spec so a framing or acknowledgement change needs a new endpoint rather than a spec bump. Nothing calls it yet.
## Test Plan
Protocol tests added, including a byte-for-byte check against workflow-server's canonical frame fixture.
* fix(swc-plugin): register class expressions via an IIFE and reject unnameable classes
Class expressions with "use step" methods or custom serialization were
registered by module-level statements referencing the class by name. When no
module-scope binding could be resolved the plugin fell back to a placeholder
`AnonymousClass` identifier, which is a guaranteed ReferenceError at module
evaluation (vercel/workflow#3929). Other shapes were silently wrong as well:
`var A = class {}, B = class {}` registered A's steps under B, `X = class {}`
assignments and classes nested inside functions emitted unresolvable
references.
Class expressions are now wrapped in a single IIFE that receives the class,
performs every registration recorded for it, and returns it, so the
registration no longer depends on a name being in scope. The class name is
still needed for step/class IDs and is derived from the assigned variable,
the class's own identifier, or the property key it is assigned to
(`exports.Foo = class {}`, `{ Foo: class {} }`). When none is available, or
the class is declared inside a function, the plugin emits a compile error
instead of broken code.
Class declarations keep their existing module-level output; the emitters
were factored so both paths share the same statement builders.
* fix(swc-plugin): generate names for anonymous class expressions instead of erroring
With registration happening inside the IIFE, an anonymous class expression
in a position that provides no name (`foo(class { ... })`, an array element,
a conditional branch) only needs a name for its step/class IDs. Generate a
deterministic `AnonymousClass<N>`, counting only anonymous classes that have
something to register, instead of rejecting them. Classes declared inside a
function remain an error.
Dead-code elimination now keeps module-level declarations whose initializer
contains a wrapped class expression: evaluating the initializer is what
registers the class, and the binding may be otherwise unreferenced.
Copy-edit only, no behavior or API changes:
- "it's a only a short step" -> "it's only a short step" (ai/index)
- "When you tool needs" -> "When your tool needs" (ai/defining-tools)
- "Workflow operation that suspend" -> "operations that suspend" (ai/sleep-and-delays)
- "extend out ... to use emit" -> "extend our ... to emit", and
"other tools calls ... inject out own" -> "other tool calls ... inject our own"
(ai/streaming-updates-from-tools)
- "non-yet-standard" -> "not-yet-standard" (how-it-works/understanding-directives)
- "apps ... and needs no special configuration" -> "and need no special
configuration" on the five getting-started pages that disagreed with the
other five (express, fastify, hono, nuxt, vite)
- drop the orphan "needed." line after "No separate command is required."
and fix "the local installed version" -> "the locally installed version"
(observability/index)
- "Time between emissions of a chunk" -> "emission of a chunk"
(observability/tracing)
- "determine that is safe" -> "determine that it is safe" (whats-new)
- "three rules bind an implementation" -> "four rules": the list has four
bullets, and skills/migrating-world-v4-to-v5 already says four
(worlds/upgrading-to-v5)
- drop the stray duplicate "Workflow" before the Workflow SDK link in
@workflow/swc-plugin's README, and the duplicated horizontal rule before
"## Detect mode" in its spec
Each fix is applied to both the v4 and v5 copies wherever the same text
exists in both.
Co-authored-by: vercel[bot] <35613825+vercel[bot]@users.noreply.github.com>
`changeset version` assembles a release plan from every pending changeset
before it bumps anything, and throws on a changeset it cannot place there:
one naming a package outside the workspace, or one mixing a package from
the `ignore` list with published ones. #3938 shipped the latter and every
push to main since has failed to publish (#3963). Nothing at PR time ran
that step.
scripts/check-changesets.mjs runs the same assembly on the same inputs,
resolving the libraries from @changesets/cli's own install so the check
uses exactly the versions the Release job does, and stops before the
network-bound changelog generation. lint.yml runs it on every PR.
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
## Summary & Motivation
Stream infrastructure failures (HTTP/2 session wedges, transport timeouts, non-2xx stream responses) surfaced as plain `Error`, so terminal classification attributed them to customer code as `USER_ERROR`. They now carry a catchable `StreamError` with a `STREAM_ERROR` run error code, attributed to the SDK and retried when transport-level or 5xx.
The v4 events response body is wrapped so a post-header stream failure is classified and reported to the dispatcher recycler — a response header arriving is not yet a successful streamed request.
## Test Plan
Unit tests added across classification, serialization round-trip, the streamer, and the v4 transport; 331 `@workflow/core` and 123 `@workflow/world-vercel` focused tests pass.
Changesets refuses a changeset that mixes packages in the `ignore` list
with published ones, so `changeset version` has failed on every push to
main since #3938 and nothing has been published.
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Pranay Prakash