* Statically inject workflow world target
* Fix static world injection in host bundles
* Fix static world injection gaps
* Fix Vite Nitro server startup
* Fix Nitro pg-native aliasing
* Fix static world target CI gaps
* Fix static world dev rebuild gaps
* Avoid broad runtime alias in Nitro
* Refresh Next dev route for step HMR
* Externalize Nest target world
* Use canary HMR rediscovery timeout
* Bundle local world in Nest builds
* Dedupe world target helpers and fix SvelteKit chunk patch guard
* Optimize Next dev HMR rebuilds
* Fix Next dev HMR CI coverage
* Gate dev HMR logs behind opt-in flag
* Match workflow dev build logs to Next style
* Fix Next dev HMR changed-file classification
* Fix Windows port detection
* Relax HMR log wait in dev e2e
* Avoid canary workflow execution cache flakes
* Allow slower Turbopack HMR propagation in e2e
* Scope canary HMR fuzz execution assertions
* Redrive on transient workflow-server transport failures instead of failing the run
A firewall in front of workflow-server shedding load with sustained 429/503
makes undici's shared RetryAgent exhaust its retries and throw
UND_ERR_REQ_RETRY. That raw error was rethrown unwrapped, so it was
classified as USER_ERROR and the replay terminal branch wrote run_failed —
permanently failing a run on a transient blip (or, in an outage, falling back
to the ~5min queue visibility-timeout redrive).
- world-vercel: map exhausted-retry / socket / connect / DNS / timeout
failures to a typed WorkflowWorldError (code TRANSPORT/TIMEOUT) by walking
the fetch() cause chain.
- core: add isRetryableWorldError (429 / 5xx / TRANSPORT / TIMEOUT) and
rethrow such errors from the replay terminal branch so the queue redrives
quickly (1s->60s backoff) instead of failing the run. Reuse it in start()
and step_started handling.
- world-vercel: surface the Vercel firewall x-vercel-mitigated
(challenge/deny) header alongside x-vercel-id in error diagnostics and logs.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* Address review: back off + cap on every retry path; fix mock; refine scope
Revises the transport-error handling per PR review (VaguelySerious,
karthikscale3).
Blocking fix — step_started no longer self-enqueues a throttled defer for
transient world errors. Returning `{ type: 'throttled', timeoutSeconds: 1 }`
acked the delivery and enqueued a fresh message, resetting the delivery count
so the path never backed off and never reached MAX_QUEUE_DELIVERIES — an
unbounded flat-1s loop if step_started kept failing. It now throws, so the
error flows through the replay loop's retryable-world-error rethrow and earns
both the delivery-count backoff and the max-delivery cap. Throwing is safe on
step_started (the body hasn't run; a write that landed dedupes to skipped).
Also in this revision:
- Backoff that lasts: raise the queue handler-error retry ceiling 60s -> 900s.
VQS clamps each redelivery to its 900s SQS limit and adds its own post-32
exponential, so ramping our base toward 900s stretches survival from ~3.7h to
most of the 24h message-visibility window. Corrected the stale
MAX_QUEUE_DELIVERIES comment to match the real VQS schedule.
- Stop amplifying firewall challenges: the undici RetryAgent no longer retries
429 in-process (a challenge is a 429 the client can't solve). 429s surface
immediately as ThrottleError carrying x-vercel-mitigated / x-vercel-id, so
the diagnostic header now reaches us for the challenge case too.
- Track world faults as WORLD_CONTRACT_ERROR (not USER_ERROR) in
classifyRunError so an outage isn't attributed to user code.
- Fix queue.test.ts mock that `biome check --write` had rewritten from a
newable `function` into an arrow (broke `new QueueClient`); pin with a
biome-ignore.
All Vercel-specific logic stays in @workflow/world-vercel; @workflow/core
operates only on the generic WorkflowWorldError abstraction.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* Update .changeset/transport-error-redrive.md
Signed-off-by: Peter Wielander <mittgfu@gmail.com>
* Trim changeset to a single sentence per review
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* Route firewall-challenge 429s to the retryable transport path, not ThrottleError
A 429 carrying `x-vercel-mitigated: challenge` is a firewall challenge our
server-to-server client cannot solve, so it recurs for the life of the
incident. Mapping it to `ThrottleError` meant the `step_started` write deferred
it as `{ type: 'throttled' }`, which self-enqueues a FRESH queue message and
resets the delivery count — so it never backed off past `retryAfter` and never
reached `MAX_QUEUE_DELIVERIES`, hot-looping against an already-overloaded
firewall (the exact amplification this PR set out to remove, and contrary to
the "step_started can't loop unbounded" invariant, which only held for 5xx).
Map a challenge to a retryable transport `WorkflowWorldError` (`code:
'TRANSPORT'`) in both the v3 `makeRequest` and v4 `throwForErrorResponse`
(the hot event-write path) error mappings, via a shared `isFirewallChallenge429`
helper. It then propagates through the V1/V2 step paths and the replay loop's
retryable-world-error rethrow, earning the delivery-count backoff AND the
delivery cap. A genuine application-level 429 (no `challenge` mitigation) stays
a `ThrottleError` and keeps its `Retry-After`-paced defer.
Also correct the survival-window comments: with the 900s ceiling,
MAX_QUEUE_DELIVERIES=48 spans ~9-10h (~35,000s), not "the better part of 24h";
reaching 24h would need a higher delivery cap, not a higher per-hop ceiling.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Signed-off-by: Peter Wielander <mittgfu@gmail.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Co-authored-by: Peter Wielander <mittgfu@gmail.com>
Co-authored-by: Peter Wielander <peter.wielander@vercel.com>
* perf(core): memoize step return value hydration across replays
The inline replay loop re-executes the workflow body and re-consumes the
full event log on every iteration. For each already-completed step, the
step consumer re-decrypted and re-devalue-parsed the serialized result on
every replay — O(N^2) decrypt+parse operations across a single
invocation of a sequential N-step workflow.
Add a per-run memoization cache, owned by the inline loop in runtime.ts
(alongside cachedEvents) so it survives across replay iterations of the
same run but never leaks across runs. It is threaded into runWorkflow and
stored on the orchestrator context, and consulted in the step_completed
path keyed by the persisted event id. This makes a completed step's
hydrated result O(1) on subsequent replays, turning the aggregate cost
into O(N).
Determinism is preserved: the cache lookup happens inside the existing
ctx.promiseQueue slot and still resolves via the same resolve(), so a
cache hit occupies the identical position in the ordered delivery chain a
re-hydrate would have — pendingDeliveries accounting, delivery barriers,
and Promise.race/all replay are untouched.
Identity safety: hydrateStepReturnValue returns a fresh object graph each
call and each replay runs in a fresh VM, so sharing an object reference
across replays could let one replay's mutation leak into the next. Only
primitive results are memoized (immutable, reference-share == re-parse);
non-primitives re-hydrate fresh every replay, exactly as before. Hook,
wait, and abort hydration paths are intentionally left uncached.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* perf(core): bound memoized step-hydration cache by primitive size
Address the review note that the per-run step hydration cache was never
size-bounded: cached entries hold the decrypted/parsed plaintext of a
primitive step result for the whole invocation, on top of the serialized
bytes already retained in cachedEvents, so a long run returning large
strings could roughly double peak retained memory for those results.
Document the cache's memory characteristic (per-invocation, freed when the
invocation ends, bounded by primitive-returning step count) and cap the
only primitive types that can carry a large payload: string/bigint results
longer than MAX_MEMOIZED_PRIMITIVE_LENGTH (4 KiB) fall through to the
existing per-replay re-hydrate path instead of being memoized. Large
payloads are cheap to re-hydrate relative to their footprint, so this caps
the worst case at negligible cost. Other primitives are inherently small
and always memoized.
The cap only ever reduces what is cached, so deterministic replay is
unaffected: oversized values take the already-correct re-hydrate path.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
On the first delivery of a run's first invocation, background run_started,
skip the initial event-log load, and force optimistic inline start so the run
reaches its first steps with no preceding network round-trips. Safe because the
first delivery has no concurrent handler to race the step create-claim; turbo
exits the moment a suspension creates a hook or wait, and is a no-op for every
other invocation. On by default; disable with WORKFLOW_TURBO=0.
* Default source maps to dev-on / prod-off
Inline source maps are embedded in the step bundle and the intermediate
workflow VM bundle, which bloats production function bundles (a problem for
the Vercel 250MB limit) even though maps only help when reading a stack trace.
Make the default environment-aware in @workflow/builders: inline in
development (next dev / nitro dev / Vite-based dev servers, detected via
config.watch or NODE_ENV=development) and off in production. The `sourcemap`
config option and `WORKFLOW_SOURCEMAP` env var still override in either
environment. A production build with no override also drops the
source-map-support shim from the Vercel step function.
Keep runtime stack remapping graceful and fast when maps are absent
(@workflow/core): short-circuit when no frame references the workflow file
and memoize the parsed map (or its absence) per bundle, so production failures
don't rescan the bundle.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* test(e2e): make source-map expectations match dev-on/prod-off default
The e2e error-stack tests gate source-map assertions on hasWorkflowSourceMaps()
and hasStepSourceMaps(). Now that source maps default to off in production
builds, update those helpers:
- hasWorkflowSourceMaps(): false for all production builds (local prod,
postgres, Vercel — keyed off DEV_TEST_CONFIG), and exclude nest in dev (the
Nest integration builds with watch:false / no NODE_ENV=development, so its
bundles have no maps).
- hasStepSourceMaps(): nest now resolves to a production build (maps off) in
both dev and prod.
Add unit cases for the dev-vs-prod and nest behavior.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
* otel: nest linked-mode invocations under the delivery context
Follow-up to the linked-trace mode shipped in #2363. In linked mode the
queue-delivered workflow.execute / step.execute spans were created as new
trace roots (root: true) with span links to both the delivery context and
the run-origin context. That split a single local invocation across two
traces: the framework route/server span and the workflow execution span
ended up in different traces connected only by a link.
This nests each invocation under its local delivery context instead:
- Drop `root: true` on the queue-delivered workflow.execute / step.execute
spans so they become children of the active context — the framework
route/server span when one exists, otherwise a clean root. One invocation
(route handler, replay, inline steps, event writes) is now a single
bounded trace.
- buildInvocationSpanLinks in linked mode now returns only the run-origin
link; the delivery context is the parent, so it is no longer also a link.
The run-origin context remains a link (never a parent) and re-enqueues still
forward the original carrier unchanged, so a long-running run is still never
stitched into one giant trace across invocations. continuous mode is
unchanged. Everything remains a no-op when no OpenTelemetry SDK is
registered, and there is no dependency on any particular framework: with no
route/server span active, the invocation span is a clean root rather than an
orphan.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Update .changeset/nest-linked-invocations-under-delivery.md
Co-authored-by: Peter Wielander <mittgfu@gmail.com>
Signed-off-by: Karthik Kalyan <105607645+karthikscale3@users.noreply.github.com>
---------
Signed-off-by: Karthik Kalyan <105607645+karthikscale3@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: Peter Wielander <mittgfu@gmail.com>
Derive the deterministic RNG seed from `runId:workflowName:deploymentId`
and the VM's initial fixed clock from the ULID timestamp embedded in
`runId` (via the new `runIdCreatedAt` helper). All of these inputs are
available the instant a queue message arrives, so the VM seed and clock
no longer depend on `startedAt` (set only after the `run_started`
round-trip). This is the prerequisite for starting VM initialization
earlier on the critical path.
This changes the seed-derived value sequence for a given run, so the
affected deterministic test fixtures are regenerated accordingly.
createHook() used `options.token ?? ctx.generateNanoid()`, so a nullish
token fell back to a generated one but an empty string `""` was accepted
verbatim — a meaningless, non-deterministic token that is almost always
an accidental value (e.g. an unset variable). Throw a clear error when an
explicit empty-string token is passed; `undefined`/`null` still
auto-generate, and non-empty strings are unchanged.
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
* perf(core): cache compiled workflow-bundle vm.Script across replays
The inline replay loop calls runWorkflow on every iteration, and each call
re-parsed the entire workflow bundle string via vm.runInContext. For a bundle
containing many workflow definitions (the production shape: one workflow called
per replay), this re-scans every definition on every replay.
Cache the compiled vm.Script per process, keyed by (workflowCode, filename),
and run it against the fresh context instead of recompiling. Compilation is a
pure function of (code, filename), so the result is byte-identical to the
previous re-parse-every-time behaviour — determinism is preserved. filename is
part of the key because it drives source attribution in stack traces (consumed
by remapErrorStack).
Measured per-replay savings scale with bundle size (and multiply by replay
count): ~34% for a 50-workflow app, ~59% for 155 workflows, ~80% for 400.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* perf(core): bound script cache with LRU; soften determinism claim; add tests
Addresses review on #2471:
- Bound `scriptCache` to a small LRU (cap 8 bundle versions). Production
serves one bundle per process so the bound is never reached; it exists for
dev/watch mode, where each edit produces a new bundle string that would
otherwise be pinned forever (~0.8MB/edit, monotonic). Touch-on-access keeps
the latest bundle hot; evicting a `code` entry drops its per-filename scripts
together, restoring pre-cache GC behaviour.
- Document precisely why keying includes `filename` (intentional: drives
stack-trace attribution via `remapErrorStack`; NOT a dedupe key), and that
the whole bundle is compiled once per distinct filename.
- Soften the "byte-identical including thrown errors" claim to
same-workflow-function + same-`filename`-attribution, noting the one caveat:
a lookup-expression error's line number shifts to line 1 of the separate
lookup Script. Updated in both the code comment and the PR description.
- Add tests: cache-is-bounded regression (eviction past the cap), LRU recency
(hot bundle survives churn), and a realistic multi-workflow collision test
(distinct code/filename never returns the wrong Script, results carry their
own bundle marker).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
* docs(core): document scheduleWhenIdle macrotask is load-bearing
Revert the synchronous consume-loop drain optimization: it caused a
replay divergence (ReplayDivergenceError on step_started →
CorruptedEventLogError) in the world-testing inline-batches parallel
workflow on the Windows CI runner. The per-event `process.nextTick` in
the consume loop is load-bearing — it guarantees at most one event is
consumed per macrotask, letting the cross-VM `resolve → workflow VM body
→ subscribe()` chain register the next operation's consumer before the
drain advances. A synchronous drain races ahead of that registration.
What remains is a documentation comment on `scheduleWhenIdle` capturing
why its initial `setTimeout(0)` must not be downgraded to a microtask
(empirically: queueMicrotask breaks hook/sleep Promise.race ordering →
CorruptedEventLogError). No behavior change.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* docs(core): use empty changeset for comment-only macrotask doc
The scheduleWhenIdle change is a pure code comment with no consumer-facing
effect, so it does not warrant a patch bump / changelog entry. Replace the
patch changeset with an empty one to satisfy the changeset-bot convention
without claiming a release. Per the PR template's `pnpm changeset --empty`
guidance for non-releasing changes.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Pranay Prakash <pranay.gp@gmail.com>
* perf(core): drain consumable replay events synchronously
The EventsConsumer rescheduled `process.nextTick(this.consume)` after every
consumed event, so replaying N already-consumable events (structural
lifecycle events, step_created/step_started, completed deliveries) cost N
macrotask hops — O(N) per consume wave across a sequential replay.
Drain consecutively consumable events within a single synchronous pass
instead. This is safe because callbacks only ever consume events with a
consumer that is already registered; new consumers are registered by
workflow VM body code that runs asynchronously off ctx.promiseQueue after a
delivery resolve(). When the next event's consumer is not yet registered,
no callback consumes it and we fall through to the existing cross-VM-safe
deferred unconsumed-event check, exactly as before. A null end-of-events
sentinel never continues the drain, so it cannot spin past end-of-log.
scheduleWhenIdle is intentionally left unchanged: its initial setTimeout(0)
is load-bearing for cross-VM propagation (pendingDeliveries is already 0
between a delivery resolve() and the VM body registering its next
subscriber). Replacing it with queueMicrotask empirically breaks hook/sleep
Promise.race ordering (CorruptedEventLogError); a comment now records this.
Re-validated after a premature revert: the windows-unit flake that prompted
the revert reproduces on unmodified main at the same rate (local 8-way
harness: opt 4/80 vs main 7/80; main historical windows-unit ~13%), so it is
a pre-existing flake, not a regression from this change.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
---------
Signed-off-by: Pranay Prakash <pranay.gp@gmail.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
* perf(core): lazy inline step start to save a world round-trip per step
The owned-inline runtime path used to write step_created (suspension
handler) and then step_started (executeStep) as two separate world
round-trips for a step it already owns and is about to run inline. This
defers the step_created write: executeStep sends a single step_started
carrying the step input, and the world creates the step on the fly
(materializing the step entity plus a synthetic step_created event so
replay still observes it). Mirrors the existing resilient run_started ->
run_created pattern.
Exactly-one ownership is preserved by the world's atomic create-claim:
the loser of a concurrent lazy step_started gets EntityConflictError,
which executeStep maps to `skipped`, so it never runs the body. A lazy
step_started is only ever sent for a brand-new step (the suspension
handler defers only steps with no prior step_created), so crash recovery
still re-runs a `running` step via the normal non-lazy step_started.
Worlds updated: world-local, world-postgres (implicit create + synthetic
step_created event), world-vercel (routes the input as the v4 frame
payload and threads the server's stepCreated flag). @workflow/world adds
optional `input` to step_started and a `stepCreated` EventResult signal.
Rollout: server-first. The matching workflow-server change must deploy
before this ships; the Vercel world targets a single Vercel-operated
backend (server always >= SDK). For local/postgres the world ships in the
same package as the runtime, so there is no version skew.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(core): materialize deferred step before failing unregistered step on lazy inline path
The lazy inline step-start optimization defers a step's step_created write,
expecting executeStep to materialize the step via a lazy step_started carrying
its input. For an UNREGISTERED step, executeStep bails out before sending that
step_started and writes step_failed directly — but the step entity was never
created, so the world's "step must exist" ordering guard rejects the
step_failed and the run wedges (times out).
This regressed the StepNotRegisteredError e2e tests uniformly across every
framework/world (the ghost step never reached `failed`). Fix: on the lazy path,
send the lazy step_started first to materialize the step (entity + synthetic
step_created, keeping replay correct), then write step_failed. The lazy
step_started's atomic create-claim preserves exactly-one-owner: a concurrent
winner makes ours reject with EntityConflictError → skipped, so the failure is
never written twice.
Adds world-level regression tests (world-local, world-postgres) asserting a
lazy step_started followed by step_failed marks the step failed.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
* perf(core): skip per-step events.list via inline event-log delta
In the inline sequential loop, the runtime re-read its own just-written
step events with an incremental events.list every iteration — pure
latency on the Vercel world. Add an opt-in CreateEventParams.sinceCursor
so a step-terminal write can return the event-log delta since that cursor
(EventResult.events/cursor/hasMore), and have the inline loop consume it
in place of the fetch.
The delta is computed identically to events.list against the same log, so
the consumed prefix is byte-for-byte what a fetch would return. The fast
path is gated conservatively to the single-step sequential case with no
open hooks/waits (so no out-of-band hook_received/wait_completed can land
in the snapshot→replay window), and falls back to the normal fetch on any
World that does not return a delta. world-local implements the delta;
world-vercel/world-postgres are unchanged and fall back.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* perf(world-vercel): forward sinceCursor over the v4 wire for inline delta
Adds `sinceCursor` to the v4 POST frame meta so a step-terminal write can
ask the server for the authoritative event-log delta on the response
(events/cursor/hasMore), letting the inline loop skip a follow-up
events.list. The server-side computation ships in
vercel/workflow-server#538; older servers ignore the field and the
runtime falls back to events.list (no behavior change).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* test(inline-delta): cover truncated multi-page delta -> hasMore fallback
The inline-delta query in world-local intentionally omits a `limit`, so a
delta larger than one page is truncated and reports `hasMore: true`. The
runtime consume gate only stashes a delta when `!hasMore` and otherwise
falls back to the exhaustive `events.list` loop, so a partial page can
never be consumed as the complete delta.
Make that contract explicit with a comment at the query site, and add
tests pinning it: a world-local test proving the delta truncates and
surfaces `hasMore: true` byte-identically to `events.list(sinceCursor)`,
and an executeStep test proving `hasMore: true` is threaded verbatim.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* docs(world): clarify sinceCursor returns the first delta page, not the full set
The CreateEventParams.sinceCursor docstring said the result is "exactly
the delta an events.list(...) call would return," which read as the full
set. It is the first page of that delta; hasMore signals more. Spell out
the single-page-or-fallback contract so other World adapters implement
sinceCursor consistently, and note that an in-band burst larger than one
page bypasses the fast path (correct, but forgoes the saved round-trip).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
The gzip/zstd FORMAT_VERSION_TABLE entries were gated on 5.0.0-beta.16,
but beta.16 and beta.17 were both published (2026-06-15) before the
compression PR (#2394) merged (2026-06-16) — neither contains the
compression read path. The next published version is beta.18 (pending
Version Packages #2451), which is the first that can decode these
payloads.
With the cutoff at beta.16, getRunCapabilities() reported beta.16/.17
targets as compression-capable, so a cross-deployment start()/resumeHook()
(or a resilient-start probe resolving to such a target) would write
zstd/gzip payloads the target cannot decode — silent replay corruption,
exactly the TODO(release) hazard noted on those lines.
Bump both entries (and the doc comments) to beta.18 and extend the
capability test to assert beta.16/beta.17 are treated as incapable.
Peter Wielander