skills/vercel__workflow
1
0
Fork 0
mirror of https://github.com/vercel/workflow.git synced 2026-09-14 19:59:43 +08:00
Code Issues Packages Projects Releases Wiki Activity
1,617 Commits 647 Branches 550 Tags
codex/atomic-start-postgres
Commit Graph

1617 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Luca Maraschi 4ec7acaa71 feat(builders): observe accepted transforms (#3163) 
Signed-off-by: Luca Maraschi <luca.maraschi@gmail.com>
Co-authored-by: Peter Wielander <mittgfu@gmail.com>
 2026-08-10 13:08:53 -07:00
Peter Wielander fbebf7104d [core] Keep step results ordered behind waits parked on unread hook payloads (#3406) 2026-08-10 12:46:28 -07:00
Rich Haines 1aed119e84 [docs] upgrade geistdocs to 1.19.6 (#3407) 2026-08-10 08:22:32 -07:00
Shalabh Chaturvedi 264ddff67b Add WebSocket transport for step-execution event writes (opt-in) (#3084) 
* sdk side for workflow server websockets

Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com>

* hardcoded workflow server

Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com>

* debug info

Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com>

* more debug

Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com>

* remove unnecessary debug

Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com>

* default on websockets, and override url

Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com>

* fix for missing funcs

Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com>

* make websockets opt outo

Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com>

* enable ws again

Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com>

* [revert later] reduce test to single test, test both http and ws at the same time

Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com>

* empty

Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com>

* Run full suite with and without ws

Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com>

* empty

Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com>

* improve e2e test

Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com>

* minimize tests

Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com>

* fallback to http when proxy present

Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com>

* default to websockets, remove matrix

Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com>

* remove smoke test

Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com>

* update to new protocol

Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com>

* adjust for new protocol (runid in path)

Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com>

* fix ws transport error

Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com>

* blank

Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com>

* fix ws dep

Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com>

* fix ws external: only accelerators, not ws itself

Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com>

* add dedicated WS-transport e2e job; flip WS default back to opt-in

Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com>

* build all packages before local vercel build (needs workflow/nitro on disk)

Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com>

* force NITRO_PRESET=vercel for the local vercel build step

Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com>

* install vercel CLI once instead of npx-ing it per command

Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com>

* add changeset for WS events transport

Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com>

* Harden the WS events transport and gate its e2e jobs

Follow-ups from review of the WS transport.

CI: `e2e-vercel-ws-transport` was wired into the `summary` job but not
into `e2e-required-check`, so all three WS jobs could fail while the
required check stayed green. Added to both branches of the status
validation — including the `workflow-server-test` label branch, where
the job runs under the same gating as `e2e-vercel-prod`.

Transport:

- `reqId` and the pending-reply map are now per connection rather than
  per transport. The protocol defines `reqId` as a per-connection
  counter, so a reconnected socket restarts at 1; with one shared map
  that collided with the previous socket's still-registered waiters.
  It also makes the superseded-socket guard structural instead of
  something the close path has to remember.
- Post-open socket errors are no longer silent. The only `'error'`
  listener closed over the connect promise's `reject`, already settled
  once `'open'` fired, so every broken pipe / 1009 / protocol fault was
  swallowed and its requests hung with no per-request timeout to save
  them. Now logged, and the connection is torn down.
- An unexpected close reconnects eagerly instead of waiting for the next
  write, since a socket breaking mid-run means more writes are coming.
  Bounded by exponential backoff, an attempt cap that falls back to
  lazy reconnect, a bail-out when a newer socket is already live, and an
  `unref()`ed timer so a backoff window can't delay handler exit.
- `ws.send()` failures reject their request. `send()` doesn't throw on a
  non-OPEN socket — it reports through a callback we weren't passing —
  so the request just sat in `pending` forever.
- The reserved `reqId: -1` malformed-frame reply and undecodable frames
  are logged loudly instead of dropped.
- Auth headers resolve once per socket via a thunk, not once per event.
  The bearer only rides the upgrade, so the old code awaited
  `getVercelOidcToken()` on every write and discarded all but the first.
  Re-resolving on reconnect also means a new socket gets a fresh token.

Adapter: a reply with no numeric status now fails closed. Defaulting to
200 reported a write as applied whenever the client met a frame it
didn't understand — and the protocol is explicitly designed to grow new
response variants.

Tests: 24 new unit tests over the paths the e2e suite can't reach on
demand (send failure mid-flight, error after open, late close from a
superseded socket, reconnect backoff and give-up, sentinel/undecodable
frame logging, one-token-per-socket) plus the adapter's fail-closed and
typed-error mapping.

Co-Authored-By: Shalabh Chaturvedi <shalabhc@users.noreply.github.com>

Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com>

* ci: re-trigger to confirm the prior e2e failures were flake

No code change. The 5 failures on 0bb21e7 clustered in a ~20s window
across HTTP-path jobs (example/nuxt on the same test, sveltekit on a
timeout) and one WS job (sleepingWorkflow's clock-skew assertion), which
points at the environment rather than the transport changes. Re-running
to confirm.

Co-Authored-By: Shalabh Chaturvedi <shalabhc@users.noreply.github.com>

Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com>

* Add WS wire-contract conformance tests and pin the transport gate

Three gaps in the existing coverage.

**The HTTP path was already covered** — `events-v4.test.ts` has 22 tests,
including five directly on `createWorkflowRunEventV4` over HTTP (alias
URL, frame meta contents, response decoding, skipPreload/stateUpdatedAt
forwarding). Those run with the gate unset, so they do confirm the
two-branch refactor didn't disturb HTTP. No new tests needed there.

**But nothing pinned the gate itself.** Every HTTP assertion stays green
if the default flips to WS, because the transports are built to be
indistinguishable at the result layer — and an earlier revision of this
branch did flip the default deliberately, for benchmarking. Added tests
for `isWsEventsTransportEnabled()` across values, and one that drives a
real HTTP request through a MockAgent while asserting the WS transport is
never constructed.

**Nothing verified the bytes.** `ws-transport.test.ts` replies with
whatever the test hands it, which proves the client's lifecycle but not
that its frames are what workflow-server accepts. That's the drift the
spec doc exists to prevent, and it already happened once: event meta flat
on the frame where the server wanted it nested under `event`, with both
sides' tests passing.

`ws-protocol-conformance.test.ts` pairs the real client stack (through
`createWorkflowRunEventV4`) with a fixture mirroring the server route's
per-message handling: decode one frame, validate against a local copy of
`WsRequestFrameSchema`, dispatch, encode the reply the way `replyMeta`
does. `experimental_upgradeWebSocket` needs a real Vercel runtime, so the
socket is faked — everything above it is genuine.

Covers: the frame shape the server accepts (and that `reqId`/`type`/
`runId` don't leak into the event meta), payload passthrough, exactly one
frame per message, 409 → the same typed error HTTP raises, fail-closed on
an unknown reply variant, and reqId correlation across concurrent writes.

Plus golden byte fixtures, since the schema copy is the one thing here
that can silently drift. This is the "golden-frame interop test" the
server spec lists as an open gap; the matching half still needs to land
in workflow-server.

Verified the conformance suite is not vacuous: flattening the client's
frame meta fails 5 of its tests.

Co-Authored-By: Shalabh Chaturvedi <shalabhc@users.noreply.github.com>

Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com>

* match the HTTP RetryAgent's transient-failure policy on WS

HTTP event writes go through an undici RetryAgent (RETRY_AGENT_OPTIONS):
5xx and transient connection errors are retried in-process, honoring
Retry-After. The WS path never touches undici, so it shipped with no
transient-failure handling at all — a single 503 or a mid-write reset
surfaced straight to the step runtime and cost a whole step retry where
HTTP would have absorbed it in milliseconds.

That gap is invisible in a passing test run: writes still succeed, they
just cost far more. So copy the policy rather than reinvent it —
[500, 502, 503, 504] plus transport failures, undici's default backoff,
Retry-After honored, and 429 deliberately excluded for the same reason
RETRY_AGENT_OPTIONS excludes it (a firewall challenge this client cannot
solve, which in-process retries only amplify).

Adds WsTransportError so retryability is a typed property of the failure
rather than something the adapter infers by string-matching. Splits
resolveWsTransport()/wsReplyStatus() out of postEventFrameOverWs so the
retry loop stays readable.

The existing "fails closed on an error frame" test used status 500,
which is now absorbed by the retry — switched to 403 so it keeps
testing fail-closed rather than accidentally testing no-retry.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Co-Authored-By: Shalabh Chaturvedi <7066873+shalabhc@users.noreply.github.com>

Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com>

* lazy-load `ws` so the default HTTP path never evaluates it

events-v4.ts imports ws-transport.js unconditionally — the transport
gate is a runtime branch, not a build-time one — so a top-level
`import { WebSocket } from 'ws'` put `ws` and its optional native
accelerators on the module-init path of every deployment, including the
overwhelming majority that never opt in and never open a socket.

Defer it to the first connect, memoized as a promise so concurrent
first connects share one import. WebSocket.OPEN becomes an inlined
constant so the readyState check doesn't pull the module in just to
read it off the constructor.

This does NOT remove the need for the bufferutil/utf-8-validate
externals this branch also adds: webpack and Rollup both statically
follow a dynamic import(), so the build-time story is unchanged. What
it buys is that a deployment which never enables the transport never
*evaluates* `ws`, so a mis-bundled accelerator can't break it.

The test lives in its own file because vitest caches a vi.mock factory
result for the life of the module registry — once any test in a file
has connected, the factory never runs again and the counter can't
distinguish "loaded lazily" from "loaded at import".

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Co-Authored-By: Shalabh Chaturvedi <7066873+shalabhc@users.noreply.github.com>

Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com>

* release idle WS transports instead of renewing them forever

The transports map was never pruned and WsEventsTransport had no way to
close. Combined with eager reconnect that made a connection immortal by
construction: the server drains at its own maxDuration and closes, the
client immediately reopens, and the server pins a fresh invocation — for
a run that finished long ago. A warm container ended up holding a live
socket, and a live server invocation, for every runId it had ever
served. workflow-server#683 already lists "one invocation stays resident
per run rather than per write" as a known gap; this made it "per run,
forever".

Add close() plus a 60s idle release. There is no "run complete" signal
to hang teardown off — the events adapter is a stateless per-write call
— so idleness is the available proxy. 60s sits well below the server's
~680s drain deadline, so the client releases rather than the server
reclaiming, and well above the gap between steps of an active run.

scheduleReconnect() now bails when closed: close() closes the socket,
which fires the same close handler an unexpected drop would, and without
the guard the transport would instantly reconnect what it just released.

request() revives an idle-closed transport rather than failing the
write, re-registering itself only if nothing newer has claimed the map
slot. Eviction therefore costs one handshake, not an error.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Co-Authored-By: Shalabh Chaturvedi <7066873+shalabhc@users.noreply.github.com>

Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com>

* refresh the bearer on an auth_expiry drain

workflow-server#683 tags a drain frame with why it is closing:
max_duration means the socket aged out and a plain reconnect is right,
auth_expiry means the *bearer* ran out and reconnecting with the same
one just earns a 401. This client logged the drain and ignored the
reason, so against #683 an auth_expiry drain would burn all five
reconnect attempts against a token the server had already rejected, then
give up.

Parse the reason (absent reads as max_duration, so this stays correct
against the currently-deployed server) and thread forceRefresh through
the getHeaders thunk, which triggers @vercel/oidc's refresh path via a
wide expirationBufferMs.

Worth being precise about when that can actually help. getVercelOidcToken
resolves getContext().headers['x-vercel-oidc-token'] ?? env, and
refreshToken() only writes the env var — the request-context header
wins. So inside a deployed function there is genuinely no fresher token
mid-invocation and the refresh is a no-op; outside one (CLI, local dev,
a long-lived server) it works.

That makes the guard the load-bearing half: if the re-resolved bearer is
byte-identical, decline to reconnect, say so, and wait for the next
write — which usually arrives on a new invocation carrying a new token.
That failure is marked non-retryable so the retry loop doesn't spin on
it either.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Co-Authored-By: Shalabh Chaturvedi <7066873+shalabhc@users.noreply.github.com>

Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com>

* document the WS path's instrumentation gap

The HTTP branch goes through fetchV4 -> instrumentedFetch, which is not
just a fetch wrapper: it opens the OTEL CLIENT span, injects trace
context, sets the cache-bust header, emits the DEBUG logs, and routes
through the global fetch that Vercel's observability "outgoing requests"
view instruments. The comment on fetchV4 records why that matters —
bypassing it via undici.request() is exactly what once made v4 event
traffic disappear from the log viewer.

The WS branch bypasses all of it. With the flag on, per-event writes
have no client span, propagate no trace context to workflow-server, and
don't appear in the outgoing-requests view; the server's own
transport-tagged request metrics are the only remaining signal.

That's acceptable for an opt-in POC behind a flag and unacceptable as a
default, so write it down where someone deciding to flip the default
will read it: instrumenting the transport is a prerequisite for that,
not a follow-up nicety.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Co-Authored-By: Shalabh Chaturvedi <7066873+shalabhc@users.noreply.github.com>

Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com>

* ship the ws-accelerator externals instead of documenting a workaround

`bufferutil` and `utf-8-validate` are optional native accelerators for
`ws`, and neither is installed by default. Every bundler has to be told
to leave them alone, for two different reasons: Rollup/Vite/Nitro fail
the build outright (`Could not resolve "bufferutil" imported by "ws"`),
while webpack bundles the JS wrapper without its native `.node` binding
and throws `bufferUtil.mask is not a function` at runtime.

The webpack half shipped in `@workflow/next`. The Rollup half only
existed in `workbench/vite` and `workbench/tanstack-start` as
`nitro.rollupConfig.external` — app configs, not shipped code. So a real
user of `@workflow/vite`, `@workflow/nitro`, `@workflow/nuxt`,
`@workflow/sveltekit` or `@workflow/astro` hit the same build failure the
workbench had already worked around, and had to rediscover the fix.

Fix it where it propagates: `workflowTransformPlugin` in
`@workflow/rollup`, which all of those integrations already install. It
is already the home of exactly this pattern for the optional
`@opentelemetry/api` peer, so this sits next to its closest precedent.

Note the treatment is deliberately the inverse of the OTEL one, which is
externalized only when it *can't* be resolved. The OTEL API must load
for tracing to work, so a self-contained output has to bundle it when
present. These accelerators must specifically NOT load — they are a
performance nicety with a correct try/catch fallback in `ws` — so
unconditional external is both simpler and safer than risking a
half-bundled native module.

The two workbench configs drop their local copies, which is what proves
the shipped fix actually works rather than being masked by them.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Co-Authored-By: Shalabh Chaturvedi <7066873+shalabhc@users.noreply.github.com>

Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com>

* one retry policy for both transports, and no unanswerable waiters

Two review findings on the WS events transport.

**Retry belongs to `event-retry.ts`, not the adapter.** The WS path had its
own retry loop, justified as mirroring undici's `RetryAgent`. That
justification was wrong: `RetryHandler` defaults `methods` to GET/HEAD/
OPTIONS/PUT/DELETE/TRACE and nothing overrides it, so the `RetryAgent` never
retried an event POST on either transport — which is precisely why
`event-retry.ts` exists.

Worse, that loop sat *inside* `withEventPostRetry`, so it defeated a
compile-checked safety gate: `EVENT_RETRY_ELIGIBILITY` marks `step_started`,
`step_retrying` and `hook_received` non-retryable (a replayed `step_started`
double-increments `attempt`), and those frames were re-sent up to five times
before the gate ever saw a failure. For eligible types the two loops
multiplied: 3 outer attempts x 6 inner, with an inner backoff reaching 30s
against an outer base deliberately set to 100ms.

`postEventFrameOverWs` now makes one attempt and translates failures into the
vocabulary that policy already speaks — a transport failure becomes a
`WorkflowWorldError` with `code: 'TRANSPORT'`, exactly as `utils.ts` does for
a failed `fetch`, and `isRetryableEventPostError` gains one clause keyed on
that code. `WsTransportError` loses its `retryable` flag; its only consumer
was the deleted loop.

Two deliberate consequences. The code-keyed clause broadens HTTP in-process
retry to `UND_ERR_CONNECT`, `UND_ERR_CLOSED` and `EAI_AGAIN`, which were in
utils.ts's transient set but missing from event-retry.ts's — two
hand-maintained lists collapsed into one semantic code. And the stale-token
case (drain for auth expiry, refresh yields the same bearer) now gets two
in-process attempts that cannot succeed, ~300ms before it falls through to
queue redelivery; that is cheaper than keeping a WS-specific policy alive for
one call site. `TIMEOUT` is deliberately not in the clause: utils.ts maps a
caller-supplied `AbortError` onto it, and a cancelled write must not be
re-issued.

A status-less reply also stops being a bare `Error` — as one it failed
`WorkflowWorldError.is()` and surfaced a protocol version skew as a
USER_ERROR. It is now `code: 'PARSE_ERROR'`, the same code utils.ts uses for
an unreadable HTTP body, and for the same reason: the write may or may not
have landed.

**No waiter is left unanswerable.** An undecodable frame, the server's
malformed-frame sentinel (`reqId: -1`) and a non-numeric `reqId` were logged
and dropped. None can be correlated by construction, so the request that
provoked them stayed in `pending` with nothing in existence able to settle it
— freed only by the server's own drain (~680s from connect), typically past
the invocation's `maxDuration`. Each now fails the connection: every waiter
learns why, and the socket is replaced. A reply for an id nobody is waiting
on stays log-and-drop, deliberately — that request already settled, so
nothing is orphaned, and failing the socket would punish healthy in-flight
writes.

A per-request deadline backs that up for whatever is left, including a server
that accepts a frame and never answers it. Same knob as the HTTP path
(`WORKFLOW_REQUEST_TIMEOUT_MS`, 60s), whose doc comment already describes
this exact hang-to-SIGTERM pathology.

One existing idle-teardown test needed the deadline raised: the idle window
and the default deadline are both 60s, so a request could not outlive the
former without also outliving the latter. The test is about `inFlight > 0`
suppressing the teardown, so it now sets the deadline out of the way.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com>

* Open the ws socket when the invocation starts, not on its first write

Lazily connecting bills the whole handshake — an upgrade round-trip plus the
OIDC token mint that rides it — to whichever event a fresh invocation writes
first. When that is a `step_started` issued as the step body is already
running, the event's server-recorded timestamp lands later than the work it
describes: the step looks shorter than it was. That is the shape of the e2e
timing failure on this branch, where a 9s step measured 6.5s from
`getStepMetadata().stepStartedAt`.

The queue handler is the earliest point that knows the run id, and a message
delivered for a run means writes are coming, so `warmWsEventsTransport` starts
the handshake there. By the first write it is done or in flight, and the write
just uses it.

Nothing about it is load-bearing:

- It doesn't await, and can't fail the handler. A warm that fails logs and
  stops — a never-opened first connect is precisely the case `connect`'s close
  handler already declines to retry, so no backoff loop starts for a run that
  may never write. The first real write connects as it would have anyway,
  carrying the shared retry policy.

- No-op unless `WORKFLOW_EVENTS_TRANSPORT=ws`, and no-op for the api-workflow
  proxy World, which can't serve an upgrade at all — the same fallback the
  write path takes.

- Warming arms the idle timer as if a request had settled, so an invocation
  that warms and never writes (a health probe carrying the run id it is about
  to create) releases its socket on the usual 60s rather than stranding it.
  The socket is not `unref`'d, so a stranded one would hold this process and a
  server invocation open.

Also closes a race that warming makes reachable: `close()` can only drop the
connection it can see, so a release landing mid-handshake left the socket to
install itself afterwards onto a transport already evicted from the cache,
which nothing would then ever close. The `open` handler now declines to adopt
a socket whose transport was released while it was connecting. This was
already reachable via the eager reconnect path, just much harder to hit.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com>

* changeset: just the env var

Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com>

* inline the ws-accelerator predicate at its only call site

Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com>

* refactor(world-vercel): trim ws-transport comments

Comments were 47% of the file. Cut the historical narration, the
restatements of adjacent code, and the repeated rationale (the `unref`
reasoning appeared four times, per-connection reqId three), keeping the
non-obvious facts: `ws.send()` reports failure via callback instead of
throwing, reqId is per-connection so `pending` must be too, the
unknown-reqId case is deliberately non-fatal, the auth_expiry same-token
bail-out, and why the idle timeout exists at all.

No code changes.

Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com>

* own transport selection in the transport module

`events-v4.ts` was assembling the WS transport itself: reading the opt-in
flag, resolving the URL, deciding which Worlds can use a socket, minting the
per-connection header thunk, and holding the two once-per-process log latches.
None of that is about turning an event into a frame, which is what the rest of
that file does. Move it next to the socket it configures — `events-v4.ts` now
consumes one seam (`resolveWsTransport`) plus the gate, and `queue.ts` gets
`warmWsEventsTransport` from the module that owns the warm.

`headersToRecord` now lives in `http-core.ts` because both callers need it and
neither may import the other: `events-v4` already depends on the transport, so
the reverse edge would be a cycle.

Test fallout, and the reason the move is worth it: `events-v4-ws.test.ts`
mocked `getWsEventsTransport` to observe the resolve step, which no longer
intercepts anything now that the call is intra-module — an ESM mock replaces a
module's exports, not its own call sites. That mock's tests were only ever
about selection, so they move to `ws-transport.test.ts`, where the real
selection code runs against the existing fake-socket harness instead of a
stub. `resetWsEventsTransportsForTest` clears the log latches so the
once-per-process assertions don't depend on test order. What stays behind
mocks `resolveWsTransport` and covers what that file is actually for: reply
frame in, `Response`-shaped result out — including the null-resolve fallback to
HTTP, which nothing covered before.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com>

* import `ws` statically

The lazy `import('ws')` was there to keep the package off the module-init path
of deployments that never opt in — `events-v4.ts` imports this module
unconditionally, since the transport gate is a runtime branch. Measured, that
buys ~17ms: `require('ws')` is 16.5-18.0ms cold, 13 modules, and neither
`bufferutil` nor `utf-8-validate` loads (optional peers, absent by default).
Bundle size is identical either way — webpack and Rollup both statically follow
a dynamic `import()`, which is why the externals in `@workflow/builders` are
unaffected by this change.

For 17ms it cost a memoized promise, an inlined `WS_READY_STATE_OPEN` (so a
readyState check wouldn't force the module to load just to read a constant off
the constructor), and a whole test file — `ws-transport-lazy.test.ts` had to
live alone, because vitest caches a `vi.mock` factory result for the lifetime
of a module registry, so only a file that connects exactly once can observe
the laziness at all.

It also skewed the thing this branch exists to measure. The import lands inside
the first connect, so on a warm container it is billed to whichever event write
opens the socket, inflating the timestamp of the step it labels — the same
distortion the queue pre-warm was added to remove.

Also drops `WS_READY_STATE_OPEN` in favour of `WebSocket.OPEN`, now that
reading it is free.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com>

* tighten the comments on the ws transport

Comments only — no code changes in this commit.

Cuts ~150 lines of prose across the WS additions. The rule applied: keep the
design factors a future reader needs (why the connection is scoped to a run,
why a bad reply takes the socket down, why the accelerators are externalized
unconditionally, why `TIMEOUT` is excluded from the `TRANSPORT` classification)
and drop the narrative of how the code got here — which revision did what, what
an earlier attempt got wrong, what was measured on the way. That history lives
in the PR and the git log, where it doesn't have to be re-read on every visit
to the file.

Biggest reductions: the retry essay above `postEventFrameOverWs` (30 lines to
11), the flag's OTEL-gap note (34 to 13), the OIDC refresh explainer (26 to
14), the accelerator rationale in `@workflow/builders` (26 to 14), and the
conformance suite's header (28 to 17).

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com>

* inject W3C trace context on the ws upgrade

Frames carry no headers, so the upgrade is the only place this transport can
propagate context; the server parents a run's event spans to whichever
invocation opened the socket. Covered in trace-propagation.test.ts, both with
and without an active span.

Splits the opt-in gate into an import-free ws-transport-enabled.ts so callers
can answer it without loading this module (used by the next commit).

Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com>

* load the ws transport module only when it is enabled

Both call sites read the gate from the import-free module and dynamically
import ws-transport.js behind a true result, so a deployment on the HTTP
default never pays ws's ~17ms of module init. The queue pre-warm absorbs it
for one that opted in, keeping it off the first event write.

Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com>

* document WORKFLOW_EVENTS_TRANSPORT as experimental

Names the instrumentation gap (no client span per write) and the proxy path
where the variable is ignored.

Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com>

* correct why the ws accelerators are externalized

No bundler fails the build on the unresolvable require — verified against
Rollup 4.62. webpack half-bundles the native module and Vite substitutes a stub
that makes the require succeed; both leave bufferUtil.mask undefined and throw
only once a frame reaches the native masker at 48 bytes, which every CBOR event
frame does. Same claim was repeated in the rollup plugin and its test.

Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com>

* trim the WORKFLOW_EVENTS_TRANSPORT docs to user level

Mirrors the other Vercel World env vars: same facts on both pages, each in its
page's format. The instrumentation and socket-lifetime detail belongs in the
code, not in a user-facing reference.

Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com>

* cover the vite bundler in the ws transport lane

Vite substitutes a stub for ws's absent native accelerators rather than failing
the require, so nothing catches it until a masked frame reaches 48 bytes — and
this job's three existing lanes are esbuild, turbopack and nitro.

Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com>

* claim only what is measured about rollup and the ws accelerators

The rationale asserted plain Rollup was "safe by accident" via a mechanism
only ever observed in a minimal repro. Nitro traces and externalizes `ws` in
a production build, so the bundled path is not reached there at all.

Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com>

* Give the events socket an explicit lifetime instead of an idle timer

`openWsChannel` / `closeWsChannel` bracket one invocation of the flow
route, and are the only calls anywhere that create a channel. Writes ask
`resolveWsTransport` whether one is open — a lookup now, never a create —
and take pooled HTTP when it says no.

That removes the reason the idle timeout existed. A lazily-created socket
has no owner, so a timer was the only thing able to end it, and the socket
is not `unref`'d: the process could not exit, and a server invocation
stayed pinned, for the full window past the last write.

It also settles `run_created`. The trigger path opens no channel, so a
lone write no longer pays for a handshake it cannot amortize — `start()`
runs in an arbitrary request handler with no boundary the SDK can see.

Refcounted rather than a flag: inline step executions ride the flow topic
on per-step topics, so a run's steps can be concurrent invocations in one
instance sharing the channel, and the first to finish must not cut the
others short. A failed connect closes the channel so the invocation's
writes fall back to HTTP instead of each paying its own doomed handshake.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com>

* Name the one reply header the WS path does not map

The server copies six headers into an `event_ack`'s meta and this record
maps five. The sixth, `X-API-Deprecated`, is inert today — the v4 route's
middleware chain has no deprecation middleware to set it — but the record
is the only header source a WS reply has, so an unmapped key is gone
rather than merely unread, which is not true of the `Response` the HTTP
path returns.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com>

* docs: note that WORKFLOW_EVENTS_TRANSPORT=ws is ignored on the proxy path

The api-workflow proxy is an HTTP-only REST gateway and does not forward
a WebSocket upgrade, so a World configured with projectConfig keeps
writing events over HTTP regardless of the setting.

Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com>

* ci: gate the ws-transport e2e lanes on a label

Three real `vercel deploy`s per run is too much to charge every
unrelated PR in the repo for a transport that is off by default. PRs opt
in with `ws-transport-test` (or `workflow-server-test`, which already
exists to test the half of this the protocol lives in); main keeps the
signal on every commit.

The required aggregate has to allow the lane to be skipped in that case,
so its status is asserted only when the lane was actually supposed to
run.

Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com>

* chore: regenerate pnpm-lock against current main

main resolved `ws` to 8.20.0 as a transitive peer; this branch adds it
as a direct dependency of world-vercel and floats it forward, which
rewrites every `openai@x(ws@y)` peer key in the lockfile. Merging main
textually combined the two, leaving those keys pointing at a `ws` entry
the merged file no longer had — `--frozen-lockfile` then failed with
ERR_PNPM_LOCKFILE_MISSING_DEPENDENCY on the PR's merge ref.

Regenerated from main's lockfile so ours is a minimal delta on top of it.

Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com>

* fix(world-vercel): align ws on the version main already resolves

The lockfile broke on the PR's merge ref, not on this branch's head: main
resolves ws@8.20.0 as a transitive peer, and a `^8.21.1` direct dep here
floated it forward, rewriting all 73 `(ws@8.20.0)` peer keys. Git merged
the two lockfiles without a conflict but left main-side keys pointing at
a ws entry the merged file no longer had, so `--frozen-lockfile` failed
with ERR_PNPM_LOCKFILE_MISSING_DEPENDENCY.

`^8.20.0` resolves to the copy main already has, so the lockfile delta is
the two importer entries instead of a repo-wide rewrite that re-breaks
every time main moves. Also keeps one ws in the store rather than two.

Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com>

* Bind the channel release to the instance it claimed

closeWsChannel resolved the transport by URL, but the refcount lives on
the instance. A channel is evicted from the map as soon as it closes — a
refused upgrade does that on the connect path — so the next opener for
the same run registers a different instance under the same URL, and the
first invocation's close then decremented that one instead. It dropped a
socket a live invocation was still writing over, and for the event types
EVENT_RETRY_ELIGIBILITY marks non-retryable there is no second attempt to
carry the in-flight write over HTTP.

openWsChannel now returns an idempotent release closed over the transport
it incremented, and queue.ts holds that instead of re-resolving the run.
The close awaits the open's own promise, so it also can no longer land
ahead of the claim it releases.

Also names the scope of the connect-failure de-opt: it covers the
handshake only, so a channel that connects and then fails every write
keeps taking the WS path.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com>

* Decode a transport result, not a Response

Main extracted the v4 POST decode into a helper typed `Response` while this
branch narrowed the POST result to `FrameResponseLike`, because the WS branch
synthesizes its result rather than holding a real `Response`. The two merge
without a textual conflict and then fail to typecheck.

Widen the helper: it reads only the two members `FrameResponseLike` declares,
and a `Response` still satisfies them, so the HTTP call sites are unchanged.

Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com>

* Re-run CI

Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com>

* Re-run CI

Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com>

* Reconcile the WS transport with main's v4 POST rework

main moved the materialized POST result off the `x-wf-*` response headers
and onto a typed CBOR body, and added a second response shape: two callers
now POST with `Accept: application/vnd.workflow.v4-frames` and read back a
sentinel-terminated sequence of frames.

A frame stream has no representation in a protocol that pairs one reply
frame with one request frame, so the WS switch moves off the shared poster
and onto `createWorkflowRunEventV4` alone — the materialized write, which is
the hot per-step path this branch exists to shorten. `run_started` and the
`hook_received` preload stay on HTTP.

`decodeCreateEventResponse` takes `FrameResponseLike` rather than `Response`
because the WS branch has none to hand over; a real `Response` satisfies the
interface, so the HTTP callers are unchanged. The ids now come out of the
CBOR body, so `replyMetaToHeaderRecord` no longer maps any `x-wf-*` name —
only the two headers `errorFromV4Response` reads.

Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com>

* Re-run CI

Resample the WS-arm sleepingWorkflow failure: it has now recurred on a second
axis (nextjs-turbopack, 7709ms; previously vite, 7570ms), so the arm needs
more samples before the skew can be called WS-specific or repo-wide flake.

Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com>

* Re-run CI

Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com>

* Re-run CI

Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com>

* blank

* blank

---------

Co-authored-by: vercel[bot] <35613825+vercel[bot]@users.noreply.github.com>
 2026-08-09 17:34:44 -07:00
Nathan Colosimo 22349e95fd perf(core): load replay suffix in one request (#3205) 
* perf(core): stream replay suffix in one request

* perf(core): load replay suffix in one request

Signed-off-by: Nathan Colosimo <110621881+NathanColosimo@users.noreply.github.com>

* test(world-vercel): use streamed run start fixtures

* refactor(events): simplify return-all plumbing

Signed-off-by: Nathan Colosimo <110621881+NathanColosimo@users.noreply.github.com>

* Return complete local run preloads

* Document workflow event limit

* fix: make return-all event loading resilient

* Simplify full event listing

* refactor(world-vercel): omit event limit for full loads

* fix(world-vercel): explicitly request complete event logs

---------

Signed-off-by: Nathan Colosimo <110621881+NathanColosimo@users.noreply.github.com>
 2026-08-07 21:55:18 -07:00
Nathan Colosimo 665110b3a2 perf(core): load replay log from run_started (#3191) 
* perf(core): consume run_started replay page

* fix: preserve turbo startup while streaming replay

* refactor(world-vercel): simplify run start stream

* chore(world-vercel): sort merged imports

* refactor(world-vercel): require run start event page

Signed-off-by: Nathan Colosimo <110621881+NathanColosimo@users.noreply.github.com>

* refactor(world-vercel): require complete event pages

Signed-off-by: Nathan Colosimo <110621881+NathanColosimo@users.noreply.github.com>

* refactor(world-vercel): remove redundant event result type

Signed-off-by: Nathan Colosimo <110621881+NathanColosimo@users.noreply.github.com>

* refactor(world-vercel): narrow event stream state

* Simplify run-started event consumption

* Handle cross-region lifecycle event order

* test(world-vercel): validate malformed event frame

* Name run start result as an event stream

* Simplify run-started stream types

* Simplify optional event metadata

* Validate complete event frames

* Validate frame metadata

* Validate events at frame boundary

* Fix out-of-order run lifecycle replay

---------

Signed-off-by: Nathan Colosimo <110621881+NathanColosimo@users.noreply.github.com>
 2026-08-07 21:55:18 -07:00
Nathan Colosimo 65139acfd7 perf(core): continue partial run_started preloads from cursor (#3124) 
* perf(core): continue partial run preloads

* refactor(core): simplify preload continuation

* fix(core): preserve preload fallbacks

* chore: rerun CI

* fix(world): infer event create results

* fix(core): preserve run state during setup

* fix(world): enforce typed event results

* refactor(world): rely on event result contract

* refactor(core): unify replay event log state

* refactor(core): make replay log states exact

* fix(core): harden run start preload recovery

* test(world-local): allow slow preload coverage

* fix(core): preserve event result inference through recovery

* refactor(core): simplify preload state transition

Signed-off-by: Nathan Colosimo <110621881+NathanColosimo@users.noreply.github.com>

* refactor(runtime): reuse event pages without duplicate reads

* refactor(world-vercel): preserve opaque event payloads

* Validate v4 event create responses

* Validate v4 event frame metadata

* Remove invalid v4 response identity check

* Return validated v4 event bodies directly

* Reuse event result entity types

* Simplify event creation result types

* Use concrete run creation result

* Preserve generic event storage implementation

* Validate v4 event responses without casts

* Parse v4 event frames once

* Reuse the default v4 event body schema

* Simplify event preload state

* Narrow event page result states

* Preserve literal event result flags

* Accept hook conflict event responses

* Remove redundant optional event page schemas

* Simplify preloaded event log access

* Flatten replay event log state

* Simplify replay event log state

* Use one replay event log

* fix(next): preserve edits made during full HMR rebuilds

* chore(core): log dormant hook replays

* fix(next): commit HMR snapshots after rebuilds

* fix(next): ignore duplicate HMR file events

* test(next): expect deduplicated HMR removal event

* fix(next): distinguish duplicate HMR notifications

* fix(next): ignore HMR notifications without source changes

* chore: move Next HMR fix to separate PR

* fix(core): complete partial preloads before QuickJS replay

---------

Signed-off-by: Nathan Colosimo <110621881+NathanColosimo@users.noreply.github.com>
 2026-08-07 21:55:17 -07:00
Nathan Colosimo 74dbf81d32 fix(core): retry replay timeouts without exiting (#3385) 
* fix(core): retry replay timeouts without exiting

* refactor(world-postgres): leave existing retry limits unchanged

* test(world-postgres): remove mocked migration assertion

* chore: consolidate replay retry changesets
 2026-08-07 15:16:04 -07:00
Nathan Colosimo 4bb86d3054 feat(world-vercel): support Hook minimum retention (#3286) 
* feat(world-vercel): support Hook minimum retention

* fix(core): fail deterministic Hook validation
 2026-08-07 13:00:52 -07:00
Peter Wielander a8db185c3b [core] Fold events.create deltas into the replay log (#3382) 2026-08-07 10:12:10 -07:00
Nathan Rajlich eb9e13fd23 QuickJS engine: baseline-snapshot startup optimization (~25× faster VM boot) (#3342) 
* Add opt-in QuickJS WASM VM engine (WORKFLOW_VM=quickjs) with full event replay

* QuickJS engine: AbortController, setAttributes, terminal drain, turbo-safe requeue, stable PRNG seed

* QuickJS engine: hook.getConflict support, cross-run writable forwarding symbols

* QuickJS engine: stream framing round-trip, bound step proxies, webhook fidelity

* Apply biome fixes to QuickJS engine files

* Address review feedback: anchor source-map strip to end-of-input, use getWorkflowQueueName for conflict requeue, Buffer-free asset decoding, function replacers for payload injection, maxEventsLimit guard

* CI: include generated QuickJS source assets in shared e2e build artifacts

* Fix same-token hook ordering and conflicted-hook disposal in the QuickJS engine

* CI: run both VM engines across all frameworks and worlds; label jobs with the engine

* Fix stack overflow stripping inline source maps from webpack dev bundles; harden step-listing e2e assertions against eventually-consistent reads

* e2e: poll step listings until analytics rows include attempt (optional column can lag terminal status)

* e2e: use --withData to force storage-backed step listings for attempt assertions (analytics listing can omit attempt entirely)

* Sort imports in QuickJS serialization files (biome organizeImports)

* QuickJS engine: resolve the run's full payload-key capability so sealed (encp) hook payloads open

Main's sealed-box work (#3096) makes cross-deployment resumeHook() seal
hook payloads to the target run's published X25519 public key. The shared
start() path publishes that key regardless of engine, so QuickJS runs
receive sealed payloads too — but the QuickJS entrypoint resolved only the
bare symmetric key via importKey(), which cannot open encp envelopes. The
first sealed hook payload wedged the run right after hook_received, timing
out every hook/webhook e2e on Vercel prod (node:vm legs were fine — the
node engine resolves the full capability via memoizeEncryptionKey).

Resolve deriveRunPayloadKeys() in the entrypoint instead and widen the
runtime's key types from CryptoKey to DecryptionKey. Writes stay symmetric
(encrypt() with RunPayloadKeys takes the encr path). Regression test seals
a payload exactly as resumeHook does and round-trips it through the VM.

* Address review: crypto/process parity, loud Intl guards, lazy engine import, VM-leak guard, telemetry namespace, eval-string escaping

- Deterministic crypto.getRandomValues/randomUUID in the VM bootstrap,
  drawing from the seeded Math.random (identical sequences to the node
  engine's vm/index.ts implementations); all crypto.subtle methods throw
  with step-function guidance. process.env exposed as a frozen copy,
  matching node.
- Intl: throwing constructors (no ICU in QuickJS), and toLocale*-family
  methods (incl. localeCompare) throw when given an explicit locale so
  cross-engine divergence is loud instead of silently writing different
  values into the event log. No-argument forms keep working.
- runtime.ts lazy-imports the QuickJS entrypoint at dispatch, keeping the
  ~1.3MB embedded WASM assets out of node-engine deployments.
- runQuickJSWorkflow wraps the per-run phase so an exceptional exit
  disposes the VM instead of leaking it in a reused compute instance;
  corrected the misleading fail-loud comment (run_failed, not retry);
  warn when the event drain loop exhausts its iteration bound.
- Telemetry attributes renamed quickjs.* → workflow.vm.* to stay in the
  file's workflow.* namespace.
- Eval-string correlation-id interpolation uses JSON.stringify instead of
  quote-only escaping.
- common-vm.test.ts pins the reducer/reviver superset invariant against
  common.ts so the duplicated sets can't silently drift.
- Docs enumerate the remaining global-surface differences (subtle.digest,
  Intl, WebAssembly, Atomics); quickjs-entrypoint documents the known
  precondition-guard gap.

* QuickJS engine: implement resilient resumeHook (hookInput materialization + resumeId dedup)

#1834 made resumeHook() fall back to enqueueing the run with a hookInput
payload when the direct hook_received write fails transiently, with the
runtime materializing the missing event on delivery. Only the node:vm
path implemented it — the QuickJS dispatch returned before the node
block, so the resilient payload was silently dropped and the new e2e
timed out on every quickjs leg.

- runtime.ts threads hookInput into runWorkflowWithQuickJS; the
  entrypoint materializes the missing hook_received after loading the
  event log (resumeId-keyed dedup, occurredAt from the resumeId ULID,
  local eventData substitution for lazy/ref responses, EntityConflict /
  HookNotFound handling) — mirroring the node block.
- processEvents drops duplicate hook_received rows sharing a resumeId
  (first-in-log wins), matching the node engine's EventsConsumer dedup;
  the seen-set lives in the VM heap so it is deterministic per replay.

Verified against the dev server with WORKFLOW_VM=quickjs: the resilient
resume e2e passes and the materialization is observable in the logs; all
27 hook e2e tests green.

* QuickJS engine: inline step execution via live-VM continuation loop + WASM module caching

* Address review: exclusive inline step claims, self-write requeue, in-loop event ceiling

- Inline steps now claim via a lazy step_started carrying the input
  (step_created deferred, atomic create-claim in the world), with
  ownerMessageId stamped and authoritativeAttempt=1 — a concurrent
  invocation racing on the same fresh step loses with
  EntityConflictError and skips instead of both bare-starting the step
  and double-running the body. This also removes the stepsCreatedByUs
  set, whose 'created by us' invariant didn't survive the swallowed
  create-race conflict; redelivery backstops now key on hasCreatedEvent.
- dispatchPendingOps' createdAttributeEvent/createdGetConflictHook
  signals are consumed again: when the loop exits suspended without ever
  reading back a self-written attr_set / getConflict hook_created
  (eventually-consistent listing lag), the entrypoint requeues
  immediately instead of parking the run awaiting_external with its
  unblocking event already written.
- The server-supplied event ceiling is re-checked at the top of every
  continuation-loop turn (seenEventIds.size), so a single invocation
  fanning out inline can no longer grow the log arbitrarily past the
  operator's limit. The quickjs dispatch in runtime.ts converts
  MaxEventsExceededError into run_failed / MAX_EVENTS_EXCEEDED — the
  guard's throw previously nacked forever, parking runaway runs in
  'running'.
- Documented the deliberate decision that the platform function timeout
  is the only bound on inline chaining (budget parked per batch),
  matching the node engine.

* QuickJS engine: host-side, side-effect-free serialization via handles

(Re-applied onto the review-fixed base; original commits da2723016 +
9814ed9ac squashed.)

Replace the in-VM serde bundle with a host-side codec
(runtime/quickjs-serde.ts) built on quickjs-wasi 3.3's introspection
primitives and devalue 5.9's pluggable stringify/parse operations —
mirroring the node:vm engine's architecture.

Review fixes incorporated:
- reducer/reviver key sets are pinned against codec-devalue-vm's
  workflow mode by exhaustiveness tests (exact order for reducers —
  first match wins), so the handle-space codec can't silently drift
  from the shared value-space sets.
- the devalue entry in minimumReleaseAgeExclude is removed: the exact
  version is pinned via the workspace catalog + lockfile, so the
  cooldown waiver was unnecessary (verified with both frozen and
  regular installs).
- eval-string interpolation inherits the JSON.stringify(cid) hardening
  from the base branch.

* Address review: NUL-safe string extraction, deterministic retryAfter, pass-scoped handle disposal, byte-cache lifecycle

- NUL (U+0000) safety across the WASM boundary: handle.toString() routes
  through JS_ToCString and silently truncates at the first NUL, and the
  C-string key APIs mangle NUL-bearing property keys (drop or collide).
  guestString() detects truncation by comparing against the handle's
  true guest length and recovers via in-VM JSON.stringify escaping;
  shapeOf verifies its fast host-string key list against a guest
  Object.keys count (+ duplicate check) and re-extracts through key
  handles on mismatch; get/hasOwn route NUL-bearing keys through
  length-aware guest string handles. All string funnels (primitives,
  symbol descriptions, error fields via chained/own reads, Headers
  entries, RegExp source/flags, URL href) go through guestString.
  Regression-tested down to the truncate-vs-collide enumeration shapes;
  fixes nullByteWorkflow on the quickjs e2e legs.
- RetryableError's absent/invalid retryAfter fallback now reads the
  GUEST clock (the deterministic replay clock at the WASI layer) via a
  captured Date.now instead of the host wall clock — the in-VM reducer
  was replay-stable by construction and the host port silently lost
  that.
- Pass-scoped handle disposal: serialize/deserialize sweep every
  intermediate handle their pass creates (call/invoke results,
  descriptor reads, dups, parse-op constructions), closing the
  ~one-leaked-handle-per-value-node growth across long-lived inline
  sessions. Implemented with module-owned tracking rather than
  vm.withScope: the library scope also captures the handles the
  host-callback trampoline wraps around C-owned argv pointers, and
  disposing those (Map/Set/Headers forEach visitors run mid-pass)
  double-frees guest values — observed as WASM memory corruption.
  identities is cleared per pass so freed-pointer reuse cannot alias
  entries across passes.
- Byte-cache lifecycle: terminal drain now shares the per-VM cache with
  the suspension path (re-serializing an op at drain could re-invoke
  getters and produce different bytes for what the log treats as one
  value), and entries for settled ops — which neither collection filter
  can match again — are evicted, bounding the cache by the live pending
  set.

* Adopt quickjs-wasi 3.3.1: withScope handle sweeping, real memoryLimit accounting, loud unregistered-callback failures

3.3.1 ships the three fixes this branch surfaced upstream:

- Borrowed host-callback handles (vercel-labs/quickjs-wasi#31): the
  trampoline's this/argv handles are scope-exempt, making vm.withScope
  safe around host callbacks. The serde's module-owned pass-disposal
  apparatus (passDisposal/track/runWithPassDisposal and ~18 track()
  wraps) is replaced by withScope in serialize/deserialize — simpler,
  and strictly more complete: every handle constructed during the pass
  is swept, not just the ones our creation funnels saw. Bench parity
  confirmed (within ~10% on the 50k-node extreme case, unchanged
  elsewhere; still 2.6-100x over the in-VM codec).
- Real memoryLimit accounting (vercel-labs/quickjs-wasi#33): the
  engine's 256 MB VM ceiling now actually bounds retained guest
  allocations (usable-size was 0 on wasm32-wasi before, so the limit
  never accumulated).
- Unregistered host callbacks throw (vercel-labs/quickjs-wasi#34):
  guest calls into missing callbacks fail loud instead of silently
  returning undefined — protection this engine wants for
  snapshot-restore re-registration bugs.

Also merges origin/main (undici 7.29.0).

* QuickJS engine: baseline-snapshot startup optimization

Evaluating the workflow bundle dominates VM startup (~74ms of a ~77ms
boot for the 1.3MB e2e bundle) and full event replay pays it on EVERY
invocation — a large share of the quickjs engine's TTFS gap vs node:vm,
where V8 compiles the same script in single-digit ms. The bundle is
identical across all runs of a deployment, so the engine now hydrates
one VM per function instance (bootstrap + bundle eval), snapshots its
memory, and starts every invocation with QuickJS.restore (~3ms) instead
of re-evaluating.

Measured on the real generated e2e flow bundle (154 workflows), boot to
first suspension: fresh 79.4ms -> restored 3.2ms (24.8x). First
invocation pays hydrate+restore (85.8ms, ~= one fresh boot); every
subsequent invocation — including every replay wake — gets the
discount.

Determinism: replay requires module-scope user code to observe the
run-seeded PRNG and deterministic clock, and a restored heap carries
whatever module scope computed at hydrate time. The hydrate therefore
runs with draw-counting placeholder host fns and a read-counting clock;
a bundle that consumed either is marked ineligible and every invocation
falls back to fresh evaluation (node:vm-parity semantics preserved
exactly). When the gate passes, restore is byte-equivalent to fresh
eval: the per-run host fns (random / __generateNanoid / __generateUlid)
re-register by NAME on the restored VM before the workflow body runs,
so the seeded draw sequence — and every correlationId — is identical.
Pinned by a parity test that feeds Math.random() into a step input and
byte-compares the serialized ops across fresh, first-restore and
cached-restore invocations.

Cache: per function instance, keyed on the bundle string
(reference-stable in generated flow routes), promise-deduped for
concurrent first invocations, capped at 4 entries; hydrate rejections
evict for retry while eval failures cache as ineligible (the fresh path
re-evaluates and surfaces the real, source-mapped error).

Kill switch: WORKFLOW_QUICKJS_BASELINE_SNAPSHOT=0.

* Address review: intrinsics-replacement gate, hydrate-failure fallback, shared clock helper, review nits

- Serialization-intrinsics gate (the substantive finding): the restore
  path's serde captures intrinsics from the restored heap — AFTER module
  scope ran — while the fresh path captures before user code. A bundle
  that replaced a captured intrinsic at module scope (e.g. a
  Date.prototype.toISOString polyfill) without touching PRNG/clock
  passed the eligibility gate yet would serialize differently on the two
  paths. captureIntrinsicsSignature (exported from quickjs-serde)
  identity-fingerprints every to-be-captured value; the hydrate compares
  it before and after bundle eval and marks any replacement ineligible.
  Expression-created entries (makeSparseArray, makeThunk, hasOwnCall)
  are excluded — they get fresh identities per eval and cannot be
  replaced by user code. Gate test added with a toISOString polyfill.
- Hydrate-failure fallback: a getBaselineEntry rejection (infrastructure
  — vm.snapshot() under memory pressure, QuickJS.create failing) no
  longer fails the invocation; it logs a warning and falls back to fresh
  evaluation, with the cached promise already evicted for retry.
- initWorkflowVM now uses the shared makeDeterministicClockWasi helper
  its doc claimed it shared, so the two clock implementations cannot
  drift.
- getCompiledAssets() awaited once per call site (hydrate + restore).
- WORKFLOW_TURBO JSDoc reattached to isTurboEnabled (the baseline
  constant had been inserted between doc and function).
- Parity test saves/restores any pre-existing
  WORKFLOW_QUICKJS_BASELINE_SNAPSHOT env value instead of deleting it.

* Fix source-map remapping for workflows sharing a baseline snapshot

The baseline cache is keyed on the bundle, which every workflow in a
deployment shares — but the hydrate evaluated the bundle with the FIRST
caller's workflowId as the eval filename. That name is baked into the
snapshot's compiled code, so on the restore path every OTHER workflow's
stack frames referenced the first hydrator's id, and remapErrorStack
(which matches frames by the failing run's module specifier) never
matched them — raw bundle line numbers leaked into user-visible stacks
for any workflow outside the first hydrator's module.

Hydrate now evaluates under a workflow-independent constant
(BASELINE_BUNDLE_FILENAME), and the entrypoint's three remap sites
(failed-branch stack, hydrated error, cause chain) remap against BOTH
filename spaces — the run's module specifier covers fresh-path frames,
the constant covers snapshot-path frames; remapErrorStack early-exits
on a cheap includes() for whichever space has no frames.

Regression test: a two-module bundle hydrated under workflow A, with
workflow B failing through the restored snapshot — B's stack must
reference the constant filename and not A's id.

* Address review: lossless lone-surrogate string extraction, portable base64

P1 — the guestString length check was insufficient: JS_ToCString has
TWO corruptions (NUL truncation, lone-surrogate -> U+FFFD replacement)
and they can cancel — the replacement expansion offsets the truncation
so the extracted length matches the true guest length. A bare lone
surrogate can also replace 1:1 with no length change at all. Worse,
the JSON.stringify slow path was itself lossy for lone surrogates:
QuickJS passes them through raw, and the C-string extraction of ITS
output corrupts them.

- guestString accepts the fast value only when length matches AND it
  contains no U+FFFD (legitimate U+FFFD strings take the loss-free slow
  path); the slow path now escapes INSIDE the VM to printable ASCII via
  a new captured escapeString intrinsic (WTF-16-safe per-code-unit
  \uXXXX escaping), then JSON-parses host-side.
- shapeOf's fast-key acceptance adds a U+FFFD scan alongside the
  count/duplicate checks (lone-surrogate keys corrupt with count and
  uniqueness intact).
- get()/hasOwn() route keys through guest string handles when they
  carry a NUL or an UNPAIRED surrogate (paired surrogates - emoji keys
  - encode fine through the C-string APIs; vm.newString is verified
  WTF-16-preserving for the handle path).
- Tests: the reviewer's exact length-canceling case, bare lone
  surrogates, legit-U+FFFD passthrough, byte parity with the reference
  codec, and lone-surrogate/mixed keys.

P2 — the codec's base64 helpers no longer carry an unconditional Node
Buffer dependency: feature-detected Uint8Array.fromBase64/toBase64
when available, Buffer when present, btoa/atob loop otherwise —
keeping WASM-only/non-Node hosts (Cloudflare Workers) viable.

* Merge quickjs-host-serde (lossless surrogate extraction, portable base64) into quickjs-baseline-snapshot

The new escapeString captured intrinsic is expression-created (a fresh
guest closure per capture eval), so it joins makeSparseArray/makeThunk/
hasOwnCall in captureIntrinsicsSignature's exclusion list — without
this the baseline hydrate gate would classify every bundle ineligible
(the byte-parity test catches exactly that, as it did when hasOwnCall
was missed).

* Address review: pre-eval serde capture root, adopted by pointer from the snapshot

The intrinsics-replacement gate was structurally losing: its own
post-eval probe executed guest-reachable code (CAPTURE_INTRINSICS calls
Object.getOwnPropertyDescriptor / Object.getPrototypeOf), those
dependencies were not in the identity signature, and a module-scope
stateful wrapper around them both evaded detection AND had its side
effects baked into the snapshot — fresh returned 0 from the reviewer's
counter repro while restore returned the probe's call count.

Replace detection with prevention: ALL guest-touching serde
initialization (intrinsics capture, branded samples, well-known symbol
lookups) is bundled into one CAPTURE_ROOT expression evaluated in the
baseline VM BEFORE the bundle — the same capture-before-user-code
ordering the fresh path has always had. The container handle's box
lives in the snapshot's linear memory, its raw pointer rides the
BaselineEntry, and every restored VM re-adopts it (adoptSerdeRoot) —
serde init then performs only plain-data property reads and C-level
classId reads: NO guest code executes after user code has run, on
either path.

Consequences:
- the identity-signature gate and its expression-created skip-list are
  deleted (nothing to detect — module-scope intrinsic patching is now
  HARMLESS on the snapshot path, not merely detectable)
- polyfill bundles become ELIGIBLE for the optimization and serialize
  through pristine intrinsics identically on both paths (test flipped
  from gating to byte-equality)
- process.env injection converted from guest-source eval to
  handle-based installProcessEnv (captured Object.freeze +
  vm.hostToHandle): the old evalCode ran JSON.parse post-eval on the
  restore path only, the same observable-divergence class
- the reviewer's stateful-wrapper repro is a regression test: the
  counter must be zero and identical across fresh and restored
  invocations

* Adopt quickjs-wasi 3.4.0: delete the NUL/surrogate string machinery

3.4.0 ships lossless string transport (vercel-labs/quickjs-wasi#35 —
found by this PR's review cycle), so the SDK-side detection and escape
machinery is deleted wholesale:

- guestString (length + U+FFFD detection, in-VM escape fallback) — plain
  toString() is lossless now
- the escapeString / hasOwnCall / jsonStringify / objectKeys captured
  intrinsics
- keyNeedsHandleLookup and the handle-keyed get/hasOwn routing — the
  library routes inexpressible keys itself
- shapeOf's guest-key verification pass (count/duplicate/U+FFFD scans) —
  enumeration is lossless

Net ~130 lines and four captured intrinsics removed; the serde now uses
the plain quickjs-wasi surface everywhere.

Test honesty fix that 3.4.0 forced: the earlier lone-surrogate
round-trip tests passed only via mutual corruption — the pre-3.4.0
lossy host→guest transport corrupted the guest comparison literals
identically to the wire. With an honest transport they exposed that the
WIRE itself (devalue emits lone surrogates raw; the wire is UTF-8)
degrades lone surrogates to U+FFFD — in the node engine's reference
codec exactly as here, verified. Bug-compatible parity is the
load-bearing property (event logs replay across engines), so those
tests now assert byte parity with the reference codec plus
guest-observed equality with the reference codec's own round trip; NULs
are devalue-escaped and asserted to survive exactly. Wire-level
surrogate preservation is a product-wide devalue/UTF-8 question,
tracked separately from this engine.

* rerun CI
 2026-08-07 09:56:52 +00:00
Nathan Rajlich 19b5b85c8b QuickJS engine: host-side, side-effect-free serialization via handles (#3263) 
* Add opt-in QuickJS WASM VM engine (WORKFLOW_VM=quickjs) with full event replay

* QuickJS engine: AbortController, setAttributes, terminal drain, turbo-safe requeue, stable PRNG seed

* QuickJS engine: hook.getConflict support, cross-run writable forwarding symbols

* QuickJS engine: stream framing round-trip, bound step proxies, webhook fidelity

* Apply biome fixes to QuickJS engine files

* Address review feedback: anchor source-map strip to end-of-input, use getWorkflowQueueName for conflict requeue, Buffer-free asset decoding, function replacers for payload injection, maxEventsLimit guard

* CI: include generated QuickJS source assets in shared e2e build artifacts

* Fix same-token hook ordering and conflicted-hook disposal in the QuickJS engine

* CI: run both VM engines across all frameworks and worlds; label jobs with the engine

* Fix stack overflow stripping inline source maps from webpack dev bundles; harden step-listing e2e assertions against eventually-consistent reads

* e2e: poll step listings until analytics rows include attempt (optional column can lag terminal status)

* e2e: use --withData to force storage-backed step listings for attempt assertions (analytics listing can omit attempt entirely)

* Sort imports in QuickJS serialization files (biome organizeImports)

* QuickJS engine: resolve the run's full payload-key capability so sealed (encp) hook payloads open

Main's sealed-box work (#3096) makes cross-deployment resumeHook() seal
hook payloads to the target run's published X25519 public key. The shared
start() path publishes that key regardless of engine, so QuickJS runs
receive sealed payloads too — but the QuickJS entrypoint resolved only the
bare symmetric key via importKey(), which cannot open encp envelopes. The
first sealed hook payload wedged the run right after hook_received, timing
out every hook/webhook e2e on Vercel prod (node:vm legs were fine — the
node engine resolves the full capability via memoizeEncryptionKey).

Resolve deriveRunPayloadKeys() in the entrypoint instead and widen the
runtime's key types from CryptoKey to DecryptionKey. Writes stay symmetric
(encrypt() with RunPayloadKeys takes the encr path). Regression test seals
a payload exactly as resumeHook does and round-trips it through the VM.

* Address review: crypto/process parity, loud Intl guards, lazy engine import, VM-leak guard, telemetry namespace, eval-string escaping

- Deterministic crypto.getRandomValues/randomUUID in the VM bootstrap,
  drawing from the seeded Math.random (identical sequences to the node
  engine's vm/index.ts implementations); all crypto.subtle methods throw
  with step-function guidance. process.env exposed as a frozen copy,
  matching node.
- Intl: throwing constructors (no ICU in QuickJS), and toLocale*-family
  methods (incl. localeCompare) throw when given an explicit locale so
  cross-engine divergence is loud instead of silently writing different
  values into the event log. No-argument forms keep working.
- runtime.ts lazy-imports the QuickJS entrypoint at dispatch, keeping the
  ~1.3MB embedded WASM assets out of node-engine deployments.
- runQuickJSWorkflow wraps the per-run phase so an exceptional exit
  disposes the VM instead of leaking it in a reused compute instance;
  corrected the misleading fail-loud comment (run_failed, not retry);
  warn when the event drain loop exhausts its iteration bound.
- Telemetry attributes renamed quickjs.* → workflow.vm.* to stay in the
  file's workflow.* namespace.
- Eval-string correlation-id interpolation uses JSON.stringify instead of
  quote-only escaping.
- common-vm.test.ts pins the reducer/reviver superset invariant against
  common.ts so the duplicated sets can't silently drift.
- Docs enumerate the remaining global-surface differences (subtle.digest,
  Intl, WebAssembly, Atomics); quickjs-entrypoint documents the known
  precondition-guard gap.

* QuickJS engine: implement resilient resumeHook (hookInput materialization + resumeId dedup)

#1834 made resumeHook() fall back to enqueueing the run with a hookInput
payload when the direct hook_received write fails transiently, with the
runtime materializing the missing event on delivery. Only the node:vm
path implemented it — the QuickJS dispatch returned before the node
block, so the resilient payload was silently dropped and the new e2e
timed out on every quickjs leg.

- runtime.ts threads hookInput into runWorkflowWithQuickJS; the
  entrypoint materializes the missing hook_received after loading the
  event log (resumeId-keyed dedup, occurredAt from the resumeId ULID,
  local eventData substitution for lazy/ref responses, EntityConflict /
  HookNotFound handling) — mirroring the node block.
- processEvents drops duplicate hook_received rows sharing a resumeId
  (first-in-log wins), matching the node engine's EventsConsumer dedup;
  the seen-set lives in the VM heap so it is deterministic per replay.

Verified against the dev server with WORKFLOW_VM=quickjs: the resilient
resume e2e passes and the materialization is observable in the logs; all
27 hook e2e tests green.

* QuickJS engine: inline step execution via live-VM continuation loop + WASM module caching

* Address review: exclusive inline step claims, self-write requeue, in-loop event ceiling

- Inline steps now claim via a lazy step_started carrying the input
  (step_created deferred, atomic create-claim in the world), with
  ownerMessageId stamped and authoritativeAttempt=1 — a concurrent
  invocation racing on the same fresh step loses with
  EntityConflictError and skips instead of both bare-starting the step
  and double-running the body. This also removes the stepsCreatedByUs
  set, whose 'created by us' invariant didn't survive the swallowed
  create-race conflict; redelivery backstops now key on hasCreatedEvent.
- dispatchPendingOps' createdAttributeEvent/createdGetConflictHook
  signals are consumed again: when the loop exits suspended without ever
  reading back a self-written attr_set / getConflict hook_created
  (eventually-consistent listing lag), the entrypoint requeues
  immediately instead of parking the run awaiting_external with its
  unblocking event already written.
- The server-supplied event ceiling is re-checked at the top of every
  continuation-loop turn (seenEventIds.size), so a single invocation
  fanning out inline can no longer grow the log arbitrarily past the
  operator's limit. The quickjs dispatch in runtime.ts converts
  MaxEventsExceededError into run_failed / MAX_EVENTS_EXCEEDED — the
  guard's throw previously nacked forever, parking runaway runs in
  'running'.
- Documented the deliberate decision that the platform function timeout
  is the only bound on inline chaining (budget parked per batch),
  matching the node engine.

* QuickJS engine: host-side, side-effect-free serialization via handles

(Re-applied onto the review-fixed base; original commits da2723016 +
9814ed9ac squashed.)

Replace the in-VM serde bundle with a host-side codec
(runtime/quickjs-serde.ts) built on quickjs-wasi 3.3's introspection
primitives and devalue 5.9's pluggable stringify/parse operations —
mirroring the node:vm engine's architecture.

Review fixes incorporated:
- reducer/reviver key sets are pinned against codec-devalue-vm's
  workflow mode by exhaustiveness tests (exact order for reducers —
  first match wins), so the handle-space codec can't silently drift
  from the shared value-space sets.
- the devalue entry in minimumReleaseAgeExclude is removed: the exact
  version is pinned via the workspace catalog + lockfile, so the
  cooldown waiver was unnecessary (verified with both frozen and
  regular installs).
- eval-string interpolation inherits the JSON.stringify(cid) hardening
  from the base branch.

* Address review: NUL-safe string extraction, deterministic retryAfter, pass-scoped handle disposal, byte-cache lifecycle

- NUL (U+0000) safety across the WASM boundary: handle.toString() routes
  through JS_ToCString and silently truncates at the first NUL, and the
  C-string key APIs mangle NUL-bearing property keys (drop or collide).
  guestString() detects truncation by comparing against the handle's
  true guest length and recovers via in-VM JSON.stringify escaping;
  shapeOf verifies its fast host-string key list against a guest
  Object.keys count (+ duplicate check) and re-extracts through key
  handles on mismatch; get/hasOwn route NUL-bearing keys through
  length-aware guest string handles. All string funnels (primitives,
  symbol descriptions, error fields via chained/own reads, Headers
  entries, RegExp source/flags, URL href) go through guestString.
  Regression-tested down to the truncate-vs-collide enumeration shapes;
  fixes nullByteWorkflow on the quickjs e2e legs.
- RetryableError's absent/invalid retryAfter fallback now reads the
  GUEST clock (the deterministic replay clock at the WASI layer) via a
  captured Date.now instead of the host wall clock — the in-VM reducer
  was replay-stable by construction and the host port silently lost
  that.
- Pass-scoped handle disposal: serialize/deserialize sweep every
  intermediate handle their pass creates (call/invoke results,
  descriptor reads, dups, parse-op constructions), closing the
  ~one-leaked-handle-per-value-node growth across long-lived inline
  sessions. Implemented with module-owned tracking rather than
  vm.withScope: the library scope also captures the handles the
  host-callback trampoline wraps around C-owned argv pointers, and
  disposing those (Map/Set/Headers forEach visitors run mid-pass)
  double-frees guest values — observed as WASM memory corruption.
  identities is cleared per pass so freed-pointer reuse cannot alias
  entries across passes.
- Byte-cache lifecycle: terminal drain now shares the per-VM cache with
  the suspension path (re-serializing an op at drain could re-invoke
  getters and produce different bytes for what the log treats as one
  value), and entries for settled ops — which neither collection filter
  can match again — are evicted, bounding the cache by the live pending
  set.

* Adopt quickjs-wasi 3.3.1: withScope handle sweeping, real memoryLimit accounting, loud unregistered-callback failures

3.3.1 ships the three fixes this branch surfaced upstream:

- Borrowed host-callback handles (vercel-labs/quickjs-wasi#31): the
  trampoline's this/argv handles are scope-exempt, making vm.withScope
  safe around host callbacks. The serde's module-owned pass-disposal
  apparatus (passDisposal/track/runWithPassDisposal and ~18 track()
  wraps) is replaced by withScope in serialize/deserialize — simpler,
  and strictly more complete: every handle constructed during the pass
  is swept, not just the ones our creation funnels saw. Bench parity
  confirmed (within ~10% on the 50k-node extreme case, unchanged
  elsewhere; still 2.6-100x over the in-VM codec).
- Real memoryLimit accounting (vercel-labs/quickjs-wasi#33): the
  engine's 256 MB VM ceiling now actually bounds retained guest
  allocations (usable-size was 0 on wasm32-wasi before, so the limit
  never accumulated).
- Unregistered host callbacks throw (vercel-labs/quickjs-wasi#34):
  guest calls into missing callbacks fail loud instead of silently
  returning undefined — protection this engine wants for
  snapshot-restore re-registration bugs.

Also merges origin/main (undici 7.29.0).

* Address review: lossless lone-surrogate string extraction, portable base64

P1 — the guestString length check was insufficient: JS_ToCString has
TWO corruptions (NUL truncation, lone-surrogate -> U+FFFD replacement)
and they can cancel — the replacement expansion offsets the truncation
so the extracted length matches the true guest length. A bare lone
surrogate can also replace 1:1 with no length change at all. Worse,
the JSON.stringify slow path was itself lossy for lone surrogates:
QuickJS passes them through raw, and the C-string extraction of ITS
output corrupts them.

- guestString accepts the fast value only when length matches AND it
  contains no U+FFFD (legitimate U+FFFD strings take the loss-free slow
  path); the slow path now escapes INSIDE the VM to printable ASCII via
  a new captured escapeString intrinsic (WTF-16-safe per-code-unit
  \uXXXX escaping), then JSON-parses host-side.
- shapeOf's fast-key acceptance adds a U+FFFD scan alongside the
  count/duplicate checks (lone-surrogate keys corrupt with count and
  uniqueness intact).
- get()/hasOwn() route keys through guest string handles when they
  carry a NUL or an UNPAIRED surrogate (paired surrogates - emoji keys
  - encode fine through the C-string APIs; vm.newString is verified
  WTF-16-preserving for the handle path).
- Tests: the reviewer's exact length-canceling case, bare lone
  surrogates, legit-U+FFFD passthrough, byte parity with the reference
  codec, and lone-surrogate/mixed keys.

P2 — the codec's base64 helpers no longer carry an unconditional Node
Buffer dependency: feature-detected Uint8Array.fromBase64/toBase64
when available, Buffer when present, btoa/atob loop otherwise —
keeping WASM-only/non-Node hosts (Cloudflare Workers) viable.

* Adopt quickjs-wasi 3.4.0: delete the NUL/surrogate string machinery

3.4.0 ships lossless string transport (vercel-labs/quickjs-wasi#35 —
found by this PR's review cycle), so the SDK-side detection and escape
machinery is deleted wholesale:

- guestString (length + U+FFFD detection, in-VM escape fallback) — plain
  toString() is lossless now
- the escapeString / hasOwnCall / jsonStringify / objectKeys captured
  intrinsics
- keyNeedsHandleLookup and the handle-keyed get/hasOwn routing — the
  library routes inexpressible keys itself
- shapeOf's guest-key verification pass (count/duplicate/U+FFFD scans) —
  enumeration is lossless

Net ~130 lines and four captured intrinsics removed; the serde now uses
the plain quickjs-wasi surface everywhere.

Test honesty fix that 3.4.0 forced: the earlier lone-surrogate
round-trip tests passed only via mutual corruption — the pre-3.4.0
lossy host→guest transport corrupted the guest comparison literals
identically to the wire. With an honest transport they exposed that the
WIRE itself (devalue emits lone surrogates raw; the wire is UTF-8)
degrades lone surrogates to U+FFFD — in the node engine's reference
codec exactly as here, verified. Bug-compatible parity is the
load-bearing property (event logs replay across engines), so those
tests now assert byte parity with the reference codec plus
guest-observed equality with the reference codec's own round trip; NULs
are devalue-escaped and asserted to survive exactly. Wire-level
surrogate preservation is a product-wide devalue/UTF-8 question,
tracked separately from this engine.
 2026-08-06 14:47:08 -07:00
Karthik Kalyan 439a495a71 fix(core): pre-check deployment affinity before the lazy resume write (#3374) 
The lazy hook fast path (#3345) hoisted the consumer's hook_received
write above the deployment-affinity guard (#2960), so a misrouted lazy
resume wrote its event before the guard could re-route the delivery.

Stamp the run's pinned deployment on the resume message
(hookInput.deploymentId, from the producer's resume context) and, on
the consumer, compare it against the ambient deployment id immediately
before the fast path: a match continues with no run fetch, a mismatch
fetches the authoritative run and hands it to the existing guard —
which keeps sole ownership of re-route/fail policy and remains the
authoritative protection before replay and step execution. The
re-routed message preserves the complete hookInput (it may hold the
only copy of the resume payload). Older messages without the field, and
worlds without deployment affinity, are unchanged: they skip the
pre-check and rely on the authoritative guard, the pre-guard write
staying convergent per (runId, resumeId).

Fixes the misrouted-lazy-resume unit test broken by the #2960/#3345
ordering: a modern misrouted resume now re-routes with zero event
writes, asserted for both hook_received and run_started.

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
 2026-08-06 14:22:25 -07:00
Peter Wielander bf4dda6478 [world-vercel] Recover from wedged HTTP/2 events connections (#3370) 2026-08-06 11:45:41 -07:00
github-actions[bot] e6af70b9d9 Version Packages (beta) (#3318) 
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
workflow@5.0.0-beta.40 		2026-08-06 09:02:00 -07:00
Karthik Kalyan 9c1b3c8638 perf(core): initialize lazy hook replay from hook_received stream (#3345) 
* perf(core): initialize lazy hook replay from hook_received stream

On a lazy hook queue delivery, the consumer's idempotent hook_received
re-ensure is hoisted above run_started and doubles as the invocation's
setup request: it asks the World to return the current replay log with
the write (new advisory CreateEventParams.preloadEvents), so one HTTP
request yields the canonical event, the reconstructed run, and the
complete replay log — skipping both the run_started POST and the
initial events.list.

- world: optional `preloadEvents?: true` on CreateEventParams, the
  hook_received dual of skipPreload; Worlds may ignore it
- world-vercel: createHookReceivedPreloadEventV4 sends the frame Accept
  on eligible hook_received posts and decodes either response mode —
  frames via the response decoder extracted from the LIST consumer
  (GET behavior unchanged), CBOR via the shared materialized-response
  mapping. The run is reconstructed from run_created/run_started (plus
  attr_set folds), the canonical event found by x-wf-event-id, and
  resumeId now survives frame decoding so the runtime can match it
- core: new fast path before the generic run-state setup, guarded on
  hookInput.resumeId + payloadDigest; a validated COMPLETE preload
  (hasMore false — this path has no cursor-continuation machinery)
  initializes workflowRun/preloadedEvents/maxEventsLimit directly,
  anything else falls back to the run_started setup without re-posting
  the hook; error classification matches the existing re-ensure
  (terminal → consume, transient → redeliver); setup source reported
  via workflow.resume_setup_source (never
  workflow.hook.resilient_resume_materialized, which stays a
  recovery-only signal)
- producer resumeHook() is unchanged and never sets preloadEvents

Based directly on main (no dependency on #3124/#3191); pairs with
workflow-server's streamed hook_received replay-log response, which
deploys first — the SDK negotiates per request and falls back safely
against older servers.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* address review: lazy fallback, retryable resume, terminal telemetry

- world-vercel: the preload request keeps hook_received's lazy
  remoteRefBehavior — a supporting server owns frame-body resolution,
  while an older server now answers the CBOR fallback without resolving
  an S3-backed payload the runtime would discard
- world-vercel: the atomic lazy-resume shape (resumeId + digest) opts
  into withEventPostRetry via idempotentHookResume — the (runId,
  resumeId) claim makes the POST idempotent-on-retry; legacy/partial
  hook_received shapes stay single-attempt, definitive 4xx stays
  non-retryable (unit + adapter + trace-propagation coverage)
- core: a terminal event found in the preload records
  workflow.resume_setup_source=hook_received_stream and the run's
  actual terminal status on the span before consuming the delivery
- core: document resilient_resume_materialized as the legacy/non-atomic
  re-ensure signal (claim ownership is not observable client-side, so
  the hoisted path deliberately never emits it) and resume_setup_source
  as a latency signal, not proof of event creation; note the Option A
  skip is now unreachable for atomic resumes
- world: spell out the full preload usability contract on preloadEvents
  (complete hasMore-false log, non-null cursor, run/startedAt/maxEvents,
  lifecycle events, matching resumeId, list ordering, read-after-write
  consistency); bump @workflow/world to minor
- new QuickJS sourcing tests (VM mocked): an attested complete preload
  is used verbatim with no events.list, a non-attested hook-containing
  preload is refetched, and an attested empty preload is not trusted

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
 2026-08-05 19:24:55 -07:00
Mitul Shah 95e292e0c1 Refresh the Workflow SDK README (#3357) 
* docs: refresh Workflow SDK README

Signed-off-by: Cursor Agent <cursoragent@cursor.com>

* docs: fix README deployment link

Signed-off-by: Cursor Agent <cursoragent@cursor.com>

* docs: restore bug bounty guidance

Signed-off-by: Cursor Agent <cursoragent@cursor.com>

* docs: tighten README copy

Signed-off-by: Cursor Agent <cursoragent@cursor.com>

* docs: clarify workflow suspension copy

Signed-off-by: Cursor Agent <cursoragent@cursor.com>

* docs: add code of conduct

Signed-off-by: Cursor Agent <cursoragent@cursor.com>

* docs: remove README workflow example

Signed-off-by: Cursor Agent <cursoragent@cursor.com>

* Apply suggestions from code review

Co-authored-by: vercel[bot] <35613825+vercel[bot]@users.noreply.github.com>
Signed-off-by: Mitul Shah <mitulxshah@gmail.com>

* docs: restore security disclosure wording

Signed-off-by: Cursor Agent <cursoragent@cursor.com>

* Apply suggestions from code review

Co-authored-by: Mitul Shah <mitulxshah@gmail.com>
Signed-off-by: Mitul Shah <mitulxshah@gmail.com>

---------

Signed-off-by: Cursor Agent <cursoragent@cursor.com>
Signed-off-by: Mitul Shah <mitulxshah@gmail.com>
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: vercel[bot] <35613825+vercel[bot]@users.noreply.github.com>
 2026-08-05 14:59:11 -07:00
Karthik Kalyan 371f06e5ac feat(web): bulk-cancel selected runs from the runs table (#3349) 
* feat(cli): bulk-cancel runs in a single operation

Replace the per-run cancel loop in `workflow cancel` with one `cancelRuns`
call, validate `--limit` (1-500), print a compact outcome summary with
per-run lines for surfaced failures, and exit nonzero only when a run fails.
The bulk logic lives in a dependency-injected `performBulkCancel` helper so it
is unit-testable without an oclif harness.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* fix(cli): address bulk cancel review feedback

* feat(web): bulk-cancel selected runs in a single request

Thread a bulkCancelRuns action through the server action, RPC route,
rpc-client, and client wrappers, backed by core's cancelRuns. The runs table
now cancels the selected pending/running runs in one call, caps a batch at
BULK_CANCEL_MAX_RUN_IDS (disabling the button with guidance above the cap),
and reports a single outcome-summary toast covering only the categories that
occurred.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
 2026-08-05 14:54:33 -07:00
Karthik Kalyan 2150798ca6 feat(cli): bulk-cancel runs in a single operation (#3348) 
* feat(cli): bulk-cancel runs in a single operation

Replace the per-run cancel loop in `workflow cancel` with one `cancelRuns`
call, validate `--limit` (1-500), print a compact outcome summary with
per-run lines for surfaced failures, and exit nonzero only when a run fails.
The bulk logic lives in a dependency-injected `performBulkCancel` helper so it
is unit-testable without an oclif harness.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* fix(cli): address bulk cancel review feedback

---------

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
 2026-08-05 14:54:21 -07:00
Elliot Dauber 72efc90f28 Use runtime deadline for inline execution limit (#3360) 
* Use runtime deadline for inline execution limit

* up

* lazy import

* Update packages/world/src/interfaces.ts

Co-authored-by: Peter Wielander <mittgfu@gmail.com>
Signed-off-by: Elliot Dauber <67391073+elliotdauber@users.noreply.github.com>

---------

Signed-off-by: Elliot Dauber <67391073+elliotdauber@users.noreply.github.com>
Co-authored-by: Peter Wielander <mittgfu@gmail.com>
 2026-08-05 14:32:13 -07:00
Alex Langenfeld 79e4c04409 fix(core): re-route runs delivered to the wrong deployment (#2960) 
## Summary & Motivation

A queue callback that reaches a deployment other than the one its run is pinned to derives the per-run encryption key from the wrong master key, so the delivery fails before user code runs and the run dies as a blank "exceeded max retries". The delivery is re-enqueued explicitly addressed to the run's own deployment — strictly better-targeted than the send that misrouted — and the run is failed with the new `DEPLOYMENT_MISMATCH` error code only once `WORKFLOW_DEPLOYMENT_MISMATCH_MAX_RETRIES` (default 3) is spent. Gated on the new World capability `deploymentAffinity`, so worlds with synthetic or version-tagged deployment ids are unaffected.

## Test Plan

Unit tests added for the guard and both runtime paths; local vitest and typechecks pass.
 2026-08-05 14:57:37 -05:00
Karthik Kalyan 8d479283ca feat(world,world-vercel,core): bulk run cancellation primitive (#3347) 
* feat(world,world-vercel,core): bulk run cancellation primitive

Add a bulk cancellation contract to @workflow/world (schemas, types, and an
optional Storage['runs'].cancelMany method), implement it in
@workflow/world-vercel via a single POST /v4/runs/cancel request, and add a
cancelRuns runtime helper to @workflow/core that uses the world fast path
when available and otherwise falls back to bounded-concurrency (max 20)
single-run cancellation.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* Update packages/world/src/interfaces.ts

Co-authored-by: Peter Wielander <mittgfu@gmail.com>
Signed-off-by: Karthik Kalyan <105607645+karthikscale3@users.noreply.github.com>

---------

Signed-off-by: Karthik Kalyan <105607645+karthikscale3@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Co-authored-by: Peter Wielander <mittgfu@gmail.com>
 2026-08-05 12:06:14 -07:00
Nathan Colosimo 939ffb4f51 fix(core): reject unsupported Hook retention inside QuickJS (#3332) 
* fix(core): reject unsupported Hook retention inside QuickJS

* refactor(core): mirror World capabilities in QuickJS
 2026-08-05 10:41:03 -07:00
Peter Wielander 2eddf74cb6 Send the run id on correlation-id event reads (#3334) 2026-08-05 09:54:09 -07:00
Rich Haines a3331ac0f6 docs: add inbound cross-links to orphaned v4 docs pages (#3355) 
These pages had no inbound links from other docs pages' content (only
sidebar/card navigation), so they were unreachable through prose. Adds
one minimal cross-link each from a parent index or closely related page.
 2026-08-05 09:05:03 -07:00
Pranay Prakash 1222aab74d chore(deps): upgrade undici to 7.29.0 (#3315) 
Signed-off-by: Pranay Prakash <pranay.gp@gmail.com>
Signed-off-by: Nathan Colosimo <110621881+NathanColosimo@users.noreply.github.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: Nathan Colosimo <110621881+NathanColosimo@users.noreply.github.com>
 2026-08-04 21:59:06 +00:00
Nathan Rajlich a8bf8db84e QuickJS engine: inline step execution + WASM module caching (#3049) 
* Add opt-in QuickJS WASM VM engine (WORKFLOW_VM=quickjs) with full event replay

* QuickJS engine: AbortController, setAttributes, terminal drain, turbo-safe requeue, stable PRNG seed

* QuickJS engine: hook.getConflict support, cross-run writable forwarding symbols

* QuickJS engine: stream framing round-trip, bound step proxies, webhook fidelity

* Apply biome fixes to QuickJS engine files

* Address review feedback: anchor source-map strip to end-of-input, use getWorkflowQueueName for conflict requeue, Buffer-free asset decoding, function replacers for payload injection, maxEventsLimit guard

* CI: include generated QuickJS source assets in shared e2e build artifacts

* Fix same-token hook ordering and conflicted-hook disposal in the QuickJS engine

* CI: run both VM engines across all frameworks and worlds; label jobs with the engine

* Fix stack overflow stripping inline source maps from webpack dev bundles; harden step-listing e2e assertions against eventually-consistent reads

* e2e: poll step listings until analytics rows include attempt (optional column can lag terminal status)

* e2e: use --withData to force storage-backed step listings for attempt assertions (analytics listing can omit attempt entirely)

* Sort imports in QuickJS serialization files (biome organizeImports)

* QuickJS engine: resolve the run's full payload-key capability so sealed (encp) hook payloads open

Main's sealed-box work (#3096) makes cross-deployment resumeHook() seal
hook payloads to the target run's published X25519 public key. The shared
start() path publishes that key regardless of engine, so QuickJS runs
receive sealed payloads too — but the QuickJS entrypoint resolved only the
bare symmetric key via importKey(), which cannot open encp envelopes. The
first sealed hook payload wedged the run right after hook_received, timing
out every hook/webhook e2e on Vercel prod (node:vm legs were fine — the
node engine resolves the full capability via memoizeEncryptionKey).

Resolve deriveRunPayloadKeys() in the entrypoint instead and widen the
runtime's key types from CryptoKey to DecryptionKey. Writes stay symmetric
(encrypt() with RunPayloadKeys takes the encr path). Regression test seals
a payload exactly as resumeHook does and round-trips it through the VM.

* Address review: crypto/process parity, loud Intl guards, lazy engine import, VM-leak guard, telemetry namespace, eval-string escaping

- Deterministic crypto.getRandomValues/randomUUID in the VM bootstrap,
  drawing from the seeded Math.random (identical sequences to the node
  engine's vm/index.ts implementations); all crypto.subtle methods throw
  with step-function guidance. process.env exposed as a frozen copy,
  matching node.
- Intl: throwing constructors (no ICU in QuickJS), and toLocale*-family
  methods (incl. localeCompare) throw when given an explicit locale so
  cross-engine divergence is loud instead of silently writing different
  values into the event log. No-argument forms keep working.
- runtime.ts lazy-imports the QuickJS entrypoint at dispatch, keeping the
  ~1.3MB embedded WASM assets out of node-engine deployments.
- runQuickJSWorkflow wraps the per-run phase so an exceptional exit
  disposes the VM instead of leaking it in a reused compute instance;
  corrected the misleading fail-loud comment (run_failed, not retry);
  warn when the event drain loop exhausts its iteration bound.
- Telemetry attributes renamed quickjs.* → workflow.vm.* to stay in the
  file's workflow.* namespace.
- Eval-string correlation-id interpolation uses JSON.stringify instead of
  quote-only escaping.
- common-vm.test.ts pins the reducer/reviver superset invariant against
  common.ts so the duplicated sets can't silently drift.
- Docs enumerate the remaining global-surface differences (subtle.digest,
  Intl, WebAssembly, Atomics); quickjs-entrypoint documents the known
  precondition-guard gap.

* QuickJS engine: implement resilient resumeHook (hookInput materialization + resumeId dedup)

#1834 made resumeHook() fall back to enqueueing the run with a hookInput
payload when the direct hook_received write fails transiently, with the
runtime materializing the missing event on delivery. Only the node:vm
path implemented it — the QuickJS dispatch returned before the node
block, so the resilient payload was silently dropped and the new e2e
timed out on every quickjs leg.

- runtime.ts threads hookInput into runWorkflowWithQuickJS; the
  entrypoint materializes the missing hook_received after loading the
  event log (resumeId-keyed dedup, occurredAt from the resumeId ULID,
  local eventData substitution for lazy/ref responses, EntityConflict /
  HookNotFound handling) — mirroring the node block.
- processEvents drops duplicate hook_received rows sharing a resumeId
  (first-in-log wins), matching the node engine's EventsConsumer dedup;
  the seen-set lives in the VM heap so it is deterministic per replay.

Verified against the dev server with WORKFLOW_VM=quickjs: the resilient
resume e2e passes and the materialization is observable in the logs; all
27 hook e2e tests green.

* QuickJS engine: inline step execution via live-VM continuation loop + WASM module caching

* Address review: exclusive inline step claims, self-write requeue, in-loop event ceiling

- Inline steps now claim via a lazy step_started carrying the input
  (step_created deferred, atomic create-claim in the world), with
  ownerMessageId stamped and authoritativeAttempt=1 — a concurrent
  invocation racing on the same fresh step loses with
  EntityConflictError and skips instead of both bare-starting the step
  and double-running the body. This also removes the stepsCreatedByUs
  set, whose 'created by us' invariant didn't survive the swallowed
  create-race conflict; redelivery backstops now key on hasCreatedEvent.
- dispatchPendingOps' createdAttributeEvent/createdGetConflictHook
  signals are consumed again: when the loop exits suspended without ever
  reading back a self-written attr_set / getConflict hook_created
  (eventually-consistent listing lag), the entrypoint requeues
  immediately instead of parking the run awaiting_external with its
  unblocking event already written.
- The server-supplied event ceiling is re-checked at the top of every
  continuation-loop turn (seenEventIds.size), so a single invocation
  fanning out inline can no longer grow the log arbitrarily past the
  operator's limit. The quickjs dispatch in runtime.ts converts
  MaxEventsExceededError into run_failed / MAX_EVENTS_EXCEEDED — the
  guard's throw previously nacked forever, parking runaway runs in
  'running'.
- Documented the deliberate decision that the platform function timeout
  is the only bound on inline chaining (budget parked per batch),
  matching the node engine.

* Fix lost wait continuation for waits that elapse mid-iteration (sleepWinsRace flake)

The pre-inline wait-continuation sweep skipped waits with
resumeMs <= 0. A wait whose deadline falls between the iteration's
elapsed-wait pass (which saw it as still pending and wrote nothing)
and this sweep got NEITHER a wait_completed NOR a continuation — and
the inline batch then blocked the invocation for the full step
duration with no wake armed anywhere. For Promise.race(step, sleep)
that silently hands the race to the step: the sleep's wait_completed
is never written and the run completes with the wrong winner.

The vulnerable window spans the iteration's dispatch + feed network
round-trips, so on world-vercel a 1s sleep landed in it roughly half
the time (the ~50% sleepWinsRaceWorkflow failure rate in the Vercel
quickjs e2e legs), while world-local's sub-ms round-trips masked it
locally.

Match the node engine (Math.max(1000, resumeAtMs - now) in
suspension-handler.ts): always arm the continuation, clamping
already-elapsed waits to the 1s minimum — the continuation
invocation's pre-VM elapsed check completes them. Waits whose
wait_completed this invocation already wrote are skipped.

Diagnosed from run wrun_41KZ73HR4H0GZ6RYD1WQHZX822 (CI run
30942512953): wait_created at +0.5s for a 1s sleep, no wait_completed
ever, step_completed at +10.8s wins the race.
 2026-08-04 13:56:17 -07:00
Peter Wielander de1905f15c feat(world): require a runId on listByCorrelationId (#3280) 2026-08-04 13:09:35 -07:00
Nathan Colosimo 27a3f15a7b fix(core): preserve Hook retention in QuickJS (#3319) 
Co-authored-by: Nathan Rajlich <n@n8.io>
 2026-08-04 12:04:50 -07:00
christopherkindl 434e4bed2f [docs] upgrade geistdocs to 1.19.4 (#3330) 2026-08-04 11:50:28 -07:00
Mitul Shah 73da40cbb7 fix(web-shared): align colors with Geist (#3300) 
* fix(docs): align Geist colors with Vercel

Co-authored-by: Mitul Shah <mitulxshah@gmail.com>

* chore: add docs color changeset

Signed-off-by: Cursor Agent <cursoragent@cursor.com>

Co-authored-by: Mitul Shah <mitulxshah@gmail.com>

* fix(web-shared): align colors with Geist

Signed-off-by: Cursor Agent <cursoragent@cursor.com>

Co-authored-by: Mitul Shah <mitulxshah@gmail.com>

---------

Co-authored-by: Cursor Agent <cursoragent@cursor.com>
 2026-08-04 09:55:52 -07:00
Karthik Kalyan e084e08ac0 Reduce Vercel E2E polling load (#3316) 
* Reduce Vercel E2E polling load

* Keep Vercel E2E matrix concurrency
 2026-08-03 19:29:59 -07:00
Sepcnt c22abcd5f1 Add SurrealDB as a community world (#1579) 
Squashed and rebased onto main to resolve conflicts with the generic
docker community-world CI: the dedicated surrealdb service steps from
the original commits are replaced by the manifest-driven docker service
type, with a new optional `args` field so the service definition can
pass the `start` subcommand (and credentials) to the SurrealDB image.

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
 2026-08-03 17:43:11 -07:00
Nathan Colosimo 99f4aeb03d feat(world-postgres): support Hook minimum retention (#3276) 
* feat(world-postgres): retain hook tokens after runs end

* refactor(world-postgres): reuse terminal run statuses

* docs: note Postgres Hook retention support

* fix(world-postgres): expose hook retention deadline

* Fix: Exhaustive `Record<AttributeKey, ...>` in `attribute-panel.tsx` is missing the `tokenRetentionUntil` key that was added to `HookSchema`, causing TS2741 and breaking every Vercel build.

This commit fixes the issue reported at packages/web-shared/src/components/sidebar/attribute-panel.tsx:426

## Bug

Commit `ad58321` added `tokenRetentionUntil: z.coerce.date().optional()` to `HookSchema` in `packages/world/src/hooks.ts:106`. This adds `tokenRetentionUntil` to the inferred `Hook` type.

In `packages/web-shared/src/components/sidebar/attribute-panel.tsx`, `AttributeKey` is a union that includes `keyof Hook`, so `tokenRetentionUntil` becomes a required member of the **exhaustive** `Record<AttributeKey, (value: unknown, context?: DisplayContext) => ...>` object literal `attributeToDisplayFn` (starting at line ~426).

Because the literal had no `tokenRetentionUntil` entry, `tsc` fails:

```
src/components/sidebar/attribute-panel.tsx(426,7): error TS2741:
Property 'tokenRetentionUntil' is missing in type '{ ... }' but required in type
'Record<AttributeKey, (value: unknown, context?: DisplayContext | undefined) => ReactNode>'.
```

This breaks `@workflow/web-shared#build` and therefore every Vercel deployment (17 failing deployments observed, all with this identical error).

## Fix

Added a `tokenRetentionUntil` entry to `attributeToDisplayFn`, placed alongside the other Hook date fields (`lastReceivedAt`, `disposedAt`):

```ts
tokenRetentionUntil: timestampWithTooltipOrNull,
```

`tokenRetentionUntil` is a `Date` field, and `timestampWithTooltipOrNull` (defined at line 402) is the display helper used by all the other surfaced date fields (`createdAt`, `startedAt`, `completedAt`, `retryAfter`, `resumeAt`, `occurredAt`). Given the intent of `ad58321` was to expose the hook retention deadline, surfacing it as a tooltip-annotated timestamp is the consistent choice.

Only `attributeToDisplayFn` is a fully exhaustive `Record<AttributeKey, ...>`; the other maps are `Partial<...>` / `Set`, so no other edits are required.

## Verification

`node_modules` are not installed in this sandbox, so `tsc` could not be executed directly. Verified structurally instead: the newly added `tokenRetentionUntil` entry (line 449) references `timestampWithTooltipOrNull`, which is defined in-file at line 402 and already used by the sibling date entries, so the fix satisfies the missing-key requirement without introducing new type errors.

Co-authored-by: Vercel <vercel[bot]@users.noreply.github.com>
Co-authored-by: VaguelySerious <mittgfu@gmail.com>

* docs(world-postgres): clarify expired hook rows

* feat(world-postgres): enforce Hook retention limit

* fix(world): remove duplicate Hook retention field

* fix(web-shared): remove duplicate retention renderer

* test(world): remove redundant retention coercion case

---------

Co-authored-by: vercel[bot] <35613825+vercel[bot]@users.noreply.github.com>
Co-authored-by: Vercel <vercel[bot]@users.noreply.github.com>
Co-authored-by: VaguelySerious <mittgfu@gmail.com>
 2026-08-03 17:42:31 -07:00
Nathan Colosimo e6f1b6f548 feat(world-local): support Hook minimum retention (#2866) 
* feat(core): add hook token retention contract

* refactor(core): constrain hook retention options

* fix(core): preserve boolean hook visibility options

* revert(core): preserve HookOptions interface

* docs(core): clarify retained conflict ownership

* docs(core): retain newest-wins conflict pattern

* docs(core): simplify hook retention guidance

* docs(core): explain retained token cleanup

* docs(core): simplify idempotency guidance

* docs(core): clarify retained token results

* refactor(core): rename hook token expiration option

* chore(core): name hook expiration changeset

* docs(core): simplify Hook expiration language

* docs(core): clarify Hook expiration deadline

* docs(core): remove Hook deadline caveat

* refactor(core): align Hook expiration field names

* docs(core): narrow Hook expiration documentation

* docs(core): clarify hook expiration availability

* Update packages/core/src/workflow/hook.ts

Co-authored-by: Peter Wielander <mittgfu@gmail.com>
Signed-off-by: Nathan Colosimo <110621881+NathanColosimo@users.noreply.github.com>

* docs(core): clarify Hook token expiration behavior

* docs(core): explain active Hook expiration behavior

* feat(world): advertise hook ttl capability

* fix(core): validate hook ttl capability after main merge

* refactor(core): rename hook expiry to minimum retention

* docs: keep hook retention guidance on v5

* docs: define retained run availability

* fix(core): validate Hook retention at creation

* feat(core): define retained Hook lookup semantics

* refactor(core): simplify hook retention checks

* feat(world-local): support Hook token expiration

* fix(world-local): make hook recovery atomic

* refactor(world-local): align Hook minimum retention

* fix(world-local): preserve Hook creation order

* fix(world-local): expose retained Hooks consistently

* refactor(world-local): simplify retained hook storage

* fix(world-local): allow stale lock recovery

* refactor(world-local): simplify hook retention storage

Signed-off-by: Nathan Colosimo <110621881+NathanColosimo@users.noreply.github.com>

* fix(world-local): serialize expired hook token handoff

Signed-off-by: Nathan Colosimo <110621881+NathanColosimo@users.noreply.github.com>

* fix(world-local): preserve hook creation order

Signed-off-by: Nathan Colosimo <110621881+NathanColosimo@users.noreply.github.com>

* refactor(world-local): clarify hook availability cleanup

* docs: note Local World Hook retention support

* fix(world-local): harden hook retention persistence

* fix(web-shared): render hook retention deadline

* fix(world-postgres): exclude unsupported hook retention

* feat(world-local): enforce Hook retention limit

* docs(world-local): clarify retention limit error

* docs(world): clarify Hook retention deadline

* docs(hooks): link retention configuration

---------

Signed-off-by: Nathan Colosimo <110621881+NathanColosimo@users.noreply.github.com>
Co-authored-by: Peter Wielander <mittgfu@gmail.com>
 2026-08-03 17:42:31 -07:00
Peter Wielander cb77725960 [core] Derive correlation ids from per-kind sequences (opt-in) (#3301) 2026-08-03 16:49:15 -07:00
Nathan Rajlich f8f6e17aeb Add opt-in QuickJS WASM VM engine (WORKFLOW_VM=quickjs) (#3048) 
* Add opt-in QuickJS WASM VM engine (WORKFLOW_VM=quickjs) with full event replay

* QuickJS engine: AbortController, setAttributes, terminal drain, turbo-safe requeue, stable PRNG seed

* QuickJS engine: hook.getConflict support, cross-run writable forwarding symbols

* QuickJS engine: stream framing round-trip, bound step proxies, webhook fidelity

* Apply biome fixes to QuickJS engine files

* Address review feedback: anchor source-map strip to end-of-input, use getWorkflowQueueName for conflict requeue, Buffer-free asset decoding, function replacers for payload injection, maxEventsLimit guard

* CI: include generated QuickJS source assets in shared e2e build artifacts

* Fix same-token hook ordering and conflicted-hook disposal in the QuickJS engine

* CI: run both VM engines across all frameworks and worlds; label jobs with the engine

* Fix stack overflow stripping inline source maps from webpack dev bundles; harden step-listing e2e assertions against eventually-consistent reads

* e2e: poll step listings until analytics rows include attempt (optional column can lag terminal status)

* e2e: use --withData to force storage-backed step listings for attempt assertions (analytics listing can omit attempt entirely)

* Sort imports in QuickJS serialization files (biome organizeImports)

* QuickJS engine: resolve the run's full payload-key capability so sealed (encp) hook payloads open

Main's sealed-box work (#3096) makes cross-deployment resumeHook() seal
hook payloads to the target run's published X25519 public key. The shared
start() path publishes that key regardless of engine, so QuickJS runs
receive sealed payloads too — but the QuickJS entrypoint resolved only the
bare symmetric key via importKey(), which cannot open encp envelopes. The
first sealed hook payload wedged the run right after hook_received, timing
out every hook/webhook e2e on Vercel prod (node:vm legs were fine — the
node engine resolves the full capability via memoizeEncryptionKey).

Resolve deriveRunPayloadKeys() in the entrypoint instead and widen the
runtime's key types from CryptoKey to DecryptionKey. Writes stay symmetric
(encrypt() with RunPayloadKeys takes the encr path). Regression test seals
a payload exactly as resumeHook does and round-trips it through the VM.

* Address review: crypto/process parity, loud Intl guards, lazy engine import, VM-leak guard, telemetry namespace, eval-string escaping

- Deterministic crypto.getRandomValues/randomUUID in the VM bootstrap,
  drawing from the seeded Math.random (identical sequences to the node
  engine's vm/index.ts implementations); all crypto.subtle methods throw
  with step-function guidance. process.env exposed as a frozen copy,
  matching node.
- Intl: throwing constructors (no ICU in QuickJS), and toLocale*-family
  methods (incl. localeCompare) throw when given an explicit locale so
  cross-engine divergence is loud instead of silently writing different
  values into the event log. No-argument forms keep working.
- runtime.ts lazy-imports the QuickJS entrypoint at dispatch, keeping the
  ~1.3MB embedded WASM assets out of node-engine deployments.
- runQuickJSWorkflow wraps the per-run phase so an exceptional exit
  disposes the VM instead of leaking it in a reused compute instance;
  corrected the misleading fail-loud comment (run_failed, not retry);
  warn when the event drain loop exhausts its iteration bound.
- Telemetry attributes renamed quickjs.* → workflow.vm.* to stay in the
  file's workflow.* namespace.
- Eval-string correlation-id interpolation uses JSON.stringify instead of
  quote-only escaping.
- common-vm.test.ts pins the reducer/reviver superset invariant against
  common.ts so the duplicated sets can't silently drift.
- Docs enumerate the remaining global-surface differences (subtle.digest,
  Intl, WebAssembly, Atomics); quickjs-entrypoint documents the known
  precondition-guard gap.

* QuickJS engine: implement resilient resumeHook (hookInput materialization + resumeId dedup)

#1834 made resumeHook() fall back to enqueueing the run with a hookInput
payload when the direct hook_received write fails transiently, with the
runtime materializing the missing event on delivery. Only the node:vm
path implemented it — the QuickJS dispatch returned before the node
block, so the resilient payload was silently dropped and the new e2e
timed out on every quickjs leg.

- runtime.ts threads hookInput into runWorkflowWithQuickJS; the
  entrypoint materializes the missing hook_received after loading the
  event log (resumeId-keyed dedup, occurredAt from the resumeId ULID,
  local eventData substitution for lazy/ref responses, EntityConflict /
  HookNotFound handling) — mirroring the node block.
- processEvents drops duplicate hook_received rows sharing a resumeId
  (first-in-log wins), matching the node engine's EventsConsumer dedup;
  the seen-set lives in the VM heap so it is deterministic per replay.

Verified against the dev server with WORKFLOW_VM=quickjs: the resilient
resume e2e passes and the materialization is observable in the logs; all
27 hook e2e tests green.

* Rerun CI

* QuickJS engine: split VM-local class/step-function reducers off the hardened host codec

The hardened host-side serialization (#3257) made the shared
reducers/class.ts and reducers/step-function.ts depend on
serialization/hardened.ts, which imports node:util and captures host
intrinsics — unbundleable and meaningless inside the QuickJS guest,
where the codec already runs in the guest realm. Point the VM codec at
pre-hardening copies with identical wire format; the host/guest
boundary hardening for this engine arrives with the host-side serde
that retires the VM bundle.

* QuickJS engine: enqueue explicit wait continuations instead of same-message redelivery

Scheduling sleep wakeups by returning { timeoutSeconds } redelivers the
CURRENT queue message. When that message is a hook-resume delivery
(carrying hookInput), its redelivery re-runs the lazy-resume re-ensure
in the handler prologue; if the workflow disposed the hook during the
first delivery (dispose -> sleep), the re-ensure gets HookNotFound, the
prologue acks the message as 'nothing left to resume', and the wait
timer it carried is silently lost — the run wedges (caught by the
hookDisposeTestWorkflow e2e).

Enqueue fresh continuation messages instead, matching the node engine's
suspension handler: getWaitContinuationDispatch for pending waits
(gaining delay clamping/hop chaining and pending-wait dedup keys) and a
plain immediate message for elapsed-wait / attr_set / getConflict
requeues. A fresh message carries only runId, so its delivery always
reaches replay.

Also: read hook_received resumeId from the canonical top-level event
field (eventData.resumeId is the deprecated legacy fallback), and stop
passing hookInput into the entrypoint — the shared prologue in
runtime.ts materializes the event for both engines. Adds a VM replay
test for the hook -> dispose -> sleep shape.

* Sort imports in quickjs-entrypoint (biome organizeImports)

* Address review: dispatch inside run-level try/catch, queue namespace + run-origin trace carrier threading, configurable interrupt budget

- Move the QuickJS engine dispatch inside the replay loop's try so
  escaping engine failures (MaxEventsExceededError, WASM OOM,
  bundle-eval errors) reach the catch that classifies and records
  run_failed, instead of nacking the message and burning all 48 queue
  redeliveries into MAX_DELIVERIES_EXCEEDED. Transient world errors
  still rethrow for redelivery. Updated the two comments that describe
  the propagation.
- Thread the queue namespace from runtime.ts through
  runWorkflowWithQuickJS into every message publish (step dispatch,
  hook_conflict requeue, immediate requeue, wait continuation) —
  without it, publishes on a namespaced deployment land on
  __wkf_workflow_* while consumers listen on __<ns>_wkf_workflow_*.
- Thread the run-origin nextTraceCarrier accessor through instead of
  capturing the current invocation context, so linked-mode invocations
  form a star around workflow.start rather than chaining; the
  hook_conflict requeue now carries a traceCarrier and requestedAt.
- Replace the hardcoded 30s VM interrupt budget with the configurable
  replay budget (getReplayTimeoutMs, default 240s), matching the node
  engine.

* Sort imports in quickjs-runtime (biome organizeImports)
 2026-08-03 16:38:58 -07:00
Pranay Prakash 4a192c85c8 [v5 only] docs: restore start-in-workflow documentation (#1803) 
Co-authored-by: Peter Wielander <peter.wielander@vercel.com>
 2026-08-03 16:37:39 -07:00
Nathan Colosimo 89ede82faa feat(core): widen retained boundaries to plain data and standard built-ins (#3047) 
* gate retention on the hardened serializer's guest-code report instead of primitives-only args

Replaces the isPrimitiveStepArgument allowlist with the GuestCodeStats sink
that dehydrateStepArguments already exposes: a boundary retains unless
serializing its step inputs actually executed workflow code (getters, proxy
traps, custom serializers), plus a descriptor-walk probe for a replaced
Error.prepareStackTrace — the one execution path the sink cannot see,
because the serializer treats V8's engine stack getter as engine-provided.

Plain data and standard built-ins (Map, Set, Date, RegExp, Error, typed
arrays, URL, Headers) now stay on the fast path, including under prototype
patching and polyfills, since serialization reads them through captured
intrinsics.

* reword changeset and docs in plain language
 2026-08-03 15:49:02 -07:00
Nathan Colosimo 5d591d2886 perf(core): retain workflow VM across inline steps (primitives-gated) (#3046) 
* perf(core): retain workflow VM across inline steps

Combines the retained-session architecture from #2984 with the env kill
switch and loop-level single-VM test from #2966.

- executeWorkflow with discriminated request/result types and a
  WorkflowSession state machine (running/suspended/failed/replay/completed)
- EventsConsumer.append: only newly durable events feed the live VM
- WORKFLOW_RETAINED_VM=0 kill switch (default on)
- retained-vm-loop.test.ts: proves one VM per run and byte-identical
  output vs the from-scratch replay path

* refactor(core): simplify retained-session control flow

- executeWorkflow overloads: a fresh replay request can no longer return
  { type: 'replay' }, deleting the runtime invariant throw and
  runWorkflow's dead branch
- isSameSuspensionBoundary reduced to the steps-array comparison (all
  suspension counts are derived from steps in the constructor)
- runtime loop initializes workflowResult with a ternary

* fix(core): decline retention for VMs that ran host-timed async work

crypto.subtle.digest is the only sandbox API whose promise resolves on
host timing rather than from the event log, so a workflow racing it
against a step can advance while suspended and diverge from what replay
reconstructs. A sticky usedHostAsync bit on the VM context makes
canRetainWorkflowSession fall back to ordinary replay for such VMs;
a quiescent step-only VM remains a pure function of the consumed
event prefix and stays retainable.

* fix(core): track all host-timed async VM APIs for retention

Atomics.waitAsync (a wall-clock timer via SharedArrayBuffer) and the
async WebAssembly compilation entry points resolve on host timing just
like crypto.subtle.digest. Wrap every such intrinsic in createContext so
usedHostAsync covers the complete set; dynamic import() settles within a
microtask and cannot advance a suspended VM.

* feat(core): compute crypto.subtle.digest synchronously in the sandbox

node:crypto createHash produces byte-identical values to WebCrypto and
settles the digest promise on a deterministic microtask instead of host
threadpool timing. A digest can therefore never advance a suspended
workflow, so digest-using VMs stay retainable; only Atomics.waitAsync
and async WebAssembly compilation remain host-timed. createHash is
stable and undeprecated on Node 18-26 (DEP0179 only removed the direct
Hash constructor).

* fix(core): remove WeakRef and FinalizationRegistry from the sandbox

GC observation depends on host GC timing that neither replay nor a
retained VM can reconstruct from the event log. WeakMap/WeakSet stay
available (they do not expose GC state).

* fix(core): enforce the BufferSource contract in the sandbox digest

Reject non-BufferSource digest input with TypeError like WebCrypto does,
via the native ArrayBuffer.prototype.byteLength brand check (works
across vm realms). Previously a plain number was treated as a
Uint8Array length, turning a small input into a giant allocation.

* fix(core): demote retention when suspension serialization draws randomness

handleSuspension dehydrates step arguments with the live VM, and that
serialization can execute user code (getters, WORKFLOW_SERIALIZE hooks).
Randomness drawn there would desync the retained VM's future correlation
IDs from what a fresh replay regenerates. Count every draw from the
seeded stream at its single source in createContext and fall back to
ordinary replay if handleSuspension consumed any.

* refactor(core): make VM quiescence unconditional, cut tracking machinery

Delete Atomics.waitAsync and the async WebAssembly entry points from the
sandbox instead of tracking their use — with digest synchronous and GC
intrinsics removed, no sandbox API settles a promise on host timing, so
a suspended VM provably cannot advance. This deletes the trackHostAsync
wrapper, the usedHostAsync bit and session method, the runtime gate
clause, the session 'failed' state (unreachable), and the
background-progress test scenarios (impossible by construction).

* refactor(core): gate retention on passively cloneable step inputs

Replace the RNG draw-counter demotion with prevention: when a session is
a retention candidate, new step inputs take a passive descriptor walk
(never invoking getters; proxies, accessors, functions, custom classes,
and platform wrappers decline) and safe values are structuredClone'd
into the host realm before dehydration, so serialization never executes
workflow-owned code against a retained VM. Unsafe inputs serialize the
old way and the session falls back to ordinary replay.

* fix(core): harden the passive step-input walker

- require enumerable on array index descriptors: structuredClone drops
  non-enumerable indices that devalue persists
- read workflow globals and constructor prototypes via own-property
  descriptors only, so validation can never execute workflow-owned
  accessors on redefined globals

* fix(core): guard proxied constructors in the passive-input walker

constructorPrototype reads both realms' constructors via own-property
descriptors only and refuses proxies before any descriptor read, so a
proxied redefined global can never observe validation.

* fix(core): preserve retention gate after rebase

* fix(core): all-or-nothing clone batches; reject SAB views in digest

- A mixed step batch (one unsafe sibling input) now serializes every
  input through the ordinary VM path: a clone snapshotted before an
  unsafe sibling's serialization runs its getters could otherwise
  durably capture stale sibling state.
- crypto.subtle.digest rejects SharedArrayBuffer-backed views with
  TypeError, matching WebCrypto's BufferSource contract.

* fix(core): narrow the fast path to prototype-independent types

devalue serializes Map/Set through the realm's iterator protocol and
Date/RegExp/typed arrays through prototype getters, all of which
workflow code can mutate — so their serialization is not provably
passive and their bytes could differ between retained and cold modes.
The fast path now accepts only primitives, plain objects, and plain
arrays, which devalue traverses exclusively via own-property reads.
Slot-bearing exotics decline even with a swapped prototype.

The sandbox digest now reads view metadata (buffer/byteOffset/
byteLength) through captured intrinsic getters, so own properties
shadowing them cannot change which bytes are hashed or bypass the
SharedArrayBuffer rejection.

* fix(core): freeze serialization-consulted sandbox intrinsics

instanceof dispatch (Symbol.hasInstance via the constructor,
Function.prototype, and Object.prototype), the class reducer's
value.constructor walk, and devalue's Object/Array traversal all consult
intrinsics workflow code could redefine — legally and deterministically —
which would make the durable step input depend on WORKFLOW_RETAINED_VM
(spoofed values serialize as e.g. Maps on the cold path but as plain
clones on the retained path). Freeze Object/Array/Function (constructors
and prototypes), the VM collection constructors, and every
reducer-referenced global binding (absent ones pinned to undefined)
right before the workflow bundle evaluates, so the retained-input
equivalence holds by construction.

Host-realm constructor escapes (e.g. TextEncoder.constructor) remain
out of the determinism contract: code scheduling host timers was never
deterministic under ordinary replay either; documented on
canRetainWorkflowSession.

* fix(core): freeze every non-shared serialization constructor

Typed-array constructors (and their shared %TypedArray% parent), the
Date wrapper, and the session-local AbortController/AbortSignal/
Request/Response bindings were pinned but not frozen, so workflow code
could still add Symbol.hasInstance statics that diverge reducer dispatch
between the retained clone (host constructors) and ordinary VM
serialization. Freeze every binding value that is not the shared host
intrinsic; shared host objects are dispatched identically by both paths,
so mutations there cannot cause mode divergence.

* fix(core): build retained clones in a pristine realm

Replace structuredClone with an explicit deep copy into an SDK-private
realm: clones previously inherited host prototypes, which workflow code
can reach (e.g. via structuredClone's return values) and vandalize with
Symbol.toStringTag or constructor overrides, shifting devalue's
classification of the clone relative to the ordinary VM path. The
pristine realm is unreachable by any user code, and the explicit copy
serializes exactly what devalue traverses (own indices, own enumerable
string props). Arrays also now decline own constructor properties,
which the class reducer reads even when non-enumerable.

* fix(core): verify host dispatch pristineness before retained cloning

Host intrinsics are shared with the whole process and cannot be frozen,
but workflow code can reach them (structuredClone results, exposed host
classes) and install Symbol.hasInstance predicates that distinguish the
original from its clone — or WORKFLOW_SERIALIZE statics on host
Object/Array that the class reducer reads for host-prototype originals
(hydrated step results). prepareRetainedStepInput now verifies, via
own-descriptor reads only, that every host dispatch point is pristine
and declines retention before any clone exists — so a spoofed predicate
can never observe or capture a pristine-realm object.

* fix(core): reject symbol properties from retained step inputs

Reducers dispatch on symbol tags (e.g. the workflow abort-signal
markers) that are non-enumerable and dropped by the pristine-realm
copy, so a tagged object would serialize as an abort descriptor on the
cold path but as plain data on the retained path.

* fix(core): retained inputs accept only own enumerable data properties

Hidden own keys of any kind — non-enumerable properties, accessors,
symbols — can be observed by serialization dispatch (reducer probes
like .signal, thenable checks, the class reducer) while the pristine
clone drops them. With no hidden own keys, every probe on an accepted
object resolves deterministically through validated data or pristine
prototypes.

* fix(core): freeze binding prototype chains for hasInstance lookup

Symbol.hasInstance dispatch walks the constructor's prototype chain, so
the frozen Date wrapper still exposed the unfrozen original VM Date it
delegates statics to. Freeze each non-shared binding's full chain
(stopping at host Function/Object prototypes) and verify host
Object.prototype carries no added hasInstance on the detection side.

* refactor(core): single-path retained serialization via pinned members (v2)

Serialize step inputs for retained boundaries through the one ordinary
pipeline (original value, workflow global) instead of cloning into a
pristine realm and serializing under the host global. With a single
serialization event shared by every mode, durable bytes structurally
cannot depend on WORKFLOW_RETAINED_VM; the only property retention needs
is that serialization executes no workflow code, established by:

- the passive walker (descriptor-only, unchanged in spirit), now also
  accepting Map/Set/Date/typed arrays/ArrayBuffer — the common built-in
  step arguments — via prototype-identity checks
- vm/serialization-pins.ts: the 10 prototype members serialization
  executes for those built-ins (measured empirically), captured at
  context creation and identity-verified at each retained boundary; the
  'touches only pinned members' test instruments every member and locks
  the list against serde drift
- host-realm instances (hydrated step results) accepted without member
  verification: host members run host code, which cannot touch retained
  VM state

Deletes the pristine clone realm, the host-dispatch pristineness checks,
and the batch clone bookkeeping.

* refactor(core): freeze built-in prototypes instead of pinning members (v3)

Review found the pin approach's structural hole: the class reducer READS
value.constructor through Map.prototype — a data property when pristine
(so member instrumentation never listed it), but executable the moment
workflow code redefines it as a getter. Pinning what serialization
executes misses what it reads. Freeze the accepted built-ins' prototypes
wholesale (Map/Set/Date + iterator prototypes, %TypedArray% + subclass
prototypes, ArrayBuffer): reads and executes are both immutable, and a
patch attempt now throws loudly at the patch site instead of silently
degrading. Deletes vm/serialization-pins.ts; the walker requires
Object.isFrozen on the realm prototype (also covering realms where the
freeze never ran).

Also restores the host-dispatch pristineness check the v2 cut lost:
workflow code can reach shared host constructors (exposed classes,
structuredClone results) and plant workflow-realm Symbol.hasInstance
hooks or WORKFLOW_SERIALIZE statics that reducers would execute during
retained serialization. Host-realm built-in instances decline for the
same reason; host-realm plain data (hydrated results) stays retainable.

* fix(core): harden the passivity checker's own execution surface

- Capture Map/Set forEach and the %TypedArray% buffer getter as module-
  load primordials: the checker previously invoked live host methods that
  workflow code can reach (structuredClone(new Map()).constructor) and
  replace with delegating workflow-realm closures.
- Typed arrays must have one of the realm's real frozen subclass
  prototypes by identity — 'frozen and chains to %TypedArray%' admitted
  manufactured frozen hostile prototypes with delegating buffer getters.

* fix(core): checker uses module-load primordials; verify inherited serializer statics

- The walker resolved Object.getOwnPropertyDescriptor, Reflect.ownKeys,
  Array.isArray, Number/String helpers, and Object.getPrototypeOf/isFrozen
  from live host globals workflow code can reach and replace; all are now
  module-load captures, so the checker can never execute a planted
  delegate.
- The class reducer reads cls[WORKFLOW_SERIALIZE]/cls.classId as
  inherited Gets, so isHostDispatchPristine now also verifies host
  Function.prototype and Object.prototype carry no serializer statics.

Generic replacement of shared host statics (Object.keys, Array.from, …)
via realm escape remains the documented host-reachability boundary,
tracked by the realm-local intrinsics follow-up.

* fix(core): stale-suspension generation token; cover BigInt toString

- Suspension signals capture ctx.suspensionGeneration when scheduled and
  no-op if the session resumed past that boundary. The harmful interleaving
  was already unreachable (queue items are deleted on consume, completion
  writes state synchronously, nextTick precedes timers) — the token turns
  those ordering facts into an explicit invariant.
- The BigInt reducer calls .toString() on primitives from host code, which
  resolves on host BigInt.prototype: its identity joins the host dispatch
  check, and the VM BigInt.prototype is frozen besides.

* feat(core): deterministic sandbox hardening

- crypto.subtle.digest computes synchronously via node:crypto:
  byte-identical values, promise settles on a deterministic microtask,
  full BufferSource validation (internal-slot view reads, SAB rejection)
- Atomics.waitAsync (a wall-clock timer), async WebAssembly compilation,
  WeakRef, and FinalizationRegistry are removed from the sandbox — wall
  clock and GC observation are unreplayable; sync WebAssembly
  constructors remain
- freezeSerializationIntrinsics pins the universal dispatch surfaces:
  Object.prototype/Array.prototype/Function.prototype are frozen (every
  missed property read and hasInstance lookup terminates there) and
  serialization-referenced global bindings are non-writable. Value-type
  prototypes and constructor statics stay patchable so polyfills
  (Temporal's Date.prototype.toTemporalInstant, core-js Set.prototype
  .union / Object.groupBy) keep working — the retained-input gate
  verifies the members serialization executes per boundary instead.

Groundwork for retained-VM replay (#2990).

* feat(core): retain the workflow VM across inline steps (primitive args)

Keeps the suspended workflow VM, its events consumer, and the paused
async stack alive across inline step executions within one invocation.
Each loop iteration appends only the newly written events instead of
replaying the entire event log in a fresh VM, so step-to-step overhead
stays flat as runs grow.

- WorkflowSession wraps executeWorkflow: suspended sessions expose
  resume(events) which appends to the retained EventsConsumer and lets
  the parked run() continuation settle; any divergence (unexpected
  suspension shape, consumer error) demotes to full replay permanently
- Retention is gated per boundary: only suspensions whose queued step
  inputs are all primitives (null/undefined/boolean/number/string) are
  retainable, because serializing primitives executes no workflow code;
  a follow-up widens this to plain data and standard built-ins
- Suspensions with hooks, waits, or attributes always fall back
- A suspension generation token invalidates stale timer callbacks from
  an abandoned suspension so they cannot advance a resumed VM
- WORKFLOW_RETAINED_VM=0 kill switch; telemetry records
  workflow.execution.mode = replay | retained

Part 2 of the retained-VM stack (#2990); requires the determinism
hardening in part 1.

* chore: retrigger vercel deployments

* Drop serialization intrinsic freezing from the sandbox

The retained-VM passivity design moved from pinning/verifying the
sandbox surfaces serialization dispatches on to injecting hardened
operations into devalue itself (with taint-based de-opt), so freezing
Object/Array/Function prototypes and pinning global bindings is no
longer needed. Keep only the determinism hardening (sync digest,
removal of wall-clock/GC-observing APIs).

* Document and lock in why async crypto.subtle methods cannot break quiescence

The remaining async subtle methods reject immediately through the crypto
proxy (brand check — the receiver is not a real SubtleCrypto), so they can
never mint a host-timing promise. Narrow the quiescence comment to what the
code actually enforces and add a test so the unreachability is not silently
"fixed" later.

* simplify sandbox hardening: lean digest input conversion, async digest, explicit subtle throwers

* simplify retention: single decision site in suspension catch, steps-only allow-list gate, drop prepareForRetention param

* mark sandbox API removals as a major change

* simplify retention further: one staleness mechanism (generation bump on suspend), whole predicate in canRetainWorkflowSession, lazy hook/wait scan, prewarm on resume path

* simplify session API and tests: replace executeWorkflow overloads with replayWorkflow/resumeWorkflow, drop low-value events-consumer tests, compact session and retained-loop tests

* add parallel-batch retention test (sibling signal absorption) and document the unguarded-signaler invariant

* simplify workflow.ts types: 5 named types (WorkflowResult/WorkflowResumeResult), async resume(), rename runtime local to retainedSession

* add retention-interleaving e2e (retained/demoted/wait/hook boundaries), drop session telemetry test

* discard the retained session on every in-process 412 restart

Review finding (both panel reviewers): restartReplayInProcess — added on
main by #3145 while this branch was in flight — reset the cached log but
not the parked VM session. Any stale-snapshot continue then resumed a
session belonging to the discarded log: after a run_completed 412 the
completed session's resume() throws and the run is durably failed despite
having completed; after a suspension-create 412 the session is resumed
without ever passing the retention decision, bypassing both the
WORKFLOW_RETAINED_VM kill switch and the step-input gate. A restart now
always falls back to a fresh replay. Regression test injects a 412 on
run_completed and proves fresh-replay completion (red without the fix).

* review round 2: set suspensionGeneration in typed test harness contexts; correct the open-hook/wait scan comment (this suspension's writes are not merged into the cached log — non-step suspensions never reach the scan)

* simplify pass: reuse once() from @workflow/utils for the open-hook/wait memo; drop optional-chaining that contradicted the surrounding guards

---------

Co-authored-by: Peter Wielander <mittgfu@gmail.com>
 2026-08-03 15:49:02 -07:00
github-actions[bot] bf4a591f12 Version Packages (beta) (#3256) 
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
workflow@5.0.0-beta.39 		2026-08-03 13:36:21 -07:00
Mitul Shah d06b55e641 Rename new-trace-viewer to trace-viewer (#3298) 
* Rename new-trace-viewer to trace-viewer.

Move the directory, rename NewTraceViewer to TraceViewer across web-shared and web, and update the build script and README.

Signed-off-by: mitul-s <mitulxshah@gmail.com>
Co-authored-by: Cursor <cursoragent@cursor.com>

* Fix TraceViewer import ordering

Signed-off-by: Cursor Agent <cursoragent@cursor.com>

---------

Signed-off-by: mitul-s <mitulxshah@gmail.com>
Signed-off-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
 2026-08-03 16:34:48 -04:00
Mitul Shah 951695ba2a Remove the legacy trace viewer in favor of NewTraceViewer. (#3296) 
Drop RunTraceView and WorkflowTraceViewer, move shared Span/Trace types into lib/trace-types, and keep timing helpers under the new viewer.

Signed-off-by: mitul-s <mitulxshah@gmail.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
 2026-08-03 16:34:48 -04:00
Greg Schofield 5d986d021a Swap python team for individual members (#3303) 
* Swap python team for individual members

Signed-off-by: gscho <greg.c.schofield@gmail.com>

* Update .github/CODEOWNERS

Co-authored-by: Peter Wielander <mittgfu@gmail.com>
Signed-off-by: Greg Schofield <greg.c.schofield@gmail.com>

* Update .github/CODEOWNERS

Co-authored-by: Peter Wielander <mittgfu@gmail.com>
Signed-off-by: Greg Schofield <greg.c.schofield@gmail.com>

---------

Signed-off-by: gscho <greg.c.schofield@gmail.com>
Signed-off-by: Greg Schofield <greg.c.schofield@gmail.com>
Co-authored-by: Peter Wielander <mittgfu@gmail.com>
 2026-08-03 20:33:30 +00:00
Karthik Kalyan 27d0ce7904 Route preview benchmarks through the e2e server (#3274) 
* Expose Workflow web server override

Signed-off-by: Karthik Kalyanaraman <karthik.kalyanaraman@vercel.com>

* Use neutral workflow server test URL

Signed-off-by: Karthik Kalyanaraman <karthik.kalyanaraman@vercel.com>

* Route preview benchmarks through e2e server

Signed-off-by: Karthik Kalyanaraman <karthik.kalyanaraman@vercel.com>

* Use an empty changeset

Signed-off-by: Karthik Kalyanaraman <karthik.kalyanaraman@vercel.com>

---------

Signed-off-by: Karthik Kalyanaraman <karthik.kalyanaraman@vercel.com>
 2026-08-03 12:54:27 -07:00
Nathan Rajlich aa78a7e63c world-local: converge redelivered hook_received re-ensure on its committed resumeId claim (#3297) 
A redelivered re-ensure of an already-committed resume (same runId +
resumeId + digest) was rejected with HookNotFoundError when the hook had
since been disposed by the workflow (dispose -> sleep releases the
token while the run continues). The queue consumer treats HookNotFound
as 'nothing left to resume' and acks the delivery — silently dropping
whatever continuation the redelivered message carried and wedging the
run.

Check the (runId, resumeId) claim BEFORE the disposal/existence
rejections: a committed claim whose pinned event is journaled proves
this exact resume was accepted while the hook was alive, so return that
event as success. Claims with a mismatched hookId or payload digest
still fall through to full validation and are rejected as before, as
are genuinely new resumes of a disposed hook.
 2026-08-03 12:43:20 -07:00
Mitul Shah dc4cf944ae Align web-shared typography with Geist tokens (#3294) 
* fix(web-shared): align typography with system tokens

Signed-off-by: Cursor Agent <cursoragent@cursor.com>

* test(web-shared): enforce typography tokens

Signed-off-by: Cursor Agent <cursoragent@cursor.com>

* refactor(web-shared): simplify typography audit

Signed-off-by: Cursor Agent <cursoragent@cursor.com>

* test(web-shared): cover typography guard failures

Signed-off-by: Cursor Agent <cursoragent@cursor.com>

* test(web-shared): narrow typography guard

Signed-off-by: Cursor Agent <cursoragent@cursor.com>

* test(web-shared): remove typography guard

Signed-off-by: Cursor Agent <cursoragent@cursor.com>

---------

Signed-off-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
 2026-08-03 11:58:17 -07:00
Nathan Colosimo 679dfa9c15 simplify hardened serialization: assert intrinsic captures, drop optional-capture states, close the bound-getter reporting hole (#3288) 
- Every captured intrinsic exists on all supported engines (Node 18+), so
  the optional-capture layer (intrinsicGetter-returns-undefined, canReadUrl/
  canReadUrlSearchParams/canReadHeaders, per-use fallbacks) is replaced by
  captures that throw at import if absent.
- URLSearchParams.prototype.size (the one genuinely missing member on Node
  18) is not needed: emptiness falls out of the captured toString() result,
  which the reducer already computes. Node 18 now serializes URLSearchParams
  natively instead of falling back to devalue's default handling.
- The call/get tables and their re-export aliases flatten into direct typed
  exports; readProxyAware and the viewInfo getter-indirection unroll into
  two-branch functions.
- isEngineAccessor: drop the WeakMap memo and try/catch (descriptor getters
  are always callable); exclude bound functions, which stringify as native
  code but run their target — previously workflow code could launder a
  side-effectful getter past the report with fn.bind() (test added).
- 763 -> 625 lines, byte output unchanged (parity checked for DataView and
  typed-array subviews on top of the existing test suite).
 2026-08-03 11:26:36 -07:00
Alex Langenfeld a799025af9 Surface a Request ID in the run sidebar attribute panel (#3293) 
## Summary & Motivation

One warm compute instance serves many invocations, so the Compute Instance ID already in this panel can't distinguish steps that ran inline within a single flow-function invocation — sibling steps sharing a Request ID did. It's also the value Vercel Logs indexes by, and this panel's View Logs button is where a reader takes it next.

The key is `vercelId`, not `requestId`: that's the name world-vercel stores the SDK's request id under crossing the wire, and AnalyticsEvent's sibling `requestId` field is declared but never written. Nothing in this repository populates `vercelId` on the object the panel receives yet — only AnalyticsEvent carries it, and grouping steps by invocation needs a step-level aggregation in workflow-server first — so the row ships as forward-compatible plumbing in the slot next to Compute Instance ID.

The second commit is an independent fix: `sortByAttributeOrder` guarded `indexOf` with `|| 0`, but a miss returns -1, which is truthy, so any key absent from `attributeOrder` sorted ahead of every listed key. It can be dropped on its own.

## Test Plan

Unit tests added; the two ordering assertions fail against the unfixed comparator. The new row could not be verified by hand — no local code path populates it.
 2026-08-03 13:05:09 -05:00
Shalabh Chaturvedi ba2cddc861 [benchmarks] Log the run id and Datadog trace for each sequential-steps run (#3248) 
* [benchmarks] Link the run id and Datadog trace under the STSO histograms

The STSO distribution section added in #3213 shows the shape of the
sequential-steps run but not which run produced it, so investigating an
odd-looking bucket meant hunting for the run by deployment id and time
window.

Capture the identity alongside the samples (the mechanism prototyped on
the WIP variance branch, #3107): `/api/bench` returns the trace id of the
span @vercel/otel opened for the trigger request, the runner threads it
through the sequential iteration and records `sequentialRuns` in the
result file, and the renderer prints one line under the histograms with
the run id + Datadog trace link for this run and for the `main` run it is
diffed against.

Every part is optional — a deployment predating the route change yields a
bare run id, and a `main` baseline predating this yields only this run's
side — so the section degrades instead of breaking on mixed-vintage
artifacts.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Co-Authored-By: Shalabh Chaturvedi <7066873+shalabhc@users.noreply.github.com>

Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com>

* Log the run/trace links instead of rendering them into the comment

The run id and Datadog trace are debugging aids, not part of the
benchmark's reported result, so they belong in the job's own output
rather than in the PR comment body.

Logging them where the runs are produced also makes them available in
two cases the comment could never cover: a local `pnpm bench`, and a
job that fails before the comment step runs.

This drops the comment-rendering side entirely -- `renderSequentialRunLinks`,
the `baselineSequentialRuns` baseline plumbing in `annotateWithBaseline`,
and the `sequentialRuns` field on the result artifact, which existed only
to carry the data to the renderer.

The `main`-baseline side of the link goes away with it: which run produced
the baseline histogram is only knowable at comment-render time, where the
two artifacts are matched.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com>

* Say what the trigger trace actually contains under linked mode

The route comment claimed the trigger request's span "propagates into
the workflow's own spans". That only holds under
WORKFLOW_TRACE_MODE=continuous. Nothing in the workbench or
benchmarks.yml sets the mode, so the benchmark deployment runs the
default `linked` (packages/core/src/telemetry.ts), where each
workflow/step invocation is its own trace root and the trigger's trace
carries `workflow.start` plus span links out to those roots.

The logged link is still the right entry point -- one hop through the
links, which Datadog renders -- but the comment should describe that,
so nobody opening a trigger-only trace while debugging a histogram
concludes the run produced no spans.

Raised by @TooTallNate in review of #3248.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

Co-Authored-By: Shalabh Chaturvedi <7066873+shalabhc@users.noreply.github.com>

Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com>

* Log a Datadog span search alongside the trigger trace link

Under the default linked trace mode the trigger's trace holds only
`workflow.start` plus span links, so opening it lands one hop away from
the spans an STSO investigation needs. Log an APM search on
`@workflow.run.id:<runId>` next to it, which goes straight to the run's
execution spans.

Both links are logged rather than one replacing the other: the search
depends on `workflow.run.id` being an indexed span tag in the org, and
the permalink works regardless.

Suggested by @TooTallNate in review of #3248.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

Co-Authored-By: Shalabh Chaturvedi <7066873+shalabhc@users.noreply.github.com>

Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com>

---------

Co-authored-by: vercel[bot] <35613825+vercel[bot]@users.noreply.github.com>
 2026-08-03 08:46:53 -07:00