* feat(core): seal hook payloads to the target run's public key
This is the payoff for the sealed-box work: cross-deployment
`resumeHook()` no longer calls `getEncryptionKeyForRun`, which on Vercel
means a ~350ms `run-key` API round trip (Cosmos reads, a 50–75KB
`encrypted_env.json` fetch from S3, up to three KMS decrypts) to recover
32 bytes. When the target run publishes a public key, the resumer seals to
it using only the run entity it already fetched.
More than half of `resumeHook()` calls miss the same-deployment fast path,
so this is the dominant term in hook-resumption latency — and for Eve,
hook resumption is what an agent turn waits on.
It also reduces privilege. Fetching the symmetric key grants read access
to everything in the run; sealing grants only the ability to write one
payload to it. A resumer is now cryptographically unable to read the run
it resumes.
Sealing is gated on the presence of `encryptionPublicKey`, deliberately
*not* on the capability version table the way `encr` and `gzip` are. A run
only carries a public key if the runtime that created it could also open a
sealed payload, and runs are pinned to their creating deployment — so
presence is a stronger attestation than comparing versions, and it stays
correct when `@workflow/core` and `@workflow/world-vercel` versions drift
independently (each single-axis version gate wedges a run under one drift
direction; presence wedges under neither).
`resumeWebhook` keeps using the symmetric key it already had to fetch in
order to hydrate hook metadata — sealing there would add an ECDH without
saving a round trip.
**Read path.** Sealing is useless if the run cannot open the result, and
every reader previously resolved a bare symmetric `CryptoKey`, which by
construction cannot open `encp`. Key resolution now yields the full
capability (symmetric key + X25519 keypair) at every read site:
`memoizeEncryptionKey`, the `Run` class, `runs.ts`, and
`getHookByTokenWithKey`. Without this the first sealed hook payload would
have wedged its run with `RuntimeDecryptionError`. Holder types widen from
`CryptoKey` to `PayloadKey`, which is a pure widening.
Falls back to the symmetric path when the run has no public key (older
SDKs), when the stored value is malformed, or when encryption is off — all
covered by tests, along with an end-to-end assertion that a sealed payload
actually hydrates with the keys the owning deployment re-derives.
* review: correct stale docs on the key-resolution path
- `memoizeEncryptionKey`'s JSDoc still described importing an AES-256
`CryptoKey`. It now resolves a run's full capability (symmetric key plus
X25519 keypair), and the reason matters: a run reading its own event log
can meet sealed payloads another run wrote to it, and resolving only the
symmetric key would leave those unopenable. Documented explicitly so the
next reader does not "simplify" it back.
- A comment in `resumeHook` pointed at the `encr` capability gate as being
"above" when it is in the fallback branch below. Reworded.
- Added the `decodeRunPublicKey` boundary test that belongs with the strict
base64 work but needed this branch's code to exist.
* feat: decrypt sealed payloads in the dashboard and CLI
Without this, any payload another run sealed to this one renders as a lock
icon with no way to open it — a visible regression for anyone debugging a
run that received a cross-deployment hook resumption. The user is entitled
to read the data and has already supplied the key; only the plumbing was
missing.
`hydrateDataWithKey` now delegates to the envelope layer, which dispatches
on the format prefix, instead of unconditionally running AES-GCM. All four
o11y key-resolution sites (web-shared hydration, the web stream reader,
and both CLI `--decrypt` paths) resolve the full capability rather than
just the symmetric key. Each already had the raw 32 bytes in hand, so this
costs one extra derivation and no additional requests.
A caller that supplies only a symmetric key still gets the ciphertext
placeholder for sealed payloads rather than a decryption error, since that
key never could have opened them.
**Browser bundling.** The obvious import for the new helper is
`@workflow/core/serialization`, but that module graph reaches `node:util`
and `node:async_hooks` and cannot be bundled for the browser — which is
what `@workflow/core/serialization-format` exists to avoid. The key
helpers are re-exported from that browser-safe entrypoint instead, and the
two browser consumers import from there; the CLI keeps the direct import
since it runs on Node. Verified by walking the built import graph: the
entrypoint reaches 6 modules and zero Node built-ins.
Unrelated: `pnpm --filter @workflow/web build` currently fails on `main`
too (`reducers/common.js` importing `node:util`). Turbo caching had been
hiding it; touching core caused a cache miss that surfaced it. Not
addressed here.
* review: narrow the o11y decrypt key type and dedupe an import
- `hydrateDataWithKey` accepted `PayloadKey`, which includes `SealTarget`.
A seal target holds only a public key, so it can open neither scheme —
passing one compiled fine and then always failed at runtime. Added a
`DecryptionKey` alias (`CryptoKey | RunPayloadKeys`) and narrowed the
signature, so that misuse is now a compile error. A `@ts-expect-error`
test pins the guarantee.
- `hydrateResourceIOAsync` dynamically imported
`@workflow/core/serialization-format` twice. Destructure both bindings
from the single existing import instead.
* review: record @workflow/web in the changeset
This PR changes the dashboard's stream reader
(`packages/web/app/lib/hooks/use-stream-reader.ts`) so it dispatches on the
envelope format and can read sealed (`encp`) frames, but the changeset listed
only core, web-shared and cli.
`@workflow/web` is published, so without an entry the change would still ship —
just as an incidental dependency bump, with nothing in that package's release
notes explaining that sealed-stream decryption landed.
* feat: publish each run's X25519 public key on the run entity
A cross-run writer needs the recipient run's public key to seal a payload
to it. Derive that key at `start()` and stamp it on the run, so a hook
resumption or a forwarded-stream writer can find it on a run fetch it was
already making instead of spending ~350ms on `run-key`.
The key is derived from the per-run key material `getEncryptionKeyForRun()`
already returns, so nothing about key acquisition changes. It is not
secret: the matching private scalar is never stored anywhere, only
re-derived on demand from the deployment's own env seed. Storing it beside
run metadata therefore does not weaken the run's confidentiality.
**Presence is the writer-side gate for sealed envelopes.** A run only
carries a public key if the runtime that created it could also open one —
which holds by construction, since derivation and `encp` dispatch both
live in `@workflow/core`, so any core that can stamp can also open. Runs
are pinned to their creating deployment, so the capability this attests to
is still accurate at resume time. Writers seal iff the field is set and
otherwise fall back to the symmetric path, which makes version skew
degrade gracefully instead of wedging a run.
The field rides on `run_created`, and is mirrored onto the queued
`runInput` so the resilient-start path (server recreates the run from the
queue message when the `run_created` write failed) doesn't silently
produce a run that can't receive sealed writes.
world-vercel's compile-time wire-contract guard caught the new field
before it could be silently dropped on the v4 path, exactly as designed —
routed into the frame meta block as plaintext metadata.
Also adds browser- and VM-safe base64 helpers to `sealed-box.ts`, since
neither `Buffer` nor `btoa` can be assumed in every context that module
runs in. `base64ToBytes` returns undefined on malformed input rather than
throwing, so a corrupt stored key degrades to "no usable public key" and
falls back to the symmetric path instead of crashing a resumption. Both
are cross-validated against `Buffer` in tests.
* review: fix public-key loss on resilient start and lifecycle updates
Two real bugs found in review, both in the local worlds. Neither surfaces
as an error — a run just silently stops accepting sealed cross-run writes
and falls back to the slow symmetric path forever.
**Resilient start dropped the key.** When a `run_started` arrives for a
run that was never created, world-local and world-postgres rebuild the run
from the queued message. Neither copied `encryptionPublicKey` onto the run
row or the synthetic `run_created` event they write. That is precisely the
scenario this field exists to survive. (The equivalent server-side path was
already handled.)
**world-local also wiped the key on every lifecycle transition.** Its
run_started / run_completed / run_failed / run_cancelled handlers rewrite
the whole run document field-by-field, so any field not explicitly listed
is dropped — meaning the key was lost on the *first* `run_started`, not
just on the resilient path. All four rebuild sites now carry it.
world-postgres is safe here by construction because it issues
column-scoped SQL UPDATEs rather than rewriting the row.
**base64 decoding is now strict.** The decoder accepted shapes that
cannot describe a whole number of bytes (`length % 4 === 1`) and ignored
anything after a mid-string `=`, returning a short array instead of
`undefined`. That is worse than throwing: a corrupt stored key looked
*present*, so callers sealed to garbage rather than taking the symmetric
fallback. Now rejects out-of-alphabet characters, bad lengths, misplaced
padding, and non-zero trailing bits — with a round-trip test over every
length 0–48 to make sure the strictness does not overshoot.
* fix: send encryptionPublicKey in the v4 POST frame meta
`splitEventDataForV4` lifted the run's public key into the frame meta and
`events.ts` spread that meta into `CreateEventV4Input`, but
`buildPostFrameMeta` — which copies meta onto the wire field by field — never
forwarded `encryptionPublicKey`, and the field was missing from
`CreateEventV4Input` entirely. Because the meta is applied with a spread,
TypeScript's excess-property check doesn't fire, so the key was computed, put
in the meta, and then silently dropped before the request was sent.
The server therefore never received the key, never stored it on the run
entity, and every cross-run writer fell back to the symmetric envelope. Every
symptom pointed away from the SDK: a deliberately oversized key was accepted
rather than rejected (the field never arrived), the key was absent from the run
row, and `resumeHook()` always chose `encr`.
Add the field to `CreateEventV4Input`, forward it in `buildPostFrameMeta`, and
cover it for both `run_created` and resilient-start `run_started`. Also add a
generic guard asserting that every field the splitter puts in the meta reaches
the wire, so the next omission in this hand-maintained mapping fails a test
instead of silently degrading encryption.
* feat(core): route sealed envelopes through the serialization layer
Adds the plumbing that lets a cross-run writer emit `encp` payloads. The
serialization layer previously had no way to express "seal to this public
key": every consumer expected a symmetric `CryptoKey`, and the encrypt
primitive was unconditionally AES-GCM.
That gap was also a live footgun. A 32-byte X25519 public key is a
structurally valid AES-256 key, so `importKey(pubkey, 'AES-GCM')`
succeeds and silently produces ciphertext nobody can ever open — no
compile error, no runtime error, just unreadable data. Making public keys
reachable only through `sealTo()` turns that mistake from "avoided by
convention" into "unrepresentable".
`PayloadKey` is a widening rather than a replacement, so no existing call
site changes:
| Variant | Writes | Reads | Held by |
| ---------------- | ------ | ------------ | ----------------------- |
| `CryptoKey` | encr | encr | same-run (legacy shape) |
| `RunPayloadKeys` | encr | encr, encp | the owning run, o11y |
| `SealTarget` | encp | — | cross-run writers |
A run's own payloads deliberately stay symmetric even when the holder
could seal: sealing costs a fresh ECDH and 32 bytes per envelope and buys
nothing when the writer already holds the decryption key.
The variants are branded with `Symbol.for` (not `Symbol()`) because these
values cross the host ↔ workflow VM realm boundary, where only the global
symbol registry is shared.
Both stream directions handle sealed frames, keeping the length header in
the clear so frame boundaries stay findable without a key. Frames keep
per-frame random nonces rather than a counter — a reconnect or a durable
replay restarts the writer, and a counter would repeat `(key, nonce)` and
break AES-GCM catastrophically. Regression tests assert that 100 frames
of identical plaintext produce 100 distinct ciphertexts, and that ten
writer incarnations never share a content key.
Capability shortfalls fail loudly and specifically: opening a sealed
payload with a symmetric key (or a write-only seal target, or nothing)
reports "no run keypair is available" rather than surfacing as a
mysterious auth-tag failure further down.
* review: amortize the sealed-box KEM across stream frames
Sealing each frame independently meant a fresh X25519 keygen + ECDH +
HKDF per chunk — O(frames) crypto for a long stream. As review pointed
out, amortizing the KEM is safe here so long as nonces stay random, which
they already are; the hazard I had been guarding against was counter
nonces specifically, not a long-lived content key.
`createSealSession` performs one encapsulation per writer instance and
reuses the content key, with a fresh random nonce per frame.
`createOpenSession` mirrors it on the read side, caching decapsulation by
ephemeral public key — since every frame from one writer carries the same
key, that is an ECDH per writer rather than per frame.
Both safety properties are preserved and now asserted directly:
- nonces stay random, so sharing a content key cannot repeat
`(key, nonce)` (100 identical-plaintext frames -> 100 distinct frames)
- a session is scoped to one stream instance, so a reconnect or durable
replay never inherits a previous content key (10 writer incarnations ->
10 distinct ephemeral keys)
The envelope layout is byte-identical to one-shot `seal`, so readers
cannot tell which path produced a frame and need no matching session.
Two edge cases the read-side cache introduces, both covered: frames from
two writers interleaved on one stream (eviction on every frame, so
correctness cannot depend on hit rate), and a failed decapsulation
clearing rather than poisoning the entry.
* docs: redirect retired migration-guides URLs to comparisons
The Migration Guides section was replaced by Comparisons in #2676
without redirects, 404ing the previously indexed
/docs/migration-guides/* URLs. Add permanent redirects mapping each
migrating-from-* page to its workflow-sdk-vs-* comparison, plus a
temporary catch-all onto the comparisons index.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* docs: redirect migration guide markdown URLs
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: Karthik Kalyanaraman <karthik.kalyanaraman@vercel.com>
* feat(core): add `encp` sealed-box encryption primitive
Cross-run writes (hook resumptions targeting another run, forwarded
writable stream frames) currently require the writer to hold the
recipient run's symmetric key, which also grants decrypt capability and
costs a ~350ms `run-key` API round trip across a deployment boundary.
Add the crypto foundation for sealing those writes to a public key
instead. Both keys descend from the per-run key material `K` that
`World.getEncryptionKeyForRun()` already returns, so key acquisition, the
World interface, and the Vercel API are all untouched:
K
├── AES-256 key = K used directly → 'encr' (unchanged)
└── X25519 scalar = HKDF(K, label) → 'encp'
└── public key (published, not secret)
`sealed-box.ts` implements an ECIES-style construction over the same
primitives as HPKE base mode (DHKEM(X25519, HKDF-SHA256), AES-256-GCM),
binding both public keys into the KDF `info` as HPKE's `kem_context` does
to prevent key-substitution attacks. The deviation from strict RFC 9180
framing is documented, and the HKDF labels are versioned so a conformant
profile can be added later without touching existing payloads.
Nothing produces `encp` payloads yet — this is the primitive only. The
o11y layer is hardened defensively so sealed payloads render as
ciphertext rather than throwing `Unsupported serialization format`, and
`hydrateDataWithKey` skips the AES path for them since opening a sealed
payload needs the private scalar rather than the symmetric key.
- optional AAD on the AES helpers, used to bind `projectId|runId`
- `encapsulate`/`decapsulate` split so stream writers can amortize the
KEM across frames; documented that they must keep random per-frame
nonces and re-encapsulate per connection attempt, since a long-lived
content key plus counter nonces would repeat `(key, nonce)` after a
reconnect or a durable replay
- public key derivation is cross-validated against node:crypto's native
X25519 in tests, since it reads the public half out of a JWK export
* review: tighten sealed-box docs, key-length checks, and constants
Addresses review feedback on the sealed-box primitive:
- The module doc pointed at `getSerializeStream` as enforcing the
re-encapsulate-per-writer rule, but nothing in-tree uses `encapsulate`
yet, and the stream path added later seals per frame instead. Reworded
to state the two rules as the caller's contract, since this module
enforces neither.
- `derivePublicKeyFromScalar` now asserts the JWK-derived public key is
32 bytes. That decode is the one place this module trusts an external
encoding; a short value would otherwise fail much later inside key
agreement with a far less obvious message.
- `open()` used bare 12/16 for the nonce and tag sizes. Those now come
from exported `NONCE_LENGTH`/`TAG_BYTES` in the AES layer, so the wire
format check cannot drift from the implementation.
* feat(core): deterministic sandbox hardening
- crypto.subtle.digest computes synchronously via node:crypto:
byte-identical values, promise settles on a deterministic microtask,
full BufferSource validation (internal-slot view reads, SAB rejection)
- Atomics.waitAsync (a wall-clock timer), async WebAssembly compilation,
WeakRef, and FinalizationRegistry are removed from the sandbox — wall
clock and GC observation are unreplayable; sync WebAssembly
constructors remain
- freezeSerializationIntrinsics pins the universal dispatch surfaces:
Object.prototype/Array.prototype/Function.prototype are frozen (every
missed property read and hasInstance lookup terminates there) and
serialization-referenced global bindings are non-writable. Value-type
prototypes and constructor statics stay patchable so polyfills
(Temporal's Date.prototype.toTemporalInstant, core-js Set.prototype
.union / Object.groupBy) keep working — the retained-input gate
verifies the members serialization executes per boundary instead.
Groundwork for retained-VM replay (#2990).
* Drop serialization intrinsic freezing from the sandbox
The retained-VM passivity design moved from pinning/verifying the
sandbox surfaces serialization dispatches on to injecting hardened
operations into devalue itself (with taint-based de-opt), so freezing
Object/Array/Function prototypes and pinning global bindings is no
longer needed. Keep only the determinism hardening (sync digest,
removal of wall-clock/GC-observing APIs).
* Document and lock in why async crypto.subtle methods cannot break quiescence
The remaining async subtle methods reject immediately through the crypto
proxy (brand check — the receiver is not a real SubtleCrypto), so they can
never mint a host-timing promise. Narrow the quiescence comment to what the
code actually enforces and add a test so the unreachability is not silently
"fixed" later.
* simplify sandbox hardening: lean digest input conversion, async digest, explicit subtle throwers
* mark sandbox API removals as a major change
---------
Co-authored-by: Nathan Rajlich <n@n8.io>
* [core] Don't count racing invocations' duplicate step_started events toward the maxRetries ceiling
The retry-ceiling guard added in #3035 derives a step's attempt number from
the number of step_started events in the log. But invocations racing on the
same pending batch (stale replays, wake replays, step messages dispatched
off a lost create-claim) can each write a step_started for the same logical
attempt — worlds without an atomic start guard (world-local) let them all
through — so a healthy step's count could cross the ceiling and fail the
run with a false "exceeded max retries" (the world-testing
inline-batches-debug CI flake).
Scope the count to the starts each ceiling is actually about:
- inline owned-recovery: only starts stamped with THIS message's
ownerMessageId (each real (re)delivery of the owner re-stamps it;
racers stamp their own IDs or none)
- background steps: only bare/unstamped starts (that path never stamps
ownership, and throttle/too-early redeliveries still write no start)
Real timeout retries still bound: each recovery re-run writes another
owner-stamped (or bare) start, so genuine exhaustion still trips the guard.
* review: internalize helper, close mixed owned→bare ceiling gap, fix 'bare' terminology
- Move countStepStartedEvents to an internal module (src/runtime/
count-step-started-events.ts) instead of exporting it from the public
./runtime subpath; tests import it relatively.
- Replace the background ceiling's 'unowned' scope with 'totalAttempts'
(bare starts + largest single owner's starts), so a step that burns part
of its retry budget under inline owned recovery and then transitions to
queued/bare retries still trips the combined maxRetries ceiling, while
racers' one-off stamped duplicates still don't accumulate. Adds the
mixed-sequence regression test from review.
- Fix comments calling the owned-recovery start a 'bare step_started' —
it carries the ownerMessageId stamp; only the background path's start
is bare.
* perf(core): immediate leading-edge dispatch for idle streams (flush window default 0)
Production producer-rate data (24h of client flush spans): most agents
average 1.03-1.21 chunks per flush with 87-98% single-chunk flushes and
>70% of chunks arriving more than 10ms after the previous request had
already settled — a fixed 10ms leading window batches almost nothing
for them while adding ~20% to isolated-chunk publish latency (~50ms
median RTT). The one bursty producer (avg ~4-8 chunks/flush) gets its
batching from in-flight accumulation, which does not depend on the
window at all.
The leading chunk of an idle sink now dispatches immediately by
default (window 0): first chunk goes out at once, chunks arriving
during its request coalesce into the next group, and each settle
dispatches the accumulated group immediately — path-independent
batching with no fixed tax on slow producers. A positive
WORKFLOW_STREAM_FLUSH_INTERVAL_MS (or world.streamFlushIntervalMs,
applying from the second group) opts into a windowed leading edge for
slow-but-steady producers that prefer larger groups over first-chunk
latency. Early-ack, the durability drain barrier, wire caps, and
backpressure bounds are unchanged.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Update packages/world/src/interfaces.ts
Co-authored-by: Peter Wielander <mittgfu@gmail.com>
Signed-off-by: Karthik Kalyan <105607645+karthikscale3@users.noreply.github.com>
* review: env var overrides world streamFlushIntervalMs; world option governs the leading edge too
WORKFLOW_STREAM_FLUSH_INTERVAL_MS, when set, now takes precedence over
world.streamFlushIntervalMs; otherwise the world option applies from the
very first chunk (no more second-group lazy quirk). Deciding waits for
the world when needed, which adds no latency: sendGroup awaits the same
promise before any request can leave.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Signed-off-by: Karthik Kalyan <105607645+karthikscale3@users.noreply.github.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: Peter Wielander <mittgfu@gmail.com>
* perf(core): move stream write batching into WorkflowServerWritableStream (group commit)
Batching previously lived in flushablePipe's coalescing loop, so it only
engaged on paths that used flushablePipe (getWritable). A raw
ReadableStream crossing a workflow/step boundary is piped with native
pipeTo(), which does not pull chunk N+1 until write(chunk N) resolves —
and write() resolved only after the flush timer AND the server round
trip, so the buffer never held more than one chunk and every token
became its own server request.
The sink now group-commits:
- write() resolves when the chunk enters a bounded client buffer; the
bound counts buffered AND in-request chunks
(WORKFLOW_STREAM_MAX_INFLIGHT_CHUNKS, preserving its documented
meaning) plus a byte bound (WORKFLOW_STREAM_MAX_BUFFERED_BYTES, new,
default 8 MiB, documented in runtime-tuning). A full buffer applies
backpressure until a group lands durably.
- The flush interval is a real group-commit window; chunks arriving
while a request is in flight accumulate and form the next writeMulti
group. One request in flight at a time preserves chunk order.
- Per-request wire limits (1,000 chunks / 1 MiB) split groups exactly
as the coalescing pipe did; an oversized single chunk goes alone.
- Durability moved to an explicit barrier (STREAM_DRAIN_SYMBOL):
close() drains before closing; flushablePipe adopts the barrier so
lock-release completion (step completion) still means 'everything
written is durable'; abort() DRAINS the accepted prefix (never
closing) so a producer error after acked writes cannot lose data —
native pipeTo aborts the sink on source failure; and a failed pipe
drains before settling so a step failure is not persisted ahead of
the emitted prefix. A dispatch failure retains the group, poisons
the sink, and surfaces at the next write/close/drain.
flushablePipe is now a plain per-chunk pump responsible only for
lock-release completion and durability tracking; its coalescing
machinery and STREAM_WRITE_BATCH_SYMBOL are removed.
Covered: native-pipeTo batching (the regression), awaited per-chunk
loops coalescing into one writeMulti, in-flight accumulation, wire-cap
splits (count/byte/oversized), in-flight-inclusive backpressure for
both bounds, sequential fallback without writeMulti, source-error
prefix delivery through abort, failed-pipe drain-before-reject,
early-ack sticky errors, turbo run-ready barrier gating (incl. dwell
telemetry), drain-barrier adoption/rejection, and group-level flush
spans. 1,572 core unit tests pass; e2e tier requires a deployment and
was not run here.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(core): re-dispatch a chunk buffered in the request settle gap
Review (bot): a write landing between the dispatch loop's empty-buffer
exit and the reaction clearing the in-flight marker armed no timer
(scheduleGroupCommit saw the marker set) and was never dispatched on an
open stream — only a later write/close/drain would pick it up. The
settle reaction now re-dispatches when the buffer is non-empty, treating
the chunk as an in-request arrival; drain waiters settle with the new
chain. Regression test aims a write at the settle gap and asserts both
chunks flush without a close.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* docs(core): document abort-drain boundedness and terminal-run conflict handling
Review note: the abort-path drain is deliberately un-timeboxed (a bound
would drop acked chunks); its worst case is owned by the World
transport's finite timeout/retry budget, and a teardown-driven drain
into an already-terminal run rejects into the existing catch.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* test(core): poll instead of fixed sleeps for dispatch assertions
The native-pipeTo batching test flaked on a slow CI runner: a fixed
25ms wait raced the 10ms commit window plus scheduler jitter. All
'dispatch has happened' assertions now poll the expectation (bounded);
intentional negatives keep their fixed windows.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* test(e2e): poll the events readback in stepFunctionPassingWorkflow
The events listing prefers the analytics store, which ingests
asynchronously. Reading it immediately after run completion can miss
the freshest events (the page is non-empty, so the storage fallback
does not trigger), failing the step_completed assertion. Poll for up
to 20s so ingestion has time to land; the assertion itself is
unchanged.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* test(e2e): force storage-backed inspect listings via WORKFLOW_DISABLE_ANALYTICS_READS
The analytics store ingests asynchronously; e2e assertions read events
and steps immediately after run completion and can catch a page missing
the freshest rows (observed as stepFunctionPassingWorkflow's
step_completed readback returning empty, and the same race on steps
listings in other suites). Instead of polling each readback, disable
the analytics namespace for the e2e's CLI invocations so every inspect
listing is served read-your-writes from primary storage. Replaces the
earlier bounded poll with the deterministic mechanism.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Enforces the published per-run events limit, which was previously not enforced. The server supplies the limit on the run_started response (separate change); once a run's event log reaches it, the runtime throws MaxEventsExceededError at the top of the replay loop, and the existing terminal-error path records it as run_failed with a new MAX_EVENTS_EXCEEDED code — instead of letting a runaway workflow (e.g. an unbounded step loop) grow the event log without bound.
Adds a new client side WORKFLOW_MAX_EVENTS_OVERRIDE env var which can override the server side provided value (lower only).
* [core] Enforce maxRetries for steps that time out
A step that is hard-killed by the platform function timeout writes no
step_failed/step_retrying, so `step.error` stays null and the error-based
max-retries guards never fire. Each redelivery re-runs step_started
(incrementing the attempt), so a timing-out step retried without bound
instead of stopping at maxRetries.
Enforce the retry ceiling BEFORE running the body, via a new
`authoritativeAttempt` param on executeStep:
- Inline (combined handler): count the step_started events already in the
event log for the step (+1 for this attempt). The log is authoritative
because the optimistic-start path synthesizes step.attempt = 1.
- Background (queue-dispatched): the queue delivery count (metadata.attempt),
which increments on the visibility-timeout redelivery a timed-out step
produces.
When the attempt exceeds maxRetries + 1 the step is failed without starting
another attempt. Thrown-error exhaustion is unchanged — it still terminates
via the post-body guard one attempt earlier, with the thrown error as cause.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Apply suggestions from code review
Co-authored-by: Peter Wielander <mittgfu@gmail.com>
Signed-off-by: Peter Wielander <mittgfu@gmail.com>
* [core] Verify step_started count before failing a backgrounded step
`metadata.attempt` (queue delivery count) also advances for redeliveries
that never run the step body (ThrottleError, TooEarlyError, other pre-body
failures), so trusting it directly could fail a step as "exceeded max
retries" before the body ever ran.
Use the delivery count only as a fast gate: while it is at or under the
ceiling the step can't be exhausted, so proceed without touching the log.
Only once it crosses the ceiling, load the full event log and derive the
authoritative attempt from the recorded step_started count (which only real
attempts write) — excluding throttle/too-early redeliveries. The load also
primes the replay's cachedEvents/eventsCursor.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* [core] Skip the event-log scan for brand-new inline steps
Deriving an inline step's attempt number by scanning the cumulative event
log for step_started events ran for every inline execution, which is O(n²)
across a long sequential workflow.
A lazy inline step is brand-new by construction (it only enters the batch
with no step_created yet), so it has zero prior starts and is always attempt
1 — no scan needed. Reserve the scan for owned-recovery re-runs (this
message re-executing a step it crashed/timed out on), which are uncommon and
few per batch.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Signed-off-by: Peter Wielander <mittgfu@gmail.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* [world-vercel] Idempotent retry policy for stream close (5xx retriable)
Stream close is the one idempotent stream PUT: a duplicate close of a
completed stream early-returns on the server, and the close-barrier
protocol's durable `closing` fence is an if_not_exists stamp that a
re-entered close resumes. The barrier protocol relies on close retrying
5xx: transient reconciliation failures — and unsafe close shapes
awaiting in-flight backups — surface as retriable 503s with the stream
left durably closing, expecting the writer to close again. Under the
write dispatcher's no-5xx policy (correct for non-idempotent chunk
appends), that 503 rejected writer.close() outright and left the stream
fenced until run expiry.
Close now uses its own shared RetryAgent (429 + 5xx + transient
connection errors, Retry-After honored); chunk writes keep the narrowed
no-5xx policy unchanged. Contract pinned by tests.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* changeset for stream close retry
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* concise changeset
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* docs: replace migration guides with a Comparisons section
Add a Comparisons section (v4 + v5) with an index/snapshot across all frameworks and deep-dive pages for Temporal, Cloudflare Workflows, AWS Step Functions, AWS Bedrock AgentCore, Inngest, and trigger.dev. Remove the old migration-guides section, folding its concept-mapping and migration content into the relevant comparison pages, and repoint top-level nav in both versions.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* docs: refresh comparison pages with current facts (July 2026)
Re-verified each comparison against the vendor's current public docs and
updated what changed since the June 2026 snapshot:
- Temporal: Worker Versioning is now GA; Serverless Workers (AWS Lambda,
pre-release) scale to zero, so soften the blanket "no scale-to-zero";
drop the unsubstantiated "Uber" customer claim (Uber is Cadence's origin,
not a Temporal customer).
- Cloudflare Workflows: note the new per-step billing dimension (500K/mo
included, then $0.80/100K) landing no earlier than Aug 10, 2026; note the
50K concurrency ceiling was raised from 4,500 at GA.
- AWS Bedrock AgentCore: add newer GA modules (Harness, Policy, Evaluations);
correct compliance (SOC/PCI/ISO under internal assessment, audits pending;
FedRAMP not yet authorized; drop GovCloud claim); refresh languages
(@aws/agentcore CLI scaffolds TS or Python); "some modules preview" is stale.
- Inngest: Pro pricing $75 -> $99/mo; encryption middleware now TS + Python;
AgentKit/Realtime are Developer Preview and Connect is public beta; self-host
is community/best-effort (not "unsupported"); Free-tier run duration 30 days
vs 366 on Pro; soften funding to ~$30M+.
- AWS Step Functions & trigger.dev: facts re-confirmed; date stamp only.
Bumped every "as of June 2026" stamp to July 2026. v4 and v5 kept identical.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs: present comparisons in the present tense; v5-only; Pro usage-based pricing
Follow-up pass on the comparison pages:
- Remove all date references (past and future). Anything that lands on a
date is stated as already in effect: Cloudflare's per-step billing, Temporal
Serverless Workers and GA Worker Versioning, AgentCore's Harness/Policy/
Evaluations modules. Dropped "as of July 2026" stamps, founding/GA years,
funding round dates, and roadmap/"being added" phrasing.
- Workflow SDK: reference v5 only and treat it as GA (was "v4 GA / v5 beta").
- Pricing and limits: quote the Pro/paid tier only and usage-based rates only;
drop plan-included quotas and free-tier allowances (Step Functions 4K/mo free,
Cloudflare 500K steps/mo included, Inngest 50K free execs, Inngest/Free 30-day
run cap, Temporal $100/mo plan minimum).
v4 and v5 kept identical.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs: tighten comparison maturity/status wording
- Drop dateless "upcoming change" phrasing: AgentCore compliance now states
current facts only (no "self-assessed"/audit-pending implication); remove
Inngest's "SSPL → Apache after 3 yrs" license-conversion note.
- Don't label the Workflow SDK "GA" — non-beta is the default; also drop bare
"GA" where it only meant "not beta" (Temporal "7 SDKs", Inngest "TypeScript",
competitor maturity cells).
- Maturity cells no longer cite version numbers; they describe backing/track
record instead (e.g. "Built and maintained by Vercel", "Backed by AWS").
v4 and v5 kept identical.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs: make "features without a 1:1 equivalent" sections directional
Rename each heading to name the competitor that has the feature (e.g.
"Temporal features without a direct Workflow SDK equivalent") and add a
lead-in clarifying these are the competitor's capabilities the Workflow SDK
doesn't replicate one-to-one, with how to cover each on the Workflow SDK side.
v4 and v5 kept identical.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs: point world links at the /worlds routes
The comparison pages linked to /docs/deploying/world/* and
/docs/deploying/building-a-world, which no longer exist in the docs
trees (the Docs Links check rejects them on v5 pages, where /docs hrefs
are render-rewritten and skip the legacy redirects). Link the canonical
/worlds/* routes directly, in both body links and frontmatter refs.
v4 and v5 kept identical.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* docs: address toolbar review feedback on the comparison pages
- Drop the Maturity row from every at-a-glance table
- Add a "what the limits mean in practice" paragraph to each comparison,
spelling out what the competitor's caps mean for long-running AI
workloads, and link Vercel World limits to the pricing doc
- Security cells: lead with zero-config per-run E2E encryption and note
platform security is per-World, instead of the VM-sandbox framing
- Temporal: drop the throughput sentence and the still-in-preview
Serverless Workers mention from the performance cell
- Cloudflare: end the recommendation on "already all-in on Cloudflare"
- Convert the "features without a direct equivalent" bullet lists into
two-column tables so it's unambiguous which product owns each feature
- Fix the Inngest page's "no step cap" cell (Vercel World caps runs at
10K steps per the pricing doc)
v4 and v5 kept identical.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Co-authored-by: Peter Wielander <peter.wielander@vercel.com>
* test: regression coverage for hook.resume() from isolated route bundles (o2flow beta.26 incident)
Reproduces the o2flow v5 upgrade failure (workflow@5.0.0-beta.26, fixed by
#2752 in beta.28): a plain API route importing defineHook() from the root
`workflow` entry and calling .resume() failed with Turbopack's
"Cannot find module as expression is too dynamic" stub, because the world
registration was tree-shaken out of the route bundle and getWorldLazy()'s
dynamic-import fallback got stubbed.
The bug only manifests when a route bundle loads in isolation (a Vercel
lambda): local `next dev`/`next start` evaluates next.config.ts, whose
workflow/next import chain registers the world process-wide and masks it —
which is why no existing server-driven suite caught it.
- route-bundle-isolation.test.ts: production Turbopack build of the
nextjs-turbopack workbench, then loads ONLY the compiled route bundle in a
bare Node subprocess (cold-lambda simulation) and invokes its POST handler.
Fails with the exact incident error on regressed code; passes on main.
Wired into the build-error-messages CI job.
- e2e: plainModuleDoneHook round-trip through a plain API route on the two
Next workbenches (deployed matrix covers real lambda isolation).
- Workbench fixtures mirroring o2flow: a directive-less defineHook module
shared by a workflow (create) and a plain route (resume). The webpack
workbench gets a real route file because `next dev` (webpack) does not
serve directory-symlinked app routes.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Pranay Prakash <pranay.gp@gmail.com>
* test: authenticate plain hook resume request
* test: address review — marker-based harness output parsing, changeset summary
- route-bundle-isolation: prefix the harness result line with a unique
marker and locate it explicitly instead of JSON.parse()ing the last
stdout line, so stray logging from the route bundle or the world can't
break parsing; failures now include the full subprocess stdout.
- changeset: add a human-readable summary to the (release-less) changeset.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Pranay Prakash <pranay.gp@gmail.com>
---------
Signed-off-by: Pranay Prakash <pranay.gp@gmail.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: Karthik Kalyanaraman <karthik.kalyanaraman@vercel.com>
Co-authored-by: Karthik Kalyan <105607645+karthikscale3@users.noreply.github.com>
Nathan Rajlich