* feat: publish each run's X25519 public key on the run entity
A cross-run writer needs the recipient run's public key to seal a payload
to it. Derive that key at `start()` and stamp it on the run, so a hook
resumption or a forwarded-stream writer can find it on a run fetch it was
already making instead of spending ~350ms on `run-key`.
The key is derived from the per-run key material `getEncryptionKeyForRun()`
already returns, so nothing about key acquisition changes. It is not
secret: the matching private scalar is never stored anywhere, only
re-derived on demand from the deployment's own env seed. Storing it beside
run metadata therefore does not weaken the run's confidentiality.
**Presence is the writer-side gate for sealed envelopes.** A run only
carries a public key if the runtime that created it could also open one —
which holds by construction, since derivation and `encp` dispatch both
live in `@workflow/core`, so any core that can stamp can also open. Runs
are pinned to their creating deployment, so the capability this attests to
is still accurate at resume time. Writers seal iff the field is set and
otherwise fall back to the symmetric path, which makes version skew
degrade gracefully instead of wedging a run.
The field rides on `run_created`, and is mirrored onto the queued
`runInput` so the resilient-start path (server recreates the run from the
queue message when the `run_created` write failed) doesn't silently
produce a run that can't receive sealed writes.
world-vercel's compile-time wire-contract guard caught the new field
before it could be silently dropped on the v4 path, exactly as designed —
routed into the frame meta block as plaintext metadata.
Also adds browser- and VM-safe base64 helpers to `sealed-box.ts`, since
neither `Buffer` nor `btoa` can be assumed in every context that module
runs in. `base64ToBytes` returns undefined on malformed input rather than
throwing, so a corrupt stored key degrades to "no usable public key" and
falls back to the symmetric path instead of crashing a resumption. Both
are cross-validated against `Buffer` in tests.
* review: fix public-key loss on resilient start and lifecycle updates
Two real bugs found in review, both in the local worlds. Neither surfaces
as an error — a run just silently stops accepting sealed cross-run writes
and falls back to the slow symmetric path forever.
**Resilient start dropped the key.** When a `run_started` arrives for a
run that was never created, world-local and world-postgres rebuild the run
from the queued message. Neither copied `encryptionPublicKey` onto the run
row or the synthetic `run_created` event they write. That is precisely the
scenario this field exists to survive. (The equivalent server-side path was
already handled.)
**world-local also wiped the key on every lifecycle transition.** Its
run_started / run_completed / run_failed / run_cancelled handlers rewrite
the whole run document field-by-field, so any field not explicitly listed
is dropped — meaning the key was lost on the *first* `run_started`, not
just on the resilient path. All four rebuild sites now carry it.
world-postgres is safe here by construction because it issues
column-scoped SQL UPDATEs rather than rewriting the row.
**base64 decoding is now strict.** The decoder accepted shapes that
cannot describe a whole number of bytes (`length % 4 === 1`) and ignored
anything after a mid-string `=`, returning a short array instead of
`undefined`. That is worse than throwing: a corrupt stored key looked
*present*, so callers sealed to garbage rather than taking the symmetric
fallback. Now rejects out-of-alphabet characters, bad lengths, misplaced
padding, and non-zero trailing bits — with a round-trip test over every
length 0–48 to make sure the strictness does not overshoot.
* fix: send encryptionPublicKey in the v4 POST frame meta
`splitEventDataForV4` lifted the run's public key into the frame meta and
`events.ts` spread that meta into `CreateEventV4Input`, but
`buildPostFrameMeta` — which copies meta onto the wire field by field — never
forwarded `encryptionPublicKey`, and the field was missing from
`CreateEventV4Input` entirely. Because the meta is applied with a spread,
TypeScript's excess-property check doesn't fire, so the key was computed, put
in the meta, and then silently dropped before the request was sent.
The server therefore never received the key, never stored it on the run
entity, and every cross-run writer fell back to the symmetric envelope. Every
symptom pointed away from the SDK: a deliberately oversized key was accepted
rather than rejected (the field never arrived), the key was absent from the run
row, and `resumeHook()` always chose `encr`.
Add the field to `CreateEventV4Input`, forward it in `buildPostFrameMeta`, and
cover it for both `run_created` and resilient-start `run_started`. Also add a
generic guard asserting that every field the splitter puts in the meta reaches
the wire, so the next omission in this hand-maintained mapping fails a test
instead of silently degrading encryption.
* feat(core): route sealed envelopes through the serialization layer
Adds the plumbing that lets a cross-run writer emit `encp` payloads. The
serialization layer previously had no way to express "seal to this public
key": every consumer expected a symmetric `CryptoKey`, and the encrypt
primitive was unconditionally AES-GCM.
That gap was also a live footgun. A 32-byte X25519 public key is a
structurally valid AES-256 key, so `importKey(pubkey, 'AES-GCM')`
succeeds and silently produces ciphertext nobody can ever open — no
compile error, no runtime error, just unreadable data. Making public keys
reachable only through `sealTo()` turns that mistake from "avoided by
convention" into "unrepresentable".
`PayloadKey` is a widening rather than a replacement, so no existing call
site changes:
| Variant | Writes | Reads | Held by |
| ---------------- | ------ | ------------ | ----------------------- |
| `CryptoKey` | encr | encr | same-run (legacy shape) |
| `RunPayloadKeys` | encr | encr, encp | the owning run, o11y |
| `SealTarget` | encp | — | cross-run writers |
A run's own payloads deliberately stay symmetric even when the holder
could seal: sealing costs a fresh ECDH and 32 bytes per envelope and buys
nothing when the writer already holds the decryption key.
The variants are branded with `Symbol.for` (not `Symbol()`) because these
values cross the host ↔ workflow VM realm boundary, where only the global
symbol registry is shared.
Both stream directions handle sealed frames, keeping the length header in
the clear so frame boundaries stay findable without a key. Frames keep
per-frame random nonces rather than a counter — a reconnect or a durable
replay restarts the writer, and a counter would repeat `(key, nonce)` and
break AES-GCM catastrophically. Regression tests assert that 100 frames
of identical plaintext produce 100 distinct ciphertexts, and that ten
writer incarnations never share a content key.
Capability shortfalls fail loudly and specifically: opening a sealed
payload with a symmetric key (or a write-only seal target, or nothing)
reports "no run keypair is available" rather than surfacing as a
mysterious auth-tag failure further down.
* review: amortize the sealed-box KEM across stream frames
Sealing each frame independently meant a fresh X25519 keygen + ECDH +
HKDF per chunk — O(frames) crypto for a long stream. As review pointed
out, amortizing the KEM is safe here so long as nonces stay random, which
they already are; the hazard I had been guarding against was counter
nonces specifically, not a long-lived content key.
`createSealSession` performs one encapsulation per writer instance and
reuses the content key, with a fresh random nonce per frame.
`createOpenSession` mirrors it on the read side, caching decapsulation by
ephemeral public key — since every frame from one writer carries the same
key, that is an ECDH per writer rather than per frame.
Both safety properties are preserved and now asserted directly:
- nonces stay random, so sharing a content key cannot repeat
`(key, nonce)` (100 identical-plaintext frames -> 100 distinct frames)
- a session is scoped to one stream instance, so a reconnect or durable
replay never inherits a previous content key (10 writer incarnations ->
10 distinct ephemeral keys)
The envelope layout is byte-identical to one-shot `seal`, so readers
cannot tell which path produced a frame and need no matching session.
Two edge cases the read-side cache introduces, both covered: frames from
two writers interleaved on one stream (eviction on every frame, so
correctness cannot depend on hit rate), and a failed decapsulation
clearing rather than poisoning the entry.
* feat(core): add `encp` sealed-box encryption primitive
Cross-run writes (hook resumptions targeting another run, forwarded
writable stream frames) currently require the writer to hold the
recipient run's symmetric key, which also grants decrypt capability and
costs a ~350ms `run-key` API round trip across a deployment boundary.
Add the crypto foundation for sealing those writes to a public key
instead. Both keys descend from the per-run key material `K` that
`World.getEncryptionKeyForRun()` already returns, so key acquisition, the
World interface, and the Vercel API are all untouched:
K
├── AES-256 key = K used directly → 'encr' (unchanged)
└── X25519 scalar = HKDF(K, label) → 'encp'
└── public key (published, not secret)
`sealed-box.ts` implements an ECIES-style construction over the same
primitives as HPKE base mode (DHKEM(X25519, HKDF-SHA256), AES-256-GCM),
binding both public keys into the KDF `info` as HPKE's `kem_context` does
to prevent key-substitution attacks. The deviation from strict RFC 9180
framing is documented, and the HKDF labels are versioned so a conformant
profile can be added later without touching existing payloads.
Nothing produces `encp` payloads yet — this is the primitive only. The
o11y layer is hardened defensively so sealed payloads render as
ciphertext rather than throwing `Unsupported serialization format`, and
`hydrateDataWithKey` skips the AES path for them since opening a sealed
payload needs the private scalar rather than the symmetric key.
- optional AAD on the AES helpers, used to bind `projectId|runId`
- `encapsulate`/`decapsulate` split so stream writers can amortize the
KEM across frames; documented that they must keep random per-frame
nonces and re-encapsulate per connection attempt, since a long-lived
content key plus counter nonces would repeat `(key, nonce)` after a
reconnect or a durable replay
- public key derivation is cross-validated against node:crypto's native
X25519 in tests, since it reads the public half out of a JWK export
* review: tighten sealed-box docs, key-length checks, and constants
Addresses review feedback on the sealed-box primitive:
- The module doc pointed at `getSerializeStream` as enforcing the
re-encapsulate-per-writer rule, but nothing in-tree uses `encapsulate`
yet, and the stream path added later seals per frame instead. Reworded
to state the two rules as the caller's contract, since this module
enforces neither.
- `derivePublicKeyFromScalar` now asserts the JWK-derived public key is
32 bytes. That decode is the one place this module trusts an external
encoding; a short value would otherwise fail much later inside key
agreement with a far less obvious message.
- `open()` used bare 12/16 for the nonce and tag sizes. Those now come
from exported `NONCE_LENGTH`/`TAG_BYTES` in the AES layer, so the wire
format check cannot drift from the implementation.
* feat(core): deterministic sandbox hardening
- crypto.subtle.digest computes synchronously via node:crypto:
byte-identical values, promise settles on a deterministic microtask,
full BufferSource validation (internal-slot view reads, SAB rejection)
- Atomics.waitAsync (a wall-clock timer), async WebAssembly compilation,
WeakRef, and FinalizationRegistry are removed from the sandbox — wall
clock and GC observation are unreplayable; sync WebAssembly
constructors remain
- freezeSerializationIntrinsics pins the universal dispatch surfaces:
Object.prototype/Array.prototype/Function.prototype are frozen (every
missed property read and hasInstance lookup terminates there) and
serialization-referenced global bindings are non-writable. Value-type
prototypes and constructor statics stay patchable so polyfills
(Temporal's Date.prototype.toTemporalInstant, core-js Set.prototype
.union / Object.groupBy) keep working — the retained-input gate
verifies the members serialization executes per boundary instead.
Groundwork for retained-VM replay (#2990).
* Drop serialization intrinsic freezing from the sandbox
The retained-VM passivity design moved from pinning/verifying the
sandbox surfaces serialization dispatches on to injecting hardened
operations into devalue itself (with taint-based de-opt), so freezing
Object/Array/Function prototypes and pinning global bindings is no
longer needed. Keep only the determinism hardening (sync digest,
removal of wall-clock/GC-observing APIs).
* Document and lock in why async crypto.subtle methods cannot break quiescence
The remaining async subtle methods reject immediately through the crypto
proxy (brand check — the receiver is not a real SubtleCrypto), so they can
never mint a host-timing promise. Narrow the quiescence comment to what the
code actually enforces and add a test so the unreachability is not silently
"fixed" later.
* simplify sandbox hardening: lean digest input conversion, async digest, explicit subtle throwers
* mark sandbox API removals as a major change
---------
Co-authored-by: Nathan Rajlich <n@n8.io>
* [core] Don't count racing invocations' duplicate step_started events toward the maxRetries ceiling
The retry-ceiling guard added in #3035 derives a step's attempt number from
the number of step_started events in the log. But invocations racing on the
same pending batch (stale replays, wake replays, step messages dispatched
off a lost create-claim) can each write a step_started for the same logical
attempt — worlds without an atomic start guard (world-local) let them all
through — so a healthy step's count could cross the ceiling and fail the
run with a false "exceeded max retries" (the world-testing
inline-batches-debug CI flake).
Scope the count to the starts each ceiling is actually about:
- inline owned-recovery: only starts stamped with THIS message's
ownerMessageId (each real (re)delivery of the owner re-stamps it;
racers stamp their own IDs or none)
- background steps: only bare/unstamped starts (that path never stamps
ownership, and throttle/too-early redeliveries still write no start)
Real timeout retries still bound: each recovery re-run writes another
owner-stamped (or bare) start, so genuine exhaustion still trips the guard.
* review: internalize helper, close mixed owned→bare ceiling gap, fix 'bare' terminology
- Move countStepStartedEvents to an internal module (src/runtime/
count-step-started-events.ts) instead of exporting it from the public
./runtime subpath; tests import it relatively.
- Replace the background ceiling's 'unowned' scope with 'totalAttempts'
(bare starts + largest single owner's starts), so a step that burns part
of its retry budget under inline owned recovery and then transitions to
queued/bare retries still trips the combined maxRetries ceiling, while
racers' one-off stamped duplicates still don't accumulate. Adds the
mixed-sequence regression test from review.
- Fix comments calling the owned-recovery start a 'bare step_started' —
it carries the ownerMessageId stamp; only the background path's start
is bare.
* perf(core): immediate leading-edge dispatch for idle streams (flush window default 0)
Production producer-rate data (24h of client flush spans): most agents
average 1.03-1.21 chunks per flush with 87-98% single-chunk flushes and
>70% of chunks arriving more than 10ms after the previous request had
already settled — a fixed 10ms leading window batches almost nothing
for them while adding ~20% to isolated-chunk publish latency (~50ms
median RTT). The one bursty producer (avg ~4-8 chunks/flush) gets its
batching from in-flight accumulation, which does not depend on the
window at all.
The leading chunk of an idle sink now dispatches immediately by
default (window 0): first chunk goes out at once, chunks arriving
during its request coalesce into the next group, and each settle
dispatches the accumulated group immediately — path-independent
batching with no fixed tax on slow producers. A positive
WORKFLOW_STREAM_FLUSH_INTERVAL_MS (or world.streamFlushIntervalMs,
applying from the second group) opts into a windowed leading edge for
slow-but-steady producers that prefer larger groups over first-chunk
latency. Early-ack, the durability drain barrier, wire caps, and
backpressure bounds are unchanged.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Update packages/world/src/interfaces.ts
Co-authored-by: Peter Wielander <mittgfu@gmail.com>
Signed-off-by: Karthik Kalyan <105607645+karthikscale3@users.noreply.github.com>
* review: env var overrides world streamFlushIntervalMs; world option governs the leading edge too
WORKFLOW_STREAM_FLUSH_INTERVAL_MS, when set, now takes precedence over
world.streamFlushIntervalMs; otherwise the world option applies from the
very first chunk (no more second-group lazy quirk). Deciding waits for
the world when needed, which adds no latency: sendGroup awaits the same
promise before any request can leave.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Signed-off-by: Karthik Kalyan <105607645+karthikscale3@users.noreply.github.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: Peter Wielander <mittgfu@gmail.com>
* perf(core): move stream write batching into WorkflowServerWritableStream (group commit)
Batching previously lived in flushablePipe's coalescing loop, so it only
engaged on paths that used flushablePipe (getWritable). A raw
ReadableStream crossing a workflow/step boundary is piped with native
pipeTo(), which does not pull chunk N+1 until write(chunk N) resolves —
and write() resolved only after the flush timer AND the server round
trip, so the buffer never held more than one chunk and every token
became its own server request.
The sink now group-commits:
- write() resolves when the chunk enters a bounded client buffer; the
bound counts buffered AND in-request chunks
(WORKFLOW_STREAM_MAX_INFLIGHT_CHUNKS, preserving its documented
meaning) plus a byte bound (WORKFLOW_STREAM_MAX_BUFFERED_BYTES, new,
default 8 MiB, documented in runtime-tuning). A full buffer applies
backpressure until a group lands durably.
- The flush interval is a real group-commit window; chunks arriving
while a request is in flight accumulate and form the next writeMulti
group. One request in flight at a time preserves chunk order.
- Per-request wire limits (1,000 chunks / 1 MiB) split groups exactly
as the coalescing pipe did; an oversized single chunk goes alone.
- Durability moved to an explicit barrier (STREAM_DRAIN_SYMBOL):
close() drains before closing; flushablePipe adopts the barrier so
lock-release completion (step completion) still means 'everything
written is durable'; abort() DRAINS the accepted prefix (never
closing) so a producer error after acked writes cannot lose data —
native pipeTo aborts the sink on source failure; and a failed pipe
drains before settling so a step failure is not persisted ahead of
the emitted prefix. A dispatch failure retains the group, poisons
the sink, and surfaces at the next write/close/drain.
flushablePipe is now a plain per-chunk pump responsible only for
lock-release completion and durability tracking; its coalescing
machinery and STREAM_WRITE_BATCH_SYMBOL are removed.
Covered: native-pipeTo batching (the regression), awaited per-chunk
loops coalescing into one writeMulti, in-flight accumulation, wire-cap
splits (count/byte/oversized), in-flight-inclusive backpressure for
both bounds, sequential fallback without writeMulti, source-error
prefix delivery through abort, failed-pipe drain-before-reject,
early-ack sticky errors, turbo run-ready barrier gating (incl. dwell
telemetry), drain-barrier adoption/rejection, and group-level flush
spans. 1,572 core unit tests pass; e2e tier requires a deployment and
was not run here.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(core): re-dispatch a chunk buffered in the request settle gap
Review (bot): a write landing between the dispatch loop's empty-buffer
exit and the reaction clearing the in-flight marker armed no timer
(scheduleGroupCommit saw the marker set) and was never dispatched on an
open stream — only a later write/close/drain would pick it up. The
settle reaction now re-dispatches when the buffer is non-empty, treating
the chunk as an in-request arrival; drain waiters settle with the new
chain. Regression test aims a write at the settle gap and asserts both
chunks flush without a close.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* docs(core): document abort-drain boundedness and terminal-run conflict handling
Review note: the abort-path drain is deliberately un-timeboxed (a bound
would drop acked chunks); its worst case is owned by the World
transport's finite timeout/retry budget, and a teardown-driven drain
into an already-terminal run rejects into the existing catch.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* test(core): poll instead of fixed sleeps for dispatch assertions
The native-pipeTo batching test flaked on a slow CI runner: a fixed
25ms wait raced the 10ms commit window plus scheduler jitter. All
'dispatch has happened' assertions now poll the expectation (bounded);
intentional negatives keep their fixed windows.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* test(e2e): poll the events readback in stepFunctionPassingWorkflow
The events listing prefers the analytics store, which ingests
asynchronously. Reading it immediately after run completion can miss
the freshest events (the page is non-empty, so the storage fallback
does not trigger), failing the step_completed assertion. Poll for up
to 20s so ingestion has time to land; the assertion itself is
unchanged.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* test(e2e): force storage-backed inspect listings via WORKFLOW_DISABLE_ANALYTICS_READS
The analytics store ingests asynchronously; e2e assertions read events
and steps immediately after run completion and can catch a page missing
the freshest rows (observed as stepFunctionPassingWorkflow's
step_completed readback returning empty, and the same race on steps
listings in other suites). Instead of polling each readback, disable
the analytics namespace for the e2e's CLI invocations so every inspect
listing is served read-your-writes from primary storage. Replaces the
earlier bounded poll with the deterministic mechanism.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Enforces the published per-run events limit, which was previously not enforced. The server supplies the limit on the run_started response (separate change); once a run's event log reaches it, the runtime throws MaxEventsExceededError at the top of the replay loop, and the existing terminal-error path records it as run_failed with a new MAX_EVENTS_EXCEEDED code — instead of letting a runaway workflow (e.g. an unbounded step loop) grow the event log without bound.
Adds a new client side WORKFLOW_MAX_EVENTS_OVERRIDE env var which can override the server side provided value (lower only).
* [core] Enforce maxRetries for steps that time out
A step that is hard-killed by the platform function timeout writes no
step_failed/step_retrying, so `step.error` stays null and the error-based
max-retries guards never fire. Each redelivery re-runs step_started
(incrementing the attempt), so a timing-out step retried without bound
instead of stopping at maxRetries.
Enforce the retry ceiling BEFORE running the body, via a new
`authoritativeAttempt` param on executeStep:
- Inline (combined handler): count the step_started events already in the
event log for the step (+1 for this attempt). The log is authoritative
because the optimistic-start path synthesizes step.attempt = 1.
- Background (queue-dispatched): the queue delivery count (metadata.attempt),
which increments on the visibility-timeout redelivery a timed-out step
produces.
When the attempt exceeds maxRetries + 1 the step is failed without starting
another attempt. Thrown-error exhaustion is unchanged — it still terminates
via the post-body guard one attempt earlier, with the thrown error as cause.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Apply suggestions from code review
Co-authored-by: Peter Wielander <mittgfu@gmail.com>
Signed-off-by: Peter Wielander <mittgfu@gmail.com>
* [core] Verify step_started count before failing a backgrounded step
`metadata.attempt` (queue delivery count) also advances for redeliveries
that never run the step body (ThrottleError, TooEarlyError, other pre-body
failures), so trusting it directly could fail a step as "exceeded max
retries" before the body ever ran.
Use the delivery count only as a fast gate: while it is at or under the
ceiling the step can't be exhausted, so proceed without touching the log.
Only once it crosses the ceiling, load the full event log and derive the
authoritative attempt from the recorded step_started count (which only real
attempts write) — excluding throttle/too-early redeliveries. The load also
primes the replay's cachedEvents/eventsCursor.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* [core] Skip the event-log scan for brand-new inline steps
Deriving an inline step's attempt number by scanning the cumulative event
log for step_started events ran for every inline execution, which is O(n²)
across a long sequential workflow.
A lazy inline step is brand-new by construction (it only enters the batch
with no step_created yet), so it has zero prior starts and is always attempt
1 — no scan needed. Reserve the scan for owned-recovery re-runs (this
message re-executing a step it crashed/timed out on), which are uncommon and
few per batch.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Signed-off-by: Peter Wielander <mittgfu@gmail.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* test: regression coverage for hook.resume() from isolated route bundles (o2flow beta.26 incident)
Reproduces the o2flow v5 upgrade failure (workflow@5.0.0-beta.26, fixed by
#2752 in beta.28): a plain API route importing defineHook() from the root
`workflow` entry and calling .resume() failed with Turbopack's
"Cannot find module as expression is too dynamic" stub, because the world
registration was tree-shaken out of the route bundle and getWorldLazy()'s
dynamic-import fallback got stubbed.
The bug only manifests when a route bundle loads in isolation (a Vercel
lambda): local `next dev`/`next start` evaluates next.config.ts, whose
workflow/next import chain registers the world process-wide and masks it —
which is why no existing server-driven suite caught it.
- route-bundle-isolation.test.ts: production Turbopack build of the
nextjs-turbopack workbench, then loads ONLY the compiled route bundle in a
bare Node subprocess (cold-lambda simulation) and invokes its POST handler.
Fails with the exact incident error on regressed code; passes on main.
Wired into the build-error-messages CI job.
- e2e: plainModuleDoneHook round-trip through a plain API route on the two
Next workbenches (deployed matrix covers real lambda isolation).
- Workbench fixtures mirroring o2flow: a directive-less defineHook module
shared by a workflow (create) and a plain route (resume). The webpack
workbench gets a real route file because `next dev` (webpack) does not
serve directory-symlinked app routes.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Pranay Prakash <pranay.gp@gmail.com>
* test: authenticate plain hook resume request
* test: address review — marker-based harness output parsing, changeset summary
- route-bundle-isolation: prefix the harness result line with a unique
marker and locate it explicitly instead of JSON.parse()ing the last
stdout line, so stray logging from the route bundle or the world can't
break parsing; failures now include the full subprocess stdout.
- changeset: add a human-readable summary to the (release-less) changeset.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Pranay Prakash <pranay.gp@gmail.com>
---------
Signed-off-by: Pranay Prakash <pranay.gp@gmail.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: Karthik Kalyanaraman <karthik.kalyanaraman@vercel.com>
Co-authored-by: Karthik Kalyan <105607645+karthikscale3@users.noreply.github.com>
* fix(deps): dedupe @opentelemetry/api to a single workspace instance
The lockfile resolved both 1.9.0 and 1.9.1, so the copy that registers
the tracer provider (via @vercel/otel in the app) and the copy a package
imports could differ. The API's global-registration version check rejects
a consumer newer than the registered copy and silently hands back a noop
tracer — which is why world-vercel's spans (workflow.stream.write/
chunk_rtt, read.connect, its http spans) never reached Datadog from
deployed apps while core's spans flowed in the same process. Root-caused
via the DEBUG=workflow:* run on #2900: import succeeds, no warn, spans
dropped.
Pin a single version via a workspace override so every bundle shares one
API instance.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* debug: one-shot OTEL runtime diagnostic in core + world-vercel; DEBUG on turbopack workbench
The dedupe alone did not restore world-vercel span emission (verified on
this PR's own preview: stream traffic flowed, zero workflow.stream.write
spans). Under DEBUG=workflow:*, both packages now log once how their
module instance of @opentelemetry/api sees the world — global
registration version, provider/delegate/tracer/probe constructor names,
and whether a probe span is recording. Diffing the core line (spans work)
against the world-vercel line (spans dropped) in one deployment's logs
pinpoints the divergence.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* debug: log span identity for named world-vercel spans; namespace otel probes per package
Diag round 1 showed world-vercel's tracer records and instrumentedFetch
handles the stream PUTs, yet the named spans are unfindable in the
backend. Round 2: log traceId/spanId/isRecording for every named
instrumentedFetch span under DEBUG so export can be checked for a
specific span id, and split the probe span names (.core /
.world_vercel) so per-package export is attributable.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* telemetry: emit stream RPC latencies from core (chunk_rtt, connect_ms, close span)
world-vercel's instrumentedFetch spans never export from deployed apps
(root cause still open — see PR discussion), so the operationally
needed client-side latency signals move one layer up to core, whose
spans are proven to export:
- workflow.stream.write.chunk_rtt on the workflow.stream.flush span:
the World write RPC duration, network included (same attribute key as
world-vercel's per-request span so queries are layer-agnostic).
- workflow.stream.read.connect_ms on the workflow.stream.read span:
the world.streams.get await (read dispatch -> stream handle).
- new workflow.stream.close span: the close RPC round trip.
Bonus: measured at the World interface, these cover world-local and
world-postgres too, not just Vercel deployments.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* telemetry: emit read-completion span (total duration, chunks, bytes)
Completes the read-side picture: workflow.stream.read.complete is
back-dated to the read dispatch so its duration is the total read, with
chunk/byte counts for throughput. Cancelled reads emit nothing.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* chore: drop DEBUG from turbopack workbench; tighten changeset
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* telemetry: cover createReconnectingFramedStream in read telemetry
Ordinary serialized streams read through createReconnectingFramedStream
(which calls world.streams.get directly), so connect_ms / ttfc /
read.complete never fired for that path — only WorkflowServerReadableStream
was instrumented. Wire the same helpers into the framed reader: first-
connect duration, first-frame TTFC, and completion totals — plus
workflow.stream.read.reconnects, which only this path can know.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* [world-vercel] Add /run-id sub-export with tagged ULID encode/decode
Encodes a tag bit, 5-bit version, and 6-bit Vercel region ID into a
ULID-shaped string used for workflow run IDs. Tagged values remain
valid 26-char Crockford-Base32 ULIDs so they still sort and round-trip
through any system that accepts ULIDs.
* [world-vercel] Add string-value assertions to run-id tests
Add exact-string expectations for encoded outputs at known inputs,
covering the default region/version pair, numeric region IDs, version
overrides, boundary values (all-zero, all-max), the dirty-input
overwrite case, and the lexicographic-order checks. Also adds an
explicit byte-array expectation for the canonical ULID-spec example
string and an additional first-char-range coverage test for isTagged.
* [world-vercel] Remove internal-repo reference from regions doc comment
* [world-vercel] Address PR review feedback on run-id sub-export
- isTaggedString now fully validates the input as a 26-char Crockford
Base32 ULID (delegating to ulidToBytes) instead of only inspecting
the first character. This fixes false positives on inputs like
'4UUUU...' that have a valid tag-bit position but invalid chars
later in the string.
- isTagged() now accepts `unknown` to match its documented behavior
of safely rejecting non-string inputs without requiring callers to
cast.
- Introduce `RegionKey` for the full set of keys including 'unknown',
and narrow `RegionCode` to `Exclude<RegionKey, 'unknown'>` so the
return type of `lookupRegion` and the `DecodedRunId.region` field
accurately reflect that 'unknown' is never produced. Updates
`encode` to reject 'unknown' as a region code string at runtime
(callers wanting the unknown sentinel should pass numeric 0).
* [world] [core] [world-vercel] Add World.createRunId() and region-aware queue routing
- @workflow/world: add optional createRunId(input?) to the World
interface so worlds can mint run IDs with embedded metadata, and
add an optional 'region' field to QueueOptions for per-message
routing hints.
- @workflow/core: start() now delegates run ID generation to
world.createRunId() when defined (falling back to a monotonic
ULID otherwise), and accepts a new 'runIdInput' option that is
forwarded verbatim to createRunId. When runIdInput.region is a
string, it is also threaded onto the queue options so the initial
workflow message is dispatched to the matching region.
- @workflow/world-vercel: implement createRunId() to mint
region-tagged ULIDs, preferring an explicit runIdInput.region and
falling back to the VERCEL_REGION env var. The queue now resolves
its destination region from (in order): an explicit opts.region,
the region embedded in the payload's tagged run ID, the
VERCEL_REGION env var, and finally a hardcoded 'iad1' default.
This replaces the previous unconditional 'iad1' region passed to
the @vercel/queue client.
Monotonicity within a process is preserved by tracking the last
emitted run ID and bumping the bit immediately above the 11-bit
metadata window when a same-ms collision would otherwise occur,
then re-stamping the requested region/version on top so metadata
remains stable.
* [core] [world] [world-vercel] Pass full StartOptions to World.createRunId
Drop the dedicated 'runIdInput' field on StartOptions and forward the
entire options bag to world.createRunId() instead. This keeps the
public API surface smaller and lets each World pick the fields it
recognises (e.g. world-vercel reads 'region'). The top-level 'region'
option remains on StartOptionsBase and is also threaded onto the
queue's per-call region opt when set.
* Address review feedback: doc fixes and deterministic same-ms tests
- Document the final iad1 fallback in QueueOptions.region (world)
- Correct the World.createRunId doc: start() always passes an object
- Fix the clientOptions comment: the handler client omits region and
relies on SDK auto-detection + the ce-vqsregion header for acks
- Fix a misleading QueueClient-construction comment in queue.test.ts
- Freeze time in the same-ms monotonicity test so it deterministically
exercises the intended path, and add a test covering the
bump-above-metadata fallback when the region changes mid-millisecond
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* test: keep workflow-server override rewrite-compatible
Export WORKFLOW_SERVER_URL_OVERRIDE while keeping the one-line const shape
that workflow-server's cross-repo e2e test automation rewrites. Update
world-vercel tests to import that exported value for mock origins and URL
expectations instead of duplicating the temporary preview URL.
* fix(world): clear region tag bit before ULID timestamp validation
Region-tagged run IDs set the high bit of the ULID timestamp byte. The
shared world timestamp validator used raw decodeTime(), so current tagged
run IDs appeared thousands of years in the future and were rejected before
reaching workflow-server. Clear the tag bit before decoding, matching the
workflow-server behavior, and cover tagged IDs in tests.
* fix(world-vercel): validate tagged runId timestamps via run-id decode
Keep @workflow/world's ULID helpers generic; they should not know about
world-vercel's region-tagged run ID layout. Instead, world-vercel decodes
its tagged runId to the original ULID before using the shared timestamp
validator for run_created events. Add a world-vercel regression test that a
current sfo1-tagged runId passes validation.
* fix(world-vercel): default run ID region to iad1 instead of unknown
When neither an explicit region option nor VERCEL_REGION is available,
createRunId minted a tagged ULID with the unknown (0) region sentinel,
producing the tagged: true, region: null state. The server already
resolves unknown/untagged runs to DEFAULT_VERCEL_REGION (iad1), so mint
a concrete iad1 tag instead, keeping every run ID self-describing and
routable.
* test(e2e): use verbose reporter + per-test start heartbeat
The default vitest reporter buffers per-file output, so a stalling e2e
test produces no output until its timeout — making CI look like a silent
30-minute hang. Switch the e2e CI invocations to the verbose reporter
(prints each test result as it completes) and emit a '[e2e] ▶ start:'
heartbeat to stdout at the start of every test (bypassing vitest's console
buffering) so a stuck test is immediately identifiable in the live CI log.
* test(world-vercel): point WORKFLOW_SERVER_URL_OVERRIDE at combined 527+529 preview
Temporarily target the workflow-server combined-527-529-preview deployment,
which bundles platform-directed multi-region routing (vercel/workflow-server#527,
incl. the iad1 hook pin) and durable stream state (vercel/workflow-server#529),
so e2e can validate the full multi-region path end-to-end. Revert to empty on main.
* fix(core): region-tag the health-check correlationId
The health-check response is delivered over a Redis stream whose name (and
synthetic run ID) embed the correlationId. Under platform-directed routing
the responding endpoint and the polling reader can be served from different
physical regions; Redis is physical-region-local, so the correlationId must
carry the region for both sides to resolve the same backend.
Generate the correlationId via world.createRunId() (a region-tagged ULID)
when the world provides it, falling back to a plain ULID for worlds that
don't tag IDs (e.g. local, single-region). The synthetic wrun_hc_<id> run ID
then carries the region; workflow-server's region middleware decodes it.
* Address review feedback: validate region overrides, reset server override
- queue: validate opts.region and VERCEL_REGION against the known region
table before routing, ignoring unrecognised codes so a bad override
can't clobber the payload-derived region (Copilot)
- add isKnownRegionCode() runtime guard to run-id/regions
- reset WORKFLOW_SERVER_URL_OVERRIDE to '' (must be empty on main)
- fold the within-PR iad1-default changeset into the main world-vercel
changeset and delete it (review)
- start.test: declare specVersion on createRunId mock worlds now that
the merged world-compatibility check requires it
- cover the new region-validation fall-through paths in queue.test
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* test(world-vercel): point WORKFLOW_SERVER_URL_OVERRIDE at wave-1 multi-region preview
BRANCH-ONLY — revert the override to '' before merge (lint enforces).
Points this PR's e2e/benchmark runs at the wave-1 multi-region
workflow-server preview (vercel/workflow-server#590: iad1+sfo1+fra1
serving, staging data backends) so region-tagged runs are validated
against real multi-region serving end-to-end.
Also makes the unit-test mock origins in events-v4.test.ts and
trace-propagation.test.ts override-aware (same pattern the rest of
the file and utils.test.ts already use), so the suite passes whether
or not the override is set — these two files were the only spots
hardcoding https://vercel-workflow.com.
* test(e2e): Vercel multi-region suite for start()'s region option
Adds a dedicated e2e suite validating @workflow/world-vercel region
routing end to end, run as its own CI job (e2e-vercel-multi-region)
against the nextjs-turbopack workbench only — deliberately separate
from e2e.test.ts, which runs as a matrix across all worlds/frameworks
where Vercel-specific multi-region behavior doesn't apply.
- workbench/nextjs-turbopack/vercel.json: deploy to iad1+sfo1+fra1 so
region-routed flow messages have a function to land on in each region.
- workflows/99_e2e.ts: regionProbeWorkflow returns the VERCEL_REGION
observed by both the workflow and a step, so tests can assert the run
EXECUTED in the intended region (not just that it was tagged).
- packages/core/e2e/e2e-region.test.ts: per-region cases assert
1) start(..., { region }) mints a region-tagged run ID (decoded via
@workflow/world-vercel/run-id),
2) the workflow + step both observed VERCEL_REGION === region,
3) the server reports the run completed;
plus a concurrent all-regions case guarding against cross-region
misrouting under simultaneous multi-region traffic. Skips on local
deployments.
- .github/workflows/tests.yml: new e2e-vercel-multi-region job
mirroring e2e-vercel-prod's env/deployment-wait, running only the
new suite.
* test(e2e): start region probes in-function; fix getWorld await
The first multi-region CI run surfaced two issues:
1. sfo1/fra1-tagged runs executed in iad1. The suite started runs from
the external test process, which uses the api.vercel.com token proxy
— and the proxy's queues path forwards every send to the region-less
VQS host (the world's proxy-mode resolveBaseUrl ignores the region
argument, and the proxy's x-vercel-vqs-api-url escape hatch only
allowlists vqs-server-*.vercel.sh preview hosts). Production traffic
publishes IN-FUNCTION (direct regional queue routing), so the suite
now triggers start() through a new workbench route
(/api/e2e-region-start) and rehydrates the run with getRun() —
testing the path production actually takes. Proxy-mode regional
queue routing is a known gap to address separately in api-workflow.
2. TypeError on world.runs.get: getWorld() is async and was called
without await.
* test(e2e): cover explicit and implicit region starts in the multi-region suite
With regional VQS routing now working through the api.vercel.com proxy
(vercel/api#79056 + #2789 + this branch's per-send region resolution),
the suite covers both start configurations, asserting the same three
properties for each (region-tagged run ID, execution in the intended
region via VERCEL_REGION echoed in the return value, server-side
completion):
1. EXPLICIT: start(..., { region }) called directly in the vitest
runner — publishes through the token proxy, per-send region carried
by x-vercel-queue-region. Restores the direct-start shape the suite
had originally, plus the concurrent all-regions case.
2. IMPLICIT: dedicated per-region workbench routes
(/api/e2e-region-implicit/{iad1,sfo1,fra1}), each pinned to a single
region via a per-function 'regions' entry in the workbench
vercel.json, calling start() with NO region option — createRunId
derives the tag from the minting function's VERCEL_REGION. The test
also asserts the route reported executing in its pinned region, so
the implicit-tagging assertion can't pass vacuously.
Replaces the interim /api/e2e-region-start route (explicit region via
request body), which existed to work around the pre-#79056 proxy gap.
* Revert WORKFLOW_SERVER_URL_OVERRIDE to '' — wave-1 multi-region serving is in production
workflow-server#590 (iad1+sfo1+fra1 serving) merged and deployed to
production and the e2e backend, so this branch's e2e/benchmark runs no
longer need to target the wave-1 preview. Restores the empty override
the No Test Overrides lint job enforces for merge.
The override-aware unit-test origins (events-v4/trace-propagation)
stay — they are correct under any override value.
* test(e2e): cross-region stream visibility (iad1 writer, sfo1 reader)
Regression coverage for a backend bug that made cross-region stream
reads report zero chunks on IN-PROGRESS streams (completed streams were
unaffected), which forced the multi-region serving rollback.
The new case exercises exactly that geometry:
- crossRegionStreamWorkflow (99_e2e.ts) writes N chunks to the default
output stream, then holds the stream OPEN for 45s before closing —
the in-progress window is the point, since completed streams are the
easy case.
- The e2e starts it with region iad1, waits (same-region, via the
api.vercel.com proxy) until all chunks are written, asserts the run
is still 'running', then reads through a new sfo1-pinned workbench
route (/api/e2e-stream-read/sfo1) that returns getTailIndex() plus
its VERCEL_REGION. The reader's region served none of the stream's
writes, so the reported chunk count must come from the backend's
cross-region stream metadata. The test fails loudly if the route
isn't actually executing in sfo1.
Also bumps the explicit-region test timeout to 120s: the first case in
the file absorbs every cold start at once (fresh workbench instances
in up to three regions plus a cold backend preview) and was observed
just over the 60s default.
BRANCH-ONLY (revert before merge, lint enforces):
WORKFLOW_SERVER_URL_OVERRIDE points at a multi-region backend preview
that includes the fix, so this validates cross-region stream
visibility end-to-end before multi-region serving is re-enabled.
* test(e2e): extend multi-region suite to all 19 provisioned regions
Points the suite at an all-regions backend preview and widens coverage
from the wave-1 trio to every provisioned region:
- Explicit path: a single concurrent all-regions case starts one
tagged run per region (one shared cold-start window instead of 19
sequential ones) and aggregates per-region failures so a single
region's breakage reports alongside the full picture. The trio keeps
its detailed per-region cases and the 9-way concurrent-isolation
case.
- Implicit path: workbench gains a region-pinned
/api/e2e-region-implicit/<region> route per provisioned region (19
total, shared handler), the workbench itself now deploys to all of
them, and the test.each covers the full set with per-case timeouts
for regional cold starts.
- Multi-region CI job timeout 20m -> 35m for the sequential implicit
cases.
BRANCH-ONLY (revert before merge, lint enforces):
WORKFLOW_SERVER_URL_OVERRIDE now targets the all-regions backend
preview instead of the previous (stale, since-merged) fix preview.
* test(e2e): tolerate geo-adjacent execution of queue callbacks
The first all-regions run surfaced a subtle execution-locality
behavior: queue delivery is guaranteed to the tagged region's
dataplane and the delivery callback egresses from that region, but the
consumer invocation's execution region is chosen by where that
callback enters Vercel's edge — and adjacent regions can geo-resolve
to each other's functions. Observed live: kix1-tagged runs (callback
egressing from Osaka) deterministically executing in hnd1/Tokyo on
both the explicit and implicit paths, with tagging, data placement,
and completion all still strictly kix1.
expectRunInRegion now asserts execution lands in the tagged region OR
one of its geographic neighbors (EXECUTION_ADJACENCY), while run-ID
tagging and server-side completion remain strictly the requested
region. Gross misrouting (e.g. kix1 -> iad1) still fails.
* Revert WORKFLOW_SERVER_URL_OVERRIDE to '' — all-regions serving is in production
The all-regions workflow-server rollout is deployed and serving
production traffic from every Vercel region, so this branch's e2e no
longer needs to target a branch preview. Restores the empty override
the No Test Overrides lint enforces for merge.
With this the PR is complete: region-tagged run IDs, region-aware
queue routing, and the multi-region e2e suite (explicit + implicit +
all-regions + cross-region streams) all validate against the
production-default backends.
* docs: fix three stale comments flagged in review
- start.ts: StartOptionsBase.region fallback is iad1, not the unknown
sentinel (createRunId always mints a concrete routable region)
- queue.ts: example used a nonexistent start({ runIdInput }) API; the
real option is start({ region })
- events.ts: decode() clears only the tag bit (top bit of the 48-bit
timestamp field) — it does not restore the original untagged ULID;
reword to say what actually matters for timestamp validation
* test(e2e): cover hook resolve/resume for runs owned by non-iad1 regions
Hooks are resolved by opaque token, which carries no region hint, so
lookup and resume must work regardless of which region owns the run's
data. Exercises the full follow-up-message path on sfo1- and
fra1-tagged runs: create inside the workflow, resolve by token from
the test process, resume twice sequentially, and assert payload order
and completion.
Regression coverage for the failure mode where the first message to a
hook-driven app on a non-iad1 run worked but every follow-up failed
with 'Hook not found'.
---------
Co-authored-by: Peter Wielander <peter.wielander@vercel.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* build: declare typescript (catalog:) in every package that runs tsc
Twenty packages invoke tsc in their build/typecheck scripts without
declaring a typescript dependency, resolving whatever tsc pnpm happens
to leave reachable. That broke locally after the TypeScript 6 upgrade
(#2700): base.json now uses the TS6-only 'types': ['*'] wildcard, and
worktrees carrying pre-upgrade node_modules/.bin/tsc shims (orphaned
typescript@5.9.3 bins that pnpm never refreshes for an undeclared
dependency) fail with TS2688 'Cannot find type definition file for *'.
Declaring 'typescript': 'catalog:' (the convention nest already
follows) makes pnpm own each package's tsc bin, so version upgrades
refresh the shims and this staleness class cannot recur. Packages
without tsc in their scripts are left unchanged.
Full pnpm build: 27/27 tasks green.
* Address review: drop duplicate zod devDep; regenerate lockfile minimally
- packages/world listed zod in both dependencies and devDependencies
(pre-existing on main, surfaced by the devDependencies sort) — keep
the runtime dependency only.
- Regenerate pnpm-lock.yaml from a pristine main baseline with
--lockfile-only (a clean-main run produces zero diff, so main has no
drift). Remaining non-typescript changes are mechanical consequences
of the change itself: typescript is an (optional) peer of several
tooling dependencies, so declaring it in 20 importers creates new
peer-resolution snapshot variants and prunes the now-orphaned old
ones; plus one radix-ui 1.6.1->1.6.2 refresh in docs caused by its
floating 'latest' specifier.
- Validated: pnpm install --frozen-lockfile succeeds; full build 27/27.
* telemetry: emit client-observed workflow.stream.write span per flush batch
Complements the existing workflow.stream.read TTFC span: each flushed
batch emits a back-dated CLIENT span covering the app-perceived write
latency (buffer dwell + RPC), with buffer_dwell_ms / chunks / bytes
attributes so client-side batching cost (flush timer, turbo run-ready
barrier) can be told apart from network/server time. Failed flushes
keep the batch's original t0 so a retried batch reports its full dwell.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* chore: tighten changeset
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* telemetry: rename flush span to workflow.stream.flush; DEBUG-log world-vercel OTEL load failure
- workflow.stream.write is taken by world-vercel's per-request RPC span
(#2857, chunk_rtt); the per-batch flush span gets its own name so the
two stay distinguishable in trace queries. Attributes move to
workflow.stream.flush.{buffer_dwell_ms,chunks,bytes}, operation=flush.
- world-vercel's @opentelemetry/api load failure was silently latched as
null, which also swallows bundler/resolution failures in apps that DO
register a tracer (observed in production: workbench apps emit core
spans but none of world-vercel's). Log the reason under
DEBUG=workflow:* so the failure mode is diagnosable.
- Document workflow.stream.flush in the tracing docs.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* chore: tighten changeset
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* feat(core): record replay lineage in run executionContext
recreateRunFromExisting now stamps the source run id into the new run's
executionContext as `replayedFromRunId`, and `start` accepts a matching
option. This lets tooling (e.g. the dashboard runs list) surface a run as
a replay and link back to the run it was replayed from — previously a
replay started a brand-new run with no link to its origin.
Co-authored-by: Cursor <cursoragent@cursor.com>
* Apply suggestion from @mitul-s
Signed-off-by: Mitul Shah <mitulxshah@gmail.com>
* cleanup
* Update runs.test.ts
* update
---------
Signed-off-by: Mitul Shah <mitulxshah@gmail.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
* [core] Thread queue namespace through o11y run actions and bound healthCheck stream reads
Deployments that use a queue namespace (e.g. eve's __eve_wkf_workflow_*
topics) could not be targeted by cross-context callers: start(),
recreateRunFromExisting(), reenqueueRun(), and wakeUpRun() always built
queue names from the caller's WORKFLOW_QUEUE_NAMESPACE env, so dashboard
replays published to topics the target deployment has no consumer for.
Additionally, healthCheck()'s poll loop only checked its timeout between
iterations while world.streams.get() itself was unbounded — against
workflow-server, which holds unwritten streams open for ~2 minutes, a
2s capability probe hung until the caller's function timed out (the
observed 30s 504s on dashboard replay).
- Add a `namespace` option to StartOptionsBase, RecreateRunOptions,
StopSleepOptions, and new ReenqueueRunOptions; thread it into
getWorkflowQueueName() and the cross-deployment capability probe.
- Fold the inline `namespace` param into HealthCheckOptions.
- Race streams.get() against the remaining health-check budget.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Shorten changeset
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Nathan Colosimo