* origin/main:
test: regression coverage for hook.resume() from isolated route bundles (o2flow beta.26 incident) (#3001)
docs(agents): note lint/format/typecheck are advisory, not blocking (#2886)
Retry transient connection timeouts (#3013)
fix(world-vercel): append caller User-Agent products instead of discarding them (#2998)
[ci] Enable NestJS e2e-vercel-prod and add to docs as "experimental" (#3011)
[ci] Benchmark comment: Best column + best/p75/p99 deltas (drop Avg/P10) (#3005)
docs: fall back to first child page for sidebar folders without an index (#3009)
[nest] Fix NestJS Vercel build output (#2988)
Avoid resolving run data for background steps (#2993)
chore(docs): update @vercel/geistdocs to 1.14.0 (#3002)
fix(docs): add version-switcher fallback redirects for pages missing in one version (#3003)
ci: update opencode to 1.18.4 and switch backport AI model to claude-fable-5 (#3006)
fix(core): batch stream writes via writeMulti (#2995)
Owner feedback on the opt-in fast path: stream-based `await run.returnValue`
should be on by default and work on every World, since all worlds already
implement streams.
- `WORKFLOW_RETURN_VALUE_STREAM` is now a default-on emergency kill switch
(`=0`/`false` restores the pure fixed 1s poll), mirroring `WORKFLOW_TURBO`
and `WORKFLOW_INLINE_OWNERSHIP`.
- Drop the `returnValueSignalStream` World capability gate. Verified the
marker-write + close / `startIndex: 0` catch-up contract the waiter relies on
works out of the box on world-local (filesystem) and world-postgres
(NOTIFY + durable rows); world-vercel relies on workflow-server's durable
chunk replay to late readers (documented). Nothing needed fixing.
- Keep the 5s fallback poll (`WORKFLOW_RETURN_VALUE_FALLBACK_POLL_MS`) as the
internal never-hang backstop.
- Tests: kill-switch-off is byte-identical to the legacy poll; per-world
catch-up tests for local (in core) and postgres (testcontainers); drop the
missing-capability tests.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* test: regression coverage for hook.resume() from isolated route bundles (o2flow beta.26 incident)
Reproduces the o2flow v5 upgrade failure (workflow@5.0.0-beta.26, fixed by
#2752 in beta.28): a plain API route importing defineHook() from the root
`workflow` entry and calling .resume() failed with Turbopack's
"Cannot find module as expression is too dynamic" stub, because the world
registration was tree-shaken out of the route bundle and getWorldLazy()'s
dynamic-import fallback got stubbed.
The bug only manifests when a route bundle loads in isolation (a Vercel
lambda): local `next dev`/`next start` evaluates next.config.ts, whose
workflow/next import chain registers the world process-wide and masks it —
which is why no existing server-driven suite caught it.
- route-bundle-isolation.test.ts: production Turbopack build of the
nextjs-turbopack workbench, then loads ONLY the compiled route bundle in a
bare Node subprocess (cold-lambda simulation) and invokes its POST handler.
Fails with the exact incident error on regressed code; passes on main.
Wired into the build-error-messages CI job.
- e2e: plainModuleDoneHook round-trip through a plain API route on the two
Next workbenches (deployed matrix covers real lambda isolation).
- Workbench fixtures mirroring o2flow: a directive-less defineHook module
shared by a workflow (create) and a plain route (resume). The webpack
workbench gets a real route file because `next dev` (webpack) does not
serve directory-symlinked app routes.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Pranay Prakash <pranay.gp@gmail.com>
* test: authenticate plain hook resume request
* test: address review — marker-based harness output parsing, changeset summary
- route-bundle-isolation: prefix the harness result line with a unique
marker and locate it explicitly instead of JSON.parse()ing the last
stdout line, so stray logging from the route bundle or the world can't
break parsing; failures now include the full subprocess stdout.
- changeset: add a human-readable summary to the (release-less) changeset.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Pranay Prakash <pranay.gp@gmail.com>
---------
Signed-off-by: Pranay Prakash <pranay.gp@gmail.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: Karthik Kalyanaraman <karthik.kalyanaraman@vercel.com>
Co-authored-by: Karthik Kalyan <105607645+karthikscale3@users.noreply.github.com>
`await run.returnValue` polls the run record on a fixed ~1s interval, adding
up to a second of quantization latency for a run that finishes mid-interval
(production trace 60cc034ba74b04e0fa2f70aa302a027f).
Add an opt-in fast path: when `WORKFLOW_RETURN_VALUE_STREAM` is on and the
World declares the new `returnValueSignalStream` capability, the poll loop
waits on a run-scoped system stream (`strm_…_system_return`) instead of the
fixed sleep. Every terminal transition in the live flow runtime — plus both
cancellation entry points — writes a tiny marker to that stream and closes it,
waking the waiter within a stream round-trip. The stream is a signal only: the
waiter always re-reads the authoritative run via `runs.get`, and a slow
fallback poll backstops any missed signal (crash between the terminal event
write and the stream write, transient stream failures, pre-feature runs).
Flag off, or a World without the capability, is byte-identical to the previous
fixed-poll behavior. world-vercel declares the capability.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Hook timeline bars were amber/yellow via RESOURCE_COLORS; passive
spans should be gray to match the event list icons and minimap.
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
* Give Metadata Token and Hook ID copy + truncation
Add token to the copyable metadata attributes set and constrain
copyable key-value rows so MiddleTruncate can shrink long IDs.
* Remove AttributePanel copy unit tests
The Metadata Token/Hook ID copy change is small enough that the
dedicated panel render tests are unnecessary.
---------
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
* fix(nitro): support React Router Vite builds
* refactor(nitro): simplify React Router cleanup
* docs(react-router): specify cleanup version
* fix(nitro): close temporary Vite servers
The recoverActiveRuns factory option had no environment variable, so
disabling startup re-enqueueing of pending/running runs required a custom
world module via WORKFLOW_TARGET_WORLD. Wire an env fallback
(0/false disables, 1/true enables, explicit factory option wins) and
document it in the worlds configuration reference and local world guide.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* fix(deps): dedupe @opentelemetry/api to a single workspace instance
The lockfile resolved both 1.9.0 and 1.9.1, so the copy that registers
the tracer provider (via @vercel/otel in the app) and the copy a package
imports could differ. The API's global-registration version check rejects
a consumer newer than the registered copy and silently hands back a noop
tracer — which is why world-vercel's spans (workflow.stream.write/
chunk_rtt, read.connect, its http spans) never reached Datadog from
deployed apps while core's spans flowed in the same process. Root-caused
via the DEBUG=workflow:* run on #2900: import succeeds, no warn, spans
dropped.
Pin a single version via a workspace override so every bundle shares one
API instance.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* debug: one-shot OTEL runtime diagnostic in core + world-vercel; DEBUG on turbopack workbench
The dedupe alone did not restore world-vercel span emission (verified on
this PR's own preview: stream traffic flowed, zero workflow.stream.write
spans). Under DEBUG=workflow:*, both packages now log once how their
module instance of @opentelemetry/api sees the world — global
registration version, provider/delegate/tracer/probe constructor names,
and whether a probe span is recording. Diffing the core line (spans work)
against the world-vercel line (spans dropped) in one deployment's logs
pinpoints the divergence.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* debug: log span identity for named world-vercel spans; namespace otel probes per package
Diag round 1 showed world-vercel's tracer records and instrumentedFetch
handles the stream PUTs, yet the named spans are unfindable in the
backend. Round 2: log traceId/spanId/isRecording for every named
instrumentedFetch span under DEBUG so export can be checked for a
specific span id, and split the probe span names (.core /
.world_vercel) so per-package export is attributable.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* telemetry: emit stream RPC latencies from core (chunk_rtt, connect_ms, close span)
world-vercel's instrumentedFetch spans never export from deployed apps
(root cause still open — see PR discussion), so the operationally
needed client-side latency signals move one layer up to core, whose
spans are proven to export:
- workflow.stream.write.chunk_rtt on the workflow.stream.flush span:
the World write RPC duration, network included (same attribute key as
world-vercel's per-request span so queries are layer-agnostic).
- workflow.stream.read.connect_ms on the workflow.stream.read span:
the world.streams.get await (read dispatch -> stream handle).
- new workflow.stream.close span: the close RPC round trip.
Bonus: measured at the World interface, these cover world-local and
world-postgres too, not just Vercel deployments.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* telemetry: emit read-completion span (total duration, chunks, bytes)
Completes the read-side picture: workflow.stream.read.complete is
back-dated to the read dispatch so its duration is the total read, with
chunk/byte counts for throughput. Cancelled reads emit nothing.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* chore: drop DEBUG from turbopack workbench; tighten changeset
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* telemetry: cover createReconnectingFramedStream in read telemetry
Ordinary serialized streams read through createReconnectingFramedStream
(which calls world.streams.get directly), so connect_ms / ttfc /
read.complete never fired for that path — only WorkflowServerReadableStream
was instrumented. Wire the same helpers into the framed reader: first-
connect duration, first-frame TTFC, and completion totals — plus
workflow.stream.read.reconnects, which only this path can know.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* cli: show run region in inspect output via World.regionForRunId
Adds an optional reverse-lookup hook to the World interface —
regionForRunId(runId): string | null — so tooling can display a run's
region generically. Worlds without a regional dimension simply omit
the hook and no region output appears.
- @workflow/world: new optional interface member (documented: must not
throw; null = undeterminable)
- @workflow/world-vercel: implements it from the run-ID region tag
(tagged -> embedded region, untagged legacy -> default region,
malformed -> null)
- @workflow/cli: 'workflow inspect runs' gains a region column
(between workflowName and status) and 'workflow inspect run <id>'
a region property, in both table and JSON output — only when the
world defines the hook
* Generalize the inspect hook: World.describeRun display fields
Replaces regionForRunId on the World interface with describeRun, per
review: worlds may want to expose more than a region, and the
information need not be encoded in the run ID — describeRun receives
the run entity itself (loosely typed, mirroring createRunId), so a
world can derive fields from executionContext or any other property.
Each returned key becomes an inspect column/property; null values are
preserved in structured output ('applicable but undeterminable' vs.
the hook being absent entirely).
- world-vercel: describeRun returns { region } decoded from the run
ID tag (regionForRunId stays exported as a utility); entities
without a usable runId contribute nothing
- CLI listing: columns come from the union of keys the world returns
for the page, inserted before status; both analytics and storage
paths; detached call site binds this
- CLI showRun: merges the world fields into detail/JSON output via a
method-style call (preserves this), keeping nulls
- tests: field merging (multi-key), null preservation in JSON, hook
absent, and world-vercel describeRun coverage incl. no-runId
entities
* cli: evaluate describeRun defensively
Per review: the World interface says describeRun is pure and must not
throw, but it is an external extension point and the CLI should not
trust that. New safeWorldFields helper, used by both the listing and
showRun paths:
- a throwing implementation contributes no fields instead of crashing
the inspect command
- keys that already exist on the run row are dropped, so a world can
never overwrite canonical fields (status, runId, ...) in output
Tests: canonical fields survive a clobbering describeRun (extra keys
still merged); a throwing describeRun leaves rows untouched and the
command succeeds.
* Allow async describeRun implementations
Per review: widening a sync signature to async later would break every
consumer, while accepting sync-or-async from day one is free — sync
implementations (like world-vercel's) remain valid, and consumers
simply await, which handles both. The performance intent lives on as
documented guidance: the hook is called once per displayed run, so
implementations should stay cheap and avoid I/O; the CLI evaluates a
page's rows concurrently so an async world costs one await per page,
not per row. Promise rejections get the same treatment as throws:
no fields, never a crash.
* Update packages/world/src/interfaces.ts
Co-authored-by: Peter Wielander <mittgfu@gmail.com>
Signed-off-by: Nathan Rajlich <n@n8.io>
---------
Signed-off-by: Nathan Rajlich <n@n8.io>
Co-authored-by: Peter Wielander <mittgfu@gmail.com>
* Add attribute discovery and filtering to world.analytics
- analytics.attributes.list() — distinct attribute keys observed on runs
in the window, with run counts and first/last seen timestamps
(GET /v2/analytics/attributes).
- analytics.attributes.listValues({ key }) — distinct values for one key
with latest-write-wins run counts (GET /v2/analytics/attributes/values).
- analytics.runs.list({ attributes: { key: value } }) — restrict the runs
listing to runs whose latest attribute snapshot matches every provided
pair (JSON-encoded query param, up to 8 pairs; $-prefixed framework
keys allowed in read filters).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Shorten changeset; document world.analytics in the World SDK reference
The analytics namespace was previously undocumented. Adds a full
reference page (runs, attributes, steps/events/hooks/waits, lookback
windows and pageInfo), links it from the World SDK index and meta, and
replaces the stale 'in the future you'll be able to search by
attributes' line in the attributes guide with a filtering section.
Extends the docs-typecheck ambient world global with the analytics
namespace so reference snippets typecheck.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Drop analytics.attributes.listValues
Not a derived requirement: the agent-runs UI filters by known constant
values and reads per-run values via batch attribute fetches; it never
enumerates distinct values for a key.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Feature-detect world.analytics in the attributes guide example
The snippet dereferenced the optional analytics namespace without the
runtime check its skip-typecheck annotation claimed, and would throw on
local/Postgres/custom Worlds. Guard it and drop the annotation — the
block is now genuinely typechecked.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Fix stale 'querying attributes not available' bullet in the guide
The Behavior list still claimed a query API was only planned, directly
contradicting the Searching and filtering section above it. State what
is implemented: attributes are readable on run objects everywhere, and
key discovery / key=value run filtering are available through the
optional Analytics API.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Surface the analytics namespace on the API reference index pages
The Analytics page sits under workflow/runtime > World SDK, but neither
the API reference index card, the workflow/runtime World SDK card, nor
the World SDK overview mentioned analytics — making the new reference
effectively undiscoverable from /docs/api-reference. Mention it at each
level of the path.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Bump changeset to minor
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* [world-vercel] Add /run-id sub-export with tagged ULID encode/decode
Encodes a tag bit, 5-bit version, and 6-bit Vercel region ID into a
ULID-shaped string used for workflow run IDs. Tagged values remain
valid 26-char Crockford-Base32 ULIDs so they still sort and round-trip
through any system that accepts ULIDs.
* [world-vercel] Add string-value assertions to run-id tests
Add exact-string expectations for encoded outputs at known inputs,
covering the default region/version pair, numeric region IDs, version
overrides, boundary values (all-zero, all-max), the dirty-input
overwrite case, and the lexicographic-order checks. Also adds an
explicit byte-array expectation for the canonical ULID-spec example
string and an additional first-char-range coverage test for isTagged.
* [world-vercel] Remove internal-repo reference from regions doc comment
* [world-vercel] Address PR review feedback on run-id sub-export
- isTaggedString now fully validates the input as a 26-char Crockford
Base32 ULID (delegating to ulidToBytes) instead of only inspecting
the first character. This fixes false positives on inputs like
'4UUUU...' that have a valid tag-bit position but invalid chars
later in the string.
- isTagged() now accepts `unknown` to match its documented behavior
of safely rejecting non-string inputs without requiring callers to
cast.
- Introduce `RegionKey` for the full set of keys including 'unknown',
and narrow `RegionCode` to `Exclude<RegionKey, 'unknown'>` so the
return type of `lookupRegion` and the `DecodedRunId.region` field
accurately reflect that 'unknown' is never produced. Updates
`encode` to reject 'unknown' as a region code string at runtime
(callers wanting the unknown sentinel should pass numeric 0).
* [world] [core] [world-vercel] Add World.createRunId() and region-aware queue routing
- @workflow/world: add optional createRunId(input?) to the World
interface so worlds can mint run IDs with embedded metadata, and
add an optional 'region' field to QueueOptions for per-message
routing hints.
- @workflow/core: start() now delegates run ID generation to
world.createRunId() when defined (falling back to a monotonic
ULID otherwise), and accepts a new 'runIdInput' option that is
forwarded verbatim to createRunId. When runIdInput.region is a
string, it is also threaded onto the queue options so the initial
workflow message is dispatched to the matching region.
- @workflow/world-vercel: implement createRunId() to mint
region-tagged ULIDs, preferring an explicit runIdInput.region and
falling back to the VERCEL_REGION env var. The queue now resolves
its destination region from (in order): an explicit opts.region,
the region embedded in the payload's tagged run ID, the
VERCEL_REGION env var, and finally a hardcoded 'iad1' default.
This replaces the previous unconditional 'iad1' region passed to
the @vercel/queue client.
Monotonicity within a process is preserved by tracking the last
emitted run ID and bumping the bit immediately above the 11-bit
metadata window when a same-ms collision would otherwise occur,
then re-stamping the requested region/version on top so metadata
remains stable.
* [core] [world] [world-vercel] Pass full StartOptions to World.createRunId
Drop the dedicated 'runIdInput' field on StartOptions and forward the
entire options bag to world.createRunId() instead. This keeps the
public API surface smaller and lets each World pick the fields it
recognises (e.g. world-vercel reads 'region'). The top-level 'region'
option remains on StartOptionsBase and is also threaded onto the
queue's per-call region opt when set.
* Address review feedback: doc fixes and deterministic same-ms tests
- Document the final iad1 fallback in QueueOptions.region (world)
- Correct the World.createRunId doc: start() always passes an object
- Fix the clientOptions comment: the handler client omits region and
relies on SDK auto-detection + the ce-vqsregion header for acks
- Fix a misleading QueueClient-construction comment in queue.test.ts
- Freeze time in the same-ms monotonicity test so it deterministically
exercises the intended path, and add a test covering the
bump-above-metadata fallback when the region changes mid-millisecond
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* test: keep workflow-server override rewrite-compatible
Export WORKFLOW_SERVER_URL_OVERRIDE while keeping the one-line const shape
that workflow-server's cross-repo e2e test automation rewrites. Update
world-vercel tests to import that exported value for mock origins and URL
expectations instead of duplicating the temporary preview URL.
* fix(world): clear region tag bit before ULID timestamp validation
Region-tagged run IDs set the high bit of the ULID timestamp byte. The
shared world timestamp validator used raw decodeTime(), so current tagged
run IDs appeared thousands of years in the future and were rejected before
reaching workflow-server. Clear the tag bit before decoding, matching the
workflow-server behavior, and cover tagged IDs in tests.
* fix(world-vercel): validate tagged runId timestamps via run-id decode
Keep @workflow/world's ULID helpers generic; they should not know about
world-vercel's region-tagged run ID layout. Instead, world-vercel decodes
its tagged runId to the original ULID before using the shared timestamp
validator for run_created events. Add a world-vercel regression test that a
current sfo1-tagged runId passes validation.
* fix(world-vercel): default run ID region to iad1 instead of unknown
When neither an explicit region option nor VERCEL_REGION is available,
createRunId minted a tagged ULID with the unknown (0) region sentinel,
producing the tagged: true, region: null state. The server already
resolves unknown/untagged runs to DEFAULT_VERCEL_REGION (iad1), so mint
a concrete iad1 tag instead, keeping every run ID self-describing and
routable.
* test(e2e): use verbose reporter + per-test start heartbeat
The default vitest reporter buffers per-file output, so a stalling e2e
test produces no output until its timeout — making CI look like a silent
30-minute hang. Switch the e2e CI invocations to the verbose reporter
(prints each test result as it completes) and emit a '[e2e] ▶ start:'
heartbeat to stdout at the start of every test (bypassing vitest's console
buffering) so a stuck test is immediately identifiable in the live CI log.
* test(world-vercel): point WORKFLOW_SERVER_URL_OVERRIDE at combined 527+529 preview
Temporarily target the workflow-server combined-527-529-preview deployment,
which bundles platform-directed multi-region routing (vercel/workflow-server#527,
incl. the iad1 hook pin) and durable stream state (vercel/workflow-server#529),
so e2e can validate the full multi-region path end-to-end. Revert to empty on main.
* fix(core): region-tag the health-check correlationId
The health-check response is delivered over a Redis stream whose name (and
synthetic run ID) embed the correlationId. Under platform-directed routing
the responding endpoint and the polling reader can be served from different
physical regions; Redis is physical-region-local, so the correlationId must
carry the region for both sides to resolve the same backend.
Generate the correlationId via world.createRunId() (a region-tagged ULID)
when the world provides it, falling back to a plain ULID for worlds that
don't tag IDs (e.g. local, single-region). The synthetic wrun_hc_<id> run ID
then carries the region; workflow-server's region middleware decodes it.
* Address review feedback: validate region overrides, reset server override
- queue: validate opts.region and VERCEL_REGION against the known region
table before routing, ignoring unrecognised codes so a bad override
can't clobber the payload-derived region (Copilot)
- add isKnownRegionCode() runtime guard to run-id/regions
- reset WORKFLOW_SERVER_URL_OVERRIDE to '' (must be empty on main)
- fold the within-PR iad1-default changeset into the main world-vercel
changeset and delete it (review)
- start.test: declare specVersion on createRunId mock worlds now that
the merged world-compatibility check requires it
- cover the new region-validation fall-through paths in queue.test
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* test(world-vercel): point WORKFLOW_SERVER_URL_OVERRIDE at wave-1 multi-region preview
BRANCH-ONLY — revert the override to '' before merge (lint enforces).
Points this PR's e2e/benchmark runs at the wave-1 multi-region
workflow-server preview (vercel/workflow-server#590: iad1+sfo1+fra1
serving, staging data backends) so region-tagged runs are validated
against real multi-region serving end-to-end.
Also makes the unit-test mock origins in events-v4.test.ts and
trace-propagation.test.ts override-aware (same pattern the rest of
the file and utils.test.ts already use), so the suite passes whether
or not the override is set — these two files were the only spots
hardcoding https://vercel-workflow.com.
* test(e2e): Vercel multi-region suite for start()'s region option
Adds a dedicated e2e suite validating @workflow/world-vercel region
routing end to end, run as its own CI job (e2e-vercel-multi-region)
against the nextjs-turbopack workbench only — deliberately separate
from e2e.test.ts, which runs as a matrix across all worlds/frameworks
where Vercel-specific multi-region behavior doesn't apply.
- workbench/nextjs-turbopack/vercel.json: deploy to iad1+sfo1+fra1 so
region-routed flow messages have a function to land on in each region.
- workflows/99_e2e.ts: regionProbeWorkflow returns the VERCEL_REGION
observed by both the workflow and a step, so tests can assert the run
EXECUTED in the intended region (not just that it was tagged).
- packages/core/e2e/e2e-region.test.ts: per-region cases assert
1) start(..., { region }) mints a region-tagged run ID (decoded via
@workflow/world-vercel/run-id),
2) the workflow + step both observed VERCEL_REGION === region,
3) the server reports the run completed;
plus a concurrent all-regions case guarding against cross-region
misrouting under simultaneous multi-region traffic. Skips on local
deployments.
- .github/workflows/tests.yml: new e2e-vercel-multi-region job
mirroring e2e-vercel-prod's env/deployment-wait, running only the
new suite.
* test(e2e): start region probes in-function; fix getWorld await
The first multi-region CI run surfaced two issues:
1. sfo1/fra1-tagged runs executed in iad1. The suite started runs from
the external test process, which uses the api.vercel.com token proxy
— and the proxy's queues path forwards every send to the region-less
VQS host (the world's proxy-mode resolveBaseUrl ignores the region
argument, and the proxy's x-vercel-vqs-api-url escape hatch only
allowlists vqs-server-*.vercel.sh preview hosts). Production traffic
publishes IN-FUNCTION (direct regional queue routing), so the suite
now triggers start() through a new workbench route
(/api/e2e-region-start) and rehydrates the run with getRun() —
testing the path production actually takes. Proxy-mode regional
queue routing is a known gap to address separately in api-workflow.
2. TypeError on world.runs.get: getWorld() is async and was called
without await.
* test(e2e): cover explicit and implicit region starts in the multi-region suite
With regional VQS routing now working through the api.vercel.com proxy
(vercel/api#79056 + #2789 + this branch's per-send region resolution),
the suite covers both start configurations, asserting the same three
properties for each (region-tagged run ID, execution in the intended
region via VERCEL_REGION echoed in the return value, server-side
completion):
1. EXPLICIT: start(..., { region }) called directly in the vitest
runner — publishes through the token proxy, per-send region carried
by x-vercel-queue-region. Restores the direct-start shape the suite
had originally, plus the concurrent all-regions case.
2. IMPLICIT: dedicated per-region workbench routes
(/api/e2e-region-implicit/{iad1,sfo1,fra1}), each pinned to a single
region via a per-function 'regions' entry in the workbench
vercel.json, calling start() with NO region option — createRunId
derives the tag from the minting function's VERCEL_REGION. The test
also asserts the route reported executing in its pinned region, so
the implicit-tagging assertion can't pass vacuously.
Replaces the interim /api/e2e-region-start route (explicit region via
request body), which existed to work around the pre-#79056 proxy gap.
* Revert WORKFLOW_SERVER_URL_OVERRIDE to '' — wave-1 multi-region serving is in production
workflow-server#590 (iad1+sfo1+fra1 serving) merged and deployed to
production and the e2e backend, so this branch's e2e/benchmark runs no
longer need to target the wave-1 preview. Restores the empty override
the No Test Overrides lint job enforces for merge.
The override-aware unit-test origins (events-v4/trace-propagation)
stay — they are correct under any override value.
* test(e2e): cross-region stream visibility (iad1 writer, sfo1 reader)
Regression coverage for a backend bug that made cross-region stream
reads report zero chunks on IN-PROGRESS streams (completed streams were
unaffected), which forced the multi-region serving rollback.
The new case exercises exactly that geometry:
- crossRegionStreamWorkflow (99_e2e.ts) writes N chunks to the default
output stream, then holds the stream OPEN for 45s before closing —
the in-progress window is the point, since completed streams are the
easy case.
- The e2e starts it with region iad1, waits (same-region, via the
api.vercel.com proxy) until all chunks are written, asserts the run
is still 'running', then reads through a new sfo1-pinned workbench
route (/api/e2e-stream-read/sfo1) that returns getTailIndex() plus
its VERCEL_REGION. The reader's region served none of the stream's
writes, so the reported chunk count must come from the backend's
cross-region stream metadata. The test fails loudly if the route
isn't actually executing in sfo1.
Also bumps the explicit-region test timeout to 120s: the first case in
the file absorbs every cold start at once (fresh workbench instances
in up to three regions plus a cold backend preview) and was observed
just over the 60s default.
BRANCH-ONLY (revert before merge, lint enforces):
WORKFLOW_SERVER_URL_OVERRIDE points at a multi-region backend preview
that includes the fix, so this validates cross-region stream
visibility end-to-end before multi-region serving is re-enabled.
* test(e2e): extend multi-region suite to all 19 provisioned regions
Points the suite at an all-regions backend preview and widens coverage
from the wave-1 trio to every provisioned region:
- Explicit path: a single concurrent all-regions case starts one
tagged run per region (one shared cold-start window instead of 19
sequential ones) and aggregates per-region failures so a single
region's breakage reports alongside the full picture. The trio keeps
its detailed per-region cases and the 9-way concurrent-isolation
case.
- Implicit path: workbench gains a region-pinned
/api/e2e-region-implicit/<region> route per provisioned region (19
total, shared handler), the workbench itself now deploys to all of
them, and the test.each covers the full set with per-case timeouts
for regional cold starts.
- Multi-region CI job timeout 20m -> 35m for the sequential implicit
cases.
BRANCH-ONLY (revert before merge, lint enforces):
WORKFLOW_SERVER_URL_OVERRIDE now targets the all-regions backend
preview instead of the previous (stale, since-merged) fix preview.
* test(e2e): tolerate geo-adjacent execution of queue callbacks
The first all-regions run surfaced a subtle execution-locality
behavior: queue delivery is guaranteed to the tagged region's
dataplane and the delivery callback egresses from that region, but the
consumer invocation's execution region is chosen by where that
callback enters Vercel's edge — and adjacent regions can geo-resolve
to each other's functions. Observed live: kix1-tagged runs (callback
egressing from Osaka) deterministically executing in hnd1/Tokyo on
both the explicit and implicit paths, with tagging, data placement,
and completion all still strictly kix1.
expectRunInRegion now asserts execution lands in the tagged region OR
one of its geographic neighbors (EXECUTION_ADJACENCY), while run-ID
tagging and server-side completion remain strictly the requested
region. Gross misrouting (e.g. kix1 -> iad1) still fails.
* Revert WORKFLOW_SERVER_URL_OVERRIDE to '' — all-regions serving is in production
The all-regions workflow-server rollout is deployed and serving
production traffic from every Vercel region, so this branch's e2e no
longer needs to target a branch preview. Restores the empty override
the No Test Overrides lint enforces for merge.
With this the PR is complete: region-tagged run IDs, region-aware
queue routing, and the multi-region e2e suite (explicit + implicit +
all-regions + cross-region streams) all validate against the
production-default backends.
* docs: fix three stale comments flagged in review
- start.ts: StartOptionsBase.region fallback is iad1, not the unknown
sentinel (createRunId always mints a concrete routable region)
- queue.ts: example used a nonexistent start({ runIdInput }) API; the
real option is start({ region })
- events.ts: decode() clears only the tag bit (top bit of the 48-bit
timestamp field) — it does not restore the original untagged ULID;
reword to say what actually matters for timestamp validation
* test(e2e): cover hook resolve/resume for runs owned by non-iad1 regions
Hooks are resolved by opaque token, which carries no region hint, so
lookup and resume must work regardless of which region owns the run's
data. Exercises the full follow-up-message path on sfo1- and
fra1-tagged runs: create inside the workflow, resolve by token from
the test process, resume twice sequentially, and assert payload order
and completion.
Regression coverage for the failure mode where the first message to a
hook-driven app on a non-iad1 run worked but every follow-up failed
with 'Hook not found'.
---------
Co-authored-by: Peter Wielander <peter.wielander@vercel.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* build: declare typescript (catalog:) in every package that runs tsc
Twenty packages invoke tsc in their build/typecheck scripts without
declaring a typescript dependency, resolving whatever tsc pnpm happens
to leave reachable. That broke locally after the TypeScript 6 upgrade
(#2700): base.json now uses the TS6-only 'types': ['*'] wildcard, and
worktrees carrying pre-upgrade node_modules/.bin/tsc shims (orphaned
typescript@5.9.3 bins that pnpm never refreshes for an undeclared
dependency) fail with TS2688 'Cannot find type definition file for *'.
Declaring 'typescript': 'catalog:' (the convention nest already
follows) makes pnpm own each package's tsc bin, so version upgrades
refresh the shims and this staleness class cannot recur. Packages
without tsc in their scripts are left unchanged.
Full pnpm build: 27/27 tasks green.
* Address review: drop duplicate zod devDep; regenerate lockfile minimally
- packages/world listed zod in both dependencies and devDependencies
(pre-existing on main, surfaced by the devDependencies sort) — keep
the runtime dependency only.
- Regenerate pnpm-lock.yaml from a pristine main baseline with
--lockfile-only (a clean-main run produces zero diff, so main has no
drift). Remaining non-typescript changes are mechanical consequences
of the change itself: typescript is an (optional) peer of several
tooling dependencies, so declaring it in 20 importers creates new
peer-resolution snapshot variants and prunes the now-orphaned old
ones; plus one radix-ui 1.6.1->1.6.2 refresh in docs caused by its
floating 'latest' specifier.
- Validated: pnpm install --frozen-lockfile succeeds; full build 27/27.
* fix(world): parse timezone-naive analytics timestamps as UTC
ClickHouse-backed analytics endpoints serialize DateTime64 values as
timezone-naive strings ('2026-07-13 17:09:11.593'), UTC by convention.
The analytics schemas coerced them with z.coerce.date(), i.e.
new Date(value), which interprets naive strings in the process's LOCAL
timezone. That is only correct when the process runs in UTC — the
deployed observability web app's server actions, which is why the web
UI appears unaffected — and wrong by the local UTC offset everywhere
else: the CLI on a laptop showed runs 'starting in about 7 hours'
(PDT), and 'workflow web --localUi' shares the bug.
Replace the coercion with a preprocess that normalizes naive datetime
strings to an explicit Z designator before parsing. Values already
carrying timezone info (Z or ±hh:mm) and non-string inputs (Date,
epoch) pass through unchanged.
Tests pin the contract and were verified under TZ=America/Los_Angeles
and TZ=Asia/Tokyo (3 of 4 fail against the old coercion in PDT; plain
UTC CI cannot distinguish the two, which is how this shipped).
* Potential fix for pull request finding
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Signed-off-by: Nathan Rajlich <n@n8.io>
---------
Signed-off-by: Nathan Rajlich <n@n8.io>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
* telemetry: emit client-observed workflow.stream.write span per flush batch
Complements the existing workflow.stream.read TTFC span: each flushed
batch emits a back-dated CLIENT span covering the app-perceived write
latency (buffer dwell + RPC), with buffer_dwell_ms / chunks / bytes
attributes so client-side batching cost (flush timer, turbo run-ready
barrier) can be told apart from network/server time. Failed flushes
keep the batch's original t0 so a retried batch reports its full dwell.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* chore: tighten changeset
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* telemetry: rename flush span to workflow.stream.flush; DEBUG-log world-vercel OTEL load failure
- workflow.stream.write is taken by world-vercel's per-request RPC span
(#2857, chunk_rtt); the per-batch flush span gets its own name so the
two stay distinguishable in trace queries. Attributes move to
workflow.stream.flush.{buffer_dwell_ms,chunks,bytes}, operation=flush.
- world-vercel's @opentelemetry/api load failure was silently latched as
null, which also swallows bundler/resolution failures in apps that DO
register a tracer (observed in production: workbench apps emit core
spans but none of world-vercel's). Log the reason under
DEBUG=workflow:* so the failure mode is diagnosable.
- Document workflow.stream.flush in the tracing docs.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* chore: tighten changeset
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Pranay Prakash