Hook timeline bars were amber/yellow via RESOURCE_COLORS; passive
spans should be gray to match the event list icons and minimap.
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
* Give Metadata Token and Hook ID copy + truncation
Add token to the copyable metadata attributes set and constrain
copyable key-value rows so MiddleTruncate can shrink long IDs.
* Remove AttributePanel copy unit tests
The Metadata Token/Hook ID copy change is small enough that the
dedicated panel render tests are unnecessary.
---------
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
* fix(nitro): support React Router Vite builds
* refactor(nitro): simplify React Router cleanup
* docs(react-router): specify cleanup version
* fix(nitro): close temporary Vite servers
The recoverActiveRuns factory option had no environment variable, so
disabling startup re-enqueueing of pending/running runs required a custom
world module via WORKFLOW_TARGET_WORLD. Wire an env fallback
(0/false disables, 1/true enables, explicit factory option wins) and
document it in the worlds configuration reference and local world guide.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* fix(deps): dedupe @opentelemetry/api to a single workspace instance
The lockfile resolved both 1.9.0 and 1.9.1, so the copy that registers
the tracer provider (via @vercel/otel in the app) and the copy a package
imports could differ. The API's global-registration version check rejects
a consumer newer than the registered copy and silently hands back a noop
tracer — which is why world-vercel's spans (workflow.stream.write/
chunk_rtt, read.connect, its http spans) never reached Datadog from
deployed apps while core's spans flowed in the same process. Root-caused
via the DEBUG=workflow:* run on #2900: import succeeds, no warn, spans
dropped.
Pin a single version via a workspace override so every bundle shares one
API instance.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* debug: one-shot OTEL runtime diagnostic in core + world-vercel; DEBUG on turbopack workbench
The dedupe alone did not restore world-vercel span emission (verified on
this PR's own preview: stream traffic flowed, zero workflow.stream.write
spans). Under DEBUG=workflow:*, both packages now log once how their
module instance of @opentelemetry/api sees the world — global
registration version, provider/delegate/tracer/probe constructor names,
and whether a probe span is recording. Diffing the core line (spans work)
against the world-vercel line (spans dropped) in one deployment's logs
pinpoints the divergence.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* debug: log span identity for named world-vercel spans; namespace otel probes per package
Diag round 1 showed world-vercel's tracer records and instrumentedFetch
handles the stream PUTs, yet the named spans are unfindable in the
backend. Round 2: log traceId/spanId/isRecording for every named
instrumentedFetch span under DEBUG so export can be checked for a
specific span id, and split the probe span names (.core /
.world_vercel) so per-package export is attributable.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* telemetry: emit stream RPC latencies from core (chunk_rtt, connect_ms, close span)
world-vercel's instrumentedFetch spans never export from deployed apps
(root cause still open — see PR discussion), so the operationally
needed client-side latency signals move one layer up to core, whose
spans are proven to export:
- workflow.stream.write.chunk_rtt on the workflow.stream.flush span:
the World write RPC duration, network included (same attribute key as
world-vercel's per-request span so queries are layer-agnostic).
- workflow.stream.read.connect_ms on the workflow.stream.read span:
the world.streams.get await (read dispatch -> stream handle).
- new workflow.stream.close span: the close RPC round trip.
Bonus: measured at the World interface, these cover world-local and
world-postgres too, not just Vercel deployments.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* telemetry: emit read-completion span (total duration, chunks, bytes)
Completes the read-side picture: workflow.stream.read.complete is
back-dated to the read dispatch so its duration is the total read, with
chunk/byte counts for throughput. Cancelled reads emit nothing.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* chore: drop DEBUG from turbopack workbench; tighten changeset
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* telemetry: cover createReconnectingFramedStream in read telemetry
Ordinary serialized streams read through createReconnectingFramedStream
(which calls world.streams.get directly), so connect_ms / ttfc /
read.complete never fired for that path — only WorkflowServerReadableStream
was instrumented. Wire the same helpers into the framed reader: first-
connect duration, first-frame TTFC, and completion totals — plus
workflow.stream.read.reconnects, which only this path can know.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
- analytics: attribute the serving path to the Vercel observability data
pipeline instead of naming the ClickHouse store, drop the Vercel-only
"Plan-bounded lookback" bullet, and mark the pageInfo lookback ceiling
and upgradeAvailable comments as Vercel-specific.
- sidebar: folder rows without an index page render as <button>s, which
don't stretch to the row width like the <a> folder links, leaving their
ms-auto chevron hugging the label. Stretch those triggers to full width
so every chevron sits at the end of the row.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Audited every vercel.com link in docs/content (20 unique URLs,
HTTP-validated including anchor fragments):
- project-configuration#regions (2x): the #regions anchor no longer
exists on that page — content moved to the vercel-json subpage; now
links project-configuration/vercel-json#regions
- gateway/api-reference/overview (2x): hard 404; the AI Gateway docs
restructured — 'get an API key' context now points at
ai-gateway/authentication
- observability/otel-overview (1x): redirects to
tracing/instrumentation; link the final URL
- docs/workflow and docs/workflow/python (8x): redirect to the plural
docs/workflows paths; link the final URLs (also drops a redundant
?language=py param that the redirect discards)
All other links (queues, queues/pricing + anchors, plans/hobby,
limits, regions, sandbox, workflows/pricing#storage-retention,
cli/project-linking, audit-log, home, help, blog) verified 200 with
live anchors.
* cli: show run region in inspect output via World.regionForRunId
Adds an optional reverse-lookup hook to the World interface —
regionForRunId(runId): string | null — so tooling can display a run's
region generically. Worlds without a regional dimension simply omit
the hook and no region output appears.
- @workflow/world: new optional interface member (documented: must not
throw; null = undeterminable)
- @workflow/world-vercel: implements it from the run-ID region tag
(tagged -> embedded region, untagged legacy -> default region,
malformed -> null)
- @workflow/cli: 'workflow inspect runs' gains a region column
(between workflowName and status) and 'workflow inspect run <id>'
a region property, in both table and JSON output — only when the
world defines the hook
* Generalize the inspect hook: World.describeRun display fields
Replaces regionForRunId on the World interface with describeRun, per
review: worlds may want to expose more than a region, and the
information need not be encoded in the run ID — describeRun receives
the run entity itself (loosely typed, mirroring createRunId), so a
world can derive fields from executionContext or any other property.
Each returned key becomes an inspect column/property; null values are
preserved in structured output ('applicable but undeterminable' vs.
the hook being absent entirely).
- world-vercel: describeRun returns { region } decoded from the run
ID tag (regionForRunId stays exported as a utility); entities
without a usable runId contribute nothing
- CLI listing: columns come from the union of keys the world returns
for the page, inserted before status; both analytics and storage
paths; detached call site binds this
- CLI showRun: merges the world fields into detail/JSON output via a
method-style call (preserves this), keeping nulls
- tests: field merging (multi-key), null preservation in JSON, hook
absent, and world-vercel describeRun coverage incl. no-runId
entities
* cli: evaluate describeRun defensively
Per review: the World interface says describeRun is pure and must not
throw, but it is an external extension point and the CLI should not
trust that. New safeWorldFields helper, used by both the listing and
showRun paths:
- a throwing implementation contributes no fields instead of crashing
the inspect command
- keys that already exist on the run row are dropped, so a world can
never overwrite canonical fields (status, runId, ...) in output
Tests: canonical fields survive a clobbering describeRun (extra keys
still merged); a throwing describeRun leaves rows untouched and the
command succeeds.
* Allow async describeRun implementations
Per review: widening a sync signature to async later would break every
consumer, while accepting sync-or-async from day one is free — sync
implementations (like world-vercel's) remain valid, and consumers
simply await, which handles both. The performance intent lives on as
documented guidance: the hook is called once per displayed run, so
implementations should stay cheap and avoid I/O; the CLI evaluates a
page's rows concurrently so an async world costs one await per page,
not per row. Promise rejections get the same treatment as throws:
no fields, never a crash.
* Update packages/world/src/interfaces.ts
Co-authored-by: Peter Wielander <mittgfu@gmail.com>
Signed-off-by: Nathan Rajlich <n@n8.io>
---------
Signed-off-by: Nathan Rajlich <n@n8.io>
Co-authored-by: Peter Wielander <mittgfu@gmail.com>
* Add attribute discovery and filtering to world.analytics
- analytics.attributes.list() — distinct attribute keys observed on runs
in the window, with run counts and first/last seen timestamps
(GET /v2/analytics/attributes).
- analytics.attributes.listValues({ key }) — distinct values for one key
with latest-write-wins run counts (GET /v2/analytics/attributes/values).
- analytics.runs.list({ attributes: { key: value } }) — restrict the runs
listing to runs whose latest attribute snapshot matches every provided
pair (JSON-encoded query param, up to 8 pairs; $-prefixed framework
keys allowed in read filters).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Shorten changeset; document world.analytics in the World SDK reference
The analytics namespace was previously undocumented. Adds a full
reference page (runs, attributes, steps/events/hooks/waits, lookback
windows and pageInfo), links it from the World SDK index and meta, and
replaces the stale 'in the future you'll be able to search by
attributes' line in the attributes guide with a filtering section.
Extends the docs-typecheck ambient world global with the analytics
namespace so reference snippets typecheck.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Drop analytics.attributes.listValues
Not a derived requirement: the agent-runs UI filters by known constant
values and reads per-run values via batch attribute fetches; it never
enumerates distinct values for a key.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Feature-detect world.analytics in the attributes guide example
The snippet dereferenced the optional analytics namespace without the
runtime check its skip-typecheck annotation claimed, and would throw on
local/Postgres/custom Worlds. Guard it and drop the annotation — the
block is now genuinely typechecked.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Fix stale 'querying attributes not available' bullet in the guide
The Behavior list still claimed a query API was only planned, directly
contradicting the Searching and filtering section above it. State what
is implemented: attributes are readable on run objects everywhere, and
key discovery / key=value run filtering are available through the
optional Analytics API.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Surface the analytics namespace on the API reference index pages
The Analytics page sits under workflow/runtime > World SDK, but neither
the API reference index card, the workflow/runtime World SDK card, nor
the World SDK overview mentioned analytics — making the new reference
effectively undiscoverable from /docs/api-reference. Mention it at each
level of the path.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Bump changeset to minor
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* docs: document multi-region support in the Vercel World
The Vercel World Limitations section still described the backend as
iad1-only, which is no longer true — it now runs in every Vercel
Function region, with each run pinned to a single region at creation.
- Replace the single-region / iad1 data-residency limitations with a
Multi-region section: automatic pinning from the creating function's
region (single- and multi-region deployments), explicit per-run
selection via start(..., { region }), routing semantics for readers,
and 4.x/pre-existing-run behavior (iad1, no migration).
- Version requirement called out as workflow 5.x beta with a TODO to
pin the exact minimum version once released; 4.x will not support
region pinning.
- Limitations now lists the one that remains: a run's region is fixed
at creation (no migration).
- /docs/deploying: add a Multi-region bullet to the Vercel World
feature list, deferring details to /worlds/vercel.
* docs: clarify region option scope — data + queue dispatch, not code placement
The start({ region }) option pins where the run's data is stored and
where its queue messages dispatch from; it does not deploy code.
Execution happens in the regions the application is deployed to, so
region-local execution requires deploying the app to the desired
region (vercel.json regions or the project's Function Regions
setting). Adds a warning callout to the explicit-selection section and
tightens the intro to only claim region-local execution for the
automatic case.
* docs(v4): point default-version pages at the v5 multi-region docs
The v4 docs are the default version, so readers landing on
/worlds/vercel (which renders v4 content) or /docs/deploying would
never discover multi-region exists behind the v5 switcher.
- v4 Vercel World Limitations: add a callout that multi-region ships
with workflow 5.x (deep link to the v5 Multi-region section), and
scope the iad1-only statements to the 4.x release line — they remain
true there; 4.x will not gain multi-region.
- v4 /docs/deploying: matching callout after the Vercel World blurb.
* docs(v5): surface the world pages in the Deploying sidebar
The v5 world pages (vercel-world, postgres-world, local-world) were
orphaned: the Deploying sidebar only listed 'Building a World', and
the dedicated /worlds/:id route renders v4 content — so there was no
navigation path to the v5 Vercel World page (and its new Multi-region
section) at all.
- v5/deploying/meta.json: add the world folder to the sidebar
(Worlds group between the section index and Building a World)
- world/meta.json: group title 'World' -> 'Worlds'
- v5 /docs/deploying callout: link to the v5 world page (with a
multi-region deep link) instead of /worlds/vercel, which silently
drops the reader into v4-rendered content
Verified via dev server: the Worlds group (all three pages) renders in
the v5 sidebar and the Multi-region section renders on the page.
Note for docs owners: other v5 content still links to the versionless
/worlds/* routes, which render v4 content — probably worth a broader
pass or making /worlds version-aware.
* docs(v4): surface the world pages in the Deploying sidebar
Same fix as v5: the v4 Deploying sidebar only listed 'Building a
World', leaving the world pages reachable only via the /worlds/:id
routes with no sidebar path. Adds the Worlds group (Vercel World,
Postgres World, Local World) between the section index and Building a
World, and pluralizes the group title.
Verified via dev server: the group renders, and the sidebar items land
on /worlds/:id through the existing permanent redirects (same v4
content).
* docs: pin multi-region minimum version to workflow 5.0.0-beta.33
vercel/workflow#1981 merged; the first release carrying multi-region
support is 5.0.0-beta.33. Replace the '5.x (currently in beta)'
placeholders (and the TODO marker) with the exact minimum version on
all four touched pages (v5 world page + deploying bullet, v4
breadcrumb callouts).
* docs: note hook-token data residency in the multi-region section
Hook tokens carry no region information, so the token-to-run mapping
behind getHookByToken()/resumeHook() is currently stored in iad1 for
every run regardless of its region. Hook payloads are unaffected —
a received payload lands on the run's event log in the run's region
like all other run data. Noted as potentially becoming a
project-level setting in the future.
* docs: address review — execution-locality wording + dedupe limitation
- /docs/deploying multi-region bullet: 'execution' -> 'queuing'; the
region option pins data/queue/streams, while execution follows the
app's deployed regions (matching the v4 callout and the world page's
own warning callout)
- v5 world page: drop 'region is fixed for its entire lifetime' from
the Good to know bullet — the Limitations section already owns that
statement; the bullet now covers only routing semantics
* [world-vercel] Add /run-id sub-export with tagged ULID encode/decode
Encodes a tag bit, 5-bit version, and 6-bit Vercel region ID into a
ULID-shaped string used for workflow run IDs. Tagged values remain
valid 26-char Crockford-Base32 ULIDs so they still sort and round-trip
through any system that accepts ULIDs.
* [world-vercel] Add string-value assertions to run-id tests
Add exact-string expectations for encoded outputs at known inputs,
covering the default region/version pair, numeric region IDs, version
overrides, boundary values (all-zero, all-max), the dirty-input
overwrite case, and the lexicographic-order checks. Also adds an
explicit byte-array expectation for the canonical ULID-spec example
string and an additional first-char-range coverage test for isTagged.
* [world-vercel] Remove internal-repo reference from regions doc comment
* [world-vercel] Address PR review feedback on run-id sub-export
- isTaggedString now fully validates the input as a 26-char Crockford
Base32 ULID (delegating to ulidToBytes) instead of only inspecting
the first character. This fixes false positives on inputs like
'4UUUU...' that have a valid tag-bit position but invalid chars
later in the string.
- isTagged() now accepts `unknown` to match its documented behavior
of safely rejecting non-string inputs without requiring callers to
cast.
- Introduce `RegionKey` for the full set of keys including 'unknown',
and narrow `RegionCode` to `Exclude<RegionKey, 'unknown'>` so the
return type of `lookupRegion` and the `DecodedRunId.region` field
accurately reflect that 'unknown' is never produced. Updates
`encode` to reject 'unknown' as a region code string at runtime
(callers wanting the unknown sentinel should pass numeric 0).
* [world] [core] [world-vercel] Add World.createRunId() and region-aware queue routing
- @workflow/world: add optional createRunId(input?) to the World
interface so worlds can mint run IDs with embedded metadata, and
add an optional 'region' field to QueueOptions for per-message
routing hints.
- @workflow/core: start() now delegates run ID generation to
world.createRunId() when defined (falling back to a monotonic
ULID otherwise), and accepts a new 'runIdInput' option that is
forwarded verbatim to createRunId. When runIdInput.region is a
string, it is also threaded onto the queue options so the initial
workflow message is dispatched to the matching region.
- @workflow/world-vercel: implement createRunId() to mint
region-tagged ULIDs, preferring an explicit runIdInput.region and
falling back to the VERCEL_REGION env var. The queue now resolves
its destination region from (in order): an explicit opts.region,
the region embedded in the payload's tagged run ID, the
VERCEL_REGION env var, and finally a hardcoded 'iad1' default.
This replaces the previous unconditional 'iad1' region passed to
the @vercel/queue client.
Monotonicity within a process is preserved by tracking the last
emitted run ID and bumping the bit immediately above the 11-bit
metadata window when a same-ms collision would otherwise occur,
then re-stamping the requested region/version on top so metadata
remains stable.
* [core] [world] [world-vercel] Pass full StartOptions to World.createRunId
Drop the dedicated 'runIdInput' field on StartOptions and forward the
entire options bag to world.createRunId() instead. This keeps the
public API surface smaller and lets each World pick the fields it
recognises (e.g. world-vercel reads 'region'). The top-level 'region'
option remains on StartOptionsBase and is also threaded onto the
queue's per-call region opt when set.
* Address review feedback: doc fixes and deterministic same-ms tests
- Document the final iad1 fallback in QueueOptions.region (world)
- Correct the World.createRunId doc: start() always passes an object
- Fix the clientOptions comment: the handler client omits region and
relies on SDK auto-detection + the ce-vqsregion header for acks
- Fix a misleading QueueClient-construction comment in queue.test.ts
- Freeze time in the same-ms monotonicity test so it deterministically
exercises the intended path, and add a test covering the
bump-above-metadata fallback when the region changes mid-millisecond
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* test: keep workflow-server override rewrite-compatible
Export WORKFLOW_SERVER_URL_OVERRIDE while keeping the one-line const shape
that workflow-server's cross-repo e2e test automation rewrites. Update
world-vercel tests to import that exported value for mock origins and URL
expectations instead of duplicating the temporary preview URL.
* fix(world): clear region tag bit before ULID timestamp validation
Region-tagged run IDs set the high bit of the ULID timestamp byte. The
shared world timestamp validator used raw decodeTime(), so current tagged
run IDs appeared thousands of years in the future and were rejected before
reaching workflow-server. Clear the tag bit before decoding, matching the
workflow-server behavior, and cover tagged IDs in tests.
* fix(world-vercel): validate tagged runId timestamps via run-id decode
Keep @workflow/world's ULID helpers generic; they should not know about
world-vercel's region-tagged run ID layout. Instead, world-vercel decodes
its tagged runId to the original ULID before using the shared timestamp
validator for run_created events. Add a world-vercel regression test that a
current sfo1-tagged runId passes validation.
* fix(world-vercel): default run ID region to iad1 instead of unknown
When neither an explicit region option nor VERCEL_REGION is available,
createRunId minted a tagged ULID with the unknown (0) region sentinel,
producing the tagged: true, region: null state. The server already
resolves unknown/untagged runs to DEFAULT_VERCEL_REGION (iad1), so mint
a concrete iad1 tag instead, keeping every run ID self-describing and
routable.
* test(e2e): use verbose reporter + per-test start heartbeat
The default vitest reporter buffers per-file output, so a stalling e2e
test produces no output until its timeout — making CI look like a silent
30-minute hang. Switch the e2e CI invocations to the verbose reporter
(prints each test result as it completes) and emit a '[e2e] ▶ start:'
heartbeat to stdout at the start of every test (bypassing vitest's console
buffering) so a stuck test is immediately identifiable in the live CI log.
* test(world-vercel): point WORKFLOW_SERVER_URL_OVERRIDE at combined 527+529 preview
Temporarily target the workflow-server combined-527-529-preview deployment,
which bundles platform-directed multi-region routing (vercel/workflow-server#527,
incl. the iad1 hook pin) and durable stream state (vercel/workflow-server#529),
so e2e can validate the full multi-region path end-to-end. Revert to empty on main.
* fix(core): region-tag the health-check correlationId
The health-check response is delivered over a Redis stream whose name (and
synthetic run ID) embed the correlationId. Under platform-directed routing
the responding endpoint and the polling reader can be served from different
physical regions; Redis is physical-region-local, so the correlationId must
carry the region for both sides to resolve the same backend.
Generate the correlationId via world.createRunId() (a region-tagged ULID)
when the world provides it, falling back to a plain ULID for worlds that
don't tag IDs (e.g. local, single-region). The synthetic wrun_hc_<id> run ID
then carries the region; workflow-server's region middleware decodes it.
* Address review feedback: validate region overrides, reset server override
- queue: validate opts.region and VERCEL_REGION against the known region
table before routing, ignoring unrecognised codes so a bad override
can't clobber the payload-derived region (Copilot)
- add isKnownRegionCode() runtime guard to run-id/regions
- reset WORKFLOW_SERVER_URL_OVERRIDE to '' (must be empty on main)
- fold the within-PR iad1-default changeset into the main world-vercel
changeset and delete it (review)
- start.test: declare specVersion on createRunId mock worlds now that
the merged world-compatibility check requires it
- cover the new region-validation fall-through paths in queue.test
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* test(world-vercel): point WORKFLOW_SERVER_URL_OVERRIDE at wave-1 multi-region preview
BRANCH-ONLY — revert the override to '' before merge (lint enforces).
Points this PR's e2e/benchmark runs at the wave-1 multi-region
workflow-server preview (vercel/workflow-server#590: iad1+sfo1+fra1
serving, staging data backends) so region-tagged runs are validated
against real multi-region serving end-to-end.
Also makes the unit-test mock origins in events-v4.test.ts and
trace-propagation.test.ts override-aware (same pattern the rest of
the file and utils.test.ts already use), so the suite passes whether
or not the override is set — these two files were the only spots
hardcoding https://vercel-workflow.com.
* test(e2e): Vercel multi-region suite for start()'s region option
Adds a dedicated e2e suite validating @workflow/world-vercel region
routing end to end, run as its own CI job (e2e-vercel-multi-region)
against the nextjs-turbopack workbench only — deliberately separate
from e2e.test.ts, which runs as a matrix across all worlds/frameworks
where Vercel-specific multi-region behavior doesn't apply.
- workbench/nextjs-turbopack/vercel.json: deploy to iad1+sfo1+fra1 so
region-routed flow messages have a function to land on in each region.
- workflows/99_e2e.ts: regionProbeWorkflow returns the VERCEL_REGION
observed by both the workflow and a step, so tests can assert the run
EXECUTED in the intended region (not just that it was tagged).
- packages/core/e2e/e2e-region.test.ts: per-region cases assert
1) start(..., { region }) mints a region-tagged run ID (decoded via
@workflow/world-vercel/run-id),
2) the workflow + step both observed VERCEL_REGION === region,
3) the server reports the run completed;
plus a concurrent all-regions case guarding against cross-region
misrouting under simultaneous multi-region traffic. Skips on local
deployments.
- .github/workflows/tests.yml: new e2e-vercel-multi-region job
mirroring e2e-vercel-prod's env/deployment-wait, running only the
new suite.
* test(e2e): start region probes in-function; fix getWorld await
The first multi-region CI run surfaced two issues:
1. sfo1/fra1-tagged runs executed in iad1. The suite started runs from
the external test process, which uses the api.vercel.com token proxy
— and the proxy's queues path forwards every send to the region-less
VQS host (the world's proxy-mode resolveBaseUrl ignores the region
argument, and the proxy's x-vercel-vqs-api-url escape hatch only
allowlists vqs-server-*.vercel.sh preview hosts). Production traffic
publishes IN-FUNCTION (direct regional queue routing), so the suite
now triggers start() through a new workbench route
(/api/e2e-region-start) and rehydrates the run with getRun() —
testing the path production actually takes. Proxy-mode regional
queue routing is a known gap to address separately in api-workflow.
2. TypeError on world.runs.get: getWorld() is async and was called
without await.
* test(e2e): cover explicit and implicit region starts in the multi-region suite
With regional VQS routing now working through the api.vercel.com proxy
(vercel/api#79056 + #2789 + this branch's per-send region resolution),
the suite covers both start configurations, asserting the same three
properties for each (region-tagged run ID, execution in the intended
region via VERCEL_REGION echoed in the return value, server-side
completion):
1. EXPLICIT: start(..., { region }) called directly in the vitest
runner — publishes through the token proxy, per-send region carried
by x-vercel-queue-region. Restores the direct-start shape the suite
had originally, plus the concurrent all-regions case.
2. IMPLICIT: dedicated per-region workbench routes
(/api/e2e-region-implicit/{iad1,sfo1,fra1}), each pinned to a single
region via a per-function 'regions' entry in the workbench
vercel.json, calling start() with NO region option — createRunId
derives the tag from the minting function's VERCEL_REGION. The test
also asserts the route reported executing in its pinned region, so
the implicit-tagging assertion can't pass vacuously.
Replaces the interim /api/e2e-region-start route (explicit region via
request body), which existed to work around the pre-#79056 proxy gap.
* Revert WORKFLOW_SERVER_URL_OVERRIDE to '' — wave-1 multi-region serving is in production
workflow-server#590 (iad1+sfo1+fra1 serving) merged and deployed to
production and the e2e backend, so this branch's e2e/benchmark runs no
longer need to target the wave-1 preview. Restores the empty override
the No Test Overrides lint job enforces for merge.
The override-aware unit-test origins (events-v4/trace-propagation)
stay — they are correct under any override value.
* test(e2e): cross-region stream visibility (iad1 writer, sfo1 reader)
Regression coverage for a backend bug that made cross-region stream
reads report zero chunks on IN-PROGRESS streams (completed streams were
unaffected), which forced the multi-region serving rollback.
The new case exercises exactly that geometry:
- crossRegionStreamWorkflow (99_e2e.ts) writes N chunks to the default
output stream, then holds the stream OPEN for 45s before closing —
the in-progress window is the point, since completed streams are the
easy case.
- The e2e starts it with region iad1, waits (same-region, via the
api.vercel.com proxy) until all chunks are written, asserts the run
is still 'running', then reads through a new sfo1-pinned workbench
route (/api/e2e-stream-read/sfo1) that returns getTailIndex() plus
its VERCEL_REGION. The reader's region served none of the stream's
writes, so the reported chunk count must come from the backend's
cross-region stream metadata. The test fails loudly if the route
isn't actually executing in sfo1.
Also bumps the explicit-region test timeout to 120s: the first case in
the file absorbs every cold start at once (fresh workbench instances
in up to three regions plus a cold backend preview) and was observed
just over the 60s default.
BRANCH-ONLY (revert before merge, lint enforces):
WORKFLOW_SERVER_URL_OVERRIDE points at a multi-region backend preview
that includes the fix, so this validates cross-region stream
visibility end-to-end before multi-region serving is re-enabled.
* test(e2e): extend multi-region suite to all 19 provisioned regions
Points the suite at an all-regions backend preview and widens coverage
from the wave-1 trio to every provisioned region:
- Explicit path: a single concurrent all-regions case starts one
tagged run per region (one shared cold-start window instead of 19
sequential ones) and aggregates per-region failures so a single
region's breakage reports alongside the full picture. The trio keeps
its detailed per-region cases and the 9-way concurrent-isolation
case.
- Implicit path: workbench gains a region-pinned
/api/e2e-region-implicit/<region> route per provisioned region (19
total, shared handler), the workbench itself now deploys to all of
them, and the test.each covers the full set with per-case timeouts
for regional cold starts.
- Multi-region CI job timeout 20m -> 35m for the sequential implicit
cases.
BRANCH-ONLY (revert before merge, lint enforces):
WORKFLOW_SERVER_URL_OVERRIDE now targets the all-regions backend
preview instead of the previous (stale, since-merged) fix preview.
* test(e2e): tolerate geo-adjacent execution of queue callbacks
The first all-regions run surfaced a subtle execution-locality
behavior: queue delivery is guaranteed to the tagged region's
dataplane and the delivery callback egresses from that region, but the
consumer invocation's execution region is chosen by where that
callback enters Vercel's edge — and adjacent regions can geo-resolve
to each other's functions. Observed live: kix1-tagged runs (callback
egressing from Osaka) deterministically executing in hnd1/Tokyo on
both the explicit and implicit paths, with tagging, data placement,
and completion all still strictly kix1.
expectRunInRegion now asserts execution lands in the tagged region OR
one of its geographic neighbors (EXECUTION_ADJACENCY), while run-ID
tagging and server-side completion remain strictly the requested
region. Gross misrouting (e.g. kix1 -> iad1) still fails.
* Revert WORKFLOW_SERVER_URL_OVERRIDE to '' — all-regions serving is in production
The all-regions workflow-server rollout is deployed and serving
production traffic from every Vercel region, so this branch's e2e no
longer needs to target a branch preview. Restores the empty override
the No Test Overrides lint enforces for merge.
With this the PR is complete: region-tagged run IDs, region-aware
queue routing, and the multi-region e2e suite (explicit + implicit +
all-regions + cross-region streams) all validate against the
production-default backends.
* docs: fix three stale comments flagged in review
- start.ts: StartOptionsBase.region fallback is iad1, not the unknown
sentinel (createRunId always mints a concrete routable region)
- queue.ts: example used a nonexistent start({ runIdInput }) API; the
real option is start({ region })
- events.ts: decode() clears only the tag bit (top bit of the 48-bit
timestamp field) — it does not restore the original untagged ULID;
reword to say what actually matters for timestamp validation
* test(e2e): cover hook resolve/resume for runs owned by non-iad1 regions
Hooks are resolved by opaque token, which carries no region hint, so
lookup and resume must work regardless of which region owns the run's
data. Exercises the full follow-up-message path on sfo1- and
fra1-tagged runs: create inside the workflow, resolve by token from
the test process, resume twice sequentially, and assert payload order
and completion.
Regression coverage for the failure mode where the first message to a
hook-driven app on a non-iad1 run worked but every follow-up failed
with 'Hook not found'.
---------
Co-authored-by: Peter Wielander <peter.wielander@vercel.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* build: declare typescript (catalog:) in every package that runs tsc
Twenty packages invoke tsc in their build/typecheck scripts without
declaring a typescript dependency, resolving whatever tsc pnpm happens
to leave reachable. That broke locally after the TypeScript 6 upgrade
(#2700): base.json now uses the TS6-only 'types': ['*'] wildcard, and
worktrees carrying pre-upgrade node_modules/.bin/tsc shims (orphaned
typescript@5.9.3 bins that pnpm never refreshes for an undeclared
dependency) fail with TS2688 'Cannot find type definition file for *'.
Declaring 'typescript': 'catalog:' (the convention nest already
follows) makes pnpm own each package's tsc bin, so version upgrades
refresh the shims and this staleness class cannot recur. Packages
without tsc in their scripts are left unchanged.
Full pnpm build: 27/27 tasks green.
* Address review: drop duplicate zod devDep; regenerate lockfile minimally
- packages/world listed zod in both dependencies and devDependencies
(pre-existing on main, surfaced by the devDependencies sort) — keep
the runtime dependency only.
- Regenerate pnpm-lock.yaml from a pristine main baseline with
--lockfile-only (a clean-main run produces zero diff, so main has no
drift). Remaining non-typescript changes are mechanical consequences
of the change itself: typescript is an (optional) peer of several
tooling dependencies, so declaring it in 20 importers creates new
peer-resolution snapshot variants and prunes the now-orphaned old
ones; plus one radix-ui 1.6.1->1.6.2 refresh in docs caused by its
floating 'latest' specifier.
- Validated: pnpm install --frozen-lockfile succeeds; full build 27/27.
* fix(world): parse timezone-naive analytics timestamps as UTC
ClickHouse-backed analytics endpoints serialize DateTime64 values as
timezone-naive strings ('2026-07-13 17:09:11.593'), UTC by convention.
The analytics schemas coerced them with z.coerce.date(), i.e.
new Date(value), which interprets naive strings in the process's LOCAL
timezone. That is only correct when the process runs in UTC — the
deployed observability web app's server actions, which is why the web
UI appears unaffected — and wrong by the local UTC offset everywhere
else: the CLI on a laptop showed runs 'starting in about 7 hours'
(PDT), and 'workflow web --localUi' shares the bug.
Replace the coercion with a preprocess that normalizes naive datetime
strings to an explicit Z designator before parsing. Values already
carrying timezone info (Z or ±hh:mm) and non-string inputs (Date,
epoch) pass through unchanged.
Tests pin the contract and were verified under TZ=America/Los_Angeles
and TZ=Asia/Tokyo (3 of 4 fail against the old coercion in PDT; plain
UTC CI cannot distinguish the two, which is how this shipped).
* Potential fix for pull request finding
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Signed-off-by: Nathan Rajlich <n@n8.io>
---------
Signed-off-by: Nathan Rajlich <n@n8.io>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
* telemetry: emit client-observed workflow.stream.write span per flush batch
Complements the existing workflow.stream.read TTFC span: each flushed
batch emits a back-dated CLIENT span covering the app-perceived write
latency (buffer dwell + RPC), with buffer_dwell_ms / chunks / bytes
attributes so client-side batching cost (flush timer, turbo run-ready
barrier) can be told apart from network/server time. Failed flushes
keep the batch's original t0 so a retried batch reports its full dwell.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* chore: tighten changeset
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* telemetry: rename flush span to workflow.stream.flush; DEBUG-log world-vercel OTEL load failure
- workflow.stream.write is taken by world-vercel's per-request RPC span
(#2857, chunk_rtt); the per-batch flush span gets its own name so the
two stay distinguishable in trace queries. Attributes move to
workflow.stream.flush.{buffer_dwell_ms,chunks,bytes}, operation=flush.
- world-vercel's @opentelemetry/api load failure was silently latched as
null, which also swallows bundler/resolution failures in apps that DO
register a tracer (observed in production: workbench apps emit core
spans but none of world-vercel's). Log the reason under
DEBUG=workflow:* so the failure mode is diagnosable.
- Document workflow.stream.flush in the tracing docs.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* chore: tighten changeset
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* feat(core): record replay lineage in run executionContext
recreateRunFromExisting now stamps the source run id into the new run's
executionContext as `replayedFromRunId`, and `start` accepts a matching
option. This lets tooling (e.g. the dashboard runs list) surface a run as
a replay and link back to the run it was replayed from — previously a
replay started a brand-new run with no link to its origin.
Co-authored-by: Cursor <cursoragent@cursor.com>
* Apply suggestion from @mitul-s
Signed-off-by: Mitul Shah <mitulxshah@gmail.com>
* cleanup
* Update runs.test.ts
* update
---------
Signed-off-by: Mitul Shah <mitulxshah@gmail.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
The row is already right-aligned via the parent's justify-between, so
flex-1 was only forcing the value column to stretch. Drop it so values
hug their content.
Co-authored-by: Cursor <cursoragent@cursor.com>
* [core] Thread queue namespace through o11y run actions and bound healthCheck stream reads
Deployments that use a queue namespace (e.g. eve's __eve_wkf_workflow_*
topics) could not be targeted by cross-context callers: start(),
recreateRunFromExisting(), reenqueueRun(), and wakeUpRun() always built
queue names from the caller's WORKFLOW_QUEUE_NAMESPACE env, so dashboard
replays published to topics the target deployment has no consumer for.
Additionally, healthCheck()'s poll loop only checked its timeout between
iterations while world.streams.get() itself was unbounded — against
workflow-server, which holds unwritten streams open for ~2 minutes, a
2s capability probe hung until the caller's function timed out (the
observed 30s 504s on dashboard replay).
- Add a `namespace` option to StartOptionsBase, RecreateRunOptions,
StopSleepOptions, and new ReenqueueRunOptions; thread it into
getWorkflowQueueName() and the cross-deployment capability probe.
- Fold the inline `namespace` param into HealthCheckOptions.
- Race streams.get() against the remaining health-check budget.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Shorten changeset
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* feat(world-vercel): name stream client spans + add stream attributes
Stream write/read requests already share the instrumented HTTP envelope
(a CLIENT span + W3C trace-context injection), but the spans were named
for the bare HTTP verb (`http PUT`/`http GET`) and carried only generic
HTTP attributes — so stream latency couldn't be sliced per run/stream.
Name these spans for their operation (`workflow.stream.write` /
`workflow.stream.read`) and tag them with `workflow.run.id`,
`workflow.stream.name`, `workflow.stream.operation`
(write | write_multi | close | read), and `workflow.stream.start_index`
(read). Implemented via new optional `spanName`/`attributes` fields on
`instrumentedFetch`, so other callers are unaffected.
Additive OTEL only: no behavior change when no OpenTelemetry SDK is
registered (the span is undefined and attributes are dropped).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* Shorten changeset summary
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* Add client-observed end-to-end read TTFC span
The read GET can't report a client-measured latency back to the server (the
value only exists after the response starts streaming), so capture it purely
in the SDK's own OTEL: watch response.body for the first non-empty chunk and
emit a workflow.stream.read span back-dated to read dispatch, whose duration is
the end-to-end time-to-first-chunk (incl. the network hop) via
workflow.stream.read.ttfc_ms. Rename the fetch/connect span to
workflow.stream.read.connect. No-op without an OTEL SDK registered.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* Add client-observed e2e write latency attribute
The write PUT is request/response and the server acks only after capturing the
chunk, so the workflow.stream.write span duration already equals the
client->server write latency. Expose it as a named attribute
workflow.stream.write.e2e_ms (via a durationAttribute option on
instrumentedFetch) for direct querying, parallel to the read ttfc_ms.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* docs: document stream spans and latency attributes
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* Emit read TTFC span from the core reader instead of a world-vercel transform
Per review: move the client-observed time-to-first-chunk measurement out of a
TransformStream wrapper in world-vercel and into WorkflowServerReadableStream in
core, emitting workflow.stream.read on the first non-empty chunk reaching the
consumer. Removes the passthrough, measures at the reader abstraction, and is
backend-agnostic. world-vercel keeps the workflow.stream.read.connect HTTP span;
the recordElapsedSpan helper now lives in @workflow/core.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* Apply suggestion from @VaguelySerious
Signed-off-by: Peter Wielander <mittgfu@gmail.com>
* Update docs/content/docs/v5/observability/tracing.mdx
Co-authored-by: Peter Wielander <mittgfu@gmail.com>
Signed-off-by: Karthik Kalyan <105607645+karthikscale3@users.noreply.github.com>
* Address review: rename write.e2e_ms -> write.chunk_rtt; docs + changeset wording
Per review, rename the write attribute to workflow.stream.write.chunk_rtt (it's
a per-chunk client<->server round-trip, not a full e2e), update the docs row
wording for both write and read attributes, and shorten the changeset.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
---------
Signed-off-by: Peter Wielander <mittgfu@gmail.com>
Signed-off-by: Karthik Kalyan <105607645+karthikscale3@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Co-authored-by: Peter Wielander <mittgfu@gmail.com>
* Anchor trace viewer shortcut helper to the timeline's left edge
Render the shortcut helper inside the timeline column instead of as a
pane-root overlay, so it aligns to the timeline's left edge (tracking
the divider for free) and sticks to the bottom of the viewport while
the pane scrolls. Gate visibility on a container-query width
(`@container` on the timeline column + `@min-[420px]`) rather than the
viewport `md` breakpoint, so it hides based on the timeline's own width.
No SplitPane API change required.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Raise shortcut-helper container-query threshold to 480px
At 420px the helper could still show when the detail panel squeezes the
timeline, colliding with the zoom controls in the bottom-right. 480px
leaves comfortable clearance so it hides once the timeline is narrow.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Update trace-viewer.tsx
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
github-actions[bot]