* fix(deps): dedupe @opentelemetry/api to a single workspace instance
The lockfile resolved both 1.9.0 and 1.9.1, so the copy that registers
the tracer provider (via @vercel/otel in the app) and the copy a package
imports could differ. The API's global-registration version check rejects
a consumer newer than the registered copy and silently hands back a noop
tracer — which is why world-vercel's spans (workflow.stream.write/
chunk_rtt, read.connect, its http spans) never reached Datadog from
deployed apps while core's spans flowed in the same process. Root-caused
via the DEBUG=workflow:* run on #2900: import succeeds, no warn, spans
dropped.
Pin a single version via a workspace override so every bundle shares one
API instance.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* debug: one-shot OTEL runtime diagnostic in core + world-vercel; DEBUG on turbopack workbench
The dedupe alone did not restore world-vercel span emission (verified on
this PR's own preview: stream traffic flowed, zero workflow.stream.write
spans). Under DEBUG=workflow:*, both packages now log once how their
module instance of @opentelemetry/api sees the world — global
registration version, provider/delegate/tracer/probe constructor names,
and whether a probe span is recording. Diffing the core line (spans work)
against the world-vercel line (spans dropped) in one deployment's logs
pinpoints the divergence.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* debug: log span identity for named world-vercel spans; namespace otel probes per package
Diag round 1 showed world-vercel's tracer records and instrumentedFetch
handles the stream PUTs, yet the named spans are unfindable in the
backend. Round 2: log traceId/spanId/isRecording for every named
instrumentedFetch span under DEBUG so export can be checked for a
specific span id, and split the probe span names (.core /
.world_vercel) so per-package export is attributable.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* telemetry: emit stream RPC latencies from core (chunk_rtt, connect_ms, close span)
world-vercel's instrumentedFetch spans never export from deployed apps
(root cause still open — see PR discussion), so the operationally
needed client-side latency signals move one layer up to core, whose
spans are proven to export:
- workflow.stream.write.chunk_rtt on the workflow.stream.flush span:
the World write RPC duration, network included (same attribute key as
world-vercel's per-request span so queries are layer-agnostic).
- workflow.stream.read.connect_ms on the workflow.stream.read span:
the world.streams.get await (read dispatch -> stream handle).
- new workflow.stream.close span: the close RPC round trip.
Bonus: measured at the World interface, these cover world-local and
world-postgres too, not just Vercel deployments.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* telemetry: emit read-completion span (total duration, chunks, bytes)
Completes the read-side picture: workflow.stream.read.complete is
back-dated to the read dispatch so its duration is the total read, with
chunk/byte counts for throughput. Cancelled reads emit nothing.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* chore: drop DEBUG from turbopack workbench; tighten changeset
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* telemetry: cover createReconnectingFramedStream in read telemetry
Ordinary serialized streams read through createReconnectingFramedStream
(which calls world.streams.get directly), so connect_ms / ttfc /
read.complete never fired for that path — only WorkflowServerReadableStream
was instrumented. Wire the same helpers into the framed reader: first-
connect duration, first-frame TTFC, and completion totals — plus
workflow.stream.read.reconnects, which only this path can know.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* cli: show run region in inspect output via World.regionForRunId
Adds an optional reverse-lookup hook to the World interface —
regionForRunId(runId): string | null — so tooling can display a run's
region generically. Worlds without a regional dimension simply omit
the hook and no region output appears.
- @workflow/world: new optional interface member (documented: must not
throw; null = undeterminable)
- @workflow/world-vercel: implements it from the run-ID region tag
(tagged -> embedded region, untagged legacy -> default region,
malformed -> null)
- @workflow/cli: 'workflow inspect runs' gains a region column
(between workflowName and status) and 'workflow inspect run <id>'
a region property, in both table and JSON output — only when the
world defines the hook
* Generalize the inspect hook: World.describeRun display fields
Replaces regionForRunId on the World interface with describeRun, per
review: worlds may want to expose more than a region, and the
information need not be encoded in the run ID — describeRun receives
the run entity itself (loosely typed, mirroring createRunId), so a
world can derive fields from executionContext or any other property.
Each returned key becomes an inspect column/property; null values are
preserved in structured output ('applicable but undeterminable' vs.
the hook being absent entirely).
- world-vercel: describeRun returns { region } decoded from the run
ID tag (regionForRunId stays exported as a utility); entities
without a usable runId contribute nothing
- CLI listing: columns come from the union of keys the world returns
for the page, inserted before status; both analytics and storage
paths; detached call site binds this
- CLI showRun: merges the world fields into detail/JSON output via a
method-style call (preserves this), keeping nulls
- tests: field merging (multi-key), null preservation in JSON, hook
absent, and world-vercel describeRun coverage incl. no-runId
entities
* cli: evaluate describeRun defensively
Per review: the World interface says describeRun is pure and must not
throw, but it is an external extension point and the CLI should not
trust that. New safeWorldFields helper, used by both the listing and
showRun paths:
- a throwing implementation contributes no fields instead of crashing
the inspect command
- keys that already exist on the run row are dropped, so a world can
never overwrite canonical fields (status, runId, ...) in output
Tests: canonical fields survive a clobbering describeRun (extra keys
still merged); a throwing describeRun leaves rows untouched and the
command succeeds.
* Allow async describeRun implementations
Per review: widening a sync signature to async later would break every
consumer, while accepting sync-or-async from day one is free — sync
implementations (like world-vercel's) remain valid, and consumers
simply await, which handles both. The performance intent lives on as
documented guidance: the hook is called once per displayed run, so
implementations should stay cheap and avoid I/O; the CLI evaluates a
page's rows concurrently so an async world costs one await per page,
not per row. Promise rejections get the same treatment as throws:
no fields, never a crash.
* Update packages/world/src/interfaces.ts
Co-authored-by: Peter Wielander <mittgfu@gmail.com>
Signed-off-by: Nathan Rajlich <n@n8.io>
---------
Signed-off-by: Nathan Rajlich <n@n8.io>
Co-authored-by: Peter Wielander <mittgfu@gmail.com>
* Add attribute discovery and filtering to world.analytics
- analytics.attributes.list() — distinct attribute keys observed on runs
in the window, with run counts and first/last seen timestamps
(GET /v2/analytics/attributes).
- analytics.attributes.listValues({ key }) — distinct values for one key
with latest-write-wins run counts (GET /v2/analytics/attributes/values).
- analytics.runs.list({ attributes: { key: value } }) — restrict the runs
listing to runs whose latest attribute snapshot matches every provided
pair (JSON-encoded query param, up to 8 pairs; $-prefixed framework
keys allowed in read filters).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Shorten changeset; document world.analytics in the World SDK reference
The analytics namespace was previously undocumented. Adds a full
reference page (runs, attributes, steps/events/hooks/waits, lookback
windows and pageInfo), links it from the World SDK index and meta, and
replaces the stale 'in the future you'll be able to search by
attributes' line in the attributes guide with a filtering section.
Extends the docs-typecheck ambient world global with the analytics
namespace so reference snippets typecheck.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Drop analytics.attributes.listValues
Not a derived requirement: the agent-runs UI filters by known constant
values and reads per-run values via batch attribute fetches; it never
enumerates distinct values for a key.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Feature-detect world.analytics in the attributes guide example
The snippet dereferenced the optional analytics namespace without the
runtime check its skip-typecheck annotation claimed, and would throw on
local/Postgres/custom Worlds. Guard it and drop the annotation — the
block is now genuinely typechecked.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Fix stale 'querying attributes not available' bullet in the guide
The Behavior list still claimed a query API was only planned, directly
contradicting the Searching and filtering section above it. State what
is implemented: attributes are readable on run objects everywhere, and
key discovery / key=value run filtering are available through the
optional Analytics API.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Surface the analytics namespace on the API reference index pages
The Analytics page sits under workflow/runtime > World SDK, but neither
the API reference index card, the workflow/runtime World SDK card, nor
the World SDK overview mentioned analytics — making the new reference
effectively undiscoverable from /docs/api-reference. Mention it at each
level of the path.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Bump changeset to minor
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* [world-vercel] Add /run-id sub-export with tagged ULID encode/decode
Encodes a tag bit, 5-bit version, and 6-bit Vercel region ID into a
ULID-shaped string used for workflow run IDs. Tagged values remain
valid 26-char Crockford-Base32 ULIDs so they still sort and round-trip
through any system that accepts ULIDs.
* [world-vercel] Add string-value assertions to run-id tests
Add exact-string expectations for encoded outputs at known inputs,
covering the default region/version pair, numeric region IDs, version
overrides, boundary values (all-zero, all-max), the dirty-input
overwrite case, and the lexicographic-order checks. Also adds an
explicit byte-array expectation for the canonical ULID-spec example
string and an additional first-char-range coverage test for isTagged.
* [world-vercel] Remove internal-repo reference from regions doc comment
* [world-vercel] Address PR review feedback on run-id sub-export
- isTaggedString now fully validates the input as a 26-char Crockford
Base32 ULID (delegating to ulidToBytes) instead of only inspecting
the first character. This fixes false positives on inputs like
'4UUUU...' that have a valid tag-bit position but invalid chars
later in the string.
- isTagged() now accepts `unknown` to match its documented behavior
of safely rejecting non-string inputs without requiring callers to
cast.
- Introduce `RegionKey` for the full set of keys including 'unknown',
and narrow `RegionCode` to `Exclude<RegionKey, 'unknown'>` so the
return type of `lookupRegion` and the `DecodedRunId.region` field
accurately reflect that 'unknown' is never produced. Updates
`encode` to reject 'unknown' as a region code string at runtime
(callers wanting the unknown sentinel should pass numeric 0).
* [world] [core] [world-vercel] Add World.createRunId() and region-aware queue routing
- @workflow/world: add optional createRunId(input?) to the World
interface so worlds can mint run IDs with embedded metadata, and
add an optional 'region' field to QueueOptions for per-message
routing hints.
- @workflow/core: start() now delegates run ID generation to
world.createRunId() when defined (falling back to a monotonic
ULID otherwise), and accepts a new 'runIdInput' option that is
forwarded verbatim to createRunId. When runIdInput.region is a
string, it is also threaded onto the queue options so the initial
workflow message is dispatched to the matching region.
- @workflow/world-vercel: implement createRunId() to mint
region-tagged ULIDs, preferring an explicit runIdInput.region and
falling back to the VERCEL_REGION env var. The queue now resolves
its destination region from (in order): an explicit opts.region,
the region embedded in the payload's tagged run ID, the
VERCEL_REGION env var, and finally a hardcoded 'iad1' default.
This replaces the previous unconditional 'iad1' region passed to
the @vercel/queue client.
Monotonicity within a process is preserved by tracking the last
emitted run ID and bumping the bit immediately above the 11-bit
metadata window when a same-ms collision would otherwise occur,
then re-stamping the requested region/version on top so metadata
remains stable.
* [core] [world] [world-vercel] Pass full StartOptions to World.createRunId
Drop the dedicated 'runIdInput' field on StartOptions and forward the
entire options bag to world.createRunId() instead. This keeps the
public API surface smaller and lets each World pick the fields it
recognises (e.g. world-vercel reads 'region'). The top-level 'region'
option remains on StartOptionsBase and is also threaded onto the
queue's per-call region opt when set.
* Address review feedback: doc fixes and deterministic same-ms tests
- Document the final iad1 fallback in QueueOptions.region (world)
- Correct the World.createRunId doc: start() always passes an object
- Fix the clientOptions comment: the handler client omits region and
relies on SDK auto-detection + the ce-vqsregion header for acks
- Fix a misleading QueueClient-construction comment in queue.test.ts
- Freeze time in the same-ms monotonicity test so it deterministically
exercises the intended path, and add a test covering the
bump-above-metadata fallback when the region changes mid-millisecond
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* test: keep workflow-server override rewrite-compatible
Export WORKFLOW_SERVER_URL_OVERRIDE while keeping the one-line const shape
that workflow-server's cross-repo e2e test automation rewrites. Update
world-vercel tests to import that exported value for mock origins and URL
expectations instead of duplicating the temporary preview URL.
* fix(world): clear region tag bit before ULID timestamp validation
Region-tagged run IDs set the high bit of the ULID timestamp byte. The
shared world timestamp validator used raw decodeTime(), so current tagged
run IDs appeared thousands of years in the future and were rejected before
reaching workflow-server. Clear the tag bit before decoding, matching the
workflow-server behavior, and cover tagged IDs in tests.
* fix(world-vercel): validate tagged runId timestamps via run-id decode
Keep @workflow/world's ULID helpers generic; they should not know about
world-vercel's region-tagged run ID layout. Instead, world-vercel decodes
its tagged runId to the original ULID before using the shared timestamp
validator for run_created events. Add a world-vercel regression test that a
current sfo1-tagged runId passes validation.
* fix(world-vercel): default run ID region to iad1 instead of unknown
When neither an explicit region option nor VERCEL_REGION is available,
createRunId minted a tagged ULID with the unknown (0) region sentinel,
producing the tagged: true, region: null state. The server already
resolves unknown/untagged runs to DEFAULT_VERCEL_REGION (iad1), so mint
a concrete iad1 tag instead, keeping every run ID self-describing and
routable.
* test(e2e): use verbose reporter + per-test start heartbeat
The default vitest reporter buffers per-file output, so a stalling e2e
test produces no output until its timeout — making CI look like a silent
30-minute hang. Switch the e2e CI invocations to the verbose reporter
(prints each test result as it completes) and emit a '[e2e] ▶ start:'
heartbeat to stdout at the start of every test (bypassing vitest's console
buffering) so a stuck test is immediately identifiable in the live CI log.
* test(world-vercel): point WORKFLOW_SERVER_URL_OVERRIDE at combined 527+529 preview
Temporarily target the workflow-server combined-527-529-preview deployment,
which bundles platform-directed multi-region routing (vercel/workflow-server#527,
incl. the iad1 hook pin) and durable stream state (vercel/workflow-server#529),
so e2e can validate the full multi-region path end-to-end. Revert to empty on main.
* fix(core): region-tag the health-check correlationId
The health-check response is delivered over a Redis stream whose name (and
synthetic run ID) embed the correlationId. Under platform-directed routing
the responding endpoint and the polling reader can be served from different
physical regions; Redis is physical-region-local, so the correlationId must
carry the region for both sides to resolve the same backend.
Generate the correlationId via world.createRunId() (a region-tagged ULID)
when the world provides it, falling back to a plain ULID for worlds that
don't tag IDs (e.g. local, single-region). The synthetic wrun_hc_<id> run ID
then carries the region; workflow-server's region middleware decodes it.
* Address review feedback: validate region overrides, reset server override
- queue: validate opts.region and VERCEL_REGION against the known region
table before routing, ignoring unrecognised codes so a bad override
can't clobber the payload-derived region (Copilot)
- add isKnownRegionCode() runtime guard to run-id/regions
- reset WORKFLOW_SERVER_URL_OVERRIDE to '' (must be empty on main)
- fold the within-PR iad1-default changeset into the main world-vercel
changeset and delete it (review)
- start.test: declare specVersion on createRunId mock worlds now that
the merged world-compatibility check requires it
- cover the new region-validation fall-through paths in queue.test
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* test(world-vercel): point WORKFLOW_SERVER_URL_OVERRIDE at wave-1 multi-region preview
BRANCH-ONLY — revert the override to '' before merge (lint enforces).
Points this PR's e2e/benchmark runs at the wave-1 multi-region
workflow-server preview (vercel/workflow-server#590: iad1+sfo1+fra1
serving, staging data backends) so region-tagged runs are validated
against real multi-region serving end-to-end.
Also makes the unit-test mock origins in events-v4.test.ts and
trace-propagation.test.ts override-aware (same pattern the rest of
the file and utils.test.ts already use), so the suite passes whether
or not the override is set — these two files were the only spots
hardcoding https://vercel-workflow.com.
* test(e2e): Vercel multi-region suite for start()'s region option
Adds a dedicated e2e suite validating @workflow/world-vercel region
routing end to end, run as its own CI job (e2e-vercel-multi-region)
against the nextjs-turbopack workbench only — deliberately separate
from e2e.test.ts, which runs as a matrix across all worlds/frameworks
where Vercel-specific multi-region behavior doesn't apply.
- workbench/nextjs-turbopack/vercel.json: deploy to iad1+sfo1+fra1 so
region-routed flow messages have a function to land on in each region.
- workflows/99_e2e.ts: regionProbeWorkflow returns the VERCEL_REGION
observed by both the workflow and a step, so tests can assert the run
EXECUTED in the intended region (not just that it was tagged).
- packages/core/e2e/e2e-region.test.ts: per-region cases assert
1) start(..., { region }) mints a region-tagged run ID (decoded via
@workflow/world-vercel/run-id),
2) the workflow + step both observed VERCEL_REGION === region,
3) the server reports the run completed;
plus a concurrent all-regions case guarding against cross-region
misrouting under simultaneous multi-region traffic. Skips on local
deployments.
- .github/workflows/tests.yml: new e2e-vercel-multi-region job
mirroring e2e-vercel-prod's env/deployment-wait, running only the
new suite.
* test(e2e): start region probes in-function; fix getWorld await
The first multi-region CI run surfaced two issues:
1. sfo1/fra1-tagged runs executed in iad1. The suite started runs from
the external test process, which uses the api.vercel.com token proxy
— and the proxy's queues path forwards every send to the region-less
VQS host (the world's proxy-mode resolveBaseUrl ignores the region
argument, and the proxy's x-vercel-vqs-api-url escape hatch only
allowlists vqs-server-*.vercel.sh preview hosts). Production traffic
publishes IN-FUNCTION (direct regional queue routing), so the suite
now triggers start() through a new workbench route
(/api/e2e-region-start) and rehydrates the run with getRun() —
testing the path production actually takes. Proxy-mode regional
queue routing is a known gap to address separately in api-workflow.
2. TypeError on world.runs.get: getWorld() is async and was called
without await.
* test(e2e): cover explicit and implicit region starts in the multi-region suite
With regional VQS routing now working through the api.vercel.com proxy
(vercel/api#79056 + #2789 + this branch's per-send region resolution),
the suite covers both start configurations, asserting the same three
properties for each (region-tagged run ID, execution in the intended
region via VERCEL_REGION echoed in the return value, server-side
completion):
1. EXPLICIT: start(..., { region }) called directly in the vitest
runner — publishes through the token proxy, per-send region carried
by x-vercel-queue-region. Restores the direct-start shape the suite
had originally, plus the concurrent all-regions case.
2. IMPLICIT: dedicated per-region workbench routes
(/api/e2e-region-implicit/{iad1,sfo1,fra1}), each pinned to a single
region via a per-function 'regions' entry in the workbench
vercel.json, calling start() with NO region option — createRunId
derives the tag from the minting function's VERCEL_REGION. The test
also asserts the route reported executing in its pinned region, so
the implicit-tagging assertion can't pass vacuously.
Replaces the interim /api/e2e-region-start route (explicit region via
request body), which existed to work around the pre-#79056 proxy gap.
* Revert WORKFLOW_SERVER_URL_OVERRIDE to '' — wave-1 multi-region serving is in production
workflow-server#590 (iad1+sfo1+fra1 serving) merged and deployed to
production and the e2e backend, so this branch's e2e/benchmark runs no
longer need to target the wave-1 preview. Restores the empty override
the No Test Overrides lint job enforces for merge.
The override-aware unit-test origins (events-v4/trace-propagation)
stay — they are correct under any override value.
* test(e2e): cross-region stream visibility (iad1 writer, sfo1 reader)
Regression coverage for a backend bug that made cross-region stream
reads report zero chunks on IN-PROGRESS streams (completed streams were
unaffected), which forced the multi-region serving rollback.
The new case exercises exactly that geometry:
- crossRegionStreamWorkflow (99_e2e.ts) writes N chunks to the default
output stream, then holds the stream OPEN for 45s before closing —
the in-progress window is the point, since completed streams are the
easy case.
- The e2e starts it with region iad1, waits (same-region, via the
api.vercel.com proxy) until all chunks are written, asserts the run
is still 'running', then reads through a new sfo1-pinned workbench
route (/api/e2e-stream-read/sfo1) that returns getTailIndex() plus
its VERCEL_REGION. The reader's region served none of the stream's
writes, so the reported chunk count must come from the backend's
cross-region stream metadata. The test fails loudly if the route
isn't actually executing in sfo1.
Also bumps the explicit-region test timeout to 120s: the first case in
the file absorbs every cold start at once (fresh workbench instances
in up to three regions plus a cold backend preview) and was observed
just over the 60s default.
BRANCH-ONLY (revert before merge, lint enforces):
WORKFLOW_SERVER_URL_OVERRIDE points at a multi-region backend preview
that includes the fix, so this validates cross-region stream
visibility end-to-end before multi-region serving is re-enabled.
* test(e2e): extend multi-region suite to all 19 provisioned regions
Points the suite at an all-regions backend preview and widens coverage
from the wave-1 trio to every provisioned region:
- Explicit path: a single concurrent all-regions case starts one
tagged run per region (one shared cold-start window instead of 19
sequential ones) and aggregates per-region failures so a single
region's breakage reports alongside the full picture. The trio keeps
its detailed per-region cases and the 9-way concurrent-isolation
case.
- Implicit path: workbench gains a region-pinned
/api/e2e-region-implicit/<region> route per provisioned region (19
total, shared handler), the workbench itself now deploys to all of
them, and the test.each covers the full set with per-case timeouts
for regional cold starts.
- Multi-region CI job timeout 20m -> 35m for the sequential implicit
cases.
BRANCH-ONLY (revert before merge, lint enforces):
WORKFLOW_SERVER_URL_OVERRIDE now targets the all-regions backend
preview instead of the previous (stale, since-merged) fix preview.
* test(e2e): tolerate geo-adjacent execution of queue callbacks
The first all-regions run surfaced a subtle execution-locality
behavior: queue delivery is guaranteed to the tagged region's
dataplane and the delivery callback egresses from that region, but the
consumer invocation's execution region is chosen by where that
callback enters Vercel's edge — and adjacent regions can geo-resolve
to each other's functions. Observed live: kix1-tagged runs (callback
egressing from Osaka) deterministically executing in hnd1/Tokyo on
both the explicit and implicit paths, with tagging, data placement,
and completion all still strictly kix1.
expectRunInRegion now asserts execution lands in the tagged region OR
one of its geographic neighbors (EXECUTION_ADJACENCY), while run-ID
tagging and server-side completion remain strictly the requested
region. Gross misrouting (e.g. kix1 -> iad1) still fails.
* Revert WORKFLOW_SERVER_URL_OVERRIDE to '' — all-regions serving is in production
The all-regions workflow-server rollout is deployed and serving
production traffic from every Vercel region, so this branch's e2e no
longer needs to target a branch preview. Restores the empty override
the No Test Overrides lint enforces for merge.
With this the PR is complete: region-tagged run IDs, region-aware
queue routing, and the multi-region e2e suite (explicit + implicit +
all-regions + cross-region streams) all validate against the
production-default backends.
* docs: fix three stale comments flagged in review
- start.ts: StartOptionsBase.region fallback is iad1, not the unknown
sentinel (createRunId always mints a concrete routable region)
- queue.ts: example used a nonexistent start({ runIdInput }) API; the
real option is start({ region })
- events.ts: decode() clears only the tag bit (top bit of the 48-bit
timestamp field) — it does not restore the original untagged ULID;
reword to say what actually matters for timestamp validation
* test(e2e): cover hook resolve/resume for runs owned by non-iad1 regions
Hooks are resolved by opaque token, which carries no region hint, so
lookup and resume must work regardless of which region owns the run's
data. Exercises the full follow-up-message path on sfo1- and
fra1-tagged runs: create inside the workflow, resolve by token from
the test process, resume twice sequentially, and assert payload order
and completion.
Regression coverage for the failure mode where the first message to a
hook-driven app on a non-iad1 run worked but every follow-up failed
with 'Hook not found'.
---------
Co-authored-by: Peter Wielander <peter.wielander@vercel.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* build: declare typescript (catalog:) in every package that runs tsc
Twenty packages invoke tsc in their build/typecheck scripts without
declaring a typescript dependency, resolving whatever tsc pnpm happens
to leave reachable. That broke locally after the TypeScript 6 upgrade
(#2700): base.json now uses the TS6-only 'types': ['*'] wildcard, and
worktrees carrying pre-upgrade node_modules/.bin/tsc shims (orphaned
typescript@5.9.3 bins that pnpm never refreshes for an undeclared
dependency) fail with TS2688 'Cannot find type definition file for *'.
Declaring 'typescript': 'catalog:' (the convention nest already
follows) makes pnpm own each package's tsc bin, so version upgrades
refresh the shims and this staleness class cannot recur. Packages
without tsc in their scripts are left unchanged.
Full pnpm build: 27/27 tasks green.
* Address review: drop duplicate zod devDep; regenerate lockfile minimally
- packages/world listed zod in both dependencies and devDependencies
(pre-existing on main, surfaced by the devDependencies sort) — keep
the runtime dependency only.
- Regenerate pnpm-lock.yaml from a pristine main baseline with
--lockfile-only (a clean-main run produces zero diff, so main has no
drift). Remaining non-typescript changes are mechanical consequences
of the change itself: typescript is an (optional) peer of several
tooling dependencies, so declaring it in 20 importers creates new
peer-resolution snapshot variants and prunes the now-orphaned old
ones; plus one radix-ui 1.6.1->1.6.2 refresh in docs caused by its
floating 'latest' specifier.
- Validated: pnpm install --frozen-lockfile succeeds; full build 27/27.
* telemetry: emit client-observed workflow.stream.write span per flush batch
Complements the existing workflow.stream.read TTFC span: each flushed
batch emits a back-dated CLIENT span covering the app-perceived write
latency (buffer dwell + RPC), with buffer_dwell_ms / chunks / bytes
attributes so client-side batching cost (flush timer, turbo run-ready
barrier) can be told apart from network/server time. Failed flushes
keep the batch's original t0 so a retried batch reports its full dwell.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* chore: tighten changeset
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* telemetry: rename flush span to workflow.stream.flush; DEBUG-log world-vercel OTEL load failure
- workflow.stream.write is taken by world-vercel's per-request RPC span
(#2857, chunk_rtt); the per-batch flush span gets its own name so the
two stay distinguishable in trace queries. Attributes move to
workflow.stream.flush.{buffer_dwell_ms,chunks,bytes}, operation=flush.
- world-vercel's @opentelemetry/api load failure was silently latched as
null, which also swallows bundler/resolution failures in apps that DO
register a tracer (observed in production: workbench apps emit core
spans but none of world-vercel's). Log the reason under
DEBUG=workflow:* so the failure mode is diagnosable.
- Document workflow.stream.flush in the tracing docs.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* chore: tighten changeset
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* feat(world-vercel): name stream client spans + add stream attributes
Stream write/read requests already share the instrumented HTTP envelope
(a CLIENT span + W3C trace-context injection), but the spans were named
for the bare HTTP verb (`http PUT`/`http GET`) and carried only generic
HTTP attributes — so stream latency couldn't be sliced per run/stream.
Name these spans for their operation (`workflow.stream.write` /
`workflow.stream.read`) and tag them with `workflow.run.id`,
`workflow.stream.name`, `workflow.stream.operation`
(write | write_multi | close | read), and `workflow.stream.start_index`
(read). Implemented via new optional `spanName`/`attributes` fields on
`instrumentedFetch`, so other callers are unaffected.
Additive OTEL only: no behavior change when no OpenTelemetry SDK is
registered (the span is undefined and attributes are dropped).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* Shorten changeset summary
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* Add client-observed end-to-end read TTFC span
The read GET can't report a client-measured latency back to the server (the
value only exists after the response starts streaming), so capture it purely
in the SDK's own OTEL: watch response.body for the first non-empty chunk and
emit a workflow.stream.read span back-dated to read dispatch, whose duration is
the end-to-end time-to-first-chunk (incl. the network hop) via
workflow.stream.read.ttfc_ms. Rename the fetch/connect span to
workflow.stream.read.connect. No-op without an OTEL SDK registered.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* Add client-observed e2e write latency attribute
The write PUT is request/response and the server acks only after capturing the
chunk, so the workflow.stream.write span duration already equals the
client->server write latency. Expose it as a named attribute
workflow.stream.write.e2e_ms (via a durationAttribute option on
instrumentedFetch) for direct querying, parallel to the read ttfc_ms.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* docs: document stream spans and latency attributes
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* Emit read TTFC span from the core reader instead of a world-vercel transform
Per review: move the client-observed time-to-first-chunk measurement out of a
TransformStream wrapper in world-vercel and into WorkflowServerReadableStream in
core, emitting workflow.stream.read on the first non-empty chunk reaching the
consumer. Removes the passthrough, measures at the reader abstraction, and is
backend-agnostic. world-vercel keeps the workflow.stream.read.connect HTTP span;
the recordElapsedSpan helper now lives in @workflow/core.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* Apply suggestion from @VaguelySerious
Signed-off-by: Peter Wielander <mittgfu@gmail.com>
* Update docs/content/docs/v5/observability/tracing.mdx
Co-authored-by: Peter Wielander <mittgfu@gmail.com>
Signed-off-by: Karthik Kalyan <105607645+karthikscale3@users.noreply.github.com>
* Address review: rename write.e2e_ms -> write.chunk_rtt; docs + changeset wording
Per review, rename the write attribute to workflow.stream.write.chunk_rtt (it's
a per-chunk client<->server round-trip, not a full e2e), update the docs row
wording for both write and read attributes, and shorten the changeset.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
---------
Signed-off-by: Peter Wielander <mittgfu@gmail.com>
Signed-off-by: Karthik Kalyan <105607645+karthikscale3@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Co-authored-by: Peter Wielander <mittgfu@gmail.com>
* web: infinite scroll for the runs table
Replace Previous/Next cursor paging with front-style infinite scroll:
a useInfiniteList hook accumulates cursor pages with per-run dedup and
generation-guarded resets, and useLoadMoreOnScroll drives loadMore from
an IntersectionObserver sentinel (400px prefetch margin, guarded against
double-fetch, observed against the table's scroll container). Footer now
shows the loaded count and the analytics lookback window.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* web: back the runs infinite list with SWR so tab switches serve from cache
Rewrite useInfiniteList on useSWRInfinite: pages are keyed by
[cacheKey, cursor] in SWR's global cache, so unmount/remount (switching
tabs) restores fetched pages instantly instead of refetching. Revalidation
is conservative because analytics list queries are expensive:
revalidateFirstPage and revalidateIfStale are off; freshness comes from
the Refresh button and the visibility-change auto-reload, which map to
reload() (reset to first page + revalidate).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* world: expose startTime/endTime on analytics runs listing
The workflow-server /v2/analytics/runs endpoint has accepted a bounded
startTime/endTime window since it shipped, and is significantly faster
with one (the window prunes the ClickHouse scan: ~2s for 12h vs ~8s for
the default 30-day entitlement window). The world client never exposed
the params, so the CLI and web UI could only issue windowless requests.
Pass them through so clients can send bounded windows (e.g. a period
picker like front's workflows o11y).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* web: front-style period picker for the runs list
Add a time-window picker (1h/6h/24h/3d/7d/30d, default 24h, URL-backed
via ?period=) that sends an explicit startTime/endTime window through
fetchRuns -> world.analytics.runs.list, keeping the ClickHouse scan
bounded. The window is frozen per selection/refresh so all cursor pages
share the same bounds, and it participates in the SWR cache key.
Plan tiers are honored data-driven from the server's pageInfo: presets
longer than the plan's observability lookback are disabled in the picker
(labeled Observability Plus when an upgrade is available), and a 402
observability-upgrade-required response renders through the existing
upgrade-required error handling. The footer now labels the selected
window instead of the plan lookback. The runtime (local) fallback path
ignores the window since the storage API has no time filter.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* web: allow status filtering without a workflow name filter
The status dropdown was disabled on Vercel backends until a workflow was
selected — a limitation of the runtime DynamoDB API's index design. The
runs list now reads via world.analytics, whose ClickHouse query filters
derived status independently of workflowName, so drop the guard.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* cli: time-window flags for runs listing; widen name lookups past the default window
The analytics backend now defaults windowless runs listings to the
trailing 24h. Replicate the web's window support in the CLI:
- 'workflow inspect runs' gains --since/--until (relative durations like
30m/12h/7d/2w, or timestamps) which are sent as an explicit
startTime/endTime window. Out-of-plan windows surface through the
existing observability-upgrade-required handling; non-analytics
backends warn that the flags are ignored.
- 'workflow start <name>' resolves the workflow's latest run via a
windowless (default-window) listing and now retries across the plan's
whole observability window on a miss, so names idle for more than a
day keep resolving.
- Bulk 'workflow cancel' matches across the plan window up front — a run
can sleep or wait on a hook for days without recent events, so the
default recent window must not bound cancellation matching.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* chore: tighten changeset descriptions
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* web: persist frozen listing windows across remounts; minimize lockfile diff
Address review findings:
- The frozen startTime/endTime lived in component state, but RunsTable
fully remounts on tab switches, so every remount minted a new SWR cache
key — the cached-pages restore never hit and cache entries grew
unboundedly (one per key, including every 5s local-backend poll tick).
Move the frozen windows to a module-scope store keyed by period: a
remount reuses the stored window (same cache key, instant restore), and
the window only advances on explicit refresh/reload. Non-analytics
backends now send no window at all (the runtime APIs ignore it anyway),
which also hides the period picker and window label there.
- Regenerate pnpm-lock.yaml from main so the diff contains only the swr
addition (plus its own use-sync-external-store dependency), dropping
the unrelated docs-importer radix-ui re-resolutions.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* feat(world-vercel): send x-vercel-queue-region on proxy-mode queue sends
Token/proxy clients (CLI, dashboards, external scripts) send queue
messages through api.vercel.com's /v1/workflow proxy, where the fixed
resolveBaseUrl override replaces the queue SDK's own
region -> <region>.vercel-queue.com base-URL resolution — so every
proxied send landed on the region-less VQS host, while the direct
in-function path already dials the regional host (iad1 today).
The proxy now routes sends to the region's VQS dataplane when the
x-vercel-queue-region header is present (vercel/api#79056, deployed).
Set it on every proxy-mode send from the region the QueueClient is
constructed with, giving proxy clients parity with the direct path.
Today that region is the static 'iad1'; when per-send region
resolution lands (#1981), the header automatically carries the
tag/option-derived region — this change is the transport mechanism,
not the policy.
Tests: proxy-mode construction carries the header matching the
client's region; the direct path doesn't.
* chore: trim comments, drop private repo reference
* chore: changeset for proxy-mode x-vercel-queue-region header
* Statically inject workflow world target
* Fix static world injection in host bundles
* Fix static world injection gaps
* Fix Vite Nitro server startup
* Fix Nitro pg-native aliasing
* Fix static world target CI gaps
* Fix static world dev rebuild gaps
* Avoid broad runtime alias in Nitro
* Refresh Next dev route for step HMR
* Externalize Nest target world
* Use canary HMR rediscovery timeout
* Bundle local world in Nest builds
* Dedupe world target helpers and fix SvelteKit chunk patch guard
* Fix compressed workflow data display
* Add OSS web no-key hydration regression
* Scope compression normalization to read paths; tidy hydration
Address review feedback on the compressed-data fix:
- world-vercel: keep gzip/zstd decompression on the o11y/display read
paths (getStep/getRun/getEvent/getWorkflowRunEvents/getHook) but not on
the runtime event-append path (world.events.create, createStep,
updateStep). That path is runtime-only and re-hydrates every payload via
the decompress-aware helpers, so decompressing at the adapter was
redundant work on the TTFB-sensitive run_started/inline-delta path and
skewed the runtime's deserialize compression telemetry to `codec: none`.
deserializeStep is now shape-only; normalizeStepData runs in the read
filter. Adds a regression test pinning the write-path pass-through.
- serialized-data: drop dead `errorRef`/`metadataRef` normalization (refs
are descriptor objects, never compressed byte payloads).
- web: in the wait-entity path, filter events by correlationId before
hydrating so an encryption key doesn't decrypt the whole event page.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Peter Wielander <peter.wielander@vercel.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Redrive on transient workflow-server transport failures instead of failing the run
A firewall in front of workflow-server shedding load with sustained 429/503
makes undici's shared RetryAgent exhaust its retries and throw
UND_ERR_REQ_RETRY. That raw error was rethrown unwrapped, so it was
classified as USER_ERROR and the replay terminal branch wrote run_failed —
permanently failing a run on a transient blip (or, in an outage, falling back
to the ~5min queue visibility-timeout redrive).
- world-vercel: map exhausted-retry / socket / connect / DNS / timeout
failures to a typed WorkflowWorldError (code TRANSPORT/TIMEOUT) by walking
the fetch() cause chain.
- core: add isRetryableWorldError (429 / 5xx / TRANSPORT / TIMEOUT) and
rethrow such errors from the replay terminal branch so the queue redrives
quickly (1s->60s backoff) instead of failing the run. Reuse it in start()
and step_started handling.
- world-vercel: surface the Vercel firewall x-vercel-mitigated
(challenge/deny) header alongside x-vercel-id in error diagnostics and logs.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* Address review: back off + cap on every retry path; fix mock; refine scope
Revises the transport-error handling per PR review (VaguelySerious,
karthikscale3).
Blocking fix — step_started no longer self-enqueues a throttled defer for
transient world errors. Returning `{ type: 'throttled', timeoutSeconds: 1 }`
acked the delivery and enqueued a fresh message, resetting the delivery count
so the path never backed off and never reached MAX_QUEUE_DELIVERIES — an
unbounded flat-1s loop if step_started kept failing. It now throws, so the
error flows through the replay loop's retryable-world-error rethrow and earns
both the delivery-count backoff and the max-delivery cap. Throwing is safe on
step_started (the body hasn't run; a write that landed dedupes to skipped).
Also in this revision:
- Backoff that lasts: raise the queue handler-error retry ceiling 60s -> 900s.
VQS clamps each redelivery to its 900s SQS limit and adds its own post-32
exponential, so ramping our base toward 900s stretches survival from ~3.7h to
most of the 24h message-visibility window. Corrected the stale
MAX_QUEUE_DELIVERIES comment to match the real VQS schedule.
- Stop amplifying firewall challenges: the undici RetryAgent no longer retries
429 in-process (a challenge is a 429 the client can't solve). 429s surface
immediately as ThrottleError carrying x-vercel-mitigated / x-vercel-id, so
the diagnostic header now reaches us for the challenge case too.
- Track world faults as WORLD_CONTRACT_ERROR (not USER_ERROR) in
classifyRunError so an outage isn't attributed to user code.
- Fix queue.test.ts mock that `biome check --write` had rewritten from a
newable `function` into an arrow (broke `new QueueClient`); pin with a
biome-ignore.
All Vercel-specific logic stays in @workflow/world-vercel; @workflow/core
operates only on the generic WorkflowWorldError abstraction.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* Update .changeset/transport-error-redrive.md
Signed-off-by: Peter Wielander <mittgfu@gmail.com>
* Trim changeset to a single sentence per review
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* Route firewall-challenge 429s to the retryable transport path, not ThrottleError
A 429 carrying `x-vercel-mitigated: challenge` is a firewall challenge our
server-to-server client cannot solve, so it recurs for the life of the
incident. Mapping it to `ThrottleError` meant the `step_started` write deferred
it as `{ type: 'throttled' }`, which self-enqueues a FRESH queue message and
resets the delivery count — so it never backed off past `retryAfter` and never
reached `MAX_QUEUE_DELIVERIES`, hot-looping against an already-overloaded
firewall (the exact amplification this PR set out to remove, and contrary to
the "step_started can't loop unbounded" invariant, which only held for 5xx).
Map a challenge to a retryable transport `WorkflowWorldError` (`code:
'TRANSPORT'`) in both the v3 `makeRequest` and v4 `throwForErrorResponse`
(the hot event-write path) error mappings, via a shared `isFirewallChallenge429`
helper. It then propagates through the V1/V2 step paths and the replay loop's
retryable-world-error rethrow, earning the delivery-count backoff AND the
delivery cap. A genuine application-level 429 (no `challenge` mitigation) stays
a `ThrottleError` and keeps its `Retry-After`-paced defer.
Also correct the survival-window comments: with the 900s ceiling,
MAX_QUEUE_DELIVERIES=48 spans ~9-10h (~35,000s), not "the better part of 24h";
reaching 24h would need a higher delivery cap, not a higher per-hop ceiling.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Signed-off-by: Peter Wielander <mittgfu@gmail.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Co-authored-by: Peter Wielander <mittgfu@gmail.com>
Co-authored-by: Peter Wielander <peter.wielander@vercel.com>
* fix(world-vercel): retry idempotent event POSTs in-process to avoid step re-execution
undici's RetryAgent never retries a POST, so a transient transport blip (UND_ERR_REQ_RETRY, ECONNRESET, socket/headers timeout, transient 5xx) when committing a step's terminal event bubbles out, the queue redelivers, and the step's user code re-executes with attempt++ even though it already ran to completion.
workflow-server makes these writes idempotent in outcome: entity handlers run before the event-log row is inserted and state transitions are conditional writes excluding terminal states, so a retry whose original landed throws before any row is written and surfaces as a 409 the SDK already handles. This adds a bounded in-process retry (new event-retry.ts) gated by a validated per-event EVENT_RETRY_ELIGIBILITY map, excluding step_started (double-increments attempt), step_retrying (appends a duplicate row), and hook_received (no server guard).
Complements #2666, which routes completion-persistence failures to queue redelivery instead of recording them as user failures; this avoids the redelivery (and re-execution) for transient blips.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(world-vercel): address review on event-POST retry
- Don't retry external cancellation: drop AbortError from the transient set (keep self-timeout TimeoutError), so a caller-requested abort isn't re-issued or stalled by the backoff budget.
- Add DEBUG-gated logging on each retry and on retry exhaustion so an in-process retry vs. a fall-through to queue redelivery is distinguishable in logs.
- Clarify docs: a landed retry surfaces as 409 for most types, but run_started/attr_set return 200 success (not a 409).
- Add createWorkflowRunEvent integration tests (events-retry.test.ts): eventType is threaded into the retry wrapper, and the 404->HookNotFoundError mapping still fires after the retry loop.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Co-authored-by: Peter Wielander <mittgfu@gmail.com>
* otel(world-vercel): inject trace context on v4 event requests
The v4 event path (createEvent / getEvent / listEvents) routes through
fetchV4 → global fetch with a custom undici dispatcher, bypassing both the
makeRequest path (where the explicit W3C trace-context injection lives) and
ambient undici auto-instrumentation. As a result, v4 event traffic from the
flow route carried no traceparent, so workflow-server could not parent its
spans to the invocation — its spans never joined the /flow execution trace,
even though v2/v3 reads/writes (via makeRequest) did join.
fetchV4 now calls injectTraceContextIntoHeaders before fetch, the single
choke point for all v4 create/get/list requests, mirroring makeRequest.
No-op when no OpenTelemetry SDK is registered.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs(agents): require trace-context injection on new world-vercel HTTP paths
Codify the guardrail that the v4 regression revealed: any outgoing
world-vercel request must call injectTraceContextIntoHeaders (auto-
instrumentation can't be relied on with the custom dispatcher / global fetch),
with a test in trace-propagation.test.ts.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* changeset: make v4 trace-propagation note concise
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* perf(core): lazy inline step start to save a world round-trip per step
The owned-inline runtime path used to write step_created (suspension
handler) and then step_started (executeStep) as two separate world
round-trips for a step it already owns and is about to run inline. This
defers the step_created write: executeStep sends a single step_started
carrying the step input, and the world creates the step on the fly
(materializing the step entity plus a synthetic step_created event so
replay still observes it). Mirrors the existing resilient run_started ->
run_created pattern.
Exactly-one ownership is preserved by the world's atomic create-claim:
the loser of a concurrent lazy step_started gets EntityConflictError,
which executeStep maps to `skipped`, so it never runs the body. A lazy
step_started is only ever sent for a brand-new step (the suspension
handler defers only steps with no prior step_created), so crash recovery
still re-runs a `running` step via the normal non-lazy step_started.
Worlds updated: world-local, world-postgres (implicit create + synthetic
step_created event), world-vercel (routes the input as the v4 frame
payload and threads the server's stepCreated flag). @workflow/world adds
optional `input` to step_started and a `stepCreated` EventResult signal.
Rollout: server-first. The matching workflow-server change must deploy
before this ships; the Vercel world targets a single Vercel-operated
backend (server always >= SDK). For local/postgres the world ships in the
same package as the runtime, so there is no version skew.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(core): materialize deferred step before failing unregistered step on lazy inline path
The lazy inline step-start optimization defers a step's step_created write,
expecting executeStep to materialize the step via a lazy step_started carrying
its input. For an UNREGISTERED step, executeStep bails out before sending that
step_started and writes step_failed directly — but the step entity was never
created, so the world's "step must exist" ordering guard rejects the
step_failed and the run wedges (times out).
This regressed the StepNotRegisteredError e2e tests uniformly across every
framework/world (the ghost step never reached `failed`). Fix: on the lazy path,
send the lazy step_started first to materialize the step (entity + synthetic
step_created, keeping replay correct), then write step_failed. The lazy
step_started's atomic create-claim preserves exactly-one-owner: a concurrent
winner makes ours reject with EntityConflictError → skipped, so the failure is
never written twice.
Adds world-level regression tests (world-local, world-postgres) asserting a
lazy step_started followed by step_failed marks the step failed.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
* perf(core): skip per-step events.list via inline event-log delta
In the inline sequential loop, the runtime re-read its own just-written
step events with an incremental events.list every iteration — pure
latency on the Vercel world. Add an opt-in CreateEventParams.sinceCursor
so a step-terminal write can return the event-log delta since that cursor
(EventResult.events/cursor/hasMore), and have the inline loop consume it
in place of the fetch.
The delta is computed identically to events.list against the same log, so
the consumed prefix is byte-for-byte what a fetch would return. The fast
path is gated conservatively to the single-step sequential case with no
open hooks/waits (so no out-of-band hook_received/wait_completed can land
in the snapshot→replay window), and falls back to the normal fetch on any
World that does not return a delta. world-local implements the delta;
world-vercel/world-postgres are unchanged and fall back.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* perf(world-vercel): forward sinceCursor over the v4 wire for inline delta
Adds `sinceCursor` to the v4 POST frame meta so a step-terminal write can
ask the server for the authoritative event-log delta on the response
(events/cursor/hasMore), letting the inline loop skip a follow-up
events.list. The server-side computation ships in
vercel/workflow-server#538; older servers ignore the field and the
runtime falls back to events.list (no behavior change).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* test(inline-delta): cover truncated multi-page delta -> hasMore fallback
The inline-delta query in world-local intentionally omits a `limit`, so a
delta larger than one page is truncated and reports `hasMore: true`. The
runtime consume gate only stashes a delta when `!hasMore` and otherwise
falls back to the exhaustive `events.list` loop, so a partial page can
never be consumed as the complete delta.
Make that contract explicit with a comment at the query site, and add
tests pinning it: a world-local test proving the delta truncates and
surfaces `hasMore: true` byte-identically to `events.list(sinceCursor)`,
and an executeStep test proving `hasMore: true` is threaded verbatim.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* docs(world): clarify sinceCursor returns the first delta page, not the full set
The CreateEventParams.sinceCursor docstring said the result is "exactly
the delta an events.list(...) call would return," which read as the full
set. It is the first page of that delta; hasMore signals more. Spell out
the single-page-or-fallback contract so other World adapters implement
sinceCursor consistently, and note that an in-band burst larger than one
page bypasses the fast path (correct, but forgoes the saved round-trip).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Karthik Kalyan