mirror of
https://github.com/vercel/workflow.git
synced 2026-09-14 19:59:43 +08:00
workflow-scale-limits-doc
419 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 vercel[bot]
|
57dce98fe4 |
[docs] Ship the World pages in the bundle, and fix two stale claims
The run-limit guidance added here hangs off `/worlds/vercel#per-run-limits`, and the skill sends agents to `node_modules/workflow/docs/`. But `docs/content/worlds/` is a sibling of `docs/content/docs/`, so `packages/workflow`'s prepack never copied it: every `/worlds/...` link in the bundled pages was a dead end for an agent, including the hub this guidance points at. Copy `worlds/v5` into `docs/worlds/` and list it in the skill's documentation-structure map. Two claims contradicted pages this PR already touches: - `child-workflows.mdx` opened by calling a 500-child spawn practical, while its own Tips and its "chunked spawning" section say to start them in chunks of 10-50. Point the bullet at that section. - `whats-new.mdx` said the event ceiling can be tuned "as a fallback", but `runtime-tuning.mdx` states that `WORKFLOW_MAX_EVENTS` does not override the Vercel World's service-owned value. Scope the claim to the Worlds where it is true. Guard: assert prepack copies the worlds tree and the skill lists it, and extend the anchor check to same-page `(#anchor)` jumps. Both fail when the change is reverted. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Signed-off-by: Pranay Prakash <1797812+pranaygp@users.noreply.github.com> Co-Authored-By: Pranay Prakash <1797812+pranaygp@users.noreply.github.com> |
||
|
vercel[bot]
|
631f1035a0 |
[docs] Correct the run-limit guidance against the published limits
Review of the guidance added here turned up several claims that disagree with https://vercel.com/docs/workflows/pricing#workflow-run-limits, the page this change designates as the single source of truth. - Steps per run are capped too. The skill said "deep runs are fine — a long sequential chain of steps is not itself a problem", but the pricing page publishes a separate steps-per-run ceiling, which the v5 comparison tables already cite. Size all three axes, not two. - The ceilings are raisable. The skill said "raising the limit is not an option"; the pricing table annotates both the event and step limits with "reach out if you need a higher limit". The env-var reasoning was right, the conclusion was not. - Split threshold. Six places recommended child workflows at "tens of thousands of events". The pricing page recommends them past a few thousand, because replay slows down long before the run fails — and since the ceiling itself is tens of thousands, the old advice fired once the run was already failing. Calibrate to the degradation point. - step_retrying is optional. "A retry adds a step_retrying and another step_started" contradicted the event table eight lines above it, in the one section whose purpose is counting events. Guard test: - Resolve anchors. Renaming the Vercel World heading previously broke four inbound links with the suite still green; two new cases catch it. The section is now "Per-run limits", since it covers steps as well. - Close the literal bypass. The forbidden spellings were derived from the Local World's SDK default, which only happens to match the managed ceiling; deriving from it meant an SDK bump would silently re-point the guard. They are now literal, cover the "25K" form, and span every docs tree rather than v5 alone. - Cover sequential-and-parallel.mdx, which presents the ceiling but was missing from the list. - Replace the "~1000 steps in flight" assertion. That figure was "~100" two commits ago and has no published source; the guard now rejects unsourced steps-in-flight thresholds instead of pinning one. Skill snippets now typecheck: processItem and handle were undefined, and the bundled example was never called from a workflow. Signed-off-by: Pranay Prakash <1797812+pranaygp@users.noreply.github.com> Co-Authored-By: Pranay Prakash <1797812+pranaygp@users.noreply.github.com> |
||
|
Peter Wielander
|
93f7ffc185 |
Update docs/content/docs/v5/cookbook/advanced/child-workflows.mdx
Signed-off-by: Peter Wielander <mittgfu@gmail.com> |
||
|
Peter Wielander
|
55a65c5327 |
Update docs/content/docs/v5/configuration/runtime-tuning.mdx
Signed-off-by: Peter Wielander <mittgfu@gmail.com> |
||
|
Peter Wielander
|
ac4ccf3c31 |
Update docs/content/docs/v5/cookbook/common-patterns/sequential-and-parallel.mdx
Signed-off-by: Peter Wielander <mittgfu@gmail.com> |
||
|
vercel[bot]
|
5f56d28ee0 |
[docs] Defer run limits to the pricing page, and compact the guidance
Review feedback, applied: - Drop the `foundations/workflows-and-steps` section entirely (@pranaygp: doesn't belong in foundations that early). Nothing links to it now. - Remove the claim that step boundaries get slower past ~100 steps in a run. @pranaygp is right that this is wrong — durabench STSO measurements show deep 1000-step runs are fine — so it should not be published. What is left is the accurate part: a wide fan-out is about *concurrency*, and the fix is to batch or bundle work into fewer, larger units. The docs also no longer suggest one child run per item as a concurrency fix, since that fan-out is exactly as wide (@pranaygp again). - Stop stating any event or parallelism number in user-facing docs. The published Workflow run limits are the source of truth; pages now link there or to the Vercel World's section (@VaguelySerious). - Recommend child workflows when a run is expected to accumulate tens of thousands of events, and batching/bundling when concurrency is high. Two pre-existing pages also stated "25,000 on the Local and Vercel Worlds" in prose (`whats-new`, `foundations/errors-and-retries`); both now defer to the pricing page for the Vercel World. The `configuration/` pages keep the number, because there it documents `WORKFLOW_MAX_EVENTS`, an SDK-owned env default for the Local World rather than the managed ceiling. The skill is updated separately, since it is agent-facing rather than published: it keeps a "~1000 steps in flight" rule of thumb for when to batch or bundle, and defers to the pricing page for the event ceiling. Replaces the old guard test, which pinned six pages to the literal "25,000" and would now fail (as the Vercel Agent review flagged). The new guard enforces the invariant the reviewers actually asked for: pages that present the ceiling must link the pricing page, user-facing docs must not hardcode the runtime's ceiling outside the env-var reference, and the ~1000 heuristic must stay in the skill. Signed-off-by: Peter Wielander <29887157+VaguelySerious@users.noreply.github.com> Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Peter Wielander <29887157+VaguelySerious@users.noreply.github.com> Co-Authored-By: Peter Wielander <29887157+VaguelySerious@users.noreply.github.com> |
||
|
Peter Wielander
|
c82673a3c7 |
Apply batched suggestions from code review
Co-authored-by: Peter Wielander <mittgfu@gmail.com> Signed-off-by: Peter Wielander <mittgfu@gmail.com> |
||
|
vercel[bot]
|
30b9e88968 |
[docs] Document the 25,000-event run ceiling and the ~100-step fan-out limit
Agents building on the SDK had no documented answer to "how big can one
run get?", so they reach for a single run with one step per input row and
one wide `Promise.all`. Both limits that make that a bad shape were only
visible in the source: `DEFAULT_MAX_EVENTS_PER_RUN` (25,000, enforced by
the Vercel World and surfaced as `MAX_EVENTS_EXCEEDED`), and the fact that
replay cost grows with the event log, which is what makes a wide fan-out
stop paying for itself somewhere around 100 steps in a run.
Both now have a canonical home in `foundations/workflows-and-steps`
("How much work fits in one run"), which the Vercel World page, the
event-sourcing guide, and the batching / sequential-and-parallel /
child-workflows recipes link to. The `workflow` skill gets the same
guidance as a table plus the three shapes (flat fan-out, bounded batches,
child workflows), and the `sequential-and-parallel` CopyPrompt now tells
the agent to bound its fan-out.
Two things the docs were quiet about and are worth stating plainly:
events are not steps — three per successful step, so 25,000 events is
reached at roughly 8,000 steps, not 25,000 — and raising the limit is not
an option, since the override only clamps down and the Vercel World's
ceiling is service-owned.
A guard test pins the documented figure to the enforced one: it reads the
constant out of `world` and `world-sim`, asserts they agree, and asserts
every page that quotes the ceiling quotes that number.
Signed-off-by: Peter Wielander <29887157+VaguelySerious@users.noreply.github.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Peter Wielander <29887157+VaguelySerious@users.noreply.github.com>
Co-Authored-By: Peter Wielander <29887157+VaguelySerious@users.noreply.github.com>
|
||
|
Shalabh Chaturvedi
|
c129332923 |
[docs] Fix prose typos across v4/v5 docs and the SWC plugin README (#3948)
Copy-edit only, no behavior or API changes: - "it's a only a short step" -> "it's only a short step" (ai/index) - "When you tool needs" -> "When your tool needs" (ai/defining-tools) - "Workflow operation that suspend" -> "operations that suspend" (ai/sleep-and-delays) - "extend out ... to use emit" -> "extend our ... to emit", and "other tools calls ... inject out own" -> "other tool calls ... inject our own" (ai/streaming-updates-from-tools) - "non-yet-standard" -> "not-yet-standard" (how-it-works/understanding-directives) - "apps ... and needs no special configuration" -> "and need no special configuration" on the five getting-started pages that disagreed with the other five (express, fastify, hono, nuxt, vite) - drop the orphan "needed." line after "No separate command is required." and fix "the local installed version" -> "the locally installed version" (observability/index) - "Time between emissions of a chunk" -> "emission of a chunk" (observability/tracing) - "determine that is safe" -> "determine that it is safe" (whats-new) - "three rules bind an implementation" -> "four rules": the list has four bullets, and skills/migrating-world-v4-to-v5 already says four (worlds/upgrading-to-v5) - drop the stray duplicate "Workflow" before the Workflow SDK link in @workflow/swc-plugin's README, and the duplicated horizontal rule before "## Detect mode" in its spec Each fix is applied to both the v4 and v5 copies wherever the same text exists in both. Co-authored-by: vercel[bot] <35613825+vercel[bot]@users.noreply.github.com> |
||
|
Karthik Kalyan
|
f9073d0739 |
Add attribute inspection to the CLI (#3950)
* Add attribute inspection to the CLI and probe the cancel window once
`wf inspect attributes` lists the distinct attribute keys on a project's
runs with their run counts and first/last seen times, and
`wf inspect runs --attribute key=value` filters by them. Between them
they turn attributes from something you can only write into something
you can discover and query. Both are analytics-only — storage has no
cross-run attribute index — so the listing says so rather than falling
back, and the filter warns and is ignored the way --since/--until
already do.
The flag is parsed and bounded in lib/inspect so the error names
--attribute rather than the parameter it becomes, and so it is testable
next to the other inspect flag helpers. It splits on the first `=` only,
since a value may contain one, and keeps an empty value, which matches
runs whose attribute was set to the empty string.
`wf cancel` also probed the plan's listing window inside its per-status
fan-out, so a four-status cancel issued four identical probes. The
window is a property of the plan rather than of a status, so the probe
is hoisted above the fan-out: eight requests become five. The harness
only ever modelled the storage path, so that probe logic had no
coverage; the new test fails with two probes before the change.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* Do not depend on an unreleased world export for the flag cap
The --attribute cap was imported from @workflow/world, where the
constant is added by a different branch, so on main it resolved to
undefined and `values.length > undefined` was always false: the flag
accepted any number of pairs and the test for it never threw.
Declare the cap in the CLI instead. The World and the backend enforce
the same bound independently, and this copy exists only so the error can
name the flag the user typed rather than the parameter it becomes.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* Degrade sleeps to the event log and bound the inspect flags
`wf inspect sleeps` was the only list path that could not degrade: it
branched on analytics being present and either returned or exited, so on
any backend providing analytics the storage branch below it was
unreachable and an analytics failure ended the command. It now warns and
falls through, like the run, step, and event listings. An argument the
World rejected is not retried — the same argument fails either path, so
falling back would trade a precise message for a slower failure.
handleApiError also only recognised errors carrying an HTTP status.
A client-side argument rejection has none, because no request was made,
so it fell past every branch and was rethrown as an unhandled error. It
is now reported as given: the message already names the method, the
parameter, and what it received.
--limit and --runId are checked before any backend setup so a mistyped
value names the flag and costs no round trip. The limit bound is
deliberately looser than the per-endpoint caps, which differ by resource
and stay with the World; this one catches a typo'd digit or a negative.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* Scope --attribute to inspect and document the inspect flags
--attribute was added to the shared cliFlags, which cancel, health,
start, and web all spread — so `workflow health --attribute k=v` parsed
and was silently ignored. It belongs with the other inspect-only
filters in the command's own flags, next to --runId and --since.
The configuration reference documented every shared flag but none of
the inspect-only ones, so --runId, --stepId, --hookId, --since/--until,
--withData and --decrypt had no entries at all. They now do, in an
Inspect filtering section, alongside --attribute. --status and
--workflowName were documented under bulk cancel only; both also filter
inspect listings, which is now noted where they are.
--limit's entry described a default with no bound and is now rejected
outside 1 to 1000, so it says so, and points out that individual
listings cap lower.
The attributes guide claimed filtering was available "through the
Analytics API", which is no longer the whole story: the CLI can now
discover keys and filter by them, so that section splits into a CLI half
and an API half.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* Stop dropping inspect flags silently
Three flags the caller typed were being discarded without saying so —
the same failure the World argument guards were added to remove,
reintroduced one layer up.
--attribute and --since/--until warned that the backend has no
analytics read path, but that condition is also false when --withData
asks for payloads, which only storage carries. Blaming the backend for
the caller's own flag sends them looking in the wrong place, so the
warning now names whichever applies.
inspect attributes dropped --sort entirely, explained only by a code
comment. It is forwarded now, and still left unset when absent so the
backend's alphabetical key order stands rather than the `desc` the
time-ordered listings impose.
A repeated --attribute key silently kept the last value, and a test
asserted that as if it were intended. Matching is per-key, so resolving
it means discarding a filter the caller typed: it is rejected instead.
The shared --limit entry also stated the 1-to-1000 bound that only
inspect enforces, which is wrong for cancel's own 1-to-500. The bound
moves to an inspect entry and the shared one points at both.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* Reject --attribute on listings that cannot use it
Only the runs listing filters by attributes, but the flag was parsed for
every inspect resource: `inspect steps --attribute tenant=acme` returned
a normal, unfiltered step list with no warning, as did events, hooks,
attributes, and `inspect run <id>`, which already names one run. That is
the silent drop the preceding commit set out to remove, missed one layer
up in the command itself.
Validated alongside the other flag bounds, before any backend setup, so
a flag on the wrong subcommand costs no round trip.
Covered at the command level as well as in the unit, since the defect
was not in the validator but in nothing calling it: the tests drive
`Inspect.run` with a mocked setup module and assert the backend is never
reached. Five of them fail without this change.
Reported in review; verified against a real project rather than found
by the suite, which is why the command-level coverage goes in with it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* Resolve the test's oclif root without a URL pathname
`new URL('../..', import.meta.url).pathname` yields `/D:/a/...` on
Windows — a leading slash before the drive letter — so `Config.load`
could not find package.json and every command-level test failed there
while passing on Linux. `fileURLToPath` handles both.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* Address review on the attribute flag
Attribute keys naming an Object.prototype member were rejected as
duplicates before anything was stored, because the duplicate check used
`in`, which walks the prototype. `--attribute toString=v` failed on
first sight, and `__proto__=v` would have set the prototype rather than
stored a value had it got that far. The map is null-prototype now and
the check uses Object.hasOwn.
--url and --web return before the filter is parsed, and neither
forwards it, so `inspect runs --attribute k=v --url` opened an
unfiltered view and a malformed pair skipped validation entirely. Both
are rejected: the dashboard takes no attribute filter.
--sort carried an oclif default of desc, so the "forward only when
asked" check in the attribute listing was always true and overrode the
backend's alphabetical key order. Every time-ordered listing already
falls back to desc itself, so the parser-level default is gone and the
flag now means what it says.
The docs claimed --since and --until must be supplied together, but the
CLI resolves the pair before the World sees it: --since alone is valid
and --until defaults to now. Only --until alone is rejected.
The vercel[bot] comment about ANALYTICS_MAX_ATTRIBUTE_FILTERS not being
exported was already addressed in
|
||
|
Pranay Prakash
|
7cc5c88a8b |
[core] Settle a hook's awaiter in-process instead of re-invoking, on creation and on conflict (#3938)
* [core] Settle a hook's awaiter in-process instead of re-invoking, on creation and on conflict * [core] Address review: deterministic hook signal tests, split changesets, document the boundary - hook.test.ts: drive the idle poll with explicit macrotask turns instead of a fixed 20ms sleep (Copilot) - Split the changeset so each package's entry says only what changed in it - runtime-tuning docs: hook-only suspensions no longer always park; the hook write continuation is the one exception Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com> |
||
|
Karthik Kalyan
|
fbfb9fe869 |
Validate world.analytics arguments up front (#3943)
* Validate world.analytics arguments up front
Every analytics method now checks its arguments before making a request
and throws a RangeError naming the limit it broke: the ids, the
pagination limit against the cap for that listing, and the attribute
filter's pair count, key length and value size. Because analytics is an
optional capability, callers wrap it in a catch, which turned an invalid
argument into what looked like an empty result rather than an error.
Two arguments that used to be dropped silently now fail too. A limit of 0
fell back to the default page size, and a startTime without a matching
endTime turned a listing you meant to bound into an unbounded one that
looked like a normal answer.
Exports ANALYTICS_RUN_SCOPED_PAGE_LIMIT, ANALYTICS_PAGE_LIMIT and
ANALYTICS_MAX_ATTRIBUTE_FILTERS so callers can check the bounds
themselves.
Deprecates analytics.events.listByCorrelationId() in favour of
analytics.events.list({ runId, correlationId }), which issues the same
request and also accepts an eventType filter. It keeps its own
implementation rather than delegating: list() treats correlationId as
optional and skips an empty one, so a delegation would turn an empty id
into an unfiltered listing of the run.
Documents every analytics method in the reference. events.getMany() was
missing entirely, seven methods shared one code block with no parameters
or return shapes, and none of the limits were written down.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* Parse attribute key timestamps as UTC
firstSeenAt and lastSeenAt were the only analytics timestamps still on a
plain date coercion. The values arrive without a timezone designator, so
that read them in the process's local zone and every other field in the
namespace read them as UTC — a seven-hour skew on those two fields alone
for anyone running outside UTC.
The added test fails without the fix under TZ=America/Los_Angeles.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* docs: drop the deprecated correlation-id listing from the reference
A reference page should describe the API you should reach for. The
deprecation notice lives on the method itself, so editors surface it
where it matters without the page advertising a method nobody should
start using. Also drops it from the page-limit table.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* Close two gaps in the analytics argument guards
Run ids were validated with workflowRunIdSchema while every other id used
a regex mirroring the backend. Those disagree: z.ulid() accepts a
lowercase body and a first character above 7, and the backend accepts
neither, so the most-used parameter had the leakiest guard and still
produced the 400 this is meant to prevent. Run ids now use the same
pattern as the rest.
A supplied-but-empty filter value was also still being dropped —
correlationId, the optional runId scope on hooks.get, and workflowName
all tested truthiness. Dropping one widens the result set rather than
narrowing it, so an empty correlationId listed the whole run and an
empty workflowName listed every workflow. That is the same failure the
limit and time-window guards were added to prevent, and the comment on
listByCorrelationId already described the hazard. They now compare
against undefined, so an empty id throws and an empty name is forwarded
for the backend to match.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* Raise argument rejections as a typed, non-retryable error
The guards threw bare RangeErrors, which left a caller — or an agent
driving this API — parsing English to decide whether to fix the call or
retry it. They now raise WorkflowWorldError with
code: 'INVALID_ARGUMENT', the code the rest of this client already uses
for its transport and throttle failures, so the retry decision is a
field lookup. normalizeEventIds moves with them rather than staying the
one guard that throws a different type.
Also sharpens the four messages that made a caller do the work:
a half-open window now names the bound that is missing rather than
restating the rule, an inverted window prints both ends, and the
attribute-value and event-id batch errors report the size measured
rather than only the bound they broke.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* Name the method and the field on an argument rejection
Two things a caller could not get without reading prose. The same guard
runs behind several methods, so `runId must be a workflow run id` was
identical whether it came from events.list or steps.get — fine with a
stack, lossy once the error has crossed a log line. And the offending
argument was only available as the first token of the message, which is
the part most likely to be reworded.
Messages now open with the method, and WorkflowWorldError carries an
optional `field`:
analytics.runs.list: pagination.limit must be an integer between 1
and 100 (received 9999)
→ code: 'INVALID_ARGUMENT', field: 'pagination.limit'
`field` is additive on the error class and set only by these guards, so
nothing that reads the existing properties changes.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
||
|
Fantix King
|
aba7c8d5d0 | docs: expand Python workflow guide (#3923) | ||
|
Nathan Colosimo
|
3c08778905 |
[core] Retain workflow VMs across waits (#3892)
* perf(core): retain workflow VMs across waits * test(core): cover retained wait wake races Signed-off-by: Nathan Colosimo <110621881+NathanColosimo@users.noreply.github.com> --------- Signed-off-by: Nathan Colosimo <110621881+NathanColosimo@users.noreply.github.com> |
||
|
Michael J. Sullivan
|
40bed1777a |
python docs: add some missing material (#3875)
This merges a bunch of stuff currently on the vercel.com docs. I'm going to go remove those next to centralize things for now. |
||
|
Karthik Kalyan
|
2668e3325b |
Durable hook resume: write, then wake (#3841)
* test(core): reproduce lazy resume disposal race * Fix durable hook resume race * Fail closed on unknown hook wakes * Improve unsupported hook wake diagnostics * Address durable hook resume review feedback * Harden producer-committed wake handling * Serialize durable hook resume: write, then wake resumeHook() now dispatches strictly serially: the hook_received event is made durable first, and the workflow wake is published only after the write is acknowledged. The wake is a plain runId message (the shape the sequential path always published), so the producer-committed wake barrier, its queue-message field, and the HOOK_RESUME_INPUT_VERSION bump are all removed — no consumer or backend coordination is needed, and either side rolls back independently to today's behavior. The pre-write ops flush now partitions serialization ops: producer-push uploads are awaited before the event commits (the payload must not point at bytes still in flight), while consumer-settled reader ops — a dehydrated WritableStream, e.g. a manual webhook's responseWritable — are backgrounded. Awaiting those deadlocked the resume against its own wake (webhookWorkflow failing across the whole e2e matrix). Also: wake retries stop on definitive 4xx errors instead of burning the retry budget; WORKFLOW_DISABLE_LAZY_HOOK_RESUME no longer gates anything and is ignored; the internal resumeHookDurable alias is removed. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * Address review: retry classification, wake dedup, 409 passthrough - Wake retry classification now actually fires against @vercel/queue: its errors carry no status field, so classify by the World's deployment-unavailable hook, then numeric status, then the queue client's definitive-4xx error names. - The wake publish carries idempotencyKey `hook-<resumeId>` on the claim path, so a retried publish whose response was lost dedups instead of costing a duplicate full replay. - EntityConflictError (HTTP 409) from the durable write is no longer re-keyed to HookNotFoundError: every 409 the backend emits on this write today is transient (slot conflict past the server's retry budget, claim race) and committed nothing, so it surfaces retryable instead of presenting as a permanent 404. - Stamp workflow.hook.resume_committed / wake_published span attributes after each leg resolves, making stranded resumes (committed event, no wake) queryable from traces. - Document on the public resumeHook signature that passing the token (not a cached Hook) is what makes the write idempotent-on-retry. - Changeset/changelog: note the ended-run behavior change (late webhook deliveries to finished runs now 404 instead of 202) and the 409 passthrough. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Pranay Prakash <pranay.gp@gmail.com> Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Pranay Prakash
|
ffc58078d0 |
Stop logging on healthy workflow execution (#3878)
A successful run printed several lines that described the runtime working correctly. Most of it was fallout from defaulting the events transport to WebSockets (#3702): three breadcrumbs written while the transport was opt-in became default-path output, because each one reported a choice the caller no longer makes. - `world-vercel: using ws events transport (…)` ran once per cold start on every deployment, naming the transport it was always going to use. - The `projectConfig` proxy fallback warned once per process. That World cannot hold a socket, so with WS on by default every CLI command and the observability app warned about a fallback nobody asked for and nobody can act on. Debug-gated and reworded from "requested but" to "unavailable for". - The `max_duration` / `auth_expiry` drain notice is routine: the transport reconnects from the close that follows and no write is lost. Swept for the same shape elsewhere: - `world-local`'s queue-concurrency notice fired per message once a fan-out exceeded the limit — the semaphore doing its job. - `@workflow/world`'s active-run recovery line printed on every dev-server restart with work in flight. The re-enqueue *failure* above it stays unconditional; that one leaves a run unresumed. - The port-detection diagnostics in `@workflow/utils` keyed off `NODE_ENV=development`, which is the only environment that reaches them, so the gate made them unconditional for their whole audience. All of it moves behind `DEBUG=workflow:*` via a new `debugLog` in `@workflow/utils`, joining world-vercel's existing `httpLog` and `logRetry` output under one selector. Warnings and errors are untouched, so a run that actually goes wrong is no quieter than before — the ws-transport tests that assert failures are never silent still pass unchanged. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Co-Authored-By: Pranay Prakash <1797812+pranaygp@users.noreply.github.com> |
||
|
Peter Wielander
|
1c44cc8c3f |
[world-vercel] Fail the run on a lost event payload instead of retrying forever (#3742)
* fix(world-vercel): fail the run on a lost event payload instead of retrying
A frame stream that dies mid-body reaches us as a truncated response, which
is exactly what a dropped socket looks like. So an event whose stored payload
is permanently gone was indistinguishable from a transient blip, and the
runtime kept redelivering a replay that could never succeed: one run re-read
a single missing payload 12,932 times in 26 minutes, and the backend query
behind each attempt throttled its table.
The World now sends a terminal `{_error: 1, code}` frame for failures that a
retry cannot fix. Handle it:
- `payload-missing` raises `CorruptedEventLogError`, so the run fails with
`CORRUPTED_EVENT_LOG` rather than looping. The log does reference a payload
nothing can produce.
- An unknown code raises a `WorkflowWorldError` with no retryable code and no
status, which is also terminal. A future code stays safe without needing a
client release first.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* Revert the world-vercel URL override to empty
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* fix(world-vercel): classify terminal stream errors
Signed-off-by: Alex Langenfeld <alex.langenfeld@vercel.com>
---------
Signed-off-by: Alex Langenfeld <alex.langenfeld@vercel.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Alex Langenfeld <alex.langenfeld@vercel.com>
|
||
|
Peter Wielander
|
cc6eb7e837 | [world-vercel] Fix deploymentId "latest" resolving against the wrong team (#3844) | ||
|
Peter Wielander
|
d9e0777eb8 | [core] Never write hook_received eagerly on the lazy resume path (#3794) | ||
|
Nathan Colosimo
|
556f3f080a |
[core] Retain workflow VMs across hooks (#3604)
* Retain workflow VMs across hooks * Refine retained VM decisions and diagnostics * Fix hook suspension assertion * Harden retained hook race coverage * Simplify retention blocker log metadata * Preserve workflow suspension compatibility * Bound retained VM serialization diagnostics * Clarify bounded serialization diagnostics |
||
|
christopherkindl
|
91eb1ae924 |
chore(docs): use geistdocs 1.23.1 (#3777)
Co-authored-by: Peter Wielander <mittgfu@gmail.com> Co-authored-by: Peter Wielander <peter.wielander@vercel.com> |
||
|
Nathan Colosimo
|
d62b44473b |
[core] Prune schema modules from workflow bundles (#3550)
* [core] Prune schema modules from workflow bundles * [world] Inline one-off validation options * refactor(world): simplify event schema boundaries * refactor(world): simplify event schema boundaries * fix(world): keep noop metadata schema-free * refactor(world): drop zod 4.4 compatibility * test(builders): cover workflow API bundle boundary |
||
|
Peter Wielander
|
7e48e7b4de |
Re-enable the sealed log by default (#3737)
* Revert "[world] Make the sealed log opt-in instead of default-on (#3735)" Reverts |
||
|
Shalabh Chaturvedi
|
dc68611fbf |
Default the events transport to WebSockets (#3702)
* Default the events transport to WebSockets WORKFLOW_EVENTS_TRANSPORT=http is the opt-out. Only that exact value disables it, so a typo'd or empty value fails toward the default rather than quietly pinning a deployment to HTTP. The prerequisite the gate named for defaulting on is met: postEventFrameOverWs opens a client span per frame. What is still missing is Vercel's outgoing-requests view, which reads instrumented fetch calls rather than spans and so cannot show a transport that issues no request. Co-Authored-By: opencode <opencode@vercel.com> Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * docs: WORKFLOW_EVENTS_TRANSPORT defaults to ws Three places still documented http as the default. Each now states the opt-out is the exact value http, rather than leaving 'default: ws' to imply that anything non-ws disables it — the asymmetry is deliberate in the code and is the part a reader would otherwise get wrong. Also drops 'Experimental' from the Vercel World page: a setting that is on for everyone by default is not opt-in experimental, whatever else it is. Co-Authored-By: opencode <opencode@vercel.com> Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * Fix the gate's own unit tests for the flipped default Five tests in ws-transport.test.ts still encoded the opt-in semantics. Three were the isWsEventsTransportEnabled table itself; the other two (openWsChannel 'does nothing when the gate is off', and the channel release equivalent) relied on the suite's ambient unset environment meaning 'off', which it no longer does. Both now set http explicitly. Two tests in ws-transport-spans.test.ts asserted HTTP-side span behaviour the same way. The write one would have kept passing by falling through resolveWsTransport's null rather than because the gate was off - passing for the wrong reason, which is what this file exists to catch. Also makes the opt-out case-insensitive and trimmed. The gate is deliberately asymmetric - unrecognized values take the default - but that asymmetry should not extend to swallowing HTTP or ' http '. Whoever reaches for the escape hatch is plausibly mid-incident, and silently ignoring their opt-out over a capital letter is the same class of silent-wrong-transport bug this flip is meant to stop shipping. 554 tests pass in packages/world-vercel. Co-Authored-By: opencode <opencode@vercel.com> Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * ci: add a required forced-HTTP e2e lane (#3703) Flipping the default makes e2e-vercel-prod a WebSocket lane: it sets no WORKFLOW_EVENTS_TRANSPORT, and unset now means ws. Nothing in the file would exercise the HTTP events transport against a real deployment any more, so this is not additive coverage — it replaces coverage the flip silently removed. Unconditional and required rather than label-gated like the WS lane. HTTP is now the fallback, and the fallback is silent: resolveWsTransport returning null costs a write nothing and logs nothing, which is the shape of the durabench bug this stack came out of. Two apps rather than the WS lane's four, since every row is a real vercel deploy charged to every PR. nextjs-turbopack is the only fixture emitting OTEL spans, so it is the one that can show which transport actually ran; express covers the non-Next server path. Also corrects the WS lane's docblock, which claimed every other job exercises HTTP only. That stopped being true one commit ago. Co-authored-by: vercel[bot] <35613825+vercel[bot]@users.noreply.github.com> Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * Fail loudly when step_completed falls back to HTTP under a strict flag The WS e2e lane asserts that the transport is harmless, not that it is used: an event written over HTTP produces the same run outcome as one written over the socket, so the lane stayed green through the entire period the transport was silently demoted. WORKFLOW_INTERNAL_EVENTS_TRANSPORT_STRICT turns that one case into a failed run, and the WS lane now sets it. Scoped to step_completed alone, because most fallback is legitimate: run_created is written outside any invocation that opens a channel; run_started routinely lands before the channel is registered (34% HTTP on a healthy deployment); step_created and wait_created mostly fold into events.createBatch, which is not wired to the socket; and a write after the invocation released its claim falls back by design. step_completed is issued after a step body has run, and was 100% ws across every WS-enabled deployment measured on two SDK versions. The flag reads as off unless the value is exactly 1 or true - the opposite asymmetry from the transport gate, which treats an unrecognized value as on. That gate risks a deployment sitting quietly on the wrong transport; this one fails runs, and should not be acquired by a typo. Co-Authored-By: opencode <opencode@vercel.com> Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * ci: run the WS transport lane on every PR It was opt-in behind ws-transport-test because four real vercel deploys were too much to charge an unrelated PR for a transport that was off by default. Flipping the default expires that reasoning from both ends: the cost is no longer for someone else's feature, and this is now the only lane that asserts the socket carried the events. e2e-vercel-prod inherits the new default but checks nothing, so behind a label the average PR would move every deployment onto WebSockets with nothing verifying they were used. Drops WS_REQUIRED from the gate along with it. That existed only to let the lane be legitimately skipped on an unlabelled PR; with no label the lane is required unconditionally, like e2e-vercel-prod and the HTTP lane, and the skipped case is now a failure rather than a warning. Gate script extracted and run against the cases that matter: ws skipped fails on a standard PR, ws skipped fails under workflow-server-test, and all-green passes. Co-Authored-By: opencode <opencode@vercel.com> Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * ci: widen the HTTP transport lane to six server shapes Before the flip, HTTP was the default and all 28 e2e-vercel-prod lane-runs covered it. After the flip they cover WebSockets instead, and this lane is the entirety of the HTTP coverage - two apps was too thin for a transport that is still supported. Six, not the full 14, because every row is a real vercel deploy charged to every PR. Chosen by server shape rather than count: example (baseline), nextjs-turbopack (Next, and the only fixture emitting OTEL spans), vite (Vite SSR), express (Node req/res), nitro (h3, also covers nuxt) and hono (fetch-API Request/Response, a different mount shape from express). The rest duplicate a shape already covered; python is left out because it has no conformance gate and needs routes this suite does not serve. The first four match the WS lane's matrix on purpose, so the same fixture runs on both transports and a failure on one can be read against the other. Project ids and slugs are copied from e2e-vercel-prod and verified equal to it; both lanes already use the same team and token. Co-Authored-By: opencode <opencode@vercel.com> Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> --------- Co-authored-by: vercel[bot] <35613825+vercel[bot]@users.noreply.github.com> |
||
|
Pranay Prakash
|
f771585486 |
fix(world-vercel,world-local): hold process-wide state on globalThis (#3728)
* fix(world-vercel,world-local): hold process-wide state on globalThis Both packages are bundled into the host application's server build, and a bundler keys module identity on (resource, layer) — Next.js alone builds `instrument`, app-route, `ssr` and `edge` layers, so one process holds one copy of each of these modules per layer. Every module-scope `const`/`let` in them was therefore per-copy state wearing the costume of a process singleton. vercel/workflow#3493 made `@workflow/world-vercel` bundled rather than external and the events WebSocket transport regressed to HTTP for exactly this reason: the queue consumer registered its channel in the `instrument` copy's `Map` and the write path looked it up in the route copy's empty one. A deterministic miss, for the life of the process. `@workflow/world-local` had the same exposure all along — including `runFileLocks`, where a duplicated mutex simply stops mutually excluding. Add `globalSingleton()` to `@workflow/utils` (the primitive `@workflow/core` already hand-rolls for its World cache) and route every mutable module-scope binding in both worlds through it. Regression cover, in three layers: - `global-singleton.test.ts` pins the primitive's semantics. - `ws-transport-module-copies.test.ts` imports the module twice in one process and asserts a transport registered by one copy is found by the other — it fails on a plain module-scope `Map`, which is the shipped bug. - `scripts/lint/module-scope-state.mjs` fails the class: an AST rule banning mutable module-scope state in these packages, with `// per-copy-ok: <why>` as the deliberate escape. Wired into both packages' `vitest run src`, with fixture self-tests so it cannot rot into a no-op. * test(world-postgres): pin the module-scope-state rule for the postgres world It is deduped today only because `getRuntimeRequire()` loads it — a property of how it is loaded, not how it is written, and exactly what changed for world-vercel in #3493. The package is already clean; this keeps it that way. * docs(worlds): codify "a world must not hold mutable module state" A world package is loaded one of two ways, and only one of them gives it a single module instance: a runtime `require()` (deduped by Node) or the host's bundler (one copy per layer). Which one you get is a property of how the world is loaded, not of how it is written, and it changed under `world-vercel` in #3493 — so the rule has to be "never rely on module scope", not "rely on it until someone flips a config". Written down in the four places someone can meet it: - `docs/content/worlds/{v4,v5}/building-a-world.mdx` — a "Process-wide state" section for custom-world authors, with the loading modes spelled out and a nudge to prefer World-instance state over a global. - `packages/world/README.md` — the same constraint on the contract package. - `CLAUDE.md` — so the next contributor working in these packages sees it. - `packages/core/src/runtime/world.ts` — at the two static imports, which is where the difference between a bundled world and a required one originates. The rule's own error message now teaches it too, rather than naming a helper. Consolidates the guard while here: `@workflow/utils` owns the rule and its fixture self-tests, and sweeps every *published* `packages/world-*` discovered at runtime, so a world package added later is covered without anyone remembering. Each world keeps a one-assertion mirror for locality. * style: drop prose em dashes from this branch's new text #3704 landed a repo-wide writing pass hours after this branch was written and took `world-vercel/src` from 406 em dashes to 130 (`ws-transport.ts` alone went 35 to 1). This branch's docs section, README, comments and lint messages were written before that and would have put 36 of them straight back into the files that were just cleaned. Rewritten sentence by sentence rather than by substitution: an em dash becomes a colon, a comma, a full stop or a parenthetical depending on what it was doing. Also fixes a real defect the sweep surfaced: `world-postgres`'s guard test was generated through a shell heredoc and had literal backslash-backticks in its doc comment. * Update .changeset/world-module-scope-state.md Co-authored-by: Peter Wielander <mittgfu@gmail.com> Signed-off-by: Pranay Prakash <pranay.gp@gmail.com> * fix(core): build the entrypoint's queue handler from getWorld() Adopted from #3666 by @MintedKenny, which implements #3665 and could not run CI as a fork PR. One line of behavior: `workflowEntrypoint`'s lazy handler init calls `getWorld()` rather than `getWorldHandlers()`. `getWorldHandlers()` owns a second, build-time-safe cache, so calling it from the runtime route built a *second* World in the same process. That costs a stateful World duplicate resources on every instance — world-postgres eagerly constructs a `pg.Pool` (default `max: 10`) and a nested world-local World in `createWorld()`, so self-hosted users have been paying for two of each — and, for a bundled world package, the two Worlds are built by two different module copies, which is the mechanism behind the WS transport regression the rest of this branch contains. The public `getWorldHandlers()` and its separate build-time cache are unchanged; only the runtime route stops using it. Kept from the original: the regression test asserting the factory runs exactly once, and the api-reference wording (re-applied over #3704's list punctuation). Not taken: renaming the `workflow.route.get_world_handlers` span. It is a distinct span from the per-request `workflow.route.get_world` at the top of the flow route, and reusing that name would collide with it in traces and in `runtime-trace-mode.test.ts`; a comment records why the name outlived the call. Co-authored-by: Kenneth <kenneth@standardforensics.com> Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: address AI review on the module-scope work Two blocking findings, both real: - **Cross-version state sharing** (`ws-transport.ts`). A process can hold two *published versions* of `@workflow/world-vercel` (a transitive dependency pinning an older `@workflow/core`, which depends on this package by exact version). Both wrote to the same unversioned `Symbol.for` key, so one version's write path could be handed a `WsEventsTransport` built by the other's class and frame against a protocol it may not share — with no version negotiation on the socket to catch it. `shapeVersion` cannot express this: the container is stable, the hazard is its contents. The registry and the events dispatcher recycler are now keyed by package version. The plain connection pools stay unversioned; sharing those across copies is the point. - **The documented pattern failed the rule this PR adds.** The custom-world docs teach `store[StateKey] ??= …`, which the rule flagged as a field write. It now recognizes state rooted at `globalThis`, following one alias hop, which is also what `core/private.ts:23` and `next/src/index.ts:58` are already doing correctly (core drops 26 findings to 22, next 7 to 6). The docs also now say outright that `globalSingleton()` is the same thing, since AGENTS.md prescribes it and the page did not mention it. Rule precision, from the review's probes: - `.mts`/`.cts` are scanned. `@workflow/world-testing` is authored in `.mts`, so its entry in the sweep was passing vacuously — with the walk fixed it reports a real finding, now annotated (it is a standalone `serve()` entry). - Mutations in top-level statements no longer count. A table filled at module evaluation is identical in every copy; divergence needs a later write. - `static` class fields are collected, attributed to the class name. - An *exported* binding initialized to an empty collection is a finding on its own, which approximates the cross-file case the walk cannot resolve. Six fixtures pin the new behavior. The rule's header now states what it does not see, and AGENTS.md states where the sweep stops and why core is not gated yet. Also tags `resetGlobalSingletonForTest` `@internal`. * fix(lint): attribute a static-field write to the field, not the class The static-field support added in the previous commit keyed `declared` on the class name, so a class carrying more than one mutable static reported one finding instead of one per field, and labelled the survivor with whichever mutation was seen first. On a two-static fixture it reported `static Registry.latch (`.set()`)`: the name of one field, the reason belonging to the other, pointing the reader at the wrong line. Key static fields `Class.field` and resolve a write to the same shape, via a new `memberPath()` that takes the first two segments of a member chain and tries that key before the bare root identifier. Two follow-ons fall out of having the path: - `this.field` inside a `static` member resolves to the class, which is the ordinary way to write the mutation. `staticClassOf()` returns nothing for an instance member, where `this` is an instance and the state is per-instance rather than per-copy, and nothing inside a nested `function`, which rebinds `this`. - `state.count++` is now a finding, like the `state.count += 1` that `assignment()` already reported. Fixtures pin all four, including the instance-field case that must stay clean. The four world packages still report zero, and the extracted `recordMutation()` keeps the file at its previous two Biome complexity warnings. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: make module duplication inert across every bundled package `@workflow/core` is bundled into the host server build the same way the worlds are, and always has been — the original repro measured three live copies in every arm, including the pre-#3493 external one. One instance is not reachable: layers cannot share a module, and core cannot be external because it *is* workflow code (`runtime/start.ts:253` and nine methods in `runtime/run.ts` are `'use step'`), so it must go through the SWC loader. The Next integration already encodes that rule by removing workflow-bearing packages from `serverExternalPackages`. So the duplication stays and the hazard is removed instead, everywhere the duplication can happen. `@workflow/core` (22 findings to 0): warn-once latches in `constants.ts`, `start.ts` and `telemetry.ts`; the source-map tracer cache; the VM script cache; the QuickJS compiled-assets and baseline caches; the dev-server port cache (its own comment already said "per process"); the text codecs; the zstd browser decoder; and the `useStep` closure brand, where a function marked by one copy was invisible to another. The one with teeth was `step-single-flight.ts`: a per-copy map is not single-flight. Two invocations reaching it through different layers would each believe they were alone in the process and both run the step body, silently degrading in-process dedup to the cross-process residual its own doc scopes out to the ownership lease. Also `@workflow/world` (a warn-once set, hand-rolled onto `globalThis` to keep that package dependency-free), `@workflow/ai` (the lazy OTel API), and `@workflow/nest` (bootstrap config in a module-level `let` and two static class fields — configure one copy, read another, and the controller is unconfigured for the life of the process). Five sites are deliberately per-copy and now say why: state keyed on objects that never cross copies (the barrier safety-net `WeakSet`, the QuickJS pending byte `WeakMap`), the synchronously-scoped guest-code sink, and the OTel diagnostic that reports what *this* copy sees. The sweep now covers all of it. Packages with a single module graph stay out (build-time code, the CLI, the o11y UI, the test runner) and AGENTS.md records which and why. Found while doing this: two static fields on one class collapsed into a single entry in the rule, so `WorkflowModule.options` was invisible behind `WorkflowModule.outDir`. Statics are now keyed `Class.field`. * fix(world): suppress noAssignInExpressions on the globalThis idiom The hand-rolled form trips Biome, as it does in `packages/core/src/private.ts`, which carries the same suppression. Restructuring it into a helper function instead would hide the state behind a call the module-scope rule cannot follow, so the binding would stop being recognized as off-module and the package would report a finding for correct code. * fix: sweep every bundled package, and mark utils side-effect free @shalabhc asked on review whether `@workflow/utils` needs this too. It does, and so do three others: `utils`, `errors`, `serde` and `workflow` all end up in the host application's server build and none were in the sweep. All four report zero today, which is exactly the state `world-testing` appeared to be in before the `.mts` walk was fixed and it turned out to have a real finding. Being clean and being *checked* are different properties, and only the second one survives the next contributor. `sideEffects: false` on `@workflow/utils`: verified that every module in the package only declares (no import-time work), so a bundler can now drop the unused parts of the barrel instead of keeping all ~64 KB of it because three packages import one 476-byte function. --------- Signed-off-by: Pranay Prakash <pranay.gp@gmail.com> Co-authored-by: Peter Wielander <mittgfu@gmail.com> Co-authored-by: Kenneth <kenneth@standardforensics.com> Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> Co-authored-by: Peter Wielander <peter.wielander@vercel.com> |
||
|
Peter Wielander
|
b2cac623d3 | [world] Make the sealed log opt-in instead of default-on (#3735) | ||
|
Nathan Rajlich
|
e1e64e3de3 |
docs: apply Vercel technical writing standards (#3704)
* docs: apply Vercel technical writing standards Audit the complete documentation corpus, package READMEs, skills, and source TSDoc/comments against the vercel-technical-writing skill and style-rules.md. Normalize sentence-case headings without changing published anchors, remove prose em dashes and filler wording, improve active voice and self-contained phrasing, standardize product/brand capitalization, American English, list punctuation, units, and code fence languages, and preserve exact runtime strings/table placeholders. All executable code is unchanged. Modified skills have their metadata versions bumped. * docs: extend writing audit to repository Markdown Apply the same technical-writing rules to design documents, compiler specifications, workbench guides, package changelogs, and the remaining tracked Markdown outside the deployed docs corpus. Preserve historical meaning, commands, output literals, table placeholders, and heading anchors. * docs: exclude generated package changelogs from audit |
||
|
Pranay Prakash
|
7b79ba37cc |
Add support for 'noop' event type - spec version 7 (#3634)
Co-authored-by: Peter Wielander <peter.wielander@vercel.com> |
||
|
Peter Wielander
|
b3dbc6d264 | [docs] v5 changes docs: what's new, world upgrade guide, migration skills (#3100) | ||
|
Rich Haines
|
d5d19fc8d5 | docs: update Geistdocs to 1.20.4 (#3654) | ||
|
Peter Wielander
|
9b1b8c7111 | [core] Pin correlation-id draw order to event-log order (#3700) | ||
|
Pranay Prakash
|
9454d51db0 |
feat(core): resolve run.returnValue via a World long poll instead of a 1s poll (#3570)
Co-authored-by: Peter Wielander <mittgfu@gmail.com> Co-authored-by: Peter Wielander <peter.wielander@vercel.com> |
||
|
Nathan Rajlich
|
5b5a926f88 |
fix(core): make step-argument serialization failures catchable in workflow code (#3675)
* fix(core): make step-argument serialization failures catchable in workflow code
A step whose arguments fail to serialize is now finalized by the
suspension handler as step_created + step_failed (mirroring a step-body
failure) instead of rejecting the whole suspension. The next replay —
forced in-process, since no step message is dispatched for the failed
step — rejects the step's promise with the SerializationError, so a
try/catch around the step call observes it. Uncaught, the error
propagates out of the workflow body and fails the run as a fatal
USER_ERROR immediately, instead of redelivering the orchestrator
message until max deliveries (49/48) as reported in production on v4.
* Serialize the step_failed error with the VM global; one-sentence changeset
Addresses review feedback: dehydrateStepError in
finalizeUnserializableStep now receives suspension.globalThis like every
other dehydration in this file. Error detection is realm-independent, so
the host-created SerializationError serializes identically, but VM-realm
values guest code threw into the cause chain are now detected by the
realm-sensitive reducers.
* Address review: QuickJS engine support, deferred-batch join, drain gate, placeholder marker, telemetry, docs
- QuickJS: dumpPendingOps now catches a step input's serialization
failure per-op, reframes it as a SerializationError with the same
framed message as dehydrateStepArguments, and surfaces it on the
pending op instead of failing the whole collection. The entrypoint's
dispatchPendingOps finalizes such steps as step_created (placeholder
input) + step_failed, excludes them from inline claims and queue
publishes, marks them handled, and raises the requeue signal so the
failure is observed even when the feed lags — mirroring the node:vm
engine, so both engines agree: catchable in workflow code, USER_ERROR
with the framed message when uncaught. Both step-argument e2e tests
now pass on WORKFLOW_VM=quickjs.
- runtime.ts: the failed-step replay path now joins
suspensionResult.deferredBatchWork before continuing, so a trailing
chunk commit or step-message publish rejection propagates instead of
being swallowed after ack; committed inline claims are documented as
deliberately handed to owned recovery.
- Terminal drain: finalization is gated on a stepDispatch target. The
drain caller has no replay to observe a finalization, so a completed
run no longer gains failed-step rows for an unawaited unserializable
step — the rethrown error is swallowed by the drain's catch,
preserving its pre-existing behavior.
- The placeholder input now carries a marker string ('[input
unavailable: step argument serialization failed]', shared via
runtime/unserializable-step.ts) so inspect/o11y don't render the
failed step as a genuine zero-argument call.
- New workflow.steps.failed_serialization span attribute on the
suspension span, so occurrence is measurable without log search.
- Docs: v5 serialization-failed error page documents where each
boundary's failure surfaces (catchable step failure vs run failure)
and the no-retry USER_ERROR semantics; foundations/errors-and-retries
gains a Serialization Failures section with the try/catch shape.
* Guard the finalization crash window; self-contained docs samples
- A crash or transient failure between finalization's two durable
writes leaves a lone placeholder step_created, and redelivery then
dispatches the step through normal crash recovery — previously
running user code with the placeholder arguments. The placeholder
now carries a structural flag on the input triple's top level (which
user code never controls, so no false positives), and the step
executor checks it after hydration: instead of running the body, it
throws the intended fatal SerializationError, completing the
interrupted finalization as step_failed. Applies to both engines
(they share the placeholder and the executor).
- Regression tests: executor fails a placeholder-input step without
running the body (and doesn't trip on a genuine argument equal to
the display marker); handleSuspension rejects for redelivery when
step_failed can't be written after step_created landed, leaving the
recoverable placeholder behind; mixed bad-step + large fan-out
returns the failure set alongside still-pending deferredBatchWork
whose rejection surfaces — the contract the runtime's failed-step
join (added previously) relies on.
- Docs: the two new code samples are now self-contained so the docs
code-sample typecheck passes.
|
||
|
Peter Wielander
|
0b2797bbac | [next] Bundle the Vercel world into the Next.js server output (#3493) | ||
|
Pranay Prakash
|
37e1d9e5a9 |
Batch: pre-claim inline steps in the same batch (#3568)
* Pre-claim inline steps inside the suspension batch (born-running pairs) Restacked onto main after #3025's squash-merge; folds in the review-round changes to the flush loop (per-write requestId attribution on createBatch, and the seeded/advancing slot-bump expectation, now shared with the pre-claim ceiling). Fold each lazy-inline step's deferred writes into the batched fan-out as an adjacent [step_created, step_started] pair: the created row carries the input, the started row is a bare ownership-stamped claim the server folds into one born-running create. The whole scheduling turn commits as ONE durable write, inline bodies start straight off that commit (in parallel with the VQS publishes for backgrounded steps), and executeStep gains a pre-claimed mode that runs or skips the body off the batch's per-event verdict - a pair 409 is the same skipped outcome as losing the lazy claim. The lone-inline case keeps the optimistic lazy path (a pair-only batch buys nothing over the single claim). Also threads per-event computeInstanceId through the World batch request, and folds the batch's committed slot ceiling into the inline slot snapshot so terminal writes stop being answered with reports echoing the batch's own events. * Parallel chunk commits, per-chunk continuation, batch span attributes Production trace of a 67-event fan-out showed the three batch chunks POSTing back-to-back (~230ms each) with no bodies or queue messages until all three settled (~670ms). Three changes: - Chunks now POST concurrently. Slot assignment is the server's, so parallel chunks race for slot ranges exactly like the pre-fold path's parallel single writes did; entity conditions, not commit order, carry correctness. The foreign-interleaving diagnostic is computed once over the whole fold (committed span vs seed) instead of per chunk. - Per-chunk continuation: each chunk's step-execution queue messages publish the moment ITS creates are durable (in-flush, via stepDispatch, same message shape and idempotency key as the caller's dispatch pass - the affected steps are pre-reported in queuedStepCorrelationIds so the caller skips them). Only the chunk carrying the inline pairs gates handleSuspension's return (opt-in via allowDeferredBatchWork); trailing chunk commits + all publishes ride result.deferredBatchWork, which the runtime joins next to the dispatch join before it can ack - the every-create-durable-before-ack contract is unchanged, the bodies just start off the pair chunk instead of the slowest chunk. - OTel: batch identity attributes (workflow.batch.size, per-type workflow.batch.shape) now live on the world.events.createBatch span (instrumentObject) instead of the http POST span, which keeps only wire-level facts (transport, bytes) and no longer sets workflow.event.type - that attribute names a single event write and tagging a batch with its first event's type misclassifies traffic. * Address review: settle deferred fold on failure, drop pair-batch retry Three fixes from review of the deferred/parallel-chunk fold. 1. A pair-chunk rejection escaped `handleSuspension` while the trailing chunks' commits and publishes were still in flight. `deferredBatchWork` never reaches the caller once the handler throws, so nothing joined that work — exactly the state `settlePhase` exists to prevent: a sibling create landing after the rejection commits an event from the abandoned replay's seeded sequence and races the caller's restart reload. The failure path now settles `trailing` before rethrowing. 2. Every pair-carrying chunk gates the return, not just the first. Pairs sort to the front and two rows per inline step fit inside one chunk, so this is one commit today, but `findIndex` silently degraded if either cap moved: a pair in an unawaited chunk yields no `inlineClaims` entry, the caller falls back to a lazy `step_started`, and that races this same fold's in-flight pair for the same step. constants.test.ts now pins the cap relationship. 3. A batch carrying a `step_started` is no longer retried in-process. The born-running pair does converge to a 409, but the pre-claim caller reads a pair 409 as "a concurrent writer owns this step" and skips the body — and on a retry that is indistinguishable from "my own first attempt committed the pair". Skipping there stranded a running step stamped with this invocation's own message id until the ownership lease expired (860s), where the single-POST path deliberately fails the delivery and recovers through owned-recovery in seconds. Same reasoning `EVENT_RETRY_ELIGIBILITY` already applies to `step_started`. Also asserts `lazyStepInput` / `preclaimedStart` mutual exclusivity in executeStep instead of only documenting it, and adds the changeset. Tests: +1 suspension-handler (pair-chunk failure settles the trailing chunk before escaping — fails without fix 1), +1 constants (cap relationship), +1 world-vercel (a born-running pair batch is single-attempt), and the existing batch-retry test retargeted at an entity-conditioned batch. Full @workflow/core unit suite 2178 green, @workflow/world-vercel 514 green, typecheck green across core / world / world-vercel. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * Guard inline bodies against unhandledRejection; review follow-ups The dispatch/deferred-batch joins now sit between the step promises' creation and the `Promise.all` that reads them, so a body rejecting in that window had no handler attached at the microtask checkpoint — an unhandledRejection, fatal under Node's default --unhandled-rejections=throw. A 412 fenced claim races exactly that window, and `deferredBatchWork` widens it by a trailing-chunk round trip. Attach a no-op catch at creation, the same way `dispatchesSettled` already does two lines up; the awaits below still decide the outcome. Review follow-ups: - `workflow.batch.shape` is sorted by event type. Map iteration is first-seen order, so a pre-claimed fold and a pure eager fold rendered the same composition as different strings, which is not groupable as a dimension. - A lost pre-claim reports StepSkipReason `running`, not `completed`. The pair's 409 says the step already exists and its claim winner is executing; the other skip site is a genuine terminal-state conflict, and tagging both `completed` left the attribute unable to separate the two. - `batchCommittedSlotCeiling`'s docstring now says the echo is only fully suppressed for a single-chunk fold: on a multi-chunk fan-out an inline terminal write issued before the trailing chunks land still names a position below them and still draws a report. - The defensive throw on a missing dehydrated input records where it lands — the pair is already durable, so it fails with the step claimed and its body unrun, recovered on redelivery via owned-recovery rather than failing cleanly. No regression test for the unhandledRejection: the existing inlineClaimRejectionScenario runs both steps inline, so `dispatches` is empty and the join resolves in a microtask — the window never opens and a test there passes with or without the fix. Reproducing it needs a scenario with a backgrounded step and a slow queue publish alongside the fenced claim. Full @workflow/core unit suite 2178 green, @workflow/world-vercel 514 green, typecheck and biome clean. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * Pin per-event computeInstanceId on the batch wire Batch encoding is a separate path from the single-event POST, so the frame meta had no coverage: the only assertion was at the World-call boundary. Adds a wire-level test that a pre-claimed pair's step_started half carries computeInstanceId in its frame meta and the step_created half does not. Verified it fails when the threading in createWorkflowRunEventBatch is removed. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * Guard the pre-claim path as inert on Worlds without createBatch world-local and world-postgres do not implement createBatch, so the fold never engages there — but the runtime passes ownerMessageId and allowDeferredBatchWork unconditionally. The existing "keeps the single path when the World lacks createBatch" test passed neither, so it never covered the pre-claim path at all. Assert the inertness with the params the runtime actually sends: no claims, no deferred work, no slot ceiling, the lazy-inline step still carrying its input, and no step_started reaching the world. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Peter Wielander <peter.wielander@vercel.com> Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Peter Wielander
|
04e060a0ec | [world] Add WORKFLOW_NODE_HTTP to run the HTTP Worlds on node:http (#3461) | ||
|
Alex Langenfeld
|
df0103bc3c |
Document stream reader request cancellation (#3581)
## Summary & Motivation The earlier timeout-docs attempt in #534 was closed without merging. Without `supportsCancellation`, a browser disconnect leaves the stream reader reconnecting until the function hits `FUNCTION_INVOCATION_TIMEOUT`, so the streaming guide now documents the `vercel.json` opt-in, with a warning that it terminates everything matching the configured path, and the resumable-streams guide points at it. ## Test Plan Docs only; no tests. ## Docs Preview | Page | v4 | v5 | | --- | --- | --- | | Streaming | [Preview](https://workflow-docs-git-alangenfeld-timeout-docs.vercel.sh/docs/foundations/streaming#avoiding-function-timeouts-after-client-disconnects) | [Preview](https://workflow-docs-git-alangenfeld-timeout-docs.vercel.sh/v5/docs/foundations/streaming#avoiding-function-timeouts-after-client-disconnects) | | Resumable Streams | [Preview](https://workflow-docs-git-alangenfeld-timeout-docs.vercel.sh/docs/ai/resumable-streams) | [Preview](https://workflow-docs-git-alangenfeld-timeout-docs.vercel.sh/v5/docs/ai/resumable-streams) | Signed-off-by: Alex Langenfeld <alex.langenfeld@vercel.com> |
||
|
Pranay Prakash
|
b0adb50bce |
feat(world,world-vercel): createBatch — ordered batch event write with per-event results (#3025)
## createBatch: the client half of the v4 batch event write (per-event
results, no fence)
> **Note:** this PR was rebuilt from scratch. The previous revision
implemented the retired "v2 suspension fence" design
(`expectedRunVersion` / `batchId` / `logicalCreatedAt`, a
grammar-validated collect mode, world-postgres migration 0016,
world-local claim machinery). The server redesigned its endpoint in
place (vercel/workflow-server#646, merged and deployed) and this branch
now targets that contract on top of current `main` (specVersion 6 slot
identity). The old head is tagged `batch-client-v2-fence-design`; prior
review threads reference deleted code.
### The server contract this targets
`POST /api/v4/runs/:runId/events/batch` (workflow-server#646): an
**ordered** list of v4 frames — byte-identical to single-event POST
frames, **no batch-level meta** — committed in one DynamoDB transaction
per attempt, answered with HTTP 200 + `{ results }`: one entry per
frame, in request order. Each event reports what its own single POST
would have returned: `200` + the materialized entity, or the single-path
status/code (e.g. `409`/`conflict` for an event an earlier delivery
already applied). A transport retry of a committed batch converges to
all-409s with nothing written twice — idempotency comes from per-entity
conditions, not batch bookkeeping. Slot-identity runs only (specVersion
≥ 6 — what `world-vercel` stamps on every new run since #3389).
### What this revision ships
1. **`@workflow/world` — the spec addition.** `Storage['events']` gains
one optional method; **method presence is the capability declaration**
(no capability flag, no stub required):
```ts
createBatch?(
runId: string,
events: BatchEventRequest[],
params?: CreateEventBatchParams
): Promise<EventBatchResult>;
interface BatchEventRequest {
event: CreateEventRequest; // same discriminated union as the single create
occurredAt?: Date; // under slot identity: the source of the durable createdAt
}
type BatchEventItemResult = // one per submitted event, in request order
| { status: 200; event: Event; run?: WorkflowRun; step?: Step; wait?: Wait }
| { status: number; error: string; message: string };
interface EventBatchResult { results: BatchEventItemResult[] }
```
Contract: **ordered** (events land in the log in request order),
**per-event outcomes** (each event reports what its own single `create`
would have returned — success discriminated by `error === undefined`),
**idempotent on retry** (per-entity conditions make a retried committed
batch converge to per-event 409s). Worlds that don't implement it keep
the single-event path. `world-local` and `world-postgres` deliberately
do NOT implement it — batching a local/in-process write buys nothing
(this deletes the old revision's riskiest surface: the hand-written
postgres migration and the world-local claim machinery).
2. **`@workflow/world-vercel`** — the wire adapter: per-event frames
concatenated in order (reusing the single-frame encoder; each frame
carries its own `occurredAt`, which under slot identity is the source of
the durable `createdAt` — this natively closes the replay-clock question
the old `logicalCreatedAt` field existed for), CBOR `{ results }`
decoded against the **same per-type zod schemas as the single POST**,
loud `SCHEMA_VALIDATION` on any malformed response (wrong length,
invalid item), and the standard typed error mapping for request-level
failures.
3. **Retry policy** — a `batchIdempotent` override in the event-retry
eligibility machinery: the whole batch POST retries transient transport
failures/5xx (and waits out 429 `Retry-After` per #3504) regardless of
the contained event types, because per-event entity conditions make the
retry converge; the per-type non-retryability matrix guards *single*
posts (where e.g. a retried bare `step_started` would increment
`attempt`) and doesn't apply inside a batch.
Tests: 7 wire tests — frame encoding/ordering + **no fence fields on the
wire**, per-event result mapping (successes typed, failures passed
through), malformed-response failures (length mismatch, invalid item
body with index), typed request-level 400s, in-process 5xx retry,
empty-batch guard — plus 9 suspension-handler tests for the runtime
fold: ordering (steps then waits), per-event 409 tolerance, non-409
failure propagation, every gate exclusion (flag off / no `createBatch` /
pre-slot run / hook writes), 32-cap chunking, and lazy-inline exclusion.
Full `world-vercel` suite: 508 passed; full `@workflow/core` suite: 2126
passed.
### The runtime integration: batched suspension fan-out (ON by default)
The suspension handler folds a **clean fan-out** — the suspension's
eager `step_created` + `wait_created` writes — into `createBatch` calls
of at most **32 events**, and uses the batch endpoint **exactly when two
or more batchable eager events exist**: a lone eager event takes the
ordinary single write (same round trip, and it keeps the slot-snapshot +
bump-and-report the single path provides) (mirroring the server's
transaction budgets: 2 items/event against the 100-item cap, 768 KB
inline-byte budget; larger fan-outs commit in successive batches). The
gate requires: World implements `createBatch` ∧ run on slot identity
(specVersion ≥ 6) ∧ no attribute writes ∧ no hook writes ∧ no resilient
step dispatch. **Everything outside the gate keeps the single-event path
byte-for-byte**, and lazy-inline steps keep deferring their
`step_created` to the lazy start exactly as before.
Per-event semantics mirror the single path: a `409` is the same
already-exists tolerance as `EntityConflictError` (the conflicted step
is not marked owned); any other per-event failure fails the suspension
write the way a single-path rejection would. Slot bumps (the batch
endpoint has no bump-and-report) are tolerated and logged — the same
accepted exposure as a dropped truncated skipped-slot report on the
single path.
**On by default**, with the `WORKFLOW_TURBO`-shaped kill switch as the
operator escape hatch: **`WORKFLOW_BATCH_TRANSITIONS=0`** (or `false`)
disables batching and restores the exact prior one-write-per-event path.
Documented in the worlds configuration reference and the changelog
entry. Burn-in watch: the `event_batch`-tagged slot-conflict metrics and
DynamoDB throttle monitors on the server side.
### Docs
- New v5 changelog entry **`changelog/batched-event-writes`**
documenting the World spec addition (full `createBatch` signature +
contract — the signature block is compile-checked against
`@workflow/world` by the docs code-sample checker), the runtime fold,
and the follow-up.
- `configuration/worlds` gains the **`WORKFLOW_BATCH_TRANSITIONS`**
reference entry: default on, `=0`/`false` as the documented escape
hatch.
### Staged follow-up: the deferred sequential transition (the STSO win)
Hold `step_completed(N)` across the replay turn and commit
`[step_completed(N), step_created(N+1), step_started(N+1)]` as one batch
at the next lazy start (the server folds the pair born-running). This
needs the synthetic-completion replay machinery rebuilt against today's
runtime (parallel inline batches, turbo's run-ready barrier, optimistic
starts, slot bookkeeping) — it stays a separate PR so the SDK's most
sensitive replay path gets its own focused review. Its acceptance
criteria are already agreed: the runtime eligibility matrix as unit
tests, and an e2e that asserts ≥1 POST to `/events/batch` and **zero**
single-event POSTs for the batched transitions.
### Compatibility
- Old servers: no `/batch` route → 404/405 → callers fall back to
single-event posts (the runtime PRs will latch this per run).
- Pre-slot runs: request-level 400 (`batch-requires-slot-identity`) →
same fallback.
- No `WORKFLOW_SERVER_URL_OVERRIDE` pin this time — the server endpoint
is merged and deployed to production.
Refs: vercel/workflow-server#646 (endpoint), vercel/workflow-server#780
(unbatchable-types design space), #3389 (slot identity), #3504 (429
retry).
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
|
||
|
Peter Wielander
|
1321570464 | [docs] Document duplicate-event handling, and describe webhook token generation accurately (#3497) | ||
|
Peter Wielander
|
de2a86c61c | [world] Make spec version 6 the current version (#3542) | ||
|
Peter Wielander
|
dc85865718 | [core] Drop pre-slot event ID support and preconditionGuard capability (#3519) | ||
|
Shalabh Chaturvedi
|
01991edeeb |
feat(world-vercel): synthesize per-event client spans on the WS transport (#3452)
* feat(world-vercel): synthesize per-event client spans on the WS transport PR #3084 added the opt-in `WORKFLOW_EVENTS_TRANSPORT=ws` path and listed "no client-side span on the WS path" as a known limitation. Because event writes become multiplexed frames on one long-lived socket rather than individual `fetch` calls, the per-event `http POST` CLIENT span that the HTTP transport produced simply disappeared — traces went from one span per event to nothing between the invocation and the server. Restore it by synthesizing a request-shaped span around each frame, and give the upgrade its own span: - Extract `withHttpClientSpan` / `recordClientSpanStatus` from `instrumentedFetch` in `http-core.ts` so the synthetic span is emitted by the same envelope as the real one and cannot drift from it. `InstrumentedFetchOptions` now extends `HttpClientSpanOptions`. - `postEventFrameOverWs` opens `http POST` with `url.full` pointing at the v4 REST endpoint the frame is forwarded into, so per-event traces and latency dashboards keep working across the flag. Extract `eventsV4Url` so that URL cannot drift from the one the HTTP path actually requests. - Tag both transports with `workflow.events.transport` (`http` | `ws`) and `workflow.event.type`; the WS path additionally sets `network.protocol.name=websocket`, `workflow.events.ws.url` (the real wire destination) and `workflow.events.ws.req_id` (join key to the server's log line for the frame), so the span is never mistaken for a real HTTP request. - Add a `workflow.events.ws.connect` span around the upgrade — the one genuinely-HTTP request here, previously the invisible half of every WS write's latency — carrying `workflow.events.ws.reconnect_attempt`. This also puts `resolveUpgradeHeaders`' trace-context injection inside a client span, as AGENTS.md requires. - Fix `parseServer` to treat `wss:` as TLS (port 443, not 80). Out of scope, deliberately: per-frame `traceparent` (needs a frame-meta field plus a server change) and Vercel's outgoing-requests view (that instruments global `fetch`, so a frame structurally cannot appear there). Covered by `ws-transport-spans.test.ts`, which drives the real selection + transport + adapter stack over a fake socket and asserts span shape, failure reporting, retry behaviour and HTTP/WS parity. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Co-Authored-By: shalabhchaturvedi-7802 <shalabh.chaturvedi@vercel.com> Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * chore: trim WS spans changeset to the user-facing summary Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Co-Authored-By: shalabhchaturvedi-7802 <shalabh.chaturvedi@vercel.com> Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * fix(world-vercel): only tag event-write spans with transport Signed-off-by: Shalabh Chaturvedi <shalabh.chaturvedi@vercel.com> Co-Authored-By: shalabhchaturvedi-7802 <shalabh.chaturvedi@vercel.com> * fix(world-vercel): format WS transport span regression test Signed-off-by: Shalabh Chaturvedi <shalabh.chaturvedi@vercel.com> Co-Authored-By: Shalabh Chaturvedi <shalabh.chaturvedi@vercel.com> --------- Co-authored-by: vercel[bot] <35613825+vercel[bot]@users.noreply.github.com> |
||
|
Karthik Kalyan
|
f1ef0cbf03 |
Deprecate world.runs.list for observability (#3404)
* Deprecate storage run listing for observability * Document analytics run listing guidance |
||
|
Peter Wielander
|
b589460ce8 | [core] Report the replay position on every event write (#3479) | ||
|
Nathan Rajlich
|
7683130461 |
Resilient step dispatch: parallelize step_created writes with queue publishes (#3365)
* feat(world,world-vercel,core): resilient step dispatch (parallel step_created + queue publish) Newly created steps are handed to the queue in parallel with their step_created event write, with the serialized input carried on the message (stepInput) so the queue consumer can idempotently re-ensure the event when the direct write failed transiently — mirroring resilient start (runInput) and resilient hook resume (hookInput). - @workflow/world: stepInput on WorkflowInvokePayload, CreateEventParams.viaStepDispatch, WorldCapabilities.resilientStepDispatch - core (node:vm): suspension handler publishes eligible steps alongside their create; the dispatch pass skips them (queuedStepCorrelationIds) - core (quickjs): dispatchPendingOps does the same for overflow steps; the ineligible fallback is now published in parallel too (removes the serial per-step enqueue loop) - consumer: on a redelivery, a stepInput-carrying message re-ensures step_created (marked viaStepDispatch) before executing - under an enforced precondition guard the parallel path requires backend cooperation (capabilities.resilientStepDispatch, declared by world-vercel): a 412-rejected step's in-flight dispatch is revoked server-side and its re-ensure refused - step dispatch/retry idempotency keys are step-identity-scoped (cid + hashed step name) so a revoked message for a reassigned correlation id cannot absorb the corrected schedule's dispatch - kill switch: WORKFLOW_RESILIENT_STEP_DISPATCH=0 * Validate stepInput.input as Uint8Array at the schema boundary Review feedback: producers only attach stepInput when the dehydrated input is binary and the queue transport preserves bytes (CBOR), so a non-binary value means the payload was mangled in transit. Enforcing Uint8Array in StepDispatchInputSchema fails the message parse instead of silently writing non-binary data into a step_created, and types the consumer's re-ensure so the unchecked 'as SerializedData' cast goes away. * Keep sequential dispatch under an enforced precondition guard (drop the resilientStepDispatch capability lift) Review feedback (two P1s): backend-side revocation bookkeeping cannot carry the guard's correctness property across the queue side-channel — - nothing orders a slow guarded create's eventual 412 (the moment the backend learns the dispatch is poisoned and records the revocation marker) before the consumer's redelivery re-ensure, so attempt > 1 is a probabilistic mitigation, not a happens-before; and - a best-effort marker that fails open (Redis loss) cannot back a capability the SDK treats as a correctness attestation. Only sequencing the publish after the create gives the message a happens-after edge over the create's guard verdict, so the guard gate is now unconditional: worlds that enforce the precondition guard keep the sequential create-then-publish dispatch. The parallel resilient path remains for unguarded writes (the quickjs engine everywhere, and worlds without the guard). Removes WorldCapabilities.resilientStepDispatch and world-vercel's declaration; the viaStepDispatch flag is kept and re-documented as advisory (server-side defense-in-depth only). This also dissolves the reviewed dedupe hazard on the step-identity- scoped dispatch keys: with no 410-ack path in any real SDK flow, a message for a never-created step keeps redelivering until an entity exists, execution always hydrates input from the committed entity (never the message), and a name-mismatched stale start is skipped by the server's stepName fence. * Correct the MAX_RESILIENT_STEP_INPUT_BYTES rationale: VQS has no hard message-size cap 256 KB is the queue's inline-vs-S3 threshold, not a rejection limit (payloads above it spill to S3-backed storage transparently). The 128 KiB bound is a cost/latency choice — keep step messages on the inline path rather than paying an S3 double-hop for bytes that already live in the event log. * Recover a missing step in-band when a stepInput-carrying delivery beats its create Durabench parallel sweeps (guard-off, node engine) caught ~4-8% of fan-out runs stalling one branch for ~306s on the resilient dispatch path. Root cause: the consumer's step_created re-ensure was gated on metadata.attempt > 1, but world-vercel's failure-retry path re-enqueues a FRESH message whose attempt resets to 1 — so when a delivery beat the producer's parallel step_created write, every fast retry hit the same 'step not found' rejection with attempt 1, and the step only recovered when the ORIGINAL message's ~300s visibility-timeout redelivery finally arrived with attempt 2. The recovery is now in-band and attempt-independent: when a stepInput-carrying execution rejects with the step-missing signature (WorkflowWorldError, 404 or the local worlds' message shape), the consumer materializes the step_created from the message payload and retries the execution once within the same delivery. The eager attempt>1 ensure is kept as a round-trip saver on genuine redeliveries. Sweep effect expected: the 305-306s TTLS outliers disappear while the resilient path keeps its p50 win (1054ms vs 1425ms at 64 branches). |
||
|
Peter Wielander
|
6786db9953 | World-side incrementing event ID (specVersion 6) (#3389) | ||
|
Rich Haines
|
1aed119e84 | [docs] upgrade geistdocs to 1.19.6 (#3407) | ||
|
Shalabh Chaturvedi
|
264ddff67b |
Add WebSocket transport for step-execution event writes (opt-in) (#3084)
* sdk side for workflow server websockets Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * hardcoded workflow server Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * debug info Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * more debug Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * remove unnecessary debug Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * default on websockets, and override url Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * fix for missing funcs Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * make websockets opt outo Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * enable ws again Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * [revert later] reduce test to single test, test both http and ws at the same time Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * empty Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * Run full suite with and without ws Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * empty Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * improve e2e test Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * minimize tests Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * fallback to http when proxy present Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * default to websockets, remove matrix Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * remove smoke test Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * update to new protocol Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * adjust for new protocol (runid in path) Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * fix ws transport error Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * blank Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * fix ws dep Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * fix ws external: only accelerators, not ws itself Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * add dedicated WS-transport e2e job; flip WS default back to opt-in Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * build all packages before local vercel build (needs workflow/nitro on disk) Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * force NITRO_PRESET=vercel for the local vercel build step Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * install vercel CLI once instead of npx-ing it per command Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * add changeset for WS events transport Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * Harden the WS events transport and gate its e2e jobs Follow-ups from review of the WS transport. CI: `e2e-vercel-ws-transport` was wired into the `summary` job but not into `e2e-required-check`, so all three WS jobs could fail while the required check stayed green. Added to both branches of the status validation — including the `workflow-server-test` label branch, where the job runs under the same gating as `e2e-vercel-prod`. Transport: - `reqId` and the pending-reply map are now per connection rather than per transport. The protocol defines `reqId` as a per-connection counter, so a reconnected socket restarts at 1; with one shared map that collided with the previous socket's still-registered waiters. It also makes the superseded-socket guard structural instead of something the close path has to remember. - Post-open socket errors are no longer silent. The only `'error'` listener closed over the connect promise's `reject`, already settled once `'open'` fired, so every broken pipe / 1009 / protocol fault was swallowed and its requests hung with no per-request timeout to save them. Now logged, and the connection is torn down. - An unexpected close reconnects eagerly instead of waiting for the next write, since a socket breaking mid-run means more writes are coming. Bounded by exponential backoff, an attempt cap that falls back to lazy reconnect, a bail-out when a newer socket is already live, and an `unref()`ed timer so a backoff window can't delay handler exit. - `ws.send()` failures reject their request. `send()` doesn't throw on a non-OPEN socket — it reports through a callback we weren't passing — so the request just sat in `pending` forever. - The reserved `reqId: -1` malformed-frame reply and undecodable frames are logged loudly instead of dropped. - Auth headers resolve once per socket via a thunk, not once per event. The bearer only rides the upgrade, so the old code awaited `getVercelOidcToken()` on every write and discarded all but the first. Re-resolving on reconnect also means a new socket gets a fresh token. Adapter: a reply with no numeric status now fails closed. Defaulting to 200 reported a write as applied whenever the client met a frame it didn't understand — and the protocol is explicitly designed to grow new response variants. Tests: 24 new unit tests over the paths the e2e suite can't reach on demand (send failure mid-flight, error after open, late close from a superseded socket, reconnect backoff and give-up, sentinel/undecodable frame logging, one-token-per-socket) plus the adapter's fail-closed and typed-error mapping. Co-Authored-By: Shalabh Chaturvedi <shalabhc@users.noreply.github.com> Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * ci: re-trigger to confirm the prior e2e failures were flake No code change. The 5 failures on 0bb21e7 clustered in a ~20s window across HTTP-path jobs (example/nuxt on the same test, sveltekit on a timeout) and one WS job (sleepingWorkflow's clock-skew assertion), which points at the environment rather than the transport changes. Re-running to confirm. Co-Authored-By: Shalabh Chaturvedi <shalabhc@users.noreply.github.com> Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * Add WS wire-contract conformance tests and pin the transport gate Three gaps in the existing coverage. **The HTTP path was already covered** — `events-v4.test.ts` has 22 tests, including five directly on `createWorkflowRunEventV4` over HTTP (alias URL, frame meta contents, response decoding, skipPreload/stateUpdatedAt forwarding). Those run with the gate unset, so they do confirm the two-branch refactor didn't disturb HTTP. No new tests needed there. **But nothing pinned the gate itself.** Every HTTP assertion stays green if the default flips to WS, because the transports are built to be indistinguishable at the result layer — and an earlier revision of this branch did flip the default deliberately, for benchmarking. Added tests for `isWsEventsTransportEnabled()` across values, and one that drives a real HTTP request through a MockAgent while asserting the WS transport is never constructed. **Nothing verified the bytes.** `ws-transport.test.ts` replies with whatever the test hands it, which proves the client's lifecycle but not that its frames are what workflow-server accepts. That's the drift the spec doc exists to prevent, and it already happened once: event meta flat on the frame where the server wanted it nested under `event`, with both sides' tests passing. `ws-protocol-conformance.test.ts` pairs the real client stack (through `createWorkflowRunEventV4`) with a fixture mirroring the server route's per-message handling: decode one frame, validate against a local copy of `WsRequestFrameSchema`, dispatch, encode the reply the way `replyMeta` does. `experimental_upgradeWebSocket` needs a real Vercel runtime, so the socket is faked — everything above it is genuine. Covers: the frame shape the server accepts (and that `reqId`/`type`/ `runId` don't leak into the event meta), payload passthrough, exactly one frame per message, 409 → the same typed error HTTP raises, fail-closed on an unknown reply variant, and reqId correlation across concurrent writes. Plus golden byte fixtures, since the schema copy is the one thing here that can silently drift. This is the "golden-frame interop test" the server spec lists as an open gap; the matching half still needs to land in workflow-server. Verified the conformance suite is not vacuous: flattening the client's frame meta fails 5 of its tests. Co-Authored-By: Shalabh Chaturvedi <shalabhc@users.noreply.github.com> Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * match the HTTP RetryAgent's transient-failure policy on WS HTTP event writes go through an undici RetryAgent (RETRY_AGENT_OPTIONS): 5xx and transient connection errors are retried in-process, honoring Retry-After. The WS path never touches undici, so it shipped with no transient-failure handling at all — a single 503 or a mid-write reset surfaced straight to the step runtime and cost a whole step retry where HTTP would have absorbed it in milliseconds. That gap is invisible in a passing test run: writes still succeed, they just cost far more. So copy the policy rather than reinvent it — [500, 502, 503, 504] plus transport failures, undici's default backoff, Retry-After honored, and 429 deliberately excluded for the same reason RETRY_AGENT_OPTIONS excludes it (a firewall challenge this client cannot solve, which in-process retries only amplify). Adds WsTransportError so retryability is a typed property of the failure rather than something the adapter infers by string-matching. Splits resolveWsTransport()/wsReplyStatus() out of postEventFrameOverWs so the retry loop stays readable. The existing "fails closed on an error frame" test used status 500, which is now absorbed by the retry — switched to 403 so it keeps testing fail-closed rather than accidentally testing no-retry. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Co-Authored-By: Shalabh Chaturvedi <7066873+shalabhc@users.noreply.github.com> Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * lazy-load `ws` so the default HTTP path never evaluates it events-v4.ts imports ws-transport.js unconditionally — the transport gate is a runtime branch, not a build-time one — so a top-level `import { WebSocket } from 'ws'` put `ws` and its optional native accelerators on the module-init path of every deployment, including the overwhelming majority that never opt in and never open a socket. Defer it to the first connect, memoized as a promise so concurrent first connects share one import. WebSocket.OPEN becomes an inlined constant so the readyState check doesn't pull the module in just to read it off the constructor. This does NOT remove the need for the bufferutil/utf-8-validate externals this branch also adds: webpack and Rollup both statically follow a dynamic import(), so the build-time story is unchanged. What it buys is that a deployment which never enables the transport never *evaluates* `ws`, so a mis-bundled accelerator can't break it. The test lives in its own file because vitest caches a vi.mock factory result for the life of the module registry — once any test in a file has connected, the factory never runs again and the counter can't distinguish "loaded lazily" from "loaded at import". Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Co-Authored-By: Shalabh Chaturvedi <7066873+shalabhc@users.noreply.github.com> Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * release idle WS transports instead of renewing them forever The transports map was never pruned and WsEventsTransport had no way to close. Combined with eager reconnect that made a connection immortal by construction: the server drains at its own maxDuration and closes, the client immediately reopens, and the server pins a fresh invocation — for a run that finished long ago. A warm container ended up holding a live socket, and a live server invocation, for every runId it had ever served. workflow-server#683 already lists "one invocation stays resident per run rather than per write" as a known gap; this made it "per run, forever". Add close() plus a 60s idle release. There is no "run complete" signal to hang teardown off — the events adapter is a stateless per-write call — so idleness is the available proxy. 60s sits well below the server's ~680s drain deadline, so the client releases rather than the server reclaiming, and well above the gap between steps of an active run. scheduleReconnect() now bails when closed: close() closes the socket, which fires the same close handler an unexpected drop would, and without the guard the transport would instantly reconnect what it just released. request() revives an idle-closed transport rather than failing the write, re-registering itself only if nothing newer has claimed the map slot. Eviction therefore costs one handshake, not an error. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Co-Authored-By: Shalabh Chaturvedi <7066873+shalabhc@users.noreply.github.com> Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * refresh the bearer on an auth_expiry drain workflow-server#683 tags a drain frame with why it is closing: max_duration means the socket aged out and a plain reconnect is right, auth_expiry means the *bearer* ran out and reconnecting with the same one just earns a 401. This client logged the drain and ignored the reason, so against #683 an auth_expiry drain would burn all five reconnect attempts against a token the server had already rejected, then give up. Parse the reason (absent reads as max_duration, so this stays correct against the currently-deployed server) and thread forceRefresh through the getHeaders thunk, which triggers @vercel/oidc's refresh path via a wide expirationBufferMs. Worth being precise about when that can actually help. getVercelOidcToken resolves getContext().headers['x-vercel-oidc-token'] ?? env, and refreshToken() only writes the env var — the request-context header wins. So inside a deployed function there is genuinely no fresher token mid-invocation and the refresh is a no-op; outside one (CLI, local dev, a long-lived server) it works. That makes the guard the load-bearing half: if the re-resolved bearer is byte-identical, decline to reconnect, say so, and wait for the next write — which usually arrives on a new invocation carrying a new token. That failure is marked non-retryable so the retry loop doesn't spin on it either. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Co-Authored-By: Shalabh Chaturvedi <7066873+shalabhc@users.noreply.github.com> Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * document the WS path's instrumentation gap The HTTP branch goes through fetchV4 -> instrumentedFetch, which is not just a fetch wrapper: it opens the OTEL CLIENT span, injects trace context, sets the cache-bust header, emits the DEBUG logs, and routes through the global fetch that Vercel's observability "outgoing requests" view instruments. The comment on fetchV4 records why that matters — bypassing it via undici.request() is exactly what once made v4 event traffic disappear from the log viewer. The WS branch bypasses all of it. With the flag on, per-event writes have no client span, propagate no trace context to workflow-server, and don't appear in the outgoing-requests view; the server's own transport-tagged request metrics are the only remaining signal. That's acceptable for an opt-in POC behind a flag and unacceptable as a default, so write it down where someone deciding to flip the default will read it: instrumenting the transport is a prerequisite for that, not a follow-up nicety. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Co-Authored-By: Shalabh Chaturvedi <7066873+shalabhc@users.noreply.github.com> Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * ship the ws-accelerator externals instead of documenting a workaround `bufferutil` and `utf-8-validate` are optional native accelerators for `ws`, and neither is installed by default. Every bundler has to be told to leave them alone, for two different reasons: Rollup/Vite/Nitro fail the build outright (`Could not resolve "bufferutil" imported by "ws"`), while webpack bundles the JS wrapper without its native `.node` binding and throws `bufferUtil.mask is not a function` at runtime. The webpack half shipped in `@workflow/next`. The Rollup half only existed in `workbench/vite` and `workbench/tanstack-start` as `nitro.rollupConfig.external` — app configs, not shipped code. So a real user of `@workflow/vite`, `@workflow/nitro`, `@workflow/nuxt`, `@workflow/sveltekit` or `@workflow/astro` hit the same build failure the workbench had already worked around, and had to rediscover the fix. Fix it where it propagates: `workflowTransformPlugin` in `@workflow/rollup`, which all of those integrations already install. It is already the home of exactly this pattern for the optional `@opentelemetry/api` peer, so this sits next to its closest precedent. Note the treatment is deliberately the inverse of the OTEL one, which is externalized only when it *can't* be resolved. The OTEL API must load for tracing to work, so a self-contained output has to bundle it when present. These accelerators must specifically NOT load — they are a performance nicety with a correct try/catch fallback in `ws` — so unconditional external is both simpler and safer than risking a half-bundled native module. The two workbench configs drop their local copies, which is what proves the shipped fix actually works rather than being masked by them. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Co-Authored-By: Shalabh Chaturvedi <7066873+shalabhc@users.noreply.github.com> Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * one retry policy for both transports, and no unanswerable waiters Two review findings on the WS events transport. **Retry belongs to `event-retry.ts`, not the adapter.** The WS path had its own retry loop, justified as mirroring undici's `RetryAgent`. That justification was wrong: `RetryHandler` defaults `methods` to GET/HEAD/ OPTIONS/PUT/DELETE/TRACE and nothing overrides it, so the `RetryAgent` never retried an event POST on either transport — which is precisely why `event-retry.ts` exists. Worse, that loop sat *inside* `withEventPostRetry`, so it defeated a compile-checked safety gate: `EVENT_RETRY_ELIGIBILITY` marks `step_started`, `step_retrying` and `hook_received` non-retryable (a replayed `step_started` double-increments `attempt`), and those frames were re-sent up to five times before the gate ever saw a failure. For eligible types the two loops multiplied: 3 outer attempts x 6 inner, with an inner backoff reaching 30s against an outer base deliberately set to 100ms. `postEventFrameOverWs` now makes one attempt and translates failures into the vocabulary that policy already speaks — a transport failure becomes a `WorkflowWorldError` with `code: 'TRANSPORT'`, exactly as `utils.ts` does for a failed `fetch`, and `isRetryableEventPostError` gains one clause keyed on that code. `WsTransportError` loses its `retryable` flag; its only consumer was the deleted loop. Two deliberate consequences. The code-keyed clause broadens HTTP in-process retry to `UND_ERR_CONNECT`, `UND_ERR_CLOSED` and `EAI_AGAIN`, which were in utils.ts's transient set but missing from event-retry.ts's — two hand-maintained lists collapsed into one semantic code. And the stale-token case (drain for auth expiry, refresh yields the same bearer) now gets two in-process attempts that cannot succeed, ~300ms before it falls through to queue redelivery; that is cheaper than keeping a WS-specific policy alive for one call site. `TIMEOUT` is deliberately not in the clause: utils.ts maps a caller-supplied `AbortError` onto it, and a cancelled write must not be re-issued. A status-less reply also stops being a bare `Error` — as one it failed `WorkflowWorldError.is()` and surfaced a protocol version skew as a USER_ERROR. It is now `code: 'PARSE_ERROR'`, the same code utils.ts uses for an unreadable HTTP body, and for the same reason: the write may or may not have landed. **No waiter is left unanswerable.** An undecodable frame, the server's malformed-frame sentinel (`reqId: -1`) and a non-numeric `reqId` were logged and dropped. None can be correlated by construction, so the request that provoked them stayed in `pending` with nothing in existence able to settle it — freed only by the server's own drain (~680s from connect), typically past the invocation's `maxDuration`. Each now fails the connection: every waiter learns why, and the socket is replaced. A reply for an id nobody is waiting on stays log-and-drop, deliberately — that request already settled, so nothing is orphaned, and failing the socket would punish healthy in-flight writes. A per-request deadline backs that up for whatever is left, including a server that accepts a frame and never answers it. Same knob as the HTTP path (`WORKFLOW_REQUEST_TIMEOUT_MS`, 60s), whose doc comment already describes this exact hang-to-SIGTERM pathology. One existing idle-teardown test needed the deadline raised: the idle window and the default deadline are both 60s, so a request could not outlive the former without also outliving the latter. The test is about `inFlight > 0` suppressing the teardown, so it now sets the deadline out of the way. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * Open the ws socket when the invocation starts, not on its first write Lazily connecting bills the whole handshake — an upgrade round-trip plus the OIDC token mint that rides it — to whichever event a fresh invocation writes first. When that is a `step_started` issued as the step body is already running, the event's server-recorded timestamp lands later than the work it describes: the step looks shorter than it was. That is the shape of the e2e timing failure on this branch, where a 9s step measured 6.5s from `getStepMetadata().stepStartedAt`. The queue handler is the earliest point that knows the run id, and a message delivered for a run means writes are coming, so `warmWsEventsTransport` starts the handshake there. By the first write it is done or in flight, and the write just uses it. Nothing about it is load-bearing: - It doesn't await, and can't fail the handler. A warm that fails logs and stops — a never-opened first connect is precisely the case `connect`'s close handler already declines to retry, so no backoff loop starts for a run that may never write. The first real write connects as it would have anyway, carrying the shared retry policy. - No-op unless `WORKFLOW_EVENTS_TRANSPORT=ws`, and no-op for the api-workflow proxy World, which can't serve an upgrade at all — the same fallback the write path takes. - Warming arms the idle timer as if a request had settled, so an invocation that warms and never writes (a health probe carrying the run id it is about to create) releases its socket on the usual 60s rather than stranding it. The socket is not `unref`'d, so a stranded one would hold this process and a server invocation open. Also closes a race that warming makes reachable: `close()` can only drop the connection it can see, so a release landing mid-handshake left the socket to install itself afterwards onto a transport already evicted from the cache, which nothing would then ever close. The `open` handler now declines to adopt a socket whose transport was released while it was connecting. This was already reachable via the eager reconnect path, just much harder to hit. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * changeset: just the env var Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * inline the ws-accelerator predicate at its only call site Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * refactor(world-vercel): trim ws-transport comments Comments were 47% of the file. Cut the historical narration, the restatements of adjacent code, and the repeated rationale (the `unref` reasoning appeared four times, per-connection reqId three), keeping the non-obvious facts: `ws.send()` reports failure via callback instead of throwing, reqId is per-connection so `pending` must be too, the unknown-reqId case is deliberately non-fatal, the auth_expiry same-token bail-out, and why the idle timeout exists at all. No code changes. Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * own transport selection in the transport module `events-v4.ts` was assembling the WS transport itself: reading the opt-in flag, resolving the URL, deciding which Worlds can use a socket, minting the per-connection header thunk, and holding the two once-per-process log latches. None of that is about turning an event into a frame, which is what the rest of that file does. Move it next to the socket it configures — `events-v4.ts` now consumes one seam (`resolveWsTransport`) plus the gate, and `queue.ts` gets `warmWsEventsTransport` from the module that owns the warm. `headersToRecord` now lives in `http-core.ts` because both callers need it and neither may import the other: `events-v4` already depends on the transport, so the reverse edge would be a cycle. Test fallout, and the reason the move is worth it: `events-v4-ws.test.ts` mocked `getWsEventsTransport` to observe the resolve step, which no longer intercepts anything now that the call is intra-module — an ESM mock replaces a module's exports, not its own call sites. That mock's tests were only ever about selection, so they move to `ws-transport.test.ts`, where the real selection code runs against the existing fake-socket harness instead of a stub. `resetWsEventsTransportsForTest` clears the log latches so the once-per-process assertions don't depend on test order. What stays behind mocks `resolveWsTransport` and covers what that file is actually for: reply frame in, `Response`-shaped result out — including the null-resolve fallback to HTTP, which nothing covered before. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * import `ws` statically The lazy `import('ws')` was there to keep the package off the module-init path of deployments that never opt in — `events-v4.ts` imports this module unconditionally, since the transport gate is a runtime branch. Measured, that buys ~17ms: `require('ws')` is 16.5-18.0ms cold, 13 modules, and neither `bufferutil` nor `utf-8-validate` loads (optional peers, absent by default). Bundle size is identical either way — webpack and Rollup both statically follow a dynamic `import()`, which is why the externals in `@workflow/builders` are unaffected by this change. For 17ms it cost a memoized promise, an inlined `WS_READY_STATE_OPEN` (so a readyState check wouldn't force the module to load just to read a constant off the constructor), and a whole test file — `ws-transport-lazy.test.ts` had to live alone, because vitest caches a `vi.mock` factory result for the lifetime of a module registry, so only a file that connects exactly once can observe the laziness at all. It also skewed the thing this branch exists to measure. The import lands inside the first connect, so on a warm container it is billed to whichever event write opens the socket, inflating the timestamp of the step it labels — the same distortion the queue pre-warm was added to remove. Also drops `WS_READY_STATE_OPEN` in favour of `WebSocket.OPEN`, now that reading it is free. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * tighten the comments on the ws transport Comments only — no code changes in this commit. Cuts ~150 lines of prose across the WS additions. The rule applied: keep the design factors a future reader needs (why the connection is scoped to a run, why a bad reply takes the socket down, why the accelerators are externalized unconditionally, why `TIMEOUT` is excluded from the `TRANSPORT` classification) and drop the narrative of how the code got here — which revision did what, what an earlier attempt got wrong, what was measured on the way. That history lives in the PR and the git log, where it doesn't have to be re-read on every visit to the file. Biggest reductions: the retry essay above `postEventFrameOverWs` (30 lines to 11), the flag's OTEL-gap note (34 to 13), the OIDC refresh explainer (26 to 14), the accelerator rationale in `@workflow/builders` (26 to 14), and the conformance suite's header (28 to 17). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * inject W3C trace context on the ws upgrade Frames carry no headers, so the upgrade is the only place this transport can propagate context; the server parents a run's event spans to whichever invocation opened the socket. Covered in trace-propagation.test.ts, both with and without an active span. Splits the opt-in gate into an import-free ws-transport-enabled.ts so callers can answer it without loading this module (used by the next commit). Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * load the ws transport module only when it is enabled Both call sites read the gate from the import-free module and dynamically import ws-transport.js behind a true result, so a deployment on the HTTP default never pays ws's ~17ms of module init. The queue pre-warm absorbs it for one that opted in, keeping it off the first event write. Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * document WORKFLOW_EVENTS_TRANSPORT as experimental Names the instrumentation gap (no client span per write) and the proxy path where the variable is ignored. Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * correct why the ws accelerators are externalized No bundler fails the build on the unresolvable require — verified against Rollup 4.62. webpack half-bundles the native module and Vite substitutes a stub that makes the require succeed; both leave bufferUtil.mask undefined and throw only once a frame reaches the native masker at 48 bytes, which every CBOR event frame does. Same claim was repeated in the rollup plugin and its test. Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * trim the WORKFLOW_EVENTS_TRANSPORT docs to user level Mirrors the other Vercel World env vars: same facts on both pages, each in its page's format. The instrumentation and socket-lifetime detail belongs in the code, not in a user-facing reference. Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * cover the vite bundler in the ws transport lane Vite substitutes a stub for ws's absent native accelerators rather than failing the require, so nothing catches it until a masked frame reaches 48 bytes — and this job's three existing lanes are esbuild, turbopack and nitro. Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * claim only what is measured about rollup and the ws accelerators The rationale asserted plain Rollup was "safe by accident" via a mechanism only ever observed in a minimal repro. Nitro traces and externalizes `ws` in a production build, so the bundled path is not reached there at all. Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * Give the events socket an explicit lifetime instead of an idle timer `openWsChannel` / `closeWsChannel` bracket one invocation of the flow route, and are the only calls anywhere that create a channel. Writes ask `resolveWsTransport` whether one is open — a lookup now, never a create — and take pooled HTTP when it says no. That removes the reason the idle timeout existed. A lazily-created socket has no owner, so a timer was the only thing able to end it, and the socket is not `unref`'d: the process could not exit, and a server invocation stayed pinned, for the full window past the last write. It also settles `run_created`. The trigger path opens no channel, so a lone write no longer pays for a handshake it cannot amortize — `start()` runs in an arbitrary request handler with no boundary the SDK can see. Refcounted rather than a flag: inline step executions ride the flow topic on per-step topics, so a run's steps can be concurrent invocations in one instance sharing the channel, and the first to finish must not cut the others short. A failed connect closes the channel so the invocation's writes fall back to HTTP instead of each paying its own doomed handshake. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * Name the one reply header the WS path does not map The server copies six headers into an `event_ack`'s meta and this record maps five. The sixth, `X-API-Deprecated`, is inert today — the v4 route's middleware chain has no deprecation middleware to set it — but the record is the only header source a WS reply has, so an unmapped key is gone rather than merely unread, which is not true of the `Response` the HTTP path returns. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * docs: note that WORKFLOW_EVENTS_TRANSPORT=ws is ignored on the proxy path The api-workflow proxy is an HTTP-only REST gateway and does not forward a WebSocket upgrade, so a World configured with projectConfig keeps writing events over HTTP regardless of the setting. Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * ci: gate the ws-transport e2e lanes on a label Three real `vercel deploy`s per run is too much to charge every unrelated PR in the repo for a transport that is off by default. PRs opt in with `ws-transport-test` (or `workflow-server-test`, which already exists to test the half of this the protocol lives in); main keeps the signal on every commit. The required aggregate has to allow the lane to be skipped in that case, so its status is asserted only when the lane was actually supposed to run. Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * chore: regenerate pnpm-lock against current main main resolved `ws` to 8.20.0 as a transitive peer; this branch adds it as a direct dependency of world-vercel and floats it forward, which rewrites every `openai@x(ws@y)` peer key in the lockfile. Merging main textually combined the two, leaving those keys pointing at a `ws` entry the merged file no longer had — `--frozen-lockfile` then failed with ERR_PNPM_LOCKFILE_MISSING_DEPENDENCY on the PR's merge ref. Regenerated from main's lockfile so ours is a minimal delta on top of it. Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * fix(world-vercel): align ws on the version main already resolves The lockfile broke on the PR's merge ref, not on this branch's head: main resolves ws@8.20.0 as a transitive peer, and a `^8.21.1` direct dep here floated it forward, rewriting all 73 `(ws@8.20.0)` peer keys. Git merged the two lockfiles without a conflict but left main-side keys pointing at a ws entry the merged file no longer had, so `--frozen-lockfile` failed with ERR_PNPM_LOCKFILE_MISSING_DEPENDENCY. `^8.20.0` resolves to the copy main already has, so the lockfile delta is the two importer entries instead of a repo-wide rewrite that re-breaks every time main moves. Also keeps one ws in the store rather than two. Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * Bind the channel release to the instance it claimed closeWsChannel resolved the transport by URL, but the refcount lives on the instance. A channel is evicted from the map as soon as it closes — a refused upgrade does that on the connect path — so the next opener for the same run registers a different instance under the same URL, and the first invocation's close then decremented that one instead. It dropped a socket a live invocation was still writing over, and for the event types EVENT_RETRY_ELIGIBILITY marks non-retryable there is no second attempt to carry the in-flight write over HTTP. openWsChannel now returns an idempotent release closed over the transport it incremented, and queue.ts holds that instead of re-resolving the run. The close awaits the open's own promise, so it also can no longer land ahead of the claim it releases. Also names the scope of the connect-failure de-opt: it covers the handshake only, so a channel that connects and then fails every write keeps taking the WS path. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * Decode a transport result, not a Response Main extracted the v4 POST decode into a helper typed `Response` while this branch narrowed the POST result to `FrameResponseLike`, because the WS branch synthesizes its result rather than holding a real `Response`. The two merge without a textual conflict and then fail to typecheck. Widen the helper: it reads only the two members `FrameResponseLike` declares, and a `Response` still satisfies them, so the HTTP call sites are unchanged. Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * Re-run CI Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * Re-run CI Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * Reconcile the WS transport with main's v4 POST rework main moved the materialized POST result off the `x-wf-*` response headers and onto a typed CBOR body, and added a second response shape: two callers now POST with `Accept: application/vnd.workflow.v4-frames` and read back a sentinel-terminated sequence of frames. A frame stream has no representation in a protocol that pairs one reply frame with one request frame, so the WS switch moves off the shared poster and onto `createWorkflowRunEventV4` alone — the materialized write, which is the hot per-step path this branch exists to shorten. `run_started` and the `hook_received` preload stay on HTTP. `decodeCreateEventResponse` takes `FrameResponseLike` rather than `Response` because the WS branch has none to hand over; a real `Response` satisfies the interface, so the HTTP callers are unchanged. The ids now come out of the CBOR body, so `replyMetaToHeaderRecord` no longer maps any `x-wf-*` name — only the two headers `errorFromV4Response` reads. Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * Re-run CI Resample the WS-arm sleepingWorkflow failure: it has now recurred on a second axis (nextjs-turbopack, 7709ms; previously vite, 7570ms), so the arm needs more samples before the skew can be called WS-specific or repo-wide flake. Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * Re-run CI Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * Re-run CI Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * blank * blank --------- Co-authored-by: vercel[bot] <35613825+vercel[bot]@users.noreply.github.com> |
||
|
Nathan Colosimo
|
22349e95fd |
perf(core): load replay suffix in one request (#3205)
* perf(core): stream replay suffix in one request * perf(core): load replay suffix in one request Signed-off-by: Nathan Colosimo <110621881+NathanColosimo@users.noreply.github.com> * test(world-vercel): use streamed run start fixtures * refactor(events): simplify return-all plumbing Signed-off-by: Nathan Colosimo <110621881+NathanColosimo@users.noreply.github.com> * Return complete local run preloads * Document workflow event limit * fix: make return-all event loading resilient * Simplify full event listing * refactor(world-vercel): omit event limit for full loads * fix(world-vercel): explicitly request complete event logs --------- Signed-off-by: Nathan Colosimo <110621881+NathanColosimo@users.noreply.github.com> |