mirror of
https://github.com/vercel/workflow.git
synced 2026-09-14 19:59:43 +08:00
workflow-auth-docs
154 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Pranay Prakash
|
6cc851c342 |
[core] Stop sending a slot snapshot on step executor writes (#4096)
The only thing a World does with `eventCount` is bump-and-report: when the write lands above the position named, it reads the events in between and returns them so the writer can merge them without a second round-trip. The replay loop and the suspension handler merge that page into their loaded log. The step executor has no log to merge into, so it took the page's highest position and discarded the rest. In production that discarded read fell on a third of all `step_started` writes (10.8M of 13.4M skipped-slot report reads per day were on executor event types), each a strongly consistent DynamoDB query on the run partition with resolved refs, on the response path. This removes the executor's `knownSlot` / `observeSlot` machinery, the `slotSnapshot` executor param, and the `batchCommittedSlotCeiling` the suspension handler computed only to seed it. The loop's and the suspension handler's own snapshots are unchanged; they consume their reports. The World contract already describes omitting the count for a caller with no loaded log to be stale against; the executor now matches it. Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com> |
||
|
Pranay Prakash
|
e00b1a57ee |
perf(world-vercel): batch a fan-out's step-execution queue publishes (#3838)
* perf(world-vercel): batch a fan-out's step-execution queue publishes A `Promise.all` fan-out dispatched one queue message per branch. Those publishes ride the shared default undici agent (8 connections, HTTP/1.1, `pipelining: 1` — see `getQueueDispatcher`), and `handleSuspension` is awaited in full before the first inline step body runs, so an N-branch fan-out paid ~N/8 serialized round trips straight onto time-to-first-step. The `step_created` writes were already batched and HTTP/2-multiplexed; the publishes were the remaining per-branch round trip. Adds an optional `Queue.queueBatch`, implemented on `@vercel/queue`'s `experimental_sendBatch` (0.5.1), and uses it for the batched fan-out fold's publishes. Each commit chunk now publishes in one request instead of up to 32. `queueBatch` reports per-entry outcomes rather than throwing, because a batch can partially fail. `queueMessages` in core keeps the previous all-or-nothing behavior for this call site: it rejects if any entry failed, so the delivery is redelivered and republishes the set, deduped by the per-step `idempotencyKey` the caller already passed. Worlds without `queueBatch` fall back to concurrent single sends. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(core): reject a short queueBatch result set instead of reading it as success `queueMessages` only inspected `error`, so a World whose `queueBatch` returned fewer results than it was given messages reported success for the whole batch. The omitted entries were never published and nothing raised: `handleSuspension` resolved, the delivery was acked, and those steps were never dispatched, so the run stalls with no error recorded anywhere. Reproduced at 64 branches against a World returning half its results: 32 of 63 steps silently lost. world-vercel guards this internally and `@vercel/queue` length-checks its own response, so it was not reachable through the world added here. It is reachable through the interface `building-a-world` opens to third-party worlds, which is where the check belongs. Documented on the interface and in the guide alongside it. Also notes that the batch grouping degenerates to one request per message under WORKFLOW_SEQUENTIAL_REPLAYS=1 (per-step physical topics are one of the routing dimensions groups split on), and corrects the comment claiming the error's `retryable` flag is consumed downstream: nothing reads it yet. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(world-vercel): carry trace context on each batched queue message `experimental_sendBatch` injects the active trace context into the multipart REQUEST headers, and the per-part headers it builds never see it. VQS stores headers per message and re-emits a stored `traceparent` at delivery as `x-vercel-queue-traceparent`, which is what lets a consumer attach a span link back to its producer, so a batched message arrived with no producer context and its `vqs.process` span got no link. `send()` is unaffected: for a single message the request headers ARE that message's headers. At 64 branches that was 63 of 64 step dispatches losing the transport-level producer link. The run's own step tracing was never affected: that carrier travels in the message payload (`WorkflowInvokePayload.traceCarrier`), which is what the consumer builds its trace context from, not a header. Injects the active context into each entry's headers in `queueBatch` — last, so it wins over caller-supplied `opts.headers` exactly as the SDK's own injection does — and honors VERCEL_QUEUE_TRACE_PROPAGATION so that kill switch still covers both paths. `getTraceContextHeaders()` is factored out of `injectTraceContextIntoHeaders` so the two share one source. Verified on the wire against a stub VQS speaking the real batch endpoint: `traceparent` carrying the producer's traceId/spanId lands on all 64 multipart parts through the real SDK, with the per-message idempotency keys still alongside it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> Co-authored-by: Karthik Kalyanaraman <karthik.kalyanaraman@vercel.com> |
||
|
Nathan Colosimo
|
7a46a81a53 |
Upgrade to Zod 4.5 and compile schemas (#3902)
Co-authored-by: VaguelySerious <mittgfu@gmail.com> Co-authored-by: Peter Wielander <peter.wielander@vercel.com> |
||
|
Peter Wielander
|
ec57aff3be | [core] Log pending consumers in divergence diagnostics (#4021) | ||
|
Alex Langenfeld
|
4547e1a7a9 |
feat(streams): add writer session seam (#3832)
## Summary & Motivation Gives one in-memory stream writer a stable identity and its own sequence space, so a transport can preserve chunk ordering across a mid-stream HTTP/WebSocket transition. `Streamer.streams.createWriteSession` is optional — Worlds that don't implement it keep using `write`/`writeMulti`/`close` unchanged. Abort disposes the session rather than closing it, since a producer failure is transport cleanup, not stream completion. ## Test Plan Tests added, plus the full `@workflow/world-vercel` suite and package builds/typecheck pass. Root build/typecheck is blocked locally by a missing Rust toolchain for the unrelated `@workflow/swc-plugin`. |
||
|
Peter Wielander
|
61fb1f93bd | [core] Add a retention option to start() (#3787) | ||
|
Pranay Prakash
|
7cc5c88a8b |
[core] Settle a hook's awaiter in-process instead of re-invoking, on creation and on conflict (#3938)
* [core] Settle a hook's awaiter in-process instead of re-invoking, on creation and on conflict * [core] Address review: deterministic hook signal tests, split changesets, document the boundary - hook.test.ts: drive the idle poll with explicit macrotask turns instead of a fixed 20ms sleep (Copilot) - Split the changeset so each package's entry says only what changed in it - runtime-tuning docs: hook-only suspensions no longer always park; the hook write continuation is the one exception Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com> |
||
|
Karthik Kalyan
|
fbfb9fe869 |
Validate world.analytics arguments up front (#3943)
* Validate world.analytics arguments up front
Every analytics method now checks its arguments before making a request
and throws a RangeError naming the limit it broke: the ids, the
pagination limit against the cap for that listing, and the attribute
filter's pair count, key length and value size. Because analytics is an
optional capability, callers wrap it in a catch, which turned an invalid
argument into what looked like an empty result rather than an error.
Two arguments that used to be dropped silently now fail too. A limit of 0
fell back to the default page size, and a startTime without a matching
endTime turned a listing you meant to bound into an unbounded one that
looked like a normal answer.
Exports ANALYTICS_RUN_SCOPED_PAGE_LIMIT, ANALYTICS_PAGE_LIMIT and
ANALYTICS_MAX_ATTRIBUTE_FILTERS so callers can check the bounds
themselves.
Deprecates analytics.events.listByCorrelationId() in favour of
analytics.events.list({ runId, correlationId }), which issues the same
request and also accepts an eventType filter. It keeps its own
implementation rather than delegating: list() treats correlationId as
optional and skips an empty one, so a delegation would turn an empty id
into an unfiltered listing of the run.
Documents every analytics method in the reference. events.getMany() was
missing entirely, seven methods shared one code block with no parameters
or return shapes, and none of the limits were written down.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* Parse attribute key timestamps as UTC
firstSeenAt and lastSeenAt were the only analytics timestamps still on a
plain date coercion. The values arrive without a timezone designator, so
that read them in the process's local zone and every other field in the
namespace read them as UTC — a seven-hour skew on those two fields alone
for anyone running outside UTC.
The added test fails without the fix under TZ=America/Los_Angeles.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* docs: drop the deprecated correlation-id listing from the reference
A reference page should describe the API you should reach for. The
deprecation notice lives on the method itself, so editors surface it
where it matters without the page advertising a method nobody should
start using. Also drops it from the page-limit table.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* Close two gaps in the analytics argument guards
Run ids were validated with workflowRunIdSchema while every other id used
a regex mirroring the backend. Those disagree: z.ulid() accepts a
lowercase body and a first character above 7, and the backend accepts
neither, so the most-used parameter had the leakiest guard and still
produced the 400 this is meant to prevent. Run ids now use the same
pattern as the rest.
A supplied-but-empty filter value was also still being dropped —
correlationId, the optional runId scope on hooks.get, and workflowName
all tested truthiness. Dropping one widens the result set rather than
narrowing it, so an empty correlationId listed the whole run and an
empty workflowName listed every workflow. That is the same failure the
limit and time-window guards were added to prevent, and the comment on
listByCorrelationId already described the hazard. They now compare
against undefined, so an empty id throws and an empty name is forwarded
for the backend to match.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* Raise argument rejections as a typed, non-retryable error
The guards threw bare RangeErrors, which left a caller — or an agent
driving this API — parsing English to decide whether to fix the call or
retry it. They now raise WorkflowWorldError with
code: 'INVALID_ARGUMENT', the code the rest of this client already uses
for its transport and throttle failures, so the retry decision is a
field lookup. normalizeEventIds moves with them rather than staying the
one guard that throws a different type.
Also sharpens the four messages that made a caller do the work:
a half-open window now names the bound that is missing rather than
restating the rule, an inverted window prints both ends, and the
attribute-value and event-id batch errors report the size measured
rather than only the bound they broke.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* Name the method and the field on an argument rejection
Two things a caller could not get without reading prose. The same guard
runs behind several methods, so `runId must be a workflow run id` was
identical whether it came from events.list or steps.get — fine with a
stack, lossy once the error has crossed a log line. And the offending
argument was only available as the first token of the message, which is
the part most likely to be reworded.
Messages now open with the method, and WorkflowWorldError carries an
optional `field`:
analytics.runs.list: pagination.limit must be an integer between 1
and 100 (received 9999)
→ code: 'INVALID_ARGUMENT', field: 'pagination.limit'
`field` is additive on the error class and set only by these guards, so
nothing that reads the existing properties changes.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
||
|
Shalabh Chaturvedi
|
4a18b0133a |
fix(world): accept lazy terminal run data (#3914)
Co-authored-by: vercel[bot] <35613825+vercel[bot]@users.noreply.github.com> |
||
|
Karthik Kalyan
|
2668e3325b |
Durable hook resume: write, then wake (#3841)
* test(core): reproduce lazy resume disposal race * Fix durable hook resume race * Fail closed on unknown hook wakes * Improve unsupported hook wake diagnostics * Address durable hook resume review feedback * Harden producer-committed wake handling * Serialize durable hook resume: write, then wake resumeHook() now dispatches strictly serially: the hook_received event is made durable first, and the workflow wake is published only after the write is acknowledged. The wake is a plain runId message (the shape the sequential path always published), so the producer-committed wake barrier, its queue-message field, and the HOOK_RESUME_INPUT_VERSION bump are all removed — no consumer or backend coordination is needed, and either side rolls back independently to today's behavior. The pre-write ops flush now partitions serialization ops: producer-push uploads are awaited before the event commits (the payload must not point at bytes still in flight), while consumer-settled reader ops — a dehydrated WritableStream, e.g. a manual webhook's responseWritable — are backgrounded. Awaiting those deadlocked the resume against its own wake (webhookWorkflow failing across the whole e2e matrix). Also: wake retries stop on definitive 4xx errors instead of burning the retry budget; WORKFLOW_DISABLE_LAZY_HOOK_RESUME no longer gates anything and is ignored; the internal resumeHookDurable alias is removed. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * Address review: retry classification, wake dedup, 409 passthrough - Wake retry classification now actually fires against @vercel/queue: its errors carry no status field, so classify by the World's deployment-unavailable hook, then numeric status, then the queue client's definitive-4xx error names. - The wake publish carries idempotencyKey `hook-<resumeId>` on the claim path, so a retried publish whose response was lost dedups instead of costing a duplicate full replay. - EntityConflictError (HTTP 409) from the durable write is no longer re-keyed to HookNotFoundError: every 409 the backend emits on this write today is transient (slot conflict past the server's retry budget, claim race) and committed nothing, so it surfaces retryable instead of presenting as a permanent 404. - Stamp workflow.hook.resume_committed / wake_published span attributes after each leg resolves, making stranded resumes (committed event, no wake) queryable from traces. - Document on the public resumeHook signature that passing the token (not a cached Hook) is what makes the write idempotent-on-retry. - Changeset/changelog: note the ended-run behavior change (late webhook deliveries to finished runs now 404 instead of 202) and the 409 passthrough. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Pranay Prakash <pranay.gp@gmail.com> Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Nathan Colosimo
|
e9d5c56701 |
[core] Prepare replay payloads as event frames arrive (#3548)
* Prepare replay payloads from streamed events Signed-off-by: Nathan Colosimo <110621881+NathanColosimo@users.noreply.github.com> * Fix streamed replay preparation invariants * fix(core): gate replay startup work by VM engine * refactor(core): simplify replay startup state * fix(core): preserve replay startup ordering * refactor(core): simplify setup failure handling * fix(core): observe replay load after setup failure * refactor(core): simplify replay encryption key promise * fix(core): scan only appended replay events * refactor(world-vercel): type replay stream outcomes --------- Signed-off-by: Nathan Colosimo <110621881+NathanColosimo@users.noreply.github.com> |
||
|
Peter Wielander
|
855e47990c |
[core] Make a duplicate attr_set inert instead of terminal (#3849)
* [core] Make a duplicate attr_set inert instead of terminal A workflow-body attribute write draws a correlation id that resolves exactly once: the dispatcher's consumer takes the matching event and deregisters. A second event under that id therefore has no callback left and never will. `attr_set` had no entry in ENTITY_EVENT_CLASS_BY_TYPE, so the duplicate skip could not take it, and `PARKABLE_EVENT_TYPES` does list the type, so it was parked for a consumer that could never come. Parking is settled by the workflow function returning, and a survivor there is reported through `strandedEvent` as a replay divergence. So the run did all of its work, every step succeeded, and the final replay failed it, deterministically enough to burn the whole replay-divergence recovery budget and terminate with CORRUPTED_EVENT_LOG. Give `attr_set` a class so the straggler is skipped like every other one: committed but inert. Parking still covers the first arrival, for a replay that walks past an attribute event before the body reaches the call that claims it. An attribute write from a step body carries no correlation id and is consumed by the structural lifecycle consumer, so it is unaffected. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * [core] Release a parked duplicate, and agree with the UI about one The class map alone decides a straggler only where the walk meets it after a consumption recorded the class. When neither copy has a consumer yet both park — the walk steps over the first and re-enters in the same tick, with nothing consumed and so no class recorded — and the drain then claims one and holds the other for a callback that will never be registered. That survivor is `strandedEvent`, which is the CORRUPTED_EVENT_LOG this branch set out to stop, reached by the other road. `dropParkedDuplicates` releases it on the same terms the walk skips one. Not an `attr_set` property: `wait_completed` parks in pairs too, and `ONE_SHOT_EVENT_TYPES` only sees the order where the consumption came first. Giving `attr_set` a class also moved the observability UI, which reads the same `entityEventClass` to grey out events a run passed over. It kept treating the straggler as live, because its terminal-class set had no `attr_set` while the dispatcher's consumer does deregister on the first event under an id. The two now share `classifyEntityEvent` and `TERMINAL_EVENT_CLASSES` rather than each keeping a copy of the rule. That sharing needs the entity rule to be exact, because a step-written `attr_set` carries no correlation id: keyed on the run it would collapse every attribute write a run made into one class, and a captured production log in `__fixtures__` holds forty. `classifyEntityEvent` gives such an event no class at all, so neither side can read the second as a repeat of the first. The shared fixture corpus had nothing for `attr_set`, which is why the drift between the two halves went unseen. It has four now, and each of them fails on both sides without the fix above it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> Co-authored-by: Pranay Prakash <pranay.gp@gmail.com> |
||
|
Pranay Prakash
|
ffc58078d0 |
Stop logging on healthy workflow execution (#3878)
A successful run printed several lines that described the runtime working correctly. Most of it was fallout from defaulting the events transport to WebSockets (#3702): three breadcrumbs written while the transport was opt-in became default-path output, because each one reported a choice the caller no longer makes. - `world-vercel: using ws events transport (…)` ran once per cold start on every deployment, naming the transport it was always going to use. - The `projectConfig` proxy fallback warned once per process. That World cannot hold a socket, so with WS on by default every CLI command and the observability app warned about a fallback nobody asked for and nobody can act on. Debug-gated and reworded from "requested but" to "unavailable for". - The `max_duration` / `auth_expiry` drain notice is routine: the transport reconnects from the close that follows and no write is lost. Swept for the same shape elsewhere: - `world-local`'s queue-concurrency notice fired per message once a fan-out exceeded the limit — the semaphore doing its job. - `@workflow/world`'s active-run recovery line printed on every dev-server restart with work in flight. The re-enqueue *failure* above it stays unconditional; that one leaves a run unresumed. - The port-detection diagnostics in `@workflow/utils` keyed off `NODE_ENV=development`, which is the only environment that reaches them, so the gate made them unconditional for their whole audience. All of it moves behind `DEBUG=workflow:*` via a new `debugLog` in `@workflow/utils`, joining world-vercel's existing `httpLog` and `logRetry` output under one selector. Warnings and errors are untouched, so a run that actually goes wrong is no quieter than before — the ws-transport tests that assert failures are never silent still pass unchanged. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Co-Authored-By: Pranay Prakash <1797812+pranaygp@users.noreply.github.com> |
||
|
Peter Wielander
|
d9e0777eb8 | [core] Never write hook_received eagerly on the lazy resume path (#3794) | ||
|
Nathan Colosimo
|
d62b44473b |
[core] Prune schema modules from workflow bundles (#3550)
* [core] Prune schema modules from workflow bundles * [world] Inline one-off validation options * refactor(world): simplify event schema boundaries * refactor(world): simplify event schema boundaries * fix(world): keep noop metadata schema-free * refactor(world): drop zod 4.4 compatibility * test(builders): cover workflow API bundle boundary |
||
|
Peter Wielander
|
bf9de1cd81 | [core] Re-arm a wait continuation delivered before its wait elapses (#3743) | ||
|
Peter Wielander
|
7e48e7b4de |
Re-enable the sealed log by default (#3737)
* Revert "[world] Make the sealed log opt-in instead of default-on (#3735)" Reverts |
||
|
Pranay Prakash
|
f771585486 |
fix(world-vercel,world-local): hold process-wide state on globalThis (#3728)
* fix(world-vercel,world-local): hold process-wide state on globalThis Both packages are bundled into the host application's server build, and a bundler keys module identity on (resource, layer) — Next.js alone builds `instrument`, app-route, `ssr` and `edge` layers, so one process holds one copy of each of these modules per layer. Every module-scope `const`/`let` in them was therefore per-copy state wearing the costume of a process singleton. vercel/workflow#3493 made `@workflow/world-vercel` bundled rather than external and the events WebSocket transport regressed to HTTP for exactly this reason: the queue consumer registered its channel in the `instrument` copy's `Map` and the write path looked it up in the route copy's empty one. A deterministic miss, for the life of the process. `@workflow/world-local` had the same exposure all along — including `runFileLocks`, where a duplicated mutex simply stops mutually excluding. Add `globalSingleton()` to `@workflow/utils` (the primitive `@workflow/core` already hand-rolls for its World cache) and route every mutable module-scope binding in both worlds through it. Regression cover, in three layers: - `global-singleton.test.ts` pins the primitive's semantics. - `ws-transport-module-copies.test.ts` imports the module twice in one process and asserts a transport registered by one copy is found by the other — it fails on a plain module-scope `Map`, which is the shipped bug. - `scripts/lint/module-scope-state.mjs` fails the class: an AST rule banning mutable module-scope state in these packages, with `// per-copy-ok: <why>` as the deliberate escape. Wired into both packages' `vitest run src`, with fixture self-tests so it cannot rot into a no-op. * test(world-postgres): pin the module-scope-state rule for the postgres world It is deduped today only because `getRuntimeRequire()` loads it — a property of how it is loaded, not how it is written, and exactly what changed for world-vercel in #3493. The package is already clean; this keeps it that way. * docs(worlds): codify "a world must not hold mutable module state" A world package is loaded one of two ways, and only one of them gives it a single module instance: a runtime `require()` (deduped by Node) or the host's bundler (one copy per layer). Which one you get is a property of how the world is loaded, not of how it is written, and it changed under `world-vercel` in #3493 — so the rule has to be "never rely on module scope", not "rely on it until someone flips a config". Written down in the four places someone can meet it: - `docs/content/worlds/{v4,v5}/building-a-world.mdx` — a "Process-wide state" section for custom-world authors, with the loading modes spelled out and a nudge to prefer World-instance state over a global. - `packages/world/README.md` — the same constraint on the contract package. - `CLAUDE.md` — so the next contributor working in these packages sees it. - `packages/core/src/runtime/world.ts` — at the two static imports, which is where the difference between a bundled world and a required one originates. The rule's own error message now teaches it too, rather than naming a helper. Consolidates the guard while here: `@workflow/utils` owns the rule and its fixture self-tests, and sweeps every *published* `packages/world-*` discovered at runtime, so a world package added later is covered without anyone remembering. Each world keeps a one-assertion mirror for locality. * style: drop prose em dashes from this branch's new text #3704 landed a repo-wide writing pass hours after this branch was written and took `world-vercel/src` from 406 em dashes to 130 (`ws-transport.ts` alone went 35 to 1). This branch's docs section, README, comments and lint messages were written before that and would have put 36 of them straight back into the files that were just cleaned. Rewritten sentence by sentence rather than by substitution: an em dash becomes a colon, a comma, a full stop or a parenthetical depending on what it was doing. Also fixes a real defect the sweep surfaced: `world-postgres`'s guard test was generated through a shell heredoc and had literal backslash-backticks in its doc comment. * Update .changeset/world-module-scope-state.md Co-authored-by: Peter Wielander <mittgfu@gmail.com> Signed-off-by: Pranay Prakash <pranay.gp@gmail.com> * fix(core): build the entrypoint's queue handler from getWorld() Adopted from #3666 by @MintedKenny, which implements #3665 and could not run CI as a fork PR. One line of behavior: `workflowEntrypoint`'s lazy handler init calls `getWorld()` rather than `getWorldHandlers()`. `getWorldHandlers()` owns a second, build-time-safe cache, so calling it from the runtime route built a *second* World in the same process. That costs a stateful World duplicate resources on every instance — world-postgres eagerly constructs a `pg.Pool` (default `max: 10`) and a nested world-local World in `createWorld()`, so self-hosted users have been paying for two of each — and, for a bundled world package, the two Worlds are built by two different module copies, which is the mechanism behind the WS transport regression the rest of this branch contains. The public `getWorldHandlers()` and its separate build-time cache are unchanged; only the runtime route stops using it. Kept from the original: the regression test asserting the factory runs exactly once, and the api-reference wording (re-applied over #3704's list punctuation). Not taken: renaming the `workflow.route.get_world_handlers` span. It is a distinct span from the per-request `workflow.route.get_world` at the top of the flow route, and reusing that name would collide with it in traces and in `runtime-trace-mode.test.ts`; a comment records why the name outlived the call. Co-authored-by: Kenneth <kenneth@standardforensics.com> Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: address AI review on the module-scope work Two blocking findings, both real: - **Cross-version state sharing** (`ws-transport.ts`). A process can hold two *published versions* of `@workflow/world-vercel` (a transitive dependency pinning an older `@workflow/core`, which depends on this package by exact version). Both wrote to the same unversioned `Symbol.for` key, so one version's write path could be handed a `WsEventsTransport` built by the other's class and frame against a protocol it may not share — with no version negotiation on the socket to catch it. `shapeVersion` cannot express this: the container is stable, the hazard is its contents. The registry and the events dispatcher recycler are now keyed by package version. The plain connection pools stay unversioned; sharing those across copies is the point. - **The documented pattern failed the rule this PR adds.** The custom-world docs teach `store[StateKey] ??= …`, which the rule flagged as a field write. It now recognizes state rooted at `globalThis`, following one alias hop, which is also what `core/private.ts:23` and `next/src/index.ts:58` are already doing correctly (core drops 26 findings to 22, next 7 to 6). The docs also now say outright that `globalSingleton()` is the same thing, since AGENTS.md prescribes it and the page did not mention it. Rule precision, from the review's probes: - `.mts`/`.cts` are scanned. `@workflow/world-testing` is authored in `.mts`, so its entry in the sweep was passing vacuously — with the walk fixed it reports a real finding, now annotated (it is a standalone `serve()` entry). - Mutations in top-level statements no longer count. A table filled at module evaluation is identical in every copy; divergence needs a later write. - `static` class fields are collected, attributed to the class name. - An *exported* binding initialized to an empty collection is a finding on its own, which approximates the cross-file case the walk cannot resolve. Six fixtures pin the new behavior. The rule's header now states what it does not see, and AGENTS.md states where the sweep stops and why core is not gated yet. Also tags `resetGlobalSingletonForTest` `@internal`. * fix(lint): attribute a static-field write to the field, not the class The static-field support added in the previous commit keyed `declared` on the class name, so a class carrying more than one mutable static reported one finding instead of one per field, and labelled the survivor with whichever mutation was seen first. On a two-static fixture it reported `static Registry.latch (`.set()`)`: the name of one field, the reason belonging to the other, pointing the reader at the wrong line. Key static fields `Class.field` and resolve a write to the same shape, via a new `memberPath()` that takes the first two segments of a member chain and tries that key before the bare root identifier. Two follow-ons fall out of having the path: - `this.field` inside a `static` member resolves to the class, which is the ordinary way to write the mutation. `staticClassOf()` returns nothing for an instance member, where `this` is an instance and the state is per-instance rather than per-copy, and nothing inside a nested `function`, which rebinds `this`. - `state.count++` is now a finding, like the `state.count += 1` that `assignment()` already reported. Fixtures pin all four, including the instance-field case that must stay clean. The four world packages still report zero, and the extracted `recordMutation()` keeps the file at its previous two Biome complexity warnings. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: make module duplication inert across every bundled package `@workflow/core` is bundled into the host server build the same way the worlds are, and always has been — the original repro measured three live copies in every arm, including the pre-#3493 external one. One instance is not reachable: layers cannot share a module, and core cannot be external because it *is* workflow code (`runtime/start.ts:253` and nine methods in `runtime/run.ts` are `'use step'`), so it must go through the SWC loader. The Next integration already encodes that rule by removing workflow-bearing packages from `serverExternalPackages`. So the duplication stays and the hazard is removed instead, everywhere the duplication can happen. `@workflow/core` (22 findings to 0): warn-once latches in `constants.ts`, `start.ts` and `telemetry.ts`; the source-map tracer cache; the VM script cache; the QuickJS compiled-assets and baseline caches; the dev-server port cache (its own comment already said "per process"); the text codecs; the zstd browser decoder; and the `useStep` closure brand, where a function marked by one copy was invisible to another. The one with teeth was `step-single-flight.ts`: a per-copy map is not single-flight. Two invocations reaching it through different layers would each believe they were alone in the process and both run the step body, silently degrading in-process dedup to the cross-process residual its own doc scopes out to the ownership lease. Also `@workflow/world` (a warn-once set, hand-rolled onto `globalThis` to keep that package dependency-free), `@workflow/ai` (the lazy OTel API), and `@workflow/nest` (bootstrap config in a module-level `let` and two static class fields — configure one copy, read another, and the controller is unconfigured for the life of the process). Five sites are deliberately per-copy and now say why: state keyed on objects that never cross copies (the barrier safety-net `WeakSet`, the QuickJS pending byte `WeakMap`), the synchronously-scoped guest-code sink, and the OTel diagnostic that reports what *this* copy sees. The sweep now covers all of it. Packages with a single module graph stay out (build-time code, the CLI, the o11y UI, the test runner) and AGENTS.md records which and why. Found while doing this: two static fields on one class collapsed into a single entry in the rule, so `WorkflowModule.options` was invisible behind `WorkflowModule.outDir`. Statics are now keyed `Class.field`. * fix(world): suppress noAssignInExpressions on the globalThis idiom The hand-rolled form trips Biome, as it does in `packages/core/src/private.ts`, which carries the same suppression. Restructuring it into a helper function instead would hide the state behind a call the module-scope rule cannot follow, so the binding would stop being recognized as off-module and the package would report a finding for correct code. * fix: sweep every bundled package, and mark utils side-effect free @shalabhc asked on review whether `@workflow/utils` needs this too. It does, and so do three others: `utils`, `errors`, `serde` and `workflow` all end up in the host application's server build and none were in the sweep. All four report zero today, which is exactly the state `world-testing` appeared to be in before the `.mts` walk was fixed and it turned out to have a real finding. Being clean and being *checked* are different properties, and only the second one survives the next contributor. `sideEffects: false` on `@workflow/utils`: verified that every module in the package only declares (no import-time work), so a bundler can now drop the unused parts of the barrel instead of keeping all ~64 KB of it because three packages import one 476-byte function. --------- Signed-off-by: Pranay Prakash <pranay.gp@gmail.com> Co-authored-by: Peter Wielander <mittgfu@gmail.com> Co-authored-by: Kenneth <kenneth@standardforensics.com> Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> Co-authored-by: Peter Wielander <peter.wielander@vercel.com> |
||
|
Peter Wielander
|
b2cac623d3 | [world] Make the sealed log opt-in instead of default-on (#3735) | ||
|
Nathan Rajlich
|
e1e64e3de3 |
docs: apply Vercel technical writing standards (#3704)
* docs: apply Vercel technical writing standards Audit the complete documentation corpus, package READMEs, skills, and source TSDoc/comments against the vercel-technical-writing skill and style-rules.md. Normalize sentence-case headings without changing published anchors, remove prose em dashes and filler wording, improve active voice and self-contained phrasing, standardize product/brand capitalization, American English, list punctuation, units, and code fence languages, and preserve exact runtime strings/table placeholders. All executable code is unchanged. Modified skills have their metadata versions bumped. * docs: extend writing audit to repository Markdown Apply the same technical-writing rules to design documents, compiler specifications, workbench guides, package changelogs, and the remaining tracked Markdown outside the deployed docs corpus. Preserve historical meaning, commands, output literals, table placeholders, and heading anchors. * docs: exclude generated package changelogs from audit |
||
|
Pranay Prakash
|
7b79ba37cc |
Add support for 'noop' event type - spec version 7 (#3634)
Co-authored-by: Peter Wielander <peter.wielander@vercel.com> |
||
|
Pranay Prakash
|
9454d51db0 |
feat(core): resolve run.returnValue via a World long poll instead of a 1s poll (#3570)
Co-authored-by: Peter Wielander <mittgfu@gmail.com> Co-authored-by: Peter Wielander <peter.wielander@vercel.com> |
||
|
Pranay Prakash
|
37e1d9e5a9 |
Batch: pre-claim inline steps in the same batch (#3568)
* Pre-claim inline steps inside the suspension batch (born-running pairs) Restacked onto main after #3025's squash-merge; folds in the review-round changes to the flush loop (per-write requestId attribution on createBatch, and the seeded/advancing slot-bump expectation, now shared with the pre-claim ceiling). Fold each lazy-inline step's deferred writes into the batched fan-out as an adjacent [step_created, step_started] pair: the created row carries the input, the started row is a bare ownership-stamped claim the server folds into one born-running create. The whole scheduling turn commits as ONE durable write, inline bodies start straight off that commit (in parallel with the VQS publishes for backgrounded steps), and executeStep gains a pre-claimed mode that runs or skips the body off the batch's per-event verdict - a pair 409 is the same skipped outcome as losing the lazy claim. The lone-inline case keeps the optimistic lazy path (a pair-only batch buys nothing over the single claim). Also threads per-event computeInstanceId through the World batch request, and folds the batch's committed slot ceiling into the inline slot snapshot so terminal writes stop being answered with reports echoing the batch's own events. * Parallel chunk commits, per-chunk continuation, batch span attributes Production trace of a 67-event fan-out showed the three batch chunks POSTing back-to-back (~230ms each) with no bodies or queue messages until all three settled (~670ms). Three changes: - Chunks now POST concurrently. Slot assignment is the server's, so parallel chunks race for slot ranges exactly like the pre-fold path's parallel single writes did; entity conditions, not commit order, carry correctness. The foreign-interleaving diagnostic is computed once over the whole fold (committed span vs seed) instead of per chunk. - Per-chunk continuation: each chunk's step-execution queue messages publish the moment ITS creates are durable (in-flush, via stepDispatch, same message shape and idempotency key as the caller's dispatch pass - the affected steps are pre-reported in queuedStepCorrelationIds so the caller skips them). Only the chunk carrying the inline pairs gates handleSuspension's return (opt-in via allowDeferredBatchWork); trailing chunk commits + all publishes ride result.deferredBatchWork, which the runtime joins next to the dispatch join before it can ack - the every-create-durable-before-ack contract is unchanged, the bodies just start off the pair chunk instead of the slowest chunk. - OTel: batch identity attributes (workflow.batch.size, per-type workflow.batch.shape) now live on the world.events.createBatch span (instrumentObject) instead of the http POST span, which keeps only wire-level facts (transport, bytes) and no longer sets workflow.event.type - that attribute names a single event write and tagging a batch with its first event's type misclassifies traffic. * Address review: settle deferred fold on failure, drop pair-batch retry Three fixes from review of the deferred/parallel-chunk fold. 1. A pair-chunk rejection escaped `handleSuspension` while the trailing chunks' commits and publishes were still in flight. `deferredBatchWork` never reaches the caller once the handler throws, so nothing joined that work — exactly the state `settlePhase` exists to prevent: a sibling create landing after the rejection commits an event from the abandoned replay's seeded sequence and races the caller's restart reload. The failure path now settles `trailing` before rethrowing. 2. Every pair-carrying chunk gates the return, not just the first. Pairs sort to the front and two rows per inline step fit inside one chunk, so this is one commit today, but `findIndex` silently degraded if either cap moved: a pair in an unawaited chunk yields no `inlineClaims` entry, the caller falls back to a lazy `step_started`, and that races this same fold's in-flight pair for the same step. constants.test.ts now pins the cap relationship. 3. A batch carrying a `step_started` is no longer retried in-process. The born-running pair does converge to a 409, but the pre-claim caller reads a pair 409 as "a concurrent writer owns this step" and skips the body — and on a retry that is indistinguishable from "my own first attempt committed the pair". Skipping there stranded a running step stamped with this invocation's own message id until the ownership lease expired (860s), where the single-POST path deliberately fails the delivery and recovers through owned-recovery in seconds. Same reasoning `EVENT_RETRY_ELIGIBILITY` already applies to `step_started`. Also asserts `lazyStepInput` / `preclaimedStart` mutual exclusivity in executeStep instead of only documenting it, and adds the changeset. Tests: +1 suspension-handler (pair-chunk failure settles the trailing chunk before escaping — fails without fix 1), +1 constants (cap relationship), +1 world-vercel (a born-running pair batch is single-attempt), and the existing batch-retry test retargeted at an entity-conditioned batch. Full @workflow/core unit suite 2178 green, @workflow/world-vercel 514 green, typecheck green across core / world / world-vercel. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * Guard inline bodies against unhandledRejection; review follow-ups The dispatch/deferred-batch joins now sit between the step promises' creation and the `Promise.all` that reads them, so a body rejecting in that window had no handler attached at the microtask checkpoint — an unhandledRejection, fatal under Node's default --unhandled-rejections=throw. A 412 fenced claim races exactly that window, and `deferredBatchWork` widens it by a trailing-chunk round trip. Attach a no-op catch at creation, the same way `dispatchesSettled` already does two lines up; the awaits below still decide the outcome. Review follow-ups: - `workflow.batch.shape` is sorted by event type. Map iteration is first-seen order, so a pre-claimed fold and a pure eager fold rendered the same composition as different strings, which is not groupable as a dimension. - A lost pre-claim reports StepSkipReason `running`, not `completed`. The pair's 409 says the step already exists and its claim winner is executing; the other skip site is a genuine terminal-state conflict, and tagging both `completed` left the attribute unable to separate the two. - `batchCommittedSlotCeiling`'s docstring now says the echo is only fully suppressed for a single-chunk fold: on a multi-chunk fan-out an inline terminal write issued before the trailing chunks land still names a position below them and still draws a report. - The defensive throw on a missing dehydrated input records where it lands — the pair is already durable, so it fails with the step claimed and its body unrun, recovered on redelivery via owned-recovery rather than failing cleanly. No regression test for the unhandledRejection: the existing inlineClaimRejectionScenario runs both steps inline, so `dispatches` is empty and the join resolves in a microtask — the window never opens and a test there passes with or without the fix. Reproducing it needs a scenario with a backgrounded step and a slow queue publish alongside the fenced claim. Full @workflow/core unit suite 2178 green, @workflow/world-vercel 514 green, typecheck and biome clean. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * Pin per-event computeInstanceId on the batch wire Batch encoding is a separate path from the single-event POST, so the frame meta had no coverage: the only assertion was at the World-call boundary. Adds a wire-level test that a pre-claimed pair's step_started half carries computeInstanceId in its frame meta and the step_created half does not. Verified it fails when the threading in createWorkflowRunEventBatch is removed. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * Guard the pre-claim path as inert on Worlds without createBatch world-local and world-postgres do not implement createBatch, so the fold never engages there — but the runtime passes ownerMessageId and allowDeferredBatchWork unconditionally. The existing "keeps the single path when the World lacks createBatch" test passed neither, so it never covered the pre-claim path at all. Assert the inertness with the params the runtime actually sends: no claims, no deferred work, no slot ceiling, the lazy-inline step still carrying its input, and no step_started reaching the world. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Peter Wielander <peter.wielander@vercel.com> Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Peter Wielander
|
04e060a0ec | [world] Add WORKFLOW_NODE_HTTP to run the HTTP Worlds on node:http (#3461) | ||
|
Pranay Prakash
|
b0adb50bce |
feat(world,world-vercel): createBatch — ordered batch event write with per-event results (#3025)
## createBatch: the client half of the v4 batch event write (per-event
results, no fence)
> **Note:** this PR was rebuilt from scratch. The previous revision
implemented the retired "v2 suspension fence" design
(`expectedRunVersion` / `batchId` / `logicalCreatedAt`, a
grammar-validated collect mode, world-postgres migration 0016,
world-local claim machinery). The server redesigned its endpoint in
place (vercel/workflow-server#646, merged and deployed) and this branch
now targets that contract on top of current `main` (specVersion 6 slot
identity). The old head is tagged `batch-client-v2-fence-design`; prior
review threads reference deleted code.
### The server contract this targets
`POST /api/v4/runs/:runId/events/batch` (workflow-server#646): an
**ordered** list of v4 frames — byte-identical to single-event POST
frames, **no batch-level meta** — committed in one DynamoDB transaction
per attempt, answered with HTTP 200 + `{ results }`: one entry per
frame, in request order. Each event reports what its own single POST
would have returned: `200` + the materialized entity, or the single-path
status/code (e.g. `409`/`conflict` for an event an earlier delivery
already applied). A transport retry of a committed batch converges to
all-409s with nothing written twice — idempotency comes from per-entity
conditions, not batch bookkeeping. Slot-identity runs only (specVersion
≥ 6 — what `world-vercel` stamps on every new run since #3389).
### What this revision ships
1. **`@workflow/world` — the spec addition.** `Storage['events']` gains
one optional method; **method presence is the capability declaration**
(no capability flag, no stub required):
```ts
createBatch?(
runId: string,
events: BatchEventRequest[],
params?: CreateEventBatchParams
): Promise<EventBatchResult>;
interface BatchEventRequest {
event: CreateEventRequest; // same discriminated union as the single create
occurredAt?: Date; // under slot identity: the source of the durable createdAt
}
type BatchEventItemResult = // one per submitted event, in request order
| { status: 200; event: Event; run?: WorkflowRun; step?: Step; wait?: Wait }
| { status: number; error: string; message: string };
interface EventBatchResult { results: BatchEventItemResult[] }
```
Contract: **ordered** (events land in the log in request order),
**per-event outcomes** (each event reports what its own single `create`
would have returned — success discriminated by `error === undefined`),
**idempotent on retry** (per-entity conditions make a retried committed
batch converge to per-event 409s). Worlds that don't implement it keep
the single-event path. `world-local` and `world-postgres` deliberately
do NOT implement it — batching a local/in-process write buys nothing
(this deletes the old revision's riskiest surface: the hand-written
postgres migration and the world-local claim machinery).
2. **`@workflow/world-vercel`** — the wire adapter: per-event frames
concatenated in order (reusing the single-frame encoder; each frame
carries its own `occurredAt`, which under slot identity is the source of
the durable `createdAt` — this natively closes the replay-clock question
the old `logicalCreatedAt` field existed for), CBOR `{ results }`
decoded against the **same per-type zod schemas as the single POST**,
loud `SCHEMA_VALIDATION` on any malformed response (wrong length,
invalid item), and the standard typed error mapping for request-level
failures.
3. **Retry policy** — a `batchIdempotent` override in the event-retry
eligibility machinery: the whole batch POST retries transient transport
failures/5xx (and waits out 429 `Retry-After` per #3504) regardless of
the contained event types, because per-event entity conditions make the
retry converge; the per-type non-retryability matrix guards *single*
posts (where e.g. a retried bare `step_started` would increment
`attempt`) and doesn't apply inside a batch.
Tests: 7 wire tests — frame encoding/ordering + **no fence fields on the
wire**, per-event result mapping (successes typed, failures passed
through), malformed-response failures (length mismatch, invalid item
body with index), typed request-level 400s, in-process 5xx retry,
empty-batch guard — plus 9 suspension-handler tests for the runtime
fold: ordering (steps then waits), per-event 409 tolerance, non-409
failure propagation, every gate exclusion (flag off / no `createBatch` /
pre-slot run / hook writes), 32-cap chunking, and lazy-inline exclusion.
Full `world-vercel` suite: 508 passed; full `@workflow/core` suite: 2126
passed.
### The runtime integration: batched suspension fan-out (ON by default)
The suspension handler folds a **clean fan-out** — the suspension's
eager `step_created` + `wait_created` writes — into `createBatch` calls
of at most **32 events**, and uses the batch endpoint **exactly when two
or more batchable eager events exist**: a lone eager event takes the
ordinary single write (same round trip, and it keeps the slot-snapshot +
bump-and-report the single path provides) (mirroring the server's
transaction budgets: 2 items/event against the 100-item cap, 768 KB
inline-byte budget; larger fan-outs commit in successive batches). The
gate requires: World implements `createBatch` ∧ run on slot identity
(specVersion ≥ 6) ∧ no attribute writes ∧ no hook writes ∧ no resilient
step dispatch. **Everything outside the gate keeps the single-event path
byte-for-byte**, and lazy-inline steps keep deferring their
`step_created` to the lazy start exactly as before.
Per-event semantics mirror the single path: a `409` is the same
already-exists tolerance as `EntityConflictError` (the conflicted step
is not marked owned); any other per-event failure fails the suspension
write the way a single-path rejection would. Slot bumps (the batch
endpoint has no bump-and-report) are tolerated and logged — the same
accepted exposure as a dropped truncated skipped-slot report on the
single path.
**On by default**, with the `WORKFLOW_TURBO`-shaped kill switch as the
operator escape hatch: **`WORKFLOW_BATCH_TRANSITIONS=0`** (or `false`)
disables batching and restores the exact prior one-write-per-event path.
Documented in the worlds configuration reference and the changelog
entry. Burn-in watch: the `event_batch`-tagged slot-conflict metrics and
DynamoDB throttle monitors on the server side.
### Docs
- New v5 changelog entry **`changelog/batched-event-writes`**
documenting the World spec addition (full `createBatch` signature +
contract — the signature block is compile-checked against
`@workflow/world` by the docs code-sample checker), the runtime fold,
and the follow-up.
- `configuration/worlds` gains the **`WORKFLOW_BATCH_TRANSITIONS`**
reference entry: default on, `=0`/`false` as the documented escape
hatch.
### Staged follow-up: the deferred sequential transition (the STSO win)
Hold `step_completed(N)` across the replay turn and commit
`[step_completed(N), step_created(N+1), step_started(N+1)]` as one batch
at the next lazy start (the server folds the pair born-running). This
needs the synthetic-completion replay machinery rebuilt against today's
runtime (parallel inline batches, turbo's run-ready barrier, optimistic
starts, slot bookkeeping) — it stays a separate PR so the SDK's most
sensitive replay path gets its own focused review. Its acceptance
criteria are already agreed: the runtime eligibility matrix as unit
tests, and an e2e that asserts ≥1 POST to `/events/batch` and **zero**
single-event POSTs for the batched transitions.
### Compatibility
- Old servers: no `/batch` route → 404/405 → callers fall back to
single-event posts (the runtime PRs will latch this per run).
- Pre-slot runs: request-level 400 (`batch-requires-slot-identity`) →
same fallback.
- No `WORKFLOW_SERVER_URL_OVERRIDE` pin this time — the server endpoint
is merged and deployed to production.
Refs: vercel/workflow-server#646 (endpoint), vercel/workflow-server#780
(unbatchable-types design space), #3389 (slot identity), #3504 (429
retry).
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
|
||
|
Peter Wielander
|
1321570464 | [docs] Document duplicate-event handling, and describe webhook token generation accurately (#3497) | ||
|
Peter Wielander
|
de2a86c61c | [world] Make spec version 6 the current version (#3542) | ||
|
Peter Wielander
|
dc85865718 | [core] Drop pre-slot event ID support and preconditionGuard capability (#3519) | ||
|
Peter Wielander
|
834d1f945f | [web-shared] Mark ignored duplicate events in the observability UI (#3467) | ||
|
Karthik Kalyan
|
f1ef0cbf03 |
Deprecate world.runs.list for observability (#3404)
* Deprecate storage run listing for observability * Document analytics run listing guidance |
||
|
Peter Wielander
|
b589460ce8 | [core] Report the replay position on every event write (#3479) | ||
|
Karthik Kalyan
|
a0ccfe0f50 |
feat(core): measure hook-triggered time to resume (#3437)
Report end-to-end TTR for a hook resumption — entry into the public resume
API through to the first line of the next durable step — on that step's
`step.execute` span, decomposed into non-overlapping phases that sum
exactly to the total:
workflow.resume.total_ms
workflow.resume.phase.{producer_prep,queue_delivery,resume_setup,
replay,step_dispatch,step_claim,step_prepare}_ms
dimensioned by trigger, dispatch strategy, setup source, and whether the
step ran inline or was dispatched to another invocation.
T0 is stamped by whichever public entry point the caller used, so
`resumeWebhook` — which does its own by-token lookup and key resolution
before reaching the shared implementation — measures the same window as
`resumeHook` rather than a systematically shorter one. T7 is taken inside
`contextStorage.run`, immediately before `stepFn.apply()`, so the
`step_prepare` phase covers the step-context setup it is defined to cover.
`resumeHook()` puts the producer boundaries on an optional
`hookResumeTiming` field on the queue message (both dispatch paths); the
consuming invocation adds its own and hands them to the execution that
will actually ATTEMPT the next durable step. That decision is made against
the dispatch loop's own classification, so an owned-recovery step keeps
the measurement here instead of it riding off on a queued sibling, and a
step converted into a delayed backstop wake — which this delivery does not
attempt — never takes it. Within an inline batch the tracking is shared
and a one-shot latch picks the single step that reaches user code, so the
sample survives the batch's first step losing its create-claim. A
deployment-affinity re-route forwards the timing untouched, keeping the
wasted hop inside `queue_delivery`.
The field is optional in every direction (new producer/old consumer, new
consumer/old message, no workflow-server change) and parses with
`.catch(undefined)` so a malformed value can never fail a delivery. A
sample is emitted only when every required boundary is present, finite,
and monotonic — a skewed or incomplete set is dropped rather than
reported as a negative phase.
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
||
|
Peter Wielander
|
efbc408c4e | [core] Ignore duplicate events per event class instead of failing the run (#3381) | ||
|
Alex Langenfeld
|
600b096d2f |
feat(world): batch analytics event lookups (#3390)
## Summary & Motivation Adds `events.getMany` to the analytics API, letting dashboard consumers enrich canonical event pages with ClickHouse provenance in one bounded, deduplicated batch request instead of N point lookups. Coordinated with a workflow-server batch endpoint. ## Test Plan Added unit tests for the client and schema; also ran Biome checks, `git diff --check`, and world-vercel typecheck. Signed-off-by: Alex Langenfeld <alex.langenfeld@vercel.com> |
||
|
Nathan Rajlich
|
7683130461 |
Resilient step dispatch: parallelize step_created writes with queue publishes (#3365)
* feat(world,world-vercel,core): resilient step dispatch (parallel step_created + queue publish) Newly created steps are handed to the queue in parallel with their step_created event write, with the serialized input carried on the message (stepInput) so the queue consumer can idempotently re-ensure the event when the direct write failed transiently — mirroring resilient start (runInput) and resilient hook resume (hookInput). - @workflow/world: stepInput on WorkflowInvokePayload, CreateEventParams.viaStepDispatch, WorldCapabilities.resilientStepDispatch - core (node:vm): suspension handler publishes eligible steps alongside their create; the dispatch pass skips them (queuedStepCorrelationIds) - core (quickjs): dispatchPendingOps does the same for overflow steps; the ineligible fallback is now published in parallel too (removes the serial per-step enqueue loop) - consumer: on a redelivery, a stepInput-carrying message re-ensures step_created (marked viaStepDispatch) before executing - under an enforced precondition guard the parallel path requires backend cooperation (capabilities.resilientStepDispatch, declared by world-vercel): a 412-rejected step's in-flight dispatch is revoked server-side and its re-ensure refused - step dispatch/retry idempotency keys are step-identity-scoped (cid + hashed step name) so a revoked message for a reassigned correlation id cannot absorb the corrected schedule's dispatch - kill switch: WORKFLOW_RESILIENT_STEP_DISPATCH=0 * Validate stepInput.input as Uint8Array at the schema boundary Review feedback: producers only attach stepInput when the dehydrated input is binary and the queue transport preserves bytes (CBOR), so a non-binary value means the payload was mangled in transit. Enforcing Uint8Array in StepDispatchInputSchema fails the message parse instead of silently writing non-binary data into a step_created, and types the consumer's re-ensure so the unchecked 'as SerializedData' cast goes away. * Keep sequential dispatch under an enforced precondition guard (drop the resilientStepDispatch capability lift) Review feedback (two P1s): backend-side revocation bookkeeping cannot carry the guard's correctness property across the queue side-channel — - nothing orders a slow guarded create's eventual 412 (the moment the backend learns the dispatch is poisoned and records the revocation marker) before the consumer's redelivery re-ensure, so attempt > 1 is a probabilistic mitigation, not a happens-before; and - a best-effort marker that fails open (Redis loss) cannot back a capability the SDK treats as a correctness attestation. Only sequencing the publish after the create gives the message a happens-after edge over the create's guard verdict, so the guard gate is now unconditional: worlds that enforce the precondition guard keep the sequential create-then-publish dispatch. The parallel resilient path remains for unguarded writes (the quickjs engine everywhere, and worlds without the guard). Removes WorldCapabilities.resilientStepDispatch and world-vercel's declaration; the viaStepDispatch flag is kept and re-documented as advisory (server-side defense-in-depth only). This also dissolves the reviewed dedupe hazard on the step-identity- scoped dispatch keys: with no 410-ack path in any real SDK flow, a message for a never-created step keeps redelivering until an entity exists, execution always hydrates input from the committed entity (never the message), and a name-mismatched stale start is skipped by the server's stepName fence. * Correct the MAX_RESILIENT_STEP_INPUT_BYTES rationale: VQS has no hard message-size cap 256 KB is the queue's inline-vs-S3 threshold, not a rejection limit (payloads above it spill to S3-backed storage transparently). The 128 KiB bound is a cost/latency choice — keep step messages on the inline path rather than paying an S3 double-hop for bytes that already live in the event log. * Recover a missing step in-band when a stepInput-carrying delivery beats its create Durabench parallel sweeps (guard-off, node engine) caught ~4-8% of fan-out runs stalling one branch for ~306s on the resilient dispatch path. Root cause: the consumer's step_created re-ensure was gated on metadata.attempt > 1, but world-vercel's failure-retry path re-enqueues a FRESH message whose attempt resets to 1 — so when a delivery beat the producer's parallel step_created write, every fast retry hit the same 'step not found' rejection with attempt 1, and the step only recovered when the ORIGINAL message's ~300s visibility-timeout redelivery finally arrived with attempt 2. The recovery is now in-band and attempt-independent: when a stepInput-carrying execution rejects with the step-missing signature (WorkflowWorldError, 404 or the local worlds' message shape), the consumer materializes the step_created from the message payload and retries the execution once within the same delivery. The eager attempt>1 ensure is kept as a round-trip saver on genuine redeliveries. Sweep effect expected: the 305-306s TTLS outliers disappear while the resilient path keeps its p50 win (1054ms vs 1425ms at 64 branches). |
||
|
Peter Wielander
|
6786db9953 | World-side incrementing event ID (specVersion 6) (#3389) | ||
|
Nathan Colosimo
|
22349e95fd |
perf(core): load replay suffix in one request (#3205)
* perf(core): stream replay suffix in one request * perf(core): load replay suffix in one request Signed-off-by: Nathan Colosimo <110621881+NathanColosimo@users.noreply.github.com> * test(world-vercel): use streamed run start fixtures * refactor(events): simplify return-all plumbing Signed-off-by: Nathan Colosimo <110621881+NathanColosimo@users.noreply.github.com> * Return complete local run preloads * Document workflow event limit * fix: make return-all event loading resilient * Simplify full event listing * refactor(world-vercel): omit event limit for full loads * fix(world-vercel): explicitly request complete event logs --------- Signed-off-by: Nathan Colosimo <110621881+NathanColosimo@users.noreply.github.com> |
||
|
Nathan Colosimo
|
65139acfd7 |
perf(core): continue partial run_started preloads from cursor (#3124)
* perf(core): continue partial run preloads * refactor(core): simplify preload continuation * fix(core): preserve preload fallbacks * chore: rerun CI * fix(world): infer event create results * fix(core): preserve run state during setup * fix(world): enforce typed event results * refactor(world): rely on event result contract * refactor(core): unify replay event log state * refactor(core): make replay log states exact * fix(core): harden run start preload recovery * test(world-local): allow slow preload coverage * fix(core): preserve event result inference through recovery * refactor(core): simplify preload state transition Signed-off-by: Nathan Colosimo <110621881+NathanColosimo@users.noreply.github.com> * refactor(runtime): reuse event pages without duplicate reads * refactor(world-vercel): preserve opaque event payloads * Validate v4 event create responses * Validate v4 event frame metadata * Remove invalid v4 response identity check * Return validated v4 event bodies directly * Reuse event result entity types * Simplify event creation result types * Use concrete run creation result * Preserve generic event storage implementation * Validate v4 event responses without casts * Parse v4 event frames once * Reuse the default v4 event body schema * Simplify event preload state * Narrow event page result states * Preserve literal event result flags * Accept hook conflict event responses * Remove redundant optional event page schemas * Simplify preloaded event log access * Flatten replay event log state * Simplify replay event log state * Use one replay event log * fix(next): preserve edits made during full HMR rebuilds * chore(core): log dormant hook replays * fix(next): commit HMR snapshots after rebuilds * fix(next): ignore duplicate HMR file events * test(next): expect deduplicated HMR removal event * fix(next): distinguish duplicate HMR notifications * fix(next): ignore HMR notifications without source changes * chore: move Next HMR fix to separate PR * fix(core): complete partial preloads before QuickJS replay --------- Signed-off-by: Nathan Colosimo <110621881+NathanColosimo@users.noreply.github.com> |
||
|
Nathan Colosimo
|
74dbf81d32 |
fix(core): retry replay timeouts without exiting (#3385)
* fix(core): retry replay timeouts without exiting * refactor(world-postgres): leave existing retry limits unchanged * test(world-postgres): remove mocked migration assertion * chore: consolidate replay retry changesets |
||
|
Peter Wielander
|
a8db185c3b | [core] Fold events.create deltas into the replay log (#3382) | ||
|
Karthik Kalyan
|
439a495a71 |
fix(core): pre-check deployment affinity before the lazy resume write (#3374)
The lazy hook fast path (#3345) hoisted the consumer's hook_received write above the deployment-affinity guard (#2960), so a misrouted lazy resume wrote its event before the guard could re-route the delivery. Stamp the run's pinned deployment on the resume message (hookInput.deploymentId, from the producer's resume context) and, on the consumer, compare it against the ambient deployment id immediately before the fast path: a match continues with no run fetch, a mismatch fetches the authoritative run and hands it to the existing guard — which keeps sole ownership of re-route/fail policy and remains the authoritative protection before replay and step execution. The re-routed message preserves the complete hookInput (it may hold the only copy of the resume payload). Older messages without the field, and worlds without deployment affinity, are unchanged: they skip the pre-check and rely on the authoritative guard, the pre-guard write staying convergent per (runId, resumeId). Fixes the misrouted-lazy-resume unit test broken by the #2960/#3345 ordering: a modern misrouted resume now re-routes with zero event writes, asserted for both hook_received and run_started. Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |
||
|
Karthik Kalyan
|
9c1b3c8638 |
perf(core): initialize lazy hook replay from hook_received stream (#3345)
* perf(core): initialize lazy hook replay from hook_received stream On a lazy hook queue delivery, the consumer's idempotent hook_received re-ensure is hoisted above run_started and doubles as the invocation's setup request: it asks the World to return the current replay log with the write (new advisory CreateEventParams.preloadEvents), so one HTTP request yields the canonical event, the reconstructed run, and the complete replay log — skipping both the run_started POST and the initial events.list. - world: optional `preloadEvents?: true` on CreateEventParams, the hook_received dual of skipPreload; Worlds may ignore it - world-vercel: createHookReceivedPreloadEventV4 sends the frame Accept on eligible hook_received posts and decodes either response mode — frames via the response decoder extracted from the LIST consumer (GET behavior unchanged), CBOR via the shared materialized-response mapping. The run is reconstructed from run_created/run_started (plus attr_set folds), the canonical event found by x-wf-event-id, and resumeId now survives frame decoding so the runtime can match it - core: new fast path before the generic run-state setup, guarded on hookInput.resumeId + payloadDigest; a validated COMPLETE preload (hasMore false — this path has no cursor-continuation machinery) initializes workflowRun/preloadedEvents/maxEventsLimit directly, anything else falls back to the run_started setup without re-posting the hook; error classification matches the existing re-ensure (terminal → consume, transient → redeliver); setup source reported via workflow.resume_setup_source (never workflow.hook.resilient_resume_materialized, which stays a recovery-only signal) - producer resumeHook() is unchanged and never sets preloadEvents Based directly on main (no dependency on #3124/#3191); pairs with workflow-server's streamed hook_received replay-log response, which deploys first — the SDK negotiates per request and falls back safely against older servers. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * address review: lazy fallback, retryable resume, terminal telemetry - world-vercel: the preload request keeps hook_received's lazy remoteRefBehavior — a supporting server owns frame-body resolution, while an older server now answers the CBOR fallback without resolving an S3-backed payload the runtime would discard - world-vercel: the atomic lazy-resume shape (resumeId + digest) opts into withEventPostRetry via idempotentHookResume — the (runId, resumeId) claim makes the POST idempotent-on-retry; legacy/partial hook_received shapes stay single-attempt, definitive 4xx stays non-retryable (unit + adapter + trace-propagation coverage) - core: a terminal event found in the preload records workflow.resume_setup_source=hook_received_stream and the run's actual terminal status on the span before consuming the delivery - core: document resilient_resume_materialized as the legacy/non-atomic re-ensure signal (claim ownership is not observable client-side, so the hoisted path deliberately never emits it) and resume_setup_source as a latency signal, not proof of event creation; note the Option A skip is now unreachable for atomic resumes - world: spell out the full preload usability contract on preloadEvents (complete hasMore-false log, non-null cursor, run/startedAt/maxEvents, lifecycle events, matching resumeId, list ordering, read-after-write consistency); bump @workflow/world to minor - new QuickJS sourcing tests (VM mocked): an attested complete preload is used verbatim with no events.list, a non-attested hook-containing preload is refetched, and an attested empty preload is not trusted Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |
||
|
Elliot Dauber
|
72efc90f28 |
Use runtime deadline for inline execution limit (#3360)
* Use runtime deadline for inline execution limit * up * lazy import * Update packages/world/src/interfaces.ts Co-authored-by: Peter Wielander <mittgfu@gmail.com> Signed-off-by: Elliot Dauber <67391073+elliotdauber@users.noreply.github.com> --------- Signed-off-by: Elliot Dauber <67391073+elliotdauber@users.noreply.github.com> Co-authored-by: Peter Wielander <mittgfu@gmail.com> |
||
|
Alex Langenfeld
|
79e4c04409 |
fix(core): re-route runs delivered to the wrong deployment (#2960)
## Summary & Motivation A queue callback that reaches a deployment other than the one its run is pinned to derives the per-run encryption key from the wrong master key, so the delivery fails before user code runs and the run dies as a blank "exceeded max retries". The delivery is re-enqueued explicitly addressed to the run's own deployment — strictly better-targeted than the send that misrouted — and the run is failed with the new `DEPLOYMENT_MISMATCH` error code only once `WORKFLOW_DEPLOYMENT_MISMATCH_MAX_RETRIES` (default 3) is spent. Gated on the new World capability `deploymentAffinity`, so worlds with synthetic or version-tagged deployment ids are unaffected. ## Test Plan Unit tests added for the guard and both runtime paths; local vitest and typechecks pass. |
||
|
Karthik Kalyan
|
8d479283ca |
feat(world,world-vercel,core): bulk run cancellation primitive (#3347)
* feat(world,world-vercel,core): bulk run cancellation primitive Add a bulk cancellation contract to @workflow/world (schemas, types, and an optional Storage['runs'].cancelMany method), implement it in @workflow/world-vercel via a single POST /v4/runs/cancel request, and add a cancelRuns runtime helper to @workflow/core that uses the world fast path when available and otherwise falls back to bounded-concurrency (max 20) single-run cancellation. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * Update packages/world/src/interfaces.ts Co-authored-by: Peter Wielander <mittgfu@gmail.com> Signed-off-by: Karthik Kalyan <105607645+karthikscale3@users.noreply.github.com> --------- Signed-off-by: Karthik Kalyan <105607645+karthikscale3@users.noreply.github.com> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> Co-authored-by: Peter Wielander <mittgfu@gmail.com> |
||
|
Peter Wielander
|
de1905f15c | feat(world): require a runId on listByCorrelationId (#3280) | ||
|
Nathan Colosimo
|
e6f1b6f548 |
feat(world-local): support Hook minimum retention (#2866)
* feat(core): add hook token retention contract * refactor(core): constrain hook retention options * fix(core): preserve boolean hook visibility options * revert(core): preserve HookOptions interface * docs(core): clarify retained conflict ownership * docs(core): retain newest-wins conflict pattern * docs(core): simplify hook retention guidance * docs(core): explain retained token cleanup * docs(core): simplify idempotency guidance * docs(core): clarify retained token results * refactor(core): rename hook token expiration option * chore(core): name hook expiration changeset * docs(core): simplify Hook expiration language * docs(core): clarify Hook expiration deadline * docs(core): remove Hook deadline caveat * refactor(core): align Hook expiration field names * docs(core): narrow Hook expiration documentation * docs(core): clarify hook expiration availability * Update packages/core/src/workflow/hook.ts Co-authored-by: Peter Wielander <mittgfu@gmail.com> Signed-off-by: Nathan Colosimo <110621881+NathanColosimo@users.noreply.github.com> * docs(core): clarify Hook token expiration behavior * docs(core): explain active Hook expiration behavior * feat(world): advertise hook ttl capability * fix(core): validate hook ttl capability after main merge * refactor(core): rename hook expiry to minimum retention * docs: keep hook retention guidance on v5 * docs: define retained run availability * fix(core): validate Hook retention at creation * feat(core): define retained Hook lookup semantics * refactor(core): simplify hook retention checks * feat(world-local): support Hook token expiration * fix(world-local): make hook recovery atomic * refactor(world-local): align Hook minimum retention * fix(world-local): preserve Hook creation order * fix(world-local): expose retained Hooks consistently * refactor(world-local): simplify retained hook storage * fix(world-local): allow stale lock recovery * refactor(world-local): simplify hook retention storage Signed-off-by: Nathan Colosimo <110621881+NathanColosimo@users.noreply.github.com> * fix(world-local): serialize expired hook token handoff Signed-off-by: Nathan Colosimo <110621881+NathanColosimo@users.noreply.github.com> * fix(world-local): preserve hook creation order Signed-off-by: Nathan Colosimo <110621881+NathanColosimo@users.noreply.github.com> * refactor(world-local): clarify hook availability cleanup * docs: note Local World Hook retention support * fix(world-local): harden hook retention persistence * fix(web-shared): render hook retention deadline * fix(world-postgres): exclude unsupported hook retention * feat(world-local): enforce Hook retention limit * docs(world-local): clarify retention limit error * docs(world): clarify Hook retention deadline * docs(hooks): link retention configuration --------- Signed-off-by: Nathan Colosimo <110621881+NathanColosimo@users.noreply.github.com> Co-authored-by: Peter Wielander <mittgfu@gmail.com> |
||
|
Karthik Kalyan
|
31f92df10d |
Lazy hook resumption: parallel event write + queue publish (#3230)
* feat(core): lazy hook resumption via parallel event write + queue publish (rebased onto #1834 + #3145) Rebase of #3230 onto current main ( |
||
|
Pranay Prakash
|
ee944d2476 |
feat(core): stamp creator environment into runInput and reject cross-environment queue deliveries client-side (#3244)
* feat(core): stamp creator environment into runInput and reject cross-environment queue deliveries client-side `start()` makes two writes that have to land in the same tenant: the `run_created` event, attributed to whatever environment the caller authenticates as, and the queue message, pinned to a deployment. A misconfigured caller can split them — writing the run to one environment while addressing the message to a deployment in another. The consumer finds no run under its own tenant, the backend's resilient start (`run_started` creates the run when `run_created` was never seen) mints a second copy of the same run id in the consumer's environment, and both copies are real: the creator's sits pending forever while the other executes. The deployment id is not the discriminator — it matched end to end in the incident that motivated this. The environment is. So carry it: add an optional `World.getEnvironment()`, implement it in world-vercel from the same resolution that produces the `x-vercel-environment` header, and stamp it into the queue message's `runInput`. The consuming deployment already knows its own environment, so it can refuse the delivery itself with no server coordination — and refuse before `run_started`, the write that would create the fork. The refusal acks the message instead of throwing: the mismatch is baked into the message, so every redelivery would reach the same verdict and throwing would hot-loop until MAX_QUEUE_DELIVERIES. Both sides must be known for the check to run, so worlds with a single tenant (local, Postgres) and runs started by an older SDK behave exactly as before. A companion diagnostic logs a deployment-id mismatch without refusing, since deployment ids differ for benign reasons too. Signed-off-by: Pranay Prakash <pranay.gp@gmail.com> * fix(world-vercel): resolve the runtime environment from VERCEL_TARGET_ENV For a deployment in a Vercel custom environment, the OIDC token's environment claim is the custom environment's slug (the platform mints `customEnvironment?.slug ?? envTarget`) while VERCEL_ENV reports 'preview' — so keying the cross-environment guard on VERCEL_ENV could false-refuse a legitimate delivery, e.g. a CLI client attributed to 'staging' starting a run on the staging deployment. VERCEL_TARGET_ENV is populated from exactly the same slug-or-target pair as the claim, so prefer it, keeping VERCEL_ENV as the fallback for contexts that don't inject it. Also sorts runtime.ts imports per the Biome rule that landed on main in #3241. Signed-off-by: Pranay Prakash <pranay.gp@gmail.com> --------- Signed-off-by: Pranay Prakash <pranay.gp@gmail.com> |
||
|
Peter Wielander
|
1471f252fa | [core] Gate event creation on the loaded event count and restart replays in-process (#3145) |