mirror of
https://github.com/vercel/workflow.git
synced 2026-09-14 19:59:43 +08:00
peter/windows-preload-timeout
1633 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Peter Wielander
|
a0bb3121fc |
[core] Poll for the deferred check instead of sleeping past it
`Unit Tests (windows-latest)` has a second failure family alongside the world-local preload timeout: `events-consumer.test.ts`, rotating between lines 1005 and 1123 across main runs. Eleven call sites slept `MIN_DEFERRED_CHECK_DELAY_MS * 4` and then asserted an outcome the deferred check produces. That check is not on a fixed schedule. It waits for delivery to go idle, which is its own poll loop, and only then arms a `getDeferredCheckDelayMs()` timer, so the sleep is a lower bound on when the timer becomes eligible rather than a guarantee it has run. The helper's own comment already said assertions that a check DID fire should poll instead; every one of the eleven ignored it. Replace the sleep with `afterDeferredCheck`, which polls the assertion block. The positive assertions gate the poll and the negatives alongside them are evaluated once the check is known to have fired, which is what they mean. Reproduced without a Windows runner by raising the stubbed delay 20x, which makes the fixed sleep too short by construction: the old file fails exactly at 1005 and 1123, the two lines CI reports, and the new file passes 45/45 at the same delay. Also drop the changeset to an empty one. Neither change affects users. |
||
|
Peter Wielander
|
104c2f9e17 |
[world-local] Give the complete-preload test a budget the Windows runner can meet
`Unit Tests (windows-latest)` has been red on main since 2026-08-11 with
Error: Test timed out in 120000ms.
❯ src/storage.test.ts:1284:7
That is `returns the complete preload when run_started is retried`. It writes
a thousand events sequentially, one file write each, then asserts the retry
returns the whole log.
It is a marginal budget rather than a step regression. Same 245 tests either
side of the boundary, whole-file duration on the Windows runner went 143491ms
(last green) to 154816ms (first red), 8% apart. An added fs op per write would
show a far larger jump. On macOS the writes run at ~1ms each and the test
finishes in about a second, so it only ever bites on that runner.
Raise the budget to 300s and say why in the test, including that batching the
writes with `Promise.all` measures slower rather than faster: the writers then
contend for the same event slot and re-probe.
The count is load-bearing and was a bare 999/1001. It sits one past the event
cache so the preload cannot be served from cached entries alone and has to
read at least one back from disk. Below the ceiling the test still passes and
silently stops covering that fallback, so export the ceiling and size the test
from it.
Unit Tests (windows-latest) is an input to E2E Required Check, so this has been
failing the required aggregate on every open PR.
|
||
|
Mitul Shah
|
14b52ac04b |
[1/4] Replace shared observability component styles with Tailwind (#3295)
* Replace hard-coded UI styles with Tailwind Signed-off-by: Cursor Agent <cursoragent@cursor.com> * Update Tailwind migration verification Signed-off-by: Cursor Agent <cursoragent@cursor.com> * Remove documentation changes from style migration Signed-off-by: Cursor Agent <cursoragent@cursor.com> * Split trace, graph, and workbench style changes Signed-off-by: Cursor Agent <cursoragent@cursor.com> --------- Signed-off-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: mitul-s <19615826+mitul-s@users.noreply.github.com> |
||
|
Karthik Kalyan
|
a0ccfe0f50 |
feat(core): measure hook-triggered time to resume (#3437)
Report end-to-end TTR for a hook resumption — entry into the public resume
API through to the first line of the next durable step — on that step's
`step.execute` span, decomposed into non-overlapping phases that sum
exactly to the total:
workflow.resume.total_ms
workflow.resume.phase.{producer_prep,queue_delivery,resume_setup,
replay,step_dispatch,step_claim,step_prepare}_ms
dimensioned by trigger, dispatch strategy, setup source, and whether the
step ran inline or was dispatched to another invocation.
T0 is stamped by whichever public entry point the caller used, so
`resumeWebhook` — which does its own by-token lookup and key resolution
before reaching the shared implementation — measures the same window as
`resumeHook` rather than a systematically shorter one. T7 is taken inside
`contextStorage.run`, immediately before `stepFn.apply()`, so the
`step_prepare` phase covers the step-context setup it is defined to cover.
`resumeHook()` puts the producer boundaries on an optional
`hookResumeTiming` field on the queue message (both dispatch paths); the
consuming invocation adds its own and hands them to the execution that
will actually ATTEMPT the next durable step. That decision is made against
the dispatch loop's own classification, so an owned-recovery step keeps
the measurement here instead of it riding off on a queued sibling, and a
step converted into a delayed backstop wake — which this delivery does not
attempt — never takes it. Within an inline batch the tracking is shared
and a one-shot latch picks the single step that reaches user code, so the
sample survives the batch's first step losing its create-claim. A
deployment-affinity re-route forwards the timing untouched, keeping the
wasted hop inside `queue_delivery`.
The field is optional in every direction (new producer/old consumer, new
consumer/old message, no workflow-server change) and parses with
`.catch(undefined)` so a malformed value can never fail a delivery. A
sample is emitted only when every required boundary is present, finite,
and monotonic — a skewed or incomplete set is dropped rather than
reported as a negative phase.
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
||
|
Peter Wielander
|
efbc408c4e | [core] Ignore duplicate events per event class instead of failing the run (#3381) | ||
|
Alex Langenfeld
|
600b096d2f |
feat(world): batch analytics event lookups (#3390)
## Summary & Motivation Adds `events.getMany` to the analytics API, letting dashboard consumers enrich canonical event pages with ClickHouse provenance in one bounded, deduplicated batch request instead of N point lookups. Coordinated with a workflow-server batch endpoint. ## Test Plan Added unit tests for the client and schema; also ran Biome checks, `git diff --check`, and world-vercel typecheck. Signed-off-by: Alex Langenfeld <alex.langenfeld@vercel.com> |
||
|
Nathan Colosimo
|
9add9d782d |
Bound decoded sparse-array lengths (#3462)
## Summary - reject compact sparse arrays above the supported logical length at the main devalue hydration boundary - delegate accepted sparse-array construction to devalue - cover both current binary payloads and legacy flattened payloads ## Why Compact sparse-array encodings can represent a logical length that is disproportionate to the stored payload. Applying one codec-level bound keeps hydration predictable before downstream consumers process the decoded value. ## Impact Compact sparse arrays with logical lengths above 100,000 now fail hydration with a `RangeError`. Other payloads are unchanged. ## Verification - `pnpm --filter @workflow/core test` — 2,023 passed, 3 expected failures - `pnpm --filter @workflow/core typecheck` - `pnpm --filter @workflow/core build` - focused serialization suite — 154 passed - direct root-argument, bound-step, and aggregate-error payload checks - reuse, quality, and efficiency review |
||
|
Alex Langenfeld
|
c1a5c74edb |
fix(streams): surface typed retention expiry errors (#3410)
## Summary & Motivation Adds `StreamExpiredError` to `@workflow/errors`, carrying the run, stream, and server-reported expiry timestamp from workflow-server's 410 `stream-expired` envelope. The reconnect loop rethrows it instead of retrying, since retention expiry is terminal and a retry budget would only convert it into a generic exhaustion error. ## Test Plan Unit tests added for the 410 decoding path and the reconnect rethrow; typechecks pass across the touched packages. |
||
|
Nathan Rajlich
|
7683130461 |
Resilient step dispatch: parallelize step_created writes with queue publishes (#3365)
* feat(world,world-vercel,core): resilient step dispatch (parallel step_created + queue publish) Newly created steps are handed to the queue in parallel with their step_created event write, with the serialized input carried on the message (stepInput) so the queue consumer can idempotently re-ensure the event when the direct write failed transiently — mirroring resilient start (runInput) and resilient hook resume (hookInput). - @workflow/world: stepInput on WorkflowInvokePayload, CreateEventParams.viaStepDispatch, WorldCapabilities.resilientStepDispatch - core (node:vm): suspension handler publishes eligible steps alongside their create; the dispatch pass skips them (queuedStepCorrelationIds) - core (quickjs): dispatchPendingOps does the same for overflow steps; the ineligible fallback is now published in parallel too (removes the serial per-step enqueue loop) - consumer: on a redelivery, a stepInput-carrying message re-ensures step_created (marked viaStepDispatch) before executing - under an enforced precondition guard the parallel path requires backend cooperation (capabilities.resilientStepDispatch, declared by world-vercel): a 412-rejected step's in-flight dispatch is revoked server-side and its re-ensure refused - step dispatch/retry idempotency keys are step-identity-scoped (cid + hashed step name) so a revoked message for a reassigned correlation id cannot absorb the corrected schedule's dispatch - kill switch: WORKFLOW_RESILIENT_STEP_DISPATCH=0 * Validate stepInput.input as Uint8Array at the schema boundary Review feedback: producers only attach stepInput when the dehydrated input is binary and the queue transport preserves bytes (CBOR), so a non-binary value means the payload was mangled in transit. Enforcing Uint8Array in StepDispatchInputSchema fails the message parse instead of silently writing non-binary data into a step_created, and types the consumer's re-ensure so the unchecked 'as SerializedData' cast goes away. * Keep sequential dispatch under an enforced precondition guard (drop the resilientStepDispatch capability lift) Review feedback (two P1s): backend-side revocation bookkeeping cannot carry the guard's correctness property across the queue side-channel — - nothing orders a slow guarded create's eventual 412 (the moment the backend learns the dispatch is poisoned and records the revocation marker) before the consumer's redelivery re-ensure, so attempt > 1 is a probabilistic mitigation, not a happens-before; and - a best-effort marker that fails open (Redis loss) cannot back a capability the SDK treats as a correctness attestation. Only sequencing the publish after the create gives the message a happens-after edge over the create's guard verdict, so the guard gate is now unconditional: worlds that enforce the precondition guard keep the sequential create-then-publish dispatch. The parallel resilient path remains for unguarded writes (the quickjs engine everywhere, and worlds without the guard). Removes WorldCapabilities.resilientStepDispatch and world-vercel's declaration; the viaStepDispatch flag is kept and re-documented as advisory (server-side defense-in-depth only). This also dissolves the reviewed dedupe hazard on the step-identity- scoped dispatch keys: with no 410-ack path in any real SDK flow, a message for a never-created step keeps redelivering until an entity exists, execution always hydrates input from the committed entity (never the message), and a name-mismatched stale start is skipped by the server's stepName fence. * Correct the MAX_RESILIENT_STEP_INPUT_BYTES rationale: VQS has no hard message-size cap 256 KB is the queue's inline-vs-S3 threshold, not a rejection limit (payloads above it spill to S3-backed storage transparently). The 128 KiB bound is a cost/latency choice — keep step messages on the inline path rather than paying an S3 double-hop for bytes that already live in the event log. * Recover a missing step in-band when a stepInput-carrying delivery beats its create Durabench parallel sweeps (guard-off, node engine) caught ~4-8% of fan-out runs stalling one branch for ~306s on the resilient dispatch path. Root cause: the consumer's step_created re-ensure was gated on metadata.attempt > 1, but world-vercel's failure-retry path re-enqueues a FRESH message whose attempt resets to 1 — so when a delivery beat the producer's parallel step_created write, every fast retry hit the same 'step not found' rejection with attempt 1, and the step only recovered when the ORIGINAL message's ~300s visibility-timeout redelivery finally arrived with attempt 2. The recovery is now in-band and attempt-independent: when a stepInput-carrying execution rejects with the step-missing signature (WorkflowWorldError, 404 or the local worlds' message shape), the consumer materializes the step_created from the message payload and retries the execution once within the same delivery. The eager attempt>1 ensure is kept as a round-trip saver on genuine redeliveries. Sweep effect expected: the 305-306s TTLS outliers disappear while the resilient path keeps its p50 win (1054ms vs 1425ms at 64 branches). |
||
|
github-actions[bot]
|
9f5015b805 | Version Packages (beta) (#3378) workflow@5.0.0-beta.41 | ||
|
Peter Wielander
|
6786db9953 | World-side incrementing event ID (specVersion 6) (#3389) | ||
|
Shalabh Chaturvedi
|
b7591fbf21 |
sim-world - deterministic scenario testing for race conditions (#3328)
* Add world-sim: a deterministic simulation World for concurrency scenarios `@workflow/world-sim` is an in-process World implementation whose point is that nothing in it races. Every world call is a stoppable point with `before`/`positioned`/`after` phases, every call is attributed to a writer (orchestrator, a named step body, or an out-of-band external client), and the clock is virtual — a thirty-day sleep costs no wall time. A scenario script says "stop this writer here, do that, let it go", so an interleaving that a real deployment leaves to chance becomes something you can name. The model follows workflow-server where it matters: - Event ids are minted at the handler boundary, not at storage append. DynamoDB does not generate ids, so the server does, in the request handler — and that id is the log's sort key. This is what makes "earlier log position, later commit" expressible, and it is the hazard the fence scenarios are about. - The out-of-band write marker is keyed on the event's own ULID time and is forward-only, over `hook_received`, `step_completed` and `step_failed`. - Both halves of the staleness fence are modelled behind flags: the watermark (`preconditionGuard`) that clients send today, and the count (`countGuard`) that they do not. The count is synthesized on the caller's behalf and keyed by run rather than by writer, since an orchestrator and its inline step bodies are one process sharing one loaded log. `workbench/sim-world` is the scenario book — 39 of them, each a workflow plus a script. Several are pinned corruptions rather than passing assertions: they record what the runtime does today, so that a change in behaviour shows up as a diff. The doc-29/30/31 trio is the argument for the count guard, one flag apart: (B, A, C) corrupts under the watermark alone, is fenced once the count is on, and (B, C, A) corrupts with both on. That last one suspends mid-run on purpose. The fence is a conditional append evaluated inside the storage write, so there is no checked-but-uncommitted moment to slip past; the window nothing can close is the quiescent gap between deliveries, where the run makes no writes and so meets no checks. Both packages are private and unpublished, so no changeset. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * Add the new sim-world workspaces to the lockfile Kept separate from the source commit because it is not a clean diff. The two new importers are the part that belongs to this branch; the rest is re-resolution churn from running `pnpm install` at a later date than whoever last touched the file — `latest` specifiers like docs' `radix-ui` move on their own. Drop or regenerate this commit if the churn is unwelcome; the source commit stands on its own. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * Make a consistency violation fail the scenario that tripped it `expect.violations` let a scenario declare the corruption it reproduced and pass for reproducing it. That is a green suite describing a broken system, and it goes red on the day someone *fixes* the bug — backwards, and the opposite of what a test is for. The field is gone. A scenario now states the outcome the run should have reached, which for a corruption means the branch its own durable log implies, and stays red until the runtime gets there. Any violation fails the scenario. The six reproductions read as ordinary failures now: expected output "afterSlow:doc-26", got "afterFast:doc-26" Six scenarios are therefore red, and `run.ts` exits non-zero. The count is the signal: seven is a regression, five means something was fixed and a scenario is ready to retire. Five of the six have known fixes — four predate the count guard, and doc-29 goes green the moment a client sends `stateEventCount`. doc-31 has none, which is the point of it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * Document the world-sim implementation in DESIGN.md Folds the working notes into a checked-in design doc: module map, the runtime model the simulator has to match, the interception model and its three call phases, the determinism machinery, the store's guards and fault injection, the writer vocabulary, the termination budgets, both consistency checkers, and current test status. Links it from both READMEs, and updates the workbench's doc-31 note now that the append-tail fence it needs exists as a proposal. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * Say which fix closes each red scenario, and which are demonstrated The fix column read "predates the count guard" on three rows, which scans as "the count guard is involved" when it meant the opposite. It was also wrong on one: `two racing STEPS, no hook anywhere` cannot be fixed by anything predating the count guard, since the row below it is that same fault with `preconditionGuard` on and failing identically. The column now names the specific change, and a second column separates a fix that is argued for from one that is shown — a passing scenario that is the red one with the fix armed, same tempo, one flag apart. Two of the six have that; three name a fix with no paired scenario yet; doc-31 has none. Also states the thing the column could imply but does not mean: `countGuard` requires `stateEventCount`, which no client sends, so three of the five identified fixes are dark in production. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * Give scenarios stable ids, refer to events by log position, add colour Four changes to make a rendered scenario something you can cite and read. Scenario ids. Every scenario gets a hyphenated `id` next to its prose name — `in-flight-after-decision`, `stale-read-step-count-fork`. The name is a sentence and will be reworded; the id is what a commit message, a bug report or `pnpm sim <id>` refers to. The runner matches the id first and falls back to the name, so existing invocations keep working, and the table of the six red scenarios in DESIGN.md now cites ids. One reference scheme. Events were referred to three ways at once: the trace printed raw correlation ULIDs, violation messages carried `evnt_<ulid>` event ids, and the docs spoke of "position 7". Now there is only log position. `#12` is the twelfth event in the durable log sorted the way `events.list` sorts it, `@7` is the resource created at position 7, and ids inside violation messages are rewritten on the way out. That numbering is deliberately in *log* order while the trace prints in *commit* order, which makes the subject of the red scenarios visible on the page: a run whose log disagrees with the order its writers committed in shows positions counting backwards. Colour, by event family, and only when the destination is a terminal. Off under `NO_COLOR` or `--no-color`, forced on with `--color`. With colour off the output is the same plain ASCII, so it stays usable as a golden file. The workflows under test are now one file. Nothing imported across them and scenarios read them alongside the tempo that steers them, so five modules were a two-file hop for no benefit. Unit tests 61/61. Scenario book unchanged at 33 passed, 6 failed. Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * Add an append-only log world, and split the book into one file per scenario Three things the book could not do before. `appendOnlyLog` gives an event its log position at *commit* instead of when its handler minted one. That is the single change that makes a stale read impossible — the log can be behind, never wrong — so playing the same 39 scenarios with and without it is how you tell which of the six reds the change would actually close. All six: 33 pass / 6 violations mint-ordered, 39 pass / 0 violations append-only. It bundles two effects that have to move together: an overtaken write re-mints at the tail, and a withheld read returns a prefix rather than a hole, which is why `onStaleRead` now carries `{eventId, hidden, truncated}` and the trace distinguishes a lagging read from a stale one. `preconditionGuard` on `RunScenarioOptions` forces the fence off across the book, asking whether anything relies on it. Violations 6 -> 8 mint-ordered, so it is load-bearing there; 0 -> 0 append-only, so it is dead weight once positions are assigned at commit. Both flags are tri-state: `undefined` leaves it to the spec, which is not the same as `false`. The scenario book was a 1020-line file; it is now 39 files and an index that only decides reading order. Same 39 ids. This is the whole answer to "how do I add a scenario" — copy the file next door — and it is why the README can be short. Three `in-flight-*` scenarios are reworded so that no expectation is restated per world: a scenario is one sequence of movements, the only thing a world changes is what a read returns, and what catches the fault in both is the invariant that a run's log must replay back into that run. Also: `loadFlowHandler` moves out of `build.ts` into `load.ts`, so playing scenarios no longer drags SWC and esbuild into the module graph; the CLI gains `--report-only`, `--summary-file` and `--detail-file` for CI, with the default still exiting non-zero; and the workbench's `test` script points at `--report-only` so a recursive `pnpm -r test` does not go red for the six. Docs are split by task: the workbench README is how to add a scenario, and the package README plus DESIGN.md are how to change the simulator. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * Publish the scenario book from CI, without blocking a merge Plays the book on every pull request, once per log world, and posts both summaries as one sticky comment. It never gates. Six scenarios are red on purpose — each is a reproduction of a corruption the runtime can still produce, stating the outcome its own durable log implies and staying red until the runtime gets there — so a lane that failed on them would be red on every PR and read as broken rather than as informative. What it publishes is the pair of counts, and the thing to look at is whether they still say 33/6 and 39/0. A seventh red is a regression; five means something got fixed and a scenario is ready to retire. The append-only column is the measurement the pair exists for: it says which of the six would close if positions were assigned at commit. Non-blocking at the job level rather than only on the two sim steps, so that a failed install or a PR comment the token cannot write does not turn this into a red X either. The steps themselves keep their real exit codes, so the run still says which world was clean. `--title` is new on the CLI: two summaries land in one comment, and two headings reading "world-sim scenario book" would leave the chips line as the only way to tell them apart. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * Make the Sim World comment four lines until you open it Two collapsed folds, one per world, each with its count and a green or orange dot on the visible line; the table is behind them. Nothing else above the fold. The failures list is gone. Six scenarios are red on purpose, so a comment that led with them led with the part that was not news, and grew a wall of text on exactly the PRs that changed nothing. The count is the signal — and anyone who wants the names can open the table, which has always had them. `renderMarkdownSummary` now renders no heading of its own, since it is built to be stacked under one; the workflow supplies the heading and the one-line description. `mdCell`, `clip` and `dedupe` existed only for the failures table and go with it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * docs(sim-world): drop the six-red list, add an API reference The "six red scenarios" section was a second copy of something `pnpm sim` already prints exactly, and the copy that goes stale. What replaces it says how to read a red — an open bug stating the outcome its own durable log implies, green when fixed rather than when seen again — and points at DESIGN.md for the part that is not re-derivable from a run. The reference has three tables. Writers: what a concurrent thread of execution is here, and the four ids one can have. Movements: the instruction that moves a writer to a place and holds it, described against the world boundary, the position in the event log, and the commit to storage — three moments, which is why there are three stops rather than one. Withholdings: the two ways to change what a reader sees without holding anybody. Terminology unified on hold/held (the word `Held` and the trace already use) and on "assigned a position in the event log" / "committed to storage", which also fixes the `runToEventProduced` docstring: the event has crossed the world boundary there, so "submitted" was the misleading half. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * docs(sim-world): move the API reference into the package, uniform vocabulary The reference documents `@workflow/world-sim`, so it belongs beside the API and not in the workbench that happens to be its first caller. It replaces the old `## Writers` section, which said most of the same things in a different order and a different vocabulary. Three nouns, one meaning each. A *writer* is a thread of execution — the table lists all three kinds with the handle that names each and the events each one writes. An *advance* moves one writer to a named place and holds it; the three `runTo*` stops are the three moments a write has (crossed the world boundary, assigned a position in the event log, committed to storage), which is why there are three rather than one, and the table says which way a concurrent commit sorts at each. A *withholding* hides something from readers without holding anybody, which is what `withholdNextEvent` and `beginHookDelivery` have in common and why the latter is not an advance. "Movements" is gone in favour of "advances", which the code and the older prose already used. "Held" replaces "stopped"/"paused" throughout, matching `Held` and `isHeld()`. The workbench README loses the duplicate tables and keeps the four rules that bite on a first scenario, so it is a guide to adding one and links out for the rest. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * docs(world-sim): rename "Two logs" to "World behaviors", one-line motivations The section described two logs, but there are four world behaviors a scenario can pick between: the two log orderings and the two guards. Renamed and regrouped so each one is described by what it does rather than by which of the current reds it closes. Measured counts and "production" / "what happens today" framing are out of this README throughout — they belong to a run of the book, and a doc that carries them is stale the first time a scenario changes colour. The workbench README and the CI lane still print them, where they read as measurements. Every section's motivation is one line; the file-top motivation is unchanged. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * docs(world-sim): Usage shows both halves — the workflow and the script The example was a script with no workflow beside it, which hid the fact that you write both. Replaced with the smallest pair that has something to control: two steps in flight at once, and a script that decides which of them reaches the log first. The scenario in it was run before it was written down — `prepare` held before it takes a position, `finalize` committed into the earlier slot, #6 then #7 — so the positions the prose cites are the ones the trace prints. Link to the workbench moved to the end of the section, where it reads as "where to go next" rather than as an aside mid-example. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * docs(world-sim): say what "arm" meant instead of using the word The READMEs told a reader to arm a wait without ever saying that calling an advance and awaiting it are two separate things. That is the whole mechanism — `runTo` registers its watch synchronously when called, and the returned promise only reports arrival — so it is stated plainly once in Advances and the jargon is dropped everywhere it stood in for the explanation. "Armed" survives only where it means a guard is switched on, which is a different word doing a different job. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * docs(world-sim): two symmetric watches in the Usage example The example started one watch and awaited the other inline, which is exactly the shape a reader cannot generalise from — it looks like the two calls do different things. Both are now started, then both awaited, so the sentence above the block and the code below it say the same thing. Re-run before committing: same log, finalize at #6 and prepare at #7. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * docs(world-sim): neutral names in the Usage example `prepare` / `finalize` / `held` / `committed` made a reader decode four domain-ish names to see a mechanism that has nothing to do with any of them. stepA and stepB, watchA and watchB: the only thing left to notice is that the two watches stop at different points, which is the whole lesson. This detaches the example from the workbench's `parallelStepsWorkflow`, so it was verified against a throwaway copy of the workflow rather than assumed — same shape, stepB at #6 and stepA at #7, replay ok. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * refactor(world-sim): one event fold, two call phases, a smaller entry Three reductions to the same end — less surface to keep consistent — with the scenario book as the check that none of them changed behaviour. **One fold.** `create` validated *and* applied an event in a 17-case switch; `foldSeededEvent` applied it again in 15 cases without validating. Two copies of the event -> entity state machine that only a replay compares, and any divergence between them makes a replay disagree with the run it is checking for a reason that is not the runtime's fault. Both paths now end in one total `applyEvent`: the write path validates first and calls it, `seedFromLog` calls it with no validation at all, because those events were accepted once already and re-litigating them would reject legitimate history. store.ts loses 175 lines. Characterization tests for the seeded path went in first and were confirmed to bite before the refactor started. **Two phases.** `CallPhase` had a `positioned` phase between `before` and `after`, and `runToPositionMinted` / `runToCall` to park on it. Nothing in the book used any of the three: the mint/commit gap they existed for is reached through `beginHookDelivery`, which owns both halves explicitly and does not block the writer that made the write. Removing the phase also removes an `await` from the interception path, so this was measured rather than reasoned about — all four book runs are unchanged. **A smaller entry.** `index.ts` re-exported the construction kit — `createSimWorld`, `createSimStore`, `driveQueue`, `verifyReplay`, `checkInvariants`, the clock — which nothing outside the package imports and which made every one of their signatures a compatibility promise. The entry is now the scenario surface; extenders import from the module. `InFlightWrite` joins it, having been missing though `beginHookDelivery` returns it. Unchanged, and the point of saying so: 33/6/6 mint-ordered, 39/0/0 append-only, 8 violations with `--no-fence`, 0 with both. 70 unit tests green (up from 67), tsc and biome clean in both workspaces. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * fix(world-sim): follow the new event-result contract Rebase fixups for four core replay commits, no behaviour change of our own. `EventResult` is now a union — a populated page (`events` + `cursor` + `hasMore`) or all three absent — so the three fields cannot be assembled from three independent variables the type has no way to see agree. They travel as one `deltaPage` object, spread as a whole or not at all. The spread has to be conditional rather than optional: `...maybe` widens each field to `T | undefined`, which is neither arm. `processExitTriggersQueueRedelivery` is gone from the `World` interface — #3385 stopped exiting on an exhausted replay budget, so there is nothing left to tell it not to. The book is unmoved across the rebase, which is the thing worth reporting: 33/6/6 mint-ordered, 39/0/0 append-only, 8 violations with `--no-fence`, 0 with both — and the same six ids red, not a swap that nets to the same count. 70 unit tests green. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * feat(world-sim): a scenario for the unclaimed-payload delivery race Reproduces vercel/workflow#3406 as a pair of scenarios. A hook payload nobody reads registers an unarmed delivery barrier, and a step result is allowed to skip it — otherwise the workflow stalls until the barrier registry idles. The skip is transitive, so the step also skips an armed `wait_completed` merely parked behind that payload. Both branches then draw their next `step_created` id in the order the log does not record. The replay invariant cannot see this: live and replay run the same code, make the same mistake, and agree. What is checkable is the log disagreeing with itself — `wait_completed` is committed first, yet the step branch draws the earlier id. Verified red on the current runtime and green with #3406's diff applied. `claimed-payload-under-fork` is the control: same steps, same tempo, same log order through the step result, one branch awaiting the hook so the payload lands claimed. It passes in both cases. Getting there needed one new primitive. The delivery loop is serial, so a held inline step body stops the loop inside its own delivery and no timer can fire — every interleaving where a `wait_completed` lands while a step result is outstanding was unreachable. `sim.deliverQueued(select?)` takes a message out of the pending set and delivers it from the script, concurrently with the hold; `takeById` removes it first, so the loop can never pick up the same message. The book is now 41 scenarios: 34/7 with 6 violations mint-ordered, 40/1 with 0 violations append-only. The six violation reds and the --no-fence 6 -> 8 / 0 -> 0 measurements are unchanged. The new red is the first that stays red in both worlds, because no log position is wrong in it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * fix(world-sim): address review on the simulation world Fifteen findings from review, and one measurement that moved because of them. Correctness: - tempo: `abort()` stored the raw `reject` rather than the wrapper that also disposes the watch, so a point reached after teardown blocked a world call on a promise nobody could release. The deadline is one-shot and already spent by then, so that is a hang. Route abort through the disposing path and make the action a no-op once aborted. - world: `externalDepth` was a plain counter, so a step body writing concurrently with a scripted `deliverHook` was attributed to the script — invisible to any armed `runTo`, and `ext` in the trace. Measured: one misattribution book-wide. Scope it with `AsyncLocalStorage` instead. - writers: `runToEventCommitted` matched a *rejected* create, so in fence scenarios a script could wake believing a write was durable when it had 412'd. Record `failed` on the observed point and require `failed: false`. - clock/ids: reject fractional `advanceBy`, and floor in `ulid()` — a fractional millisecond silently minted `undefinedundefined…` ids that failed much later at `z.ulid()`. - streams: literal NUL bytes as key separators, `limit: 0` paging forever, and a garbled cursor slicing the array away as `NaN`. - scenario: register the handler under the exported `WORKFLOW_QUEUE_PREFIX`. The count guard, which is the substantive one. Since #3145 `@workflow/core` sends `stateEventCount` on every replay-context create and the server's count guard defaults on, so the sim's "no client sends the count" claim was stale in six places. `countGuard` now follows the fence, the sim prefers the runtime's own count over its reconstruction, and the two scenarios whose subject is isolating the watermark half say `countGuard: false` explicitly. The scoreboard does not move under the production-shaped default, which is itself the answer to the review's question. `log.monotonic-order` could never fire: its only caller fed it the sorted array. It now takes commit order, supplied only by a world that promises the two agree — under a mint-ordered log an out-of-order commit is the premise the scenario injected, not a defect. Docs: DESIGN §5 gains the two ways the sim's guards are stronger than production's (FIFO-vs-mint-order pruning, and a fence that is exact in-process where production's is region-local and fails open); §9's scoreboard is regenerated and its "no fix armed anywhere real" claim corrected; §10 gains the parallel hook-resume path, which no sim delivery takes. `step-vs-step-fork` and its twin now say who the withheld reader is in production. `unclaimed-payload-under-fork` is green after the rebase — #3406 fixed the delivery-barrier ordering — so the book is back to six reds and stays there. Counts refreshed everywhere they appear: 35/6/6 mint-ordered, 41/0/0 append-only. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * fix(world-sim): make the README's code samples type-check `packages/docs-typecheck` globs `packages/*/README.md`, so the new README went into the Docs Code Samples job and five of its `ts` blocks failed. Four were genuine excerpts with an unbound `sim` or `spec`; one was a match- object *sketch* containing a `…`, so not TypeScript at all — that one loses its `ts` tag, joining the eleven other untagged blocks in the file. The rest name what they use. That needed two `paths` entries in the checker: `@workflow/world-sim` was unmapped, and an unresolved import is deliberately tolerated there (`isExpectedMissingModule`), so every identifier in those samples was `any` — they would have gone green while checking nothing. Mapped, they are checked against the real declarations: seeding `id: 12345`, `readyAtMsTYPO`, `dirsTYPO` and `runToEventCommittedTYPO` is caught against `ScenarioSpec`, `PendingMessageView`, `SimBuildOptions` and `Writer`. The lead teaser keeps its four unadorned lines and takes a `@skip-typecheck` marker instead; the same calls appear in full, checked form further down. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> --------- Co-authored-by: vercel[bot] <35613825+vercel[bot]@users.noreply.github.com> |
||
|
Peter Wielander
|
69c30ff49e | Gate the unconsumed-event check on delivery idleness (#3439) | ||
|
Caleb An
|
2d5ca54086 | Set vercel approvers to workflow team (#3435) | ||
|
Makoto Arata
|
1a64f68472 |
fix(core): preserve new.target in the deterministic Date override so Date subclasses work (#3372)
* test: add failing test for Date subclassing in workflow VM Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: ar_tama <arata.makoto@gmail.com> * fix(core): preserve `new.target` in the deterministic `Date` override so `Date` subclasses work in workflow functions The VM's `Date` override was a plain function, so `class X extends Date` lost the subclass identity: `super()` returned a fresh plain `Date` that became `this`, dropping the subclass's methods and fields. This silently broke `Date` subclasses like `TZDate` from `@date-fns/tz`. Using `class Date extends Date_` keeps `new.target` intact, and `extends` already wires up the prototype chain and statics, so the manual `prototype` assignment and `Object.setPrototypeOf` fix-ups are no longer needed. Determinism is unchanged: zero-arg construction still returns the fixed timestamp and `Date.now()` is still overridden. Fixes #3371 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: ar_tama <arata.makoto@gmail.com> * test: add failing test for calling `Date()` without `new` Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: ar_tama <arata.makoto@gmail.com> * fix(core): keep `Date()` callable without `new` Use a plain function that branches on `new.target` and constructs via `Reflect.construct(Date_, args, new.target)` instead of a class: subclassing still works (`new.target` is forwarded), and calling `Date()` without `new` now matches the spec — arguments are ignored and the (fixed) time string is returned, where the previous override returned a `Date` object. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: ar_tama <arata.makoto@gmail.com> * chore: update changeset to match the final `Reflect.construct` implementation Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: ar_tama <arata.makoto@gmail.com> --------- Signed-off-by: ar_tama <arata.makoto@gmail.com> Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |
||
|
Peter Wielander
|
dc61ea1b13 | docs(builders): make the onAfterTransform sample self-contained (#3424) | ||
|
Luca Maraschi
|
4ec7acaa71 |
feat(builders): observe accepted transforms (#3163)
Signed-off-by: Luca Maraschi <luca.maraschi@gmail.com> Co-authored-by: Peter Wielander <mittgfu@gmail.com> |
||
|
Peter Wielander
|
fbebf7104d | [core] Keep step results ordered behind waits parked on unread hook payloads (#3406) | ||
|
Rich Haines
|
1aed119e84 | [docs] upgrade geistdocs to 1.19.6 (#3407) | ||
|
Shalabh Chaturvedi
|
264ddff67b |
Add WebSocket transport for step-execution event writes (opt-in) (#3084)
* sdk side for workflow server websockets Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * hardcoded workflow server Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * debug info Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * more debug Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * remove unnecessary debug Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * default on websockets, and override url Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * fix for missing funcs Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * make websockets opt outo Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * enable ws again Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * [revert later] reduce test to single test, test both http and ws at the same time Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * empty Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * Run full suite with and without ws Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * empty Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * improve e2e test Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * minimize tests Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * fallback to http when proxy present Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * default to websockets, remove matrix Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * remove smoke test Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * update to new protocol Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * adjust for new protocol (runid in path) Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * fix ws transport error Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * blank Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * fix ws dep Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * fix ws external: only accelerators, not ws itself Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * add dedicated WS-transport e2e job; flip WS default back to opt-in Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * build all packages before local vercel build (needs workflow/nitro on disk) Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * force NITRO_PRESET=vercel for the local vercel build step Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * install vercel CLI once instead of npx-ing it per command Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * add changeset for WS events transport Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * Harden the WS events transport and gate its e2e jobs Follow-ups from review of the WS transport. CI: `e2e-vercel-ws-transport` was wired into the `summary` job but not into `e2e-required-check`, so all three WS jobs could fail while the required check stayed green. Added to both branches of the status validation — including the `workflow-server-test` label branch, where the job runs under the same gating as `e2e-vercel-prod`. Transport: - `reqId` and the pending-reply map are now per connection rather than per transport. The protocol defines `reqId` as a per-connection counter, so a reconnected socket restarts at 1; with one shared map that collided with the previous socket's still-registered waiters. It also makes the superseded-socket guard structural instead of something the close path has to remember. - Post-open socket errors are no longer silent. The only `'error'` listener closed over the connect promise's `reject`, already settled once `'open'` fired, so every broken pipe / 1009 / protocol fault was swallowed and its requests hung with no per-request timeout to save them. Now logged, and the connection is torn down. - An unexpected close reconnects eagerly instead of waiting for the next write, since a socket breaking mid-run means more writes are coming. Bounded by exponential backoff, an attempt cap that falls back to lazy reconnect, a bail-out when a newer socket is already live, and an `unref()`ed timer so a backoff window can't delay handler exit. - `ws.send()` failures reject their request. `send()` doesn't throw on a non-OPEN socket — it reports through a callback we weren't passing — so the request just sat in `pending` forever. - The reserved `reqId: -1` malformed-frame reply and undecodable frames are logged loudly instead of dropped. - Auth headers resolve once per socket via a thunk, not once per event. The bearer only rides the upgrade, so the old code awaited `getVercelOidcToken()` on every write and discarded all but the first. Re-resolving on reconnect also means a new socket gets a fresh token. Adapter: a reply with no numeric status now fails closed. Defaulting to 200 reported a write as applied whenever the client met a frame it didn't understand — and the protocol is explicitly designed to grow new response variants. Tests: 24 new unit tests over the paths the e2e suite can't reach on demand (send failure mid-flight, error after open, late close from a superseded socket, reconnect backoff and give-up, sentinel/undecodable frame logging, one-token-per-socket) plus the adapter's fail-closed and typed-error mapping. Co-Authored-By: Shalabh Chaturvedi <shalabhc@users.noreply.github.com> Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * ci: re-trigger to confirm the prior e2e failures were flake No code change. The 5 failures on 0bb21e7 clustered in a ~20s window across HTTP-path jobs (example/nuxt on the same test, sveltekit on a timeout) and one WS job (sleepingWorkflow's clock-skew assertion), which points at the environment rather than the transport changes. Re-running to confirm. Co-Authored-By: Shalabh Chaturvedi <shalabhc@users.noreply.github.com> Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * Add WS wire-contract conformance tests and pin the transport gate Three gaps in the existing coverage. **The HTTP path was already covered** — `events-v4.test.ts` has 22 tests, including five directly on `createWorkflowRunEventV4` over HTTP (alias URL, frame meta contents, response decoding, skipPreload/stateUpdatedAt forwarding). Those run with the gate unset, so they do confirm the two-branch refactor didn't disturb HTTP. No new tests needed there. **But nothing pinned the gate itself.** Every HTTP assertion stays green if the default flips to WS, because the transports are built to be indistinguishable at the result layer — and an earlier revision of this branch did flip the default deliberately, for benchmarking. Added tests for `isWsEventsTransportEnabled()` across values, and one that drives a real HTTP request through a MockAgent while asserting the WS transport is never constructed. **Nothing verified the bytes.** `ws-transport.test.ts` replies with whatever the test hands it, which proves the client's lifecycle but not that its frames are what workflow-server accepts. That's the drift the spec doc exists to prevent, and it already happened once: event meta flat on the frame where the server wanted it nested under `event`, with both sides' tests passing. `ws-protocol-conformance.test.ts` pairs the real client stack (through `createWorkflowRunEventV4`) with a fixture mirroring the server route's per-message handling: decode one frame, validate against a local copy of `WsRequestFrameSchema`, dispatch, encode the reply the way `replyMeta` does. `experimental_upgradeWebSocket` needs a real Vercel runtime, so the socket is faked — everything above it is genuine. Covers: the frame shape the server accepts (and that `reqId`/`type`/ `runId` don't leak into the event meta), payload passthrough, exactly one frame per message, 409 → the same typed error HTTP raises, fail-closed on an unknown reply variant, and reqId correlation across concurrent writes. Plus golden byte fixtures, since the schema copy is the one thing here that can silently drift. This is the "golden-frame interop test" the server spec lists as an open gap; the matching half still needs to land in workflow-server. Verified the conformance suite is not vacuous: flattening the client's frame meta fails 5 of its tests. Co-Authored-By: Shalabh Chaturvedi <shalabhc@users.noreply.github.com> Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * match the HTTP RetryAgent's transient-failure policy on WS HTTP event writes go through an undici RetryAgent (RETRY_AGENT_OPTIONS): 5xx and transient connection errors are retried in-process, honoring Retry-After. The WS path never touches undici, so it shipped with no transient-failure handling at all — a single 503 or a mid-write reset surfaced straight to the step runtime and cost a whole step retry where HTTP would have absorbed it in milliseconds. That gap is invisible in a passing test run: writes still succeed, they just cost far more. So copy the policy rather than reinvent it — [500, 502, 503, 504] plus transport failures, undici's default backoff, Retry-After honored, and 429 deliberately excluded for the same reason RETRY_AGENT_OPTIONS excludes it (a firewall challenge this client cannot solve, which in-process retries only amplify). Adds WsTransportError so retryability is a typed property of the failure rather than something the adapter infers by string-matching. Splits resolveWsTransport()/wsReplyStatus() out of postEventFrameOverWs so the retry loop stays readable. The existing "fails closed on an error frame" test used status 500, which is now absorbed by the retry — switched to 403 so it keeps testing fail-closed rather than accidentally testing no-retry. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Co-Authored-By: Shalabh Chaturvedi <7066873+shalabhc@users.noreply.github.com> Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * lazy-load `ws` so the default HTTP path never evaluates it events-v4.ts imports ws-transport.js unconditionally — the transport gate is a runtime branch, not a build-time one — so a top-level `import { WebSocket } from 'ws'` put `ws` and its optional native accelerators on the module-init path of every deployment, including the overwhelming majority that never opt in and never open a socket. Defer it to the first connect, memoized as a promise so concurrent first connects share one import. WebSocket.OPEN becomes an inlined constant so the readyState check doesn't pull the module in just to read it off the constructor. This does NOT remove the need for the bufferutil/utf-8-validate externals this branch also adds: webpack and Rollup both statically follow a dynamic import(), so the build-time story is unchanged. What it buys is that a deployment which never enables the transport never *evaluates* `ws`, so a mis-bundled accelerator can't break it. The test lives in its own file because vitest caches a vi.mock factory result for the life of the module registry — once any test in a file has connected, the factory never runs again and the counter can't distinguish "loaded lazily" from "loaded at import". Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Co-Authored-By: Shalabh Chaturvedi <7066873+shalabhc@users.noreply.github.com> Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * release idle WS transports instead of renewing them forever The transports map was never pruned and WsEventsTransport had no way to close. Combined with eager reconnect that made a connection immortal by construction: the server drains at its own maxDuration and closes, the client immediately reopens, and the server pins a fresh invocation — for a run that finished long ago. A warm container ended up holding a live socket, and a live server invocation, for every runId it had ever served. workflow-server#683 already lists "one invocation stays resident per run rather than per write" as a known gap; this made it "per run, forever". Add close() plus a 60s idle release. There is no "run complete" signal to hang teardown off — the events adapter is a stateless per-write call — so idleness is the available proxy. 60s sits well below the server's ~680s drain deadline, so the client releases rather than the server reclaiming, and well above the gap between steps of an active run. scheduleReconnect() now bails when closed: close() closes the socket, which fires the same close handler an unexpected drop would, and without the guard the transport would instantly reconnect what it just released. request() revives an idle-closed transport rather than failing the write, re-registering itself only if nothing newer has claimed the map slot. Eviction therefore costs one handshake, not an error. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Co-Authored-By: Shalabh Chaturvedi <7066873+shalabhc@users.noreply.github.com> Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * refresh the bearer on an auth_expiry drain workflow-server#683 tags a drain frame with why it is closing: max_duration means the socket aged out and a plain reconnect is right, auth_expiry means the *bearer* ran out and reconnecting with the same one just earns a 401. This client logged the drain and ignored the reason, so against #683 an auth_expiry drain would burn all five reconnect attempts against a token the server had already rejected, then give up. Parse the reason (absent reads as max_duration, so this stays correct against the currently-deployed server) and thread forceRefresh through the getHeaders thunk, which triggers @vercel/oidc's refresh path via a wide expirationBufferMs. Worth being precise about when that can actually help. getVercelOidcToken resolves getContext().headers['x-vercel-oidc-token'] ?? env, and refreshToken() only writes the env var — the request-context header wins. So inside a deployed function there is genuinely no fresher token mid-invocation and the refresh is a no-op; outside one (CLI, local dev, a long-lived server) it works. That makes the guard the load-bearing half: if the re-resolved bearer is byte-identical, decline to reconnect, say so, and wait for the next write — which usually arrives on a new invocation carrying a new token. That failure is marked non-retryable so the retry loop doesn't spin on it either. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Co-Authored-By: Shalabh Chaturvedi <7066873+shalabhc@users.noreply.github.com> Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * document the WS path's instrumentation gap The HTTP branch goes through fetchV4 -> instrumentedFetch, which is not just a fetch wrapper: it opens the OTEL CLIENT span, injects trace context, sets the cache-bust header, emits the DEBUG logs, and routes through the global fetch that Vercel's observability "outgoing requests" view instruments. The comment on fetchV4 records why that matters — bypassing it via undici.request() is exactly what once made v4 event traffic disappear from the log viewer. The WS branch bypasses all of it. With the flag on, per-event writes have no client span, propagate no trace context to workflow-server, and don't appear in the outgoing-requests view; the server's own transport-tagged request metrics are the only remaining signal. That's acceptable for an opt-in POC behind a flag and unacceptable as a default, so write it down where someone deciding to flip the default will read it: instrumenting the transport is a prerequisite for that, not a follow-up nicety. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Co-Authored-By: Shalabh Chaturvedi <7066873+shalabhc@users.noreply.github.com> Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * ship the ws-accelerator externals instead of documenting a workaround `bufferutil` and `utf-8-validate` are optional native accelerators for `ws`, and neither is installed by default. Every bundler has to be told to leave them alone, for two different reasons: Rollup/Vite/Nitro fail the build outright (`Could not resolve "bufferutil" imported by "ws"`), while webpack bundles the JS wrapper without its native `.node` binding and throws `bufferUtil.mask is not a function` at runtime. The webpack half shipped in `@workflow/next`. The Rollup half only existed in `workbench/vite` and `workbench/tanstack-start` as `nitro.rollupConfig.external` — app configs, not shipped code. So a real user of `@workflow/vite`, `@workflow/nitro`, `@workflow/nuxt`, `@workflow/sveltekit` or `@workflow/astro` hit the same build failure the workbench had already worked around, and had to rediscover the fix. Fix it where it propagates: `workflowTransformPlugin` in `@workflow/rollup`, which all of those integrations already install. It is already the home of exactly this pattern for the optional `@opentelemetry/api` peer, so this sits next to its closest precedent. Note the treatment is deliberately the inverse of the OTEL one, which is externalized only when it *can't* be resolved. The OTEL API must load for tracing to work, so a self-contained output has to bundle it when present. These accelerators must specifically NOT load — they are a performance nicety with a correct try/catch fallback in `ws` — so unconditional external is both simpler and safer than risking a half-bundled native module. The two workbench configs drop their local copies, which is what proves the shipped fix actually works rather than being masked by them. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Co-Authored-By: Shalabh Chaturvedi <7066873+shalabhc@users.noreply.github.com> Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * one retry policy for both transports, and no unanswerable waiters Two review findings on the WS events transport. **Retry belongs to `event-retry.ts`, not the adapter.** The WS path had its own retry loop, justified as mirroring undici's `RetryAgent`. That justification was wrong: `RetryHandler` defaults `methods` to GET/HEAD/ OPTIONS/PUT/DELETE/TRACE and nothing overrides it, so the `RetryAgent` never retried an event POST on either transport — which is precisely why `event-retry.ts` exists. Worse, that loop sat *inside* `withEventPostRetry`, so it defeated a compile-checked safety gate: `EVENT_RETRY_ELIGIBILITY` marks `step_started`, `step_retrying` and `hook_received` non-retryable (a replayed `step_started` double-increments `attempt`), and those frames were re-sent up to five times before the gate ever saw a failure. For eligible types the two loops multiplied: 3 outer attempts x 6 inner, with an inner backoff reaching 30s against an outer base deliberately set to 100ms. `postEventFrameOverWs` now makes one attempt and translates failures into the vocabulary that policy already speaks — a transport failure becomes a `WorkflowWorldError` with `code: 'TRANSPORT'`, exactly as `utils.ts` does for a failed `fetch`, and `isRetryableEventPostError` gains one clause keyed on that code. `WsTransportError` loses its `retryable` flag; its only consumer was the deleted loop. Two deliberate consequences. The code-keyed clause broadens HTTP in-process retry to `UND_ERR_CONNECT`, `UND_ERR_CLOSED` and `EAI_AGAIN`, which were in utils.ts's transient set but missing from event-retry.ts's — two hand-maintained lists collapsed into one semantic code. And the stale-token case (drain for auth expiry, refresh yields the same bearer) now gets two in-process attempts that cannot succeed, ~300ms before it falls through to queue redelivery; that is cheaper than keeping a WS-specific policy alive for one call site. `TIMEOUT` is deliberately not in the clause: utils.ts maps a caller-supplied `AbortError` onto it, and a cancelled write must not be re-issued. A status-less reply also stops being a bare `Error` — as one it failed `WorkflowWorldError.is()` and surfaced a protocol version skew as a USER_ERROR. It is now `code: 'PARSE_ERROR'`, the same code utils.ts uses for an unreadable HTTP body, and for the same reason: the write may or may not have landed. **No waiter is left unanswerable.** An undecodable frame, the server's malformed-frame sentinel (`reqId: -1`) and a non-numeric `reqId` were logged and dropped. None can be correlated by construction, so the request that provoked them stayed in `pending` with nothing in existence able to settle it — freed only by the server's own drain (~680s from connect), typically past the invocation's `maxDuration`. Each now fails the connection: every waiter learns why, and the socket is replaced. A reply for an id nobody is waiting on stays log-and-drop, deliberately — that request already settled, so nothing is orphaned, and failing the socket would punish healthy in-flight writes. A per-request deadline backs that up for whatever is left, including a server that accepts a frame and never answers it. Same knob as the HTTP path (`WORKFLOW_REQUEST_TIMEOUT_MS`, 60s), whose doc comment already describes this exact hang-to-SIGTERM pathology. One existing idle-teardown test needed the deadline raised: the idle window and the default deadline are both 60s, so a request could not outlive the former without also outliving the latter. The test is about `inFlight > 0` suppressing the teardown, so it now sets the deadline out of the way. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * Open the ws socket when the invocation starts, not on its first write Lazily connecting bills the whole handshake — an upgrade round-trip plus the OIDC token mint that rides it — to whichever event a fresh invocation writes first. When that is a `step_started` issued as the step body is already running, the event's server-recorded timestamp lands later than the work it describes: the step looks shorter than it was. That is the shape of the e2e timing failure on this branch, where a 9s step measured 6.5s from `getStepMetadata().stepStartedAt`. The queue handler is the earliest point that knows the run id, and a message delivered for a run means writes are coming, so `warmWsEventsTransport` starts the handshake there. By the first write it is done or in flight, and the write just uses it. Nothing about it is load-bearing: - It doesn't await, and can't fail the handler. A warm that fails logs and stops — a never-opened first connect is precisely the case `connect`'s close handler already declines to retry, so no backoff loop starts for a run that may never write. The first real write connects as it would have anyway, carrying the shared retry policy. - No-op unless `WORKFLOW_EVENTS_TRANSPORT=ws`, and no-op for the api-workflow proxy World, which can't serve an upgrade at all — the same fallback the write path takes. - Warming arms the idle timer as if a request had settled, so an invocation that warms and never writes (a health probe carrying the run id it is about to create) releases its socket on the usual 60s rather than stranding it. The socket is not `unref`'d, so a stranded one would hold this process and a server invocation open. Also closes a race that warming makes reachable: `close()` can only drop the connection it can see, so a release landing mid-handshake left the socket to install itself afterwards onto a transport already evicted from the cache, which nothing would then ever close. The `open` handler now declines to adopt a socket whose transport was released while it was connecting. This was already reachable via the eager reconnect path, just much harder to hit. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * changeset: just the env var Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * inline the ws-accelerator predicate at its only call site Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * refactor(world-vercel): trim ws-transport comments Comments were 47% of the file. Cut the historical narration, the restatements of adjacent code, and the repeated rationale (the `unref` reasoning appeared four times, per-connection reqId three), keeping the non-obvious facts: `ws.send()` reports failure via callback instead of throwing, reqId is per-connection so `pending` must be too, the unknown-reqId case is deliberately non-fatal, the auth_expiry same-token bail-out, and why the idle timeout exists at all. No code changes. Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * own transport selection in the transport module `events-v4.ts` was assembling the WS transport itself: reading the opt-in flag, resolving the URL, deciding which Worlds can use a socket, minting the per-connection header thunk, and holding the two once-per-process log latches. None of that is about turning an event into a frame, which is what the rest of that file does. Move it next to the socket it configures — `events-v4.ts` now consumes one seam (`resolveWsTransport`) plus the gate, and `queue.ts` gets `warmWsEventsTransport` from the module that owns the warm. `headersToRecord` now lives in `http-core.ts` because both callers need it and neither may import the other: `events-v4` already depends on the transport, so the reverse edge would be a cycle. Test fallout, and the reason the move is worth it: `events-v4-ws.test.ts` mocked `getWsEventsTransport` to observe the resolve step, which no longer intercepts anything now that the call is intra-module — an ESM mock replaces a module's exports, not its own call sites. That mock's tests were only ever about selection, so they move to `ws-transport.test.ts`, where the real selection code runs against the existing fake-socket harness instead of a stub. `resetWsEventsTransportsForTest` clears the log latches so the once-per-process assertions don't depend on test order. What stays behind mocks `resolveWsTransport` and covers what that file is actually for: reply frame in, `Response`-shaped result out — including the null-resolve fallback to HTTP, which nothing covered before. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * import `ws` statically The lazy `import('ws')` was there to keep the package off the module-init path of deployments that never opt in — `events-v4.ts` imports this module unconditionally, since the transport gate is a runtime branch. Measured, that buys ~17ms: `require('ws')` is 16.5-18.0ms cold, 13 modules, and neither `bufferutil` nor `utf-8-validate` loads (optional peers, absent by default). Bundle size is identical either way — webpack and Rollup both statically follow a dynamic `import()`, which is why the externals in `@workflow/builders` are unaffected by this change. For 17ms it cost a memoized promise, an inlined `WS_READY_STATE_OPEN` (so a readyState check wouldn't force the module to load just to read a constant off the constructor), and a whole test file — `ws-transport-lazy.test.ts` had to live alone, because vitest caches a `vi.mock` factory result for the lifetime of a module registry, so only a file that connects exactly once can observe the laziness at all. It also skewed the thing this branch exists to measure. The import lands inside the first connect, so on a warm container it is billed to whichever event write opens the socket, inflating the timestamp of the step it labels — the same distortion the queue pre-warm was added to remove. Also drops `WS_READY_STATE_OPEN` in favour of `WebSocket.OPEN`, now that reading it is free. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * tighten the comments on the ws transport Comments only — no code changes in this commit. Cuts ~150 lines of prose across the WS additions. The rule applied: keep the design factors a future reader needs (why the connection is scoped to a run, why a bad reply takes the socket down, why the accelerators are externalized unconditionally, why `TIMEOUT` is excluded from the `TRANSPORT` classification) and drop the narrative of how the code got here — which revision did what, what an earlier attempt got wrong, what was measured on the way. That history lives in the PR and the git log, where it doesn't have to be re-read on every visit to the file. Biggest reductions: the retry essay above `postEventFrameOverWs` (30 lines to 11), the flag's OTEL-gap note (34 to 13), the OIDC refresh explainer (26 to 14), the accelerator rationale in `@workflow/builders` (26 to 14), and the conformance suite's header (28 to 17). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * inject W3C trace context on the ws upgrade Frames carry no headers, so the upgrade is the only place this transport can propagate context; the server parents a run's event spans to whichever invocation opened the socket. Covered in trace-propagation.test.ts, both with and without an active span. Splits the opt-in gate into an import-free ws-transport-enabled.ts so callers can answer it without loading this module (used by the next commit). Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * load the ws transport module only when it is enabled Both call sites read the gate from the import-free module and dynamically import ws-transport.js behind a true result, so a deployment on the HTTP default never pays ws's ~17ms of module init. The queue pre-warm absorbs it for one that opted in, keeping it off the first event write. Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * document WORKFLOW_EVENTS_TRANSPORT as experimental Names the instrumentation gap (no client span per write) and the proxy path where the variable is ignored. Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * correct why the ws accelerators are externalized No bundler fails the build on the unresolvable require — verified against Rollup 4.62. webpack half-bundles the native module and Vite substitutes a stub that makes the require succeed; both leave bufferUtil.mask undefined and throw only once a frame reaches the native masker at 48 bytes, which every CBOR event frame does. Same claim was repeated in the rollup plugin and its test. Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * trim the WORKFLOW_EVENTS_TRANSPORT docs to user level Mirrors the other Vercel World env vars: same facts on both pages, each in its page's format. The instrumentation and socket-lifetime detail belongs in the code, not in a user-facing reference. Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * cover the vite bundler in the ws transport lane Vite substitutes a stub for ws's absent native accelerators rather than failing the require, so nothing catches it until a masked frame reaches 48 bytes — and this job's three existing lanes are esbuild, turbopack and nitro. Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * claim only what is measured about rollup and the ws accelerators The rationale asserted plain Rollup was "safe by accident" via a mechanism only ever observed in a minimal repro. Nitro traces and externalizes `ws` in a production build, so the bundled path is not reached there at all. Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * Give the events socket an explicit lifetime instead of an idle timer `openWsChannel` / `closeWsChannel` bracket one invocation of the flow route, and are the only calls anywhere that create a channel. Writes ask `resolveWsTransport` whether one is open — a lookup now, never a create — and take pooled HTTP when it says no. That removes the reason the idle timeout existed. A lazily-created socket has no owner, so a timer was the only thing able to end it, and the socket is not `unref`'d: the process could not exit, and a server invocation stayed pinned, for the full window past the last write. It also settles `run_created`. The trigger path opens no channel, so a lone write no longer pays for a handshake it cannot amortize — `start()` runs in an arbitrary request handler with no boundary the SDK can see. Refcounted rather than a flag: inline step executions ride the flow topic on per-step topics, so a run's steps can be concurrent invocations in one instance sharing the channel, and the first to finish must not cut the others short. A failed connect closes the channel so the invocation's writes fall back to HTTP instead of each paying its own doomed handshake. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * Name the one reply header the WS path does not map The server copies six headers into an `event_ack`'s meta and this record maps five. The sixth, `X-API-Deprecated`, is inert today — the v4 route's middleware chain has no deprecation middleware to set it — but the record is the only header source a WS reply has, so an unmapped key is gone rather than merely unread, which is not true of the `Response` the HTTP path returns. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * docs: note that WORKFLOW_EVENTS_TRANSPORT=ws is ignored on the proxy path The api-workflow proxy is an HTTP-only REST gateway and does not forward a WebSocket upgrade, so a World configured with projectConfig keeps writing events over HTTP regardless of the setting. Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * ci: gate the ws-transport e2e lanes on a label Three real `vercel deploy`s per run is too much to charge every unrelated PR in the repo for a transport that is off by default. PRs opt in with `ws-transport-test` (or `workflow-server-test`, which already exists to test the half of this the protocol lives in); main keeps the signal on every commit. The required aggregate has to allow the lane to be skipped in that case, so its status is asserted only when the lane was actually supposed to run. Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * chore: regenerate pnpm-lock against current main main resolved `ws` to 8.20.0 as a transitive peer; this branch adds it as a direct dependency of world-vercel and floats it forward, which rewrites every `openai@x(ws@y)` peer key in the lockfile. Merging main textually combined the two, leaving those keys pointing at a `ws` entry the merged file no longer had — `--frozen-lockfile` then failed with ERR_PNPM_LOCKFILE_MISSING_DEPENDENCY on the PR's merge ref. Regenerated from main's lockfile so ours is a minimal delta on top of it. Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * fix(world-vercel): align ws on the version main already resolves The lockfile broke on the PR's merge ref, not on this branch's head: main resolves ws@8.20.0 as a transitive peer, and a `^8.21.1` direct dep here floated it forward, rewriting all 73 `(ws@8.20.0)` peer keys. Git merged the two lockfiles without a conflict but left main-side keys pointing at a ws entry the merged file no longer had, so `--frozen-lockfile` failed with ERR_PNPM_LOCKFILE_MISSING_DEPENDENCY. `^8.20.0` resolves to the copy main already has, so the lockfile delta is the two importer entries instead of a repo-wide rewrite that re-breaks every time main moves. Also keeps one ws in the store rather than two. Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * Bind the channel release to the instance it claimed closeWsChannel resolved the transport by URL, but the refcount lives on the instance. A channel is evicted from the map as soon as it closes — a refused upgrade does that on the connect path — so the next opener for the same run registers a different instance under the same URL, and the first invocation's close then decremented that one instead. It dropped a socket a live invocation was still writing over, and for the event types EVENT_RETRY_ELIGIBILITY marks non-retryable there is no second attempt to carry the in-flight write over HTTP. openWsChannel now returns an idempotent release closed over the transport it incremented, and queue.ts holds that instead of re-resolving the run. The close awaits the open's own promise, so it also can no longer land ahead of the claim it releases. Also names the scope of the connect-failure de-opt: it covers the handshake only, so a channel that connects and then fails every write keeps taking the WS path. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * Decode a transport result, not a Response Main extracted the v4 POST decode into a helper typed `Response` while this branch narrowed the POST result to `FrameResponseLike`, because the WS branch synthesizes its result rather than holding a real `Response`. The two merge without a textual conflict and then fail to typecheck. Widen the helper: it reads only the two members `FrameResponseLike` declares, and a `Response` still satisfies them, so the HTTP call sites are unchanged. Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * Re-run CI Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * Re-run CI Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * Reconcile the WS transport with main's v4 POST rework main moved the materialized POST result off the `x-wf-*` response headers and onto a typed CBOR body, and added a second response shape: two callers now POST with `Accept: application/vnd.workflow.v4-frames` and read back a sentinel-terminated sequence of frames. A frame stream has no representation in a protocol that pairs one reply frame with one request frame, so the WS switch moves off the shared poster and onto `createWorkflowRunEventV4` alone — the materialized write, which is the hot per-step path this branch exists to shorten. `run_started` and the `hook_received` preload stay on HTTP. `decodeCreateEventResponse` takes `FrameResponseLike` rather than `Response` because the WS branch has none to hand over; a real `Response` satisfies the interface, so the HTTP callers are unchanged. The ids now come out of the CBOR body, so `replyMetaToHeaderRecord` no longer maps any `x-wf-*` name — only the two headers `errorFromV4Response` reads. Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * Re-run CI Resample the WS-arm sleepingWorkflow failure: it has now recurred on a second axis (nextjs-turbopack, 7709ms; previously vite, 7570ms), so the arm needs more samples before the skew can be called WS-specific or repo-wide flake. Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * Re-run CI Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * Re-run CI Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * blank * blank --------- Co-authored-by: vercel[bot] <35613825+vercel[bot]@users.noreply.github.com> |
||
|
Nathan Colosimo
|
22349e95fd |
perf(core): load replay suffix in one request (#3205)
* perf(core): stream replay suffix in one request * perf(core): load replay suffix in one request Signed-off-by: Nathan Colosimo <110621881+NathanColosimo@users.noreply.github.com> * test(world-vercel): use streamed run start fixtures * refactor(events): simplify return-all plumbing Signed-off-by: Nathan Colosimo <110621881+NathanColosimo@users.noreply.github.com> * Return complete local run preloads * Document workflow event limit * fix: make return-all event loading resilient * Simplify full event listing * refactor(world-vercel): omit event limit for full loads * fix(world-vercel): explicitly request complete event logs --------- Signed-off-by: Nathan Colosimo <110621881+NathanColosimo@users.noreply.github.com> |
||
|
Nathan Colosimo
|
665110b3a2 |
perf(core): load replay log from run_started (#3191)
* perf(core): consume run_started replay page * fix: preserve turbo startup while streaming replay * refactor(world-vercel): simplify run start stream * chore(world-vercel): sort merged imports * refactor(world-vercel): require run start event page Signed-off-by: Nathan Colosimo <110621881+NathanColosimo@users.noreply.github.com> * refactor(world-vercel): require complete event pages Signed-off-by: Nathan Colosimo <110621881+NathanColosimo@users.noreply.github.com> * refactor(world-vercel): remove redundant event result type Signed-off-by: Nathan Colosimo <110621881+NathanColosimo@users.noreply.github.com> * refactor(world-vercel): narrow event stream state * Simplify run-started event consumption * Handle cross-region lifecycle event order * test(world-vercel): validate malformed event frame * Name run start result as an event stream * Simplify run-started stream types * Simplify optional event metadata * Validate complete event frames * Validate frame metadata * Validate events at frame boundary * Fix out-of-order run lifecycle replay --------- Signed-off-by: Nathan Colosimo <110621881+NathanColosimo@users.noreply.github.com> |
||
|
Nathan Colosimo
|
65139acfd7 |
perf(core): continue partial run_started preloads from cursor (#3124)
* perf(core): continue partial run preloads * refactor(core): simplify preload continuation * fix(core): preserve preload fallbacks * chore: rerun CI * fix(world): infer event create results * fix(core): preserve run state during setup * fix(world): enforce typed event results * refactor(world): rely on event result contract * refactor(core): unify replay event log state * refactor(core): make replay log states exact * fix(core): harden run start preload recovery * test(world-local): allow slow preload coverage * fix(core): preserve event result inference through recovery * refactor(core): simplify preload state transition Signed-off-by: Nathan Colosimo <110621881+NathanColosimo@users.noreply.github.com> * refactor(runtime): reuse event pages without duplicate reads * refactor(world-vercel): preserve opaque event payloads * Validate v4 event create responses * Validate v4 event frame metadata * Remove invalid v4 response identity check * Return validated v4 event bodies directly * Reuse event result entity types * Simplify event creation result types * Use concrete run creation result * Preserve generic event storage implementation * Validate v4 event responses without casts * Parse v4 event frames once * Reuse the default v4 event body schema * Simplify event preload state * Narrow event page result states * Preserve literal event result flags * Accept hook conflict event responses * Remove redundant optional event page schemas * Simplify preloaded event log access * Flatten replay event log state * Simplify replay event log state * Use one replay event log * fix(next): preserve edits made during full HMR rebuilds * chore(core): log dormant hook replays * fix(next): commit HMR snapshots after rebuilds * fix(next): ignore duplicate HMR file events * test(next): expect deduplicated HMR removal event * fix(next): distinguish duplicate HMR notifications * fix(next): ignore HMR notifications without source changes * chore: move Next HMR fix to separate PR * fix(core): complete partial preloads before QuickJS replay --------- Signed-off-by: Nathan Colosimo <110621881+NathanColosimo@users.noreply.github.com> |
||
|
Nathan Colosimo
|
74dbf81d32 |
fix(core): retry replay timeouts without exiting (#3385)
* fix(core): retry replay timeouts without exiting * refactor(world-postgres): leave existing retry limits unchanged * test(world-postgres): remove mocked migration assertion * chore: consolidate replay retry changesets |
||
|
Nathan Colosimo
|
4bb86d3054 |
feat(world-vercel): support Hook minimum retention (#3286)
* feat(world-vercel): support Hook minimum retention * fix(core): fail deterministic Hook validation |
||
|
Peter Wielander
|
a8db185c3b | [core] Fold events.create deltas into the replay log (#3382) | ||
|
Nathan Rajlich
|
eb9e13fd23 |
QuickJS engine: baseline-snapshot startup optimization (~25× faster VM boot) (#3342)
* Add opt-in QuickJS WASM VM engine (WORKFLOW_VM=quickjs) with full event replay * QuickJS engine: AbortController, setAttributes, terminal drain, turbo-safe requeue, stable PRNG seed * QuickJS engine: hook.getConflict support, cross-run writable forwarding symbols * QuickJS engine: stream framing round-trip, bound step proxies, webhook fidelity * Apply biome fixes to QuickJS engine files * Address review feedback: anchor source-map strip to end-of-input, use getWorkflowQueueName for conflict requeue, Buffer-free asset decoding, function replacers for payload injection, maxEventsLimit guard * CI: include generated QuickJS source assets in shared e2e build artifacts * Fix same-token hook ordering and conflicted-hook disposal in the QuickJS engine * CI: run both VM engines across all frameworks and worlds; label jobs with the engine * Fix stack overflow stripping inline source maps from webpack dev bundles; harden step-listing e2e assertions against eventually-consistent reads * e2e: poll step listings until analytics rows include attempt (optional column can lag terminal status) * e2e: use --withData to force storage-backed step listings for attempt assertions (analytics listing can omit attempt entirely) * Sort imports in QuickJS serialization files (biome organizeImports) * QuickJS engine: resolve the run's full payload-key capability so sealed (encp) hook payloads open Main's sealed-box work (#3096) makes cross-deployment resumeHook() seal hook payloads to the target run's published X25519 public key. The shared start() path publishes that key regardless of engine, so QuickJS runs receive sealed payloads too — but the QuickJS entrypoint resolved only the bare symmetric key via importKey(), which cannot open encp envelopes. The first sealed hook payload wedged the run right after hook_received, timing out every hook/webhook e2e on Vercel prod (node:vm legs were fine — the node engine resolves the full capability via memoizeEncryptionKey). Resolve deriveRunPayloadKeys() in the entrypoint instead and widen the runtime's key types from CryptoKey to DecryptionKey. Writes stay symmetric (encrypt() with RunPayloadKeys takes the encr path). Regression test seals a payload exactly as resumeHook does and round-trips it through the VM. * Address review: crypto/process parity, loud Intl guards, lazy engine import, VM-leak guard, telemetry namespace, eval-string escaping - Deterministic crypto.getRandomValues/randomUUID in the VM bootstrap, drawing from the seeded Math.random (identical sequences to the node engine's vm/index.ts implementations); all crypto.subtle methods throw with step-function guidance. process.env exposed as a frozen copy, matching node. - Intl: throwing constructors (no ICU in QuickJS), and toLocale*-family methods (incl. localeCompare) throw when given an explicit locale so cross-engine divergence is loud instead of silently writing different values into the event log. No-argument forms keep working. - runtime.ts lazy-imports the QuickJS entrypoint at dispatch, keeping the ~1.3MB embedded WASM assets out of node-engine deployments. - runQuickJSWorkflow wraps the per-run phase so an exceptional exit disposes the VM instead of leaking it in a reused compute instance; corrected the misleading fail-loud comment (run_failed, not retry); warn when the event drain loop exhausts its iteration bound. - Telemetry attributes renamed quickjs.* → workflow.vm.* to stay in the file's workflow.* namespace. - Eval-string correlation-id interpolation uses JSON.stringify instead of quote-only escaping. - common-vm.test.ts pins the reducer/reviver superset invariant against common.ts so the duplicated sets can't silently drift. - Docs enumerate the remaining global-surface differences (subtle.digest, Intl, WebAssembly, Atomics); quickjs-entrypoint documents the known precondition-guard gap. * QuickJS engine: implement resilient resumeHook (hookInput materialization + resumeId dedup) #1834 made resumeHook() fall back to enqueueing the run with a hookInput payload when the direct hook_received write fails transiently, with the runtime materializing the missing event on delivery. Only the node:vm path implemented it — the QuickJS dispatch returned before the node block, so the resilient payload was silently dropped and the new e2e timed out on every quickjs leg. - runtime.ts threads hookInput into runWorkflowWithQuickJS; the entrypoint materializes the missing hook_received after loading the event log (resumeId-keyed dedup, occurredAt from the resumeId ULID, local eventData substitution for lazy/ref responses, EntityConflict / HookNotFound handling) — mirroring the node block. - processEvents drops duplicate hook_received rows sharing a resumeId (first-in-log wins), matching the node engine's EventsConsumer dedup; the seen-set lives in the VM heap so it is deterministic per replay. Verified against the dev server with WORKFLOW_VM=quickjs: the resilient resume e2e passes and the materialization is observable in the logs; all 27 hook e2e tests green. * QuickJS engine: inline step execution via live-VM continuation loop + WASM module caching * Address review: exclusive inline step claims, self-write requeue, in-loop event ceiling - Inline steps now claim via a lazy step_started carrying the input (step_created deferred, atomic create-claim in the world), with ownerMessageId stamped and authoritativeAttempt=1 — a concurrent invocation racing on the same fresh step loses with EntityConflictError and skips instead of both bare-starting the step and double-running the body. This also removes the stepsCreatedByUs set, whose 'created by us' invariant didn't survive the swallowed create-race conflict; redelivery backstops now key on hasCreatedEvent. - dispatchPendingOps' createdAttributeEvent/createdGetConflictHook signals are consumed again: when the loop exits suspended without ever reading back a self-written attr_set / getConflict hook_created (eventually-consistent listing lag), the entrypoint requeues immediately instead of parking the run awaiting_external with its unblocking event already written. - The server-supplied event ceiling is re-checked at the top of every continuation-loop turn (seenEventIds.size), so a single invocation fanning out inline can no longer grow the log arbitrarily past the operator's limit. The quickjs dispatch in runtime.ts converts MaxEventsExceededError into run_failed / MAX_EVENTS_EXCEEDED — the guard's throw previously nacked forever, parking runaway runs in 'running'. - Documented the deliberate decision that the platform function timeout is the only bound on inline chaining (budget parked per batch), matching the node engine. * QuickJS engine: host-side, side-effect-free serialization via handles (Re-applied onto the review-fixed base; original commits da2723016 + 9814ed9ac squashed.) Replace the in-VM serde bundle with a host-side codec (runtime/quickjs-serde.ts) built on quickjs-wasi 3.3's introspection primitives and devalue 5.9's pluggable stringify/parse operations — mirroring the node:vm engine's architecture. Review fixes incorporated: - reducer/reviver key sets are pinned against codec-devalue-vm's workflow mode by exhaustiveness tests (exact order for reducers — first match wins), so the handle-space codec can't silently drift from the shared value-space sets. - the devalue entry in minimumReleaseAgeExclude is removed: the exact version is pinned via the workspace catalog + lockfile, so the cooldown waiver was unnecessary (verified with both frozen and regular installs). - eval-string interpolation inherits the JSON.stringify(cid) hardening from the base branch. * Address review: NUL-safe string extraction, deterministic retryAfter, pass-scoped handle disposal, byte-cache lifecycle - NUL (U+0000) safety across the WASM boundary: handle.toString() routes through JS_ToCString and silently truncates at the first NUL, and the C-string key APIs mangle NUL-bearing property keys (drop or collide). guestString() detects truncation by comparing against the handle's true guest length and recovers via in-VM JSON.stringify escaping; shapeOf verifies its fast host-string key list against a guest Object.keys count (+ duplicate check) and re-extracts through key handles on mismatch; get/hasOwn route NUL-bearing keys through length-aware guest string handles. All string funnels (primitives, symbol descriptions, error fields via chained/own reads, Headers entries, RegExp source/flags, URL href) go through guestString. Regression-tested down to the truncate-vs-collide enumeration shapes; fixes nullByteWorkflow on the quickjs e2e legs. - RetryableError's absent/invalid retryAfter fallback now reads the GUEST clock (the deterministic replay clock at the WASI layer) via a captured Date.now instead of the host wall clock — the in-VM reducer was replay-stable by construction and the host port silently lost that. - Pass-scoped handle disposal: serialize/deserialize sweep every intermediate handle their pass creates (call/invoke results, descriptor reads, dups, parse-op constructions), closing the ~one-leaked-handle-per-value-node growth across long-lived inline sessions. Implemented with module-owned tracking rather than vm.withScope: the library scope also captures the handles the host-callback trampoline wraps around C-owned argv pointers, and disposing those (Map/Set/Headers forEach visitors run mid-pass) double-frees guest values — observed as WASM memory corruption. identities is cleared per pass so freed-pointer reuse cannot alias entries across passes. - Byte-cache lifecycle: terminal drain now shares the per-VM cache with the suspension path (re-serializing an op at drain could re-invoke getters and produce different bytes for what the log treats as one value), and entries for settled ops — which neither collection filter can match again — are evicted, bounding the cache by the live pending set. * Adopt quickjs-wasi 3.3.1: withScope handle sweeping, real memoryLimit accounting, loud unregistered-callback failures 3.3.1 ships the three fixes this branch surfaced upstream: - Borrowed host-callback handles (vercel-labs/quickjs-wasi#31): the trampoline's this/argv handles are scope-exempt, making vm.withScope safe around host callbacks. The serde's module-owned pass-disposal apparatus (passDisposal/track/runWithPassDisposal and ~18 track() wraps) is replaced by withScope in serialize/deserialize — simpler, and strictly more complete: every handle constructed during the pass is swept, not just the ones our creation funnels saw. Bench parity confirmed (within ~10% on the 50k-node extreme case, unchanged elsewhere; still 2.6-100x over the in-VM codec). - Real memoryLimit accounting (vercel-labs/quickjs-wasi#33): the engine's 256 MB VM ceiling now actually bounds retained guest allocations (usable-size was 0 on wasm32-wasi before, so the limit never accumulated). - Unregistered host callbacks throw (vercel-labs/quickjs-wasi#34): guest calls into missing callbacks fail loud instead of silently returning undefined — protection this engine wants for snapshot-restore re-registration bugs. Also merges origin/main (undici 7.29.0). * QuickJS engine: baseline-snapshot startup optimization Evaluating the workflow bundle dominates VM startup (~74ms of a ~77ms boot for the 1.3MB e2e bundle) and full event replay pays it on EVERY invocation — a large share of the quickjs engine's TTFS gap vs node:vm, where V8 compiles the same script in single-digit ms. The bundle is identical across all runs of a deployment, so the engine now hydrates one VM per function instance (bootstrap + bundle eval), snapshots its memory, and starts every invocation with QuickJS.restore (~3ms) instead of re-evaluating. Measured on the real generated e2e flow bundle (154 workflows), boot to first suspension: fresh 79.4ms -> restored 3.2ms (24.8x). First invocation pays hydrate+restore (85.8ms, ~= one fresh boot); every subsequent invocation — including every replay wake — gets the discount. Determinism: replay requires module-scope user code to observe the run-seeded PRNG and deterministic clock, and a restored heap carries whatever module scope computed at hydrate time. The hydrate therefore runs with draw-counting placeholder host fns and a read-counting clock; a bundle that consumed either is marked ineligible and every invocation falls back to fresh evaluation (node:vm-parity semantics preserved exactly). When the gate passes, restore is byte-equivalent to fresh eval: the per-run host fns (random / __generateNanoid / __generateUlid) re-register by NAME on the restored VM before the workflow body runs, so the seeded draw sequence — and every correlationId — is identical. Pinned by a parity test that feeds Math.random() into a step input and byte-compares the serialized ops across fresh, first-restore and cached-restore invocations. Cache: per function instance, keyed on the bundle string (reference-stable in generated flow routes), promise-deduped for concurrent first invocations, capped at 4 entries; hydrate rejections evict for retry while eval failures cache as ineligible (the fresh path re-evaluates and surfaces the real, source-mapped error). Kill switch: WORKFLOW_QUICKJS_BASELINE_SNAPSHOT=0. * Address review: intrinsics-replacement gate, hydrate-failure fallback, shared clock helper, review nits - Serialization-intrinsics gate (the substantive finding): the restore path's serde captures intrinsics from the restored heap — AFTER module scope ran — while the fresh path captures before user code. A bundle that replaced a captured intrinsic at module scope (e.g. a Date.prototype.toISOString polyfill) without touching PRNG/clock passed the eligibility gate yet would serialize differently on the two paths. captureIntrinsicsSignature (exported from quickjs-serde) identity-fingerprints every to-be-captured value; the hydrate compares it before and after bundle eval and marks any replacement ineligible. Expression-created entries (makeSparseArray, makeThunk, hasOwnCall) are excluded — they get fresh identities per eval and cannot be replaced by user code. Gate test added with a toISOString polyfill. - Hydrate-failure fallback: a getBaselineEntry rejection (infrastructure — vm.snapshot() under memory pressure, QuickJS.create failing) no longer fails the invocation; it logs a warning and falls back to fresh evaluation, with the cached promise already evicted for retry. - initWorkflowVM now uses the shared makeDeterministicClockWasi helper its doc claimed it shared, so the two clock implementations cannot drift. - getCompiledAssets() awaited once per call site (hydrate + restore). - WORKFLOW_TURBO JSDoc reattached to isTurboEnabled (the baseline constant had been inserted between doc and function). - Parity test saves/restores any pre-existing WORKFLOW_QUICKJS_BASELINE_SNAPSHOT env value instead of deleting it. * Fix source-map remapping for workflows sharing a baseline snapshot The baseline cache is keyed on the bundle, which every workflow in a deployment shares — but the hydrate evaluated the bundle with the FIRST caller's workflowId as the eval filename. That name is baked into the snapshot's compiled code, so on the restore path every OTHER workflow's stack frames referenced the first hydrator's id, and remapErrorStack (which matches frames by the failing run's module specifier) never matched them — raw bundle line numbers leaked into user-visible stacks for any workflow outside the first hydrator's module. Hydrate now evaluates under a workflow-independent constant (BASELINE_BUNDLE_FILENAME), and the entrypoint's three remap sites (failed-branch stack, hydrated error, cause chain) remap against BOTH filename spaces — the run's module specifier covers fresh-path frames, the constant covers snapshot-path frames; remapErrorStack early-exits on a cheap includes() for whichever space has no frames. Regression test: a two-module bundle hydrated under workflow A, with workflow B failing through the restored snapshot — B's stack must reference the constant filename and not A's id. * Address review: lossless lone-surrogate string extraction, portable base64 P1 — the guestString length check was insufficient: JS_ToCString has TWO corruptions (NUL truncation, lone-surrogate -> U+FFFD replacement) and they can cancel — the replacement expansion offsets the truncation so the extracted length matches the true guest length. A bare lone surrogate can also replace 1:1 with no length change at all. Worse, the JSON.stringify slow path was itself lossy for lone surrogates: QuickJS passes them through raw, and the C-string extraction of ITS output corrupts them. - guestString accepts the fast value only when length matches AND it contains no U+FFFD (legitimate U+FFFD strings take the loss-free slow path); the slow path now escapes INSIDE the VM to printable ASCII via a new captured escapeString intrinsic (WTF-16-safe per-code-unit \uXXXX escaping), then JSON-parses host-side. - shapeOf's fast-key acceptance adds a U+FFFD scan alongside the count/duplicate checks (lone-surrogate keys corrupt with count and uniqueness intact). - get()/hasOwn() route keys through guest string handles when they carry a NUL or an UNPAIRED surrogate (paired surrogates - emoji keys - encode fine through the C-string APIs; vm.newString is verified WTF-16-preserving for the handle path). - Tests: the reviewer's exact length-canceling case, bare lone surrogates, legit-U+FFFD passthrough, byte parity with the reference codec, and lone-surrogate/mixed keys. P2 — the codec's base64 helpers no longer carry an unconditional Node Buffer dependency: feature-detected Uint8Array.fromBase64/toBase64 when available, Buffer when present, btoa/atob loop otherwise — keeping WASM-only/non-Node hosts (Cloudflare Workers) viable. * Merge quickjs-host-serde (lossless surrogate extraction, portable base64) into quickjs-baseline-snapshot The new escapeString captured intrinsic is expression-created (a fresh guest closure per capture eval), so it joins makeSparseArray/makeThunk/ hasOwnCall in captureIntrinsicsSignature's exclusion list — without this the baseline hydrate gate would classify every bundle ineligible (the byte-parity test catches exactly that, as it did when hasOwnCall was missed). * Address review: pre-eval serde capture root, adopted by pointer from the snapshot The intrinsics-replacement gate was structurally losing: its own post-eval probe executed guest-reachable code (CAPTURE_INTRINSICS calls Object.getOwnPropertyDescriptor / Object.getPrototypeOf), those dependencies were not in the identity signature, and a module-scope stateful wrapper around them both evaded detection AND had its side effects baked into the snapshot — fresh returned 0 from the reviewer's counter repro while restore returned the probe's call count. Replace detection with prevention: ALL guest-touching serde initialization (intrinsics capture, branded samples, well-known symbol lookups) is bundled into one CAPTURE_ROOT expression evaluated in the baseline VM BEFORE the bundle — the same capture-before-user-code ordering the fresh path has always had. The container handle's box lives in the snapshot's linear memory, its raw pointer rides the BaselineEntry, and every restored VM re-adopts it (adoptSerdeRoot) — serde init then performs only plain-data property reads and C-level classId reads: NO guest code executes after user code has run, on either path. Consequences: - the identity-signature gate and its expression-created skip-list are deleted (nothing to detect — module-scope intrinsic patching is now HARMLESS on the snapshot path, not merely detectable) - polyfill bundles become ELIGIBLE for the optimization and serialize through pristine intrinsics identically on both paths (test flipped from gating to byte-equality) - process.env injection converted from guest-source eval to handle-based installProcessEnv (captured Object.freeze + vm.hostToHandle): the old evalCode ran JSON.parse post-eval on the restore path only, the same observable-divergence class - the reviewer's stateful-wrapper repro is a regression test: the counter must be zero and identical across fresh and restored invocations * Adopt quickjs-wasi 3.4.0: delete the NUL/surrogate string machinery 3.4.0 ships lossless string transport (vercel-labs/quickjs-wasi#35 — found by this PR's review cycle), so the SDK-side detection and escape machinery is deleted wholesale: - guestString (length + U+FFFD detection, in-VM escape fallback) — plain toString() is lossless now - the escapeString / hasOwnCall / jsonStringify / objectKeys captured intrinsics - keyNeedsHandleLookup and the handle-keyed get/hasOwn routing — the library routes inexpressible keys itself - shapeOf's guest-key verification pass (count/duplicate/U+FFFD scans) — enumeration is lossless Net ~130 lines and four captured intrinsics removed; the serde now uses the plain quickjs-wasi surface everywhere. Test honesty fix that 3.4.0 forced: the earlier lone-surrogate round-trip tests passed only via mutual corruption — the pre-3.4.0 lossy host→guest transport corrupted the guest comparison literals identically to the wire. With an honest transport they exposed that the WIRE itself (devalue emits lone surrogates raw; the wire is UTF-8) degrades lone surrogates to U+FFFD — in the node engine's reference codec exactly as here, verified. Bug-compatible parity is the load-bearing property (event logs replay across engines), so those tests now assert byte parity with the reference codec plus guest-observed equality with the reference codec's own round trip; NULs are devalue-escaped and asserted to survive exactly. Wire-level surrogate preservation is a product-wide devalue/UTF-8 question, tracked separately from this engine. * rerun CI |
||
|
Nathan Rajlich
|
19b5b85c8b |
QuickJS engine: host-side, side-effect-free serialization via handles (#3263)
* Add opt-in QuickJS WASM VM engine (WORKFLOW_VM=quickjs) with full event replay * QuickJS engine: AbortController, setAttributes, terminal drain, turbo-safe requeue, stable PRNG seed * QuickJS engine: hook.getConflict support, cross-run writable forwarding symbols * QuickJS engine: stream framing round-trip, bound step proxies, webhook fidelity * Apply biome fixes to QuickJS engine files * Address review feedback: anchor source-map strip to end-of-input, use getWorkflowQueueName for conflict requeue, Buffer-free asset decoding, function replacers for payload injection, maxEventsLimit guard * CI: include generated QuickJS source assets in shared e2e build artifacts * Fix same-token hook ordering and conflicted-hook disposal in the QuickJS engine * CI: run both VM engines across all frameworks and worlds; label jobs with the engine * Fix stack overflow stripping inline source maps from webpack dev bundles; harden step-listing e2e assertions against eventually-consistent reads * e2e: poll step listings until analytics rows include attempt (optional column can lag terminal status) * e2e: use --withData to force storage-backed step listings for attempt assertions (analytics listing can omit attempt entirely) * Sort imports in QuickJS serialization files (biome organizeImports) * QuickJS engine: resolve the run's full payload-key capability so sealed (encp) hook payloads open Main's sealed-box work (#3096) makes cross-deployment resumeHook() seal hook payloads to the target run's published X25519 public key. The shared start() path publishes that key regardless of engine, so QuickJS runs receive sealed payloads too — but the QuickJS entrypoint resolved only the bare symmetric key via importKey(), which cannot open encp envelopes. The first sealed hook payload wedged the run right after hook_received, timing out every hook/webhook e2e on Vercel prod (node:vm legs were fine — the node engine resolves the full capability via memoizeEncryptionKey). Resolve deriveRunPayloadKeys() in the entrypoint instead and widen the runtime's key types from CryptoKey to DecryptionKey. Writes stay symmetric (encrypt() with RunPayloadKeys takes the encr path). Regression test seals a payload exactly as resumeHook does and round-trips it through the VM. * Address review: crypto/process parity, loud Intl guards, lazy engine import, VM-leak guard, telemetry namespace, eval-string escaping - Deterministic crypto.getRandomValues/randomUUID in the VM bootstrap, drawing from the seeded Math.random (identical sequences to the node engine's vm/index.ts implementations); all crypto.subtle methods throw with step-function guidance. process.env exposed as a frozen copy, matching node. - Intl: throwing constructors (no ICU in QuickJS), and toLocale*-family methods (incl. localeCompare) throw when given an explicit locale so cross-engine divergence is loud instead of silently writing different values into the event log. No-argument forms keep working. - runtime.ts lazy-imports the QuickJS entrypoint at dispatch, keeping the ~1.3MB embedded WASM assets out of node-engine deployments. - runQuickJSWorkflow wraps the per-run phase so an exceptional exit disposes the VM instead of leaking it in a reused compute instance; corrected the misleading fail-loud comment (run_failed, not retry); warn when the event drain loop exhausts its iteration bound. - Telemetry attributes renamed quickjs.* → workflow.vm.* to stay in the file's workflow.* namespace. - Eval-string correlation-id interpolation uses JSON.stringify instead of quote-only escaping. - common-vm.test.ts pins the reducer/reviver superset invariant against common.ts so the duplicated sets can't silently drift. - Docs enumerate the remaining global-surface differences (subtle.digest, Intl, WebAssembly, Atomics); quickjs-entrypoint documents the known precondition-guard gap. * QuickJS engine: implement resilient resumeHook (hookInput materialization + resumeId dedup) #1834 made resumeHook() fall back to enqueueing the run with a hookInput payload when the direct hook_received write fails transiently, with the runtime materializing the missing event on delivery. Only the node:vm path implemented it — the QuickJS dispatch returned before the node block, so the resilient payload was silently dropped and the new e2e timed out on every quickjs leg. - runtime.ts threads hookInput into runWorkflowWithQuickJS; the entrypoint materializes the missing hook_received after loading the event log (resumeId-keyed dedup, occurredAt from the resumeId ULID, local eventData substitution for lazy/ref responses, EntityConflict / HookNotFound handling) — mirroring the node block. - processEvents drops duplicate hook_received rows sharing a resumeId (first-in-log wins), matching the node engine's EventsConsumer dedup; the seen-set lives in the VM heap so it is deterministic per replay. Verified against the dev server with WORKFLOW_VM=quickjs: the resilient resume e2e passes and the materialization is observable in the logs; all 27 hook e2e tests green. * QuickJS engine: inline step execution via live-VM continuation loop + WASM module caching * Address review: exclusive inline step claims, self-write requeue, in-loop event ceiling - Inline steps now claim via a lazy step_started carrying the input (step_created deferred, atomic create-claim in the world), with ownerMessageId stamped and authoritativeAttempt=1 — a concurrent invocation racing on the same fresh step loses with EntityConflictError and skips instead of both bare-starting the step and double-running the body. This also removes the stepsCreatedByUs set, whose 'created by us' invariant didn't survive the swallowed create-race conflict; redelivery backstops now key on hasCreatedEvent. - dispatchPendingOps' createdAttributeEvent/createdGetConflictHook signals are consumed again: when the loop exits suspended without ever reading back a self-written attr_set / getConflict hook_created (eventually-consistent listing lag), the entrypoint requeues immediately instead of parking the run awaiting_external with its unblocking event already written. - The server-supplied event ceiling is re-checked at the top of every continuation-loop turn (seenEventIds.size), so a single invocation fanning out inline can no longer grow the log arbitrarily past the operator's limit. The quickjs dispatch in runtime.ts converts MaxEventsExceededError into run_failed / MAX_EVENTS_EXCEEDED — the guard's throw previously nacked forever, parking runaway runs in 'running'. - Documented the deliberate decision that the platform function timeout is the only bound on inline chaining (budget parked per batch), matching the node engine. * QuickJS engine: host-side, side-effect-free serialization via handles (Re-applied onto the review-fixed base; original commits da2723016 + 9814ed9ac squashed.) Replace the in-VM serde bundle with a host-side codec (runtime/quickjs-serde.ts) built on quickjs-wasi 3.3's introspection primitives and devalue 5.9's pluggable stringify/parse operations — mirroring the node:vm engine's architecture. Review fixes incorporated: - reducer/reviver key sets are pinned against codec-devalue-vm's workflow mode by exhaustiveness tests (exact order for reducers — first match wins), so the handle-space codec can't silently drift from the shared value-space sets. - the devalue entry in minimumReleaseAgeExclude is removed: the exact version is pinned via the workspace catalog + lockfile, so the cooldown waiver was unnecessary (verified with both frozen and regular installs). - eval-string interpolation inherits the JSON.stringify(cid) hardening from the base branch. * Address review: NUL-safe string extraction, deterministic retryAfter, pass-scoped handle disposal, byte-cache lifecycle - NUL (U+0000) safety across the WASM boundary: handle.toString() routes through JS_ToCString and silently truncates at the first NUL, and the C-string key APIs mangle NUL-bearing property keys (drop or collide). guestString() detects truncation by comparing against the handle's true guest length and recovers via in-VM JSON.stringify escaping; shapeOf verifies its fast host-string key list against a guest Object.keys count (+ duplicate check) and re-extracts through key handles on mismatch; get/hasOwn route NUL-bearing keys through length-aware guest string handles. All string funnels (primitives, symbol descriptions, error fields via chained/own reads, Headers entries, RegExp source/flags, URL href) go through guestString. Regression-tested down to the truncate-vs-collide enumeration shapes; fixes nullByteWorkflow on the quickjs e2e legs. - RetryableError's absent/invalid retryAfter fallback now reads the GUEST clock (the deterministic replay clock at the WASI layer) via a captured Date.now instead of the host wall clock — the in-VM reducer was replay-stable by construction and the host port silently lost that. - Pass-scoped handle disposal: serialize/deserialize sweep every intermediate handle their pass creates (call/invoke results, descriptor reads, dups, parse-op constructions), closing the ~one-leaked-handle-per-value-node growth across long-lived inline sessions. Implemented with module-owned tracking rather than vm.withScope: the library scope also captures the handles the host-callback trampoline wraps around C-owned argv pointers, and disposing those (Map/Set/Headers forEach visitors run mid-pass) double-frees guest values — observed as WASM memory corruption. identities is cleared per pass so freed-pointer reuse cannot alias entries across passes. - Byte-cache lifecycle: terminal drain now shares the per-VM cache with the suspension path (re-serializing an op at drain could re-invoke getters and produce different bytes for what the log treats as one value), and entries for settled ops — which neither collection filter can match again — are evicted, bounding the cache by the live pending set. * Adopt quickjs-wasi 3.3.1: withScope handle sweeping, real memoryLimit accounting, loud unregistered-callback failures 3.3.1 ships the three fixes this branch surfaced upstream: - Borrowed host-callback handles (vercel-labs/quickjs-wasi#31): the trampoline's this/argv handles are scope-exempt, making vm.withScope safe around host callbacks. The serde's module-owned pass-disposal apparatus (passDisposal/track/runWithPassDisposal and ~18 track() wraps) is replaced by withScope in serialize/deserialize — simpler, and strictly more complete: every handle constructed during the pass is swept, not just the ones our creation funnels saw. Bench parity confirmed (within ~10% on the 50k-node extreme case, unchanged elsewhere; still 2.6-100x over the in-VM codec). - Real memoryLimit accounting (vercel-labs/quickjs-wasi#33): the engine's 256 MB VM ceiling now actually bounds retained guest allocations (usable-size was 0 on wasm32-wasi before, so the limit never accumulated). - Unregistered host callbacks throw (vercel-labs/quickjs-wasi#34): guest calls into missing callbacks fail loud instead of silently returning undefined — protection this engine wants for snapshot-restore re-registration bugs. Also merges origin/main (undici 7.29.0). * Address review: lossless lone-surrogate string extraction, portable base64 P1 — the guestString length check was insufficient: JS_ToCString has TWO corruptions (NUL truncation, lone-surrogate -> U+FFFD replacement) and they can cancel — the replacement expansion offsets the truncation so the extracted length matches the true guest length. A bare lone surrogate can also replace 1:1 with no length change at all. Worse, the JSON.stringify slow path was itself lossy for lone surrogates: QuickJS passes them through raw, and the C-string extraction of ITS output corrupts them. - guestString accepts the fast value only when length matches AND it contains no U+FFFD (legitimate U+FFFD strings take the loss-free slow path); the slow path now escapes INSIDE the VM to printable ASCII via a new captured escapeString intrinsic (WTF-16-safe per-code-unit \uXXXX escaping), then JSON-parses host-side. - shapeOf's fast-key acceptance adds a U+FFFD scan alongside the count/duplicate checks (lone-surrogate keys corrupt with count and uniqueness intact). - get()/hasOwn() route keys through guest string handles when they carry a NUL or an UNPAIRED surrogate (paired surrogates - emoji keys - encode fine through the C-string APIs; vm.newString is verified WTF-16-preserving for the handle path). - Tests: the reviewer's exact length-canceling case, bare lone surrogates, legit-U+FFFD passthrough, byte parity with the reference codec, and lone-surrogate/mixed keys. P2 — the codec's base64 helpers no longer carry an unconditional Node Buffer dependency: feature-detected Uint8Array.fromBase64/toBase64 when available, Buffer when present, btoa/atob loop otherwise — keeping WASM-only/non-Node hosts (Cloudflare Workers) viable. * Adopt quickjs-wasi 3.4.0: delete the NUL/surrogate string machinery 3.4.0 ships lossless string transport (vercel-labs/quickjs-wasi#35 — found by this PR's review cycle), so the SDK-side detection and escape machinery is deleted wholesale: - guestString (length + U+FFFD detection, in-VM escape fallback) — plain toString() is lossless now - the escapeString / hasOwnCall / jsonStringify / objectKeys captured intrinsics - keyNeedsHandleLookup and the handle-keyed get/hasOwn routing — the library routes inexpressible keys itself - shapeOf's guest-key verification pass (count/duplicate/U+FFFD scans) — enumeration is lossless Net ~130 lines and four captured intrinsics removed; the serde now uses the plain quickjs-wasi surface everywhere. Test honesty fix that 3.4.0 forced: the earlier lone-surrogate round-trip tests passed only via mutual corruption — the pre-3.4.0 lossy host→guest transport corrupted the guest comparison literals identically to the wire. With an honest transport they exposed that the WIRE itself (devalue emits lone surrogates raw; the wire is UTF-8) degrades lone surrogates to U+FFFD — in the node engine's reference codec exactly as here, verified. Bug-compatible parity is the load-bearing property (event logs replay across engines), so those tests now assert byte parity with the reference codec plus guest-observed equality with the reference codec's own round trip; NULs are devalue-escaped and asserted to survive exactly. Wire-level surrogate preservation is a product-wide devalue/UTF-8 question, tracked separately from this engine. |
||
|
Karthik Kalyan
|
439a495a71 |
fix(core): pre-check deployment affinity before the lazy resume write (#3374)
The lazy hook fast path (#3345) hoisted the consumer's hook_received write above the deployment-affinity guard (#2960), so a misrouted lazy resume wrote its event before the guard could re-route the delivery. Stamp the run's pinned deployment on the resume message (hookInput.deploymentId, from the producer's resume context) and, on the consumer, compare it against the ambient deployment id immediately before the fast path: a match continues with no run fetch, a mismatch fetches the authoritative run and hands it to the existing guard — which keeps sole ownership of re-route/fail policy and remains the authoritative protection before replay and step execution. The re-routed message preserves the complete hookInput (it may hold the only copy of the resume payload). Older messages without the field, and worlds without deployment affinity, are unchanged: they skip the pre-check and rely on the authoritative guard, the pre-guard write staying convergent per (runId, resumeId). Fixes the misrouted-lazy-resume unit test broken by the #2960/#3345 ordering: a modern misrouted resume now re-routes with zero event writes, asserted for both hook_received and run_started. Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |
||
|
Peter Wielander
|
bf4dda6478 | [world-vercel] Recover from wedged HTTP/2 events connections (#3370) | ||
|
github-actions[bot]
|
e6af70b9d9 |
Version Packages (beta) (#3318)
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>workflow@5.0.0-beta.40 |
||
|
Karthik Kalyan
|
9c1b3c8638 |
perf(core): initialize lazy hook replay from hook_received stream (#3345)
* perf(core): initialize lazy hook replay from hook_received stream On a lazy hook queue delivery, the consumer's idempotent hook_received re-ensure is hoisted above run_started and doubles as the invocation's setup request: it asks the World to return the current replay log with the write (new advisory CreateEventParams.preloadEvents), so one HTTP request yields the canonical event, the reconstructed run, and the complete replay log — skipping both the run_started POST and the initial events.list. - world: optional `preloadEvents?: true` on CreateEventParams, the hook_received dual of skipPreload; Worlds may ignore it - world-vercel: createHookReceivedPreloadEventV4 sends the frame Accept on eligible hook_received posts and decodes either response mode — frames via the response decoder extracted from the LIST consumer (GET behavior unchanged), CBOR via the shared materialized-response mapping. The run is reconstructed from run_created/run_started (plus attr_set folds), the canonical event found by x-wf-event-id, and resumeId now survives frame decoding so the runtime can match it - core: new fast path before the generic run-state setup, guarded on hookInput.resumeId + payloadDigest; a validated COMPLETE preload (hasMore false — this path has no cursor-continuation machinery) initializes workflowRun/preloadedEvents/maxEventsLimit directly, anything else falls back to the run_started setup without re-posting the hook; error classification matches the existing re-ensure (terminal → consume, transient → redeliver); setup source reported via workflow.resume_setup_source (never workflow.hook.resilient_resume_materialized, which stays a recovery-only signal) - producer resumeHook() is unchanged and never sets preloadEvents Based directly on main (no dependency on #3124/#3191); pairs with workflow-server's streamed hook_received replay-log response, which deploys first — the SDK negotiates per request and falls back safely against older servers. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * address review: lazy fallback, retryable resume, terminal telemetry - world-vercel: the preload request keeps hook_received's lazy remoteRefBehavior — a supporting server owns frame-body resolution, while an older server now answers the CBOR fallback without resolving an S3-backed payload the runtime would discard - world-vercel: the atomic lazy-resume shape (resumeId + digest) opts into withEventPostRetry via idempotentHookResume — the (runId, resumeId) claim makes the POST idempotent-on-retry; legacy/partial hook_received shapes stay single-attempt, definitive 4xx stays non-retryable (unit + adapter + trace-propagation coverage) - core: a terminal event found in the preload records workflow.resume_setup_source=hook_received_stream and the run's actual terminal status on the span before consuming the delivery - core: document resilient_resume_materialized as the legacy/non-atomic re-ensure signal (claim ownership is not observable client-side, so the hoisted path deliberately never emits it) and resume_setup_source as a latency signal, not proof of event creation; note the Option A skip is now unreachable for atomic resumes - world: spell out the full preload usability contract on preloadEvents (complete hasMore-false log, non-null cursor, run/startedAt/maxEvents, lifecycle events, matching resumeId, list ordering, read-after-write consistency); bump @workflow/world to minor - new QuickJS sourcing tests (VM mocked): an attested complete preload is used verbatim with no events.list, a non-attested hook-containing preload is refetched, and an attested empty preload is not trusted Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |
||
|
Mitul Shah
|
95e292e0c1 |
Refresh the Workflow SDK README (#3357)
* docs: refresh Workflow SDK README Signed-off-by: Cursor Agent <cursoragent@cursor.com> * docs: fix README deployment link Signed-off-by: Cursor Agent <cursoragent@cursor.com> * docs: restore bug bounty guidance Signed-off-by: Cursor Agent <cursoragent@cursor.com> * docs: tighten README copy Signed-off-by: Cursor Agent <cursoragent@cursor.com> * docs: clarify workflow suspension copy Signed-off-by: Cursor Agent <cursoragent@cursor.com> * docs: add code of conduct Signed-off-by: Cursor Agent <cursoragent@cursor.com> * docs: remove README workflow example Signed-off-by: Cursor Agent <cursoragent@cursor.com> * Apply suggestions from code review Co-authored-by: vercel[bot] <35613825+vercel[bot]@users.noreply.github.com> Signed-off-by: Mitul Shah <mitulxshah@gmail.com> * docs: restore security disclosure wording Signed-off-by: Cursor Agent <cursoragent@cursor.com> * Apply suggestions from code review Co-authored-by: Mitul Shah <mitulxshah@gmail.com> Signed-off-by: Mitul Shah <mitulxshah@gmail.com> --------- Signed-off-by: Cursor Agent <cursoragent@cursor.com> Signed-off-by: Mitul Shah <mitulxshah@gmail.com> Co-authored-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: vercel[bot] <35613825+vercel[bot]@users.noreply.github.com> |
||
|
Karthik Kalyan
|
371f06e5ac |
feat(web): bulk-cancel selected runs from the runs table (#3349)
* feat(cli): bulk-cancel runs in a single operation Replace the per-run cancel loop in `workflow cancel` with one `cancelRuns` call, validate `--limit` (1-500), print a compact outcome summary with per-run lines for surfaced failures, and exit nonzero only when a run fails. The bulk logic lives in a dependency-injected `performBulkCancel` helper so it is unit-testable without an oclif harness. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(cli): address bulk cancel review feedback * feat(web): bulk-cancel selected runs in a single request Thread a bulkCancelRuns action through the server action, RPC route, rpc-client, and client wrappers, backed by core's cancelRuns. The runs table now cancels the selected pending/running runs in one call, caps a batch at BULK_CANCEL_MAX_RUN_IDS (disabling the button with guidance above the cap), and reports a single outcome-summary toast covering only the categories that occurred. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
Karthik Kalyan
|
2150798ca6 |
feat(cli): bulk-cancel runs in a single operation (#3348)
* feat(cli): bulk-cancel runs in a single operation Replace the per-run cancel loop in `workflow cancel` with one `cancelRuns` call, validate `--limit` (1-500), print a compact outcome summary with per-run lines for surfaced failures, and exit nonzero only when a run fails. The bulk logic lives in a dependency-injected `performBulkCancel` helper so it is unit-testable without an oclif harness. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(cli): address bulk cancel review feedback --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
Elliot Dauber
|
72efc90f28 |
Use runtime deadline for inline execution limit (#3360)
* Use runtime deadline for inline execution limit * up * lazy import * Update packages/world/src/interfaces.ts Co-authored-by: Peter Wielander <mittgfu@gmail.com> Signed-off-by: Elliot Dauber <67391073+elliotdauber@users.noreply.github.com> --------- Signed-off-by: Elliot Dauber <67391073+elliotdauber@users.noreply.github.com> Co-authored-by: Peter Wielander <mittgfu@gmail.com> |
||
|
Alex Langenfeld
|
79e4c04409 |
fix(core): re-route runs delivered to the wrong deployment (#2960)
## Summary & Motivation A queue callback that reaches a deployment other than the one its run is pinned to derives the per-run encryption key from the wrong master key, so the delivery fails before user code runs and the run dies as a blank "exceeded max retries". The delivery is re-enqueued explicitly addressed to the run's own deployment — strictly better-targeted than the send that misrouted — and the run is failed with the new `DEPLOYMENT_MISMATCH` error code only once `WORKFLOW_DEPLOYMENT_MISMATCH_MAX_RETRIES` (default 3) is spent. Gated on the new World capability `deploymentAffinity`, so worlds with synthetic or version-tagged deployment ids are unaffected. ## Test Plan Unit tests added for the guard and both runtime paths; local vitest and typechecks pass. |
||
|
Karthik Kalyan
|
8d479283ca |
feat(world,world-vercel,core): bulk run cancellation primitive (#3347)
* feat(world,world-vercel,core): bulk run cancellation primitive Add a bulk cancellation contract to @workflow/world (schemas, types, and an optional Storage['runs'].cancelMany method), implement it in @workflow/world-vercel via a single POST /v4/runs/cancel request, and add a cancelRuns runtime helper to @workflow/core that uses the world fast path when available and otherwise falls back to bounded-concurrency (max 20) single-run cancellation. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * Update packages/world/src/interfaces.ts Co-authored-by: Peter Wielander <mittgfu@gmail.com> Signed-off-by: Karthik Kalyan <105607645+karthikscale3@users.noreply.github.com> --------- Signed-off-by: Karthik Kalyan <105607645+karthikscale3@users.noreply.github.com> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> Co-authored-by: Peter Wielander <mittgfu@gmail.com> |
||
|
Nathan Colosimo
|
939ffb4f51 |
fix(core): reject unsupported Hook retention inside QuickJS (#3332)
* fix(core): reject unsupported Hook retention inside QuickJS * refactor(core): mirror World capabilities in QuickJS |
||
|
Peter Wielander
|
2eddf74cb6 | Send the run id on correlation-id event reads (#3334) | ||
|
Rich Haines
|
a3331ac0f6 |
docs: add inbound cross-links to orphaned v4 docs pages (#3355)
These pages had no inbound links from other docs pages' content (only sidebar/card navigation), so they were unreachable through prose. Adds one minimal cross-link each from a parent index or closely related page. |
||
|
Pranay Prakash
|
1222aab74d |
chore(deps): upgrade undici to 7.29.0 (#3315)
Signed-off-by: Pranay Prakash <pranay.gp@gmail.com> Signed-off-by: Nathan Colosimo <110621881+NathanColosimo@users.noreply.github.com> Co-authored-by: Claude Fable 5 <noreply@anthropic.com> Co-authored-by: Nathan Colosimo <110621881+NathanColosimo@users.noreply.github.com> |
||
|
Nathan Rajlich
|
a8bf8db84e |
QuickJS engine: inline step execution + WASM module caching (#3049)
* Add opt-in QuickJS WASM VM engine (WORKFLOW_VM=quickjs) with full event replay * QuickJS engine: AbortController, setAttributes, terminal drain, turbo-safe requeue, stable PRNG seed * QuickJS engine: hook.getConflict support, cross-run writable forwarding symbols * QuickJS engine: stream framing round-trip, bound step proxies, webhook fidelity * Apply biome fixes to QuickJS engine files * Address review feedback: anchor source-map strip to end-of-input, use getWorkflowQueueName for conflict requeue, Buffer-free asset decoding, function replacers for payload injection, maxEventsLimit guard * CI: include generated QuickJS source assets in shared e2e build artifacts * Fix same-token hook ordering and conflicted-hook disposal in the QuickJS engine * CI: run both VM engines across all frameworks and worlds; label jobs with the engine * Fix stack overflow stripping inline source maps from webpack dev bundles; harden step-listing e2e assertions against eventually-consistent reads * e2e: poll step listings until analytics rows include attempt (optional column can lag terminal status) * e2e: use --withData to force storage-backed step listings for attempt assertions (analytics listing can omit attempt entirely) * Sort imports in QuickJS serialization files (biome organizeImports) * QuickJS engine: resolve the run's full payload-key capability so sealed (encp) hook payloads open Main's sealed-box work (#3096) makes cross-deployment resumeHook() seal hook payloads to the target run's published X25519 public key. The shared start() path publishes that key regardless of engine, so QuickJS runs receive sealed payloads too — but the QuickJS entrypoint resolved only the bare symmetric key via importKey(), which cannot open encp envelopes. The first sealed hook payload wedged the run right after hook_received, timing out every hook/webhook e2e on Vercel prod (node:vm legs were fine — the node engine resolves the full capability via memoizeEncryptionKey). Resolve deriveRunPayloadKeys() in the entrypoint instead and widen the runtime's key types from CryptoKey to DecryptionKey. Writes stay symmetric (encrypt() with RunPayloadKeys takes the encr path). Regression test seals a payload exactly as resumeHook does and round-trips it through the VM. * Address review: crypto/process parity, loud Intl guards, lazy engine import, VM-leak guard, telemetry namespace, eval-string escaping - Deterministic crypto.getRandomValues/randomUUID in the VM bootstrap, drawing from the seeded Math.random (identical sequences to the node engine's vm/index.ts implementations); all crypto.subtle methods throw with step-function guidance. process.env exposed as a frozen copy, matching node. - Intl: throwing constructors (no ICU in QuickJS), and toLocale*-family methods (incl. localeCompare) throw when given an explicit locale so cross-engine divergence is loud instead of silently writing different values into the event log. No-argument forms keep working. - runtime.ts lazy-imports the QuickJS entrypoint at dispatch, keeping the ~1.3MB embedded WASM assets out of node-engine deployments. - runQuickJSWorkflow wraps the per-run phase so an exceptional exit disposes the VM instead of leaking it in a reused compute instance; corrected the misleading fail-loud comment (run_failed, not retry); warn when the event drain loop exhausts its iteration bound. - Telemetry attributes renamed quickjs.* → workflow.vm.* to stay in the file's workflow.* namespace. - Eval-string correlation-id interpolation uses JSON.stringify instead of quote-only escaping. - common-vm.test.ts pins the reducer/reviver superset invariant against common.ts so the duplicated sets can't silently drift. - Docs enumerate the remaining global-surface differences (subtle.digest, Intl, WebAssembly, Atomics); quickjs-entrypoint documents the known precondition-guard gap. * QuickJS engine: implement resilient resumeHook (hookInput materialization + resumeId dedup) #1834 made resumeHook() fall back to enqueueing the run with a hookInput payload when the direct hook_received write fails transiently, with the runtime materializing the missing event on delivery. Only the node:vm path implemented it — the QuickJS dispatch returned before the node block, so the resilient payload was silently dropped and the new e2e timed out on every quickjs leg. - runtime.ts threads hookInput into runWorkflowWithQuickJS; the entrypoint materializes the missing hook_received after loading the event log (resumeId-keyed dedup, occurredAt from the resumeId ULID, local eventData substitution for lazy/ref responses, EntityConflict / HookNotFound handling) — mirroring the node block. - processEvents drops duplicate hook_received rows sharing a resumeId (first-in-log wins), matching the node engine's EventsConsumer dedup; the seen-set lives in the VM heap so it is deterministic per replay. Verified against the dev server with WORKFLOW_VM=quickjs: the resilient resume e2e passes and the materialization is observable in the logs; all 27 hook e2e tests green. * QuickJS engine: inline step execution via live-VM continuation loop + WASM module caching * Address review: exclusive inline step claims, self-write requeue, in-loop event ceiling - Inline steps now claim via a lazy step_started carrying the input (step_created deferred, atomic create-claim in the world), with ownerMessageId stamped and authoritativeAttempt=1 — a concurrent invocation racing on the same fresh step loses with EntityConflictError and skips instead of both bare-starting the step and double-running the body. This also removes the stepsCreatedByUs set, whose 'created by us' invariant didn't survive the swallowed create-race conflict; redelivery backstops now key on hasCreatedEvent. - dispatchPendingOps' createdAttributeEvent/createdGetConflictHook signals are consumed again: when the loop exits suspended without ever reading back a self-written attr_set / getConflict hook_created (eventually-consistent listing lag), the entrypoint requeues immediately instead of parking the run awaiting_external with its unblocking event already written. - The server-supplied event ceiling is re-checked at the top of every continuation-loop turn (seenEventIds.size), so a single invocation fanning out inline can no longer grow the log arbitrarily past the operator's limit. The quickjs dispatch in runtime.ts converts MaxEventsExceededError into run_failed / MAX_EVENTS_EXCEEDED — the guard's throw previously nacked forever, parking runaway runs in 'running'. - Documented the deliberate decision that the platform function timeout is the only bound on inline chaining (budget parked per batch), matching the node engine. * Fix lost wait continuation for waits that elapse mid-iteration (sleepWinsRace flake) The pre-inline wait-continuation sweep skipped waits with resumeMs <= 0. A wait whose deadline falls between the iteration's elapsed-wait pass (which saw it as still pending and wrote nothing) and this sweep got NEITHER a wait_completed NOR a continuation — and the inline batch then blocked the invocation for the full step duration with no wake armed anywhere. For Promise.race(step, sleep) that silently hands the race to the step: the sleep's wait_completed is never written and the run completes with the wrong winner. The vulnerable window spans the iteration's dispatch + feed network round-trips, so on world-vercel a 1s sleep landed in it roughly half the time (the ~50% sleepWinsRaceWorkflow failure rate in the Vercel quickjs e2e legs), while world-local's sub-ms round-trips masked it locally. Match the node engine (Math.max(1000, resumeAtMs - now) in suspension-handler.ts): always arm the continuation, clamping already-elapsed waits to the 1s minimum — the continuation invocation's pre-VM elapsed check completes them. Waits whose wait_completed this invocation already wrote are skipped. Diagnosed from run wrun_41KZ73HR4H0GZ6RYD1WQHZX822 (CI run 30942512953): wait_created at +0.5s for a 1s sleep, no wait_completed ever, step_completed at +10.8s wins the race. |
||
|
Peter Wielander
|
de1905f15c | feat(world): require a runId on listByCorrelationId (#3280) | ||
|
Nathan Colosimo
|
27a3f15a7b |
fix(core): preserve Hook retention in QuickJS (#3319)
Co-authored-by: Nathan Rajlich <n@n8.io> |
||
|
christopherkindl
|
434e4bed2f | [docs] upgrade geistdocs to 1.19.4 (#3330) | ||
|
Mitul Shah
|
73da40cbb7 |
fix(web-shared): align colors with Geist (#3300)
* fix(docs): align Geist colors with Vercel Co-authored-by: Mitul Shah <mitulxshah@gmail.com> * chore: add docs color changeset Signed-off-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Mitul Shah <mitulxshah@gmail.com> * fix(web-shared): align colors with Geist Signed-off-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Mitul Shah <mitulxshah@gmail.com> --------- Co-authored-by: Cursor Agent <cursoragent@cursor.com> |
||
|
Karthik Kalyan
|
e084e08ac0 |
Reduce Vercel E2E polling load (#3316)
* Reduce Vercel E2E polling load * Keep Vercel E2E matrix concurrency |
||
|
Sepcnt
|
c22abcd5f1 |
Add SurrealDB as a community world (#1579)
Squashed and rebased onto main to resolve conflicts with the generic docker community-world CI: the dedicated surrealdb service steps from the original commits are replaced by the manifest-driven docker service type, with a new optional `args` field so the service definition can pass the `start` subcommand (and credentials) to the SurrealDB image. Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |
||
|
Nathan Colosimo
|
99f4aeb03d |
feat(world-postgres): support Hook minimum retention (#3276)
* feat(world-postgres): retain hook tokens after runs end
* refactor(world-postgres): reuse terminal run statuses
* docs: note Postgres Hook retention support
* fix(world-postgres): expose hook retention deadline
* Fix: Exhaustive `Record<AttributeKey, ...>` in `attribute-panel.tsx` is missing the `tokenRetentionUntil` key that was added to `HookSchema`, causing TS2741 and breaking every Vercel build.
This commit fixes the issue reported at packages/web-shared/src/components/sidebar/attribute-panel.tsx:426
## Bug
Commit `ad58321` added `tokenRetentionUntil: z.coerce.date().optional()` to `HookSchema` in `packages/world/src/hooks.ts:106`. This adds `tokenRetentionUntil` to the inferred `Hook` type.
In `packages/web-shared/src/components/sidebar/attribute-panel.tsx`, `AttributeKey` is a union that includes `keyof Hook`, so `tokenRetentionUntil` becomes a required member of the **exhaustive** `Record<AttributeKey, (value: unknown, context?: DisplayContext) => ...>` object literal `attributeToDisplayFn` (starting at line ~426).
Because the literal had no `tokenRetentionUntil` entry, `tsc` fails:
```
src/components/sidebar/attribute-panel.tsx(426,7): error TS2741:
Property 'tokenRetentionUntil' is missing in type '{ ... }' but required in type
'Record<AttributeKey, (value: unknown, context?: DisplayContext | undefined) => ReactNode>'.
```
This breaks `@workflow/web-shared#build` and therefore every Vercel deployment (17 failing deployments observed, all with this identical error).
## Fix
Added a `tokenRetentionUntil` entry to `attributeToDisplayFn`, placed alongside the other Hook date fields (`lastReceivedAt`, `disposedAt`):
```ts
tokenRetentionUntil: timestampWithTooltipOrNull,
```
`tokenRetentionUntil` is a `Date` field, and `timestampWithTooltipOrNull` (defined at line 402) is the display helper used by all the other surfaced date fields (`createdAt`, `startedAt`, `completedAt`, `retryAfter`, `resumeAt`, `occurredAt`). Given the intent of `ad58321` was to expose the hook retention deadline, surfacing it as a tooltip-annotated timestamp is the consistent choice.
Only `attributeToDisplayFn` is a fully exhaustive `Record<AttributeKey, ...>`; the other maps are `Partial<...>` / `Set`, so no other edits are required.
## Verification
`node_modules` are not installed in this sandbox, so `tsc` could not be executed directly. Verified structurally instead: the newly added `tokenRetentionUntil` entry (line 449) references `timestampWithTooltipOrNull`, which is defined in-file at line 402 and already used by the sibling date entries, so the fix satisfies the missing-key requirement without introducing new type errors.
Co-authored-by: Vercel <vercel[bot]@users.noreply.github.com>
Co-authored-by: VaguelySerious <mittgfu@gmail.com>
* docs(world-postgres): clarify expired hook rows
* feat(world-postgres): enforce Hook retention limit
* fix(world): remove duplicate Hook retention field
* fix(web-shared): remove duplicate retention renderer
* test(world): remove redundant retention coercion case
---------
Co-authored-by: vercel[bot] <35613825+vercel[bot]@users.noreply.github.com>
Co-authored-by: Vercel <vercel[bot]@users.noreply.github.com>
Co-authored-by: VaguelySerious <mittgfu@gmail.com>
|