Drop RunTraceView and WorkflowTraceViewer, move shared Span/Trace types into lib/trace-types, and keep timing helpers under the new viewer.
Signed-off-by: mitul-s <mitulxshah@gmail.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
A redelivered re-ensure of an already-committed resume (same runId +
resumeId + digest) was rejected with HookNotFoundError when the hook had
since been disposed by the workflow (dispose -> sleep releases the
token while the run continues). The queue consumer treats HookNotFound
as 'nothing left to resume' and acks the delivery — silently dropping
whatever continuation the redelivered message carried and wedging the
run.
Check the (runId, resumeId) claim BEFORE the disposal/existence
rejections: a committed claim whose pinned event is journaled proves
this exact resume was accepted while the hook was alive, so return that
event as success. Claims with a mismatched hookId or payload digest
still fall through to full validation and are rejected as before, as
are genuinely new resumes of a disposed hook.
- Every captured intrinsic exists on all supported engines (Node 18+), so
the optional-capture layer (intrinsicGetter-returns-undefined, canReadUrl/
canReadUrlSearchParams/canReadHeaders, per-use fallbacks) is replaced by
captures that throw at import if absent.
- URLSearchParams.prototype.size (the one genuinely missing member on Node
18) is not needed: emptiness falls out of the captured toString() result,
which the reducer already computes. Node 18 now serializes URLSearchParams
natively instead of falling back to devalue's default handling.
- The call/get tables and their re-export aliases flatten into direct typed
exports; readProxyAware and the viewInfo getter-indirection unroll into
two-branch functions.
- isEngineAccessor: drop the WeakMap memo and try/catch (descriptor getters
are always callable); exclude bound functions, which stringify as native
code but run their target — previously workflow code could launder a
side-effectful getter past the report with fn.bind() (test added).
- 763 -> 625 lines, byte output unchanged (parity checked for DataView and
typed-array subviews on top of the existing test suite).
## Summary & Motivation
One warm compute instance serves many invocations, so the Compute Instance ID already in this panel can't distinguish steps that ran inline within a single flow-function invocation — sibling steps sharing a Request ID did. It's also the value Vercel Logs indexes by, and this panel's View Logs button is where a reader takes it next.
The key is `vercelId`, not `requestId`: that's the name world-vercel stores the SDK's request id under crossing the wire, and AnalyticsEvent's sibling `requestId` field is declared but never written. Nothing in this repository populates `vercelId` on the object the panel receives yet — only AnalyticsEvent carries it, and grouping steps by invocation needs a step-level aggregation in workflow-server first — so the row ships as forward-compatible plumbing in the slot next to Compute Instance ID.
The second commit is an independent fix: `sortByAttributeOrder` guarded `indexOf` with `|| 0`, but a miss returns -1, which is truthy, so any key absent from `attributeOrder` sorted ahead of every listed key. It can be dropped on its own.
## Test Plan
Unit tests added; the two ordering assertions fail against the unfixed comparator. The new row could not be verified by hand — no local code path populates it.
* [benchmarks] Link the run id and Datadog trace under the STSO histograms
The STSO distribution section added in #3213 shows the shape of the
sequential-steps run but not which run produced it, so investigating an
odd-looking bucket meant hunting for the run by deployment id and time
window.
Capture the identity alongside the samples (the mechanism prototyped on
the WIP variance branch, #3107): `/api/bench` returns the trace id of the
span @vercel/otel opened for the trigger request, the runner threads it
through the sequential iteration and records `sequentialRuns` in the
result file, and the renderer prints one line under the histograms with
the run id + Datadog trace link for this run and for the `main` run it is
diffed against.
Every part is optional — a deployment predating the route change yields a
bare run id, and a `main` baseline predating this yields only this run's
side — so the section degrades instead of breaking on mixed-vintage
artifacts.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Shalabh Chaturvedi <7066873+shalabhc@users.noreply.github.com>
Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com>
* Log the run/trace links instead of rendering them into the comment
The run id and Datadog trace are debugging aids, not part of the
benchmark's reported result, so they belong in the job's own output
rather than in the PR comment body.
Logging them where the runs are produced also makes them available in
two cases the comment could never cover: a local `pnpm bench`, and a
job that fails before the comment step runs.
This drops the comment-rendering side entirely -- `renderSequentialRunLinks`,
the `baselineSequentialRuns` baseline plumbing in `annotateWithBaseline`,
and the `sequentialRuns` field on the result artifact, which existed only
to carry the data to the renderer.
The `main`-baseline side of the link goes away with it: which run produced
the baseline histogram is only knowable at comment-render time, where the
two artifacts are matched.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com>
* Say what the trigger trace actually contains under linked mode
The route comment claimed the trigger request's span "propagates into
the workflow's own spans". That only holds under
WORKFLOW_TRACE_MODE=continuous. Nothing in the workbench or
benchmarks.yml sets the mode, so the benchmark deployment runs the
default `linked` (packages/core/src/telemetry.ts), where each
workflow/step invocation is its own trace root and the trigger's trace
carries `workflow.start` plus span links out to those roots.
The logged link is still the right entry point -- one hop through the
links, which Datadog renders -- but the comment should describe that,
so nobody opening a trigger-only trace while debugging a histogram
concludes the run produced no spans.
Raised by @TooTallNate in review of #3248.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Co-Authored-By: Shalabh Chaturvedi <7066873+shalabhc@users.noreply.github.com>
Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com>
* Log a Datadog span search alongside the trigger trace link
Under the default linked trace mode the trigger's trace holds only
`workflow.start` plus span links, so opening it lands one hop away from
the spans an STSO investigation needs. Log an APM search on
`@workflow.run.id:<runId>` next to it, which goes straight to the run's
execution spans.
Both links are logged rather than one replacing the other: the search
depends on `workflow.run.id` being an indexed span tag in the org, and
the permalink works regardless.
Suggested by @TooTallNate in review of #3248.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Co-Authored-By: Shalabh Chaturvedi <7066873+shalabhc@users.noreply.github.com>
Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com>
---------
Co-authored-by: vercel[bot] <35613825+vercel[bot]@users.noreply.github.com>
* feat(core): lazy hook resumption via parallel event write + queue publish (rebased onto #1834 + #3145)
Rebase of #3230 onto current main (267765375 + #1834 resilient resumeHook
+ #3145 event-count-gated replay restart). Reconstructed as a single commit
since `git rebase -i` is unavailable in this environment.
Reconciliation vs the pre-rebase branch:
- Replaces #1834's version-prediction (`supportsQueueHookInput`,
`QUEUE_HOOK_INPUT_MIN_VERSION`) with #3230's capability protocol
(persisted `hookResumeInputVersion` + static `hookResumeDedup`).
- One idempotency protocol: a single `resumeId` + SHA-256 payload digest
per resume, sent to both the direct event write and the queue `hookInput`.
- Two execution tiers: backend+consumer attest dedup -> parallel
`Promise.allSettled(event write, queue publish)`; otherwise plain
sequential (no hookInput/resumeId, event-write errors propagate).
- Consumer re-ensures the `hook_received` event (keyed by resumeId/digest)
after event loading, before replay; skips when already preloaded.
- Preserves #3145: event-count guard, `preconditionReinvocations`,
in-process replay restart, `insertEventByEventId`.
- Removes #1834's resumeId-only test (never released); adds parallel +
consumer-preload + world-local dedup/producer-consumer suites.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(core): read top-level event.resumeId in replay dedup; reconcile unreleased #1834 docs/changeset
- hook.ts: dedup hook_received replay on top-level event.resumeId (the
backend now hoists it to a first-class column), with the legacy nested
eventData.resumeId retained as a deprecated parse-only fallback.
- workflow.test.ts: cover dedup across both top-level and legacy nested forms.
- resume-hook.ts: emit producer recovery telemetry when a transient
event-write failure is swallowed on the parallel path.
- resume-hook.consumer-preload.test.ts: add terminal-run (consume) and
transient-conflict (rethrow/redeliver) re-ensure cases.
- Consolidate the two overlapping changesets into resilient-resume-hook.md
and delete the redundant lazy-hook-resumption.md.
- Docs: return type back to Promise<Hook> (resume-hook.mdx), rewrite the
resilience changelog to the final parallel/deduplicated design, and correct
the WORKFLOW_DISABLE_LAZY_HOOK_RESUME resilience wording.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* docs,core: rename "Resilient hook resume" → "Lazy hook resume" for consistency
- changelog/index.mdx: update the changelog entry title.
- hook.ts: update the dedup comment label to "Lazy-resume dedup".
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* chore: give #3230 its own changeset instead of repurposing #1834's
The lazy-hook-resume work had been folded into #1834's pre-existing
`resilient-resume-hook.md` changeset. Give this PR its own changeset and
delete the superseded #1834 one, whose `resilientResume: true` flag promise
no longer holds (resumeHook() returns plain Promise<Hook>).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* chore: add #3230's own lazy-hook-resumption changeset
Follow-up to 63d877178, which deleted #1834's superseded changeset but did
not stage the replacement. Adds this PR's own changeset.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* chore: tighten lazy-hook-resumption changeset
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* chore: leave #1834's resilient-resume changeset/changelog/docs untouched
Restore #1834's own artifacts that #3230 had rewritten:
- .changeset/resilient-resume-hook.md (restored verbatim)
- docs/.../changelog/resilient-resume.mdx (restored verbatim)
- docs/.../changelog/index.mdx (restored verbatim)
#3230 keeps only its own changeset plus the two docs its code/config genuinely
require: the resumeHook() Promise<Hook> return type (ResumedHook is removed
from the code) and the new WORKFLOW_DISABLE_LAZY_HOOK_RESUME env var.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* Reconcile #1834 ResumedHook contract with #3230 parallel resume
Preserve the resilient-resume contract from #1834 on the parallelized
resumeHook() fast path instead of dropping it:
- Restore the `ResumedHook` type (Hook + optional `resilientResume`) and its
exports (`@workflow/core/runtime`, `workflow/api`); resumeHook/resumeHookImpl
return `Promise<ResumedHook>`.
- Set `resilientResume: true` on the swallow-recover branch (transient direct
write failure + successful queue dispatch), absent on the happy/sequential
paths.
- Restore the producer OTEL convention `workflow.hook.resilient_resume` and the
consumer `workflow.hook.resilient_resume_materialized`, wired where the
consumer re-ensures the event.
- Restore the consumer `occurredAt` derivation from the resume ULID so the
materialized hook_received is dated to resume time, not queue-round-trip time.
- Fix the #3230 changeset's contradictory "Still returns Promise<Hook>" line and
update the resilient-resume changelog + resume-hook API reference to the
shipped parallel/dedup behavior.
- Port the #1834 failure-path coverage into resume-hook.parallel.test.ts
(non-retryable event-write rethrow, both-fail prioritizes the queue error,
resilientResume flag + payload delivery on the recovered path).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* Address review: drop dead nested resumeId fallback, remove server PR link
- Drop the legacy nested `eventData.resumeId` fallback in the hook consumer.
The nested form was only ever written by unreleased preview builds and is
stripped by `EventSchema` parsing (the `hook_received` eventData schema does
not declare it), so the fallback was dead code. Dedup now keys solely off the
top-level `event.resumeId` column. Repoint the replay dedup test to the
surviving top-level path (it previously exercised the nested form only by
building unparsed Event objects in memory).
- Remove the internal workflow-server PR reference from world-vercel's
capability note (the link 404s outside the org); the note keeps the same
information without the dead link.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
* feat(core): side-effect-free serialization of workflow VM values
Serialization runs on the host but inspects values constructed inside the
node:vm sandbox, so ordinary dynamic operations dispatch into the sandbox
realm and execute workflow code: `value.toISOString()`, `Array.from(map)`,
`Object.prototype.toString` (via Symbol.toStringTag), `.source`/`.flags`,
`.href`, view `.buffer`/`.byteOffset`/`.byteLength`, and error
`.message`/`.stack`/`.cause` reads.
That is a determinism hazard. A payload is serialized exactly once and is
never re-serialized on replay, so any workflow-visible side effect it
triggers exists only on the live path — a patched `Date.prototype.toISOString`
that consumes a seeded `Math.random()` draw, for example, shifts every
subsequent draw and diverges from replay.
This makes serialization side-effect free where the data allows it, and
observable where it does not:
- Classification uses engine brand checks (node:util types, internal-slot
probes) instead of `instanceof global.X` and Object.prototype.toString, so
it is immune to Symbol.hasInstance, reassigned sandbox globals, and
Symbol.toStringTag spoofs. An unbranded value claiming a brand-decided tag
is now classified as a plain object instead of being routed into an
extractor that requires the real internal slot (unhardened devalue crashes
on that input).
- Extraction goes through intrinsics captured at module load — host boot,
before any workflow bundle runs — invoked with explicit receivers.
Internal slots are realm-agnostic, so host intrinsics read VM-realm
objects without touching the sandbox's patchable prototypes.
- Property access reads through descriptors, so plain data never invokes
anything.
Where workflow code must run because the data lives behind it — getters,
proxies, custom [WORKFLOW_SERIALIZE] methods, toString() on
toStringTag-branded objects like Temporal polyfills — the execution is
preserved for compatibility and recorded in a new `CodecOptions.guestCodeStats`
sink, surfaced as workflow.serialization.guest_code_{executions,details} span
attributes. Consumers that retain a VM across steps can treat a non-empty
report as "serialization may have perturbed VM state".
Engine-provided accessors are deliberately not reported: V8 defines `stack`
as an own accessor on every Error instance, so reporting it would flag every
serialized error. Nativeness is decided with the captured host
Function.prototype.toString; the bound-function caveat is documented in
hardened.ts.
Requires devalue 5.9.0 for the pluggable `operations` option.
* chore: shorten changeset
* fix(core): close review gaps in hardened serialization
Five correctness fixes, all with repros:
- Callable proxies were treated as engine accessors. V8 returns
`function () { [native code] }` from Function.prototype.toString for a
proxy around a function rather than throwing, so a proxy-wrapped getter
was cached as engine-provided and invoked unreported. Gate on
types.isProxy first.
- Host builtins implemented in JavaScript were reported as workflow code.
Node's DOMException.prototype.message/name are ordinary functions, so
the nativeness test failed and every serialized DOMException reported
two getter executions. They belong to the *host* realm, though, and
workflow code cannot author a host-realm function — so provenance is
now decided by nativeness OR host-realm `Function.prototype`, which are
disjoint and together cover both cases (V8 installs `stack` per realm,
so a VM error's getter is native but VM-realm).
- The extraReducers at the two VM call sites were still unhardened, and
they run on every value the earlier reducers do not claim — which is
exactly where the report has to be complete. `instanceof
global.ReadableStream/WritableStream/Request/Response` consulted
Symbol.hasInstance on the sandbox class (14 invocations for an ordinary
payload once the classes are patched), and AbortController's guard did
a bare `value.signal` read, so a non-enumerable `signal` getter ran
with an empty report. All five now walk the prototype chain and read
through descriptors.
- `__closureVarsFn` was invoked unreported on a purity argument that
nothing checked: the property is reachable from workflow code, which
can replace the compiler-generated function. step.ts now registers the
generated function as trusted when it builds the proxy, so provenance
is verified rather than assumed, and an unrecognized function is
reported.
- The URL/URLSearchParams test patched prototypes of *host* classes
injected into the sandbox, mutating them for the rest of the worker
process. Restored in a finally.
Also, per review:
- `dehydrateStepArguments` / `dehydrateWorkflowReturnValue` take an
optional GuestCodeStats out-param, so a retained-VM gate can consume
the report instead of it being spent on span attributes. The
report-completeness tests use it to exercise the real dehydrate path.
- Every intrinsic capture is now optional. The table is built at module
scope, so a missing member was an import-time crash of @workflow/core
rather than a degraded path; only SharedArrayBuffer was guarded, while
URLSearchParams.prototype.size (Node 19.8+) and the WHATWG classes were
assumed. Absent captures now make the corresponding reducer decline to
match.
- Documented that recording is not prevention (a recorded getter calling
Math.random() still advances the run's seeded PRNG), and that a
`{ kind: 'proxy' }` report implies a silent shape change (a proxied Map
serializes as a plain object).
- Parity coverage extended to DataView, boxed primitives, null-prototype
objects, setter-only properties, DOMException, AggregateError, an
accessor-valued Symbol.toStringTag, both RetryableError retryAfter
paths, and a WORKFLOW_SERIALIZE class instance.
* fix(core): keep identifying proxied host classes
Every Next.js e2e job failed on the two webhook tests: the hook POST
returned 404 because `resumeWebhook` could not serialize its step return
value ("Cannot stringify arbitrary non-POJOs"), so no hook was ever
registered.
The value was a `NextRequest`, which Next.js hands over as a **Proxy**.
`isInstanceOfPrototype` rejected proxies outright, so the Request reducer
answered "not a Request" and devalue fell through to the POJO check. The
reasoning behind rejecting them — that proxied built-ins were never
serializable, because internal-slot reads throw on a proxy receiver — is
true for `Map`/`Date`/`URL`, whose reducers read internal slots, but not
for `Request`/`Response`/streams, whose reducers read ordinary
properties. Next's proxy forwards those with the target as receiver, so
they serialized fine before this PR.
Identification now walks through proxies, matching `instanceof`, and
records the traps rather than suppressing the answer. The three reducers
that do read internal slots (URL, URLSearchParams, Headers) fall back to
the dynamic read when the value is a proxy, so their behavior is exactly
what it was before — including throwing for a bare proxy over a built-in,
which threw before too.
Verified against the real thing: the full nextjs-turbopack e2e suite
(135 tests) passes locally, having reproduced the failure first and
confirmed a reverted `serialization.ts` fixed it.
The regression test uses a receiver-correcting proxy, which is what makes
NextRequest work in practice; a comment records that a bare
`new Proxy(request, {})` throws on undici's private slots with or without
this change.
* fix(core): state what the closure-fn mark proves, and correct stale docs
- `isInstanceOfPrototype`'s JSDoc still described the behavior removed in
8bc462fb5 (proxies rejected without firing traps), which is the opposite
of what it now does.
- The `__closureVarsFn` provenance check proves the function was passed to
`useStep`, not that this package generated it: `useStep` is published on
the sandbox global, so workflow code can call it with a function of its
own and have it marked. Renamed `registerTrustedFunction` /
`isTrustedFunction` to `markUseStepClosureFn` / `isUseStepClosureFn` so
the name states the boundary, and documented the laundering caveat
alongside the existing ones. Marking still earns its keep — reporting
every step that captures a variable would bury the signal — and closing
the gap properly needs a compiler-emitted marker, which is a compiler
change.
- Added the missing coverage for both sides of that check: an unmarked
`__closureVarsFn` is invoked and reported, a marked one is invoked and
not.
- `guestCodeStats` was documented as something a retained-VM gate consumes,
but no runtime caller passes a sink; the executions reach telemetry from
every dehydrate path regardless. Reworded both docs to say that, so the
out-param is not mistaken for wiring that already exists.
- useQuickJSVm defaults to the QuickJS engine when neither the run's
stamped executionContext.workflowVm nor WORKFLOW_VM specifies one;
WORKFLOW_VM=node is the explicit node:vm opt-in.
- CI matrix inverted to match: quickjs legs leave WORKFLOW_VM unset so
the default-selection path is exercised end to end; node legs opt in
explicitly (labels/artifacts unchanged via MATRIX_VM).
- Entrypoint tests that assert node replay-loop internals against mock
worlds pin WORKFLOW_VM=node (the quickjs path would instantiate a WASM
VM per call).
- Docs + changeset updated. Runs keep the engine stamped at start().
- SnapshotMetadata gains eventCount, rngDraws and formatVersion. The
max-events guard now compares restored total + delta (both at entry
and per loop turn) — previously a run that kept snapshotting could
never accumulate enough delta to trip the ceiling it exists for.
- Correlation-id generation is position-based across snapshots: the
runtime seeds from the BASE seed and fast-forwards the persisted draw
count instead of mixing the snapshot cursor into the seed. Ids are now
identical across snapshot generations AND identical to a no-snapshot
run, so overlapping invocations straddling a snapshot save still
collide on the world's dedup (new test pins restored ids == full-replay
ids). Snapshots without a draw count fall back to full replay.
- Host callbacks are declared in ONE list that drives both the fresh-boot
install and the restore re-registration, so adding a callback can't
silently skip the restore path.
- Preloaded events are used again with snapshotting enabled (the first
qualifying suspension skips its save — no cursor yet); short runs keep
the zero-round-trip fast path.
- Snapshot persist runs off the response path (waitUntil), with a 32MB
plaintext size ceiling (skip + warn). Loads that fail format/shape
checks warn instead of silently miming a miss. Terminal deletes are
gated on a snapshot actually existing and now also fire on the runGone
path; a server-side TTL remains the backstop for unobserved
cancellations.
The security bot flagged path traversal: an unvalidated runId was
interpolated directly into {basedir}/snapshots paths, letting ../ (or /,
\, NUL, absolute fragments) escape the snapshots dir on save/load/delete.
Apply the same two-layer scheme as the other world-local storages:
assertSafeEntityId as the primary defense plus resolveWithinBase
containment at the join. Hostile-runId test covers all three operations.
- Inline steps now claim via a lazy step_started carrying the input
(step_created deferred, atomic create-claim in the world), with
ownerMessageId stamped and authoritativeAttempt=1 — a concurrent
invocation racing on the same fresh step loses with
EntityConflictError and skips instead of both bare-starting the step
and double-running the body. This also removes the stepsCreatedByUs
set, whose 'created by us' invariant didn't survive the swallowed
create-race conflict; redelivery backstops now key on hasCreatedEvent.
- dispatchPendingOps' createdAttributeEvent/createdGetConflictHook
signals are consumed again: when the loop exits suspended without ever
reading back a self-written attr_set / getConflict hook_created
(eventually-consistent listing lag), the entrypoint requeues
immediately instead of parking the run awaiting_external with its
unblocking event already written.
- The server-supplied event ceiling is re-checked at the top of every
continuation-loop turn (seenEventIds.size), so a single invocation
fanning out inline can no longer grow the log arbitrarily past the
operator's limit. The quickjs dispatch in runtime.ts converts
MaxEventsExceededError into run_failed / MAX_EVENTS_EXCEEDED — the
guard's throw previously nacked forever, parking runaway runs in
'running'.
- Documented the deliberate decision that the platform function timeout
is the only bound on inline chaining (budget parked per batch),
matching the node engine.
#1834 made resumeHook() fall back to enqueueing the run with a hookInput
payload when the direct hook_received write fails transiently, with the
runtime materializing the missing event on delivery. Only the node:vm
path implemented it — the QuickJS dispatch returned before the node
block, so the resilient payload was silently dropped and the new e2e
timed out on every quickjs leg.
- runtime.ts threads hookInput into runWorkflowWithQuickJS; the
entrypoint materializes the missing hook_received after loading the
event log (resumeId-keyed dedup, occurredAt from the resumeId ULID,
local eventData substitution for lazy/ref responses, EntityConflict /
HookNotFound handling) — mirroring the node block.
- processEvents drops duplicate hook_received rows sharing a resumeId
(first-in-log wins), matching the node engine's EventsConsumer dedup;
the seen-set lives in the VM heap so it is deterministic per replay.
Verified against the dev server with WORKFLOW_VM=quickjs: the resilient
resume e2e passes and the materialization is observable in the logs; all
27 hook e2e tests green.
- Deterministic crypto.getRandomValues/randomUUID in the VM bootstrap,
drawing from the seeded Math.random (identical sequences to the node
engine's vm/index.ts implementations); all crypto.subtle methods throw
with step-function guidance. process.env exposed as a frozen copy,
matching node.
- Intl: throwing constructors (no ICU in QuickJS), and toLocale*-family
methods (incl. localeCompare) throw when given an explicit locale so
cross-engine divergence is loud instead of silently writing different
values into the event log. No-argument forms keep working.
- runtime.ts lazy-imports the QuickJS entrypoint at dispatch, keeping the
~1.3MB embedded WASM assets out of node-engine deployments.
- runQuickJSWorkflow wraps the per-run phase so an exceptional exit
disposes the VM instead of leaking it in a reused compute instance;
corrected the misleading fail-loud comment (run_failed, not retry);
warn when the event drain loop exhausts its iteration bound.
- Telemetry attributes renamed quickjs.* → workflow.vm.* to stay in the
file's workflow.* namespace.
- Eval-string correlation-id interpolation uses JSON.stringify instead of
quote-only escaping.
- common-vm.test.ts pins the reducer/reviver superset invariant against
common.ts so the duplicated sets can't silently drift.
- Docs enumerate the remaining global-surface differences (subtle.digest,
Intl, WebAssembly, Atomics); quickjs-entrypoint documents the known
precondition-guard gap.
Main's sealed-box work (#3096) makes cross-deployment resumeHook() seal
hook payloads to the target run's published X25519 public key. The shared
start() path publishes that key regardless of engine, so QuickJS runs
receive sealed payloads too — but the QuickJS entrypoint resolved only the
bare symmetric key via importKey(), which cannot open encp envelopes. The
first sealed hook payload wedged the run right after hook_received, timing
out every hook/webhook e2e on Vercel prod (node:vm legs were fine — the
node engine resolves the full capability via memoizeEncryptionKey).
Resolve deriveRunPayloadKeys() in the entrypoint instead and widen the
runtime's key types from CryptoKey to DecryptionKey. Writes stay symmetric
(encrypt() with RunPayloadKeys takes the encr path). Regression test seals
a payload exactly as resumeHook does and round-trips it through the VM.
* feat(core): stamp creator environment into runInput and reject cross-environment queue deliveries client-side
`start()` makes two writes that have to land in the same tenant: the
`run_created` event, attributed to whatever environment the caller
authenticates as, and the queue message, pinned to a deployment. A
misconfigured caller can split them — writing the run to one environment
while addressing the message to a deployment in another. The consumer
finds no run under its own tenant, the backend's resilient start
(`run_started` creates the run when `run_created` was never seen) mints a
second copy of the same run id in the consumer's environment, and both
copies are real: the creator's sits pending forever while the other
executes.
The deployment id is not the discriminator — it matched end to end in the
incident that motivated this. The environment is. So carry it: add an
optional `World.getEnvironment()`, implement it in world-vercel from the
same resolution that produces the `x-vercel-environment` header, and stamp
it into the queue message's `runInput`.
The consuming deployment already knows its own environment, so it can
refuse the delivery itself with no server coordination — and refuse before
`run_started`, the write that would create the fork. The refusal acks the
message instead of throwing: the mismatch is baked into the message, so
every redelivery would reach the same verdict and throwing would hot-loop
until MAX_QUEUE_DELIVERIES.
Both sides must be known for the check to run, so worlds with a single
tenant (local, Postgres) and runs started by an older SDK behave exactly
as before. A companion diagnostic logs a deployment-id mismatch without
refusing, since deployment ids differ for benign reasons too.
Signed-off-by: Pranay Prakash <pranay.gp@gmail.com>
* fix(world-vercel): resolve the runtime environment from VERCEL_TARGET_ENV
For a deployment in a Vercel custom environment, the OIDC token's
environment claim is the custom environment's slug (the platform mints
`customEnvironment?.slug ?? envTarget`) while VERCEL_ENV reports
'preview' — so keying the cross-environment guard on VERCEL_ENV could
false-refuse a legitimate delivery, e.g. a CLI client attributed to
'staging' starting a run on the staging deployment. VERCEL_TARGET_ENV
is populated from exactly the same slug-or-target pair as the claim, so
prefer it, keeping VERCEL_ENV as the fallback for contexts that don't
inject it. Also sorts runtime.ts imports per the Biome rule that landed
on main in #3241.
Signed-off-by: Pranay Prakash <pranay.gp@gmail.com>
---------
Signed-off-by: Pranay Prakash <pranay.gp@gmail.com>
* Make resumeHook() resilient to transient hook_received event write failures
When events.create('hook_received') fails with a retryable error (429/5xx),
resumeHook() now dispatches the queue message with a `hookInput` payload
carrying the dehydrated hook payload. The workflow runtime materializes the
missing hook_received event from that payload on its next delivery, mirroring
the existing resilient-start behavior of start() / run_created / run_started.
Returned Hook carries a new `resilientResume: true` flag when the fallback
path was taken. Both write paths share a client-minted `resumeId` as an
idempotency key so the runtime can dedup if the direct write actually
committed but the client saw a transient error.
Uses a sequential write-then-queue flow (not parallel) to avoid a dedup race
on the happy path: hook_received events have no entity-level conflict guard
(unlike run_created), so a duplicate written before the direct write commits
would double-deliver the payload to the workflow.
* Fix resilient resume: use local payload in materialized hook_received event
The server returns a 'lazy' response for hook_received event creation,
where eventData.payload may be a RefDescriptor (when the payload
exceeded the inline size and was offloaded to blob storage) rather
than the raw bytes. Pushing this directly to the in-memory events
array caused the workflow VM to fail with 'Invalid input' when trying
to deserialize the RefDescriptor as a Uint8Array.
Substitute the eventData we already have locally so the in-memory
event matches what getWorkflowRunEvents would return after
client-side ref hydration.
* Gate resilient resume on target runtime capability; carry hook token; export ResumedHook; docs
- Only take the resilient path when the target run's recorded
@workflow/core version understands hookInput on the queue payload.
Runs keep executing on the deployment they were created on (skew
protection), and older runtimes parse the queue message with a schema
that silently strips unknown fields - the resume payload would be
lost while resumeHook() reported success. Fail fast (propagate the
original event-write error) for such runs instead, preserving the
caller's ability to retry.
- Carry the hook token on hookInput and write it into the materialized
hook_received event so it gets the same replay-divergence guard as a
directly written event (#2030 parity).
- Export ResumedHook from @workflow/core/runtime and workflow/api.
- Add changelog page and update resumeHook() API reference docs.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Address review: correct capability cutoff, drop own-version escape hatch, replay-side resumeId dedup
Review fixes for the resilient-resume capability gate and dedup:
- Bump the supportsQueueHookInput cutoff to 5.0.0-beta.39: 5.0.0-beta.38 is
published WITHOUT this feature (its queue-payload schema strips hookInput),
so classifying it as capable would silently lose resume payloads. The
cutoff is now a single exported constant (QUEUE_HOOK_INPUT_MIN_VERSION)
with a TODO(release) requiring re-verification at merge time.
- Remove the own-version exact-match escape hatch entirely: version strings
do not identify builds (a published beta.38 and a main-built tarball can
share a version string while differing in content), so the check could
declare a featureless published deployment capable. Pre-release builds now
fall back to fail-fast until the version is bumped past the cutoff — the
safe direction. Tests simulate a capable target explicitly.
- Make duplicate suppression authoritative at the replay boundary: replay
now dedups hook_received events sharing a resumeId (same resume attempt),
so even when concurrent redelivery of the same queue message
double-materializes the event (no World enforces uniqueness on
hook_received), the payload reaches workflow code exactly once. This is a
pure function of the persisted log, keeping replay deterministic. The
runtime's snapshot check remains as best-effort write suppression, with
its comment corrected to say so; the EntityConflictError catch is kept as
the forward-compatible signal for planned server-side (runId, resumeId)
uniqueness, with its comment corrected to say it is defensive today.
- Stamp materialized hook_received events with occurredAt decoded from the
resumeId ULID so resiliently-resumed hooks are timestamped at resume time
rather than after the queue round-trip.
- Pin the cross-version compat contract in a test: the direct write is
resumeId-only (no digest or negotiation fields), which later server-side
idempotency work must keep accepting.
- Exercise the published boundary (5.0.0-beta.38) in fail-fast tests, and
make the capability tests self-check against the exported cutoff constant
instead of restating literals.
- Docs: changelog date June -> July 2026, dash consistency, and document the
replay-side dedup guarantee.
* Encode release-gate and successor-rebase contracts into code comments
Comment-only changes capturing the review agreements so they survive the
parallel-resume successor rebase (no behavior change):
- capabilities.ts: the QUEUE_HOOK_INPUT_MIN_VERSION re-verification point
is the actual combined SDK release (after the successor lands and its
server-side dedup is deployed), not source-merge time — this PR merges
source-only and no SDK is published from it alone. Every Version
Packages merge in between moves the earliest possible carrier.
- workflow/hook.ts + runtime.ts: scope the replay-side resumeId dedup
honestly as defense-in-depth over the persisted log, not a
cross-invocation exactly-once guarantee — concurrent invocations
replaying pre-duplicate snapshots each see only their own row; the
storage-level (runId, resumeId) constraint in the successor work is the
correctness boundary. The set stays useful post-constraint for logs
written before it deployed.
- runtime.ts: document the EntityConflictError swallow's known gap while
the branch is defensive (this invocation's local log lacks the payload;
progress relies on the other writer's delivery or redelivery) and pin
the rebase contract for when the constraint makes it live: a matching
claim must append the canonical event locally and succeed; a real
conflict must rethrow for redelivery.
- resume-hook-resilient.test.ts: reframe the wire-shape pin as a tripwire
rather than a permanent contract — the successor deliberately widens it
(ID/digest pair + attestation) before any SDK release, so the
resumeId-only shape never ships as a published server contract.
---------
Co-authored-by: Peter Wielander <peter.wielander@vercel.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* ci: stop deploying changeset-release/main, run its e2e against production
The changesets action force-pushes `changeset-release/main`, and it can
point at exactly main's HEAD SHA. Vercel keeps one commit status per
project per SHA, so when both a production deployment (from main) and a
preview deployment (from changeset-release/main) are built for the same
commit, whichever finishes last owns the status. On 2026-07-30 the
preview finished last, so `vercel/wait-for-deployment-action` — which
reads the deployment ID out of that status — handed production e2e runs
a preview deployment ID and forked runs across environments.
Disable git deployments for that branch in every Vercel project rooted
in this repo, and give the changeset PR's Vercel e2e lanes a deployment
to test that actually exists: main's production deployment for the PR's
base SHA, resolved by SHA so a mid-flight production build is waited out
rather than silently replaced by an older one.
Signed-off-by: Pranay Prakash <pranay.gp@gmail.com>
* ci: resolve changeset-release e2e deployments with the wait action, tokenless
Per review: with changeset-release/main no longer deployed, main SHAs
can never again be deployed to a second environment of these projects,
so the per-SHA commit status the action reads is unambiguous for
exactly this lane. Reuse vercel/wait-for-deployment-action with
environment: production and sha pinned to the PR base SHA instead of
the Vercel-API polling script, drop the script and its VERCEL_TOKEN
usage, and inherit the action's inactive/skipped-build handling.
Signed-off-by: Pranay Prakash <pranay.gp@gmail.com>
---------
Signed-off-by: Pranay Prakash <pranay.gp@gmail.com>
* Fix Biome lint violations and add Biome CI check
Biome was not configured to respect .gitignore, so ~92% of the 13,355
reported diagnostics came from gitignored build artifacts. Enable VCS
integration (useIgnoreFile), apply safe auto-fixes across the repo, fix
the remaining mechanical errors by hand, downgrade judgment-call a11y /
dangerouslySetInnerHTML rules to warnings, and add a 'biome ci' job to
the Lint workflow so violations block PRs going forward.
* Use an empty changeset (no behavior change, no release needed)
* feat(world): persist the compute instance that ran each step attempt
Add CreateEventParams.computeInstanceId (ambient per-event identity, mirroring requestId) and a readable Event.computeInstanceId. Core stamps it on every step_started write; world-vercel forwards it in the v4 frame meta next to vercelId. Lets observability distinguish steps sharing a compute instance from those on different instances or invocations.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Alex Langenfeld <alex.langenfeld@vercel.com>
* test: cover computeInstanceId threading from params to v4 frame meta
world-vercel: computeInstanceId reaches the v4 frame meta, rides alongside vercelId rather than replacing it, and is omitted when unset. core: step_started carries it without displacing the stateUpdatedAt precondition guard (both share one params object).
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Alex Langenfeld <alex.langenfeld@vercel.com>
* fix(web-shared): render computeInstanceId in the attribute panel
AttributeKey derives from keyof Event, so adding computeInstanceId to the event schema widened it and left the exhaustive attributeToDisplayFn map incomplete (TS2741). Renders it beside deploymentId as 'Compute Instance ID', copyable like the other opaque ids.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Alex Langenfeld <alex.langenfeld@vercel.com>
* fix(world-postgres): exclude computeInstanceId from the events column contract
The events table asserts satisfies DrizzlishOfType<...Omit<Event, 'occurredAt'>...>, so adding computeInstanceId to the event schema broke the build (TS1360). This world does not persist it, matching how occurredAt is already handled.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Alex Langenfeld <alex.langenfeld@vercel.com>
* refactor: move computeInstanceId to the analytics read contract
The server routes computeInstanceId into ClickHouse and returns it on AnalyticsEvent/AnalyticsStep, never on the event record — so Event.computeInstanceId was dead on read and zod would strip the field off the analytics wire. Move it to AnalyticsEventSchema/AnalyticsStepSchema (beside vercelId/requestId, the same class of ambient provenance), which also drops the world-postgres column-contract exclusion entirely.
Also: hoist the duplicated step_started params into one local, extract the repeated mock-agent harness in events.test.ts, and use vi.spyOn plus an identity assertion against COMPUTE_INSTANCE_ID.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Alex Langenfeld <alex.langenfeld@vercel.com>
---------
Signed-off-by: Alex Langenfeld <alex.langenfeld@vercel.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
* Split STSO by inline vs queue-hop steps, add distribution diffs vs main
The sequential-steps benchmark's STSO metric mixed two unrelated
phenomena: gaps between steps running back-to-back in the same warm
process, and gaps across an invocation boundary (queue dispatch, client
reinit, event-log replay), which cost ~10x more. The old step-index
windows (1-20 / 101-120 / 1001-1020) sampled 19 gaps each and captured
neither cleanly: whether a boundary happened to land inside a window
moved that window's P99 by hundreds of percent, which is most of the
run-to-run variance the benchmark comment was reporting.
The workflow now tags each step with whether it was the first step body
executed in its process ('queue-hop') or a later one in the same warm
process ('inline') via a process-global, so the split is ground truth
rather than inferred from step index or trace timestamps. STSO is
reported as two rows over *every* gap in the run instead of three
sampled windows. No targets on the new rows — the old ones described the
index-bucketed grouping.
computeStats now keeps the full sorted sample array alongside the
percentiles, and the comment renders a histogram + cumulative-time diff
against `main` under the table, one per STSO kind. Percentiles alone
hide how many samples moved and by how much, which is exactly where the
variance lives. Inline rows use a fixed 50ms bin width (the adaptive
width is coarse enough to hide structure inside that cluster); queue-hop
rows keep the adaptive width. Negative gaps (clock skew between two step
bodies' clocks) get their own bucket rather than being counted with the
slow tail.
Raw samples are stripped from the comment's embedded data block — ~1000
per run would exceed GitHub's comment size limit within a couple of
history entries — so the histogram renders for the current run only,
while collapsed history keeps its tables. Until this lands on `main` no
baseline has raw samples, so the section renders this run's distribution
as a single series.
Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com>
* Clarify what stripping raw samples from the data block does not affect
Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com>
* Drop the bucket tables; fold counts and deltas into the histogram bars
Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com>
* Collapse the STSO distribution section into a dropdown
Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com>
* Fix footer assertion after the dropdown wording change
Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com>
---------
Co-authored-by: vercel[bot] <35613825+vercel[bot]@users.noreply.github.com>
* feat(core): emit faas.instance span attribute for compute instance identity
Synthesize a per-warm-instance id (cinst_<ulid>) once at module load and emit it as the OTEL faas.instance attribute on the flow and step route spans. Vercel exposes no native per-instance id under Fluid compute, so this lets traces distinguish which compute instance handled each request. Purely additive telemetry.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Alex Langenfeld <alex.langenfeld@vercel.com>
* test(core): assert faas.instance is stable across invocations
Covers the id format and, on the two-invocation warm-handler case, that both invocations report the same id — the module-scope minting contract that makes the attribute identify the instance rather than the invocation.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Alex Langenfeld <alex.langenfeld@vercel.com>
---------
Signed-off-by: Alex Langenfeld <alex.langenfeld@vercel.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
deleteJSON was the one mutation path in the fs layer that neither used
withWindowsRetry nor swallowed unlink errors. On Windows, unlink fails
with a share-violation EPERM while a concurrent reader briefly holds
the file open — hook polling races deleteAllHooksForRun by design — so
a transient EPERM surfaced as a failed operation, e.g. a failed
run.cancel().
Wrap the unlink in withWindowsRetry, matching the rename/link/unlink
guards the write pipeline already has. ENOENT is not in the retryable
set, so the already-deleted tolerance still short-circuits.
Signed-off-by: Andrew Barba <barba@hey.com>
github-actions[bot]