mirror of
https://github.com/vercel/workflow.git
synced 2026-09-14 19:59:43 +08:00
codex/atomic-start-core
1356 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Karthik Kalyan
|
27d0ce7904 |
Route preview benchmarks through the e2e server (#3274)
* Expose Workflow web server override Signed-off-by: Karthik Kalyanaraman <karthik.kalyanaraman@vercel.com> * Use neutral workflow server test URL Signed-off-by: Karthik Kalyanaraman <karthik.kalyanaraman@vercel.com> * Route preview benchmarks through e2e server Signed-off-by: Karthik Kalyanaraman <karthik.kalyanaraman@vercel.com> * Use an empty changeset Signed-off-by: Karthik Kalyanaraman <karthik.kalyanaraman@vercel.com> --------- Signed-off-by: Karthik Kalyanaraman <karthik.kalyanaraman@vercel.com> |
||
|
Nathan Rajlich
|
aa78a7e63c |
world-local: converge redelivered hook_received re-ensure on its committed resumeId claim (#3297)
A redelivered re-ensure of an already-committed resume (same runId + resumeId + digest) was rejected with HookNotFoundError when the hook had since been disposed by the workflow (dispose -> sleep releases the token while the run continues). The queue consumer treats HookNotFound as 'nothing left to resume' and acks the delivery — silently dropping whatever continuation the redelivered message carried and wedging the run. Check the (runId, resumeId) claim BEFORE the disposal/existence rejections: a committed claim whose pinned event is journaled proves this exact resume was accepted while the hook was alive, so return that event as success. Claims with a mismatched hookId or payload digest still fall through to full validation and are rejected as before, as are genuinely new resumes of a disposed hook. |
||
|
Mitul Shah
|
dc4cf944ae |
Align web-shared typography with Geist tokens (#3294)
* fix(web-shared): align typography with system tokens Signed-off-by: Cursor Agent <cursoragent@cursor.com> * test(web-shared): enforce typography tokens Signed-off-by: Cursor Agent <cursoragent@cursor.com> * refactor(web-shared): simplify typography audit Signed-off-by: Cursor Agent <cursoragent@cursor.com> * test(web-shared): cover typography guard failures Signed-off-by: Cursor Agent <cursoragent@cursor.com> * test(web-shared): narrow typography guard Signed-off-by: Cursor Agent <cursoragent@cursor.com> * test(web-shared): remove typography guard Signed-off-by: Cursor Agent <cursoragent@cursor.com> --------- Signed-off-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Cursor Agent <cursoragent@cursor.com> |
||
|
Nathan Colosimo
|
679dfa9c15 |
simplify hardened serialization: assert intrinsic captures, drop optional-capture states, close the bound-getter reporting hole (#3288)
- Every captured intrinsic exists on all supported engines (Node 18+), so the optional-capture layer (intrinsicGetter-returns-undefined, canReadUrl/ canReadUrlSearchParams/canReadHeaders, per-use fallbacks) is replaced by captures that throw at import if absent. - URLSearchParams.prototype.size (the one genuinely missing member on Node 18) is not needed: emptiness falls out of the captured toString() result, which the reducer already computes. Node 18 now serializes URLSearchParams natively instead of falling back to devalue's default handling. - The call/get tables and their re-export aliases flatten into direct typed exports; readProxyAware and the viewInfo getter-indirection unroll into two-branch functions. - isEngineAccessor: drop the WeakMap memo and try/catch (descriptor getters are always callable); exclude bound functions, which stringify as native code but run their target — previously workflow code could launder a side-effectful getter past the report with fn.bind() (test added). - 763 -> 625 lines, byte output unchanged (parity checked for DataView and typed-array subviews on top of the existing test suite). |
||
|
Alex Langenfeld
|
a799025af9 |
Surface a Request ID in the run sidebar attribute panel (#3293)
## Summary & Motivation One warm compute instance serves many invocations, so the Compute Instance ID already in this panel can't distinguish steps that ran inline within a single flow-function invocation — sibling steps sharing a Request ID did. It's also the value Vercel Logs indexes by, and this panel's View Logs button is where a reader takes it next. The key is `vercelId`, not `requestId`: that's the name world-vercel stores the SDK's request id under crossing the wire, and AnalyticsEvent's sibling `requestId` field is declared but never written. Nothing in this repository populates `vercelId` on the object the panel receives yet — only AnalyticsEvent carries it, and grouping steps by invocation needs a step-level aggregation in workflow-server first — so the row ships as forward-compatible plumbing in the slot next to Compute Instance ID. The second commit is an independent fix: `sortByAttributeOrder` guarded `indexOf` with `|| 0`, but a miss returns -1, which is truthy, so any key absent from `attributeOrder` sorted ahead of every listed key. It can be dropped on its own. ## Test Plan Unit tests added; the two ordering assertions fail against the unfixed comparator. The new row could not be verified by hand — no local code path populates it. |
||
|
Shalabh Chaturvedi
|
ba2cddc861 |
[benchmarks] Log the run id and Datadog trace for each sequential-steps run (#3248)
* [benchmarks] Link the run id and Datadog trace under the STSO histograms The STSO distribution section added in #3213 shows the shape of the sequential-steps run but not which run produced it, so investigating an odd-looking bucket meant hunting for the run by deployment id and time window. Capture the identity alongside the samples (the mechanism prototyped on the WIP variance branch, #3107): `/api/bench` returns the trace id of the span @vercel/otel opened for the trigger request, the runner threads it through the sequential iteration and records `sequentialRuns` in the result file, and the renderer prints one line under the histograms with the run id + Datadog trace link for this run and for the `main` run it is diffed against. Every part is optional — a deployment predating the route change yields a bare run id, and a `main` baseline predating this yields only this run's side — so the section degrades instead of breaking on mixed-vintage artifacts. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Co-Authored-By: Shalabh Chaturvedi <7066873+shalabhc@users.noreply.github.com> Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * Log the run/trace links instead of rendering them into the comment The run id and Datadog trace are debugging aids, not part of the benchmark's reported result, so they belong in the job's own output rather than in the PR comment body. Logging them where the runs are produced also makes them available in two cases the comment could never cover: a local `pnpm bench`, and a job that fails before the comment step runs. This drops the comment-rendering side entirely -- `renderSequentialRunLinks`, the `baselineSequentialRuns` baseline plumbing in `annotateWithBaseline`, and the `sequentialRuns` field on the result artifact, which existed only to carry the data to the renderer. The `main`-baseline side of the link goes away with it: which run produced the baseline histogram is only knowable at comment-render time, where the two artifacts are matched. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * Say what the trigger trace actually contains under linked mode The route comment claimed the trigger request's span "propagates into the workflow's own spans". That only holds under WORKFLOW_TRACE_MODE=continuous. Nothing in the workbench or benchmarks.yml sets the mode, so the benchmark deployment runs the default `linked` (packages/core/src/telemetry.ts), where each workflow/step invocation is its own trace root and the trigger's trace carries `workflow.start` plus span links out to those roots. The logged link is still the right entry point -- one hop through the links, which Datadog renders -- but the comment should describe that, so nobody opening a trigger-only trace while debugging a histogram concludes the run produced no spans. Raised by @TooTallNate in review of #3248. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Co-Authored-By: Shalabh Chaturvedi <7066873+shalabhc@users.noreply.github.com> Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> * Log a Datadog span search alongside the trigger trace link Under the default linked trace mode the trigger's trace holds only `workflow.start` plus span links, so opening it lands one hop away from the spans an STSO investigation needs. Log an APM search on `@workflow.run.id:<runId>` next to it, which goes straight to the run's execution spans. Both links are logged rather than one replacing the other: the search depends on `workflow.run.id` being an indexed span tag in the org, and the permalink works regardless. Suggested by @TooTallNate in review of #3248. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Co-Authored-By: Shalabh Chaturvedi <7066873+shalabhc@users.noreply.github.com> Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com> --------- Co-authored-by: vercel[bot] <35613825+vercel[bot]@users.noreply.github.com> |
||
|
Karthik Kalyan
|
31f92df10d |
Lazy hook resumption: parallel event write + queue publish (#3230)
* feat(core): lazy hook resumption via parallel event write + queue publish (rebased onto #1834 + #3145) Rebase of #3230 onto current main ( |
||
|
Peter Wielander
|
4174a6ea73 | [ci] Shrink the event-log race repro job 100x and add a local world-postgres runner (#3273) | ||
|
Nathan Rajlich
|
b732e91fac |
feat(core): side-effect-free serialization of workflow VM values (#3257)
* feat(core): side-effect-free serialization of workflow VM values
Serialization runs on the host but inspects values constructed inside the
node:vm sandbox, so ordinary dynamic operations dispatch into the sandbox
realm and execute workflow code: `value.toISOString()`, `Array.from(map)`,
`Object.prototype.toString` (via Symbol.toStringTag), `.source`/`.flags`,
`.href`, view `.buffer`/`.byteOffset`/`.byteLength`, and error
`.message`/`.stack`/`.cause` reads.
That is a determinism hazard. A payload is serialized exactly once and is
never re-serialized on replay, so any workflow-visible side effect it
triggers exists only on the live path — a patched `Date.prototype.toISOString`
that consumes a seeded `Math.random()` draw, for example, shifts every
subsequent draw and diverges from replay.
This makes serialization side-effect free where the data allows it, and
observable where it does not:
- Classification uses engine brand checks (node:util types, internal-slot
probes) instead of `instanceof global.X` and Object.prototype.toString, so
it is immune to Symbol.hasInstance, reassigned sandbox globals, and
Symbol.toStringTag spoofs. An unbranded value claiming a brand-decided tag
is now classified as a plain object instead of being routed into an
extractor that requires the real internal slot (unhardened devalue crashes
on that input).
- Extraction goes through intrinsics captured at module load — host boot,
before any workflow bundle runs — invoked with explicit receivers.
Internal slots are realm-agnostic, so host intrinsics read VM-realm
objects without touching the sandbox's patchable prototypes.
- Property access reads through descriptors, so plain data never invokes
anything.
Where workflow code must run because the data lives behind it — getters,
proxies, custom [WORKFLOW_SERIALIZE] methods, toString() on
toStringTag-branded objects like Temporal polyfills — the execution is
preserved for compatibility and recorded in a new `CodecOptions.guestCodeStats`
sink, surfaced as workflow.serialization.guest_code_{executions,details} span
attributes. Consumers that retain a VM across steps can treat a non-empty
report as "serialization may have perturbed VM state".
Engine-provided accessors are deliberately not reported: V8 defines `stack`
as an own accessor on every Error instance, so reporting it would flag every
serialized error. Nativeness is decided with the captured host
Function.prototype.toString; the bound-function caveat is documented in
hardened.ts.
Requires devalue 5.9.0 for the pluggable `operations` option.
* chore: shorten changeset
* fix(core): close review gaps in hardened serialization
Five correctness fixes, all with repros:
- Callable proxies were treated as engine accessors. V8 returns
`function () { [native code] }` from Function.prototype.toString for a
proxy around a function rather than throwing, so a proxy-wrapped getter
was cached as engine-provided and invoked unreported. Gate on
types.isProxy first.
- Host builtins implemented in JavaScript were reported as workflow code.
Node's DOMException.prototype.message/name are ordinary functions, so
the nativeness test failed and every serialized DOMException reported
two getter executions. They belong to the *host* realm, though, and
workflow code cannot author a host-realm function — so provenance is
now decided by nativeness OR host-realm `Function.prototype`, which are
disjoint and together cover both cases (V8 installs `stack` per realm,
so a VM error's getter is native but VM-realm).
- The extraReducers at the two VM call sites were still unhardened, and
they run on every value the earlier reducers do not claim — which is
exactly where the report has to be complete. `instanceof
global.ReadableStream/WritableStream/Request/Response` consulted
Symbol.hasInstance on the sandbox class (14 invocations for an ordinary
payload once the classes are patched), and AbortController's guard did
a bare `value.signal` read, so a non-enumerable `signal` getter ran
with an empty report. All five now walk the prototype chain and read
through descriptors.
- `__closureVarsFn` was invoked unreported on a purity argument that
nothing checked: the property is reachable from workflow code, which
can replace the compiler-generated function. step.ts now registers the
generated function as trusted when it builds the proxy, so provenance
is verified rather than assumed, and an unrecognized function is
reported.
- The URL/URLSearchParams test patched prototypes of *host* classes
injected into the sandbox, mutating them for the rest of the worker
process. Restored in a finally.
Also, per review:
- `dehydrateStepArguments` / `dehydrateWorkflowReturnValue` take an
optional GuestCodeStats out-param, so a retained-VM gate can consume
the report instead of it being spent on span attributes. The
report-completeness tests use it to exercise the real dehydrate path.
- Every intrinsic capture is now optional. The table is built at module
scope, so a missing member was an import-time crash of @workflow/core
rather than a degraded path; only SharedArrayBuffer was guarded, while
URLSearchParams.prototype.size (Node 19.8+) and the WHATWG classes were
assumed. Absent captures now make the corresponding reducer decline to
match.
- Documented that recording is not prevention (a recorded getter calling
Math.random() still advances the run's seeded PRNG), and that a
`{ kind: 'proxy' }` report implies a silent shape change (a proxied Map
serializes as a plain object).
- Parity coverage extended to DataView, boxed primitives, null-prototype
objects, setter-only properties, DOMException, AggregateError, an
accessor-valued Symbol.toStringTag, both RetryableError retryAfter
paths, and a WORKFLOW_SERIALIZE class instance.
* fix(core): keep identifying proxied host classes
Every Next.js e2e job failed on the two webhook tests: the hook POST
returned 404 because `resumeWebhook` could not serialize its step return
value ("Cannot stringify arbitrary non-POJOs"), so no hook was ever
registered.
The value was a `NextRequest`, which Next.js hands over as a **Proxy**.
`isInstanceOfPrototype` rejected proxies outright, so the Request reducer
answered "not a Request" and devalue fell through to the POJO check. The
reasoning behind rejecting them — that proxied built-ins were never
serializable, because internal-slot reads throw on a proxy receiver — is
true for `Map`/`Date`/`URL`, whose reducers read internal slots, but not
for `Request`/`Response`/streams, whose reducers read ordinary
properties. Next's proxy forwards those with the target as receiver, so
they serialized fine before this PR.
Identification now walks through proxies, matching `instanceof`, and
records the traps rather than suppressing the answer. The three reducers
that do read internal slots (URL, URLSearchParams, Headers) fall back to
the dynamic read when the value is a proxy, so their behavior is exactly
what it was before — including throwing for a bare proxy over a built-in,
which threw before too.
Verified against the real thing: the full nextjs-turbopack e2e suite
(135 tests) passes locally, having reproduced the failure first and
confirmed a reverted `serialization.ts` fixed it.
The regression test uses a receiver-correcting proxy, which is what makes
NextRequest work in practice; a comment records that a bare
`new Proxy(request, {})` throws on undici's private slots with or without
this change.
* fix(core): state what the closure-fn mark proves, and correct stale docs
- `isInstanceOfPrototype`'s JSDoc still described the behavior removed in
|
||
|
Pranay Prakash
|
ee944d2476 |
feat(core): stamp creator environment into runInput and reject cross-environment queue deliveries client-side (#3244)
* feat(core): stamp creator environment into runInput and reject cross-environment queue deliveries client-side `start()` makes two writes that have to land in the same tenant: the `run_created` event, attributed to whatever environment the caller authenticates as, and the queue message, pinned to a deployment. A misconfigured caller can split them — writing the run to one environment while addressing the message to a deployment in another. The consumer finds no run under its own tenant, the backend's resilient start (`run_started` creates the run when `run_created` was never seen) mints a second copy of the same run id in the consumer's environment, and both copies are real: the creator's sits pending forever while the other executes. The deployment id is not the discriminator — it matched end to end in the incident that motivated this. The environment is. So carry it: add an optional `World.getEnvironment()`, implement it in world-vercel from the same resolution that produces the `x-vercel-environment` header, and stamp it into the queue message's `runInput`. The consuming deployment already knows its own environment, so it can refuse the delivery itself with no server coordination — and refuse before `run_started`, the write that would create the fork. The refusal acks the message instead of throwing: the mismatch is baked into the message, so every redelivery would reach the same verdict and throwing would hot-loop until MAX_QUEUE_DELIVERIES. Both sides must be known for the check to run, so worlds with a single tenant (local, Postgres) and runs started by an older SDK behave exactly as before. A companion diagnostic logs a deployment-id mismatch without refusing, since deployment ids differ for benign reasons too. Signed-off-by: Pranay Prakash <pranay.gp@gmail.com> * fix(world-vercel): resolve the runtime environment from VERCEL_TARGET_ENV For a deployment in a Vercel custom environment, the OIDC token's environment claim is the custom environment's slug (the platform mints `customEnvironment?.slug ?? envTarget`) while VERCEL_ENV reports 'preview' — so keying the cross-environment guard on VERCEL_ENV could false-refuse a legitimate delivery, e.g. a CLI client attributed to 'staging' starting a run on the staging deployment. VERCEL_TARGET_ENV is populated from exactly the same slug-or-target pair as the claim, so prefer it, keeping VERCEL_ENV as the fallback for contexts that don't inject it. Also sorts runtime.ts imports per the Biome rule that landed on main in #3241. Signed-off-by: Pranay Prakash <pranay.gp@gmail.com> --------- Signed-off-by: Pranay Prakash <pranay.gp@gmail.com> |
||
|
Peter Wielander
|
1471f252fa | [core] Gate event creation on the loaded event count and restart replays in-process (#3145) | ||
|
Nathan Rajlich
|
438eaa6a59 |
Make resumeHook() resilient to transient hook_received event write failures (#1834)
* Make resumeHook() resilient to transient hook_received event write failures
When events.create('hook_received') fails with a retryable error (429/5xx),
resumeHook() now dispatches the queue message with a `hookInput` payload
carrying the dehydrated hook payload. The workflow runtime materializes the
missing hook_received event from that payload on its next delivery, mirroring
the existing resilient-start behavior of start() / run_created / run_started.
Returned Hook carries a new `resilientResume: true` flag when the fallback
path was taken. Both write paths share a client-minted `resumeId` as an
idempotency key so the runtime can dedup if the direct write actually
committed but the client saw a transient error.
Uses a sequential write-then-queue flow (not parallel) to avoid a dedup race
on the happy path: hook_received events have no entity-level conflict guard
(unlike run_created), so a duplicate written before the direct write commits
would double-deliver the payload to the workflow.
* Fix resilient resume: use local payload in materialized hook_received event
The server returns a 'lazy' response for hook_received event creation,
where eventData.payload may be a RefDescriptor (when the payload
exceeded the inline size and was offloaded to blob storage) rather
than the raw bytes. Pushing this directly to the in-memory events
array caused the workflow VM to fail with 'Invalid input' when trying
to deserialize the RefDescriptor as a Uint8Array.
Substitute the eventData we already have locally so the in-memory
event matches what getWorkflowRunEvents would return after
client-side ref hydration.
* Gate resilient resume on target runtime capability; carry hook token; export ResumedHook; docs
- Only take the resilient path when the target run's recorded
@workflow/core version understands hookInput on the queue payload.
Runs keep executing on the deployment they were created on (skew
protection), and older runtimes parse the queue message with a schema
that silently strips unknown fields - the resume payload would be
lost while resumeHook() reported success. Fail fast (propagate the
original event-write error) for such runs instead, preserving the
caller's ability to retry.
- Carry the hook token on hookInput and write it into the materialized
hook_received event so it gets the same replay-divergence guard as a
directly written event (#2030 parity).
- Export ResumedHook from @workflow/core/runtime and workflow/api.
- Add changelog page and update resumeHook() API reference docs.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Address review: correct capability cutoff, drop own-version escape hatch, replay-side resumeId dedup
Review fixes for the resilient-resume capability gate and dedup:
- Bump the supportsQueueHookInput cutoff to 5.0.0-beta.39: 5.0.0-beta.38 is
published WITHOUT this feature (its queue-payload schema strips hookInput),
so classifying it as capable would silently lose resume payloads. The
cutoff is now a single exported constant (QUEUE_HOOK_INPUT_MIN_VERSION)
with a TODO(release) requiring re-verification at merge time.
- Remove the own-version exact-match escape hatch entirely: version strings
do not identify builds (a published beta.38 and a main-built tarball can
share a version string while differing in content), so the check could
declare a featureless published deployment capable. Pre-release builds now
fall back to fail-fast until the version is bumped past the cutoff — the
safe direction. Tests simulate a capable target explicitly.
- Make duplicate suppression authoritative at the replay boundary: replay
now dedups hook_received events sharing a resumeId (same resume attempt),
so even when concurrent redelivery of the same queue message
double-materializes the event (no World enforces uniqueness on
hook_received), the payload reaches workflow code exactly once. This is a
pure function of the persisted log, keeping replay deterministic. The
runtime's snapshot check remains as best-effort write suppression, with
its comment corrected to say so; the EntityConflictError catch is kept as
the forward-compatible signal for planned server-side (runId, resumeId)
uniqueness, with its comment corrected to say it is defensive today.
- Stamp materialized hook_received events with occurredAt decoded from the
resumeId ULID so resiliently-resumed hooks are timestamped at resume time
rather than after the queue round-trip.
- Pin the cross-version compat contract in a test: the direct write is
resumeId-only (no digest or negotiation fields), which later server-side
idempotency work must keep accepting.
- Exercise the published boundary (5.0.0-beta.38) in fail-fast tests, and
make the capability tests self-check against the exported cutoff constant
instead of restating literals.
- Docs: changelog date June -> July 2026, dash consistency, and document the
replay-side dedup guarantee.
* Encode release-gate and successor-rebase contracts into code comments
Comment-only changes capturing the review agreements so they survive the
parallel-resume successor rebase (no behavior change):
- capabilities.ts: the QUEUE_HOOK_INPUT_MIN_VERSION re-verification point
is the actual combined SDK release (after the successor lands and its
server-side dedup is deployed), not source-merge time — this PR merges
source-only and no SDK is published from it alone. Every Version
Packages merge in between moves the earliest possible carrier.
- workflow/hook.ts + runtime.ts: scope the replay-side resumeId dedup
honestly as defense-in-depth over the persisted log, not a
cross-invocation exactly-once guarantee — concurrent invocations
replaying pre-duplicate snapshots each see only their own row; the
storage-level (runId, resumeId) constraint in the successor work is the
correctness boundary. The set stays useful post-constraint for logs
written before it deployed.
- runtime.ts: document the EntityConflictError swallow's known gap while
the branch is defensive (this invocation's local log lacks the payload;
progress relies on the other writer's delivery or redelivery) and pin
the rebase contract for when the constraint makes it live: a matching
claim must append the canonical event locally and succeed; a real
conflict must rethrow for redelivery.
- resume-hook-resilient.test.ts: reframe the wire-shape pin as a tripwire
rather than a permanent contract — the successor deliberately widens it
(ID/digest pair + attestation) before any SDK release, so the
resumeId-only shape never ships as a published server contract.
---------
Co-authored-by: Peter Wielander <peter.wielander@vercel.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
|
||
|
Alex Langenfeld
|
4017597a5f |
feat(core): report replay divergence recovery (#3208)
Signed-off-by: Alex Langenfeld <alex.langenfeld@vercel.com> |
||
|
Pranay Prakash
|
11dc036854 |
ci: stop deploying changeset-release/main, run its e2e against production (#3243)
* ci: stop deploying changeset-release/main, run its e2e against production The changesets action force-pushes `changeset-release/main`, and it can point at exactly main's HEAD SHA. Vercel keeps one commit status per project per SHA, so when both a production deployment (from main) and a preview deployment (from changeset-release/main) are built for the same commit, whichever finishes last owns the status. On 2026-07-30 the preview finished last, so `vercel/wait-for-deployment-action` — which reads the deployment ID out of that status — handed production e2e runs a preview deployment ID and forked runs across environments. Disable git deployments for that branch in every Vercel project rooted in this repo, and give the changeset PR's Vercel e2e lanes a deployment to test that actually exists: main's production deployment for the PR's base SHA, resolved by SHA so a mid-flight production build is waited out rather than silently replaced by an older one. Signed-off-by: Pranay Prakash <pranay.gp@gmail.com> * ci: resolve changeset-release e2e deployments with the wait action, tokenless Per review: with changeset-release/main no longer deployed, main SHAs can never again be deployed to a second environment of these projects, so the per-SHA commit status the action reads is unambiguous for exactly this lane. Reuse vercel/wait-for-deployment-action with environment: production and sha pinned to the PR base SHA instead of the Vercel-API polling script, drop the script and its VERCEL_TOKEN usage, and inherit the action's inactive/skipped-build handling. Signed-off-by: Pranay Prakash <pranay.gp@gmail.com> --------- Signed-off-by: Pranay Prakash <pranay.gp@gmail.com> |
||
|
Andrew Barba
|
2677653759 |
fix(world-local): bound stalled queue deliveries (#3255)
Signed-off-by: Andrew Barba <barba@hey.com> |
||
|
Peter Wielander
|
a54f2b1486 | Sort imports in runtime.ts and step-executor.ts (#3241) | ||
|
Nathan Rajlich
|
32ac8e73fd |
Fix Biome lint violations and add Biome CI check (#3222)
* Fix Biome lint violations and add Biome CI check Biome was not configured to respect .gitignore, so ~92% of the 13,355 reported diagnostics came from gitignored build artifacts. Enable VCS integration (useIgnoreFile), apply safe auto-fixes across the repo, fix the remaining mechanical errors by hand, downgrade judgment-call a11y / dangerouslySetInnerHTML rules to warnings, and add a 'biome ci' job to the Lint workflow so violations block PRs going forward. * Use an empty changeset (no behavior change, no release needed) |
||
|
Alex Langenfeld
|
4a9d26b1cb |
feat(world): persist the compute instance that ran each step attempt (#3186)
* feat(world): persist the compute instance that ran each step attempt Add CreateEventParams.computeInstanceId (ambient per-event identity, mirroring requestId) and a readable Event.computeInstanceId. Core stamps it on every step_started write; world-vercel forwards it in the v4 frame meta next to vercelId. Lets observability distinguish steps sharing a compute instance from those on different instances or invocations. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Alex Langenfeld <alex.langenfeld@vercel.com> * test: cover computeInstanceId threading from params to v4 frame meta world-vercel: computeInstanceId reaches the v4 frame meta, rides alongside vercelId rather than replacing it, and is omitted when unset. core: step_started carries it without displacing the stateUpdatedAt precondition guard (both share one params object). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Alex Langenfeld <alex.langenfeld@vercel.com> * fix(web-shared): render computeInstanceId in the attribute panel AttributeKey derives from keyof Event, so adding computeInstanceId to the event schema widened it and left the exhaustive attributeToDisplayFn map incomplete (TS2741). Renders it beside deploymentId as 'Compute Instance ID', copyable like the other opaque ids. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Alex Langenfeld <alex.langenfeld@vercel.com> * fix(world-postgres): exclude computeInstanceId from the events column contract The events table asserts satisfies DrizzlishOfType<...Omit<Event, 'occurredAt'>...>, so adding computeInstanceId to the event schema broke the build (TS1360). This world does not persist it, matching how occurredAt is already handled. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Alex Langenfeld <alex.langenfeld@vercel.com> * refactor: move computeInstanceId to the analytics read contract The server routes computeInstanceId into ClickHouse and returns it on AnalyticsEvent/AnalyticsStep, never on the event record — so Event.computeInstanceId was dead on read and zod would strip the field off the analytics wire. Move it to AnalyticsEventSchema/AnalyticsStepSchema (beside vercelId/requestId, the same class of ambient provenance), which also drops the world-postgres column-contract exclusion entirely. Also: hoist the duplicated step_started params into one local, extract the repeated mock-agent harness in events.test.ts, and use vi.spyOn plus an identity assertion against COMPUTE_INSTANCE_ID. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Alex Langenfeld <alex.langenfeld@vercel.com> --------- Signed-off-by: Alex Langenfeld <alex.langenfeld@vercel.com> Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Shalabh Chaturvedi
|
8bda7cef79 |
[benchmarks] Split STSO by inline vs queue-hop steps, add distribution diffs vs main (#3213)
* Split STSO by inline vs queue-hop steps, add distribution diffs vs main
The sequential-steps benchmark's STSO metric mixed two unrelated
phenomena: gaps between steps running back-to-back in the same warm
process, and gaps across an invocation boundary (queue dispatch, client
reinit, event-log replay), which cost ~10x more. The old step-index
windows (1-20 / 101-120 / 1001-1020) sampled 19 gaps each and captured
neither cleanly: whether a boundary happened to land inside a window
moved that window's P99 by hundreds of percent, which is most of the
run-to-run variance the benchmark comment was reporting.
The workflow now tags each step with whether it was the first step body
executed in its process ('queue-hop') or a later one in the same warm
process ('inline') via a process-global, so the split is ground truth
rather than inferred from step index or trace timestamps. STSO is
reported as two rows over *every* gap in the run instead of three
sampled windows. No targets on the new rows — the old ones described the
index-bucketed grouping.
computeStats now keeps the full sorted sample array alongside the
percentiles, and the comment renders a histogram + cumulative-time diff
against `main` under the table, one per STSO kind. Percentiles alone
hide how many samples moved and by how much, which is exactly where the
variance lives. Inline rows use a fixed 50ms bin width (the adaptive
width is coarse enough to hide structure inside that cluster); queue-hop
rows keep the adaptive width. Negative gaps (clock skew between two step
bodies' clocks) get their own bucket rather than being counted with the
slow tail.
Raw samples are stripped from the comment's embedded data block — ~1000
per run would exceed GitHub's comment size limit within a couple of
history entries — so the histogram renders for the current run only,
while collapsed history keeps its tables. Until this lands on `main` no
baseline has raw samples, so the section renders this run's distribution
as a single series.
Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com>
* Clarify what stripping raw samples from the data block does not affect
Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com>
* Drop the bucket tables; fold counts and deltas into the histogram bars
Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com>
* Collapse the STSO distribution section into a dropdown
Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com>
* Fix footer assertion after the dropdown wording change
Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com>
---------
Co-authored-by: vercel[bot] <35613825+vercel[bot]@users.noreply.github.com>
|
||
|
Alex Langenfeld
|
9cc11f5329 |
feat(core): emit faas.instance span attribute for compute instance identity (#2989)
* feat(core): emit faas.instance span attribute for compute instance identity Synthesize a per-warm-instance id (cinst_<ulid>) once at module load and emit it as the OTEL faas.instance attribute on the flow and step route spans. Vercel exposes no native per-instance id under Fluid compute, so this lets traces distinguish which compute instance handled each request. Purely additive telemetry. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Alex Langenfeld <alex.langenfeld@vercel.com> * test(core): assert faas.instance is stable across invocations Covers the id format and, on the two-invocation warm-handler case, that both invocations report the same id — the module-scope minting contract that makes the attribute identify the instance rather than the invocation. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Alex Langenfeld <alex.langenfeld@vercel.com> --------- Signed-off-by: Alex Langenfeld <alex.langenfeld@vercel.com> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Andrew Barba
|
f05f642e89 |
[world-local] Retry transient EPERM unlink failures on Windows (#3215)
deleteJSON was the one mutation path in the fs layer that neither used withWindowsRetry nor swallowed unlink errors. On Windows, unlink fails with a share-violation EPERM while a concurrent reader briefly holds the file open — hook polling races deleteAllHooksForRun by design — so a transient EPERM surfaced as a failed operation, e.g. a failed run.cancel(). Wrap the unlink in withWindowsRetry, matching the rename/link/unlink guards the write pipeline already has. ENOENT is not in the retryable set, so the already-deleted tolerance still short-circuits. Signed-off-by: Andrew Barba <barba@hey.com> |
||
|
Peter Wielander
|
c93f6f7bd0 | [world-vercel] Raise H2 receive windows on the events agent (#3212) | ||
|
github-actions[bot]
|
b12f248b66 |
Version Packages (beta) (#3185)
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> |
||
|
Mitul Shah
|
e181f64b72 |
Align Streams UI with trace viewer (#3197)
* Align streams UI with trace viewer Co-authored-by: Cursor <cursoragent@cursor.com> * cleanupp --------- Co-authored-by: Cursor <cursoragent@cursor.com> |
||
|
Nathan Rajlich
|
b92c23ccb4 | fix(core): don't observe idle while a committed delivery is parked behind its deferral (#3198) | ||
|
Peter Wielander
|
34975f6b7d | [world-vercel] Make HTTP/2 actually multiplex on the events path (#3190) | ||
|
Pranay Prakash
|
25715d4521 |
[RFC] feat(nitro): embed observability dashboard in-process at /_workflow (#2548)
* feat(nitro): embed observability dashboard in-process at /_workflow Serve the @workflow/web observability UI inside the Nitro process at a configurable route (default /_workflow) instead of spawning a separate web server and 302-redirecting to it. Enabled in dev, omitted from production builds by default (so prod bundles carry no @workflow/web import). Never mounted on Vercel deploys (use the hosted dashboard). - @workflow/web: add a framework-neutral `@workflow/web/handler` (createWorkflowWebHandler) that serves SSR + static client assets + RPC as one Web Request->Response handler under a runtime basename (asset manifest URLs + publicPath are reprefixed so the dashboard is self-contained under its mount). Add `@workflow/web/registry` for embedded-dashboard discovery; make the RPC/stream client basename-aware. - @workflow/nitro: mount the handler in-process (Nitro v2 h3 + v3 native paths), gated by a new `dashboard` option (default = dev). - @workflow/cli: `workflow web` / `inspect --web` defer to a running embedded dashboard instead of starting a redundant server; pass `--standalone` to force the standalone UI. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * refactor(nitro): normalize dashboard path once, use isNitroV2() helper Address review feedback on the embedded dashboard: - Normalize the dashboard mount path in one place before it feeds both the Nitro route registration (`[path, path + '/**']`) and the handler `basename`. Force a single leading slash, strip trailing slashes, and reject the root mount, so a custom `path` can't make the route and the handler's internal `normalizeBasename` disagree. - Replace the handler-level `!nitro.routing` v2 checks with the existing `isNitroV2()` helper for consistent v2/v3 detection. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> Co-authored-by: Peter Wielander <mittgfu@gmail.com> |
||
|
Douglas Harcourt Parsons
|
e8934ade9c |
Remove self-attribution from README (#3182)
Signed-off-by: Douglas Harcourt Parsons <dglsparsons@users.noreply.github.com> |
||
|
Peter Wielander
|
a09d00135b | Revert "Statically inject workflow world target" (#2752) (#3142) | ||
|
github-actions[bot]
|
741a0d9eaf |
Version Packages (beta) (#3087)
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> |
||
|
Pranay Prakash
|
2941b1c360 |
fix(core): order step-result deliveries against wait/hook deliveries by event-log position (#3139)
* fix(core): order step-result deliveries against wait/hook deliveries by event-log position Two production runs on `@workflow/core@5.0.0-beta.36` burned all three divergence-recovery replays at the same event and terminated with CORRUPTED_EVENT_LOG: wrun_41KYJENABV0GSF5YTE9EETV5DD (step vs wait) wrun_41KYJEE01S0GPC9RWT5MEKVCX8 (step vs hook) Replay divergence: step event step_created for step_X belongs to "A", but the current step consumer is "B" `useStep` proxies draw deterministic ULIDs in invocation order, so the ULID -> stepName allocation is a function of the order in which promise resolutions are delivered to workflow code. The delivery-barrier registry pinned that order to event-log position for hook payloads and wait completions, but step results were delivered straight off the serial `promiseQueue` — and their latency varies between replays of the SAME invocation, because the first replay pays full hydration while later replays memo-hit primitive results in the shared `ReplayPayloadCache`. A step completion adjacent in the log to a `wait_completed` was therefore delivered wait-first on a cold replay and step-first on a warm one; whichever order the invocation that wrote the follow-up `step_created` events happened to see became law, and every replay computing the other order diverged permanently. Step results and step failures now register a 'step' delivery barrier at their event-log index and resolve from a detached continuation after every relevant earlier-in-log delivery, mirroring the hook payload path: hydration stays inside the serial queue slot (which also releases `pendingDeliveries`), while the barrier wait and the resolve run off the queue so a queue slot never blocks on a resolution the queue itself drives. Waits and hook payloads likewise defer behind earlier step results. Two details are what actually make the ordering hold, and both were found by testing rather than by reading the code: The deferral set is captured while CONSUMING the event, not at the start of the hydration slot. Captured at slot start it is not merely less deterministic, it is usually empty: an earlier delivery whose own slot runs first on the serial queue has typically already resolved and deregistered its barrier before the later slot begins, so the later delivery does not defer at all. Every event in one drain window is consumed before any slot runs, so consumption time sees all of them. A delivery that had to wait then yields a macrotask before resolving. An earlier delivery being "delivered" only means its `resolve()` ran; the branch it woke may need arbitrarily many further microtask hops before it reaches its next `useStep` call (a `for await` over a hook resumes the generator, settles the promise from `next()`, and only then runs the loop body). Ordering the `resolve()` calls alone therefore buys a fixed hop or two of margin and leaves a hop-count race that holds only for the shortest consumers; yielding a macrotask lets the earlier branch drain completely, whatever its shape. One asymmetry is load-bearing: a step result skips any earlier delivery that will not resolve on its own, i.e. one blocked directly or transitively on a buffered hook payload no consumer has claimed. Such a payload is delivered only when the workflow next reads the hook, and reaching that read commonly requires the step result itself, so gating the step on it stalls the run until the barrier's idle safety net fires — which then releases every delivery queued behind that payload at once and loses the very race the ordering exists to protect. Waits and hooks keep gating on unclaimed payloads, where waiting for the claim IS the guarantee. Tests come in two files. `step-delivery-ordering.test.ts` is byte-identical to the file in the repro-only companion PR vercel/workflow#3137 apart from two `it.fails` markers there (which let a repro-only branch have green CI); `sed 's/it\.fails(/it(/g' | cmp` verifies it. Each of its five cases replays one committed log twice through a shared `ReplayPayloadCache`, and the two warm-replay cases fail on main with the production error text. `step-delivery-hop-count.test.ts` exists because those five cases cannot tell "delivered in log order" apart from "resolves a hop or two later than before". It replays logs a live run legitimately produced — the live invocation received the two events in separate deliveries, so the first branch finished long before the second event existed — while the replay receives both in one drain window, and pads the consumer with a varying number of extra awaits so hop count is the only variable. It covers step results against both wait completions and hook payloads, plus step FAILURES against wait completions, since a rejection decides whether a `catch` continuation runs and so which ULID the `useStep` there draws. All 18 cases fail on main; of the 12 that predate the macrotask, 9 still fail with the resolve-ordering-only version of this fix; all 18 pass here. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Pranay Prakash <pranay.gp@gmail.com> * fix(core): close remaining delivery-barrier ordering gaps Follow-up on the step-delivery barrier work, addressing three cases the registry did not yet cover. Each has a regression test in the new `delivery-barrier-coverage.test.ts` that reproduces the production `ReplayDivergenceError` when its fix is reverted. - Step results now defer behind earlier STEP results. The old exclusion assumed the serial `promiseQueue` fixes step-vs-step order, which stopped holding once a step began resolving from a detached continuation instead of its queue slot: two steps consumed in different drain windows can disagree on their deferral set, and the earlier one — parked on the macrotask yield — gets overtaken. - `sleep.ts` and `hook.ts` (waiting-consumer path) now capture their deferral at event-consumption time, as `step.ts` already does. Reading the registry after their queue work misses an earlier step or hook that delivered and retired its barrier in the meantime, skipping both the gate and the macrotask yield. The buffered hook payload path deliberately keeps evaluating at claim time; a consumption-time snapshot there stalls the e2e `hookWithSleepWorkflow`. - Abort deliveries participate in the registry. `_setAborted` fires the signal's listeners, which may invoke a step and draw a ULID, so an abort is as branch-deciding as any other delivery. Also memoizes `resolvesOnItsOwn`. The walk is exponential in the number of live hook/wait barriers, and the registry is not bounded — a fan-out of `Promise.race([hook, sleep])` branches accumulates one barrier per branch per kind (49 measured for 24 branches). At 40 barriers a single scan took 92s before, and is instant after. --------- Signed-off-by: Pranay Prakash <pranay.gp@gmail.com> Co-authored-by: Claude Fable 5 <noreply@anthropic.com> Co-authored-by: Nathan Rajlich <n@n8.io> Co-authored-by: Peter Wielander <mittgfu@gmail.com> |
||
|
Nathan Rajlich
|
0d6ec43877 |
Pass the webhook request through without buffering its body (#3166)
The Astro and SvelteKit webhook wrappers copied the incoming request via `normalizeRequest()`, which buffers the whole body with `arrayBuffer()`, before calling the handler that validates the webhook token. Requests carrying an unknown token therefore did unnecessary work before being rejected. The copy turns out to be unnecessary: both frameworks already hand the route a standard `Request`, so the webhook wrappers now pass it straight to the handler. The body is left untouched until `resumeWebhook()` has accepted the token. The flow route keeps `normalizeRequest()` for now — it authenticates via the queue trigger rather than a URL token, so the ordering does not matter there, and whether the shim is needed at all is a separate question. Note that the Astro dev server buffers request bodies upstream of the route handler, so the new behavior is only observable in built output; the node adapter and Vercel builds both benefit. |
||
|
Nathan Colosimo
|
5d17c609b1 |
Reuse one event-log deduplication helper (#3110)
* refactor(core): reuse event log merge helper * refactor(core): simplify event merge helper |
||
|
Nathan Colosimo
|
7959acc8cf | Remove deprecated setAttributes aliases (#3128) | ||
|
Peter Wielander
|
49276f2d0b | [utils] Fix vercel world not being selected when running build on external CI (#3144) | ||
|
Karthik Kalyan
|
d24c91cfde |
feat(core): resume hooks from stored resumeContext and seal to the run key (#3125)
Hooks can carry an optional `resumeContext` mirrored from the run at
creation time. When present, `resumeHook`/`resumeWebhook` resume directly
from it instead of fetching the full run, saving a round trip per resume.
When the context also carries the run's `encryptionPublicKey`, the resume
seals its payload (`encp`) directly to that key. Combined with the sealed
envelope work (#3093-#3096), a default webhook resume then needs neither a
run read nor a cross-deployment run-key lookup: the key is resolved only
when the hook actually stores metadata that must be hydrated symmetrically.
Everything falls back transparently to the full run fetch and symmetric
key when the context (or the public key within it) is absent, so new
clients interoperate with old servers and vice versa.
- world: optional `encryptionPublicKey` on `HookResumeContext`
- world-postgres: `resume_context` column migration
- core: combined fast-path + seal in resume-hook; fast-path control-flow
suite split from the real-serialization crypto suite
- world-vercel: cover the `getEncryptionKeyForRun(runId, { deploymentId })`
overload the fast path relies on
- web-shared: render `resumeContext` in the attribute panel
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
|
||
|
Peter Wielander
|
62c01d94b0 | [e2e] Report partial results when the event-log race repro is cut short (#3148) | ||
|
Nathan Rajlich
|
b8bcded9d9 |
fix(core): seal forwarded writes for workflow-body writables (#3155)
`getWritable()` resolves to two different functions. In a step it publishes the run's X25519 public key on the handle; in a workflow body it returns only a name, because the workflow VM holds no key material by design. Forwarding a workflow-body handle to another run therefore produced a descriptor with no public key, and the receiving run fell to tier 2/3 of `getForwardedWritableEncryptionKey`: fetch the owner's symmetric key over the `run-key` API. That is the round trip sealing exists to remove, and it also hands the writer material that could decrypt — the tiers 2/3 comment calls that an honor-system restriction, where sealing makes it cryptographic. Nothing warned; the data was correct, just slower and over-privileged. The only visible difference was an `encr` rather than `encp` prefix in storage. Publish the key when a step revives such a handle. That is the first point where the owning run's key is in scope, and it cannot be deferred to serialization: `start()` dehydrates its arguments with the CHILD's runId and key, so by then the owner's key is gone. Guarded on the stream belonging to the current run — stamping our key onto another run's stream would make the receiver seal to the wrong recipient and lock the real owner out. This is the shape eve uses: a driver workflow takes a writable in its workflow body and forwards it into per-turn child runs started with `deploymentId: 'latest'`, so the sealed path was not engaging for the workload that motivated it. |
||
|
Nathan Rajlich
|
a86035f71f |
feat: return the run public key from the capability probe (#3099)
* feat: return the run public key from the capability probe
This removes the last key-lookup request from the cross-deployment hot
paths. `start()` already blocks on a capability probe for every
cross-deployment call, and the probe responder executes *inside the target
deployment*, where the run's key material is available locally. So the
public key can ride back on a response the caller is already awaiting, at
no additional latency, and the `run-key` API request disappears.
Three properties make this better than keeping the request:
- The wait is already being paid. Folding the key into the existing
response removes the request outright rather than relocating it.
- A public key is exactly what this channel can carry. The probe response
stream is deliberately unauthenticated, which would disqualify shipping
the symmetric key over it — but a public key is not secret.
- It reduces privilege. The caller ends up able to seal the workflow
arguments but not read them back; fetching the symmetric key granted
full read access to a run it merely launched.
`runId` is now minted before the probe rather than just after it.
`createRunId()` reads only `opts`, which is fully resolved by that point,
so the move has no other dependency — and a test asserts the id sent to
the probe is the one actually created.
Everything is best-effort. The probe is already failure-tolerant (2s
timeout, errors swallowed) and is skipped entirely for same-deployment
starts and for worlds without a streams API. When no key comes back — old
target, timeout, encryption disabled, or a malformed value — `start()`
falls back to the existing lookup plus symmetric encryption. Key
derivation failures inside the responder are caught and logged so the
probe still reports health and capabilities, which callers depend on for
reasons unrelated to encryption.
* fix: keep the health-check discriminator on runId-bearing probes
`QueuePayloadSchema` is an ordered union and `z.object` strips keys the
matching member doesn't declare. Adding an optional `runId` to
`HealthCheckPayloadSchema` made a probe payload also satisfy
`WorkflowInvokePayloadSchema`, whose only required field is `runId`. Because
the invoke member came first, world-vercel's queue handler parsed a
runId-bearing probe down to `{ runId }`, dropping `__healthCheck` and
`correlationId`.
The runtime dispatches on `__healthCheck` before falling through to the
invoke schema, so the probe was reinterpreted as "replay this run": it POSTed
`run_started` for a run that does not exist yet, 404'd, failed the handler,
and retried indefinitely. The probe never answered and the cross-deployment
`start()` timed out — which also regressed the pre-existing capability
detection, not just the new key lookup.
Order the health-check member first; it requires `__healthCheck: true`, which
no invoke or step payload carries, so invoke and step payloads still resolve
to their own members.
Also reorder `getPhysicalQueueName` to match health checks before the runId
branch, so under `WORKFLOW_SEQUENTIAL_REPLAYS=1` a probe keeps its per-probe
topic instead of queueing behind the run it is preparing.
|
||
|
Nathan Rajlich
|
8a95d36c9a |
feat(core): seal forwarded stream writes to the owner's public key (#3098)
* feat(core): seal forwarded stream writes to the owner's public key
When a parent forwards a `WritableStream` into a child run, the child
writes to the parent's server stream and must encrypt with the parent's
key. The descriptor carried `{name, runId, deploymentId}` — enough to
resolve the parent's *symmetric* key, which cross-deployment means the
same ~350ms `run-key` round trip the rest of this work removes.
Carrying the parent's public key in the descriptor closes that hole. The
parent owns the stream and already has its key material resolved on the
step context when it creates the handle, so deriving and stamping the
public key there costs nothing, and the child can seal immediately.
Resolution is now three tiers, cheapest first:
1. descriptor has the owner's public key → seal, zero I/O
2. descriptor has the owner's deployment → resolve symmetric key (API call)
3. neither (older SDKs) → load the owning run, then resolve its key
Tiers 2 and 3 import the key encrypt-only, which is an honor-system
restriction — the same bytes could decrypt. Tier 1 makes it a real
guarantee: a public key cannot read anything.
**On nonce discipline.** An earlier sketch amortized one KEM across a
whole stream and used counter nonces. That is unsafe here: a stream
reconnect or a durable replay restarts the writer, and a counter would
restart at zero under a still-live content key, repeating `(key, nonce)`
— which under AES-GCM leaks the plaintext XOR and the auth subkey. This
implementation instead seals each frame independently, so no content key
outlives a single frame and the hazard cannot arise by construction. The
cost is one ECDH and 32 bytes per frame; `encapsulate`/`decapsulate`
remain available if profiling later justifies amortizing, but doing so
would need connection-scoped nonce rules to stay safe.
A regression test asserts 20 frames of identical plaintext produce 20
distinct ephemeral keys and 20 distinct ciphertexts.
* review: assert the amortized-KEM invariant for forwarded streams
The forwarded-stream test asserted a distinct ephemeral key per frame,
which held when each frame was sealed independently. Now that the KEM is
amortized per writer, the invariant that actually matters is different and
the test says so: all frames of one stream share an ephemeral key, yet
identical plaintext still yields distinct ciphertext (nonces stay random),
and a second writer incarnation — a reconnect or durable replay — gets a
different ephemeral key rather than inheriting the previous content key.
* fix: keep the owner public key when reviving a forwarded writable
Forwarding a writable to another run is two hops, not one: `start()` hands the
parent's handle to the child WORKFLOW, and the child workflow then hands it to
the step that writes. The handle is revived and re-serialized in between.
The revivers re-attached the stream name, runId and deploymentId but not the
owner's X25519 public key, so it was dropped on that middle hop. The step then
found no key on the descriptor and fell back to fetching the owner's symmetric
key — reintroducing exactly the round trip sealing exists to remove. Sealing
therefore never engaged for `start()`-forwarded streams, while the single-hop
unit test still passed.
Re-attach the key at all three reviver sites, and cover the two-hop shape so
the end-to-end path is tested rather than just one serialization round.
|
||
|
Nathan Rajlich
|
0b1ca15dec |
feat(core): seal hook payloads to the target run's public key (#3096)
* feat(core): seal hook payloads to the target run's public key This is the payoff for the sealed-box work: cross-deployment `resumeHook()` no longer calls `getEncryptionKeyForRun`, which on Vercel means a ~350ms `run-key` API round trip (Cosmos reads, a 50–75KB `encrypted_env.json` fetch from S3, up to three KMS decrypts) to recover 32 bytes. When the target run publishes a public key, the resumer seals to it using only the run entity it already fetched. More than half of `resumeHook()` calls miss the same-deployment fast path, so this is the dominant term in hook-resumption latency — and for Eve, hook resumption is what an agent turn waits on. It also reduces privilege. Fetching the symmetric key grants read access to everything in the run; sealing grants only the ability to write one payload to it. A resumer is now cryptographically unable to read the run it resumes. Sealing is gated on the presence of `encryptionPublicKey`, deliberately *not* on the capability version table the way `encr` and `gzip` are. A run only carries a public key if the runtime that created it could also open a sealed payload, and runs are pinned to their creating deployment — so presence is a stronger attestation than comparing versions, and it stays correct when `@workflow/core` and `@workflow/world-vercel` versions drift independently (each single-axis version gate wedges a run under one drift direction; presence wedges under neither). `resumeWebhook` keeps using the symmetric key it already had to fetch in order to hydrate hook metadata — sealing there would add an ECDH without saving a round trip. **Read path.** Sealing is useless if the run cannot open the result, and every reader previously resolved a bare symmetric `CryptoKey`, which by construction cannot open `encp`. Key resolution now yields the full capability (symmetric key + X25519 keypair) at every read site: `memoizeEncryptionKey`, the `Run` class, `runs.ts`, and `getHookByTokenWithKey`. Without this the first sealed hook payload would have wedged its run with `RuntimeDecryptionError`. Holder types widen from `CryptoKey` to `PayloadKey`, which is a pure widening. Falls back to the symmetric path when the run has no public key (older SDKs), when the stored value is malformed, or when encryption is off — all covered by tests, along with an end-to-end assertion that a sealed payload actually hydrates with the keys the owning deployment re-derives. * review: correct stale docs on the key-resolution path - `memoizeEncryptionKey`'s JSDoc still described importing an AES-256 `CryptoKey`. It now resolves a run's full capability (symmetric key plus X25519 keypair), and the reason matters: a run reading its own event log can meet sealed payloads another run wrote to it, and resolving only the symmetric key would leave those unopenable. Documented explicitly so the next reader does not "simplify" it back. - A comment in `resumeHook` pointed at the `encr` capability gate as being "above" when it is in the fallback branch below. Reworded. - Added the `decodeRunPublicKey` boundary test that belongs with the strict base64 work but needed this branch's code to exist. |
||
|
Peter Wielander
|
04e5ec9873 | [e2e] Rebuild the event-log corruption repro around step-count divergence (#3147) | ||
|
Nathan Rajlich
|
e8bc7d6aad |
feat: decrypt sealed payloads in the dashboard and CLI (#3146)
* feat: decrypt sealed payloads in the dashboard and CLI Without this, any payload another run sealed to this one renders as a lock icon with no way to open it — a visible regression for anyone debugging a run that received a cross-deployment hook resumption. The user is entitled to read the data and has already supplied the key; only the plumbing was missing. `hydrateDataWithKey` now delegates to the envelope layer, which dispatches on the format prefix, instead of unconditionally running AES-GCM. All four o11y key-resolution sites (web-shared hydration, the web stream reader, and both CLI `--decrypt` paths) resolve the full capability rather than just the symmetric key. Each already had the raw 32 bytes in hand, so this costs one extra derivation and no additional requests. A caller that supplies only a symmetric key still gets the ciphertext placeholder for sealed payloads rather than a decryption error, since that key never could have opened them. **Browser bundling.** The obvious import for the new helper is `@workflow/core/serialization`, but that module graph reaches `node:util` and `node:async_hooks` and cannot be bundled for the browser — which is what `@workflow/core/serialization-format` exists to avoid. The key helpers are re-exported from that browser-safe entrypoint instead, and the two browser consumers import from there; the CLI keeps the direct import since it runs on Node. Verified by walking the built import graph: the entrypoint reaches 6 modules and zero Node built-ins. Unrelated: `pnpm --filter @workflow/web build` currently fails on `main` too (`reducers/common.js` importing `node:util`). Turbo caching had been hiding it; touching core caused a cache miss that surfaced it. Not addressed here. * review: narrow the o11y decrypt key type and dedupe an import - `hydrateDataWithKey` accepted `PayloadKey`, which includes `SealTarget`. A seal target holds only a public key, so it can open neither scheme — passing one compiled fine and then always failed at runtime. Added a `DecryptionKey` alias (`CryptoKey | RunPayloadKeys`) and narrowed the signature, so that misuse is now a compile error. A `@ts-expect-error` test pins the guarantee. - `hydrateResourceIOAsync` dynamically imported `@workflow/core/serialization-format` twice. Destructure both bindings from the single existing import instead. * review: record @workflow/web in the changeset This PR changes the dashboard's stream reader (`packages/web/app/lib/hooks/use-stream-reader.ts`) so it dispatches on the envelope format and can read sealed (`encp`) frames, but the changeset listed only core, web-shared and cli. `@workflow/web` is published, so without an entry the change would still ship — just as an incidental dependency bump, with nothing in that package's release notes explaining that sealed-stream decryption landed. |
||
|
Nathan Colosimo
|
d7553c4517 |
Prewarm only appended replay payloads (#3131)
* Prewarm only appended replay payloads * Keep replay prewarm progress inside cache * Clarify replay prewarm cursor * Remove replay prewarm scan test |
||
|
Nathan Rajlich
|
b4ba79ebc5 |
feat: publish each run's X25519 public key on the run entity (#3095)
* feat: publish each run's X25519 public key on the run entity A cross-run writer needs the recipient run's public key to seal a payload to it. Derive that key at `start()` and stamp it on the run, so a hook resumption or a forwarded-stream writer can find it on a run fetch it was already making instead of spending ~350ms on `run-key`. The key is derived from the per-run key material `getEncryptionKeyForRun()` already returns, so nothing about key acquisition changes. It is not secret: the matching private scalar is never stored anywhere, only re-derived on demand from the deployment's own env seed. Storing it beside run metadata therefore does not weaken the run's confidentiality. **Presence is the writer-side gate for sealed envelopes.** A run only carries a public key if the runtime that created it could also open one — which holds by construction, since derivation and `encp` dispatch both live in `@workflow/core`, so any core that can stamp can also open. Runs are pinned to their creating deployment, so the capability this attests to is still accurate at resume time. Writers seal iff the field is set and otherwise fall back to the symmetric path, which makes version skew degrade gracefully instead of wedging a run. The field rides on `run_created`, and is mirrored onto the queued `runInput` so the resilient-start path (server recreates the run from the queue message when the `run_created` write failed) doesn't silently produce a run that can't receive sealed writes. world-vercel's compile-time wire-contract guard caught the new field before it could be silently dropped on the v4 path, exactly as designed — routed into the frame meta block as plaintext metadata. Also adds browser- and VM-safe base64 helpers to `sealed-box.ts`, since neither `Buffer` nor `btoa` can be assumed in every context that module runs in. `base64ToBytes` returns undefined on malformed input rather than throwing, so a corrupt stored key degrades to "no usable public key" and falls back to the symmetric path instead of crashing a resumption. Both are cross-validated against `Buffer` in tests. * review: fix public-key loss on resilient start and lifecycle updates Two real bugs found in review, both in the local worlds. Neither surfaces as an error — a run just silently stops accepting sealed cross-run writes and falls back to the slow symmetric path forever. **Resilient start dropped the key.** When a `run_started` arrives for a run that was never created, world-local and world-postgres rebuild the run from the queued message. Neither copied `encryptionPublicKey` onto the run row or the synthetic `run_created` event they write. That is precisely the scenario this field exists to survive. (The equivalent server-side path was already handled.) **world-local also wiped the key on every lifecycle transition.** Its run_started / run_completed / run_failed / run_cancelled handlers rewrite the whole run document field-by-field, so any field not explicitly listed is dropped — meaning the key was lost on the *first* `run_started`, not just on the resilient path. All four rebuild sites now carry it. world-postgres is safe here by construction because it issues column-scoped SQL UPDATEs rather than rewriting the row. **base64 decoding is now strict.** The decoder accepted shapes that cannot describe a whole number of bytes (`length % 4 === 1`) and ignored anything after a mid-string `=`, returning a short array instead of `undefined`. That is worse than throwing: a corrupt stored key looked *present*, so callers sealed to garbage rather than taking the symmetric fallback. Now rejects out-of-alphabet characters, bad lengths, misplaced padding, and non-zero trailing bits — with a round-trip test over every length 0–48 to make sure the strictness does not overshoot. * fix: send encryptionPublicKey in the v4 POST frame meta `splitEventDataForV4` lifted the run's public key into the frame meta and `events.ts` spread that meta into `CreateEventV4Input`, but `buildPostFrameMeta` — which copies meta onto the wire field by field — never forwarded `encryptionPublicKey`, and the field was missing from `CreateEventV4Input` entirely. Because the meta is applied with a spread, TypeScript's excess-property check doesn't fire, so the key was computed, put in the meta, and then silently dropped before the request was sent. The server therefore never received the key, never stored it on the run entity, and every cross-run writer fell back to the symmetric envelope. Every symptom pointed away from the SDK: a deliberately oversized key was accepted rather than rejected (the field never arrived), the key was absent from the run row, and `resumeHook()` always chose `encr`. Add the field to `CreateEventV4Input`, forward it in `buildPostFrameMeta`, and cover it for both `run_created` and resilient-start `run_started`. Also add a generic guard asserting that every field the splitter puts in the meta reaches the wire, so the next omission in this hand-maintained mapping fails a test instead of silently degrading encryption. |
||
|
Nathan Rajlich
|
7a6503048a |
feat(core): route sealed envelopes through the serialization layer (#3094)
* feat(core): route sealed envelopes through the serialization layer Adds the plumbing that lets a cross-run writer emit `encp` payloads. The serialization layer previously had no way to express "seal to this public key": every consumer expected a symmetric `CryptoKey`, and the encrypt primitive was unconditionally AES-GCM. That gap was also a live footgun. A 32-byte X25519 public key is a structurally valid AES-256 key, so `importKey(pubkey, 'AES-GCM')` succeeds and silently produces ciphertext nobody can ever open — no compile error, no runtime error, just unreadable data. Making public keys reachable only through `sealTo()` turns that mistake from "avoided by convention" into "unrepresentable". `PayloadKey` is a widening rather than a replacement, so no existing call site changes: | Variant | Writes | Reads | Held by | | ---------------- | ------ | ------------ | ----------------------- | | `CryptoKey` | encr | encr | same-run (legacy shape) | | `RunPayloadKeys` | encr | encr, encp | the owning run, o11y | | `SealTarget` | encp | — | cross-run writers | A run's own payloads deliberately stay symmetric even when the holder could seal: sealing costs a fresh ECDH and 32 bytes per envelope and buys nothing when the writer already holds the decryption key. The variants are branded with `Symbol.for` (not `Symbol()`) because these values cross the host ↔ workflow VM realm boundary, where only the global symbol registry is shared. Both stream directions handle sealed frames, keeping the length header in the clear so frame boundaries stay findable without a key. Frames keep per-frame random nonces rather than a counter — a reconnect or a durable replay restarts the writer, and a counter would repeat `(key, nonce)` and break AES-GCM catastrophically. Regression tests assert that 100 frames of identical plaintext produce 100 distinct ciphertexts, and that ten writer incarnations never share a content key. Capability shortfalls fail loudly and specifically: opening a sealed payload with a symmetric key (or a write-only seal target, or nothing) reports "no run keypair is available" rather than surfacing as a mysterious auth-tag failure further down. * review: amortize the sealed-box KEM across stream frames Sealing each frame independently meant a fresh X25519 keygen + ECDH + HKDF per chunk — O(frames) crypto for a long stream. As review pointed out, amortizing the KEM is safe here so long as nonces stay random, which they already are; the hazard I had been guarding against was counter nonces specifically, not a long-lived content key. `createSealSession` performs one encapsulation per writer instance and reuses the content key, with a fresh random nonce per frame. `createOpenSession` mirrors it on the read side, caching decapsulation by ephemeral public key — since every frame from one writer carries the same key, that is an ECDH per writer rather than per frame. Both safety properties are preserved and now asserted directly: - nonces stay random, so sharing a content key cannot repeat `(key, nonce)` (100 identical-plaintext frames -> 100 distinct frames) - a session is scoped to one stream instance, so a reconnect or durable replay never inherits a previous content key (10 writer incarnations -> 10 distinct ephemeral keys) The envelope layout is byte-identical to one-shot `seal`, so readers cannot tell which path produced a frame and need no matching session. Two edge cases the read-side cache introduces, both covered: frames from two writers interleaved on one stream (eviction on every frame, so correctness cannot depend on hit rate), and a failed decapsulation clearing rather than poisoning the entry. |
||
|
Nathan Rajlich
|
4ba223a01c |
feat(core): add encp sealed-box encryption primitive (#3093)
* feat(core): add `encp` sealed-box encryption primitive
Cross-run writes (hook resumptions targeting another run, forwarded
writable stream frames) currently require the writer to hold the
recipient run's symmetric key, which also grants decrypt capability and
costs a ~350ms `run-key` API round trip across a deployment boundary.
Add the crypto foundation for sealing those writes to a public key
instead. Both keys descend from the per-run key material `K` that
`World.getEncryptionKeyForRun()` already returns, so key acquisition, the
World interface, and the Vercel API are all untouched:
K
├── AES-256 key = K used directly → 'encr' (unchanged)
└── X25519 scalar = HKDF(K, label) → 'encp'
└── public key (published, not secret)
`sealed-box.ts` implements an ECIES-style construction over the same
primitives as HPKE base mode (DHKEM(X25519, HKDF-SHA256), AES-256-GCM),
binding both public keys into the KDF `info` as HPKE's `kem_context` does
to prevent key-substitution attacks. The deviation from strict RFC 9180
framing is documented, and the HKDF labels are versioned so a conformant
profile can be added later without touching existing payloads.
Nothing produces `encp` payloads yet — this is the primitive only. The
o11y layer is hardened defensively so sealed payloads render as
ciphertext rather than throwing `Unsupported serialization format`, and
`hydrateDataWithKey` skips the AES path for them since opening a sealed
payload needs the private scalar rather than the symmetric key.
- optional AAD on the AES helpers, used to bind `projectId|runId`
- `encapsulate`/`decapsulate` split so stream writers can amortize the
KEM across frames; documented that they must keep random per-frame
nonces and re-encapsulate per connection attempt, since a long-lived
content key plus counter nonces would repeat `(key, nonce)` after a
reconnect or a durable replay
- public key derivation is cross-validated against node:crypto's native
X25519 in tests, since it reads the public half out of a JWK export
* review: tighten sealed-box docs, key-length checks, and constants
Addresses review feedback on the sealed-box primitive:
- The module doc pointed at `getSerializeStream` as enforcing the
re-encapsulate-per-writer rule, but nothing in-tree uses `encapsulate`
yet, and the stream path added later seals per frame instead. Reworded
to state the two rules as the caller's contract, since this module
enforces neither.
- `derivePublicKeyFromScalar` now asserts the JWK-derived public key is
32 bytes. That decode is the one place this module trusts an external
encoding; a short value would otherwise fail much later inside key
agreement with a far less obvious message.
- `open()` used bare 12/16 for the nonce and tag sizes. Those now come
from exported `NONCE_LENGTH`/`TAG_BYTES` in the AES layer, so the wire
format check cannot drift from the implementation.
|
||
|
Nathan Colosimo
|
8c12358075 | chore(core): clarify runtime comments (#3111) | ||
|
Nathan Colosimo
|
4ada27d35a |
Remove obsolete world factory aliases (#3112)
* refactor: remove obsolete world factory aliases * test(core): remove world factory loader test |
||
|
Nathan Colosimo
|
d813fb8ee8 |
feat(core): deterministic sandbox hardening (#3045)
* feat(core): deterministic sandbox hardening - crypto.subtle.digest computes synchronously via node:crypto: byte-identical values, promise settles on a deterministic microtask, full BufferSource validation (internal-slot view reads, SAB rejection) - Atomics.waitAsync (a wall-clock timer), async WebAssembly compilation, WeakRef, and FinalizationRegistry are removed from the sandbox — wall clock and GC observation are unreplayable; sync WebAssembly constructors remain - freezeSerializationIntrinsics pins the universal dispatch surfaces: Object.prototype/Array.prototype/Function.prototype are frozen (every missed property read and hasInstance lookup terminates there) and serialization-referenced global bindings are non-writable. Value-type prototypes and constructor statics stay patchable so polyfills (Temporal's Date.prototype.toTemporalInstant, core-js Set.prototype .union / Object.groupBy) keep working — the retained-input gate verifies the members serialization executes per boundary instead. Groundwork for retained-VM replay (#2990). * Drop serialization intrinsic freezing from the sandbox The retained-VM passivity design moved from pinning/verifying the sandbox surfaces serialization dispatches on to injecting hardened operations into devalue itself (with taint-based de-opt), so freezing Object/Array/Function prototypes and pinning global bindings is no longer needed. Keep only the determinism hardening (sync digest, removal of wall-clock/GC-observing APIs). * Document and lock in why async crypto.subtle methods cannot break quiescence The remaining async subtle methods reject immediately through the crypto proxy (brand check — the receiver is not a real SubtleCrypto), so they can never mint a host-timing promise. Narrow the quiescence comment to what the code actually enforces and add a test so the unreachability is not silently "fixed" later. * simplify sandbox hardening: lean digest input conversion, async digest, explicit subtle throwers * mark sandbox API removals as a major change --------- Co-authored-by: Nathan Rajlich <n@n8.io> |