mirror of
https://github.com/vercel/workflow.git
synced 2026-09-14 19:59:43 +08:00
codex/atomic-start-postgres
181 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Nathan Colosimo
|
22349e95fd |
perf(core): load replay suffix in one request (#3205)
* perf(core): stream replay suffix in one request * perf(core): load replay suffix in one request Signed-off-by: Nathan Colosimo <110621881+NathanColosimo@users.noreply.github.com> * test(world-vercel): use streamed run start fixtures * refactor(events): simplify return-all plumbing Signed-off-by: Nathan Colosimo <110621881+NathanColosimo@users.noreply.github.com> * Return complete local run preloads * Document workflow event limit * fix: make return-all event loading resilient * Simplify full event listing * refactor(world-vercel): omit event limit for full loads * fix(world-vercel): explicitly request complete event logs --------- Signed-off-by: Nathan Colosimo <110621881+NathanColosimo@users.noreply.github.com> |
||
|
Nathan Colosimo
|
65139acfd7 |
perf(core): continue partial run_started preloads from cursor (#3124)
* perf(core): continue partial run preloads * refactor(core): simplify preload continuation * fix(core): preserve preload fallbacks * chore: rerun CI * fix(world): infer event create results * fix(core): preserve run state during setup * fix(world): enforce typed event results * refactor(world): rely on event result contract * refactor(core): unify replay event log state * refactor(core): make replay log states exact * fix(core): harden run start preload recovery * test(world-local): allow slow preload coverage * fix(core): preserve event result inference through recovery * refactor(core): simplify preload state transition Signed-off-by: Nathan Colosimo <110621881+NathanColosimo@users.noreply.github.com> * refactor(runtime): reuse event pages without duplicate reads * refactor(world-vercel): preserve opaque event payloads * Validate v4 event create responses * Validate v4 event frame metadata * Remove invalid v4 response identity check * Return validated v4 event bodies directly * Reuse event result entity types * Simplify event creation result types * Use concrete run creation result * Preserve generic event storage implementation * Validate v4 event responses without casts * Parse v4 event frames once * Reuse the default v4 event body schema * Simplify event preload state * Narrow event page result states * Preserve literal event result flags * Accept hook conflict event responses * Remove redundant optional event page schemas * Simplify preloaded event log access * Flatten replay event log state * Simplify replay event log state * Use one replay event log * fix(next): preserve edits made during full HMR rebuilds * chore(core): log dormant hook replays * fix(next): commit HMR snapshots after rebuilds * fix(next): ignore duplicate HMR file events * test(next): expect deduplicated HMR removal event * fix(next): distinguish duplicate HMR notifications * fix(next): ignore HMR notifications without source changes * chore: move Next HMR fix to separate PR * fix(core): complete partial preloads before QuickJS replay --------- Signed-off-by: Nathan Colosimo <110621881+NathanColosimo@users.noreply.github.com> |
||
|
Nathan Colosimo
|
74dbf81d32 |
fix(core): retry replay timeouts without exiting (#3385)
* fix(core): retry replay timeouts without exiting * refactor(world-postgres): leave existing retry limits unchanged * test(world-postgres): remove mocked migration assertion * chore: consolidate replay retry changesets |
||
|
Peter Wielander
|
a8db185c3b | [core] Fold events.create deltas into the replay log (#3382) | ||
|
Karthik Kalyan
|
439a495a71 |
fix(core): pre-check deployment affinity before the lazy resume write (#3374)
The lazy hook fast path (#3345) hoisted the consumer's hook_received write above the deployment-affinity guard (#2960), so a misrouted lazy resume wrote its event before the guard could re-route the delivery. Stamp the run's pinned deployment on the resume message (hookInput.deploymentId, from the producer's resume context) and, on the consumer, compare it against the ambient deployment id immediately before the fast path: a match continues with no run fetch, a mismatch fetches the authoritative run and hands it to the existing guard — which keeps sole ownership of re-route/fail policy and remains the authoritative protection before replay and step execution. The re-routed message preserves the complete hookInput (it may hold the only copy of the resume payload). Older messages without the field, and worlds without deployment affinity, are unchanged: they skip the pre-check and rely on the authoritative guard, the pre-guard write staying convergent per (runId, resumeId). Fixes the misrouted-lazy-resume unit test broken by the #2960/#3345 ordering: a modern misrouted resume now re-routes with zero event writes, asserted for both hook_received and run_started. Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |
||
|
github-actions[bot]
|
e6af70b9d9 |
Version Packages (beta) (#3318)
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> |
||
|
Karthik Kalyan
|
9c1b3c8638 |
perf(core): initialize lazy hook replay from hook_received stream (#3345)
* perf(core): initialize lazy hook replay from hook_received stream On a lazy hook queue delivery, the consumer's idempotent hook_received re-ensure is hoisted above run_started and doubles as the invocation's setup request: it asks the World to return the current replay log with the write (new advisory CreateEventParams.preloadEvents), so one HTTP request yields the canonical event, the reconstructed run, and the complete replay log — skipping both the run_started POST and the initial events.list. - world: optional `preloadEvents?: true` on CreateEventParams, the hook_received dual of skipPreload; Worlds may ignore it - world-vercel: createHookReceivedPreloadEventV4 sends the frame Accept on eligible hook_received posts and decodes either response mode — frames via the response decoder extracted from the LIST consumer (GET behavior unchanged), CBOR via the shared materialized-response mapping. The run is reconstructed from run_created/run_started (plus attr_set folds), the canonical event found by x-wf-event-id, and resumeId now survives frame decoding so the runtime can match it - core: new fast path before the generic run-state setup, guarded on hookInput.resumeId + payloadDigest; a validated COMPLETE preload (hasMore false — this path has no cursor-continuation machinery) initializes workflowRun/preloadedEvents/maxEventsLimit directly, anything else falls back to the run_started setup without re-posting the hook; error classification matches the existing re-ensure (terminal → consume, transient → redeliver); setup source reported via workflow.resume_setup_source (never workflow.hook.resilient_resume_materialized, which stays a recovery-only signal) - producer resumeHook() is unchanged and never sets preloadEvents Based directly on main (no dependency on #3124/#3191); pairs with workflow-server's streamed hook_received replay-log response, which deploys first — the SDK negotiates per request and falls back safely against older servers. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * address review: lazy fallback, retryable resume, terminal telemetry - world-vercel: the preload request keeps hook_received's lazy remoteRefBehavior — a supporting server owns frame-body resolution, while an older server now answers the CBOR fallback without resolving an S3-backed payload the runtime would discard - world-vercel: the atomic lazy-resume shape (resumeId + digest) opts into withEventPostRetry via idempotentHookResume — the (runId, resumeId) claim makes the POST idempotent-on-retry; legacy/partial hook_received shapes stay single-attempt, definitive 4xx stays non-retryable (unit + adapter + trace-propagation coverage) - core: a terminal event found in the preload records workflow.resume_setup_source=hook_received_stream and the run's actual terminal status on the span before consuming the delivery - core: document resilient_resume_materialized as the legacy/non-atomic re-ensure signal (claim ownership is not observable client-side, so the hoisted path deliberately never emits it) and resume_setup_source as a latency signal, not proof of event creation; note the Option A skip is now unreachable for atomic resumes - world: spell out the full preload usability contract on preloadEvents (complete hasMore-false log, non-null cursor, run/startedAt/maxEvents, lifecycle events, matching resumeId, list ordering, read-after-write consistency); bump @workflow/world to minor - new QuickJS sourcing tests (VM mocked): an attested complete preload is used verbatim with no events.list, a non-attested hook-containing preload is refetched, and an attested empty preload is not trusted Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |
||
|
Elliot Dauber
|
72efc90f28 |
Use runtime deadline for inline execution limit (#3360)
* Use runtime deadline for inline execution limit * up * lazy import * Update packages/world/src/interfaces.ts Co-authored-by: Peter Wielander <mittgfu@gmail.com> Signed-off-by: Elliot Dauber <67391073+elliotdauber@users.noreply.github.com> --------- Signed-off-by: Elliot Dauber <67391073+elliotdauber@users.noreply.github.com> Co-authored-by: Peter Wielander <mittgfu@gmail.com> |
||
|
Alex Langenfeld
|
79e4c04409 |
fix(core): re-route runs delivered to the wrong deployment (#2960)
## Summary & Motivation A queue callback that reaches a deployment other than the one its run is pinned to derives the per-run encryption key from the wrong master key, so the delivery fails before user code runs and the run dies as a blank "exceeded max retries". The delivery is re-enqueued explicitly addressed to the run's own deployment — strictly better-targeted than the send that misrouted — and the run is failed with the new `DEPLOYMENT_MISMATCH` error code only once `WORKFLOW_DEPLOYMENT_MISMATCH_MAX_RETRIES` (default 3) is spent. Gated on the new World capability `deploymentAffinity`, so worlds with synthetic or version-tagged deployment ids are unaffected. ## Test Plan Unit tests added for the guard and both runtime paths; local vitest and typechecks pass. |
||
|
Karthik Kalyan
|
8d479283ca |
feat(world,world-vercel,core): bulk run cancellation primitive (#3347)
* feat(world,world-vercel,core): bulk run cancellation primitive Add a bulk cancellation contract to @workflow/world (schemas, types, and an optional Storage['runs'].cancelMany method), implement it in @workflow/world-vercel via a single POST /v4/runs/cancel request, and add a cancelRuns runtime helper to @workflow/core that uses the world fast path when available and otherwise falls back to bounded-concurrency (max 20) single-run cancellation. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * Update packages/world/src/interfaces.ts Co-authored-by: Peter Wielander <mittgfu@gmail.com> Signed-off-by: Karthik Kalyan <105607645+karthikscale3@users.noreply.github.com> --------- Signed-off-by: Karthik Kalyan <105607645+karthikscale3@users.noreply.github.com> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> Co-authored-by: Peter Wielander <mittgfu@gmail.com> |
||
|
Peter Wielander
|
de1905f15c | feat(world): require a runId on listByCorrelationId (#3280) | ||
|
Nathan Colosimo
|
e6f1b6f548 |
feat(world-local): support Hook minimum retention (#2866)
* feat(core): add hook token retention contract * refactor(core): constrain hook retention options * fix(core): preserve boolean hook visibility options * revert(core): preserve HookOptions interface * docs(core): clarify retained conflict ownership * docs(core): retain newest-wins conflict pattern * docs(core): simplify hook retention guidance * docs(core): explain retained token cleanup * docs(core): simplify idempotency guidance * docs(core): clarify retained token results * refactor(core): rename hook token expiration option * chore(core): name hook expiration changeset * docs(core): simplify Hook expiration language * docs(core): clarify Hook expiration deadline * docs(core): remove Hook deadline caveat * refactor(core): align Hook expiration field names * docs(core): narrow Hook expiration documentation * docs(core): clarify hook expiration availability * Update packages/core/src/workflow/hook.ts Co-authored-by: Peter Wielander <mittgfu@gmail.com> Signed-off-by: Nathan Colosimo <110621881+NathanColosimo@users.noreply.github.com> * docs(core): clarify Hook token expiration behavior * docs(core): explain active Hook expiration behavior * feat(world): advertise hook ttl capability * fix(core): validate hook ttl capability after main merge * refactor(core): rename hook expiry to minimum retention * docs: keep hook retention guidance on v5 * docs: define retained run availability * fix(core): validate Hook retention at creation * feat(core): define retained Hook lookup semantics * refactor(core): simplify hook retention checks * feat(world-local): support Hook token expiration * fix(world-local): make hook recovery atomic * refactor(world-local): align Hook minimum retention * fix(world-local): preserve Hook creation order * fix(world-local): expose retained Hooks consistently * refactor(world-local): simplify retained hook storage * fix(world-local): allow stale lock recovery * refactor(world-local): simplify hook retention storage Signed-off-by: Nathan Colosimo <110621881+NathanColosimo@users.noreply.github.com> * fix(world-local): serialize expired hook token handoff Signed-off-by: Nathan Colosimo <110621881+NathanColosimo@users.noreply.github.com> * fix(world-local): preserve hook creation order Signed-off-by: Nathan Colosimo <110621881+NathanColosimo@users.noreply.github.com> * refactor(world-local): clarify hook availability cleanup * docs: note Local World Hook retention support * fix(world-local): harden hook retention persistence * fix(web-shared): render hook retention deadline * fix(world-postgres): exclude unsupported hook retention * feat(world-local): enforce Hook retention limit * docs(world-local): clarify retention limit error * docs(world): clarify Hook retention deadline * docs(hooks): link retention configuration --------- Signed-off-by: Nathan Colosimo <110621881+NathanColosimo@users.noreply.github.com> Co-authored-by: Peter Wielander <mittgfu@gmail.com> |
||
|
github-actions[bot]
|
bf4a591f12 |
Version Packages (beta) (#3256)
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> |
||
|
Karthik Kalyan
|
31f92df10d |
Lazy hook resumption: parallel event write + queue publish (#3230)
* feat(core): lazy hook resumption via parallel event write + queue publish (rebased onto #1834 + #3145) Rebase of #3230 onto current main ( |
||
|
Pranay Prakash
|
ee944d2476 |
feat(core): stamp creator environment into runInput and reject cross-environment queue deliveries client-side (#3244)
* feat(core): stamp creator environment into runInput and reject cross-environment queue deliveries client-side `start()` makes two writes that have to land in the same tenant: the `run_created` event, attributed to whatever environment the caller authenticates as, and the queue message, pinned to a deployment. A misconfigured caller can split them — writing the run to one environment while addressing the message to a deployment in another. The consumer finds no run under its own tenant, the backend's resilient start (`run_started` creates the run when `run_created` was never seen) mints a second copy of the same run id in the consumer's environment, and both copies are real: the creator's sits pending forever while the other executes. The deployment id is not the discriminator — it matched end to end in the incident that motivated this. The environment is. So carry it: add an optional `World.getEnvironment()`, implement it in world-vercel from the same resolution that produces the `x-vercel-environment` header, and stamp it into the queue message's `runInput`. The consuming deployment already knows its own environment, so it can refuse the delivery itself with no server coordination — and refuse before `run_started`, the write that would create the fork. The refusal acks the message instead of throwing: the mismatch is baked into the message, so every redelivery would reach the same verdict and throwing would hot-loop until MAX_QUEUE_DELIVERIES. Both sides must be known for the check to run, so worlds with a single tenant (local, Postgres) and runs started by an older SDK behave exactly as before. A companion diagnostic logs a deployment-id mismatch without refusing, since deployment ids differ for benign reasons too. Signed-off-by: Pranay Prakash <pranay.gp@gmail.com> * fix(world-vercel): resolve the runtime environment from VERCEL_TARGET_ENV For a deployment in a Vercel custom environment, the OIDC token's environment claim is the custom environment's slug (the platform mints `customEnvironment?.slug ?? envTarget`) while VERCEL_ENV reports 'preview' — so keying the cross-environment guard on VERCEL_ENV could false-refuse a legitimate delivery, e.g. a CLI client attributed to 'staging' starting a run on the staging deployment. VERCEL_TARGET_ENV is populated from exactly the same slug-or-target pair as the claim, so prefer it, keeping VERCEL_ENV as the fallback for contexts that don't inject it. Also sorts runtime.ts imports per the Biome rule that landed on main in #3241. Signed-off-by: Pranay Prakash <pranay.gp@gmail.com> --------- Signed-off-by: Pranay Prakash <pranay.gp@gmail.com> |
||
|
Peter Wielander
|
1471f252fa | [core] Gate event creation on the loaded event count and restart replays in-process (#3145) | ||
|
Nathan Rajlich
|
438eaa6a59 |
Make resumeHook() resilient to transient hook_received event write failures (#1834)
* Make resumeHook() resilient to transient hook_received event write failures
When events.create('hook_received') fails with a retryable error (429/5xx),
resumeHook() now dispatches the queue message with a `hookInput` payload
carrying the dehydrated hook payload. The workflow runtime materializes the
missing hook_received event from that payload on its next delivery, mirroring
the existing resilient-start behavior of start() / run_created / run_started.
Returned Hook carries a new `resilientResume: true` flag when the fallback
path was taken. Both write paths share a client-minted `resumeId` as an
idempotency key so the runtime can dedup if the direct write actually
committed but the client saw a transient error.
Uses a sequential write-then-queue flow (not parallel) to avoid a dedup race
on the happy path: hook_received events have no entity-level conflict guard
(unlike run_created), so a duplicate written before the direct write commits
would double-deliver the payload to the workflow.
* Fix resilient resume: use local payload in materialized hook_received event
The server returns a 'lazy' response for hook_received event creation,
where eventData.payload may be a RefDescriptor (when the payload
exceeded the inline size and was offloaded to blob storage) rather
than the raw bytes. Pushing this directly to the in-memory events
array caused the workflow VM to fail with 'Invalid input' when trying
to deserialize the RefDescriptor as a Uint8Array.
Substitute the eventData we already have locally so the in-memory
event matches what getWorkflowRunEvents would return after
client-side ref hydration.
* Gate resilient resume on target runtime capability; carry hook token; export ResumedHook; docs
- Only take the resilient path when the target run's recorded
@workflow/core version understands hookInput on the queue payload.
Runs keep executing on the deployment they were created on (skew
protection), and older runtimes parse the queue message with a schema
that silently strips unknown fields - the resume payload would be
lost while resumeHook() reported success. Fail fast (propagate the
original event-write error) for such runs instead, preserving the
caller's ability to retry.
- Carry the hook token on hookInput and write it into the materialized
hook_received event so it gets the same replay-divergence guard as a
directly written event (#2030 parity).
- Export ResumedHook from @workflow/core/runtime and workflow/api.
- Add changelog page and update resumeHook() API reference docs.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Address review: correct capability cutoff, drop own-version escape hatch, replay-side resumeId dedup
Review fixes for the resilient-resume capability gate and dedup:
- Bump the supportsQueueHookInput cutoff to 5.0.0-beta.39: 5.0.0-beta.38 is
published WITHOUT this feature (its queue-payload schema strips hookInput),
so classifying it as capable would silently lose resume payloads. The
cutoff is now a single exported constant (QUEUE_HOOK_INPUT_MIN_VERSION)
with a TODO(release) requiring re-verification at merge time.
- Remove the own-version exact-match escape hatch entirely: version strings
do not identify builds (a published beta.38 and a main-built tarball can
share a version string while differing in content), so the check could
declare a featureless published deployment capable. Pre-release builds now
fall back to fail-fast until the version is bumped past the cutoff — the
safe direction. Tests simulate a capable target explicitly.
- Make duplicate suppression authoritative at the replay boundary: replay
now dedups hook_received events sharing a resumeId (same resume attempt),
so even when concurrent redelivery of the same queue message
double-materializes the event (no World enforces uniqueness on
hook_received), the payload reaches workflow code exactly once. This is a
pure function of the persisted log, keeping replay deterministic. The
runtime's snapshot check remains as best-effort write suppression, with
its comment corrected to say so; the EntityConflictError catch is kept as
the forward-compatible signal for planned server-side (runId, resumeId)
uniqueness, with its comment corrected to say it is defensive today.
- Stamp materialized hook_received events with occurredAt decoded from the
resumeId ULID so resiliently-resumed hooks are timestamped at resume time
rather than after the queue round-trip.
- Pin the cross-version compat contract in a test: the direct write is
resumeId-only (no digest or negotiation fields), which later server-side
idempotency work must keep accepting.
- Exercise the published boundary (5.0.0-beta.38) in fail-fast tests, and
make the capability tests self-check against the exported cutoff constant
instead of restating literals.
- Docs: changelog date June -> July 2026, dash consistency, and document the
replay-side dedup guarantee.
* Encode release-gate and successor-rebase contracts into code comments
Comment-only changes capturing the review agreements so they survive the
parallel-resume successor rebase (no behavior change):
- capabilities.ts: the QUEUE_HOOK_INPUT_MIN_VERSION re-verification point
is the actual combined SDK release (after the successor lands and its
server-side dedup is deployed), not source-merge time — this PR merges
source-only and no SDK is published from it alone. Every Version
Packages merge in between moves the earliest possible carrier.
- workflow/hook.ts + runtime.ts: scope the replay-side resumeId dedup
honestly as defense-in-depth over the persisted log, not a
cross-invocation exactly-once guarantee — concurrent invocations
replaying pre-duplicate snapshots each see only their own row; the
storage-level (runId, resumeId) constraint in the successor work is the
correctness boundary. The set stays useful post-constraint for logs
written before it deployed.
- runtime.ts: document the EntityConflictError swallow's known gap while
the branch is defensive (this invocation's local log lacks the payload;
progress relies on the other writer's delivery or redelivery) and pin
the rebase contract for when the constraint makes it live: a matching
claim must append the canonical event locally and succeed; a real
conflict must rethrow for redelivery.
- resume-hook-resilient.test.ts: reframe the wire-shape pin as a tripwire
rather than a permanent contract — the successor deliberately widens it
(ID/digest pair + attestation) before any SDK release, so the
resumeId-only shape never ships as a published server contract.
---------
Co-authored-by: Peter Wielander <peter.wielander@vercel.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
|
||
|
Alex Langenfeld
|
4017597a5f |
feat(core): report replay divergence recovery (#3208)
Signed-off-by: Alex Langenfeld <alex.langenfeld@vercel.com> |
||
|
Nathan Rajlich
|
32ac8e73fd |
Fix Biome lint violations and add Biome CI check (#3222)
* Fix Biome lint violations and add Biome CI check Biome was not configured to respect .gitignore, so ~92% of the 13,355 reported diagnostics came from gitignored build artifacts. Enable VCS integration (useIgnoreFile), apply safe auto-fixes across the repo, fix the remaining mechanical errors by hand, downgrade judgment-call a11y / dangerouslySetInnerHTML rules to warnings, and add a 'biome ci' job to the Lint workflow so violations block PRs going forward. * Use an empty changeset (no behavior change, no release needed) |
||
|
Alex Langenfeld
|
4a9d26b1cb |
feat(world): persist the compute instance that ran each step attempt (#3186)
* feat(world): persist the compute instance that ran each step attempt Add CreateEventParams.computeInstanceId (ambient per-event identity, mirroring requestId) and a readable Event.computeInstanceId. Core stamps it on every step_started write; world-vercel forwards it in the v4 frame meta next to vercelId. Lets observability distinguish steps sharing a compute instance from those on different instances or invocations. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Alex Langenfeld <alex.langenfeld@vercel.com> * test: cover computeInstanceId threading from params to v4 frame meta world-vercel: computeInstanceId reaches the v4 frame meta, rides alongside vercelId rather than replacing it, and is omitted when unset. core: step_started carries it without displacing the stateUpdatedAt precondition guard (both share one params object). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Alex Langenfeld <alex.langenfeld@vercel.com> * fix(web-shared): render computeInstanceId in the attribute panel AttributeKey derives from keyof Event, so adding computeInstanceId to the event schema widened it and left the exhaustive attributeToDisplayFn map incomplete (TS2741). Renders it beside deploymentId as 'Compute Instance ID', copyable like the other opaque ids. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Alex Langenfeld <alex.langenfeld@vercel.com> * fix(world-postgres): exclude computeInstanceId from the events column contract The events table asserts satisfies DrizzlishOfType<...Omit<Event, 'occurredAt'>...>, so adding computeInstanceId to the event schema broke the build (TS1360). This world does not persist it, matching how occurredAt is already handled. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Alex Langenfeld <alex.langenfeld@vercel.com> * refactor: move computeInstanceId to the analytics read contract The server routes computeInstanceId into ClickHouse and returns it on AnalyticsEvent/AnalyticsStep, never on the event record — so Event.computeInstanceId was dead on read and zod would strip the field off the analytics wire. Move it to AnalyticsEventSchema/AnalyticsStepSchema (beside vercelId/requestId, the same class of ambient provenance), which also drops the world-postgres column-contract exclusion entirely. Also: hoist the duplicated step_started params into one local, extract the repeated mock-agent harness in events.test.ts, and use vi.spyOn plus an identity assertion against COMPUTE_INSTANCE_ID. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Alex Langenfeld <alex.langenfeld@vercel.com> --------- Signed-off-by: Alex Langenfeld <alex.langenfeld@vercel.com> Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
github-actions[bot]
|
741a0d9eaf |
Version Packages (beta) (#3087)
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> |
||
|
Karthik Kalyan
|
d24c91cfde |
feat(core): resume hooks from stored resumeContext and seal to the run key (#3125)
Hooks can carry an optional `resumeContext` mirrored from the run at
creation time. When present, `resumeHook`/`resumeWebhook` resume directly
from it instead of fetching the full run, saving a round trip per resume.
When the context also carries the run's `encryptionPublicKey`, the resume
seals its payload (`encp`) directly to that key. Combined with the sealed
envelope work (#3093-#3096), a default webhook resume then needs neither a
run read nor a cross-deployment run-key lookup: the key is resolved only
when the hook actually stores metadata that must be hydrated symmetrically.
Everything falls back transparently to the full run fetch and symmetric
key when the context (or the public key within it) is absent, so new
clients interoperate with old servers and vice versa.
- world: optional `encryptionPublicKey` on `HookResumeContext`
- world-postgres: `resume_context` column migration
- core: combined fast-path + seal in resume-hook; fast-path control-flow
suite split from the real-serialization crypto suite
- world-vercel: cover the `getEncryptionKeyForRun(runId, { deploymentId })`
overload the fast path relies on
- web-shared: render `resumeContext` in the attribute panel
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
|
||
|
Nathan Rajlich
|
a86035f71f |
feat: return the run public key from the capability probe (#3099)
* feat: return the run public key from the capability probe
This removes the last key-lookup request from the cross-deployment hot
paths. `start()` already blocks on a capability probe for every
cross-deployment call, and the probe responder executes *inside the target
deployment*, where the run's key material is available locally. So the
public key can ride back on a response the caller is already awaiting, at
no additional latency, and the `run-key` API request disappears.
Three properties make this better than keeping the request:
- The wait is already being paid. Folding the key into the existing
response removes the request outright rather than relocating it.
- A public key is exactly what this channel can carry. The probe response
stream is deliberately unauthenticated, which would disqualify shipping
the symmetric key over it — but a public key is not secret.
- It reduces privilege. The caller ends up able to seal the workflow
arguments but not read them back; fetching the symmetric key granted
full read access to a run it merely launched.
`runId` is now minted before the probe rather than just after it.
`createRunId()` reads only `opts`, which is fully resolved by that point,
so the move has no other dependency — and a test asserts the id sent to
the probe is the one actually created.
Everything is best-effort. The probe is already failure-tolerant (2s
timeout, errors swallowed) and is skipped entirely for same-deployment
starts and for worlds without a streams API. When no key comes back — old
target, timeout, encryption disabled, or a malformed value — `start()`
falls back to the existing lookup plus symmetric encryption. Key
derivation failures inside the responder are caught and logged so the
probe still reports health and capabilities, which callers depend on for
reasons unrelated to encryption.
* fix: keep the health-check discriminator on runId-bearing probes
`QueuePayloadSchema` is an ordered union and `z.object` strips keys the
matching member doesn't declare. Adding an optional `runId` to
`HealthCheckPayloadSchema` made a probe payload also satisfy
`WorkflowInvokePayloadSchema`, whose only required field is `runId`. Because
the invoke member came first, world-vercel's queue handler parsed a
runId-bearing probe down to `{ runId }`, dropping `__healthCheck` and
`correlationId`.
The runtime dispatches on `__healthCheck` before falling through to the
invoke schema, so the probe was reinterpreted as "replay this run": it POSTed
`run_started` for a run that does not exist yet, 404'd, failed the handler,
and retried indefinitely. The probe never answered and the cross-deployment
`start()` timed out — which also regressed the pre-existing capability
detection, not just the new key lookup.
Order the health-check member first; it requires `__healthCheck: true`, which
no invoke or step payload carries, so invoke and step payloads still resolve
to their own members.
Also reorder `getPhysicalQueueName` to match health checks before the runId
branch, so under `WORKFLOW_SEQUENTIAL_REPLAYS=1` a probe keeps its per-probe
topic instead of queueing behind the run it is preparing.
|
||
|
Nathan Rajlich
|
b4ba79ebc5 |
feat: publish each run's X25519 public key on the run entity (#3095)
* feat: publish each run's X25519 public key on the run entity A cross-run writer needs the recipient run's public key to seal a payload to it. Derive that key at `start()` and stamp it on the run, so a hook resumption or a forwarded-stream writer can find it on a run fetch it was already making instead of spending ~350ms on `run-key`. The key is derived from the per-run key material `getEncryptionKeyForRun()` already returns, so nothing about key acquisition changes. It is not secret: the matching private scalar is never stored anywhere, only re-derived on demand from the deployment's own env seed. Storing it beside run metadata therefore does not weaken the run's confidentiality. **Presence is the writer-side gate for sealed envelopes.** A run only carries a public key if the runtime that created it could also open one — which holds by construction, since derivation and `encp` dispatch both live in `@workflow/core`, so any core that can stamp can also open. Runs are pinned to their creating deployment, so the capability this attests to is still accurate at resume time. Writers seal iff the field is set and otherwise fall back to the symmetric path, which makes version skew degrade gracefully instead of wedging a run. The field rides on `run_created`, and is mirrored onto the queued `runInput` so the resilient-start path (server recreates the run from the queue message when the `run_created` write failed) doesn't silently produce a run that can't receive sealed writes. world-vercel's compile-time wire-contract guard caught the new field before it could be silently dropped on the v4 path, exactly as designed — routed into the frame meta block as plaintext metadata. Also adds browser- and VM-safe base64 helpers to `sealed-box.ts`, since neither `Buffer` nor `btoa` can be assumed in every context that module runs in. `base64ToBytes` returns undefined on malformed input rather than throwing, so a corrupt stored key degrades to "no usable public key" and falls back to the symmetric path instead of crashing a resumption. Both are cross-validated against `Buffer` in tests. * review: fix public-key loss on resilient start and lifecycle updates Two real bugs found in review, both in the local worlds. Neither surfaces as an error — a run just silently stops accepting sealed cross-run writes and falls back to the slow symmetric path forever. **Resilient start dropped the key.** When a `run_started` arrives for a run that was never created, world-local and world-postgres rebuild the run from the queued message. Neither copied `encryptionPublicKey` onto the run row or the synthetic `run_created` event they write. That is precisely the scenario this field exists to survive. (The equivalent server-side path was already handled.) **world-local also wiped the key on every lifecycle transition.** Its run_started / run_completed / run_failed / run_cancelled handlers rewrite the whole run document field-by-field, so any field not explicitly listed is dropped — meaning the key was lost on the *first* `run_started`, not just on the resilient path. All four rebuild sites now carry it. world-postgres is safe here by construction because it issues column-scoped SQL UPDATEs rather than rewriting the row. **base64 decoding is now strict.** The decoder accepted shapes that cannot describe a whole number of bytes (`length % 4 === 1`) and ignored anything after a mid-string `=`, returning a short array instead of `undefined`. That is worse than throwing: a corrupt stored key looked *present*, so callers sealed to garbage rather than taking the symmetric fallback. Now rejects out-of-alphabet characters, bad lengths, misplaced padding, and non-zero trailing bits — with a round-trip test over every length 0–48 to make sure the strictness does not overshoot. * fix: send encryptionPublicKey in the v4 POST frame meta `splitEventDataForV4` lifted the run's public key into the frame meta and `events.ts` spread that meta into `CreateEventV4Input`, but `buildPostFrameMeta` — which copies meta onto the wire field by field — never forwarded `encryptionPublicKey`, and the field was missing from `CreateEventV4Input` entirely. Because the meta is applied with a spread, TypeScript's excess-property check doesn't fire, so the key was computed, put in the meta, and then silently dropped before the request was sent. The server therefore never received the key, never stored it on the run entity, and every cross-run writer fell back to the symmetric envelope. Every symptom pointed away from the SDK: a deliberately oversized key was accepted rather than rejected (the field never arrived), the key was absent from the run row, and `resumeHook()` always chose `encr`. Add the field to `CreateEventV4Input`, forward it in `buildPostFrameMeta`, and cover it for both `run_created` and resilient-start `run_started`. Also add a generic guard asserting that every field the splitter puts in the meta reaches the wire, so the next omission in this hand-maintained mapping fails a test instead of silently degrading encryption. |
||
|
Nathan Colosimo
|
8c12358075 | chore(core): clarify runtime comments (#3111) | ||
|
Nathan Colosimo
|
62d570ed4b | Remove retired v1 step route plumbing (#3061) | ||
|
Karthik Kalyan
|
fc81f4502f |
perf(core): immediate leading-edge dispatch for idle streams (flush window default 0) (#3088)
* perf(core): immediate leading-edge dispatch for idle streams (flush window default 0) Production producer-rate data (24h of client flush spans): most agents average 1.03-1.21 chunks per flush with 87-98% single-chunk flushes and >70% of chunks arriving more than 10ms after the previous request had already settled — a fixed 10ms leading window batches almost nothing for them while adding ~20% to isolated-chunk publish latency (~50ms median RTT). The one bursty producer (avg ~4-8 chunks/flush) gets its batching from in-flight accumulation, which does not depend on the window at all. The leading chunk of an idle sink now dispatches immediately by default (window 0): first chunk goes out at once, chunks arriving during its request coalesce into the next group, and each settle dispatches the accumulated group immediately — path-independent batching with no fixed tax on slow producers. A positive WORKFLOW_STREAM_FLUSH_INTERVAL_MS (or world.streamFlushIntervalMs, applying from the second group) opts into a windowed leading edge for slow-but-steady producers that prefer larger groups over first-chunk latency. Early-ack, the durability drain barrier, wire caps, and backpressure bounds are unchanged. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Update packages/world/src/interfaces.ts Co-authored-by: Peter Wielander <mittgfu@gmail.com> Signed-off-by: Karthik Kalyan <105607645+karthikscale3@users.noreply.github.com> * review: env var overrides world streamFlushIntervalMs; world option governs the leading edge too WORKFLOW_STREAM_FLUSH_INTERVAL_MS, when set, now takes precedence over world.streamFlushIntervalMs; otherwise the world option applies from the very first chunk (no more second-group lazy quirk). Deciding waits for the world when needed, which adds no latency: sendGroup awaits the same promise before any request can leave. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Signed-off-by: Karthik Kalyan <105607645+karthikscale3@users.noreply.github.com> Co-authored-by: Claude Fable 5 <noreply@anthropic.com> Co-authored-by: Peter Wielander <mittgfu@gmail.com> |
||
|
github-actions[bot]
|
1225258b5d | Version Packages (beta) (#3028) | ||
|
Shalabh Chaturvedi
|
fe12b84729 |
Implement max_events per run limit (#2986)
Enforces the published per-run events limit, which was previously not enforced. The server supplies the limit on the run_started response (separate change); once a run's event log reaches it, the runtime throws MaxEventsExceededError at the top of the replay loop, and the existing terminal-error path records it as run_failed with a new MAX_EVENTS_EXCEEDED code — instead of letting a runaway workflow (e.g. an unbounded step loop) grow the event log without bound. Adds a new client side WORKFLOW_MAX_EVENTS_OVERRIDE env var which can override the server side provided value (lower only). |
||
|
Peter Wielander
|
eb8fdb9797 | Default WORKFLOW_PRECONDITION_GUARD on (#2946) | ||
|
Nathan Colosimo
|
a5e6f1167a |
feat(core): add experimental Hook minimum retention (#2865)
* feat(core): add hook token retention contract * refactor(core): constrain hook retention options * fix(core): preserve boolean hook visibility options * revert(core): preserve HookOptions interface * docs(core): clarify retained conflict ownership * docs(core): retain newest-wins conflict pattern * docs(core): simplify hook retention guidance * docs(core): explain retained token cleanup * docs(core): simplify idempotency guidance * docs(core): clarify retained token results * refactor(core): rename hook token expiration option * chore(core): name hook expiration changeset * docs(core): simplify Hook expiration language * docs(core): clarify Hook expiration deadline * docs(core): remove Hook deadline caveat * refactor(core): align Hook expiration field names * docs(core): narrow Hook expiration documentation * docs(core): clarify hook expiration availability * Update packages/core/src/workflow/hook.ts Co-authored-by: Peter Wielander <mittgfu@gmail.com> Signed-off-by: Nathan Colosimo <110621881+NathanColosimo@users.noreply.github.com> * docs(core): clarify Hook token expiration behavior * docs(core): explain active Hook expiration behavior * feat(world): advertise hook ttl capability * fix(core): validate hook ttl capability after main merge * refactor(core): rename hook expiry to minimum retention * docs: keep hook retention guidance on v5 * docs: define retained run availability * fix(core): validate Hook retention at creation * feat(core): define retained Hook lookup semantics * refactor(core): simplify hook retention checks * docs(core): simplify retained conflict example * docs(core): flatten forward-to-owner example --------- Signed-off-by: Nathan Colosimo <110621881+NathanColosimo@users.noreply.github.com> Co-authored-by: Peter Wielander <mittgfu@gmail.com> |
||
|
Peter Wielander
|
bb773e9507 | Enable additional perf optimizations when correctness guarantees are met (#2970) | ||
|
Joey Hotz
|
7d29babaef |
feat(world): add optional getMany() for batch run reads (#2915)
Signed-off-by: Joey Hotz <joeyhotz1@gmail.com> |
||
|
github-actions[bot]
|
784f03231e | Version Packages (beta) (#2919) | ||
|
Peter Wielander
|
1933e294cf | Report RSFS/replay latency telemetry on step terminal events (#2929) | ||
|
Roey D. Chasman
|
6b8efd58ce |
feat: cross-run lineage via reserved run attributes (#2153)
Signed-off-by: Roey D. Chasman <rchasman@gmail.com> |
||
|
Peter Wielander
|
a00d169470 | Add stateUpdatedAt precondition guard to event creation (#2266) | ||
|
github-actions[bot]
|
4ecef5303e |
Version Packages (beta) (#2904)
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> |
||
|
Nathan Rajlich
|
c31e30caac |
cli: show world-specific run fields in inspect output via World.describeRun (#2896)
* cli: show run region in inspect output via World.regionForRunId
Adds an optional reverse-lookup hook to the World interface —
regionForRunId(runId): string | null — so tooling can display a run's
region generically. Worlds without a regional dimension simply omit
the hook and no region output appears.
- @workflow/world: new optional interface member (documented: must not
throw; null = undeterminable)
- @workflow/world-vercel: implements it from the run-ID region tag
(tagged -> embedded region, untagged legacy -> default region,
malformed -> null)
- @workflow/cli: 'workflow inspect runs' gains a region column
(between workflowName and status) and 'workflow inspect run <id>'
a region property, in both table and JSON output — only when the
world defines the hook
* Generalize the inspect hook: World.describeRun display fields
Replaces regionForRunId on the World interface with describeRun, per
review: worlds may want to expose more than a region, and the
information need not be encoded in the run ID — describeRun receives
the run entity itself (loosely typed, mirroring createRunId), so a
world can derive fields from executionContext or any other property.
Each returned key becomes an inspect column/property; null values are
preserved in structured output ('applicable but undeterminable' vs.
the hook being absent entirely).
- world-vercel: describeRun returns { region } decoded from the run
ID tag (regionForRunId stays exported as a utility); entities
without a usable runId contribute nothing
- CLI listing: columns come from the union of keys the world returns
for the page, inserted before status; both analytics and storage
paths; detached call site binds this
- CLI showRun: merges the world fields into detail/JSON output via a
method-style call (preserves this), keeping nulls
- tests: field merging (multi-key), null preservation in JSON, hook
absent, and world-vercel describeRun coverage incl. no-runId
entities
* cli: evaluate describeRun defensively
Per review: the World interface says describeRun is pure and must not
throw, but it is an external extension point and the CLI should not
trust that. New safeWorldFields helper, used by both the listing and
showRun paths:
- a throwing implementation contributes no fields instead of crashing
the inspect command
- keys that already exist on the run row are dropped, so a world can
never overwrite canonical fields (status, runId, ...) in output
Tests: canonical fields survive a clobbering describeRun (extra keys
still merged); a throwing describeRun leaves rows untouched and the
command succeeds.
* Allow async describeRun implementations
Per review: widening a sync signature to async later would break every
consumer, while accepting sync-or-async from day one is free — sync
implementations (like world-vercel's) remain valid, and consumers
simply await, which handles both. The performance intent lives on as
documented guidance: the hook is called once per displayed run, so
implementations should stay cheap and avoid I/O; the CLI evaluates a
page's rows concurrently so an async world costs one await per page,
not per row. Promise rejections get the same treatment as throws:
no fields, never a crash.
* Update packages/world/src/interfaces.ts
Co-authored-by: Peter Wielander <mittgfu@gmail.com>
Signed-off-by: Nathan Rajlich <n@n8.io>
---------
Signed-off-by: Nathan Rajlich <n@n8.io>
Co-authored-by: Peter Wielander <mittgfu@gmail.com>
|
||
|
Karthik Kalyan
|
f2be954bb7 |
Add attribute discovery and filtering to world.analytics (#2903)
* Add attribute discovery and filtering to world.analytics
- analytics.attributes.list() — distinct attribute keys observed on runs
in the window, with run counts and first/last seen timestamps
(GET /v2/analytics/attributes).
- analytics.attributes.listValues({ key }) — distinct values for one key
with latest-write-wins run counts (GET /v2/analytics/attributes/values).
- analytics.runs.list({ attributes: { key: value } }) — restrict the runs
listing to runs whose latest attribute snapshot matches every provided
pair (JSON-encoded query param, up to 8 pairs; $-prefixed framework
keys allowed in read filters).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Shorten changeset; document world.analytics in the World SDK reference
The analytics namespace was previously undocumented. Adds a full
reference page (runs, attributes, steps/events/hooks/waits, lookback
windows and pageInfo), links it from the World SDK index and meta, and
replaces the stale 'in the future you'll be able to search by
attributes' line in the attributes guide with a filtering section.
Extends the docs-typecheck ambient world global with the analytics
namespace so reference snippets typecheck.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Drop analytics.attributes.listValues
Not a derived requirement: the agent-runs UI filters by known constant
values and reads per-run values via batch attribute fetches; it never
enumerates distinct values for a key.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Feature-detect world.analytics in the attributes guide example
The snippet dereferenced the optional analytics namespace without the
runtime check its skip-typecheck annotation claimed, and would throw on
local/Postgres/custom Worlds. Guard it and drop the annotation — the
block is now genuinely typechecked.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Fix stale 'querying attributes not available' bullet in the guide
The Behavior list still claimed a query API was only planned, directly
contradicting the Searching and filtering section above it. State what
is implemented: attributes are readable on run objects everywhere, and
key discovery / key=value run filtering are available through the
optional Analytics API.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Surface the analytics namespace on the API reference index pages
The Analytics page sits under workflow/runtime > World SDK, but neither
the API reference index card, the workflow/runtime World SDK card, nor
the World SDK overview mentioned analytics — making the new reference
effectively undiscoverable from /docs/api-reference. Mention it at each
level of the path.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Bump changeset to minor
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
|
||
|
Nathan Rajlich
|
9da2d76260 |
[core][world][world-vercel] Add World.createRunId() and region-aware queue routing (#1981)
* [world-vercel] Add /run-id sub-export with tagged ULID encode/decode Encodes a tag bit, 5-bit version, and 6-bit Vercel region ID into a ULID-shaped string used for workflow run IDs. Tagged values remain valid 26-char Crockford-Base32 ULIDs so they still sort and round-trip through any system that accepts ULIDs. * [world-vercel] Add string-value assertions to run-id tests Add exact-string expectations for encoded outputs at known inputs, covering the default region/version pair, numeric region IDs, version overrides, boundary values (all-zero, all-max), the dirty-input overwrite case, and the lexicographic-order checks. Also adds an explicit byte-array expectation for the canonical ULID-spec example string and an additional first-char-range coverage test for isTagged. * [world-vercel] Remove internal-repo reference from regions doc comment * [world-vercel] Address PR review feedback on run-id sub-export - isTaggedString now fully validates the input as a 26-char Crockford Base32 ULID (delegating to ulidToBytes) instead of only inspecting the first character. This fixes false positives on inputs like '4UUUU...' that have a valid tag-bit position but invalid chars later in the string. - isTagged() now accepts `unknown` to match its documented behavior of safely rejecting non-string inputs without requiring callers to cast. - Introduce `RegionKey` for the full set of keys including 'unknown', and narrow `RegionCode` to `Exclude<RegionKey, 'unknown'>` so the return type of `lookupRegion` and the `DecodedRunId.region` field accurately reflect that 'unknown' is never produced. Updates `encode` to reject 'unknown' as a region code string at runtime (callers wanting the unknown sentinel should pass numeric 0). * [world] [core] [world-vercel] Add World.createRunId() and region-aware queue routing - @workflow/world: add optional createRunId(input?) to the World interface so worlds can mint run IDs with embedded metadata, and add an optional 'region' field to QueueOptions for per-message routing hints. - @workflow/core: start() now delegates run ID generation to world.createRunId() when defined (falling back to a monotonic ULID otherwise), and accepts a new 'runIdInput' option that is forwarded verbatim to createRunId. When runIdInput.region is a string, it is also threaded onto the queue options so the initial workflow message is dispatched to the matching region. - @workflow/world-vercel: implement createRunId() to mint region-tagged ULIDs, preferring an explicit runIdInput.region and falling back to the VERCEL_REGION env var. The queue now resolves its destination region from (in order): an explicit opts.region, the region embedded in the payload's tagged run ID, the VERCEL_REGION env var, and finally a hardcoded 'iad1' default. This replaces the previous unconditional 'iad1' region passed to the @vercel/queue client. Monotonicity within a process is preserved by tracking the last emitted run ID and bumping the bit immediately above the 11-bit metadata window when a same-ms collision would otherwise occur, then re-stamping the requested region/version on top so metadata remains stable. * [core] [world] [world-vercel] Pass full StartOptions to World.createRunId Drop the dedicated 'runIdInput' field on StartOptions and forward the entire options bag to world.createRunId() instead. This keeps the public API surface smaller and lets each World pick the fields it recognises (e.g. world-vercel reads 'region'). The top-level 'region' option remains on StartOptionsBase and is also threaded onto the queue's per-call region opt when set. * Address review feedback: doc fixes and deterministic same-ms tests - Document the final iad1 fallback in QueueOptions.region (world) - Correct the World.createRunId doc: start() always passes an object - Fix the clientOptions comment: the handler client omits region and relies on SDK auto-detection + the ce-vqsregion header for acks - Fix a misleading QueueClient-construction comment in queue.test.ts - Freeze time in the same-ms monotonicity test so it deterministically exercises the intended path, and add a test covering the bump-above-metadata fallback when the region changes mid-millisecond Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * test: keep workflow-server override rewrite-compatible Export WORKFLOW_SERVER_URL_OVERRIDE while keeping the one-line const shape that workflow-server's cross-repo e2e test automation rewrites. Update world-vercel tests to import that exported value for mock origins and URL expectations instead of duplicating the temporary preview URL. * fix(world): clear region tag bit before ULID timestamp validation Region-tagged run IDs set the high bit of the ULID timestamp byte. The shared world timestamp validator used raw decodeTime(), so current tagged run IDs appeared thousands of years in the future and were rejected before reaching workflow-server. Clear the tag bit before decoding, matching the workflow-server behavior, and cover tagged IDs in tests. * fix(world-vercel): validate tagged runId timestamps via run-id decode Keep @workflow/world's ULID helpers generic; they should not know about world-vercel's region-tagged run ID layout. Instead, world-vercel decodes its tagged runId to the original ULID before using the shared timestamp validator for run_created events. Add a world-vercel regression test that a current sfo1-tagged runId passes validation. * fix(world-vercel): default run ID region to iad1 instead of unknown When neither an explicit region option nor VERCEL_REGION is available, createRunId minted a tagged ULID with the unknown (0) region sentinel, producing the tagged: true, region: null state. The server already resolves unknown/untagged runs to DEFAULT_VERCEL_REGION (iad1), so mint a concrete iad1 tag instead, keeping every run ID self-describing and routable. * test(e2e): use verbose reporter + per-test start heartbeat The default vitest reporter buffers per-file output, so a stalling e2e test produces no output until its timeout — making CI look like a silent 30-minute hang. Switch the e2e CI invocations to the verbose reporter (prints each test result as it completes) and emit a '[e2e] ▶ start:' heartbeat to stdout at the start of every test (bypassing vitest's console buffering) so a stuck test is immediately identifiable in the live CI log. * test(world-vercel): point WORKFLOW_SERVER_URL_OVERRIDE at combined 527+529 preview Temporarily target the workflow-server combined-527-529-preview deployment, which bundles platform-directed multi-region routing (vercel/workflow-server#527, incl. the iad1 hook pin) and durable stream state (vercel/workflow-server#529), so e2e can validate the full multi-region path end-to-end. Revert to empty on main. * fix(core): region-tag the health-check correlationId The health-check response is delivered over a Redis stream whose name (and synthetic run ID) embed the correlationId. Under platform-directed routing the responding endpoint and the polling reader can be served from different physical regions; Redis is physical-region-local, so the correlationId must carry the region for both sides to resolve the same backend. Generate the correlationId via world.createRunId() (a region-tagged ULID) when the world provides it, falling back to a plain ULID for worlds that don't tag IDs (e.g. local, single-region). The synthetic wrun_hc_<id> run ID then carries the region; workflow-server's region middleware decodes it. * Address review feedback: validate region overrides, reset server override - queue: validate opts.region and VERCEL_REGION against the known region table before routing, ignoring unrecognised codes so a bad override can't clobber the payload-derived region (Copilot) - add isKnownRegionCode() runtime guard to run-id/regions - reset WORKFLOW_SERVER_URL_OVERRIDE to '' (must be empty on main) - fold the within-PR iad1-default changeset into the main world-vercel changeset and delete it (review) - start.test: declare specVersion on createRunId mock worlds now that the merged world-compatibility check requires it - cover the new region-validation fall-through paths in queue.test Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test(world-vercel): point WORKFLOW_SERVER_URL_OVERRIDE at wave-1 multi-region preview BRANCH-ONLY — revert the override to '' before merge (lint enforces). Points this PR's e2e/benchmark runs at the wave-1 multi-region workflow-server preview (vercel/workflow-server#590: iad1+sfo1+fra1 serving, staging data backends) so region-tagged runs are validated against real multi-region serving end-to-end. Also makes the unit-test mock origins in events-v4.test.ts and trace-propagation.test.ts override-aware (same pattern the rest of the file and utils.test.ts already use), so the suite passes whether or not the override is set — these two files were the only spots hardcoding https://vercel-workflow.com. * test(e2e): Vercel multi-region suite for start()'s region option Adds a dedicated e2e suite validating @workflow/world-vercel region routing end to end, run as its own CI job (e2e-vercel-multi-region) against the nextjs-turbopack workbench only — deliberately separate from e2e.test.ts, which runs as a matrix across all worlds/frameworks where Vercel-specific multi-region behavior doesn't apply. - workbench/nextjs-turbopack/vercel.json: deploy to iad1+sfo1+fra1 so region-routed flow messages have a function to land on in each region. - workflows/99_e2e.ts: regionProbeWorkflow returns the VERCEL_REGION observed by both the workflow and a step, so tests can assert the run EXECUTED in the intended region (not just that it was tagged). - packages/core/e2e/e2e-region.test.ts: per-region cases assert 1) start(..., { region }) mints a region-tagged run ID (decoded via @workflow/world-vercel/run-id), 2) the workflow + step both observed VERCEL_REGION === region, 3) the server reports the run completed; plus a concurrent all-regions case guarding against cross-region misrouting under simultaneous multi-region traffic. Skips on local deployments. - .github/workflows/tests.yml: new e2e-vercel-multi-region job mirroring e2e-vercel-prod's env/deployment-wait, running only the new suite. * test(e2e): start region probes in-function; fix getWorld await The first multi-region CI run surfaced two issues: 1. sfo1/fra1-tagged runs executed in iad1. The suite started runs from the external test process, which uses the api.vercel.com token proxy — and the proxy's queues path forwards every send to the region-less VQS host (the world's proxy-mode resolveBaseUrl ignores the region argument, and the proxy's x-vercel-vqs-api-url escape hatch only allowlists vqs-server-*.vercel.sh preview hosts). Production traffic publishes IN-FUNCTION (direct regional queue routing), so the suite now triggers start() through a new workbench route (/api/e2e-region-start) and rehydrates the run with getRun() — testing the path production actually takes. Proxy-mode regional queue routing is a known gap to address separately in api-workflow. 2. TypeError on world.runs.get: getWorld() is async and was called without await. * test(e2e): cover explicit and implicit region starts in the multi-region suite With regional VQS routing now working through the api.vercel.com proxy (vercel/api#79056 + #2789 + this branch's per-send region resolution), the suite covers both start configurations, asserting the same three properties for each (region-tagged run ID, execution in the intended region via VERCEL_REGION echoed in the return value, server-side completion): 1. EXPLICIT: start(..., { region }) called directly in the vitest runner — publishes through the token proxy, per-send region carried by x-vercel-queue-region. Restores the direct-start shape the suite had originally, plus the concurrent all-regions case. 2. IMPLICIT: dedicated per-region workbench routes (/api/e2e-region-implicit/{iad1,sfo1,fra1}), each pinned to a single region via a per-function 'regions' entry in the workbench vercel.json, calling start() with NO region option — createRunId derives the tag from the minting function's VERCEL_REGION. The test also asserts the route reported executing in its pinned region, so the implicit-tagging assertion can't pass vacuously. Replaces the interim /api/e2e-region-start route (explicit region via request body), which existed to work around the pre-#79056 proxy gap. * Revert WORKFLOW_SERVER_URL_OVERRIDE to '' — wave-1 multi-region serving is in production workflow-server#590 (iad1+sfo1+fra1 serving) merged and deployed to production and the e2e backend, so this branch's e2e/benchmark runs no longer need to target the wave-1 preview. Restores the empty override the No Test Overrides lint job enforces for merge. The override-aware unit-test origins (events-v4/trace-propagation) stay — they are correct under any override value. * test(e2e): cross-region stream visibility (iad1 writer, sfo1 reader) Regression coverage for a backend bug that made cross-region stream reads report zero chunks on IN-PROGRESS streams (completed streams were unaffected), which forced the multi-region serving rollback. The new case exercises exactly that geometry: - crossRegionStreamWorkflow (99_e2e.ts) writes N chunks to the default output stream, then holds the stream OPEN for 45s before closing — the in-progress window is the point, since completed streams are the easy case. - The e2e starts it with region iad1, waits (same-region, via the api.vercel.com proxy) until all chunks are written, asserts the run is still 'running', then reads through a new sfo1-pinned workbench route (/api/e2e-stream-read/sfo1) that returns getTailIndex() plus its VERCEL_REGION. The reader's region served none of the stream's writes, so the reported chunk count must come from the backend's cross-region stream metadata. The test fails loudly if the route isn't actually executing in sfo1. Also bumps the explicit-region test timeout to 120s: the first case in the file absorbs every cold start at once (fresh workbench instances in up to three regions plus a cold backend preview) and was observed just over the 60s default. BRANCH-ONLY (revert before merge, lint enforces): WORKFLOW_SERVER_URL_OVERRIDE points at a multi-region backend preview that includes the fix, so this validates cross-region stream visibility end-to-end before multi-region serving is re-enabled. * test(e2e): extend multi-region suite to all 19 provisioned regions Points the suite at an all-regions backend preview and widens coverage from the wave-1 trio to every provisioned region: - Explicit path: a single concurrent all-regions case starts one tagged run per region (one shared cold-start window instead of 19 sequential ones) and aggregates per-region failures so a single region's breakage reports alongside the full picture. The trio keeps its detailed per-region cases and the 9-way concurrent-isolation case. - Implicit path: workbench gains a region-pinned /api/e2e-region-implicit/<region> route per provisioned region (19 total, shared handler), the workbench itself now deploys to all of them, and the test.each covers the full set with per-case timeouts for regional cold starts. - Multi-region CI job timeout 20m -> 35m for the sequential implicit cases. BRANCH-ONLY (revert before merge, lint enforces): WORKFLOW_SERVER_URL_OVERRIDE now targets the all-regions backend preview instead of the previous (stale, since-merged) fix preview. * test(e2e): tolerate geo-adjacent execution of queue callbacks The first all-regions run surfaced a subtle execution-locality behavior: queue delivery is guaranteed to the tagged region's dataplane and the delivery callback egresses from that region, but the consumer invocation's execution region is chosen by where that callback enters Vercel's edge — and adjacent regions can geo-resolve to each other's functions. Observed live: kix1-tagged runs (callback egressing from Osaka) deterministically executing in hnd1/Tokyo on both the explicit and implicit paths, with tagging, data placement, and completion all still strictly kix1. expectRunInRegion now asserts execution lands in the tagged region OR one of its geographic neighbors (EXECUTION_ADJACENCY), while run-ID tagging and server-side completion remain strictly the requested region. Gross misrouting (e.g. kix1 -> iad1) still fails. * Revert WORKFLOW_SERVER_URL_OVERRIDE to '' — all-regions serving is in production The all-regions workflow-server rollout is deployed and serving production traffic from every Vercel region, so this branch's e2e no longer needs to target a branch preview. Restores the empty override the No Test Overrides lint enforces for merge. With this the PR is complete: region-tagged run IDs, region-aware queue routing, and the multi-region e2e suite (explicit + implicit + all-regions + cross-region streams) all validate against the production-default backends. * docs: fix three stale comments flagged in review - start.ts: StartOptionsBase.region fallback is iad1, not the unknown sentinel (createRunId always mints a concrete routable region) - queue.ts: example used a nonexistent start({ runIdInput }) API; the real option is start({ region }) - events.ts: decode() clears only the tag bit (top bit of the 48-bit timestamp field) — it does not restore the original untagged ULID; reword to say what actually matters for timestamp validation * test(e2e): cover hook resolve/resume for runs owned by non-iad1 regions Hooks are resolved by opaque token, which carries no region hint, so lookup and resume must work regardless of which region owns the run's data. Exercises the full follow-up-message path on sfo1- and fra1-tagged runs: create inside the workflow, resolve by token from the test process, resume twice sequentially, and assert payload order and completion. Regression coverage for the failure mode where the first message to a hook-driven app on a non-iad1 run worked but every follow-up failed with 'Hook not found'. --------- Co-authored-by: Peter Wielander <peter.wielander@vercel.com> Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |
||
|
Nathan Rajlich
|
b01ed548d7 |
build: declare typescript (catalog:) in every package that runs tsc (#2898)
* build: declare typescript (catalog:) in every package that runs tsc Twenty packages invoke tsc in their build/typecheck scripts without declaring a typescript dependency, resolving whatever tsc pnpm happens to leave reachable. That broke locally after the TypeScript 6 upgrade (#2700): base.json now uses the TS6-only 'types': ['*'] wildcard, and worktrees carrying pre-upgrade node_modules/.bin/tsc shims (orphaned typescript@5.9.3 bins that pnpm never refreshes for an undeclared dependency) fail with TS2688 'Cannot find type definition file for *'. Declaring 'typescript': 'catalog:' (the convention nest already follows) makes pnpm own each package's tsc bin, so version upgrades refresh the shims and this staleness class cannot recur. Packages without tsc in their scripts are left unchanged. Full pnpm build: 27/27 tasks green. * Address review: drop duplicate zod devDep; regenerate lockfile minimally - packages/world listed zod in both dependencies and devDependencies (pre-existing on main, surfaced by the devDependencies sort) — keep the runtime dependency only. - Regenerate pnpm-lock.yaml from a pristine main baseline with --lockfile-only (a clean-main run produces zero diff, so main has no drift). Remaining non-typescript changes are mechanical consequences of the change itself: typescript is an (optional) peer of several tooling dependencies, so declaring it in 20 importers creates new peer-resolution snapshot variants and prunes the now-orphaned old ones; plus one radix-ui 1.6.1->1.6.2 refresh in docs caused by its floating 'latest' specifier. - Validated: pnpm install --frozen-lockfile succeeds; full build 27/27. |
||
|
Nathan Rajlich
|
ac41e7d1d7 |
fix(world): parse timezone-naive analytics timestamps as UTC (#2899)
* fix(world): parse timezone-naive analytics timestamps as UTC
ClickHouse-backed analytics endpoints serialize DateTime64 values as
timezone-naive strings ('2026-07-13 17:09:11.593'), UTC by convention.
The analytics schemas coerced them with z.coerce.date(), i.e.
new Date(value), which interprets naive strings in the process's LOCAL
timezone. That is only correct when the process runs in UTC — the
deployed observability web app's server actions, which is why the web
UI appears unaffected — and wrong by the local UTC offset everywhere
else: the CLI on a laptop showed runs 'starting in about 7 hours'
(PDT), and 'workflow web --localUi' shares the bug.
Replace the coercion with a preprocess that normalizes naive datetime
strings to an explicit Z designator before parsing. Values already
carrying timezone info (Z or ±hh:mm) and non-string inputs (Date,
epoch) pass through unchanged.
Tests pin the contract and were verified under TZ=America/Los_Angeles
and TZ=Asia/Tokyo (3 of 4 fail against the old coercion in PDT; plain
UTC CI cannot distinguish the two, which is how this shipped).
* Potential fix for pull request finding
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Signed-off-by: Nathan Rajlich <n@n8.io>
---------
Signed-off-by: Nathan Rajlich <n@n8.io>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
|
||
|
github-actions[bot]
|
faf3348317 |
Version Packages (beta) (#2883)
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> |
||
|
Casey Gowrie
|
7a1ea5a45a |
Fix namespaced active run recovery (#2888)
Signed-off-by: Casey Gowrie <ctgowrie@gmail.com> |
||
|
Peter Wielander
|
0b956f65cb | Rename experimental_setAttributes to setAttributes (#2882) | ||
|
github-actions[bot]
|
5de1b7a100 |
Version Packages (beta) (#2859)
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> |
||
|
Mitul Shah
|
36c63af4a8 |
feat(core): record replay lineage in run executionContext (#2872)
* feat(core): record replay lineage in run executionContext recreateRunFromExisting now stamps the source run id into the new run's executionContext as `replayedFromRunId`, and `start` accepts a matching option. This lets tooling (e.g. the dashboard runs list) surface a run as a replay and link back to the run it was replayed from — previously a replay started a brand-new run with no link to its origin. Co-authored-by: Cursor <cursoragent@cursor.com> * Apply suggestion from @mitul-s Signed-off-by: Mitul Shah <mitulxshah@gmail.com> * cleanup * Update runs.test.ts * update --------- Signed-off-by: Mitul Shah <mitulxshah@gmail.com> Co-authored-by: Cursor <cursoragent@cursor.com> |
||
|
Peter Wielander
|
66036282b5 | Fix duplicate inline step execution on mid-step wake via message ownership (#2848) | ||
|
Nathan Colosimo
|
145835b647 |
Centralize workflow event semantics (#2790)
* Centralize workflow event semantics * Simplify centralized event helper usage * refactor: finish centralizing event semantics * refactor(world): derive Hook from its schema * fix(world): preserve event helper compatibility |