mirror of
https://github.com/vercel/workflow.git
synced 2026-09-14 19:59:43 +08:00
identify-condition-causing-reported-error
33 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Shin
|
71bc027a6c |
fix(world-postgres): make step creation atomic (#3575)
Signed-off-by: Shin <128954611+shin4141@users.noreply.github.com> |
||
|
Pranay Prakash
|
7b79ba37cc |
Add support for 'noop' event type - spec version 7 (#3634)
Co-authored-by: Peter Wielander <peter.wielander@vercel.com> |
||
|
Pranay Prakash
|
9454d51db0 |
feat(core): resolve run.returnValue via a World long poll instead of a 1s poll (#3570)
Co-authored-by: Peter Wielander <mittgfu@gmail.com> Co-authored-by: Peter Wielander <peter.wielander@vercel.com> |
||
|
Peter Wielander
|
771cdb22a8 | fix(world-postgres): refuse a hook resume that races the disposal (#3645) | ||
|
Peter Wielander
|
6786db9953 | World-side incrementing event ID (specVersion 6) (#3389) | ||
|
Nathan Colosimo
|
22349e95fd |
perf(core): load replay suffix in one request (#3205)
* perf(core): stream replay suffix in one request * perf(core): load replay suffix in one request Signed-off-by: Nathan Colosimo <110621881+NathanColosimo@users.noreply.github.com> * test(world-vercel): use streamed run start fixtures * refactor(events): simplify return-all plumbing Signed-off-by: Nathan Colosimo <110621881+NathanColosimo@users.noreply.github.com> * Return complete local run preloads * Document workflow event limit * fix: make return-all event loading resilient * Simplify full event listing * refactor(world-vercel): omit event limit for full loads * fix(world-vercel): explicitly request complete event logs --------- Signed-off-by: Nathan Colosimo <110621881+NathanColosimo@users.noreply.github.com> |
||
|
Nathan Colosimo
|
65139acfd7 |
perf(core): continue partial run_started preloads from cursor (#3124)
* perf(core): continue partial run preloads * refactor(core): simplify preload continuation * fix(core): preserve preload fallbacks * chore: rerun CI * fix(world): infer event create results * fix(core): preserve run state during setup * fix(world): enforce typed event results * refactor(world): rely on event result contract * refactor(core): unify replay event log state * refactor(core): make replay log states exact * fix(core): harden run start preload recovery * test(world-local): allow slow preload coverage * fix(core): preserve event result inference through recovery * refactor(core): simplify preload state transition Signed-off-by: Nathan Colosimo <110621881+NathanColosimo@users.noreply.github.com> * refactor(runtime): reuse event pages without duplicate reads * refactor(world-vercel): preserve opaque event payloads * Validate v4 event create responses * Validate v4 event frame metadata * Remove invalid v4 response identity check * Return validated v4 event bodies directly * Reuse event result entity types * Simplify event creation result types * Use concrete run creation result * Preserve generic event storage implementation * Validate v4 event responses without casts * Parse v4 event frames once * Reuse the default v4 event body schema * Simplify event preload state * Narrow event page result states * Preserve literal event result flags * Accept hook conflict event responses * Remove redundant optional event page schemas * Simplify preloaded event log access * Flatten replay event log state * Simplify replay event log state * Use one replay event log * fix(next): preserve edits made during full HMR rebuilds * chore(core): log dormant hook replays * fix(next): commit HMR snapshots after rebuilds * fix(next): ignore duplicate HMR file events * test(next): expect deduplicated HMR removal event * fix(next): distinguish duplicate HMR notifications * fix(next): ignore HMR notifications without source changes * chore: move Next HMR fix to separate PR * fix(core): complete partial preloads before QuickJS replay --------- Signed-off-by: Nathan Colosimo <110621881+NathanColosimo@users.noreply.github.com> |
||
|
Peter Wielander
|
de1905f15c | feat(world): require a runId on listByCorrelationId (#3280) | ||
|
Nathan Colosimo
|
99f4aeb03d |
feat(world-postgres): support Hook minimum retention (#3276)
* feat(world-postgres): retain hook tokens after runs end
* refactor(world-postgres): reuse terminal run statuses
* docs: note Postgres Hook retention support
* fix(world-postgres): expose hook retention deadline
* Fix: Exhaustive `Record<AttributeKey, ...>` in `attribute-panel.tsx` is missing the `tokenRetentionUntil` key that was added to `HookSchema`, causing TS2741 and breaking every Vercel build.
This commit fixes the issue reported at packages/web-shared/src/components/sidebar/attribute-panel.tsx:426
## Bug
Commit `ad58321` added `tokenRetentionUntil: z.coerce.date().optional()` to `HookSchema` in `packages/world/src/hooks.ts:106`. This adds `tokenRetentionUntil` to the inferred `Hook` type.
In `packages/web-shared/src/components/sidebar/attribute-panel.tsx`, `AttributeKey` is a union that includes `keyof Hook`, so `tokenRetentionUntil` becomes a required member of the **exhaustive** `Record<AttributeKey, (value: unknown, context?: DisplayContext) => ...>` object literal `attributeToDisplayFn` (starting at line ~426).
Because the literal had no `tokenRetentionUntil` entry, `tsc` fails:
```
src/components/sidebar/attribute-panel.tsx(426,7): error TS2741:
Property 'tokenRetentionUntil' is missing in type '{ ... }' but required in type
'Record<AttributeKey, (value: unknown, context?: DisplayContext | undefined) => ReactNode>'.
```
This breaks `@workflow/web-shared#build` and therefore every Vercel deployment (17 failing deployments observed, all with this identical error).
## Fix
Added a `tokenRetentionUntil` entry to `attributeToDisplayFn`, placed alongside the other Hook date fields (`lastReceivedAt`, `disposedAt`):
```ts
tokenRetentionUntil: timestampWithTooltipOrNull,
```
`tokenRetentionUntil` is a `Date` field, and `timestampWithTooltipOrNull` (defined at line 402) is the display helper used by all the other surfaced date fields (`createdAt`, `startedAt`, `completedAt`, `retryAfter`, `resumeAt`, `occurredAt`). Given the intent of `ad58321` was to expose the hook retention deadline, surfacing it as a tooltip-annotated timestamp is the consistent choice.
Only `attributeToDisplayFn` is a fully exhaustive `Record<AttributeKey, ...>`; the other maps are `Partial<...>` / `Set`, so no other edits are required.
## Verification
`node_modules` are not installed in this sandbox, so `tsc` could not be executed directly. Verified structurally instead: the newly added `tokenRetentionUntil` entry (line 449) references `timestampWithTooltipOrNull`, which is defined in-file at line 402 and already used by the sibling date entries, so the fix satisfies the missing-key requirement without introducing new type errors.
Co-authored-by: Vercel <vercel[bot]@users.noreply.github.com>
Co-authored-by: VaguelySerious <mittgfu@gmail.com>
* docs(world-postgres): clarify expired hook rows
* feat(world-postgres): enforce Hook retention limit
* fix(world): remove duplicate Hook retention field
* fix(web-shared): remove duplicate retention renderer
* test(world): remove redundant retention coercion case
---------
Co-authored-by: vercel[bot] <35613825+vercel[bot]@users.noreply.github.com>
Co-authored-by: Vercel <vercel[bot]@users.noreply.github.com>
Co-authored-by: VaguelySerious <mittgfu@gmail.com>
|
||
|
Peter Wielander
|
850777a03b | [world] Guard hook_received against a concurrent run termination (#2987) | ||
|
Joey Hotz
|
3ddf42ed5f |
fix(world-postgres): throw EntityConflictError on duplicate run_created (#2983)
Signed-off-by: Joey Hotz <joeyhotz1@gmail.com> |
||
|
Joey Hotz
|
7d29babaef |
feat(world): add optional getMany() for batch run reads (#2915)
Signed-off-by: Joey Hotz <joeyhotz1@gmail.com> |
||
|
Nathan Colosimo
|
dd36e26962 |
Fix Postgres step lifecycle event ordering (#2714)
* Fix Postgres step start event ordering * Document Postgres step start transaction * Increase canary HMR e2e timeouts * Address Postgres lifecycle review comments |
||
|
Pranay Prakash
|
e7ef9d823b |
perf(core): lazy inline step start (save one world round-trip per step) (#2478)
* perf(core): lazy inline step start to save a world round-trip per step The owned-inline runtime path used to write step_created (suspension handler) and then step_started (executeStep) as two separate world round-trips for a step it already owns and is about to run inline. This defers the step_created write: executeStep sends a single step_started carrying the step input, and the world creates the step on the fly (materializing the step entity plus a synthetic step_created event so replay still observes it). Mirrors the existing resilient run_started -> run_created pattern. Exactly-one ownership is preserved by the world's atomic create-claim: the loser of a concurrent lazy step_started gets EntityConflictError, which executeStep maps to `skipped`, so it never runs the body. A lazy step_started is only ever sent for a brand-new step (the suspension handler defers only steps with no prior step_created), so crash recovery still re-runs a `running` step via the normal non-lazy step_started. Worlds updated: world-local, world-postgres (implicit create + synthetic step_created event), world-vercel (routes the input as the v4 frame payload and threads the server's stepCreated flag). @workflow/world adds optional `input` to step_started and a `stepCreated` EventResult signal. Rollout: server-first. The matching workflow-server change must deploy before this ships; the Vercel world targets a single Vercel-operated backend (server always >= SDK). For local/postgres the world ships in the same package as the runtime, so there is no version skew. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(core): materialize deferred step before failing unregistered step on lazy inline path The lazy inline step-start optimization defers a step's step_created write, expecting executeStep to materialize the step via a lazy step_started carrying its input. For an UNREGISTERED step, executeStep bails out before sending that step_started and writes step_failed directly — but the step entity was never created, so the world's "step must exist" ordering guard rejects the step_failed and the run wedges (times out). This regressed the StepNotRegisteredError e2e tests uniformly across every framework/world (the ghost step never reached `failed`). Fix: on the lazy path, send the lazy step_started first to materialize the step (entity + synthetic step_created, keeping replay correct), then write step_failed. The lazy step_started's atomic create-claim preserves exactly-one-owner: a concurrent winner makes ours reject with EntityConflictError → skipped, so the failure is never written twice. Adds world-level regression tests (world-local, world-postgres) asserting a lazy step_started followed by step_failed marks the step failed. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
Nathan Rajlich
|
f2a7bdeb0a |
fix(world-local,world-postgres): make duplicate hook_created idempotent (#2295)
* fix(world-local): make duplicate hook_created idempotent Duplicate processing of the same hook_created — same runId, hookId, and token, e.g. cross-process replay or queue redelivery — was being recorded as a hook_conflict in the event log, which then replayed as a self- conflict HookConflictError. The fix mirrors the existing step_created duplicate-correlation path: when the exclusive token claim fails and the existing claim has the same (runId, hookId), throw EntityConflictError so the runtime's existing concurrent-replay catch path swallows it. Different runId or hookId reusing the same token still produces a real hook_conflict. The persisted token claim already carried hookId; only the read schema was dropping it. The schema now preserves hookId (marked optional for backward compatibility with older claim files). Fixes #2283 * fix(world-postgres): make duplicate hook_created idempotent world-postgres has the same gap as world-local was just fixed for: the duplicate-token check in events.create unconditionally writes a hook_conflict event when an existing hook with the same token is found, even when the existing hook has the same (runId, hookId) as the incoming event. The unique partial index on workflow_events does not catch this because the duplicate path inserts hook_conflict, not hook_created. Mirror the world-local fix: when the existing hook's (runId, hookId) matches the incoming event, throw EntityConflictError so the runtime's existing concurrent-replay catch path swallows it. Different runId or hookId reusing the same token still produces a real hook_conflict. Refs #2283 * test(e2e): add regression test for hook_conflict from same-tick replay race Regression test for #1665 / #2283. A parent workflow awaits 6 child workflows with Promise.all; each child does a tiny step and creates one webhook. Awaited children flatten into the parent run, so all webhook creations land on the same workflow body. When their step resolutions align in the same tick the workflow body is re-walked and each pass submits hook_created with the same deterministic (correlationId, token). Before the world-side idempotency fix, the world wrote hook_conflict events for the duplicates and the workflow failed with HookConflictError. With the fix, duplicates throw EntityConflictError (swallowed by the suspension handler), no hook_conflict events appear in the log, and the webhooks resolve normally. Verified locally against world-local: the test fails reliably (3/3) on the unfixed code and passes reliably (5/5) on the fixed code. * test(e2e): rewrite parallelStepsThenWebhookWorkflow to match the actual #1665 repro The earlier version invoked another 'use workflow' function directly from inside the parent workflow, which is not a valid child-workflow invocation (child workflows must be spawned via start()) and didn't mirror the bug shape on #1665 anyway. Rewrite the workflow as a single 'use workflow' function that exactly mirrors Paolo's minimal repro: await Promise.all([stepA(), stepB()]); using webhook = createWebhook(); await webhook; The for-loop runs N independent iterations of that sequence in series, each disposing its webhook via 'using' before the next, to give the timing-sensitive race multiple chances to fire. The race is hard to force deterministically on fast local dev — but the same (runId, hookId) idempotency invariant is covered deterministically by the new unit tests in world-local and world-postgres. This e2e test serves as a higher-level regression net: its assertions (no hook_conflict event in the log, no HookConflictError-failed run) are correct whether the race fires or not, and will catch any future regression on a run that does hit it. * fix(world-local,world-postgres): recover crash-orphaned hook claims/rows instead of suppressing the retry Addresses review feedback on PR #2295. The original idempotency fix made duplicate same-(runId, hookId) hook_created submissions throw EntityConflictError so the suspension handler's concurrent-replay catch path swallows them. But the claim file (world-local) and hook row (world-postgres) are written before the durable hook_created event, and the writes are not atomic. A process / DB interruption between the claim/hook write and the event write leaves an orphaned claim/hook row; the retry then matched the same (runId, hookId), threw EntityConflictError, got swallowed, and the run was permanently left with no hook_created event in the log. world-local: - Add a per-(runId, hookId) in-process mutex (withHookLock) mirroring the existing withStepLock, so two same-tick concurrent calls serialize on the entity write and the dedup branch never observes an in-flight winner mid-write. - In the dedup branch, when the existing claim is for the same (runId, hookId) we are trying to create, check whether the durable hook entity actually exists on disk: - exists → real duplicate: throw EntityConflictError as before. - missing → orphaned claim from a prior crash: fall through and complete the partial write (write the hook entity with overwrite, then emit hook_created via the outer code path). world-postgres: - In the dedup branch, when the existing hook row matches the incoming (runId, hookId), check whether a hook_created event for this (runId, correlationId) already exists in the event log: - exists → real duplicate: throw EntityConflictError as before. - missing → orphaned hook row from a prior crash between hook INSERT and events INSERT: skip the hook insert (the row is already there) and let the outer code path emit hook_created, completing the partial write. Tests: - world-local: pre-seed an orphaned token claim with no matching hook entity, retry hook_created, assert hook entity and hook_created event both land (no hook_conflict, no EntityConflictError). - world-postgres: pre-seed an orphaned hook row with no matching hook_created event, retry, assert hook_created event lands (no hook_conflict, no EntityConflictError). Both tests fail on the prior implementation (EntityConflictError thrown on retry, exact symptom from the review). * fix(world-local): probe the event log (not the hook entity) to detect duplicate hook_created Addresses follow-up review on PR #2295. The previous dedup branch checked whether the durable hook entity existed on disk. But the hook entity is written before the `hook_created` event, and the two writes are not atomic, so a crash between them leaves both the claim file and the hook entity on disk with no event in the log. The dedup branch then matched on `(runId, hookId)`, found the hook entity, threw EntityConflictError, and the suspension handler swallowed the retry — permanently losing `hook_created` from the event log. The fix mirrors what the world-postgres branch already does: probe the run's event log for an existing `hook_created` event for the same `(runId, correlationId)`. The event is the durable record of a successful hook creation; the claim file and hook entity are partial- write artifacts that may exist without the event. - exists → real duplicate: throw EntityConflictError so the runtime's concurrent-replay catch path swallows it. - missing → orphaned partial write (crash at any point before the event landed): re-write the hook entity (with overwrite: true, in case a stale partial copy exists) and let the outer code path emit the hook_created event. Added a new helper findHookCreatedEvent that runs a filtered paginatedFileSystemQuery with limit:1 over the run's events. Regression test "should recover an orphaned hook entity with no matching hook_created event" added — pre-creates a hook, deletes just the hook_created event from disk to simulate a crash between the entity write and the event write, asserts the retry emits a fresh hook_created event (no hook_conflict, no swallowed EntityConflictError). I verified this test fails on the prior fix (throws `EntityConflictError: Hook "hook_orphan_entity_1" already created`, exactly as pranaygp reported) and passes on this commit. The previous test ("should recover an orphaned hook token claim with no matching hook entity") continues to pass — the event-log probe is a strict superset of the entity probe, since a missing entity always also implies a missing event. * fix(world-local): converge same-hook creation across workers via canonical eventId Addresses follow-up review on PR #2295. The previous fix made the dedup branch probe the event log to decide real-duplicate vs orphan-recovery, but the probe and the recovery write are not a single atomic operation. Two workers sharing a data directory (or two retries that lose `writeExclusive(constraintPath)` back to back) could both pass the probe (each observing no hook_created event yet), both fall through to the recovery write, and both append a hook_created event with a different eventId — producing two events in the log for the same (runId, hookId). The in-process `withHookLock` mutex does not help here because it is process-local and tag-specific. The fix persists `eventId` in the durable token claim file (written by the original `writeExclusive(constraintPath)`). On a same-(runId, hookId) dedup match, retries adopt that canonical eventId and rebuild the event with a deterministic createdAt derived from the eventId (a ULID). The outer event write switches from `writeJSON` (check-then-write, TOCTOU) to `writeExclusive` (O_CREAT|O_EXCL via temp-file + hard-link, atomic across processes). Either worker may win the publish; the other throws EntityConflictError which the runtime's existing concurrent-replay catch path swallows. Net result: exactly one hook_created event per logical creation. Backward compatibility: a claim file written before this commit lacks `eventId`. Retries that read such a claim fall back to the event-log probe + fresh-eventId recovery — the legacy behavior that does not converge across workers but cannot regress for freshly- written claims after upgrade. world-postgres already converges across workers via the partial unique index on workflow_events_entity_creation_unique (runId+correlationId+eventType for hook/step/wait_created): the loser's INSERT raises 23505 which is already translated to EntityConflictError. Regression tests: - world-local: `converges same-hook creation across workers to one event` uses two tagged storage instances sharing one data directory and fires 25 paired Promise.allSettled hook_created calls. Expected 25 hook_created events total; before this fix yielded 50. - world-postgres: `converges same-hook creation across concurrent calls to one event` exercises the same shape against the real Postgres unique index. Already converges; the test is a guard against future regressions to the catch path. Verified the world-local test fails on c7b23e1b5 with exactly the shape pranaygp reported (50 events for 25 logical creations) and passes on this commit. The earlier orphaned-claim and orphaned- entity recovery tests also continue to pass. * fix(world-local): converge legacy hook claims via recovery-marker sidecar; replace tag-proxy test with real subprocess workers Addresses follow-up review on PR #2295. Two distinct issues, both flagged by pranaygp as P1: 1. The fallback path for token claims written by versions before eventId was persisted inline (legacy claims after upgrade) still permitted the same cross-process corruption the inline fast path was fixed to prevent. Two processes both reading a legacy claim each generated their own eventId, landed their writeExclusive(eventPath) calls at different paths, and appended two hook_created events for the same (runId, hookId). Existing persisted claims after a real upgrade are exactly the state the crash-recovery branch needs to repair, so leaving the legacy path non-convergent is silent corruption, not backward compatibility. 2. The committed cross-worker convergence test used two tagged storage instances sharing one directory as a proxy for separate processes. But tags change the destination filename (events/wrun_X-evnt_Y.worker-a.json vs ...worker-b.json), so two tagged workers can each writeExclusive their own event at different paths and both fulfill. The Map-by-eventId deduplication in the assertion then masked the duplicate publication, so the test passed for the wrong reason. Implementation: - New HookRecoveryMarkerSchema (`{ eventId, hookId, runId }`) and HookRecoveryMarkerPath helper. The marker is a sidecar at hooks/tokens/<hash>.recovery.json, written via writeExclusive so the first cross-process retry pins its candidate eventId as canonical; subsequent retries read the marker and adopt that eventId. Together with the existing writeExclusive(eventPath) in the outer publish, this gives the legacy-fallback path the same single-event convergence guarantee as the inline-eventId fast path. - pinCanonicalEventIdForLegacyClaim() encapsulates the marker write-or-read. A stale marker for a different (runId, hookId) (token-reuse with leaked state) is overwritten best-effort — the common cross-worker race for the same hook still converges; only the narrow stale-token-reuse case loses convergence. - hook_disposed now also deletes the recovery marker when it deletes the token constraint file, preventing a future legacy recovery for a recycled token from latching onto a stale eventId. - The dedup branch unified: existingClaim.eventId for new claims, pinCanonicalEventIdForLegacyClaim() for legacy ones. Removed the now-redundant findHookCreatedEvent helper — the writeExclusive(eventPath) in the outer publish is the authoritative duplicate-vs-orphan detector. Tests: - New test fixture test-fixtures/hook-race-worker.ts (TypeScript, run via child_process.fork with tsx as execPath — tsx is a transitive dev dep via vitest). Each subprocess gets its own createStorage(testDir) so the in-process hookLocks Map cannot serialize across workers. - Replaced the tag-proxy test with "converges same-hook creation across separate OS processes to one event". Spawns workerCount subprocesses, releases them from a barrier into the same hook_created, asserts exactly one fulfilled + (N-1) rejected with EntityConflictError, and asserts directly on the raw events.list() result (no Map dedup) that the number of hook_created entries equals the number of logical creations. - Added "converges same-hook creation across processes when only a legacy token claim exists". Same shape, but pre-seeds the legacy claim format (`{ token, hookId, runId }` with no eventId) before each race. Verified to FAIL on 7ce66551b (both subprocesses fulfill, no convergence) and pass on this commit. - Also verified the new-eventId subprocess test FAILS when the event write is reverted to writeJSON (TOCTOU), confirming it exercises the writeExclusive-based cross-process arbitration. Both prior orphaned-claim / orphaned-entity recovery tests also continue to pass. * fix(world-local): per-lifetime recovery markers, restore event-log probe, fix CI tsx resolution Addresses three P1 review comments on PR #2295. 1. Stale recovery marker leaking across token-reuse lifetimes (pranaygp): The previous marker path used `hashToken(token)` so a stale marker for run A could leak into run B's recovery when the same token was reused after run A terminated through normal lifecycle. `deleteAllHooksForRun()` and tagged `world.clear()` deleted the token constraint and hook entity but NOT the marker sidecar, so the next legacy claim on the same token entered the stale-marker overwrite branch and the workers overwrote it non-atomically, yielding divergent publication. Fix: - Marker path now hashes `(token, runId, hookId)` together (`hookRecoveryMarkerPath` in storage/helpers.ts). Different lifetimes can never share a marker, so the stale-marker overwrite branch is removed entirely. - `hookRecoveryMarkerPath` is moved to helpers.ts and shared across events-storage.ts, hooks-storage.ts, and index.ts. - `deleteAllHooksForRun()` and tagged `world.clear()` now also delete the recovery marker for each hook (disk hygiene; per- lifetime identity makes leaks no longer corrupting). - `hook_disposed` now uses the new per-lifetime marker path too. 2. Duplicate `hook_created` event when a legacy claim's event was already published (VADE bot, also implied by pranaygp's analysis): Removing the event-log probe from the legacy fallback let a post- upgrade retry pin a new canonical eventId via the marker and publish a duplicate event at that path, even when the original pre-upgrade writer had already successfully published the event with its own (different) eventId. Fix: - Restore `findExistingHookCreatedEventId()` (renamed and made to return the eventId for clearer semantics). - Legacy fallback now probes the event log BEFORE pinning the marker; if a matching `hook_created` event already exists, throw `EntityConflictError` so the runtime's concurrent-replay catch path swallows the retry. - Inline-`eventId` fast path does NOT need the probe — the claim itself is the durable convergence key. 3. CI failure: tsx not resolvable under pnpm isolated linking (pranaygp; confirmed by ubuntu/windows unit test 60s timeouts): The previous test hard-coded `node_modules/.bin/tsx` assuming tsx would be hoisted there. But tsx was only a transitive peer dep via vitest, and pnpm's isolated linking does NOT link transitive peer deps into the workspace bin after a fresh install — so neither root nor package-local `.bin/tsx` existed in CI, the subprocess fork never started, and the barrier hung until vitest killed the test. Fix: - Add `tsx` as a direct `devDependency` of `@workflow/world- local` (pinned to 4.20.6 to match the existing transitive resolution). - Resolve via `import.meta.resolve('tsx/package.json')` and read the `bin` field dynamically, so we adapt to wherever pnpm links tsx for this package — not a hard-coded layout. - Lazy-init the resolver (no module-load IIFE) so an absent tsx fails only the convergence tests, not all 376 tests in the file. - Surface a clear error message if resolution fails, calling out the cause (transitive vs direct deps) for future readers. Also: harden the barrier helper so `error` events and pre-ready exits resolve BOTH `readyPromises` and `donePromises`, then `SIGKILL` siblings. Previously a broken child only resolved `donePromises`, leaving `Promise.all(readyPromises)` pending until the per-test timeout (60s in CI). Regression tests added: - `legacy claim whose hook_created event was already published does not append a duplicate event` — pre-seeds a legacy claim AND a pre-existing `hook_created` event with a different eventId, asserts the retry throws EntityConflictError and the log still has exactly the original event. - `converges legacy claim recovery across run lifetimes after token reuse` — runs pranaygp's full lifecycle path: race subprocess workers on run A's legacy claim, terminate run A via `run_completed` (triggers `deleteAllHooksForRun`), reuse the token in a legacy claim for run B, race subprocess workers again, asserts exactly one fulfillment + one `EntityConflictError` per race and exactly one `hook_created` event per run. Both new tests verified to fail on 2c673e436 (after rebuilding): the published-event test throws via duplicate publish instead of EntityConflictError, the token-reuse test sees both run B workers fulfill (2 events instead of 1). The existing orphaned-claim and orphaned-entity recovery tests also continue to pass. CI loop confirmed to be repaired locally by spawning subprocesses via the new resolver and intentionally breaking the worker fixture to verify the helper fails fast (~500ms) instead of hanging at the barrier. * fix(world-local): defer hook entity write until event publish commits Addresses karthikscale3's P1 review comment on PR #2295. The dedup-recovery path used to write the hook entity BEFORE the outer event publish proved whether the attempt was repairing a missing event or just colliding with an already-published `hook_created`. For already-committed duplicates, the event write then throws `EntityConflictError`, but the hook entity had already been overwritten with the retry's payload — leaving the durable hook entity and the event log inconsistent (e.g. the entity reflects the retry's metadata while the event still carries the original). karthikscale3 reproduced this on the prior head by creating `hook_created` with metadata `{ v: "a" }`, then retrying the same `(runId, hookId, token)` with metadata `{ v: "b" }` and `isWebhook: false`: the retry threw `EntityConflictError` but `hooks.get()` returned the retry's payload. Fix: defer the hook entity write until AFTER the outer `writeExclusive(eventPath)` commits. The branch now only captures the entity-to-write and its overwrite options; the actual write happens immediately after the event publish in the shared trailing block. A retry that ends in `EntityConflictError` (the event was already published) now leaves the entity untouched. The first-writer happy path and all recovery paths (orphaned- claim, orphaned-entity, cross-worker convergence, legacy claim, token-reuse across lifetimes) are unaffected — they all reach the event publish successfully, then the entity write runs as before. Regression test `does not mutate an already-committed hook entity when a duplicate hook_created retry collides` added to world-local: runs karthikscale3's exact scenario and asserts the persisted entity still carries the original metadata and isWebhook. Verified to fail on the prior commit (persisted metadata = 0xbb instead of 0xaa) and pass on this commit after rebuilding. Parallel guard test `does not mutate an already-committed hook entity when a duplicate hook_created retry collides` added to world-postgres. Postgres already protected this via `onConflictDoNothing()` on the hook INSERT, but the test guards against a future regression that adds an UPDATE/UPSERT to the dedup path. * refactor(world-local): per-instance in-process locks; drop tsx subprocess test plumbing You were right that the tsx subprocess machinery was overkill for a storage-level convergence test. Replaced with a simple two-instance in-process test that exercises the same cross-process semantics without spawning anything. The trick: `stepLocks` and `hookLocks` were module-level Maps shared by all `createEventsStorage` calls in the same process. Move them inside the function so each `createStorage(dir)` call gets its own lock map. Two storage instances sharing one data directory then behave exactly like two separate OS processes: - independent in-process `hookLocks` Maps (no in-process serialization between them), and - a shared filesystem (so the on-disk `writeExclusive` claim / marker / event publish primitives are the only thing arbitrating convergence). This is also a real architectural improvement — the global lock map was always a leaky abstraction that made unit-test simulation of the cross-process path awkward. Changes: - `stepLocks` and `hookLocks` moved from module scope into `createEventsStorage`. `withStepLock` and `withHookLock` wrappers collapsed into direct `withInProcessLock(map, key, fn)` calls at the two call sites that need them. - The three convergence regression tests in `storage.test.ts` now use `const workerA = createStorage(testDir); const workerB = createStorage(testDir);` and race `Promise.allSettled` of `events.create` from both — no subprocess, no IPC, no barrier helper, no `raceHookCreatedAcrossProcesses`. Same assertions (exactly one fulfillment + N-1 `EntityConflictError` per race, raw `events.list()` shows exactly one `hook_created` per logical creation — no Map dedup) so the regression catches are identical. - Removed: `tsx` devDep, `test-fixtures/hook-race-worker.ts`, `HOOK_RACE_WORKER` / `resolveTsxLoaderUrl` / `TSX_BIN` / `raceHookCreatedAcrossProcesses` and the `fork`/`fileURLToPath` imports they pulled in. Verified (after rebuilding world-local): - All 379 tests pass on macOS in ~1s (was ~6.7s with subprocesses). - Convergence tests confirmed to still catch the bugs: temporarily reverted the `eventId = canonicalEventId` adoption → both workers fulfilled (2 events instead of 1). Temporarily reverted the legacy-claim marker pin → same: both workers fulfilled. - No subprocess machinery means no Windows-specific quirks (cli.mjs shebang, .cmd wrappers, .bin hoisting under pnpm isolated linking, etc.) that produced the Windows CI 60s timeouts. - World-postgres still has its own parallel guard test for the karthikscale3 "no-mutate-on-duplicate" regression; that one exercises real DB concurrency and is unaffected by this change. Full repo `pnpm test` (43 packages) and the `parallelStepsThenWebhookWorkflow` e2e test against world-local both green. * fix(world-local): repair event-first hook orphans from the persisted event; skip #1665 e2e on world-postgres - A crash between the hook_created event publish and the deferred hook entity write left the event committed with the entity missing and unrepairable (retries threw EntityConflictError without materializing the entity). Retries now rebuild the entity from the PERSISTED event's payload — never the retry's eventData — via a race-safe writeExclusive, on both the canonical-eventId collision path and the legacy-claim probe path. - Skip parallelStepsThenWebhookWorkflow e2e on world-postgres: the same-tick replay pattern surfaces a separate pre-existing step_started ordering bug there (#2331). --------- Co-authored-by: Peter Wielander <peter.wielander@vercel.com> |
||
|
Pranay Prakash
|
ae8d6feeda |
Add native v4 workflow attribute events (#2226)
* Add native workflow attribute events * Fix abbreviated attributes docs sample * Document attribute replay ordering for step races * Address native attribute review feedback * Validate before claiming attr_set dedup lock; clearer start() attribute errors - world-local: claim the attr_set correlation lock only after validation, so a validation failure does not permanently mark the correlationId as written and wedge the run in a re-invoke loop on retry - world-postgres: distinguish a concurrently-deleted run from a cap violation when the guarded attributes update matches no rows - core: reject non-string initial attribute values in start() with a clear error instead of a downstream schema failure Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Add attribute edge-case tests across all layers - core: normalizeAttributeChanges unit tests (non-object inputs, FatalError wrapping, key/value/batch limits, boundary lengths, UTF-8 byte counting) - core: start() rejects reserved keys, oversized keys/values, and over-cap initial attribute batches before any write - world-local + world-postgres: per-run cap enforced against existing attributes (upsert-at-cap allowed, removal frees room), oversized values rejected on attr_set, invalid initial attributes rejected on run_created - e2e: validation DX workflow asserting every invalid write throws a catchable FatalError naming the violated rule and limit, with the run staying healthy; start() rejects invalid initial attributes client-side Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Remove accidentally committed local e2e diagnostics artifact Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Bump world-vercel to spec version 4 for native attributes The deployed workflow-server (vercel/workflow-server#469) materializes native attr_set events and accepts initial run attributes, but world-vercel still advertised spec v3 — so start(..., { attributes }) rejected itself client-side ('requires spec version 4') on every Vercel deployment, failing the new e2e seeding test across the prod matrix. New runs are now stamped v4. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Reject duplicate correlated attr_set before materializing in Postgres A redelivered duplicate — including one carrying different changes for the same correlationId — previously re-applied the run attributes update and only then failed the event insert, leaving the snapshot out of sync with the event log. Pre-check the event log for the correlationId before mutating; the unique index still guards the truly-concurrent race, which is idempotent (deterministic replay carries identical changes). Also apply the suggested docs wording for initial attributes. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Apply suggestion from @VaguelySerious Signed-off-by: Peter Wielander <mittgfu@gmail.com> * Fail the run on World-rejected attribute writes; un-nest runtime test Two fixes from review: - runtime.test.ts: the pre-existing test "propagates transient step_created failures..." was accidentally nested inside the new attribute-race test, failing the new test ("Calling the test function inside another test function is not allowed") and preventing the old test from running. Restored it verbatim at describe level. - A workflow-body attr_set the World rejects as invalid (e.g. the cumulative per-run attribute cap, which only the World can check) is deterministic: redelivering the orchestrator message replays the same write into the same rejection, wedging the run in redelivery with no terminal event. handleSuspension now wraps such rejections in FatalError, and workflowEntrypoint fails the run with the validation error instead of rejecting the delivery. Transient storage errors still propagate and retry via redelivery. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Signed-off-by: Peter Wielander <mittgfu@gmail.com> Co-authored-by: Claude Fable 5 <noreply@anthropic.com> Co-authored-by: Peter Wielander <mittgfu@gmail.com> Co-authored-by: Peter Wielander <peter.wielander@vercel.com> |
||
|
Peter Wielander
|
1e6b1fdea2 |
Attributes MVP (experimental and write-only) and CI hardening (#2134)
* fix(core): scan inline sourcemaps during error remapping * Attributes MVP (experimental and write-only) (#2088) |
||
|
Pranay Prakash
|
9d2a9261fd |
Expose conflicting run id on hook conflicts (#2012)
* Expose conflicting run id on hook conflicts * Mark hook conflict run id as future required * Address hook conflict docs review * Address hook conflict review comments * Fix hook conflict docs typecheck |
||
|
Nathan Rajlich
|
5f22832675 |
Serialize run_failed/step_failed errors through serialization pipeline (#1851)
* Serialize run_failed/step_failed errors through serialization pipeline
Switch run_failed, step_failed, and step_retrying events to persist
the full thrown value via the workflow serialization pipeline (as
SerializedData / Uint8Array) instead of a lossy { message, stack, code }
StructuredError shape. Consumers hydrate via hydrateRunError /
hydrateStepError to reconstruct the original thrown value, preserving
Error subclass identity, cause chains, and custom properties.
- WorkflowRun.error and Step.error are now SerializedData
- WorkflowRun gains a top-level errorCode plaintext field
- WorkflowRunFailedError.cause is now the hydrated thrown value
- Adds world-postgres migration 0010_add_error_code.sql
- Legacy pre-pipeline errorJson records surface as undefined on read
* Update Next.js workbenches for new WorkflowRunFailedError.cause type
cause is now `unknown` (the hydrated thrown value) rather than
`Error & { code }`. Defensively extract Error-shaped fields when the
hydrated value is an Error, otherwise round-trip the raw value, and
expose the new `errorCode` classification field.
* Update docs for WorkflowRunFailedError.cause: unknown
The hydrated `cause` is now `unknown` (the original thrown value
through the serialization pipeline) and the error classification has
moved to the top-level `errorCode` property. Update the two affected
docs pages and the `TSDoc` interface to reflect the new shape, and
narrow `cause` with `instanceof Error` before accessing fields.
* Expand test coverage for the run/step error serialization pipeline
Unit tests:
- 19 new dehydrate/hydrate{Step,Run}Error round-trip tests covering
FatalError, plain Error, built-in Error subclasses, non-Error thrown
values (string, plain object), cause chains, encryption round-trip,
the binary format prefix contract, and the unserializable / unknown-
format error paths.
- 5 new tests for Run.returnValue when the run is failed: hydrated
FatalError + cause as cause, plain Error preservation, non-Error
thrown values surfaced verbatim, cross-class cause chains, and the
hydration-failure fallback that still surfaces errorCode.
E2E tests (new, in 99_e2e.ts + e2e.test.ts):
- Step throw → workflow catch round-trips a FatalError with a TypeError
cause chain, asserting class identity, fatal marker, and cause name +
message all survive the step_failed event pipeline.
- Workflow throw → run_failed reaches status with the new
top-level errorCode metadata exposed (cause-shape coverage lives at
the unit level, since the SWC plugin's class registration is not
invoked in the plain-Node e2e runner).
- Workflow throw of a non-Error value round-trips that value verbatim
as WorkflowRunFailedError.cause.
Adjustments to existing assertions:
- error.cause is now ; tests narrow with
and use the new top-level field instead of .
- step.error / run.error from CLI --withData are now hydrated payloads:
unregistered class instances surface as Instance refs whose
carries the original message + stack.
Observability hydration:
- hydrateStepIO / hydrateWorkflowIO in serialization-format.ts now
hydrate the field via hydrateData, so the CLI and web UI
continue to surface readable run/step error messages and stacks.
* Tighten error serialization changeset description
* Trim error serialization changeset to a single sentence
* Resolve FatalError/RetryableError revivers via cross-realm registry
When a workflow runs in a Node `vm` context, its bundled
`@workflow/errors` is a different module instance than the host's
import (separate prototype chains, separate class identity). Calling
`new FatalError(...)` from the host-side reviver produces a
host-realm instance that fails `err instanceof FatalError` checks
in the workflow code — even when the serialized payload was correctly
tagged via the dedicated `FatalError` reducer.
Surfaced by the local-prod e2e "step throw round-trips FatalError"
test on Next.js Turbopack: each route gets its own bundled chunk, so
the flow handler's `@workflow/errors` and the workflow VM bundle's
`@workflow/errors` are two distinct copies of the same module.
Fix:
- Each bundled copy of `@workflow/errors` self-registers its
`FatalError` and `RetryableError` classes on `globalThis` via
`Symbol.for("@workflow/errors//FatalError")` /
`Symbol.for("@workflow/errors//RetryableError")`. First load wins
per realm; the descriptor is non-writable / non-configurable to make
accidental clobbering loud.
- The revivers in `@workflow/core`'s common reducers module read the
consumer's `globalThis` (passed in as `global`) to pick up the
realm-local class, falling back to the host-imported class when no
registration is present (e.g. in the CLI / test runner).
* Use `types.isNativeError` to remap workflow stacks across VM realms
The runtime's run-failure path computes a source-map-remapped stack
and then assigns it back onto the thrown value via `if (err
instanceof Error) err.stack = errorStack`. Workflows run inside a
Node `vm` context, so a workflow-thrown error is an instance of the
VM realm's `Error` — `instanceof` against the host realm's
`Error` returns `false`, the assignment is skipped, and the
serialized `run_failed` event carries the un-remapped (bundled-line-
number) stack instead of the source-mapped one.
Switch the gate to `types.isNativeError`, which uses V8's internal
type tag and works across realms — same approach already in place
for the serialization reducers.
Caught by the local-prod e2e "nested function calls preserve message
and stack trace" and "cross-file imports preserve message and stack
trace" tests, which assert that the persisted run-error stack
contains `99_e2e.ts` / `helpers.ts`.
* Sync CLI revivers with core + add toJSON shim for Error subclasses
Two issues with the CLI's hand-rolled reviver list:
1. It hadn't been updated for the new first-class Error subclass
reducers (`TypeError`, `RangeError`, `FatalError`, `RetryableError`,
etc.). devalue throws "Unknown type X" when it encounters a
reduced value with no matching reviver, and `hydrateResourceIO`
swallows that error and surfaces the raw `Uint8Array` payload —
so `step.error` / `run.error` showed up as raw byte dumps in
`workflow inspect` output.
2. Even with all the right revivers, `Error.prototype`'s `message`
/ `stack` / `cause` are non-enumerable, so `JSON.stringify`
(used by `workflow inspect --json`) drops them — leaving the
subclass-specific enumerable fields (e.g. `FatalError.fatal`)
visible but the actual error data missing.
Fix:
- Build the CLI reviver set on top of `getCommonRevivers()` from
`@workflow/core` so the CLI stays in sync with the runtime's
reducer set automatically. New core reducers/revivers will Just
Work without any CLI-side change.
- Wrap each Error reviver from the common set with a thin shim that
attaches a non-enumerable `toJSON` method to the produced
`Error` instance. `JSON.stringify` calls `toJSON` and gets a
full object (`name` + `message` + `stack` + `cause` + any
enumerable subclass fields like `fatal` / `retryAfter` /
`errors`); `util.inspect` ignores `toJSON` and renders the
canonical `Error: msg\\n at ...` format. Best of both worlds for
CLI output without compromising the runtime hydration path.
Caught by the local-prod e2e "basic step error preserves" and
"cross-file step error preserves" tests, which read
`failedStep.error.message` / `.stack` from the CLI's JSON output.
* Clarify parseErrorJson JSDoc to match its always-null return
The previous JSDoc described preserving legacy values "for best-effort
hydration" which contradicted the implementation, where legacy errors
are intentionally surfaced as absent (the pre-pipeline shapes can't be
hydrated by the new error revivers). Rewrite the comment so the contract
matches behavior. Also rename the now-unused parameter to `_errorJson`
to reflect that the function ignores it.
Caught by a code review on #1851.
* Refine error-handler ergonomics on the step / run hot paths
Three review-driven adjustments that all touch the queue handlers and
their interaction with the error serialization pipeline:
1. Memoize the per-run encryption key fetch. The step handler used to
eagerly fetch + import the key at the top of every step delivery so
the value would be in scope for every potential dehydrateStepError
path. That pessimized step-started early-return cases (the fetch
happens unconditionally even when the step never reaches user code)
and required duplicating the same boilerplate at four call sites in
runtime.ts. Introduce `memoizeEncryptionKey(world, run)` in
runtime/helpers.ts that returns a lazy, single-fetch accessor;
step-handler / runtime call sites use `await getEncryptionKey()`
instead. The first caller pays the fetch cost, subsequent callers
await the cached promise, and steps that fail before any
encryption-aware work happens skip the fetch entirely.
2. Preserve the prior attempt's serialized error as the cause on the
defensive max-retries-exceeded `step_failed` re-invocation guard.
The existing comment explicitly opted out of cause attachment, but
the symmetric post-failure path below already does this and the
reviewer is right that consumers shouldn't have to walk the
step_retrying event history to recover the underlying error. Best-
effort: if hydration of the prior `step.error` throws, fall back
to a FatalError without cause rather than letting the event write
itself fail.
3. Document the intentional `unflatten` throw in
`hydrateStepError` / `hydrateRunError` for non-Uint8Array input.
SDK version is pinned per workflow run via skew protection so the
non-binary branch is dead in production; if a misshapen value
reaches it, surfacing the throw via the surrounding o11y try/catch
is more debuggable than masking it. Add a comment so future
reviewers don't reach for a defensive fallback.
A standalone `falls back to plaintext` suggestion on the run_failed
key fetch was rejected: when encryption is configured we should fail
loudly rather than silently emit plaintext error data. The queue's
redelivery semantics will retry the key fetch; persistent KMS outages
get logged with the existing "persistent error preventing the run from
being terminated" message rather than a security regression.
* Hydrate `event.eventData.error` in event listings
`hydrateEventData` enumerated the per-event fields that need
hydration (`result`, `input`, `output`, `metadata`, `payload`)
but omitted the new `error` field on `step_failed`,
`step_retrying`, and `run_failed` events. Without this branch,
o11y tools that list events (e.g. `workflow inspect events`) surface
the raw `Uint8Array` payload instead of a hydrated
`{ name, message, stack, … }` object even though the entity-level
`Run.error` / `Step.error` paths already hydrate.
Mirrors the existing per-field branches; the `try/catch` leaves the
field un-hydrated on parse failure rather than failing the whole
event view. Adds a unit test.
* Use `.is()` static checks in `classifyRunError` for cross-realm safety
Workflows execute inside a separate `vm` realm: the
`WorkflowRuntimeError` class bundled into the workflow code and the
host-imported one are distinct constructors, so an
`err instanceof WorkflowRuntimeError` check on a VM-thrown error
returns `false` and we'd misclassify genuine runtime errors (corrupted
event log, missing timestamps, workflow/step not registered) as user
errors.
Switch to each subclass's `.is()` static (a name-based duck check that
works across realms). Since `WorkflowRuntimeError.is` only matches its
own concrete name, enumerate every concrete subclass we want to
recognize (`StepNotRegisteredError`, `WorkflowNotRegisteredError`)
in a `RUNTIME_ERROR_CHECKS` table; keep that table in sync with the
class hierarchy in `@workflow/errors`.
Existing `classify-error.test.ts` already covers `WorkflowRuntimeError`
and `WorkflowNotRegisteredError` cases — both still pass.
* Add e2e coverage for step throws of non-Error values
We had `errorWorkflowThrowNonErrorValue` (workflow body throws a plain
object — round-trips verbatim as `WorkflowRunFailedError.cause`) but
no symmetric coverage for the step-throw side. Step-throw goes through
a different code path: non-Error values aren't recognized as
`FatalError` (no `name === 'FatalError'`) nor `RetryableError`,
so they take the transient retry path. After max retries the runtime
wraps the original thrown value as `cause` on a fresh `FatalError`
which the workflow's catch block then sees.
Add a workflow that throws a recognizable plain object from a step
with `maxRetries = 0` (so we exhaust on first attempt and avoid a
long test wait) and a workflow that asserts the wrapped FatalError
shape: `isFatal`, `instanceof FatalError`, message includes the
original object's serialized form, `cause` is the original non-Error
object verbatim with structure preserved.
Documents the current retry-then-wrap behavior so any future change
to "non-Error throws skip retries" semantics has to update the test.
* Note legacy postgres error-data loss in the run/step error changeset
Pre-upgrade failed runs that wrote into world-postgres's deprecated
`error` text column can't be hydrated through the new pipeline (the
shape is incompatible with the new revivers). The new runtime
intentionally surfaces them as `error: undefined` on read; the
original payload is still readable directly from the `errorJson`
column for manual inspection. Add a one-sentence note to the
changeset's migration text so consumers upgrading don't get blindsided
by suddenly-empty error fields on historical runs.
|
||
|
Nathan Rajlich
|
7c45e9e213 |
Enforce per-(run, correlation) uniqueness for entity-creating events in world-postgres (#1878)
Adds a unique partial index on workflow_events(run_id, correlation_id, type) filtered to step_created/hook_created/wait_created, and translates the resulting unique-violation (pg code 23505, surfaced via DrizzleQueryError.cause) into EntityConflictError. The steps table already deduped via onConflictDoNothing, but the event row still inserted, leaving duplicate events in the log. Now both rows are kept consistent and the runtime's existing dedup catch path handles concurrent writers cleanly. |
||
|
Peter Wielander
|
66d49c0db6 | [world] Restructure stream interface, require run ID for all step and stream operations (#1293) | ||
|
Peter Wielander
|
5f138f2cee | [core] [world] Gate CBOR queue transport on specVersion (#1627) | ||
|
James Berry
|
5502438bac |
[world-postgres] Migrate client from postgres.js to pg (#1484)
|
||
|
Pranay Prakash
|
aee035f944 |
refactor: replace HTTP status code checks with semantic error types (#1342)
* feat: classify run failure error codes and improve error logging - Add RUN_ERROR_CODES (USER_ERROR, RUNTIME_ERROR) to @workflow/errors - Populate errorCode in run_failed events via classifyRunError() - Update web UI StatusBadge to show amber dot for infrastructure errors - Improve world-local queue error logging (concise, no body dump) - Improve schema validation error messages (concise, verbose behind DEBUG) - Add e2e tests for error code flow and infrastructure error retry Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat: add semantic error types to replace HTTP status code checks in runtime Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat: classify run failure error codes and improve error logging - Add RUN_ERROR_CODES (USER_ERROR, RUNTIME_ERROR) to @workflow/errors - Populate errorCode in run_failed events via classifyRunError() - Update web UI StatusBadge to show amber dot for infrastructure errors - Improve world-local queue error logging (concise, no body dump) - Improve schema validation error messages (concise, verbose behind DEBUG) - Add e2e tests for error code flow and infrastructure error retry Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat: classify run failure error codes and improve error logging - Add RUN_ERROR_CODES (USER_ERROR, RUNTIME_ERROR) to @workflow/errors - Populate errorCode in run_failed events via classifyRunError() - Update web UI StatusBadge to show amber dot for infrastructure errors - Improve world-local queue error logging (concise, no body dump) - Improve schema validation error messages (concise, verbose behind DEBUG) - Add e2e tests for error code flow and infrastructure error retry Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * address PR review comments - Remove dead `meta` option from TooEarlyError constructor (TooTallNate) - Extract `throwWithTrace` helper to deduplicate span recording in world-vercel makeRequest (TooTallNate) - Restore `maxAttempts` const for stable retry count logging (TooTallNate) - Fix behavioral regression: add WorkflowAPIError 404 fallback in suspension-handler hook disposal to handle world-vercel path where makeRequest doesn't map 404 to HookNotFoundError (TooTallNate) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: translate 404 to HookNotFoundError at the world-vercel boundary Move the 404 → HookNotFoundError translation into world-vercel's createWorkflowRunEvent, where we know the event type context. For hook-related events (hook_created, hook_disposed, hook_received, hook_conflict), a 404 from the server means the hook was not found. This removes the WorkflowAPIError 404 fallback from the runtime's suspension-handler, keeping the runtime fully decoupled from HTTP status codes. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: parse Retry-After for 425 responses and narrow hook event set - Parse Retry-After header unconditionally so TooEarlyError gets the server-provided delay instead of always falling back to ~1s - Narrow hookEventsRequiringExistence to only hook_disposed and hook_received (matching world-local's set), since hook_created and hook_conflict don't imply the hook must already exist Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * rename WorkflowAPIError to WorkflowWorldError Breaking change: rename WorkflowAPIError → WorkflowWorldError to better reflect that this error represents world (storage backend) failures, not HTTP API errors specifically. Updated across all packages: errors, core, world-local, world-vercel, world-postgres, workflow, and web. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com> |
||
|
Nathan Rajlich
|
1060f9d04a | Change user input/output to be binary data at the World interface (#853) | ||
|
Pranay Prakash
|
4966b728a8 |
implement event-sourced architecture (#621)
* perf: implement event-sourced architecture
* Apply suggestions from code review
* Improve invalid event log handling in step/hook/wait
* Handle serialized workflow run errors correctly
* log error in failing test
* Handle queue idempotency in vercel world
* hotfix for error propogation
* Fix: Incorrect HTTP status code 409 should be 410 for terminal run state rejections in postgres storage
* Fix: The code attempts to pass an unsupported `fatal` property when creating a `step_failed` event. The TypeScript schema for `step_failed` events only allows `error` and `stack` properties, so the `fatal` property causes a compilation error.
This commit fixes the issue reported at packages/core/src/runtime/step-handler.ts:133-139
## TypeScript error: Invalid property 'fatal' in step_failed event
**What fails:** TypeScript compilation fails in `packages/core` due to an invalid property in the `step_failed` event creation.
**How to reproduce:**
```bash
cd /vercel/sandbox/primary
pnpm run -F @workflow/core build
```
**Result:**
```
src/runtime/step-handler.ts(133,32): error TS2769: No overload matches this call.
Overload 1 of 2, '(runId: null, data: { eventType: "run_created"; ... }, gave the following error.
Argument of type 'string' is not assignable to parameter of type 'null'.
Overload 2 of 2, '(runId: string, data: CreateEventRequest, params?: CreateEventParams | undefined): Promise<EventResult>', gave the following error.
Object literal may only specify known properties, and 'fatal' does not exist in type '{ error: any; stack?: string | undefined; }'.
```
**Issue:** The code attempted to pass a `fatal: true` property in the `eventData` object when creating a `step_failed` event. However, the event schema defined in `packages/world/src/events.ts` for `StepFailedEventSchema` only allows `error` and `stack` properties - the `fatal` property is not part of the schema.
Co-authored-by: Vercel <vercel[bot]@users.noreply.github.com>
* Fix: Code silently skips updating workflowRun if result.run is undefined, causing workflows to be incorrectly skipped instead of throwing an error
* Add hook_conflict event type for duplicate token detection
Implements hook_conflict events across all world implementations to handle
cases where a workflow attempts to use a hook token already claimed by another
workflow. Instead of throwing errors, the system now records hook_conflict
events in the event log, enabling deterministic replay.
- Add HookConflictEvent schema to @workflow/world
- Implement hook_conflict in world-local, world-postgres, and suspension-handler
- Update hook consumer to reject promises with WorkflowRuntimeError on conflict
- Add HOOK_CONFLICT error slug with documentation
- Add e2e and unit tests for conflict scenarios
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* Add changeset for hook_conflict events
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* Add unit tests for hook_conflict handling
- Add tests for hook_conflict event in workflow.test.ts
- Fix world-postgres test to not expect removed message field
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* Improve hook-conflict.mdx error guide
- Remove redundant third point in 'Why This Happens' section
- Add example showing how to handle WorkflowRuntimeError for hook conflicts
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* Fix docs validation: add hook-conflict to errors index
- Fix broken link in hook-conflict.mdx (/docs/foundations/webhooks -> /docs/api-reference/workflow/create-webhook)
- Add hook-conflict to errors index page so it's discoverable by the docs link validator
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* Fix world-local tests for hook_conflict event behavior
Update tests to expect hook_conflict events instead of thrown errors when
duplicate hook tokens are used. This aligns with the new event-sourced
approach where conflicts are recorded as events rather than thrown.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* Add specVersion property to World interface for backwards compatibility
- Add specVersion property to World interface to track world package version
- Add specVersion to WorkflowRun schema and run_created event data
- World implementations (vercel, local, postgres) set specVersion from npm version
- Server can use specVersion to route operations based on world version
- Add specVersion display to observability UI attribute panel
- Add spec_version column to postgres runs schema
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* Add migration for spec_version column in postgres schema
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* Add drizzle migration journal and snapshot for spec_version column
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* Regenerate postgres migration using drizzle-kit
- Properly generates migration with drizzle-kit CLI
- Removes deprecated 'paused' status from enum
- Adds spec_version column
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* Add backwards compatibility for event-sourced runs
- Add RunNotSupportedError for runs requiring newer world versions
- Add semver-based version utilities (isLegacyVersion) to @workflow/world
- World implementations check specVersion and route to legacy handlers
- Legacy runs (< 4.1.0): run_cancelled skips event storage, wait_completed stores event only
- New runs always get current world version (4.1.0-beta.0)
- Make EventResult.event optional for legacy compatibility
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* Refactor spec version from semver strings to integers
Replace semver-based version compatibility with explicit integer spec versions:
- SPEC_VERSION_LEGACY (1): pre-event-sourcing runs
- SPEC_VERSION_CURRENT (2): event-sourced architecture
Use branded SpecVersion type to enforce importing from @workflow/world.
Remove semver dependency from world, world-local, and world-postgres.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* Store error as CBOR in postgres world for consistency with input/output
- Add error_cbor bytea columns to workflow_runs and workflow_steps tables
- Deprecate text error column, rename to errorJson with fallback parsing
- Remove JSON.stringify from error writes (run_failed, step_failed, step_retrying)
- Add parseErrorJson helper for backwards compatibility with legacy data
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* Use WorkflowRuntimeError and improve run entity handling in core runtime
- Replace generic Error with WorkflowRuntimeError for runtime assertions
- Add explicit check for run entity in run_created response
- Use run.runId instead of event.runId for consistency
- Use actual run status instead of hardcoded 'pending' in attributes
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* Add specVersion to Step, Hook, and Event entities
- Add specVersion field to Step, Hook, and Event interfaces in @workflow/world
- Add spec_version column to steps, hooks, events tables in postgres schema
- Set specVersion to SPEC_VERSION_CURRENT when creating entities in all worlds
- Update migration to include spec_version columns for all entity tables
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* Refactor world-local storage into modular files
Split storage.ts (1041 lines) into smaller, focused modules:
- storage/filters.ts: Data filtering helpers
- storage/helpers.ts: ULID and date utilities
- storage/hooks-storage.ts: Hook CRUD operations
- storage/legacy.ts: Legacy event handling
- storage/runs-storage.ts: Run get/list operations
- storage/steps-storage.ts: Step get/list operations
- storage/events-storage.ts: Event create/list operations
- storage/index.ts: Main composition
Also extracted test helpers to test-helpers.ts for reusability.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* Remove genversion and World.specVersion property
The World.specVersion string property was never actually read - only
the numeric SPEC_VERSION_CURRENT is used for backwards compatibility.
- Remove genversion dependency and generated version.ts from @workflow/world
- Remove specVersion property from World interface and all implementations
- Minor fix: correct error message to reference 'workflow' package
- Minor fix: correct error source priority in world-vercel events.ts
- Minor fix: update comment in runtime.ts
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* Remove genversion from world-local, world-postgres, and world-vercel
These packages no longer need genversion since we removed the
World.specVersion property in the previous commit.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* Remove version.ts from .gitignore files
No longer needed after removing genversion.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* Add legacy/backwards compatibility tests
Tests for world-local and world-postgres covering:
- Legacy runs (specVersion < 2 or null/undefined)
- run_cancelled handling (updates run, no event stored)
- wait_completed handling (stores event only)
- Rejection of unsupported events
- Hook cleanup on cancellation
- Future runs (specVersion > current)
- Rejection with RunNotSupportedError
- Current version runs (normal processing)
- Legacy error parsing (errorJson field parsing)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* Add hook_received support for legacy runs
When resumeHook() is called on a legacy run (specVersion < 2), the
hook_received event was previously rejected. This adds support for
storing hook_received events on legacy runs without entity mutation,
matching the behavior of wait_completed.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* Fix missing genversion in world-vercel
Signed-off-by: Peter Wielander <mittgfu@gmail.com>
* Remove deprecated workflow_completed, workflow_failed, and workflow_started events
Replace with run_completed, run_failed, and run_started equivalents.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* Add specVersion to EventWithRefsSchema in world-vercel
The manually-created EventWithRefsSchema was missing the specVersion field,
which caused specVersion to be stripped when using lazy (refs) mode for events.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* Wire specVersion from client through world backends
- Core runtime now sends specVersion in run_created eventData
- world-local accepts specVersion from eventData (defaults to current)
- world-postgres accepts specVersion from eventData (defaults to current)
This matches workflow-server behavior where v2 endpoints accept
specVersion from the client, while v1 endpoints default to legacy.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* Move specVersion to event object level, propagate to entities
- Add specVersion to BaseEventSchema (event level, not eventData)
- Remove specVersion from RunCreatedEventSchema.eventData
- Core runtime sends specVersion on event object
- world-local reads specVersion from event, propagates to run/step/hook entities
- world-postgres reads specVersion from event, propagates to run/step/hook entities
This ensures specVersion flows from client through event to all created entities.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* Update specVersion to be optional in types for backwards compatibility
- specVersion is optional in all entity schemas (runs, steps, hooks, events)
for backwards compatibility with legacy data in storage
- Runtime always sends specVersion on event requests
- world-local and world-postgres provide fallback to SPEC_VERSION_CURRENT
- Test helpers include specVersion in all event creation calls
- EventWithRefsSchema in world-vercel defaults specVersion to 1 for legacy
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* Fix world-vercel queue tests missing VERCEL_DEPLOYMENT_ID setup
Two tests were calling queue.queue() without setting up
VERCEL_DEPLOYMENT_ID, causing them to fail with "No deploymentId
provided" error before reaching the code they were testing.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* Add specVersion to all event creation calls in core package
The server's CreateEventSchemaV2 requires specVersion on all events,
but only run_created was sending it. This caused 400 Bad Request errors
for all other event types (run_started, run_completed, run_failed,
run_cancelled, step_created, step_started, step_completed, step_failed,
step_retrying, hook_created, hook_received, wait_created, wait_completed).
Now all event creation calls include specVersion: SPEC_VERSION_CURRENT.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
---------
Signed-off-by: Peter Wielander <mittgfu@gmail.com>
Co-authored-by: Vercel <vercel[bot]@users.noreply.github.com>
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
Co-authored-by: Peter Wielander <mittgfu@gmail.com>
|
||
|
Peter Wielander
|
dd3db13d54 | [world] Remove pause and resume events, actions and states (#751) | ||
|
Gal Schlezinger
|
10ce313d56 |
postgres: fix tests (#394)
* postgres: use non-deprecated drizzle signatures Signed-off-by: Gal Schlezinger <gal@spitfire.co.il> * postgres: store metadata in the hook Signed-off-by: Gal Schlezinger <gal@spitfire.co.il> * core: do not rely on module cache for world config. instead, use a global and a symbol. this makes sure that streamers can use in-memory event emitters and that it won't be compiled away into the different flow.js and step.js files. this was figured out when i was adding a hooks tests to world-testing. Signed-off-by: Gal Schlezinger <gal@spitfire.co.il> * add postgres world to all workbench packages we try to run them with the postgres world but it's not installed Signed-off-by: Gal Schlezinger <gal@spitfire.co.il> * Replace jsonb with cbor because zero byte does not work in jsonb :( Signed-off-by: Gal Schlezinger <gal@spitfire.co.il> * fix error handling: attempts start at 0 now, and not 1 like when we released. so initial attempt in postgres should reflect that. Signed-off-by: Gal Schlezinger <gal@spitfire.co.il> * drain stuff Signed-off-by: Gal Schlezinger <gal@spitfire.co.il> * fallback metadata to metadataJson Signed-off-by: Gal Schlezinger <gal@spitfire.co.il> * Make code more readable Signed-off-by: Gal Schlezinger <gal@spitfire.co.il> * apply Vade fix Signed-off-by: Gal Schlezinger <gal@spitfire.co.il> --------- Signed-off-by: Gal Schlezinger <gal@spitfire.co.il> |
||
|
Adrian
|
281292cdd8 |
fix: world-postgres tests (#314)
* fix: world-postgres db:push script * fix: test truncating wrong table names |
||
|
Pranay Prakash
|
00b0bb9346 |
Proper error stack propogating (#280)
* Proper stacktrace propogation in world Proper stacktrace propogation in world * Merge Reconciliation * Standardize the error type in the world spec * Deduplicate vercel world utils * fix undefined type issue --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> |
||
|
Peter Wielander
|
0ce3197d02 | Disable postgres world tests running in win32, fix container setup for ubuntu (#215) | ||
|
Copilot
|
4a821fce7c |
Fix Windows support by normalizing path separators in workflow IDs (#150)
--------- Signed-off-by: Nathan Rajlich <n@n8.io> Co-authored-by: Nathan Rajlich <n@n8.io> |
||
|
Gal Schlezinger
|
4ca9a3edbd |
Introducing Workflow DevKit
build durable, resilient, and observable workflows. Co-authored-by: Nathan Rajlich <n@n8.io> Co-authored-by: Pranay Prakash <pranay.gp@gmail.com> Co-authored-by: Adrian <me@adriandlam.com> Co-authored-by: JJ Kasper <jj@jjsweb.site> Co-authored-by: Vercel Release Bot <88769842+vercel-release-bot@users.noreply.github.com> Co-authored-by: Peter Wielander <mittgfu@gmail.com> Co-authored-by: Hayden Bleasel <hello@haydenbleasel.com> Co-authored-by: Gal Schlezinger <gal@spitfire.co.il> Co-authored-by: Manuel Muñoz Solera <mamuso@mamuso.net> Co-authored-by: Garrett <garrett.tolbert@vercel.com> Co-authored-by: Lars Grammel <lars.grammel@gmail.com> Co-authored-by: Pooya Parsa <pyapar@gmail.com> Co-authored-by: Tom Dale <tom@tomdale.net> Co-authored-by: Vishal Yathish <135551666+visyat@users.noreply.github.com> Co-authored-by: josh <144584931+dancer@users.noreply.github.com> |