* feat: enforce max queue deliveries in handlers with graceful failure
Replace VQS maxDeliveries cap with handler-level enforcement. Handlers
now gracefully fail runs/steps after excessive queue redeliveries,
preventing "phantom stuck" runs.
- Add MAX_QUEUE_DELIVERIES constant (64) and enforce in both workflow
and step handlers with run_failed/step_failed events
- Remove maxDeliveries from VQS trigger configs (builders + sveltekit)
- Improve world-local queue: safety limit loop, structured logging
with runId/stepId, backoff delay on failures
- Add MAX_DELIVERIES_EXCEEDED error code
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* address PR review comments
- Expand max-delivery comments explaining minimal-work approach
- Make workflow handler error message verbose (matching step handler)
- Fix comment: "consume the message silently"
- Reduce local queue safety limit from 1000 to 256
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* change MAX_QUEUE_DELIVERIES to 48 and use 5s linear backoff locally
VQS uses linear 5s backoff for attempts 1-32, then exponential capped
at 2h. At 48 attempts total elapsed time is ~20h, safely under the
24h message visibility limit. Local world now uses 5s linear backoff
to approximate VQS timing.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* feat: add HookConflictError for hook token conflicts
Replace WorkflowRuntimeError with a dedicated HookConflictError class
for hook token conflicts. This correctly classifies the error as a
USER_ERROR (duplicate token is a user mistake) rather than a
RUNTIME_ERROR.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix: use HookConflictError.is() instead of instanceof in docs
The .is() static method handles cross-VM/realm boundaries where
instanceof can be unreliable.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* feat: classify run failure error codes and improve error logging
- Add RUN_ERROR_CODES (USER_ERROR, RUNTIME_ERROR) to @workflow/errors
- Populate errorCode in run_failed events via classifyRunError()
- Update web UI StatusBadge to show amber dot for infrastructure errors
- Improve world-local queue error logging (concise, no body dump)
- Improve schema validation error messages (concise, verbose behind DEBUG)
- Add e2e tests for error code flow and infrastructure error retry
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* feat: add semantic error types to replace HTTP status code checks in runtime
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* feat: classify run failure error codes and improve error logging
- Add RUN_ERROR_CODES (USER_ERROR, RUNTIME_ERROR) to @workflow/errors
- Populate errorCode in run_failed events via classifyRunError()
- Update web UI StatusBadge to show amber dot for infrastructure errors
- Improve world-local queue error logging (concise, no body dump)
- Improve schema validation error messages (concise, verbose behind DEBUG)
- Add e2e tests for error code flow and infrastructure error retry
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* feat: classify run failure error codes and improve error logging
- Add RUN_ERROR_CODES (USER_ERROR, RUNTIME_ERROR) to @workflow/errors
- Populate errorCode in run_failed events via classifyRunError()
- Update web UI StatusBadge to show amber dot for infrastructure errors
- Improve world-local queue error logging (concise, no body dump)
- Improve schema validation error messages (concise, verbose behind DEBUG)
- Add e2e tests for error code flow and infrastructure error retry
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* address PR review comments
- Remove dead `meta` option from TooEarlyError constructor (TooTallNate)
- Extract `throwWithTrace` helper to deduplicate span recording in
world-vercel makeRequest (TooTallNate)
- Restore `maxAttempts` const for stable retry count logging (TooTallNate)
- Fix behavioral regression: add WorkflowAPIError 404 fallback in
suspension-handler hook disposal to handle world-vercel path where
makeRequest doesn't map 404 to HookNotFoundError (TooTallNate)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix: translate 404 to HookNotFoundError at the world-vercel boundary
Move the 404 → HookNotFoundError translation into world-vercel's
createWorkflowRunEvent, where we know the event type context. For
hook-related events (hook_created, hook_disposed, hook_received,
hook_conflict), a 404 from the server means the hook was not found.
This removes the WorkflowAPIError 404 fallback from the runtime's
suspension-handler, keeping the runtime fully decoupled from HTTP
status codes.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix: parse Retry-After for 425 responses and narrow hook event set
- Parse Retry-After header unconditionally so TooEarlyError gets
the server-provided delay instead of always falling back to ~1s
- Narrow hookEventsRequiringExistence to only hook_disposed and
hook_received (matching world-local's set), since hook_created
and hook_conflict don't imply the hook must already exist
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* rename WorkflowAPIError to WorkflowWorldError
Breaking change: rename WorkflowAPIError → WorkflowWorldError to
better reflect that this error represents world (storage backend)
failures, not HTTP API errors specifically. Updated across all
packages: errors, core, world-local, world-vercel, world-postgres,
workflow, and web.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* Add isWebhook flag to prevent hooks from being resumed via public webhook endpoint
Hooks created with createHook() are now non-resumable via the public webhook
endpoint by default (isWebhook=false). Only hooks created with createWebhook()
set isWebhook=true, allowing them to be resumed via the public URL.
Also adds HookNotFoundError thrown by all world backends when a webhook
token doesn't match any hook, and an e2e test for the new behavior.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* Fix world-local: default isWebhook to false and fix test assertions
- Default isWebhook to false at write time in events-storage
- Default isWebhook to false at read time in hooks-storage (for old data)
- Update test assertions to match HookNotFoundError message
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* Fix: default isWebhook to true for backwards compat, add postgres migration
- Revert read-side default to `isWebhook ?? true` in world-local for
backwards compatibility with existing hooks that predate the field
- Add postgres migration 0009 to add `is_webhook` column with default true
- Update drizzle schema to match
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix(core): detect and fatal error on orphaned/invalid events in EventsConsumer
When an event log has duplicate or invalid events (e.g., 2 wait_completed for a single wait_created), the EventsConsumer gets stuck: the orphaned event has no callback to consume it, so eventIndex never advances, blocking all subsequent events and hanging the workflow forever. This adds deferred orphaned event detection that raises a WorkflowRuntimeError instead.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* add corrupted-event-log error slug and docs page
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* add changeset for orphaned event detection
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* address PR feedback on docs and JSDoc
- Add JSDoc to EventsConsumer constructor's onUnconsumedEvent parameter
- Rewrite docs to clarify this is a SDK/server bug, not user code
- Remove cancel/programmatic retry advice; run is already marked failed
- Emphasize error is uncatchable inside workflow code
- Add steps: upgrade package, retry via dashboard/CLI, report issue on GitHub
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* address PR review: structured options, required callback, reorder docs
- Change EventsConsumer constructor to use structured EventsConsumerOptions object
- Make onUnconsumedEvent required (all callsites provide it)
- Move corrupted-event-log card to last position in errors index
- Update all callsites in tests
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix docs code sample to pass type-checking
Use correct imports (getRun from workflow/api) and declare variables
to satisfy the docs-typecheck CI job.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* add e2e test for parallel sleep (Promise.all with 10x sleep)
Adds parallelSleepWorkflow that does Promise.all with 10 concurrent
sleep('1s') calls, and an e2e test asserting it completes or fails
within a reasonable time (not hanging).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix parallelSleep e2e test: await returnValue instead of checking status
run.status returns immediately and the workflow is still 'running'.
Use run.returnValue which polls until completion.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix parallelSleep e2e: assert parallel completion under 10s
The test verifies 10 concurrent sleep('1s') calls complete in parallel
(~1s) rather than serially (10s) or hanging indefinitely. 30s timeout
is sufficient since the workflow should finish in ~1-2s.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
* Use `@vercel/cli-auth` for auth token reading and OAuth refresh
Replace the manual auth.json reading logic in the CLI with the `@vercel/cli-auth` package, which handles credential storage via CredentialsStore and OAuth token refresh via the OAuth client.
Previously, the CLI would read the token from disk but had no refresh logic — if the token was expired, API calls would fail.
Now, getAuthToken() checks token expiry and automatically refreshes it using the stored refresh token before returning it.
* Remove dead logging
* perf: implement event-sourced architecture
* Apply suggestions from code review
* Improve invalid event log handling in step/hook/wait
* Handle serialized workflow run errors correctly
* log error in failing test
* Handle queue idempotency in vercel world
* hotfix for error propogation
* Fix: Incorrect HTTP status code 409 should be 410 for terminal run state rejections in postgres storage
* Fix: The code attempts to pass an unsupported `fatal` property when creating a `step_failed` event. The TypeScript schema for `step_failed` events only allows `error` and `stack` properties, so the `fatal` property causes a compilation error.
This commit fixes the issue reported at packages/core/src/runtime/step-handler.ts:133-139
## TypeScript error: Invalid property 'fatal' in step_failed event
**What fails:** TypeScript compilation fails in `packages/core` due to an invalid property in the `step_failed` event creation.
**How to reproduce:**
```bash
cd /vercel/sandbox/primary
pnpm run -F @workflow/core build
```
**Result:**
```
src/runtime/step-handler.ts(133,32): error TS2769: No overload matches this call.
Overload 1 of 2, '(runId: null, data: { eventType: "run_created"; ... }, gave the following error.
Argument of type 'string' is not assignable to parameter of type 'null'.
Overload 2 of 2, '(runId: string, data: CreateEventRequest, params?: CreateEventParams | undefined): Promise<EventResult>', gave the following error.
Object literal may only specify known properties, and 'fatal' does not exist in type '{ error: any; stack?: string | undefined; }'.
```
**Issue:** The code attempted to pass a `fatal: true` property in the `eventData` object when creating a `step_failed` event. However, the event schema defined in `packages/world/src/events.ts` for `StepFailedEventSchema` only allows `error` and `stack` properties - the `fatal` property is not part of the schema.
Co-authored-by: Vercel <vercel[bot]@users.noreply.github.com>
* Fix: Code silently skips updating workflowRun if result.run is undefined, causing workflows to be incorrectly skipped instead of throwing an error
* Add hook_conflict event type for duplicate token detection
Implements hook_conflict events across all world implementations to handle
cases where a workflow attempts to use a hook token already claimed by another
workflow. Instead of throwing errors, the system now records hook_conflict
events in the event log, enabling deterministic replay.
- Add HookConflictEvent schema to @workflow/world
- Implement hook_conflict in world-local, world-postgres, and suspension-handler
- Update hook consumer to reject promises with WorkflowRuntimeError on conflict
- Add HOOK_CONFLICT error slug with documentation
- Add e2e and unit tests for conflict scenarios
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* Add changeset for hook_conflict events
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* Add unit tests for hook_conflict handling
- Add tests for hook_conflict event in workflow.test.ts
- Fix world-postgres test to not expect removed message field
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* Improve hook-conflict.mdx error guide
- Remove redundant third point in 'Why This Happens' section
- Add example showing how to handle WorkflowRuntimeError for hook conflicts
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* Fix docs validation: add hook-conflict to errors index
- Fix broken link in hook-conflict.mdx (/docs/foundations/webhooks -> /docs/api-reference/workflow/create-webhook)
- Add hook-conflict to errors index page so it's discoverable by the docs link validator
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* Fix world-local tests for hook_conflict event behavior
Update tests to expect hook_conflict events instead of thrown errors when
duplicate hook tokens are used. This aligns with the new event-sourced
approach where conflicts are recorded as events rather than thrown.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* Add specVersion property to World interface for backwards compatibility
- Add specVersion property to World interface to track world package version
- Add specVersion to WorkflowRun schema and run_created event data
- World implementations (vercel, local, postgres) set specVersion from npm version
- Server can use specVersion to route operations based on world version
- Add specVersion display to observability UI attribute panel
- Add spec_version column to postgres runs schema
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* Add migration for spec_version column in postgres schema
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* Add drizzle migration journal and snapshot for spec_version column
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* Regenerate postgres migration using drizzle-kit
- Properly generates migration with drizzle-kit CLI
- Removes deprecated 'paused' status from enum
- Adds spec_version column
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* Add backwards compatibility for event-sourced runs
- Add RunNotSupportedError for runs requiring newer world versions
- Add semver-based version utilities (isLegacyVersion) to @workflow/world
- World implementations check specVersion and route to legacy handlers
- Legacy runs (< 4.1.0): run_cancelled skips event storage, wait_completed stores event only
- New runs always get current world version (4.1.0-beta.0)
- Make EventResult.event optional for legacy compatibility
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* Refactor spec version from semver strings to integers
Replace semver-based version compatibility with explicit integer spec versions:
- SPEC_VERSION_LEGACY (1): pre-event-sourcing runs
- SPEC_VERSION_CURRENT (2): event-sourced architecture
Use branded SpecVersion type to enforce importing from @workflow/world.
Remove semver dependency from world, world-local, and world-postgres.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* Store error as CBOR in postgres world for consistency with input/output
- Add error_cbor bytea columns to workflow_runs and workflow_steps tables
- Deprecate text error column, rename to errorJson with fallback parsing
- Remove JSON.stringify from error writes (run_failed, step_failed, step_retrying)
- Add parseErrorJson helper for backwards compatibility with legacy data
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* Use WorkflowRuntimeError and improve run entity handling in core runtime
- Replace generic Error with WorkflowRuntimeError for runtime assertions
- Add explicit check for run entity in run_created response
- Use run.runId instead of event.runId for consistency
- Use actual run status instead of hardcoded 'pending' in attributes
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* Add specVersion to Step, Hook, and Event entities
- Add specVersion field to Step, Hook, and Event interfaces in @workflow/world
- Add spec_version column to steps, hooks, events tables in postgres schema
- Set specVersion to SPEC_VERSION_CURRENT when creating entities in all worlds
- Update migration to include spec_version columns for all entity tables
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* Refactor world-local storage into modular files
Split storage.ts (1041 lines) into smaller, focused modules:
- storage/filters.ts: Data filtering helpers
- storage/helpers.ts: ULID and date utilities
- storage/hooks-storage.ts: Hook CRUD operations
- storage/legacy.ts: Legacy event handling
- storage/runs-storage.ts: Run get/list operations
- storage/steps-storage.ts: Step get/list operations
- storage/events-storage.ts: Event create/list operations
- storage/index.ts: Main composition
Also extracted test helpers to test-helpers.ts for reusability.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* Remove genversion and World.specVersion property
The World.specVersion string property was never actually read - only
the numeric SPEC_VERSION_CURRENT is used for backwards compatibility.
- Remove genversion dependency and generated version.ts from @workflow/world
- Remove specVersion property from World interface and all implementations
- Minor fix: correct error message to reference 'workflow' package
- Minor fix: correct error source priority in world-vercel events.ts
- Minor fix: update comment in runtime.ts
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* Remove genversion from world-local, world-postgres, and world-vercel
These packages no longer need genversion since we removed the
World.specVersion property in the previous commit.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* Remove version.ts from .gitignore files
No longer needed after removing genversion.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* Add legacy/backwards compatibility tests
Tests for world-local and world-postgres covering:
- Legacy runs (specVersion < 2 or null/undefined)
- run_cancelled handling (updates run, no event stored)
- wait_completed handling (stores event only)
- Rejection of unsupported events
- Hook cleanup on cancellation
- Future runs (specVersion > current)
- Rejection with RunNotSupportedError
- Current version runs (normal processing)
- Legacy error parsing (errorJson field parsing)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* Add hook_received support for legacy runs
When resumeHook() is called on a legacy run (specVersion < 2), the
hook_received event was previously rejected. This adds support for
storing hook_received events on legacy runs without entity mutation,
matching the behavior of wait_completed.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* Fix missing genversion in world-vercel
Signed-off-by: Peter Wielander <mittgfu@gmail.com>
* Remove deprecated workflow_completed, workflow_failed, and workflow_started events
Replace with run_completed, run_failed, and run_started equivalents.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* Add specVersion to EventWithRefsSchema in world-vercel
The manually-created EventWithRefsSchema was missing the specVersion field,
which caused specVersion to be stripped when using lazy (refs) mode for events.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* Wire specVersion from client through world backends
- Core runtime now sends specVersion in run_created eventData
- world-local accepts specVersion from eventData (defaults to current)
- world-postgres accepts specVersion from eventData (defaults to current)
This matches workflow-server behavior where v2 endpoints accept
specVersion from the client, while v1 endpoints default to legacy.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* Move specVersion to event object level, propagate to entities
- Add specVersion to BaseEventSchema (event level, not eventData)
- Remove specVersion from RunCreatedEventSchema.eventData
- Core runtime sends specVersion on event object
- world-local reads specVersion from event, propagates to run/step/hook entities
- world-postgres reads specVersion from event, propagates to run/step/hook entities
This ensures specVersion flows from client through event to all created entities.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* Update specVersion to be optional in types for backwards compatibility
- specVersion is optional in all entity schemas (runs, steps, hooks, events)
for backwards compatibility with legacy data in storage
- Runtime always sends specVersion on event requests
- world-local and world-postgres provide fallback to SPEC_VERSION_CURRENT
- Test helpers include specVersion in all event creation calls
- EventWithRefsSchema in world-vercel defaults specVersion to 1 for legacy
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* Fix world-vercel queue tests missing VERCEL_DEPLOYMENT_ID setup
Two tests were calling queue.queue() without setting up
VERCEL_DEPLOYMENT_ID, causing them to fail with "No deploymentId
provided" error before reaching the code they were testing.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* Add specVersion to all event creation calls in core package
The server's CreateEventSchemaV2 requires specVersion on all events,
but only run_created was sending it. This caused 400 Bad Request errors
for all other event types (run_started, run_completed, run_failed,
run_cancelled, step_created, step_started, step_completed, step_failed,
step_retrying, hook_created, hook_received, wait_created, wait_completed).
Now all event creation calls include specVersion: SPEC_VERSION_CURRENT.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
---------
Signed-off-by: Peter Wielander <mittgfu@gmail.com>
Co-authored-by: Vercel <vercel[bot]@users.noreply.github.com>
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
Co-authored-by: Peter Wielander <mittgfu@gmail.com>
* Proper stacktrace propogation in world
Proper stacktrace propogation in world
* Merge Reconciliation
* Standardize the error type in the world spec
* Deduplicate vercel world utils
* fix undefined type issue
---------
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Changed `RetryableError` to use milliseconds instead of seconds for numeric `retryAfter` values.
### What changed?
- Modified the `RetryableError` class to interpret numeric `retryAfter` values as milliseconds instead of seconds
- Updated documentation and examples to reflect this change
- Added dependency on `@workflow/utils` package to use the `parseDurationToDate` function
- Updated examples in documentation to show proper usage with:
- Duration strings (e.g., "5m", "30s")
- Millisecond values (e.g., 5000 for 5 seconds)
- Date objects
### How to test?
1. Create a workflow that uses `RetryableError` with different `retryAfter` formats:
```typescript
// With milliseconds
throw new RetryableError("Test retry", { retryAfter: 5000 });
// With duration string
throw new RetryableError("Test retry", { retryAfter: "5m" });
// With Date object
throw new RetryableError("Test retry", { retryAfter: new Date(Date.now() + 10000) });
```
2. Verify that the retry behavior correctly respects the specified durations
### Why make this change?
Using milliseconds as the unit for numeric time values is more consistent with JavaScript conventions (like `setTimeout` and other timing functions). This change makes the API more intuitive for JavaScript developers and aligns with standard practices in the ecosystem.
Signed-off-by: Nathan Rajlich <n@n8.io>
Pranay Prakash