* fix: separate infrastructure vs user code error handling in runtime and step handler
Transient network errors (ECONNRESET, etc.) during infrastructure calls
(event listing, event creation) were caught by a shared try/catch that
also handles user code errors, incorrectly marking runs as run_failed
or steps as step_failed instead of letting the queue redeliver.
- runtime.ts: Move infrastructure calls outside the user-code try/catch
so errors propagate to the queue handler for automatic retry
- step-handler.ts: Same structural separation — only stepFn.apply() is
wrapped in the try/catch that produces step_failed/step_retrying
- helpers.ts: Add isTransientNetworkError() and update withServerErrorRetry
to retry network errors in addition to 5xx responses
- helpers.test.ts: Add tests for network error detection and retry
* add changeset
* remove withServerErrorRetry and isTransientNetworkError
Redundant with undici RetryAgent which already handles 5xx retries
and network error retries at the HTTP dispatcher level.
* address review feedback: move getEncryptionKeyForRun out of user-code try/catch, re-add 5xx/410 safety net in step handler, relax e2e test assertion
* remove serverError5xxRetryWorkflow e2e test
This test validated withServerErrorRetry's in-process retry behavior,
which was removed. Queue-level retry with process-scoped fault injection
is unreliable across serverless instances and too slow for e2e timeouts.
* remove serverError5xxRetryWorkflow and fault injection helpers from e2e workflows
* remove inline comment about deleted test
spawn({timeout}) sends SIGTERM by default, but the CLI process ignores it
(likely due to undici keeping active connections alive). The process then
runs for minutes until vitest or GitHub Actions eventually kills it.
Using killSignal: 'SIGKILL' ensures the process is forcefully terminated
when the 20s timeout fires, since SIGKILL cannot be caught or ignored.
Set setupGitUser: false on changesets/action to prevent it from
overwriting our git config with the hardcoded github-actions[bot]
identity. The git identity is now configured in a prior step using
the app-slug output from actions/create-github-app-token.
Set git user.name and user.email so that the 'Version Packages' commit
created by changesets/action is attributed to the app's bot account
instead of the default github-actions[bot].
The health command's `endpoint` flag and the shared `env` flag both
declared `char: 'e'`, causing ambiguity. Remove the short flag from
`endpoint` so `-e` unambiguously maps to `--env`.
The e2e tests spawn a CLI subprocess for every inspect/cancel/health call.
Each subprocess performs an npm registry version check on startup, which
can hang under load and exceed the 20s spawn timeout, causing SIGTERM.
Add WORKFLOW_NO_UPDATE_CHECK=1 env var support to skip the check, and
set it in the e2e test harness.
Replace GH_TOKEN_PULL_REQUESTS (shared org PAT) with a dedicated GitHub
App token to avoid rate limiting from noisy neighbors. The app token is
generated per-run via actions/create-github-app-token, providing its own
rate limit bucket and short-lived credentials.
The warning about WORKFLOW_VERCEL_* env vars having no effect at runtime
was incorrectly shown when the Workflow CLI or web observability app was
running. These tools intentionally set those env vars. Gate the warning
on VERCEL=1 so it only fires inside Vercel serverless functions where the
env vars are truly unnecessary.
* Fix connector line showing for run events
* Improve row toggle
* improve error reporting
* add sorting for events
* add changeset
* update search to be best match
* feat: add `exists` getter to `Run` class
Allow checking if a workflow run exists without catching errors.
Returns `true` if the run is found, `false` for WorkflowRunNotFoundError,
and re-throws any other errors.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* address PR feedback: use resolveData: 'none' in exists getter, fix docs wording
Pass { resolveData: 'none' } to world.runs.get() in the exists getter
to skip fetching input/output payloads. Update docs to say "without
throwing when the run is not found" instead of "without throwing an error".
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* Support client-side tools in DurableAgent
Tools without an `execute` function now pause the agent loop and return
`clientToolCalls` in the result, enabling human-in-the-loop and
client-side tool execution patterns. Fixes#847.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* Match AI SDK result shape for client-side tools
Use `toolCalls` and `toolResults` on DurableAgentStreamResult to match
the AI SDK's GenerateTextResult convention. Consumers find unresolved
client-side tool calls by diffing toolCalls vs toolResults — the same
pattern used with useChat/addToolOutput.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* Address PR review feedback
- Use iterMessages (not options.messages) so returned messages include
the assistant tool-call message, enabling callers to resume
- Add resolved server/provider tool results to returned messages as a
tool role message so callers don't re-execute when resuming
- Treat missing tools (!tool) as errors (will throw in executeTool),
only tools that exist but lack execute are client-side
- Track tool-output-available chunks in allUIChunks when
collectUIMessages is enabled so uiMessages is complete
- Add message assertions to tests
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* Clean up type casts in client-side tools path
Push directly to iterMessages (LanguageModelV2Prompt) instead of
double-casting through ModelMessage[]. Single cast at the return
point matches the existing pattern at the end of stream().
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* Reverse check in resolveProviderToolResult for early exit
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* feat: support `deploymentId: 'latest'` in `start()` to resolve most recent deployment
Add resolveLatestDeploymentId() to the World interface and implement it
in world-vercel, which calls the Vercel API to resolve the latest
deployment for the current environment. When start() receives
deploymentId: 'latest', it invokes this method and replaces the sentinel
with the resolved deployment ID before passing it to event creation,
queue dispatch, and encryption key derivation.
* fix: address PR review feedback
- Use encodeURIComponent() for deployment ID in API URL path
- Avoid mutating caller-provided opts; pass resolved deploymentId via
spread into getEncryptionKeyForRun context instead
- Add test asserting getEncryptionKeyForRun receives resolved ID
- Add test for OIDC token success path in resolve-latest-deployment
* first pass
* fix failing test
* Update changeset to fix SWC compiler issue
Fix bug where the SWC compiler bug prunes step-only imports in the client-mode transformation
Signed-off-by: Nathan Rajlich <n@n8.io>
* DCO Remediation Commit for nathancolosimo <nathancolosimo@gmail.com>
I, nathancolosimo <nathancolosimo@gmail.com>, hereby add my Signed-off-by to this commit: 150d175e7d
I, nathancolosimo <nathancolosimo@gmail.com>, hereby add my Signed-off-by to this commit: 429747fcc3
Signed-off-by: nathancolosimo <nathancolosimo@gmail.com>
---------
Signed-off-by: Nathan Rajlich <n@n8.io>
Signed-off-by: nathancolosimo <nathancolosimo@gmail.com>
Co-authored-by: Nathan Rajlich <n@n8.io>
* fix(web): move react-router deps to devDependencies
Add a custom entry.server.tsx (based on the default react-router template)
so that the @react-router/dev build plugin no longer requires
@react-router/node and isbot to be in the dependencies field of
package.json. This allows all react-router related packages to be
devDependencies since they are fully bundled at build time, leaving
express as the sole production dependency.
* Update packages/web/app/entry.server.tsx
Co-authored-by: Peter Wielander <mittgfu@gmail.com>
Signed-off-by: Nathan Rajlich <n@n8.io>
---------
Signed-off-by: Nathan Rajlich <n@n8.io>
Co-authored-by: Peter Wielander <mittgfu@gmail.com>
* fix(ai): use workspace:* for workflow peer dependency
The hardcoded version range in peerDependencies would fall out of sync
with the actual workflow package version, causing changeset warnings.
Using workspace:* lets pnpm resolve the correct version at publish time.
* Apply suggestion from @Copilot
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Signed-off-by: Nathan Rajlich <n@n8.io>
* add changeset for @workflow/ai peer dependency fix
---------
Signed-off-by: Nathan Rajlich <n@n8.io>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
The start() function resolved deploymentId from world.getDeploymentId() but
then passed the original opts (without the resolved value) to
getEncryptionKeyForRun. In external contexts (e2e tests, CLI) where
deploymentId is inferred from the environment rather than explicitly provided,
this caused getEncryptionKeyForRun to receive undefined deploymentId and
silently skip encryption for the initial workflow input.
Multiple layered issues caused encryption key 429 errors to produce
confusing, unrelated-looking test failures:
- awaitCommand() in e2e utils now rejects on non-zero exit codes
instead of silently resolving (the most critical fix)
- cliInspectJson/cliHealthJson throw on empty stdout instead of
falling back to '{}'
- maybeDecryptFields re-throws HTTP errors instead of silently
falling back to encrypted placeholders
- fetchRunKey includes response body and status text in errors
- Added 429 to CLI status text map
* fix(world-local): return HTTP 200 instead of 503 for queue timeout re-enqueue signals
The local queue used HTTP 503 (Service Unavailable) as an internal signal
for re-enqueueing after a timeout delay. This is misleading since 503 is
an error status, causing confusion when inspecting logs or network traffic.
Changed to return 200 with timeoutSeconds in the body, and updated the
consumer to detect the re-enqueue signal from the response body instead
of the status code.
* fix(world-local): handle timeoutSeconds: 0 and add queue re-enqueue tests
Address review feedback: timeoutSeconds === 0 should trigger an immediate
re-enqueue without delay, not be treated as a normal success. Also adds
test coverage for the handler response codes and re-enqueue behavior.
* test: enable encryption on nextjs-turbopack workbench
Set VERCEL_DEPLOYMENT_KEY to enable e2e encryption for the
nextjs-turbopack workbench project. This allows the e2e test
suite to exercise encrypted workflow runs alongside existing
unencrypted runs for backwards compatibility testing.
* fix: fetch encryption key from API in external contexts (e2e, CLI)
Use VERCEL=1 env var to distinguish serverless runtime (can use local
HKDF derivation) from external contexts like e2e test runners, CLI,
and web dashboard (must fetch derived key from Vercel API).
Previously, when VERCEL_DEPLOYMENT_ID matched but VERCEL_DEPLOYMENT_KEY
was absent (e.g., e2e runner has the deployment ID from CI but not the
key), getEncryptionKeyForRun returned undefined — silently skipping
decryption. Now external contexts always use the API fetch path.
* fix: add 429 retry with exponential backoff to fetchRunKey
The e2e runner calls fetchRunKey for every run, which can exceed the
API rate limit (30/min per team) when many tests run in parallel.
Add retry logic with exponential backoff + jitter (up to 5 retries)
and respect the Retry-After header when present.
* Fix: Missing NaN guard on `retry-after` header parsing causes `setTimeout(r, NaN)` to resolve immediately (0ms), triggering rapid-fire retries against a rate-limited API endpoint.
This commit fixes the issue reported at packages/world-vercel/src/encryption.ts:138
**Bug Analysis:**
In `packages/world-vercel/src/encryption.ts`, the `fetchRunKey` function handles HTTP 429 (rate limited) responses by reading the `retry-after` header and computing a delay. The original code was:
```js
const retryAfter = response.headers.get('retry-after');
const delay = retryAfter
? Number.parseInt(retryAfter, 10) * 1000
: BASE_DELAY_MS * 2 ** attempt + Math.random() * 500;
```
Per RFC 9110, the `Retry-After` header can contain either a number of seconds OR an HTTP date string (e.g., `"Wed, 21 Oct 2015 07:28:00 GMT"`). When the header contains a date string or any non-numeric value, `Number.parseInt(retryAfter, 10)` returns `NaN`. Since the header string is truthy (non-empty), the ternary takes the parseInt path rather than the exponential backoff fallback. `NaN * 1000` evaluates to `NaN`, and `setTimeout(r, NaN)` resolves immediately (~0ms, confirmed by testing). This causes rapid-fire retries (up to 5 iterations with zero delay) against a rate-limited API endpoint, wasting resources and potentially worsening the rate-limiting situation.
**Fix:**
Parse the integer first, then check if the result is NaN before using it. If NaN, fall back to exponential backoff with jitter. This matches the established pattern already used in `packages/world-vercel/src/utils.ts` (lines 309-315) where the same header is properly guarded with `!Number.isNaN(parsed)`.
```js
const retryAfterHeader = response.headers.get('retry-after');
const parsedRetryAfter = retryAfterHeader
? Number.parseInt(retryAfterHeader, 10)
: NaN;
const delay = !Number.isNaN(parsedRetryAfter)
? parsedRetryAfter * 1000
: BASE_DELAY_MS * 2 ** attempt + Math.random() * 500;
```
Co-authored-by: Vercel <vercel[bot]@users.noreply.github.com>
Co-authored-by: TooTallNate <n@n8.io>
* fix: add --decrypt to e2e CLI inspect calls
With encryption enabled on the nextjs-turbopack workbench, CLI inspect
output shows encrypted data as placeholders. The e2e tests check for
actual output values, so they need --decrypt to see the real data.
* fix: remove manual 429 retry, rely on undici RetryAgent; add changeset
The shared RetryAgent from getDispatcher() already handles 429/5xx
retries with exponential backoff and Retry-After header support.
The manual retry loop in fetchRunKey was redundant.
Also add changeset for the VERCEL=1 external context fix.
* refactor: use @ts-expect-error on dispatcher instead of as any cast
---------
Co-authored-by: Vercel <vercel[bot]@users.noreply.github.com>
* Stop reading WORKFLOW_VERCEL_* env vars at runtime to prevent unintended proxy routing
createWorld() in core no longer reads WORKFLOW_VERCEL_PROJECT, WORKFLOW_VERCEL_TEAM,
WORKFLOW_VERCEL_AUTH_TOKEN, WORKFLOW_VERCEL_ENV, or WORKFLOW_VERCEL_PROJECT_NAME from
process.env. These env vars are intended for CLI/observability tooling only, and
reading them at runtime caused all workflow traffic to route through the
api.vercel.com proxy when users mistakenly set them as Vercel project env vars.
The Vercel runtime already provides everything needed: OIDC tokens for auth,
VERCEL_PROJECT_ID for encryption context, and VERCEL_DEPLOYMENT_ID for world
selection. createWorld() now calls createVercelWorld() with no config.
A warning is emitted if the env vars are detected at runtime, telling users to
remove them.
The CLI and e2e tests are updated to call createVercelWorld() directly with an
explicit config object and inject via setWorld(), keeping the WORKFLOW_VERCEL_*
env vars scoped to tooling contexts only.
* Refactor inferVercelEnvVars to return config instead of relying on process.env
inferVercelEnvVars() now returns a VercelEnvVars object with the resolved
config values. setup.ts uses the returned object directly for
createVercelWorld() instead of re-reading from process.env via getEnvVars().
The writeEnvVars() calls inside inferVercelEnvVars are consolidated into a
single call at the end, retained only for the embedded web UI which reads
process.env as a fallback in its server actions. The scattered writeEnvVars
calls after each inference step are removed.
* Address review: consistent e2e gate and expanded warning
- Use WORKFLOW_VERCEL_ENV as the gate in both e2e.test.ts and
bench.bench.ts for consistency (was WORKFLOW_VERCEL_AUTH_TOKEN in
e2e.test.ts, WORKFLOW_VERCEL_ENV in bench.bench.ts)
- Expand the misconfiguration warning to also detect
WORKFLOW_VERCEL_AUTH_TOKEN and WORKFLOW_VERCEL_ENV, listing the
specific env vars that are set
* Add changeset
Signed-off-by: Nathan Rajlich <n@n8.io>
---------
Signed-off-by: Nathan Rajlich <n@n8.io>
* test: reproduce hook+sleep promiseQueue regression
Add failing tests that reproduce a regression from #1246 where the
sleep's WorkflowSuspension fires before all hook payloads are delivered
when a hook and sleep run concurrently.
Root cause: when the null event fires, the sleep queues a suspension
through promiseQueue. After the first hook payload resolves, subsequent
hook payload resolutions are queued AFTER the sleep suspension, causing
the workflow to terminate prematurely via Promise.race.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* test: expand tests to isolate bug to hook + pending entity pattern
Add control tests proving sequential steps are NOT affected by the
promiseQueue regression, isolating the bug to hooks specifically:
Failing (hook-based):
- hook + sleep: all 3 payloads → step invocation
- hook + sleep: 2 payloads → return
- hook + incomplete step: 2 payloads → return
Passing (step-based controls):
- sleep + sequential steps: both step events exist
- sleep + sequential steps: only 1st step completed
- incomplete step + sequential steps: all step events exist
- hook only (no concurrent entity): payloads + step
The bug is: any entity that queues a suspension through promiseQueue
at null-event time preempts hook payload delivery, because hooks
buffer payloads in payloadsQueue and only resolve them one-at-a-time
as the workflow code iterates. Steps are unaffected because each step
has its own events consumed before null fires.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix(core): move WorkflowSuspension calls back to setTimeout(0) from promiseQueue
The promiseQueue refactor (#1246) moved ALL promise resolutions through
the queue, including WorkflowSuspension calls. This caused premature
termination when hooks had buffered payloads and a concurrent entity
(sleep or incomplete step) was pending:
1. Null event fires → all subscribers run
2. Sleep/step queues WorkflowSuspension via promiseQueue.then()
3. Hook's next payload resolution is also queued via promiseQueue.then()
4. Suspension fires first → Promise.race terminates workflow
5. Hook payload never delivered → infinite retry loop
Fix: move WorkflowSuspension calls back to setTimeout(0) (macrotask).
Suspensions fire AFTER all microtask-based deliveries (promiseQueue
resolve/reject for step results and hook payloads) have completed.
Non-suspension resolve/reject calls remain on promiseQueue for
deterministic ordering of data delivery.
Affected paths:
- step.ts: null event handler
- sleep.ts: null event handler
- hook.ts: null event handler, eventLogEmpty suspension, dispose suspension
* fix(core): use pendingDeliveries counter + scheduleWhenIdle for suspensions
Replace all nested setTimeout/promiseQueue suspension patterns with a
clean idle-polling mechanism:
1. pendingDeliveries counter: incremented before async hydration (step
results, hook payloads), decremented in finally block after delivery
2. scheduleWhenIdle(ctx, fn): polls via setTimeout(0) → check counter →
if > 0, wait for promiseQueue.then() → repeat. Only fires fn when
pendingDeliveries reaches 0.
This correctly handles:
- Sync deserialization (no encryption): counter is 0, fires immediately
after first setTimeout(0)
- Async deserialization (with encryption): waits for decryption to
complete before firing
- Multi-round hook payload delivery: each createHookPromise() call
increments the counter, preventing premature suspension between
delivery rounds
Also adds payloadsQueue.length check to hook null handler — don't
trigger suspension if buffered payloads can satisfy pending awaits.
Tests now run in both sync and async modes (14 total = 7 scenarios x 2).
All 164 tests pass.
---------
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-authored-by: Nathan Rajlich <n@n8.io>
* feat: expose workflow and step names in metadata functions
Add `workflowName` to `WorkflowMetadata` and `stepName` to `StepMetadata`,
making them available via `getWorkflowMetadata()` and `getStepMetadata()`.
This is useful for adding context to structured logs, collating logs during
investigations, and bucketing errors by workflow/step name.
Closes#1103
* add changeset
* Apply suggestion from @TooTallNate
Signed-off-by: Nathan Rajlich <n@n8.io>
---------
Signed-off-by: Nathan Rajlich <n@n8.io>
Previously, immediate re-enqueues used `timeoutSeconds: 1` as an
arbitrary minimum delay. This changes the convention so that
`timeoutSeconds: 0` signals an immediate re-enqueue:
- world-local: fix truthiness check so 0 triggers a 503 response
- world-vercel: omit `delaySeconds` from queue options when 0
- core: use `timeoutSeconds: 0` for hook conflict re-enqueues
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* Fix docs typechecker to actually resolve workflow module types
The docs typechecker was silently failing to resolve `workflow` imports
because:
1. repoRoot was computed 4 levels up instead of 3
2. No path mappings for workspace packages
3. Missing TS lib files for disposable and DOM iterables
This meant all imported symbols became `any` and type errors in docs
code samples went undetected.
Fixes:
- Fix repoRoot: '../../../..' -> '../../..'
- Add path mappings for workflow, @workflow/core, @workflow/ai, etc.
- Add lib.dom.iterable.d.ts and lib.esnext.disposable.d.ts
- Fix regex that ate first line of some code blocks
- Auto-skip error demonstration code blocks
- Add respondWith to global Request for webhook doc patterns
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* Fix docs typecheck failures against updated base branch
- Skip incomplete code samples (with `...`/`/* ... */`) from type checking
- Fix sleep("1 month") → sleep("30d") (valid ms duration)
- Fix run.result() → run.returnValue (correct Run API)
- Add skip-typecheck for docs with intentionally simplified types
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* Fix doc code samples to typecheck instead of skipping them
- hook-conflict.mdx: use error.message.includes() instead of non-existent error.slug
- get-hook-by-token.mdx: cast hook.metadata to expected shape
- ai/index.mdx: fix tool inputSchema to match execute function params
- get-writable.mdx: use ModelMessage[], spread result.messages, use model placeholder
- Add myModel global placeholder for docs examples
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* Narrow TS2307 ignore to only non-mapped modules
Instead of blanket-ignoring all "Cannot find module" errors (TS2307),
only ignore them for modules not in our explicit paths mappings.
This ensures workspace package imports (workflow, @workflow/core, etc.)
are validated while still allowing relative imports and framework deps.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Vercel Release Bot