Throwaway diagnostic on top of debug/combined-2171-2177. When a wait_completed
is validated with hasCreatedEvent=false, force a run_failed carrying
correlationId, the consumer's subscribe-time eventIndex, the completed-at
eventIndex, whether this consumer self-handled its own wait_created (and at
what index). Persists to S3 so we can see why the wait_created bypassed the
reused-sleep consumer. Do not merge.
A reused/duration sleep races a `wait_completed` replay against a
non-deterministic, wall-clock-derived expected value, producing a false
CorruptedEventLogError on a perfectly consistent event log.
`sleep(<ms|string>)` computes its resumeAt as `Date.now() + duration` (see
parseDurationToDate). The original run records that absolute timestamp into
both wait_created and wait_completed. During replay the VM clock advances to
each event's createdAt, so a freshly-created sleep recomputes a *different*
absolute resumeAt. Normally harmless: the wait_created consumer overwrites the
queue item's resumeAt with the recorded (authoritative) value before
wait_completed is validated.
The bug: when a wait_completed is consumed by a sleep consumer that never
applied a wait_created (hasCreatedEvent=false), the queue item still holds the
freshly-recomputed value, and the resumeAt comparison fails — even though the
event log is internally consistent and the recorded resumeAt is the source of
truth. Captured in production stress runs: hasCreatedEvent=false with ~18-42s
deltas between the recomputed and recorded resumeAt.
Fix: only validate resumeAt when an authoritative recorded value is available
(hasCreatedEvent=true). When it is not, the correlationId match already
establishes the wait's identity, so skip the check rather than fail. Extracted
the validation into `detectResumeAtMismatch`, which also lowers the consumer
callback's cognitive-complexity warning (33 -> 21).
Adds a regression test that advances the replay clock (via updateTimestamp)
and asserts a consistent wait_completed with hasCreatedEvent=false no longer
raises CorruptedEventLogError. Pre-existing stable bug; independent of the
hook-vs-sleep race fix (#2171) and of the server-side work.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
A buffered hook payload (a `hook_received` consumed before the workflow
awaited the hook) was delivered to its consumer only at claim time
(`iterator.next()`/`await`), and a concurrent `wait_completed` resolved
synchronously with fewer microtask hops. When the workflow did
`Promise.race([hook, sleep])`, sleep could win a race the committed event
log says the hook won, surfacing as `CorruptedEventLogError` on replay
(observed in production: a `step_created` for one step consumed by a
different step's consumer).
Fix, in three timing-independent parts:
1. Resolve a buffered hook payload through a `promiseQueue` slot chained
at its log position (not at the later claim site), so resolution order
stays anchored to the event log regardless of hydration/decryption
time.
2. Register a per-delivery ordering barrier keyed by the source
`hook_received` eventId (`ctx.pendingHookDeliveries`). A later-in-log
entity (sleep's `wait_completed`) defers behind any earlier-in-log
in-flight hook delivery via `awaitEarlierHookDeliveries`.
3. Release the barrier on a MACROTASK (`setTimeout(0)`) after the payload
is claimed, so the consumer's branch decision — however many await
hops deep through the async hook iterator — always commits before the
deferring entity proceeds. This reuses the macrotask-boundary technique
`scheduleWhenIdle` already relies on and is fully hop-count- and
decryption-time independent (no microtask-hop heuristic).
`awaitEarlierHookDeliveries` bounds its wait with a one-macrotask
fallback so an unclaimed payload cannot deadlock a deferring entity.
Adds characterization tests covering hook-vs-sleep (the repro),
hook-vs-step, and a two-hook slow-decrypt ordering case. Pre-existing
runtime bug on `stable`, independent of the OCC/fence work.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* Add server-backed exact ID search to the Events tab.
Replace client-side substring filtering with API lookups for full correlation and event IDs so searches work beyond the first loaded page.
* Fix exact ID search dimming and support wrun_ correlation IDs.
Disable group dimming for server search results and accept run IDs in the exact ID parser so run-level correlation search works.
* Fix dimmed row when searching by event ID for run-level events.
Map selectedGroupKey to __run__ for run-level search results so the matched row is treated as related instead of dimmed.
* Remove run ID search from Events tab exact ID lookup.
Workflow-server only accepts step, wait, and hook correlation IDs — not wrun_. Update the search placeholder and validation toast accordingly.
* Harden exact ID search UX and correlation fetch limits.
Normalize lowercase ULIDs, scope Enter toasts to ID-like input, abort stale searches, disable search when unavailable, expand parser tests, and cap correlation pagination in workflow web.
* Fix search clear race and surface truncated correlation results.
Guard successful exact-ID search against aborted requests, invalidate in-flight work when the input clears, and return truncation metadata from correlation pagination.
* Differentiate exact ID search errors from not-found results.
Return a discriminated union from onExactIdSearch and show search errors in the Events tab instead of mislabeling them as missing IDs.
* Apply suggestion from @VaguelySerious
---------
Signed-off-by: Peter Wielander <mittgfu@gmail.com>
Signed-off-by: Karthik Kalyan <105607645+karthikscale3@users.noreply.github.com>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Peter Wielander <mittgfu@gmail.com>
* [world-vercel] Add /run-id sub-export with tagged ULID encode/decode
Encodes a tag bit, 5-bit version, and 6-bit Vercel region ID into a
ULID-shaped string used for workflow run IDs. Tagged values remain
valid 26-char Crockford-Base32 ULIDs so they still sort and round-trip
through any system that accepts ULIDs.
* [world-vercel] Add string-value assertions to run-id tests
Add exact-string expectations for encoded outputs at known inputs,
covering the default region/version pair, numeric region IDs, version
overrides, boundary values (all-zero, all-max), the dirty-input
overwrite case, and the lexicographic-order checks. Also adds an
explicit byte-array expectation for the canonical ULID-spec example
string and an additional first-char-range coverage test for isTagged.
* [world-vercel] Remove internal-repo reference from regions doc comment
* [world-vercel] Address PR review feedback on run-id sub-export
- isTaggedString now fully validates the input as a 26-char Crockford
Base32 ULID (delegating to ulidToBytes) instead of only inspecting
the first character. This fixes false positives on inputs like
'4UUUU...' that have a valid tag-bit position but invalid chars
later in the string.
- isTagged() now accepts `unknown` to match its documented behavior
of safely rejecting non-string inputs without requiring callers to
cast.
- Introduce `RegionKey` for the full set of keys including 'unknown',
and narrow `RegionCode` to `Exclude<RegionKey, 'unknown'>` so the
return type of `lookupRegion` and the `DecodedRunId.region` field
accurately reflect that 'unknown' is never produced. Updates
`encode` to reject 'unknown' as a region code string at runtime
(callers wanting the unknown sentinel should pass numeric 0).
* [world-vercel] Move tagged-ULID metadata to the top of randomness
Address review feedback on #1978:
1. **Metadata at top of randomness, not bottom.** Place `regionId` (6
bits) in the high bits of byte[6] and `version` (5 bits) straddling
bytes 6 and 7, leaving the bottom 69 bits of randomness untouched by
`encode`. This means a `monotonicFactory()`-style ULID generator's
intra-millisecond bottom-bit increments survive encoding intact, so
consecutive `encode(ulid(), region, { version })` calls with the
same metadata produce strictly increasing strings. Previously the
metadata sat in the bottom 11 bits — exactly the bits the monotonic
factory uses — causing same-ms collisions/inversions.
2. **DecodedRunId is now a discriminated union.** When `tagged: false`,
the `regionId`, `version`, and `region` fields are typed as
`null` instead of being populated with garbage bits from arbitrary
ULIDs. This forces callers to discriminate on `tagged` before
reading metadata.
3. **regionIdFor: keep runtime backstop, mark as ignored for coverage.**
The unreachable-in-TS branch stays as a defensive runtime check for
callers crossing a JS/TS boundary; an istanbul/c8 ignore comment
keeps coverage tools quiet.
Doc strings and tests updated accordingly. The new layout adds a test
verifying that a sequence of incrementing-bottom-bit ULIDs (simulating
`monotonicFactory()`) round-trips through `encode` as a strictly
increasing sequence.
108/108 world-vercel tests pass; typecheck clean.
Signed-off-by: Nathan Rajlich <n@n8.io>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
The World service omits input fields from run/step snapshot responses when
payloads are externalized as RemoteRef blobs. Mirrors existing
WorkflowRunWithoutData / StepWithoutData types and extends #1939's treatment
of output/error/completedAt to input.
Fixes WorkflowWorldError schema validation failures on every event
acknowledgement after the 2026-05-12 service-side RemoteRef rollout.
Fixes#1977.
Signed-off-by: Nathan Rajlich <n@n8.io>
Signed-off-by: adamiBs <bsadambs@gmail.com>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Nathan Rajlich <n@n8.io>
Translates the diagrammed pattern into 5 prefix-replay tests that run the
same workflow against progressively longer prefixes of a 5-event log
(hook_created, wait_created, hook_received A, wait_completed,
hook_received B). Each test asserts the consumer takes the same
deterministic path: suspending at the right intermediate point with the
right invocationsQueue state, or completing with race winners
[hookA, sleep]. The full-log test verifies that the trailing
hook_received B is consumed by the dangling race-2 hook awaiter without
producing an unconsumed-event error. Runs in both sync and async
deserialization modes via the existing defineTests harness.
Signed-off-by: Nathan Rajlich <n@n8.io>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
* fix(world-local): prevent path traversal via request-supplied IDs (#1829)
* fix(world-local): prevent path traversal via request-supplied IDs
Request-supplied identifiers (runId, eventId, stepId, hookId, correlationId,
stream names, and tags) flowed directly into path.join() calls, allowing a
client to send values like '../../../package' and cause the backend to read
or write files outside the workflow data directory.
Add a centralized validator (assertSafeEntityId) that rejects IDs which are
empty, start with '.', or contain path separators or NUL bytes. Apply it at
each storage-layer entry point that composes IDs into filesystem paths:
fs.taggedPath / readJSONWithFallback / paginatedFileSystemQuery, the runs /
steps / events / hooks storage methods, and the streamer.
* address review feedback
- UnsafeEntityIdError now extends WorkflowWorldError for consistency with
other storage-layer errors and the platform error-to-HTTP mapping.
- Add resolveWithinBase(basedir, ...segments) containment helper and
apply it at every taggedPath / readJSONWithFallback / .locks path
construction site in events-storage and legacy, so a forgotten
assertSafeEntityId at a future call site can't silently regress.
- Truncate attacker-controlled values in the error message.
- Drop unused assertSafeEntityIds helper and the unreachable typeof
check under the TS signature.
- Fix docstrings on assertSafeEntityId / taggedPath JSDoc example /
filePrefix validation comment to match what the code actually does.
- handleLegacyEvent now re-asserts runId locally so the invariant is
documented at the call site instead of implicitly inherited from
events.create.
---------
Co-authored-by: JJ Kasper <jj@jjsweb.site>
* fix(world-local): tighten ID validation and add streamer regression tests
Addresses code review feedback on the path-traversal backport:
- Reject dots inside entity IDs in `assertSafeEntityId`. Internal IDs
(ULIDs, step_N, etc.) never contain dots, but `stripTag()` /
`getObjectCreatedAt()` strip a trailing `.[tag]` suffix from filenames,
so a request-supplied runId like `wrun_123.foo` would be silently
mangled during listing/pagination.
- Reject empty `correlationId` on events that include one. The event
schemas only require `z.string()`, so without this check a
step_created / hook_created / wait_created request with
`correlationId: ''` would silently be written under a malformed
composite key like `${runId}-`.
- Add streamer regression tests covering writeToStream, closeStream,
listStreamsByRunId, and getStreamChunks (the v4-shape surface that
this backport touches independently of main).
---------
Co-authored-by: JJ Kasper <jj@jjsweb.site>
Turborepo replays nextjs-turbopack:build from cache without restoring the
Vercel diagnostics manifest (.vercel/output/diagnostics/workflows-manifest.json),
which causes the Vercel deployment to fail post-build. Add .vercel/output/**
to the workbench's Turbo outputs so it is persisted and replayed. Applies to
both nextjs-turbopack and nextjs-webpack (whose turbo.json is a symlink).
Signed-off-by: Nathan Rajlich <n@n8.io>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Per-deployment SDK tarballs are now built by the tarballs/ app, not by
docs/. Remove the outdated reference to docs/scripts/pack.ts (which no
longer exists) and point readers at tarballs/README.md.
The fixture was introduced by the backport of #1935 (#1945) with the
pre-namespacing step ID "step//./input//step" in the __internal_workflows
manifest comment. The subsequent backport of #1944 (#1946) shipped the
namespacing fix in lib.rs but did not update this fixture, since on main
the two PRs landed in the reverse order and #1944's diff already covered
the fixture there.
Update the expected output to match what the plugin now emits:
"step//./input//outer/step" — same value already present in main.
* [ci] Attribute backport changelog entries to original PR author
`@changesets/changelog-github` resolves a changeset commit to its
associated PR via the GitHub GraphQL `associatedPullRequests` field. For
commits landed on `stable` via our backport workflow, that resolves to
the backport PR authored by `github-actions[bot]` (since the backport
workflow uses `createCommitOnBranch` to produce signed commits), so the
generated changelog ends up with "Thanks @github-actions!" instead of
the original contributor.
This adds a small `.changeset/changelog.mjs` wrapper around
`@changesets/changelog-github` that detects backport PRs by matching the
title (`Backport #N: ...`) or body (`Automated backport of #N to
` + '`stable`' + `...`) produced by `.github/workflows/backport.yml`, resolves the
original PR number, and injects `pr:`/`commit:` directives into the
changeset summary before delegating to the upstream generator. The
result is that the rendered changelog entry attributes the change to the
original PR and author, while the commit link still points at the
backport commit on the release branch.
* Address review feedback
- Use `Bearer` instead of `Token` for the GitHub GraphQL auth header
for consistency with the rest of the repo (Copilot review on PR #2091).
- Add a defensive `formatError` helper so `console.warn` in the catch
blocks doesn't itself throw when a non-Error value is thrown (Copilot
review on PR #2091).
Signed-off-by: Nathan Rajlich <n@n8.io>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
* remove toast on succesful decrypt
* show hook name
* fix hook on detail view
Signed-off-by: Mitul Shah <mitulxshah@gmail.com>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
* [swc-plugin] Preserve imports referenced by hoisted nested steps
Dead-code elimination ran before nested step functions were hoisted out of workflow bodies, so imports referenced only by hoisted step bodies were incorrectly stripped from the step bundle, causing a ReferenceError at runtime. Move DCE to run after hoisting in visit_mut_program.
* [swc-plugin] Namespace nested step IDs under non-exported workflow functions
Anonymous steps nested inside callback properties of a non-exported workflow function were registered with an unnamespaced step ID in step mode while the workflow-mode proxy looked them up under the workflow function name, causing a runtime 'step not found' failure. Set current_workflow_function_name in visit_mut_fn_decl for non-exported workflow functions to match the behavior in visit_mut_export_decl. Also clarify the fixture comment to distinguish step-mode and workflow-mode behavior per reviewer feedback.
* [swc-plugin] Namespace nested step IDs across all workflow declaration shapes
Extends the previous fix to cover all three non-exported workflow declaration forms (async function decl, const arrow, const fn-expr) by visiting the workflow body with workflow context before replacing it, and corrects the __internal_workflows manifest comment to report the same prefixed step IDs that are registered at runtime and looked up by the workflow-mode WORKFLOW_USE_STEP proxy. Adds a dedicated regression fixture covering all three shapes.
Signed-off-by: Nathan Rajlich <n@n8.io>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Nathan Rajlich <n@n8.io>
* [swc-plugin] Capture lexical `this` for nested arrow step functions
When a nested arrow `"use step"` references the enclosing function/method's
`this`, plumb that `this` through the workflow runtime so the step body
sees the correct receiver.
- Workflow mode wraps the step proxy with `.bind(this)`, so invoking the
proxy captures the caller's `this` as `thisVal` on the queue item.
- Step mode hoists the body as a regular `function` (not an arrow) so the
runtime's `stepFn.apply(thisVal, args)` rebinds `this` inside the
hoisted body.
Detection only fires for arrows, since arrows inherit `this` lexically.
Nested non-arrow functions/methods/getters/setters introduce their own
`this`, so the detector stops at those boundaries.
The runtime already supported `thisVal` for instance-method steps; this
PR is purely a compiler change to feed the existing pipeline.
Caveat: capture works at runtime only when the captured value is
serializable across the workflow->step boundary (i.e. the enclosing
class implements `WORKFLOW_SERIALIZE`/`WORKFLOW_DESERIALIZE`).
Refs vercel/workflow#1865
* Address PR review: preserve step proxy metadata + tighter `this` detection
- core: Override `.bind` on step proxies so the bound function retains
`stepId` and `__closureVarsFn`. Without this, a bound proxy that flows
through workflow serialization (e.g. as a step argument) would be
treated as a non-serializable plain function by `getStepFunctionReducer`.
- swc-plugin: Detector now also walks `arrow.params` so `this` references
in default values / destructuring initializers (e.g. `(x = this.foo) =>
...`) trigger the `.bind(this)` path.
- swc-plugin: Class bodies inside the arrow body are now treated as
`this`-binding boundaries — `this` inside class field initializers,
methods, etc. is bound to the class instance, not the outer arrow. The
detector still walks `extends` clauses and computed property keys
because those are evaluated in the surrounding scope.
- spec.md: Sharpen the note about `this` in step bodies — it's
syntactically allowed but only meaningful for instance-method steps and
lexical-`this` arrow steps; other shapes compile but `this` will be
whatever the caller of the step proxy passes.
- Add `lexical-this-detector-edge-cases` fixture covering both the
default-param positive case and the inner-class false-positive guard.
- Strengthen the runtime test to assert `stepId` / `__closureVarsFn`
survive `.bind(...)`.
* [swc-plugin] Fix `arguments` closure-var capture; drop dead `this`/`arguments` checks
- Add `arguments` to `is_global_identifier` so it's not captured as a
closure variable. Previously a nested `function`-form step like
function step() { 'use step'; return arguments[0]; }
was hoisted with `const { arguments } = ...` (a strict-mode syntax
error) and the body's `arguments[0]` resolved against the destructured
binding instead of the function's intrinsic `arguments` object.
- Remove dead `ForbiddenExpression` checks for `this` and `arguments` in
`visit_mut_this_expr` / `visit_mut_ident`. The `'use step'` /
`'use workflow'` directives are stripped during the module-level
traversal before children are visited, so `in_step_function` /
`in_workflow_function` are never observed as true here in practice.
The existing `step-with-this-arguments-super` fixture explicitly
documents that all three identifiers are allowed in step bodies.
- Tighten the spec note about `arguments` accordingly: it works in
`function`-form steps (reflecting positional args) but is not captured
for arrow-form steps; use `...args` for that case.
- Add `nested-step-arguments` fixture pinning down the new behavior.
Signed-off-by: Nathan Rajlich <n@n8.io>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Nathan Rajlich <n@n8.io>
* Update workflow-trace-view.tsx
* Update trace viewer layout to be in a row
---------
Signed-off-by: Mitul Shah <mitulxshah@gmail.com>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Adds a unique partial index on workflow_events(run_id, correlation_id, type)
filtered to step_created/hook_created/wait_created, and translates the
resulting unique-violation (pg code 23505, surfaced via DrizzleQueryError.cause)
into EntityConflictError. The steps table already deduped via
onConflictDoNothing, but the event row still inserted, leaving duplicate
events in the log. Now both rows are kept consistent and the runtime's
existing dedup catch path handles concurrent writers cleanly.
(cherry picked from commit 7c45e9e213)
Co-authored-by: Nathan Rajlich <n@n8.io>
Sets pnpm's `minimumReleaseAge` to 2 days (company-wide standard) and
excludes internal scopes from the gate.
Signed-off-by: Karthik Kalyan <105607645+karthikscale3@users.noreply.github.com>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(world-postgres): bootstrap graphile-worker schema in setup CLI
`workflow-postgres-setup` now installs the `graphile_worker` schema in
addition to the drizzle migrations so that by the time any consumer
calls `world.start()`, both schemas already exist. This eliminates the
inter-process race on graphile-worker's `installSchema` where
concurrent `CREATE SCHEMA IF NOT EXISTS` calls could both pass the
MVCC-snapshotted existence check and one would fail with
`duplicate key value violates unique constraint "pg_namespace_nspname_index"`.
Reproduced locally against a fresh postgres:18-alpine with 8 parallel
`makeWorkerUtils().migrate()` calls — 7/8 fail without the pre-bootstrap,
0/8 fail after running `workflow-postgres-setup` first.
* Apply suggestions from code review
---------
Signed-off-by: Peter Wielander <mittgfu@gmail.com>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-authored-by: Nathan Rajlich <n@n8.io>
No need to version this app, it doesn't get published to npm.
Signed-off-by: Nathan Rajlich <n@n8.io>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
* Fix stable CI test harness failures
* Fix dev e2e cleanup races
* Restore non-Next dev cleanup
* Keep Next dev temp workflow files intact
* Keep dev test placeholders on disk
* Speed up workflow port detection
* Probe workflow health with POST
* Support HEAD workflow health checks
* Stabilize local CI health checks
* Retry flaky Vercel agent e2e
* Wrap generated framework route exports
* Relax remote addTen e2e timeout
* Materialize manual webhook responses
* Give remote CLI inspect more time
* Stabilize remote sleep and hook e2e checks
* Wait for step return streams before completion
* Stabilize hook and stream e2e waits
* Bound queue health check timeouts
* Address stable CI review feedback
* Fix route export replacement with embedded source maps
The AI SDK renamed `Experimental_Agent` to `ToolLoopAgent`. Update the
"Building Durable AI Agents" page's API route snippet (v4 and v5) so it
matches the current AI SDK API.
Signed-off-by: Karthik Kalyan <105607645+karthikscale3@users.noreply.github.com>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Nathan Rajlich