mirror of
https://github.com/vercel/workflow.git
synced 2026-09-14 19:59:43 +08:00
peter/windows-preload-timeout
37 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Alex Langenfeld
|
c1a5c74edb |
fix(streams): surface typed retention expiry errors (#3410)
## Summary & Motivation Adds `StreamExpiredError` to `@workflow/errors`, carrying the run, stream, and server-reported expiry timestamp from workflow-server's 410 `stream-expired` envelope. The reconnect loop rethrows it instead of retrying, since retention expiry is terminal and a retry budget would only convert it into a generic exhaustion error. ## Test Plan Unit tests added for the 410 decoding path and the reconnect rethrow; typechecks pass across the touched packages. |
||
|
Alex Langenfeld
|
79e4c04409 |
fix(core): re-route runs delivered to the wrong deployment (#2960)
## Summary & Motivation A queue callback that reaches a deployment other than the one its run is pinned to derives the per-run encryption key from the wrong master key, so the delivery fails before user code runs and the run dies as a blank "exceeded max retries". The delivery is re-enqueued explicitly addressed to the run's own deployment — strictly better-targeted than the send that misrouted — and the run is failed with the new `DEPLOYMENT_MISMATCH` error code only once `WORKFLOW_DEPLOYMENT_MISMATCH_MAX_RETRIES` (default 3) is spent. Gated on the new World capability `deploymentAffinity`, so worlds with synthetic or version-tagged deployment ids are unaffected. ## Test Plan Unit tests added for the guard and both runtime paths; local vitest and typechecks pass. |
||
|
Peter Wielander
|
1471f252fa | [core] Gate event creation on the loaded event count and restart replays in-process (#3145) | ||
|
Nathan Rajlich
|
32ac8e73fd |
Fix Biome lint violations and add Biome CI check (#3222)
* Fix Biome lint violations and add Biome CI check Biome was not configured to respect .gitignore, so ~92% of the 13,355 reported diagnostics came from gitignored build artifacts. Enable VCS integration (useIgnoreFile), apply safe auto-fixes across the repo, fix the remaining mechanical errors by hand, downgrade judgment-call a11y / dangerouslySetInnerHTML rules to warnings, and add a 'biome ci' job to the Lint workflow so violations block PRs going forward. * Use an empty changeset (no behavior change, no release needed) |
||
|
Nathan Colosimo
|
62d570ed4b | Remove retired v1 step route plumbing (#3061) | ||
|
Shalabh Chaturvedi
|
fe12b84729 |
Implement max_events per run limit (#2986)
Enforces the published per-run events limit, which was previously not enforced. The server supplies the limit on the run_started response (separate change); once a run's event log reaches it, the runtime throws MaxEventsExceededError at the top of the replay loop, and the existing terminal-error path records it as run_failed with a new MAX_EVENTS_EXCEEDED code — instead of letting a runaway workflow (e.g. an unbounded step loop) grow the event log without bound. Adds a new client side WORKFLOW_MAX_EVENTS_OVERRIDE env var which can override the server side provided value (lower only). |
||
|
Peter Wielander
|
eb8fdb9797 | Default WORKFLOW_PRECONDITION_GUARD on (#2946) | ||
|
Peter Wielander
|
a00d169470 | Add stateUpdatedAt precondition guard to event creation (#2266) | ||
|
Nathan Colosimo
|
692a6ac5dc |
Upgrade workspace to TypeScript 6 (#2700)
* Upgrade workspace to TypeScript 6 * Restore Nest baseUrl for SWC builds * Use empty changeset for TS6 upgrade * Remove TS6 changeset |
||
|
Pranay Prakash
|
2a3b11bcb4 |
Retry replay divergence before failing event logs (#2212)
(cherry picked from commit
|
||
|
Nathan Rajlich
|
8d0928b2a2 |
fix(core,errors): classify SDK encryption failures as RUNTIME_ERROR (#2145)
* fix(core,errors): classify SDK encryption failures as RUNTIME_ERROR SDK-level AES-GCM encrypt/decrypt failures are never the user's fault, but the run-failure classifier was tagging them as USER_ERROR because the native Web Crypto OperationError (most commonly raised by AESCipherJob.onDone on GCM auth-tag mismatch) does not match any RUNTIME_ERROR_CHECKS entry. Introduce a new RuntimeDecryptionError (subclass of WorkflowRuntimeError) that the encryption module throws when subtle.encrypt/subtle.decrypt fails, with the original DOMException as cause plus diagnostic context (operation, byteLength, printable/hex format prefix of the input header). classifyRunError now picks it up via RUNTIME_ERROR_CHECKS, so these failures surface as RUNTIME_ERROR with a proper named class for dashboards and triage. * Trim changeset description to one sentence * Trim historical-context comments * docs: add runtime-decryption-failed troubleshooting page (v4 + v5) * fix(core): round-trip RuntimeDecryptionError context, fix formatPrefix, propagate through serialization wrappers Addresses review feedback on #2145: - Add a RuntimeDecryptionError reducer/reviver (+ SerializableSpecial entry + globalThis registration) so its `context` (operation, byteLength, formatPrefix) survives the dehydrate/hydrate run-error round trip instead of being dropped by the generic Error reducer. - Stop capturing `formatPrefix` in the low-level encryption layer, which only sees the stripped AES payload (nonce bytes), not the outer `encr` marker. The serialization layer now attaches the real envelope prefix. - Rethrow RuntimeDecryptionError unchanged from the serialize/dehydrate catch blocks instead of reframing it as a SerializationError, so an encryption failure during dehydration stays a RUNTIME_ERROR rather than being misclassified as USER_ERROR. * fix(core): enrich stream decrypt errors with envelope prefix + fix lint - Mirror the catch/enrich/rethrow block from serialization/encryption.ts around the stream-path aesGcmDecrypt() call so auth-tag failures on encrypted stream frames also carry context.formatPrefix = 'encr' (addresses review feedback). Add a tampered-frame test. - Fix all auto-fixable Biome lint findings in the touched files (template literals, useless try/catch wrappers, optional chaining, non-null assertions). |
||
|
Pranay Prakash
|
1d3959eaa8 | Capture world contract failures as fatal (#2060) | ||
|
Pranay Prakash
|
ad71b58bba | Report corrupted event logs distinctly (#2046) | ||
|
Pranay Prakash
|
9d2a9261fd |
Expose conflicting run id on hook conflicts (#2012)
* Expose conflicting run id on hook conflicts * Mark hook conflict run id as future required * Address hook conflict docs review * Address hook conflict review comments * Fix hook conflict docs typecheck |
||
|
Peter Wielander
|
540a2efb99 | [errors] Replace chalk import in @workfow/errors with inline ANSI shim (#1915) | ||
|
Nathan Rajlich
|
5f22832675 |
Serialize run_failed/step_failed errors through serialization pipeline (#1851)
* Serialize run_failed/step_failed errors through serialization pipeline
Switch run_failed, step_failed, and step_retrying events to persist
the full thrown value via the workflow serialization pipeline (as
SerializedData / Uint8Array) instead of a lossy { message, stack, code }
StructuredError shape. Consumers hydrate via hydrateRunError /
hydrateStepError to reconstruct the original thrown value, preserving
Error subclass identity, cause chains, and custom properties.
- WorkflowRun.error and Step.error are now SerializedData
- WorkflowRun gains a top-level errorCode plaintext field
- WorkflowRunFailedError.cause is now the hydrated thrown value
- Adds world-postgres migration 0010_add_error_code.sql
- Legacy pre-pipeline errorJson records surface as undefined on read
* Update Next.js workbenches for new WorkflowRunFailedError.cause type
cause is now `unknown` (the hydrated thrown value) rather than
`Error & { code }`. Defensively extract Error-shaped fields when the
hydrated value is an Error, otherwise round-trip the raw value, and
expose the new `errorCode` classification field.
* Update docs for WorkflowRunFailedError.cause: unknown
The hydrated `cause` is now `unknown` (the original thrown value
through the serialization pipeline) and the error classification has
moved to the top-level `errorCode` property. Update the two affected
docs pages and the `TSDoc` interface to reflect the new shape, and
narrow `cause` with `instanceof Error` before accessing fields.
* Expand test coverage for the run/step error serialization pipeline
Unit tests:
- 19 new dehydrate/hydrate{Step,Run}Error round-trip tests covering
FatalError, plain Error, built-in Error subclasses, non-Error thrown
values (string, plain object), cause chains, encryption round-trip,
the binary format prefix contract, and the unserializable / unknown-
format error paths.
- 5 new tests for Run.returnValue when the run is failed: hydrated
FatalError + cause as cause, plain Error preservation, non-Error
thrown values surfaced verbatim, cross-class cause chains, and the
hydration-failure fallback that still surfaces errorCode.
E2E tests (new, in 99_e2e.ts + e2e.test.ts):
- Step throw → workflow catch round-trips a FatalError with a TypeError
cause chain, asserting class identity, fatal marker, and cause name +
message all survive the step_failed event pipeline.
- Workflow throw → run_failed reaches status with the new
top-level errorCode metadata exposed (cause-shape coverage lives at
the unit level, since the SWC plugin's class registration is not
invoked in the plain-Node e2e runner).
- Workflow throw of a non-Error value round-trips that value verbatim
as WorkflowRunFailedError.cause.
Adjustments to existing assertions:
- error.cause is now ; tests narrow with
and use the new top-level field instead of .
- step.error / run.error from CLI --withData are now hydrated payloads:
unregistered class instances surface as Instance refs whose
carries the original message + stack.
Observability hydration:
- hydrateStepIO / hydrateWorkflowIO in serialization-format.ts now
hydrate the field via hydrateData, so the CLI and web UI
continue to surface readable run/step error messages and stacks.
* Tighten error serialization changeset description
* Trim error serialization changeset to a single sentence
* Resolve FatalError/RetryableError revivers via cross-realm registry
When a workflow runs in a Node `vm` context, its bundled
`@workflow/errors` is a different module instance than the host's
import (separate prototype chains, separate class identity). Calling
`new FatalError(...)` from the host-side reviver produces a
host-realm instance that fails `err instanceof FatalError` checks
in the workflow code — even when the serialized payload was correctly
tagged via the dedicated `FatalError` reducer.
Surfaced by the local-prod e2e "step throw round-trips FatalError"
test on Next.js Turbopack: each route gets its own bundled chunk, so
the flow handler's `@workflow/errors` and the workflow VM bundle's
`@workflow/errors` are two distinct copies of the same module.
Fix:
- Each bundled copy of `@workflow/errors` self-registers its
`FatalError` and `RetryableError` classes on `globalThis` via
`Symbol.for("@workflow/errors//FatalError")` /
`Symbol.for("@workflow/errors//RetryableError")`. First load wins
per realm; the descriptor is non-writable / non-configurable to make
accidental clobbering loud.
- The revivers in `@workflow/core`'s common reducers module read the
consumer's `globalThis` (passed in as `global`) to pick up the
realm-local class, falling back to the host-imported class when no
registration is present (e.g. in the CLI / test runner).
* Use `types.isNativeError` to remap workflow stacks across VM realms
The runtime's run-failure path computes a source-map-remapped stack
and then assigns it back onto the thrown value via `if (err
instanceof Error) err.stack = errorStack`. Workflows run inside a
Node `vm` context, so a workflow-thrown error is an instance of the
VM realm's `Error` — `instanceof` against the host realm's
`Error` returns `false`, the assignment is skipped, and the
serialized `run_failed` event carries the un-remapped (bundled-line-
number) stack instead of the source-mapped one.
Switch the gate to `types.isNativeError`, which uses V8's internal
type tag and works across realms — same approach already in place
for the serialization reducers.
Caught by the local-prod e2e "nested function calls preserve message
and stack trace" and "cross-file imports preserve message and stack
trace" tests, which assert that the persisted run-error stack
contains `99_e2e.ts` / `helpers.ts`.
* Sync CLI revivers with core + add toJSON shim for Error subclasses
Two issues with the CLI's hand-rolled reviver list:
1. It hadn't been updated for the new first-class Error subclass
reducers (`TypeError`, `RangeError`, `FatalError`, `RetryableError`,
etc.). devalue throws "Unknown type X" when it encounters a
reduced value with no matching reviver, and `hydrateResourceIO`
swallows that error and surfaces the raw `Uint8Array` payload —
so `step.error` / `run.error` showed up as raw byte dumps in
`workflow inspect` output.
2. Even with all the right revivers, `Error.prototype`'s `message`
/ `stack` / `cause` are non-enumerable, so `JSON.stringify`
(used by `workflow inspect --json`) drops them — leaving the
subclass-specific enumerable fields (e.g. `FatalError.fatal`)
visible but the actual error data missing.
Fix:
- Build the CLI reviver set on top of `getCommonRevivers()` from
`@workflow/core` so the CLI stays in sync with the runtime's
reducer set automatically. New core reducers/revivers will Just
Work without any CLI-side change.
- Wrap each Error reviver from the common set with a thin shim that
attaches a non-enumerable `toJSON` method to the produced
`Error` instance. `JSON.stringify` calls `toJSON` and gets a
full object (`name` + `message` + `stack` + `cause` + any
enumerable subclass fields like `fatal` / `retryAfter` /
`errors`); `util.inspect` ignores `toJSON` and renders the
canonical `Error: msg\\n at ...` format. Best of both worlds for
CLI output without compromising the runtime hydration path.
Caught by the local-prod e2e "basic step error preserves" and
"cross-file step error preserves" tests, which read
`failedStep.error.message` / `.stack` from the CLI's JSON output.
* Clarify parseErrorJson JSDoc to match its always-null return
The previous JSDoc described preserving legacy values "for best-effort
hydration" which contradicted the implementation, where legacy errors
are intentionally surfaced as absent (the pre-pipeline shapes can't be
hydrated by the new error revivers). Rewrite the comment so the contract
matches behavior. Also rename the now-unused parameter to `_errorJson`
to reflect that the function ignores it.
Caught by a code review on #1851.
* Refine error-handler ergonomics on the step / run hot paths
Three review-driven adjustments that all touch the queue handlers and
their interaction with the error serialization pipeline:
1. Memoize the per-run encryption key fetch. The step handler used to
eagerly fetch + import the key at the top of every step delivery so
the value would be in scope for every potential dehydrateStepError
path. That pessimized step-started early-return cases (the fetch
happens unconditionally even when the step never reaches user code)
and required duplicating the same boilerplate at four call sites in
runtime.ts. Introduce `memoizeEncryptionKey(world, run)` in
runtime/helpers.ts that returns a lazy, single-fetch accessor;
step-handler / runtime call sites use `await getEncryptionKey()`
instead. The first caller pays the fetch cost, subsequent callers
await the cached promise, and steps that fail before any
encryption-aware work happens skip the fetch entirely.
2. Preserve the prior attempt's serialized error as the cause on the
defensive max-retries-exceeded `step_failed` re-invocation guard.
The existing comment explicitly opted out of cause attachment, but
the symmetric post-failure path below already does this and the
reviewer is right that consumers shouldn't have to walk the
step_retrying event history to recover the underlying error. Best-
effort: if hydration of the prior `step.error` throws, fall back
to a FatalError without cause rather than letting the event write
itself fail.
3. Document the intentional `unflatten` throw in
`hydrateStepError` / `hydrateRunError` for non-Uint8Array input.
SDK version is pinned per workflow run via skew protection so the
non-binary branch is dead in production; if a misshapen value
reaches it, surfacing the throw via the surrounding o11y try/catch
is more debuggable than masking it. Add a comment so future
reviewers don't reach for a defensive fallback.
A standalone `falls back to plaintext` suggestion on the run_failed
key fetch was rejected: when encryption is configured we should fail
loudly rather than silently emit plaintext error data. The queue's
redelivery semantics will retry the key fetch; persistent KMS outages
get logged with the existing "persistent error preventing the run from
being terminated" message rather than a security regression.
* Hydrate `event.eventData.error` in event listings
`hydrateEventData` enumerated the per-event fields that need
hydration (`result`, `input`, `output`, `metadata`, `payload`)
but omitted the new `error` field on `step_failed`,
`step_retrying`, and `run_failed` events. Without this branch,
o11y tools that list events (e.g. `workflow inspect events`) surface
the raw `Uint8Array` payload instead of a hydrated
`{ name, message, stack, … }` object even though the entity-level
`Run.error` / `Step.error` paths already hydrate.
Mirrors the existing per-field branches; the `try/catch` leaves the
field un-hydrated on parse failure rather than failing the whole
event view. Adds a unit test.
* Use `.is()` static checks in `classifyRunError` for cross-realm safety
Workflows execute inside a separate `vm` realm: the
`WorkflowRuntimeError` class bundled into the workflow code and the
host-imported one are distinct constructors, so an
`err instanceof WorkflowRuntimeError` check on a VM-thrown error
returns `false` and we'd misclassify genuine runtime errors (corrupted
event log, missing timestamps, workflow/step not registered) as user
errors.
Switch to each subclass's `.is()` static (a name-based duck check that
works across realms). Since `WorkflowRuntimeError.is` only matches its
own concrete name, enumerate every concrete subclass we want to
recognize (`StepNotRegisteredError`, `WorkflowNotRegisteredError`)
in a `RUNTIME_ERROR_CHECKS` table; keep that table in sync with the
class hierarchy in `@workflow/errors`.
Existing `classify-error.test.ts` already covers `WorkflowRuntimeError`
and `WorkflowNotRegisteredError` cases — both still pass.
* Add e2e coverage for step throws of non-Error values
We had `errorWorkflowThrowNonErrorValue` (workflow body throws a plain
object — round-trips verbatim as `WorkflowRunFailedError.cause`) but
no symmetric coverage for the step-throw side. Step-throw goes through
a different code path: non-Error values aren't recognized as
`FatalError` (no `name === 'FatalError'`) nor `RetryableError`,
so they take the transient retry path. After max retries the runtime
wraps the original thrown value as `cause` on a fresh `FatalError`
which the workflow's catch block then sees.
Add a workflow that throws a recognizable plain object from a step
with `maxRetries = 0` (so we exhaust on first attempt and avoid a
long test wait) and a workflow that asserts the wrapped FatalError
shape: `isFatal`, `instanceof FatalError`, message includes the
original object's serialized form, `cause` is the original non-Error
object verbatim with structure preserved.
Documents the current retry-then-wrap behavior so any future change
to "non-Error throws skip retries" semantics has to update the test.
* Note legacy postgres error-data loss in the run/step error changeset
Pre-upgrade failed runs that wrote into world-postgres's deprecated
`error` text column can't be hydrated through the new pipeline (the
shape is incompatible with the new revivers). The new runtime
intentionally surfaces them as `error: undefined` on read; the
original payload is still readable directly from the `errorJson`
column for manual inspection. Add a one-sentence note to the
changeset's migration text so consumers upgrading don't get blindsided
by suddenly-empty error fields on historical runs.
|
||
|
Pranay Prakash
|
1203dae70c |
Friendlier workflow errors (consolidated) (#1849)
* Introduce structured context-violation errors + Ansi renderer Phase 1: Add Ansi rendering helpers (frame, hint, note, help, code, inline) to @workflow/errors, and a chalk mock for readable snapshot tests. Phase 2: Add four context-violation error classes to @workflow/core (NotInWorkflowContextError, NotInStepContextError, NotInWorkflowOrStepContextError, UnavailableInWorkflowContextError) and apply them to all twelve user-facing throw sites so errors now include docs links and a structured "what/why/fix" frame. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * Address review: tighten changeset, implement ansifyName, harden Ansi - Tighten phase 1 changeset to a single sentence (per pranaygp review) and switch to double-quoted frontmatter (per Copilot + repo convention). - Implement `ansifyName` to actually apply dim styling to workflow/ / step/ prefixes; add an `Ansi.dim` helper to `@workflow/errors` so callers don't need to import chalk directly. - Remove the `void getWorkflowMetadata;` workaround in context-errors.ts by dropping the unused value import (we only needed the type and symbol). - Render the plain-Error throw in `workflow/get-workflow-metadata.ts` with `Ansi.frame` + docs link so the VM path matches the structured-class styling from the sibling step path (still uses a plain Error to avoid the module-init cycle). - Guard `buildUnderline` against zero-length markers so a stray empty token can't produce a negative `String.repeat` count. * Structured runtime logger metadata + fold in replay-timeout logging Adds a `.child()` and `.forRun(runId, workflowName)` child-logger API to the structured logger so runtime/step code doesn't have to repeat `workflowRunId`/`workflowName`/`stepId` on every call. Normalizes error metadata to structured `errorName` / `errorMessage` / `errorStack` fields instead of ad-hoc `error: err.message` strings, and adds comments to silent catches that swallow expected idempotency conflicts. Also folds in the pending changes from #1812 so that PR can be closed: - Standardize the console prefix to `[workflow-sdk]`. - Split the replay-timeout log into a warn-while-retrying vs. error-when-giving-up, and surface the underlying error when we can't mark a timed-out run as failed. - Include the error stack in the "Fatal runtime error during workflow setup" log and in the top-level user-code workflow error log so the stack surfaces in flattened log drains. - Drop the `[Workflows] "<runId>" - ` prefix from `buildWorkflowSuspensionMessage` — the structured logger now attaches run context. Supersedes #1812. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * Use double-quoted changeset frontmatter per repo convention * Add SerializationError + apply to user-facing serialization sites Phase 4 of friendlier errors: introduce a `SerializationError` class with an optional `hint` and a docs link (workflow-sdk.dev/err/serialization-failed), and adopt it at every user-facing serialization boundary in @workflow/core: - Locked ReadableStream at a workflow boundary - Unregistered class / missing `classId` / missing `WORKFLOW_DESERIALIZE` - Attempting to return step functions to clients or call workflow functions directly - Webhook `respondWith()` called outside a step - `dehydrate*` / `getSerializeStream` failures (workflow args/return, step args/return, stream chunks) Internal invariants (format prefix length checks, unknown format bytes, missing `STREAM_NAME_SYMBOL`, encryption key/size guards, etc.) now throw `WorkflowRuntimeError` instead of plain `Error` so the classifier and logger treat them consistently. `formatSerializationError` now returns `{ message, hint }` so the hint fragment can be rendered with the standard SerializationError framing instead of being baked into the message string. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * Use double-quoted changeset frontmatter per repo convention * Presentation-only user vs SDK error attribution Add describeError() that derives attribution and class-aware hints from existing error classes + RUN_ERROR_CODES — no event data changes. Wire into step failures, max-delivery exhaustion, run failures, and fatal setup errors so terminal logs include errorAttribution and a hint for known error types. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * Address review: describeError accepts precomputed errorCode + instanceof - `describeError(err, errorCode?)` now accepts an optional precomputed `RunErrorCode`. `classifyRunError(err)` only narrows to USER_ERROR / RUNTIME_ERROR, so the REPLAY_TIMEOUT and MAX_DELIVERIES_EXCEEDED branches were previously unreachable from the step / run failure log sites. Callers that know the failure category (runtime.ts for replay timeout and max-deliveries exhaustion) now pass the code in. - Context-violation checks use `instanceof` against the actual classes from context-errors.ts instead of a name-string set. Type-safe + survives class renames. - Wire the new hints through to the REPLAY_TIMEOUT and MAX_DELIVERIES_EXCEEDED log sites so those branches actually render a hint now. - 3 new tests cover the reachable code paths + precomputed-code override. - Changeset frontmatter switched to double quotes per repo convention. * Cosmetic consistency pass on remaining bare throws Internal invariants now use WorkflowRuntimeError so describeError attributes them to the SDK: missing startedAt, VM generateKey, closure-vars outside step context, ENOTSUP. defineHook().resume() formats schema validation failures as a readable list instead of a JSON blob. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * Use double-quoted changeset frontmatter per repo convention * Data-driven describeRunError + expose via @workflow/core/describe-error Observability renderers read persisted run_failed / step_failed event data, not live Error instances. describeRunError takes { errorCode, errorName } and returns the same { attribution, hint } shape as describeError, so the CLI and web UI can derive user-vs-SDK framing from the event log directly. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * Friendlier build-time errors: WorkflowBuildError class + applications Add `WorkflowBuildError` class in `@workflow/errors` with optional `hint` for an actionable next step, and apply it in `@workflow/builders` at user-facing sites: failed esbuild phases, unresolved built-in steps, and empty esbuild output now throw `WorkflowBuildError` with a hint pointing at the likely fix. Runtime invariants remain plain `Error`. * Polish friendlier-errors rendering: drop functionName leak, simplify docs link, redirect stack - Drop the readonly `functionName` param-property on context-error classes so util.inspect no longer prints a trailing `{ functionName: 'foo()' }` block. - Replace the `DocLink` ("label: https://…") shape with a plain `DocsUrl` template-literal type. Error output now renders a single clean line: `docs: https://…` (new `Ansi.docs` helper) instead of the noisier "note: Read more about foo(): https://…". - Add throw helpers (`throwNotInWorkflowContext`, etc.) that call `Error.captureStackTrace(err, stackStartFn)` on V8 engines so the top frame of the thrown error points at the user's call site instead of at the gate function inside the framework. Callers pass themselves as the boundary. - Refactor `defineHook()` (both root and `/workflow`) to use named function closures rather than `this.create`/`this.resume`, since the stack redirect relies on a stable function identity that survives destructuring. - Update context-errors.test.ts to snapshot the new `docs:` framing and to add a regression test asserting the top stack frame is the user call site. * Consolidate friendlier-errors stack: fix ANSI leak + non-retry semantics Addresses PR review feedback across the 8-phase friendlier-errors stack and fixes issues surfaced by manual testing (createHook() inside a step): - ANSI no longer leaks into .message / .stack. Context-violation errors now store plain text on .message and render the colored framed form lazily via [util.inspect.custom] / toString(). Structured logs, log drains, CBOR-serialized events, and JSON payloads no longer contain raw \x1B[...m bytes. - Context violations are now fatal. ContextViolationError sets fatal = true; FatalError.is(err) recognizes any error with a fatal: true own property. Calling createHook() from a step no longer burns three retry attempts on a guaranteed-to-fail context violation. - Ansi helpers moved to @workflow/errors/ansi subpath so imports from @workflow/errors no longer pull chalk into consumers that only want error classes (addresses reviewer VaguelySerious). - Shared redirectStackToCaller helper in packages/core/src/capture-stack.ts, used by both context-errors.ts and workflow/get-workflow-metadata.ts (addresses Copilot review on #1849). - Structured framed content: ContextViolationError now takes a structured FramedContent (title segments + detail branches) and renders plain/pretty from the same source of truth. Tightens the eight existing phase changesets to 1-2 sentences each and adds four new scoped changesets (errors-ansi-subpath, context-errors-plain-message, context-errors-fatal, capture-stack-shared) for the followup fixes, so the final changelog history stays readable. * test: update step-handler mocks for scoped forRun() logger The runtime logger now uses .forRun(runId, name, {stepId, stepName}) to attach scope context, so 409-handling log calls no longer repeat {workflowRunId, stepId} in every metadata bag — those live on the scoped logger instance. Update the mock to return itself from forRun() and tighten assertions to check both the log args (errorName/errorMessage) and the forRun() scope. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * Mark SerializationError fatal + route dehydration through step-failure path SerializationError now carries readonly fatal = true. Step-return dehydration is wrapped inside the user-code try/catch so that the resulting error flows through userCodeFailed → step_failed → FatalError.is() short-circuit instead of bubbling up as HTTP 500 and triggering a queue retry loop. Retrying a step that returned a non-POJO is guaranteed to fail the same way, so this saves ~20s and 3 near- identical error blocks per serialization failure. * Add logging snapshot tests + manual-test artifacts Snapshot tests lock in the exact shape of: - describeError() payloads (attribution, errorCode, hint) for every classification — plain Error, SerializationError, context-violation, WorkflowRuntimeError, REPLAY_TIMEOUT, MAX_DELIVERIES_EXCEEDED. - The scoped-logger call signature for the two canonical runtime failure paths (fatal-bubble and hit-max-retries), so refactors of forRun() / child() metadata merging can't silently change what users see in their log drains. SerializationError now also has a direct test for readonly fatal=true + FatalError.is() recognition. pr-artifacts/ contains real log-output snapshots from running the nextjs-turbopack workbench against five error scenarios. These are reference material for reviewers and are flagged to be removed before merge. * Readable step-fatal logs: inline stack + friendly step/workflow names The step-level fatal-error log used to embed the full stack trace inside an `errorStack` string field in the metadata object, so util.inspect rendered it as a quote-escaped, line-continuation blob when the log hit the terminal — unreadable in practice. Move framing + stack into the log *message* (matching the workflow-level log in runtime.ts) and keep the metadata object compact with only the indexable structured fields (`errorAttribution`, `errorName`, `errorMessage`, `hint`, IDs). Log drains still get the same keys; humans now see a readable stack trace. Also introduce `formatStepName` / `formatWorkflowName` in `@workflow/utils` that render machine names (`step//./workflows/1_simple//add`) as `add (./workflows/1_simple)` in log framings, using the existing `parseStepName` / `parseWorkflowName` parsers. Applied to step-fatal, hit-max-retries, exceeded-max-retries, and workflow-threw log sites. Artifacts in pr-artifacts/ updated to show the new output shape, and renamed .log → .md since they're Markdown and IDE previews are nicer that way. * Opinionated pretty formatter for runtime structured-log metadata Replace util.inspect's default object dump (which quote-escapes multi-line stacks and paragraph hints into a single-line JSON-y blob) with a workflow-aware formatter that composes the entire log line into a single string passed to console.error / console.warn. Highlights of the new output: - Per-run / per-step IDs render with their parsed friendly names so users see `wrun_… · simple (./workflows/1_simple)` instead of just the raw `workflowName: 'workflow//./workflows/1_simple//simple'`. - Color-coded attribution badge (user error red / sdk error magenta) paired with the error class in bold. - Hints render as a paragraph under `hint:` rather than a backslash- `\n`-escaped string. - Drops redundant fields (errorStack always; errorMessage when it's already in the parent message) to avoid double-printing. - Unknown fields fall through as a sorted `key value` tail so we never silently drop log information. @workflow/errors/ansi gains bold/red/magenta helpers used by the formatter. The web / web-shared packages don't consume stderr — they read structured event payloads from the World event log — so this is presentation-only at the runtime layer. * ci(benchmarks): disable pnpm cache for getCommunityWorldsMatrix The job never runs `pnpm install` (it just calls `node` against a checked-in script), so the pnpm store path never exists. The post-job `actions/setup-node@v4` cache-save then fails with `Path Validation Error: Path(s) specified in the action for caching do(es) not exist` and red-X's the entire job even though the matrix step succeeded. The setup-workflow-dev composite already has a `cache-pnpm` opt-out input for this exact case — wire it through here. * Address PR review comments: inspect dedup, cause leak, retry-loop tests - ContextViolationError: util.inspect(err) duplicated every framed detail line because the stack-tail strip only sliced the first message line. V8's Error.stack reads `Name: messageLine1\n messageLine2\n at ...`, so for our multi-line `title\n╰▶ docs: …` messages every detail line was getting prepended twice (once in the pretty form, once via the unsliced message tail). Count the actual message lines and slice past all of them. Repro test asserts `╰▶ docs:` appears exactly once. - WorkflowError: stop assigning `cause: undefined` as an enumerable own property when no cause is provided. Subclasses (every error in this PR) inherit the parent constructor; the unconditional assignment polluted `util.inspect(err)` output with `{ cause: undefined, … }` on every no-cause instance. The `super(...)` call already conditionally sets `.cause` non-enumerably when `options.cause` is provided. - step-handler.test.ts: add a regression-gate suite that exercises the fatal-vs-retryable retry-loop wiring directly. Asserts that an error with `fatal: true` produces exactly one `step_failed` event with no `step_retrying`, and that a non-fatal `Error` retries via `step_retrying` on early attempts and emits `step_failed` once the retry budget is exhausted. Catches the silent-regression case where `fatal = true` is removed from a context-violation error class but the `FatalError.is()` unit tests stay green. * Consolidate changesets + remove pr-artifacts Address review feedback to drastically shorten the changesets — fold the 15 file-by-file entries into a single user-facing changeset for @workflow/core / errors / builders / utils. Also drop the pr-artifacts/ folder (reviewer-only log captures, no longer needed). * Polish runtime error logging: layout, stack trim, hint consolidation Five user-driven fixes from manual smoke-testing of #1849: 1. Logger layout. composeLogLine() now puts the structured-fields block (attribution badge, run/step IDs, error code) **between** the framing line and the stack body, instead of after it where 30+ lines of stack buried the most useful information. The framing stays at the top, stack at the bottom, structured info readable at a glance. 2. Stack trim. Drops framework-internal frames (`node_modules/.pnpm/`, `node:internal/`, Turbopack-bundled `node_modules__pnpm_*` chunks, `_next_dist_*` chunks) and caps the surviving frame count at 6 so the stack stays compact even on heavy async wrappers. Suppressed runs emit one summary line so users know the trim happened. 3. Wrapper-route noise. The nextjs-turbopack workbench's start route was catching `WorkflowRunFailedError` rejection on `Promise.race([readLoop(), run.returnValue])` and re-logging it via `console.error('Error in workflow stream:', error)` plus `controller.error(error)` — which then triggered Next.js's `⨯ failed to pipe response` overlay. The SDK already logs the failure cleanly upstream and the runId is on the response header, so the wrapper now closes the SSE stream cleanly on WorkflowRunFailedError. 4. Consistent framed `╰▶ hint:` / `╰▶ docs:` layout for all errors that carry a hint or docs slug. WorkflowError, SerializationError, and WorkflowBuildError now share one `appendFramedDetails` helper matching the box-drawing structure that ContextViolationError already used. Was: blank-line-separated `Learn more: <url>`. Now: one tree, indistinguishable from context-violation rendering. 5. Drop the duplicate logger-side `hint` field. Hints now live on the error message only — actionable hints get serialized into the event log, rehydrated on the workflow side, and shown in observability automatically. The previous logger-only hint duplicated stderr but never made it past the step boundary. Updated SerializationError hint to point at the foundations doc ("Ensure you're returning workflow serializable types. Check the serialization docs to see what's serializable: https://workflow-sdk.dev/docs/foundations/serialization") instead of the hardcoded `(plain objects, arrays, primitives, …)` list, which drifted out of sync as the supported types grew. Same hint reuses for step args, workflow args/return, stream messages, and any other site that goes through `formatSerializationError`. Also retitled the retry summary `3 retries` → `3 max retries` since "3 retries" next to "4 attempts" was ambiguous (already-happened vs. budget). * Trim error-card title + drop machine step name from persisted error - ErrorStackBlock (web observability): show just the first non-empty trimmed line of the error message in the card title with single-line truncation. Multi-line messages (`Failed to serialize step return value\n╰▶ hint: …`) were rendering the entire framed body in the title, pushing the copy button off-screen and burying the scannability of the headline. Full message stays in the body via the stack (V8 prepends `Name: message` to `Error.stack`), so no information is lost; hover-tooltip exposes the full title text. - Persisted error message: drop the `Step "step//./.../foo"` machine name from `Step failed after N retries: …` and `Step exceeded max retries (…)` strings. Observability already attributes the event to a specific step via the UI tree, and the CLI logger emits the friendly `Step foo (./...) hit max retries` framing on its own line. Embedding the raw `step//./...` machine name in the persisted message text was duplicate noise. * Update .changeset/friendlier-errors.md Co-authored-by: Peter Wielander <mittgfu@gmail.com> Signed-off-by: Pranay Prakash <pranay.gp@gmail.com> * Update .changeset/pretty-log-format.md Co-authored-by: Peter Wielander <mittgfu@gmail.com> Signed-off-by: Pranay Prakash <pranay.gp@gmail.com> * Update SerializationError snapshot tests for slug-less message The class no longer attaches a slug-based `╰▶ docs:` line — the foundations URL is embedded directly in the hint via the `formatSerializationError` helper in @workflow/core. Update the test expectations accordingly: - bare-title case is now a single line (no docs link) - hint case renders one `╰▶ hint: …` branch (no second branch) * Update serialization.test.ts hint assertions for foundations URL Four `should throw error for an unsupported type` cases were still asserting on the old hardcoded type list. Update to the new hint phrasing that points at the foundations doc, matching the change in `formatSerializationError` (`packages/core/src/serialization/errors.ts`). --------- Signed-off-by: Pranay Prakash <pranay.gp@gmail.com> Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com> Co-authored-by: Peter Wielander <mittgfu@gmail.com> |
||
|
Nathan Rajlich
|
173756dc4d |
[docs] Rename workflowdevkit to workflowsdk and useworkflow.dev to workflow-sdk.dev (#1759)
* [docs] Rename workflowdevkit references to workflowsdk * [docs] Rename useworkflow.dev to workflow-sdk.dev * [chore] Add changeset for domain rename * [docs] Revert sitemap rewrite to useworkflow.dev (crawled-sitemap not yet available for new domain) |
||
|
Peter Wielander
|
6dc1b78582 | [core] Extend flow route duration to "max" and fail runs where replay takes too long (#1567) | ||
|
Harpreet
|
cdf90d5a38 |
Rename Workflow DevKit to Workflow SDK, remove beta badge, add tweet wall (#1541)
* Rename Workflow DevKit to Workflow SDK, remove beta badge, add tweet wall - Rename "Workflow DevKit" to "Workflow SDK" across all files (~108 files) - Rename standalone "WDK" references to "Workflow SDK" - Remove beta badge from homepage hero - Add tweet wall component to homepage with 4 builder testimonials Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * Rename Workflow DevKit to Workflow SDK, remove beta badge, add tweet wall - Rename "Workflow DevKit" to "Workflow SDK" across all files (~108 files) - Rename standalone "WDK" references to "Workflow SDK" - Remove beta badge from homepage hero - Add tweet wall component to homepage with 4 builder testimonials Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> Signed-off-by: Harpreet Arora <harpreet.txt@gmail.com> * Address review: fix missed trigger phrase renames and bump skill versions - Rename "workflow devkit" to "workflow sdk" in trigger phrases for both skill files - Bump workflow-init SKILL.md version to 1.1 - Bump workflow SKILL.md version to 1.5 - Note: CLAUDE.md is a symlink to AGENTS.md, already renamed Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> Signed-off-by: Harpreet Arora <harpreet.txt@gmail.com> * link correct tweet --------- Signed-off-by: Harpreet Arora <harpreet.txt@gmail.com> Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com> Co-authored-by: Karthik Kalyanaraman <karthik.kalyanaraman@vercel.com> |
||
|
Pranay Prakash
|
672d9195a4 |
Fix step/workflow not found errors to fail gracefully instead of queue retry (#1452)
* feat: enhance error handling for missing workflow functions Slack-Thread: https://vercel.slack.com/archives/C09G3EQAL84/p1773856370214769?thread_ts=1773856370.214769&cid=C09G3EQAL84 Co-authored-by: Pranay Prakash <1797812+pranaygp@users.noreply.github.com> * fix: update step not found handling to match FatalError pattern Move step function validation after step_started and call step_failed directly if not found. Co-authored-by: Pranay Prakash <1797812+pranaygp@users.noreply.github.com> * changes Signed-off-by: Pranay Prakash <pranay.gp@gmail.com> * feat: add StepNotRegisteredError and WorkflowNotRegisteredError semantic errors Introduce dedicated error types for when step/workflow functions are not registered in the current deployment, replacing generic WorkflowRuntimeError. These are infrastructure errors (not user code errors) with proper error slugs, docs pages, and a new FUNCTION_NOT_REGISTERED error code. Step not found fails the step (like FatalError) so the workflow can handle it gracefully. Workflow not found fails the run. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: address PR review comments - Remove FUNCTION_NOT_REGISTERED error code, use RUNTIME_ERROR instead - Use .is() instead of instanceof for WorkflowRuntimeError check in runtime.ts - Remove non-working example from WorkflowNotRegisteredError docs (custom errors not serialized yet) - Update all references from FUNCTION_NOT_REGISTERED to RUNTIME_ERROR Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat: add e2e tests for step/workflow not registered errors and fix docs typecheck E2E tests: - WorkflowNotRegisteredError: start a run with a fake workflowId, verify the run fails with RUNTIME_ERROR - StepNotRegisteredError (caught): workflow catches the step failure, verify workflow completes and step is marked failed - StepNotRegisteredError (uncaught): verify the run fails when workflow doesn't catch the error Step not registered is tested by manually invoking useStep with a non-existent step ID in the workflow VM — this is the same pattern the SWC transform generates for real step calls. Also fix docs typecheck by using declare/\@setup pattern instead of \@skip-typecheck for code samples. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: cast globalThis to any for Symbol index access in e2e workflow TypeScript's strict mode doesn't allow using a symbol to index globalThis. Cast to any since this runs in the workflow VM where the symbol is defined. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: classify WorkflowNotRegisteredError as RUNTIME_ERROR The .is() check uses name-based matching, so WorkflowNotRegisteredError (name='WorkflowNotRegisteredError') doesn't match WorkflowRuntimeError.is(). Add explicit check in classifyRunError so the error code is RUNTIME_ERROR instead of USER_ERROR. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: use instanceof for WorkflowRuntimeError checks, improve docs Address PR review feedback: 1. Revert .is() checks back to instanceof WorkflowRuntimeError in runtime.ts and classify-error.ts. instanceof catches all subclasses (current and future), which is the correct behavior for these catch blocks. 2. Remove duplicated try/catch example from step-not-registered-error API reference (troubleshooting page already has it). 3. Add Callout in API reference docs clarifying that .is() works in server-side Node.js code but not inside "use workflow" functions where errors arrive deserialized from the event log. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * changes Signed-off-by: Pranay Prakash <pranay.gp@gmail.com> --------- Signed-off-by: Pranay Prakash <pranay.gp@gmail.com> Co-authored-by: v0 <v0[bot]@users.noreply.github.com> Co-authored-by: Pranay Prakash <1797812+pranaygp@users.noreply.github.com> Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com> |
||
|
Pranay Prakash
|
2ef33d2828 |
feat: export semantic error types and add API reference docs (#1447)
* feat: export semantic error types and add API reference documentation Add missing error exports (HookNotFoundError, EntityConflictError, RunExpiredError, TooEarlyError, ThrottleError, RunNotSupportedError, WorkflowWorldError) to workflow/internal/errors. Create new error classes for world-level semantics. Tighten TSDoc comments on all error classes. Add API reference docs for all error types. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: use @setup declarations, workflow/errors import, and errors/ doc section - Replace @skip-typecheck with proper `declare` + `// @setup` lines so code samples are typechecked but setup lines hidden from readers - Add `workflow/errors` export to package.json (public API, replaces `workflow/internal/errors` in docs) - Add `workflow/errors` path mapping in docs-typecheck type-checker - Add HookConflictError to re-export list - Move all error docs under api-reference/workflow/errors/ subdirectory - Update all internal cross-references and links Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * refactor: move error docs to top-level workflow-errors section - Move semantic error docs to api-reference/workflow-errors/ (matching the workflow/errors import path, like workflow-api for workflow/api) - Keep FatalError and RetryableError in api-reference/workflow/ since they're imported from workflow, not workflow/errors - Fix all cross-reference links Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * chore: update HTTP debug logger JSDoc to clarify scope Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: make TooEarlyError.retryAfter a number (seconds) matching WorkflowWorldError TooEarlyError.retryAfter is now seconds (number) instead of a Date, consistent with ThrottleError and WorkflowWorldError. The conversion from seconds to Date is done at the consumer site (step-handler) rather than at construction time. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: address review feedback on docs accuracy - WorkflowWorldError docs: add status, code, url, retryAfter properties to TSDoc; clarify that .is() only matches direct instances (not subclasses); use instanceof in catch-all example - TooEarlyError/ThrottleError docs: mark retryAfter as optional (?) to match actual type definitions Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com> |
||
|
Pranay Prakash
|
beccbc4298 |
feat: enforce max queue deliveries in handlers with graceful failure (#1344)
* feat: enforce max queue deliveries in handlers with graceful failure Replace VQS maxDeliveries cap with handler-level enforcement. Handlers now gracefully fail runs/steps after excessive queue redeliveries, preventing "phantom stuck" runs. - Add MAX_QUEUE_DELIVERIES constant (64) and enforce in both workflow and step handlers with run_failed/step_failed events - Remove maxDeliveries from VQS trigger configs (builders + sveltekit) - Improve world-local queue: safety limit loop, structured logging with runId/stepId, backoff delay on failures - Add MAX_DELIVERIES_EXCEEDED error code Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * address PR review comments - Expand max-delivery comments explaining minimal-work approach - Make workflow handler error message verbose (matching step handler) - Fix comment: "consume the message silently" - Reduce local queue safety limit from 1000 to 256 Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * change MAX_QUEUE_DELIVERIES to 48 and use 5s linear backoff locally VQS uses linear 5s backoff for attempts 1-32, then exponential capped at 2h. At 48 attempts total elapsed time is ~20h, safely under the 24h message visibility limit. Local world now uses 5s linear backoff to approximate VQS timing. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com> |
||
|
Pranay Prakash
|
73a851ada6 |
feat: add HookConflictError for hook token conflicts (#1448)
* feat: add HookConflictError for hook token conflicts Replace WorkflowRuntimeError with a dedicated HookConflictError class for hook token conflicts. This correctly classifies the error as a USER_ERROR (duplicate token is a user mistake) rather than a RUNTIME_ERROR. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: use HookConflictError.is() instead of instanceof in docs The .is() static method handles cross-VM/realm boundaries where instanceof can be unreliable. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com> |
||
|
Pranay Prakash
|
aee035f944 |
refactor: replace HTTP status code checks with semantic error types (#1342)
* feat: classify run failure error codes and improve error logging - Add RUN_ERROR_CODES (USER_ERROR, RUNTIME_ERROR) to @workflow/errors - Populate errorCode in run_failed events via classifyRunError() - Update web UI StatusBadge to show amber dot for infrastructure errors - Improve world-local queue error logging (concise, no body dump) - Improve schema validation error messages (concise, verbose behind DEBUG) - Add e2e tests for error code flow and infrastructure error retry Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat: add semantic error types to replace HTTP status code checks in runtime Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat: classify run failure error codes and improve error logging - Add RUN_ERROR_CODES (USER_ERROR, RUNTIME_ERROR) to @workflow/errors - Populate errorCode in run_failed events via classifyRunError() - Update web UI StatusBadge to show amber dot for infrastructure errors - Improve world-local queue error logging (concise, no body dump) - Improve schema validation error messages (concise, verbose behind DEBUG) - Add e2e tests for error code flow and infrastructure error retry Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat: classify run failure error codes and improve error logging - Add RUN_ERROR_CODES (USER_ERROR, RUNTIME_ERROR) to @workflow/errors - Populate errorCode in run_failed events via classifyRunError() - Update web UI StatusBadge to show amber dot for infrastructure errors - Improve world-local queue error logging (concise, no body dump) - Improve schema validation error messages (concise, verbose behind DEBUG) - Add e2e tests for error code flow and infrastructure error retry Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * address PR review comments - Remove dead `meta` option from TooEarlyError constructor (TooTallNate) - Extract `throwWithTrace` helper to deduplicate span recording in world-vercel makeRequest (TooTallNate) - Restore `maxAttempts` const for stable retry count logging (TooTallNate) - Fix behavioral regression: add WorkflowAPIError 404 fallback in suspension-handler hook disposal to handle world-vercel path where makeRequest doesn't map 404 to HookNotFoundError (TooTallNate) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: translate 404 to HookNotFoundError at the world-vercel boundary Move the 404 → HookNotFoundError translation into world-vercel's createWorkflowRunEvent, where we know the event type context. For hook-related events (hook_created, hook_disposed, hook_received, hook_conflict), a 404 from the server means the hook was not found. This removes the WorkflowAPIError 404 fallback from the runtime's suspension-handler, keeping the runtime fully decoupled from HTTP status codes. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: parse Retry-After for 425 responses and narrow hook event set - Parse Retry-After header unconditionally so TooEarlyError gets the server-provided delay instead of always falling back to ~1s - Narrow hookEventsRequiringExistence to only hook_disposed and hook_received (matching world-local's set), since hook_created and hook_conflict don't imply the hook must already exist Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * rename WorkflowAPIError to WorkflowWorldError Breaking change: rename WorkflowAPIError → WorkflowWorldError to better reflect that this error represents world (storage backend) failures, not HTTP API errors specifically. Updated across all packages: errors, core, world-local, world-vercel, world-postgres, workflow, and web. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com> |
||
|
Pranay Prakash
|
84599b7ec5 |
feat: classify run failure error codes and improve error logging (#1340)
- Add RUN_ERROR_CODES (USER_ERROR, RUNTIME_ERROR) to @workflow/errors - Populate errorCode in run_failed events via classifyRunError() - Update web UI StatusBadge to show amber dot for infrastructure errors - Improve world-local queue error logging (concise, no body dump) - Improve schema validation error messages (concise, verbose behind DEBUG) - Add e2e tests for error code flow and infrastructure error retry Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com> |
||
|
Pranay Prakash
|
adfe8b6b11 |
Add isWebhook flag to prevent hooks from being resumed via public webhook endpoint (#1270)
* Add isWebhook flag to prevent hooks from being resumed via public webhook endpoint Hooks created with createHook() are now non-resumable via the public webhook endpoint by default (isWebhook=false). Only hooks created with createWebhook() set isWebhook=true, allowing them to be resumed via the public URL. Also adds HookNotFoundError thrown by all world backends when a webhook token doesn't match any hook, and an e2e test for the new behavior. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * Fix world-local: default isWebhook to false and fix test assertions - Default isWebhook to false at write time in events-storage - Default isWebhook to false at read time in hooks-storage (for old data) - Update test assertions to match HookNotFoundError message Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * Fix: default isWebhook to true for backwards compat, add postgres migration - Revert read-side default to `isWebhook ?? true` in world-local for backwards compatibility with existing hooks that predate the field - Add postgres migration 0009 to add `is_webhook` column with default true - Update drizzle schema to match Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com> |
||
|
Pranay Prakash
|
c2b4fe9906 |
fix(core): detect and fatal error on orphaned/invalid events (#1055)
* fix(core): detect and fatal error on orphaned/invalid events in EventsConsumer When an event log has duplicate or invalid events (e.g., 2 wait_completed for a single wait_created), the EventsConsumer gets stuck: the orphaned event has no callback to consume it, so eventIndex never advances, blocking all subsequent events and hanging the workflow forever. This adds deferred orphaned event detection that raises a WorkflowRuntimeError instead. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * add corrupted-event-log error slug and docs page Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * add changeset for orphaned event detection Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * address PR feedback on docs and JSDoc - Add JSDoc to EventsConsumer constructor's onUnconsumedEvent parameter - Rewrite docs to clarify this is a SDK/server bug, not user code - Remove cancel/programmatic retry advice; run is already marked failed - Emphasize error is uncatchable inside workflow code - Add steps: upgrade package, retry via dashboard/CLI, report issue on GitHub Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * address PR review: structured options, required callback, reorder docs - Change EventsConsumer constructor to use structured EventsConsumerOptions object - Make onUnconsumedEvent required (all callsites provide it) - Move corrupted-event-log card to last position in errors index - Update all callsites in tests Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix docs code sample to pass type-checking Use correct imports (getRun from workflow/api) and declare variables to satisfy the docs-typecheck CI job. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * add e2e test for parallel sleep (Promise.all with 10x sleep) Adds parallelSleepWorkflow that does Promise.all with 10 concurrent sleep('1s') calls, and an e2e test asserting it completes or fails within a reasonable time (not hanging). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix parallelSleep e2e test: await returnValue instead of checking status run.status returns immediately and the workflow is still 'running'. Use run.returnValue which polls until completion. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix parallelSleep e2e: assert parallel completion under 10s The test verifies 10 concurrent sleep('1s') calls complete in parallel (~1s) rather than serially (10s) or hanging indefinitely. 30s timeout is sufficient since the workflow should finish in ~1-2s. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> |
||
|
Nathan Rajlich
|
8cfb43808b |
Use @vercel/cli-auth for auth token reading and OAuth refresh (#1043)
* Use `@vercel/cli-auth` for auth token reading and OAuth refresh Replace the manual auth.json reading logic in the CLI with the `@vercel/cli-auth` package, which handles credential storage via CredentialsStore and OAuth token refresh via the OAuth client. Previously, the CLI would read the token from disk but had no refresh logic — if the token was expired, API calls would fail. Now, getAuthToken() checks token expiry and automatically refreshes it using the stored refresh token before returning it. * Remove dead logging |
||
|
Peter Wielander
|
56f22219b3 | [core] Handle 429 and 500 errors from worlds in runtime (#966) | ||
|
Pranay Prakash
|
4966b728a8 |
implement event-sourced architecture (#621)
* perf: implement event-sourced architecture
* Apply suggestions from code review
* Improve invalid event log handling in step/hook/wait
* Handle serialized workflow run errors correctly
* log error in failing test
* Handle queue idempotency in vercel world
* hotfix for error propogation
* Fix: Incorrect HTTP status code 409 should be 410 for terminal run state rejections in postgres storage
* Fix: The code attempts to pass an unsupported `fatal` property when creating a `step_failed` event. The TypeScript schema for `step_failed` events only allows `error` and `stack` properties, so the `fatal` property causes a compilation error.
This commit fixes the issue reported at packages/core/src/runtime/step-handler.ts:133-139
## TypeScript error: Invalid property 'fatal' in step_failed event
**What fails:** TypeScript compilation fails in `packages/core` due to an invalid property in the `step_failed` event creation.
**How to reproduce:**
```bash
cd /vercel/sandbox/primary
pnpm run -F @workflow/core build
```
**Result:**
```
src/runtime/step-handler.ts(133,32): error TS2769: No overload matches this call.
Overload 1 of 2, '(runId: null, data: { eventType: "run_created"; ... }, gave the following error.
Argument of type 'string' is not assignable to parameter of type 'null'.
Overload 2 of 2, '(runId: string, data: CreateEventRequest, params?: CreateEventParams | undefined): Promise<EventResult>', gave the following error.
Object literal may only specify known properties, and 'fatal' does not exist in type '{ error: any; stack?: string | undefined; }'.
```
**Issue:** The code attempted to pass a `fatal: true` property in the `eventData` object when creating a `step_failed` event. However, the event schema defined in `packages/world/src/events.ts` for `StepFailedEventSchema` only allows `error` and `stack` properties - the `fatal` property is not part of the schema.
Co-authored-by: Vercel <vercel[bot]@users.noreply.github.com>
* Fix: Code silently skips updating workflowRun if result.run is undefined, causing workflows to be incorrectly skipped instead of throwing an error
* Add hook_conflict event type for duplicate token detection
Implements hook_conflict events across all world implementations to handle
cases where a workflow attempts to use a hook token already claimed by another
workflow. Instead of throwing errors, the system now records hook_conflict
events in the event log, enabling deterministic replay.
- Add HookConflictEvent schema to @workflow/world
- Implement hook_conflict in world-local, world-postgres, and suspension-handler
- Update hook consumer to reject promises with WorkflowRuntimeError on conflict
- Add HOOK_CONFLICT error slug with documentation
- Add e2e and unit tests for conflict scenarios
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* Add changeset for hook_conflict events
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* Add unit tests for hook_conflict handling
- Add tests for hook_conflict event in workflow.test.ts
- Fix world-postgres test to not expect removed message field
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* Improve hook-conflict.mdx error guide
- Remove redundant third point in 'Why This Happens' section
- Add example showing how to handle WorkflowRuntimeError for hook conflicts
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* Fix docs validation: add hook-conflict to errors index
- Fix broken link in hook-conflict.mdx (/docs/foundations/webhooks -> /docs/api-reference/workflow/create-webhook)
- Add hook-conflict to errors index page so it's discoverable by the docs link validator
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* Fix world-local tests for hook_conflict event behavior
Update tests to expect hook_conflict events instead of thrown errors when
duplicate hook tokens are used. This aligns with the new event-sourced
approach where conflicts are recorded as events rather than thrown.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* Add specVersion property to World interface for backwards compatibility
- Add specVersion property to World interface to track world package version
- Add specVersion to WorkflowRun schema and run_created event data
- World implementations (vercel, local, postgres) set specVersion from npm version
- Server can use specVersion to route operations based on world version
- Add specVersion display to observability UI attribute panel
- Add spec_version column to postgres runs schema
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* Add migration for spec_version column in postgres schema
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* Add drizzle migration journal and snapshot for spec_version column
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* Regenerate postgres migration using drizzle-kit
- Properly generates migration with drizzle-kit CLI
- Removes deprecated 'paused' status from enum
- Adds spec_version column
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* Add backwards compatibility for event-sourced runs
- Add RunNotSupportedError for runs requiring newer world versions
- Add semver-based version utilities (isLegacyVersion) to @workflow/world
- World implementations check specVersion and route to legacy handlers
- Legacy runs (< 4.1.0): run_cancelled skips event storage, wait_completed stores event only
- New runs always get current world version (4.1.0-beta.0)
- Make EventResult.event optional for legacy compatibility
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* Refactor spec version from semver strings to integers
Replace semver-based version compatibility with explicit integer spec versions:
- SPEC_VERSION_LEGACY (1): pre-event-sourcing runs
- SPEC_VERSION_CURRENT (2): event-sourced architecture
Use branded SpecVersion type to enforce importing from @workflow/world.
Remove semver dependency from world, world-local, and world-postgres.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* Store error as CBOR in postgres world for consistency with input/output
- Add error_cbor bytea columns to workflow_runs and workflow_steps tables
- Deprecate text error column, rename to errorJson with fallback parsing
- Remove JSON.stringify from error writes (run_failed, step_failed, step_retrying)
- Add parseErrorJson helper for backwards compatibility with legacy data
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* Use WorkflowRuntimeError and improve run entity handling in core runtime
- Replace generic Error with WorkflowRuntimeError for runtime assertions
- Add explicit check for run entity in run_created response
- Use run.runId instead of event.runId for consistency
- Use actual run status instead of hardcoded 'pending' in attributes
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* Add specVersion to Step, Hook, and Event entities
- Add specVersion field to Step, Hook, and Event interfaces in @workflow/world
- Add spec_version column to steps, hooks, events tables in postgres schema
- Set specVersion to SPEC_VERSION_CURRENT when creating entities in all worlds
- Update migration to include spec_version columns for all entity tables
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* Refactor world-local storage into modular files
Split storage.ts (1041 lines) into smaller, focused modules:
- storage/filters.ts: Data filtering helpers
- storage/helpers.ts: ULID and date utilities
- storage/hooks-storage.ts: Hook CRUD operations
- storage/legacy.ts: Legacy event handling
- storage/runs-storage.ts: Run get/list operations
- storage/steps-storage.ts: Step get/list operations
- storage/events-storage.ts: Event create/list operations
- storage/index.ts: Main composition
Also extracted test helpers to test-helpers.ts for reusability.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* Remove genversion and World.specVersion property
The World.specVersion string property was never actually read - only
the numeric SPEC_VERSION_CURRENT is used for backwards compatibility.
- Remove genversion dependency and generated version.ts from @workflow/world
- Remove specVersion property from World interface and all implementations
- Minor fix: correct error message to reference 'workflow' package
- Minor fix: correct error source priority in world-vercel events.ts
- Minor fix: update comment in runtime.ts
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* Remove genversion from world-local, world-postgres, and world-vercel
These packages no longer need genversion since we removed the
World.specVersion property in the previous commit.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* Remove version.ts from .gitignore files
No longer needed after removing genversion.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* Add legacy/backwards compatibility tests
Tests for world-local and world-postgres covering:
- Legacy runs (specVersion < 2 or null/undefined)
- run_cancelled handling (updates run, no event stored)
- wait_completed handling (stores event only)
- Rejection of unsupported events
- Hook cleanup on cancellation
- Future runs (specVersion > current)
- Rejection with RunNotSupportedError
- Current version runs (normal processing)
- Legacy error parsing (errorJson field parsing)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* Add hook_received support for legacy runs
When resumeHook() is called on a legacy run (specVersion < 2), the
hook_received event was previously rejected. This adds support for
storing hook_received events on legacy runs without entity mutation,
matching the behavior of wait_completed.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* Fix missing genversion in world-vercel
Signed-off-by: Peter Wielander <mittgfu@gmail.com>
* Remove deprecated workflow_completed, workflow_failed, and workflow_started events
Replace with run_completed, run_failed, and run_started equivalents.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* Add specVersion to EventWithRefsSchema in world-vercel
The manually-created EventWithRefsSchema was missing the specVersion field,
which caused specVersion to be stripped when using lazy (refs) mode for events.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* Wire specVersion from client through world backends
- Core runtime now sends specVersion in run_created eventData
- world-local accepts specVersion from eventData (defaults to current)
- world-postgres accepts specVersion from eventData (defaults to current)
This matches workflow-server behavior where v2 endpoints accept
specVersion from the client, while v1 endpoints default to legacy.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* Move specVersion to event object level, propagate to entities
- Add specVersion to BaseEventSchema (event level, not eventData)
- Remove specVersion from RunCreatedEventSchema.eventData
- Core runtime sends specVersion on event object
- world-local reads specVersion from event, propagates to run/step/hook entities
- world-postgres reads specVersion from event, propagates to run/step/hook entities
This ensures specVersion flows from client through event to all created entities.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* Update specVersion to be optional in types for backwards compatibility
- specVersion is optional in all entity schemas (runs, steps, hooks, events)
for backwards compatibility with legacy data in storage
- Runtime always sends specVersion on event requests
- world-local and world-postgres provide fallback to SPEC_VERSION_CURRENT
- Test helpers include specVersion in all event creation calls
- EventWithRefsSchema in world-vercel defaults specVersion to 1 for legacy
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* Fix world-vercel queue tests missing VERCEL_DEPLOYMENT_ID setup
Two tests were calling queue.queue() without setting up
VERCEL_DEPLOYMENT_ID, causing them to fail with "No deploymentId
provided" error before reaching the code they were testing.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* Add specVersion to all event creation calls in core package
The server's CreateEventSchemaV2 requires specVersion on all events,
but only run_created was sending it. This caused 400 Bad Request errors
for all other event types (run_started, run_completed, run_failed,
run_cancelled, step_created, step_started, step_completed, step_failed,
step_retrying, hook_created, hook_received, wait_created, wait_completed).
Now all event creation calls include specVersion: SPEC_VERSION_CURRENT.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
---------
Signed-off-by: Peter Wielander <mittgfu@gmail.com>
Co-authored-by: Vercel <vercel[bot]@users.noreply.github.com>
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
Co-authored-by: Peter Wielander <mittgfu@gmail.com>
|
||
|
Peter Wielander
|
4bdd3e5086 | Move auth error messages into @workflow/errors package (#638) | ||
|
Copilot
|
b56aae3fe9 |
Override timeout functions in workflow VM context to throw helpful errors (#505)
* Initial plan * Override timeout functions in workflow VM context to throw helpful errors Co-authored-by: TooTallNate <71256+TooTallNate@users.noreply.github.com> * Use WorkflowRuntimeError instead of vmGlobalThis.Error for timeout functions Co-authored-by: pranaygp <1797812+pranaygp@users.noreply.github.com> * Add docs page for timeout-in-workflow error Co-authored-by: pranaygp <1797812+pranaygp@users.noreply.github.com> * Reorder unavailable functions list: set* functions first, then clear* functions Co-authored-by: TooTallNate <71256+TooTallNate@users.noreply.github.com> --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: TooTallNate <71256+TooTallNate@users.noreply.github.com> Co-authored-by: pranaygp <1797812+pranaygp@users.noreply.github.com> |
||
|
Pranay Prakash
|
00b0bb9346 |
Proper error stack propogating (#280)
* Proper stacktrace propogation in world Proper stacktrace propogation in world * Merge Reconciliation * Standardize the error type in the world spec * Deduplicate vercel world utils * fix undefined type issue --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> |
||
|
Nathan Rajlich
|
7013f29f46 |
Change RetryableError "retryAfter" option number value to represent milliseconds instead of seconds (#240)
Changed `RetryableError` to use milliseconds instead of seconds for numeric `retryAfter` values.
### What changed?
- Modified the `RetryableError` class to interpret numeric `retryAfter` values as milliseconds instead of seconds
- Updated documentation and examples to reflect this change
- Added dependency on `@workflow/utils` package to use the `parseDurationToDate` function
- Updated examples in documentation to show proper usage with:
- Duration strings (e.g., "5m", "30s")
- Millisecond values (e.g., 5000 for 5 seconds)
- Date objects
### How to test?
1. Create a workflow that uses `RetryableError` with different `retryAfter` formats:
```typescript
// With milliseconds
throw new RetryableError("Test retry", { retryAfter: 5000 });
// With duration string
throw new RetryableError("Test retry", { retryAfter: "5m" });
// With Date object
throw new RetryableError("Test retry", { retryAfter: new Date(Date.now() + 10000) });
```
2. Verify that the retry behavior correctly respects the specified durations
### Why make this change?
Using milliseconds as the unit for numeric time values is more consistent with JavaScript conventions (like `setTimeout` and other timing functions). This change makes the API more intuitive for JavaScript developers and aligns with standard practices in the ecosystem.
Signed-off-by: Nathan Rajlich <n@n8.io>
|
||
|
Nathan Rajlich
|
796fafd58d |
Remove isInstanceOf() function and utilize is() method on Error subclasses instead (#148)
|
||
|
Gal Schlezinger
|
4ca9a3edbd |
Introducing Workflow DevKit
build durable, resilient, and observable workflows. Co-authored-by: Nathan Rajlich <n@n8.io> Co-authored-by: Pranay Prakash <pranay.gp@gmail.com> Co-authored-by: Adrian <me@adriandlam.com> Co-authored-by: JJ Kasper <jj@jjsweb.site> Co-authored-by: Vercel Release Bot <88769842+vercel-release-bot@users.noreply.github.com> Co-authored-by: Peter Wielander <mittgfu@gmail.com> Co-authored-by: Hayden Bleasel <hello@haydenbleasel.com> Co-authored-by: Gal Schlezinger <gal@spitfire.co.il> Co-authored-by: Manuel Muñoz Solera <mamuso@mamuso.net> Co-authored-by: Garrett <garrett.tolbert@vercel.com> Co-authored-by: Lars Grammel <lars.grammel@gmail.com> Co-authored-by: Pooya Parsa <pyapar@gmail.com> Co-authored-by: Tom Dale <tom@tomdale.net> Co-authored-by: Vishal Yathish <135551666+visyat@users.noreply.github.com> Co-authored-by: josh <144584931+dancer@users.noreply.github.com> |