A redelivered re-ensure of an already-committed resume (same runId +
resumeId + digest) was rejected with HookNotFoundError when the hook had
since been disposed by the workflow (dispose -> sleep releases the
token while the run continues). The queue consumer treats HookNotFound
as 'nothing left to resume' and acks the delivery — silently dropping
whatever continuation the redelivered message carried and wedging the
run.
Check the (runId, resumeId) claim BEFORE the disposal/existence
rejections: a committed claim whose pinned event is journaled proves
this exact resume was accepted while the hook was alive, so return that
event as success. Claims with a mismatched hookId or payload digest
still fall through to full validation and are rejected as before, as
are genuinely new resumes of a disposed hook.
* feat(core): lazy hook resumption via parallel event write + queue publish (rebased onto #1834 + #3145)
Rebase of #3230 onto current main (267765375 + #1834 resilient resumeHook
+ #3145 event-count-gated replay restart). Reconstructed as a single commit
since `git rebase -i` is unavailable in this environment.
Reconciliation vs the pre-rebase branch:
- Replaces #1834's version-prediction (`supportsQueueHookInput`,
`QUEUE_HOOK_INPUT_MIN_VERSION`) with #3230's capability protocol
(persisted `hookResumeInputVersion` + static `hookResumeDedup`).
- One idempotency protocol: a single `resumeId` + SHA-256 payload digest
per resume, sent to both the direct event write and the queue `hookInput`.
- Two execution tiers: backend+consumer attest dedup -> parallel
`Promise.allSettled(event write, queue publish)`; otherwise plain
sequential (no hookInput/resumeId, event-write errors propagate).
- Consumer re-ensures the `hook_received` event (keyed by resumeId/digest)
after event loading, before replay; skips when already preloaded.
- Preserves #3145: event-count guard, `preconditionReinvocations`,
in-process replay restart, `insertEventByEventId`.
- Removes #1834's resumeId-only test (never released); adds parallel +
consumer-preload + world-local dedup/producer-consumer suites.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(core): read top-level event.resumeId in replay dedup; reconcile unreleased #1834 docs/changeset
- hook.ts: dedup hook_received replay on top-level event.resumeId (the
backend now hoists it to a first-class column), with the legacy nested
eventData.resumeId retained as a deprecated parse-only fallback.
- workflow.test.ts: cover dedup across both top-level and legacy nested forms.
- resume-hook.ts: emit producer recovery telemetry when a transient
event-write failure is swallowed on the parallel path.
- resume-hook.consumer-preload.test.ts: add terminal-run (consume) and
transient-conflict (rethrow/redeliver) re-ensure cases.
- Consolidate the two overlapping changesets into resilient-resume-hook.md
and delete the redundant lazy-hook-resumption.md.
- Docs: return type back to Promise<Hook> (resume-hook.mdx), rewrite the
resilience changelog to the final parallel/deduplicated design, and correct
the WORKFLOW_DISABLE_LAZY_HOOK_RESUME resilience wording.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* docs,core: rename "Resilient hook resume" → "Lazy hook resume" for consistency
- changelog/index.mdx: update the changelog entry title.
- hook.ts: update the dedup comment label to "Lazy-resume dedup".
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* chore: give #3230 its own changeset instead of repurposing #1834's
The lazy-hook-resume work had been folded into #1834's pre-existing
`resilient-resume-hook.md` changeset. Give this PR its own changeset and
delete the superseded #1834 one, whose `resilientResume: true` flag promise
no longer holds (resumeHook() returns plain Promise<Hook>).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* chore: add #3230's own lazy-hook-resumption changeset
Follow-up to 63d877178, which deleted #1834's superseded changeset but did
not stage the replacement. Adds this PR's own changeset.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* chore: tighten lazy-hook-resumption changeset
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* chore: leave #1834's resilient-resume changeset/changelog/docs untouched
Restore #1834's own artifacts that #3230 had rewritten:
- .changeset/resilient-resume-hook.md (restored verbatim)
- docs/.../changelog/resilient-resume.mdx (restored verbatim)
- docs/.../changelog/index.mdx (restored verbatim)
#3230 keeps only its own changeset plus the two docs its code/config genuinely
require: the resumeHook() Promise<Hook> return type (ResumedHook is removed
from the code) and the new WORKFLOW_DISABLE_LAZY_HOOK_RESUME env var.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* Reconcile #1834 ResumedHook contract with #3230 parallel resume
Preserve the resilient-resume contract from #1834 on the parallelized
resumeHook() fast path instead of dropping it:
- Restore the `ResumedHook` type (Hook + optional `resilientResume`) and its
exports (`@workflow/core/runtime`, `workflow/api`); resumeHook/resumeHookImpl
return `Promise<ResumedHook>`.
- Set `resilientResume: true` on the swallow-recover branch (transient direct
write failure + successful queue dispatch), absent on the happy/sequential
paths.
- Restore the producer OTEL convention `workflow.hook.resilient_resume` and the
consumer `workflow.hook.resilient_resume_materialized`, wired where the
consumer re-ensures the event.
- Restore the consumer `occurredAt` derivation from the resume ULID so the
materialized hook_received is dated to resume time, not queue-round-trip time.
- Fix the #3230 changeset's contradictory "Still returns Promise<Hook>" line and
update the resilient-resume changelog + resume-hook API reference to the
shipped parallel/dedup behavior.
- Port the #1834 failure-path coverage into resume-hook.parallel.test.ts
(non-retryable event-write rethrow, both-fail prioritizes the queue error,
resilientResume flag + payload delivery on the recovered path).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* Address review: drop dead nested resumeId fallback, remove server PR link
- Drop the legacy nested `eventData.resumeId` fallback in the hook consumer.
The nested form was only ever written by unreleased preview builds and is
stripped by `EventSchema` parsing (the `hook_received` eventData schema does
not declare it), so the fallback was dead code. Dedup now keys solely off the
top-level `event.resumeId` column. Repoint the replay dedup test to the
surviving top-level path (it previously exercised the nested form only by
building unparsed Event objects in memory).
- Remove the internal workflow-server PR reference from world-vercel's
capability note (the link 404s outside the org); the note keeps the same
information without the dead link.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
* Fix Biome lint violations and add Biome CI check
Biome was not configured to respect .gitignore, so ~92% of the 13,355
reported diagnostics came from gitignored build artifacts. Enable VCS
integration (useIgnoreFile), apply safe auto-fixes across the repo, fix
the remaining mechanical errors by hand, downgrade judgment-call a11y /
dangerouslySetInnerHTML rules to warnings, and add a 'biome ci' job to
the Lint workflow so violations block PRs going forward.
* Use an empty changeset (no behavior change, no release needed)
deleteJSON was the one mutation path in the fs layer that neither used
withWindowsRetry nor swallowed unlink errors. On Windows, unlink fails
with a share-violation EPERM while a concurrent reader briefly holds
the file open — hook polling races deleteAllHooksForRun by design — so
a transient EPERM surfaced as a failed operation, e.g. a failed
run.cancel().
Wrap the unlink in withWindowsRetry, matching the rename/link/unlink
guards the write pipeline already has. ENOENT is not in the retryable
set, so the already-deleted tolerance still short-circuits.
Signed-off-by: Andrew Barba <barba@hey.com>
* feat: publish each run's X25519 public key on the run entity
A cross-run writer needs the recipient run's public key to seal a payload
to it. Derive that key at `start()` and stamp it on the run, so a hook
resumption or a forwarded-stream writer can find it on a run fetch it was
already making instead of spending ~350ms on `run-key`.
The key is derived from the per-run key material `getEncryptionKeyForRun()`
already returns, so nothing about key acquisition changes. It is not
secret: the matching private scalar is never stored anywhere, only
re-derived on demand from the deployment's own env seed. Storing it beside
run metadata therefore does not weaken the run's confidentiality.
**Presence is the writer-side gate for sealed envelopes.** A run only
carries a public key if the runtime that created it could also open one —
which holds by construction, since derivation and `encp` dispatch both
live in `@workflow/core`, so any core that can stamp can also open. Runs
are pinned to their creating deployment, so the capability this attests to
is still accurate at resume time. Writers seal iff the field is set and
otherwise fall back to the symmetric path, which makes version skew
degrade gracefully instead of wedging a run.
The field rides on `run_created`, and is mirrored onto the queued
`runInput` so the resilient-start path (server recreates the run from the
queue message when the `run_created` write failed) doesn't silently
produce a run that can't receive sealed writes.
world-vercel's compile-time wire-contract guard caught the new field
before it could be silently dropped on the v4 path, exactly as designed —
routed into the frame meta block as plaintext metadata.
Also adds browser- and VM-safe base64 helpers to `sealed-box.ts`, since
neither `Buffer` nor `btoa` can be assumed in every context that module
runs in. `base64ToBytes` returns undefined on malformed input rather than
throwing, so a corrupt stored key degrades to "no usable public key" and
falls back to the symmetric path instead of crashing a resumption. Both
are cross-validated against `Buffer` in tests.
* review: fix public-key loss on resilient start and lifecycle updates
Two real bugs found in review, both in the local worlds. Neither surfaces
as an error — a run just silently stops accepting sealed cross-run writes
and falls back to the slow symmetric path forever.
**Resilient start dropped the key.** When a `run_started` arrives for a
run that was never created, world-local and world-postgres rebuild the run
from the queued message. Neither copied `encryptionPublicKey` onto the run
row or the synthetic `run_created` event they write. That is precisely the
scenario this field exists to survive. (The equivalent server-side path was
already handled.)
**world-local also wiped the key on every lifecycle transition.** Its
run_started / run_completed / run_failed / run_cancelled handlers rewrite
the whole run document field-by-field, so any field not explicitly listed
is dropped — meaning the key was lost on the *first* `run_started`, not
just on the resilient path. All four rebuild sites now carry it.
world-postgres is safe here by construction because it issues
column-scoped SQL UPDATEs rather than rewriting the row.
**base64 decoding is now strict.** The decoder accepted shapes that
cannot describe a whole number of bytes (`length % 4 === 1`) and ignored
anything after a mid-string `=`, returning a short array instead of
`undefined`. That is worse than throwing: a corrupt stored key looked
*present*, so callers sealed to garbage rather than taking the symmetric
fallback. Now rejects out-of-alphabet characters, bad lengths, misplaced
padding, and non-zero trailing bits — with a round-trip test over every
length 0–48 to make sure the strictness does not overshoot.
* fix: send encryptionPublicKey in the v4 POST frame meta
`splitEventDataForV4` lifted the run's public key into the frame meta and
`events.ts` spread that meta into `CreateEventV4Input`, but
`buildPostFrameMeta` — which copies meta onto the wire field by field — never
forwarded `encryptionPublicKey`, and the field was missing from
`CreateEventV4Input` entirely. Because the meta is applied with a spread,
TypeScript's excess-property check doesn't fire, so the key was computed, put
in the meta, and then silently dropped before the request was sent.
The server therefore never received the key, never stored it on the run
entity, and every cross-run writer fell back to the symmetric envelope. Every
symptom pointed away from the SDK: a deliberately oversized key was accepted
rather than rejected (the field never arrived), the key was absent from the run
row, and `resumeHook()` always chose `encr`.
Add the field to `CreateEventV4Input`, forward it in `buildPostFrameMeta`, and
cover it for both `run_created` and resilient-start `run_started`. Also add a
generic guard asserting that every field the splitter puts in the meta reaches
the wire, so the next omission in this hand-maintained mapping fails a test
instead of silently degrading encryption.
Enforces the published per-run events limit, which was previously not enforced. The server supplies the limit on the run_started response (separate change); once a run's event log reaches it, the runtime throws MaxEventsExceededError at the top of the replay loop, and the existing terminal-error path records it as run_failed with a new MAX_EVENTS_EXCEEDED code — instead of letting a runaway workflow (e.g. an unbounded step loop) grow the event log without bound.
Adds a new client side WORKFLOW_MAX_EVENTS_OVERRIDE env var which can override the server side provided value (lower only).
The recoverActiveRuns factory option had no environment variable, so
disabling startup re-enqueueing of pending/running runs required a custom
world module via WORKFLOW_TARGET_WORLD. Wire an env fallback
(0/false disables, 1/true enables, explicit factory option wins) and
document it in the worlds configuration reference and local world guide.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* build: declare typescript (catalog:) in every package that runs tsc
Twenty packages invoke tsc in their build/typecheck scripts without
declaring a typescript dependency, resolving whatever tsc pnpm happens
to leave reachable. That broke locally after the TypeScript 6 upgrade
(#2700): base.json now uses the TS6-only 'types': ['*'] wildcard, and
worktrees carrying pre-upgrade node_modules/.bin/tsc shims (orphaned
typescript@5.9.3 bins that pnpm never refreshes for an undeclared
dependency) fail with TS2688 'Cannot find type definition file for *'.
Declaring 'typescript': 'catalog:' (the convention nest already
follows) makes pnpm own each package's tsc bin, so version upgrades
refresh the shims and this staleness class cannot recur. Packages
without tsc in their scripts are left unchanged.
Full pnpm build: 27/27 tasks green.
* Address review: drop duplicate zod devDep; regenerate lockfile minimally
- packages/world listed zod in both dependencies and devDependencies
(pre-existing on main, surfaced by the devDependencies sort) — keep
the runtime dependency only.
- Regenerate pnpm-lock.yaml from a pristine main baseline with
--lockfile-only (a clean-main run produces zero diff, so main has no
drift). Remaining non-typescript changes are mechanical consequences
of the change itself: typescript is an (optional) peer of several
tooling dependencies, so declaring it in 20 importers creates new
peer-resolution snapshot variants and prunes the now-orphaned old
ones; plus one radix-ui 1.6.1->1.6.2 refresh in docs caused by its
floating 'latest' specifier.
- Validated: pnpm install --frozen-lockfile succeeds; full build 27/27.
* perf(world-local): index hook lookups instead of scanning the global event log
Hook creation, hook cache rebuilds, and token lookups previously read
and parsed every event file ever written across all runs, and run
termination read every live hook entity — making every turn of a
long-lived local session slower as history accumulated. Maintain
durable per-token / per-hookId indexes and per-run hook markers
(with a one-time backfill for pre-index data dirs), resolve tokens
through the claim file, read directory listings concurrently, reuse
compiled filename regexes, and raise the recovery page size.
* chore: trim development comments
* test(world-local): make hook-index perf guard machine-speed independent
The absolute 2s wall-clock bound timed out on Windows CI, where
filesystem operations are ~50x slower. Compare hook-creation cost
before vs. after seeding foreign event history instead.
* review: keep world package untouched, batch backfill reads
- Revert the recovery page-size change in @workflow/world; default
world-local runs.list to a 200-item page instead
- Read backfill event/hook files with bounded concurrency (32),
matching paginatedFileSystemQuery, to cut one-time migration cost
on large legacy data dirs (slow-fs platforms especially)
* Statically inject workflow world target
* Fix static world injection in host bundles
* Fix static world injection gaps
* Fix Vite Nitro server startup
* Fix Nitro pg-native aliasing
* Fix static world target CI gaps
* Fix static world dev rebuild gaps
* Avoid broad runtime alias in Nitro
* Refresh Next dev route for step HMR
* Externalize Nest target world
* Use canary HMR rediscovery timeout
* Bundle local world in Nest builds
* Dedupe world target helpers and fix SvelteKit chunk patch guard
Nathan Colosimo