mirror of
https://github.com/vercel/workflow.git
synced 2026-09-14 19:59:43 +08:00
Peter Wielander
51489b1fce
A pending step's queue dispatch is keyed by its correlation ID, and a queue dedupes that key for the lifetime of the message sent under it. So once a dispatch stops making progress, every later replay's re-send is absorbed and the run replays forever with one step nothing will finish. The watchdog already covered a step that was never delivered. It did not cover the other shape: the message is delivered, the step writes step_started, and the invocation running the body disappears before writing a terminal event. Inline ownership arms a backstop wake at the lease for exactly that, but the re-dispatch that wake triggers carried the bare correlation ID the queue had already claimed, so the recovery was deduped away and the run went silent permanently. Measured on the race repro: 432 replays in 57s, the last one 0.6s past the lease boundary, then nothing for the remaining 25 minutes of the run. Both shapes now share one deadline, dispatchLostAtMs: a watchdog interval after step_created for an unstarted step, the end of the ownership lease for a started one. Past it the key is epoch-scoped and a boundary wake is armed, and the epoch advances once per watchdog interval so a lost recovery is itself retried. Anchoring the started case on the lease rather than on a watchdog interval is what keeps healthy long-running bodies from being duplicated. A step in step_retrying stays out of scope: its retry is queued under the bare key with a backoff that can legitimately exceed either deadline. The repro harness called a run stuck after 4 minutes, well inside the runtime's own longest recovery deadline, so a run on its way back was reported as permanently stranded. Its run timeout is now derived from the lease plus a watchdog interval instead of being a second copy of a number the runtime owns.