The Astro and SvelteKit webhook wrappers copied the incoming request via
`normalizeRequest()`, which buffers the whole body with `arrayBuffer()`,
before calling the handler that validates the webhook token. Requests
carrying an unknown token therefore did unnecessary work before being
rejected.
The copy turns out to be unnecessary: both frameworks already hand the
route a standard `Request`, so the webhook wrappers now pass it straight
to the handler. The body is left untouched until `resumeWebhook()` has
accepted the token.
The flow route keeps `normalizeRequest()` for now — it authenticates via
the queue trigger rather than a URL token, so the ordering does not
matter there, and whether the shim is needed at all is a separate
question.
Note that the Astro dev server buffers request bodies upstream of the
route handler, so the new behavior is only observable in built output;
the node adapter and Vercel builds both benefit.
Signed-off-by: Nathan Rajlich <n@n8.io>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Audited every vercel.com link in docs/content (20 unique URLs,
HTTP-validated including anchor fragments):
- project-configuration#regions (2x): the #regions anchor no longer
exists on that page — content moved to the vercel-json subpage; now
links project-configuration/vercel-json#regions
- gateway/api-reference/overview (2x): hard 404; the AI Gateway docs
restructured — 'get an API key' context now points at
ai-gateway/authentication
- observability/otel-overview (1x): redirects to
tracing/instrumentation; link the final URL
- docs/workflow and docs/workflow/python (8x): redirect to the plural
docs/workflows paths; link the final URLs (also drops a redundant
?language=py param that the redirect discards)
All other links (queues, queues/pricing + anchors, plans/hobby,
limits, regions, sandbox, workflows/pricing#storage-retention,
cli/project-linking, audit-log, home, help, blog) verified 200 with
live anchors.
Signed-off-by: Nathan Rajlich <n@n8.io>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
* fix(nitro): use workspaceDir for monorepos
* test: stabilize Next canary HMR e2e
Signed-off-by: Nathan Colosimo <110621881+NathanColosimo@users.noreply.github.com>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
* Propagate trace context to vercel-workflow.com in workbench instrumentation
@vercel/otel only propagates W3C trace context to Vercel deployment URLs
by default, so outgoing requests to the workflow-server
(vercel-workflow.com) got a client span with no `traceparent` header —
breaking the APM trace link to workflow-server's spans. Add
`instrumentationConfig.fetch.propagateContextUrls` for the workflow-server
domain in every workbench that uses @vercel/otel: example,
nextjs-turbopack, nextjs-webpack, and sveltekit. The Next.js and SvelteKit
apps already declared @vercel/otel but weren't registering it at all; they
now do.
* Also propagate trace context to the Vercel Queue Service (vercel-queue.com)
The workflow-server queue path (@vercel/queue) sends to regional
vercel-queue.com subdomains (e.g. iad1.vercel-queue.com) when not using the
queues proxy, which were missing a `traceparent` header for the same reason
as vercel-workflow.com. Add `/vercel-queue\.com/` to propagateContextUrls in
all four workbench instrumentation configs.
---------
Signed-off-by: Pranay Prakash <pranay.gp@gmail.com>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Co-authored-by: Peter Wielander <mittgfu@gmail.com>
* Redrive on transient workflow-server transport failures instead of failing the run
A firewall in front of workflow-server shedding load with sustained 429/503
makes undici's shared RetryAgent exhaust its retries and throw
UND_ERR_REQ_RETRY. That raw error was rethrown unwrapped, so it was
classified as USER_ERROR and the replay terminal branch wrote run_failed —
permanently failing a run on a transient blip (or, in an outage, falling back
to the ~5min queue visibility-timeout redrive).
- world-vercel: map exhausted-retry / socket / connect / DNS / timeout
failures to a typed WorkflowWorldError (code TRANSPORT/TIMEOUT) by walking
the fetch() cause chain.
- core: add isRetryableWorldError (429 / 5xx / TRANSPORT / TIMEOUT) and
rethrow such errors from the replay terminal branch so the queue redrives
quickly (1s->60s backoff) instead of failing the run. Reuse it in start()
and step_started handling.
- world-vercel: surface the Vercel firewall x-vercel-mitigated
(challenge/deny) header alongside x-vercel-id in error diagnostics and logs.
* Address review: back off + cap on every retry path; fix mock; refine scope
Revises the transport-error handling per PR review (VaguelySerious,
karthikscale3).
Blocking fix — step_started no longer self-enqueues a throttled defer for
transient world errors. Returning `{ type: 'throttled', timeoutSeconds: 1 }`
acked the delivery and enqueued a fresh message, resetting the delivery count
so the path never backed off and never reached MAX_QUEUE_DELIVERIES — an
unbounded flat-1s loop if step_started kept failing. It now throws, so the
error flows through the replay loop's retryable-world-error rethrow and earns
both the delivery-count backoff and the max-delivery cap. Throwing is safe on
step_started (the body hasn't run; a write that landed dedupes to skipped).
Also in this revision:
- Backoff that lasts: raise the queue handler-error retry ceiling 60s -> 900s.
VQS clamps each redelivery to its 900s SQS limit and adds its own post-32
exponential, so ramping our base toward 900s stretches survival from ~3.7h to
most of the 24h message-visibility window. Corrected the stale
MAX_QUEUE_DELIVERIES comment to match the real VQS schedule.
- Stop amplifying firewall challenges: the undici RetryAgent no longer retries
429 in-process (a challenge is a 429 the client can't solve). 429s surface
immediately as ThrottleError carrying x-vercel-mitigated / x-vercel-id, so
the diagnostic header now reaches us for the challenge case too.
- Track world faults as WORLD_CONTRACT_ERROR (not USER_ERROR) in
classifyRunError so an outage isn't attributed to user code.
- Fix queue.test.ts mock that `biome check --write` had rewritten from a
newable `function` into an arrow (broke `new QueueClient`); pin with a
biome-ignore.
All Vercel-specific logic stays in @workflow/world-vercel; @workflow/core
operates only on the generic WorkflowWorldError abstraction.
* Update .changeset/transport-error-redrive.md
* Trim changeset to a single sentence per review
* Route firewall-challenge 429s to the retryable transport path, not ThrottleError
A 429 carrying `x-vercel-mitigated: challenge` is a firewall challenge our
server-to-server client cannot solve, so it recurs for the life of the
incident. Mapping it to `ThrottleError` meant the `step_started` write deferred
it as `{ type: 'throttled' }`, which self-enqueues a FRESH queue message and
resets the delivery count — so it never backed off past `retryAfter` and never
reached `MAX_QUEUE_DELIVERIES`, hot-looping against an already-overloaded
firewall (the exact amplification this PR set out to remove, and contrary to
the "step_started can't loop unbounded" invariant, which only held for 5xx).
Map a challenge to a retryable transport `WorkflowWorldError` (`code:
'TRANSPORT'`) in both the v3 `makeRequest` and v4 `throwForErrorResponse`
(the hot event-write path) error mappings, via a shared `isFirewallChallenge429`
helper. It then propagates through the V1/V2 step paths and the replay loop's
retryable-world-error rethrow, earning the delivery-count backoff AND the
delivery cap. A genuine application-level 429 (no `challenge` mitigation) stays
a `ThrottleError` and keeps its `Retry-After`-paced defer.
Also correct the survival-window comments: with the 900s ceiling,
MAX_QUEUE_DELIVERIES=48 spans ~9-10h (~35,000s), not "the better part of 24h";
reaching 24h would need a higher delivery cap, not a higher per-hop ceiling.
---------
Signed-off-by: Peter Wielander <mittgfu@gmail.com>
Signed-off-by: Pranay Prakash <pranay.gp@gmail.com>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Co-authored-by: Peter Wielander <mittgfu@gmail.com>
Co-authored-by: Peter Wielander <peter.wielander@vercel.com>
github-actions[bot]