Conflicts were all in files main also touched:
- packages/world-local/src/streamer.ts: kept main's `globalSingleton` ULID
factory alongside this branch's `StreamCursorPositionSchema` value import,
and kept the direct-index page loop (main's side was the `dataIndex` skip
loop this branch replaced).
- packages/world-vercel/src/streamer.ts: comment only; kept this branch's
text, which records that chunk snapshots now read v3.
- docs/content/worlds/v{4,5}/building-a-world.mdx: kept main's rewording and
re-applied this branch's cursor contract sentences.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* Revert "[world] Make the sealed log opt-in instead of default-on (#3735)"
Reverts b2cac623d3. New runs are stamped at spec 7 again, now that a
read which cannot see past an unfilled position waits for it instead of
reporting a log that ends there (workflow-server: derive the in-request
seal poll budget from the staleness bound).
Two things are kept from #3735 rather than reverted:
- the world-testing conformance floor at mintedSpecVersion(), which was
wrong for any staged bump and not specific to this default
- a note on mintedSpecVersion recording what default-on rests on: the
events density requirement, and that a sealed log meets it by repair
rather than by construction, so the READ has to wait
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* TEMPORARY: point world-vercel at workflow-server#839 preview
Validating the seal-poll-budget fix end to end with spec 7 on. Reverted
before merge; the override lint guard is expected to fail meanwhile.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* Revert "TEMPORARY: point world-vercel at workflow-server#839 preview"
This reverts commit 5e17cc9335.
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
* Default the events transport to WebSockets
WORKFLOW_EVENTS_TRANSPORT=http is the opt-out. Only that exact value
disables it, so a typo'd or empty value fails toward the default rather
than quietly pinning a deployment to HTTP.
The prerequisite the gate named for defaulting on is met:
postEventFrameOverWs opens a client span per frame. What is still missing
is Vercel's outgoing-requests view, which reads instrumented fetch calls
rather than spans and so cannot show a transport that issues no request.
Co-Authored-By: opencode <opencode@vercel.com>
Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com>
* docs: WORKFLOW_EVENTS_TRANSPORT defaults to ws
Three places still documented http as the default. Each now states the
opt-out is the exact value http, rather than leaving 'default: ws' to
imply that anything non-ws disables it — the asymmetry is deliberate in
the code and is the part a reader would otherwise get wrong.
Also drops 'Experimental' from the Vercel World page: a setting that is
on for everyone by default is not opt-in experimental, whatever else it
is.
Co-Authored-By: opencode <opencode@vercel.com>
Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com>
* Fix the gate's own unit tests for the flipped default
Five tests in ws-transport.test.ts still encoded the opt-in semantics.
Three were the isWsEventsTransportEnabled table itself; the other two
(openWsChannel 'does nothing when the gate is off', and the channel
release equivalent) relied on the suite's ambient unset environment
meaning 'off', which it no longer does. Both now set http explicitly.
Two tests in ws-transport-spans.test.ts asserted HTTP-side span
behaviour the same way. The write one would have kept passing by
falling through resolveWsTransport's null rather than because the gate
was off - passing for the wrong reason, which is what this file exists
to catch.
Also makes the opt-out case-insensitive and trimmed. The gate is
deliberately asymmetric - unrecognized values take the default - but
that asymmetry should not extend to swallowing HTTP or ' http '.
Whoever reaches for the escape hatch is plausibly mid-incident, and
silently ignoring their opt-out over a capital letter is the same class
of silent-wrong-transport bug this flip is meant to stop shipping.
554 tests pass in packages/world-vercel.
Co-Authored-By: opencode <opencode@vercel.com>
Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com>
* ci: add a required forced-HTTP e2e lane (#3703)
Flipping the default makes e2e-vercel-prod a WebSocket lane: it sets no
WORKFLOW_EVENTS_TRANSPORT, and unset now means ws. Nothing in the file
would exercise the HTTP events transport against a real deployment any
more, so this is not additive coverage — it replaces coverage the flip
silently removed.
Unconditional and required rather than label-gated like the WS lane.
HTTP is now the fallback, and the fallback is silent: resolveWsTransport
returning null costs a write nothing and logs nothing, which is the
shape of the durabench bug this stack came out of.
Two apps rather than the WS lane's four, since every row is a real
vercel deploy charged to every PR. nextjs-turbopack is the only fixture
emitting OTEL spans, so it is the one that can show which transport
actually ran; express covers the non-Next server path.
Also corrects the WS lane's docblock, which claimed every other job
exercises HTTP only. That stopped being true one commit ago.
Co-authored-by: vercel[bot] <35613825+vercel[bot]@users.noreply.github.com>
Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com>
* Fail loudly when step_completed falls back to HTTP under a strict flag
The WS e2e lane asserts that the transport is harmless, not that it is
used: an event written over HTTP produces the same run outcome as one
written over the socket, so the lane stayed green through the entire
period the transport was silently demoted.
WORKFLOW_INTERNAL_EVENTS_TRANSPORT_STRICT turns that one case into a
failed run, and the WS lane now sets it.
Scoped to step_completed alone, because most fallback is legitimate:
run_created is written outside any invocation that opens a channel;
run_started routinely lands before the channel is registered (34% HTTP
on a healthy deployment); step_created and wait_created mostly fold into
events.createBatch, which is not wired to the socket; and a write after
the invocation released its claim falls back by design. step_completed
is issued after a step body has run, and was 100% ws across every
WS-enabled deployment measured on two SDK versions.
The flag reads as off unless the value is exactly 1 or true - the
opposite asymmetry from the transport gate, which treats an unrecognized
value as on. That gate risks a deployment sitting quietly on the wrong
transport; this one fails runs, and should not be acquired by a typo.
Co-Authored-By: opencode <opencode@vercel.com>
Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com>
* ci: run the WS transport lane on every PR
It was opt-in behind ws-transport-test because four real vercel deploys
were too much to charge an unrelated PR for a transport that was off by
default. Flipping the default expires that reasoning from both ends: the
cost is no longer for someone else's feature, and this is now the only
lane that asserts the socket carried the events. e2e-vercel-prod
inherits the new default but checks nothing, so behind a label the
average PR would move every deployment onto WebSockets with nothing
verifying they were used.
Drops WS_REQUIRED from the gate along with it. That existed only to let
the lane be legitimately skipped on an unlabelled PR; with no label the
lane is required unconditionally, like e2e-vercel-prod and the HTTP
lane, and the skipped case is now a failure rather than a warning.
Gate script extracted and run against the cases that matter: ws skipped
fails on a standard PR, ws skipped fails under workflow-server-test, and
all-green passes.
Co-Authored-By: opencode <opencode@vercel.com>
Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com>
* ci: widen the HTTP transport lane to six server shapes
Before the flip, HTTP was the default and all 28 e2e-vercel-prod
lane-runs covered it. After the flip they cover WebSockets instead, and
this lane is the entirety of the HTTP coverage - two apps was too thin
for a transport that is still supported.
Six, not the full 14, because every row is a real vercel deploy charged
to every PR. Chosen by server shape rather than count: example
(baseline), nextjs-turbopack (Next, and the only fixture emitting OTEL
spans), vite (Vite SSR), express (Node req/res), nitro (h3, also covers
nuxt) and hono (fetch-API Request/Response, a different mount shape from
express). The rest duplicate a shape already covered; python is left out
because it has no conformance gate and needs routes this suite does not
serve.
The first four match the WS lane's matrix on purpose, so the same
fixture runs on both transports and a failure on one can be read against
the other.
Project ids and slugs are copied from e2e-vercel-prod and verified equal
to it; both lanes already use the same team and token.
Co-Authored-By: opencode <opencode@vercel.com>
Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com>
---------
Co-authored-by: vercel[bot] <35613825+vercel[bot]@users.noreply.github.com>
* fix(world-vercel,world-local): hold process-wide state on globalThis
Both packages are bundled into the host application's server build, and a
bundler keys module identity on (resource, layer) — Next.js alone builds
`instrument`, app-route, `ssr` and `edge` layers, so one process holds one
copy of each of these modules per layer. Every module-scope `const`/`let` in
them was therefore per-copy state wearing the costume of a process singleton.
vercel/workflow#3493 made `@workflow/world-vercel` bundled rather than
external and the events WebSocket transport regressed to HTTP for exactly
this reason: the queue consumer registered its channel in the `instrument`
copy's `Map` and the write path looked it up in the route copy's empty one. A
deterministic miss, for the life of the process. `@workflow/world-local` had
the same exposure all along — including `runFileLocks`, where a duplicated
mutex simply stops mutually excluding.
Add `globalSingleton()` to `@workflow/utils` (the primitive `@workflow/core`
already hand-rolls for its World cache) and route every mutable module-scope
binding in both worlds through it.
Regression cover, in three layers:
- `global-singleton.test.ts` pins the primitive's semantics.
- `ws-transport-module-copies.test.ts` imports the module twice in one
process and asserts a transport registered by one copy is found by the
other — it fails on a plain module-scope `Map`, which is the shipped bug.
- `scripts/lint/module-scope-state.mjs` fails the class: an AST rule banning
mutable module-scope state in these packages, with `// per-copy-ok: <why>`
as the deliberate escape. Wired into both packages' `vitest run src`, with
fixture self-tests so it cannot rot into a no-op.
* test(world-postgres): pin the module-scope-state rule for the postgres world
It is deduped today only because `getRuntimeRequire()` loads it — a property
of how it is loaded, not how it is written, and exactly what changed for
world-vercel in #3493. The package is already clean; this keeps it that way.
* docs(worlds): codify "a world must not hold mutable module state"
A world package is loaded one of two ways, and only one of them gives it a
single module instance: a runtime `require()` (deduped by Node) or the host's
bundler (one copy per layer). Which one you get is a property of how the world
is loaded, not of how it is written, and it changed under `world-vercel` in
#3493 — so the rule has to be "never rely on module scope", not "rely on it
until someone flips a config".
Written down in the four places someone can meet it:
- `docs/content/worlds/{v4,v5}/building-a-world.mdx` — a "Process-wide state"
section for custom-world authors, with the loading modes spelled out and a
nudge to prefer World-instance state over a global.
- `packages/world/README.md` — the same constraint on the contract package.
- `CLAUDE.md` — so the next contributor working in these packages sees it.
- `packages/core/src/runtime/world.ts` — at the two static imports, which is
where the difference between a bundled world and a required one originates.
The rule's own error message now teaches it too, rather than naming a helper.
Consolidates the guard while here: `@workflow/utils` owns the rule and its
fixture self-tests, and sweeps every *published* `packages/world-*` discovered
at runtime, so a world package added later is covered without anyone
remembering. Each world keeps a one-assertion mirror for locality.
* style: drop prose em dashes from this branch's new text
#3704 landed a repo-wide writing pass hours after this branch was written and
took `world-vercel/src` from 406 em dashes to 130 (`ws-transport.ts` alone went
35 to 1). This branch's docs section, README, comments and lint messages were
written before that and would have put 36 of them straight back into the files
that were just cleaned.
Rewritten sentence by sentence rather than by substitution: an em dash becomes a
colon, a comma, a full stop or a parenthetical depending on what it was doing.
Also fixes a real defect the sweep surfaced: `world-postgres`'s guard test was
generated through a shell heredoc and had literal backslash-backticks in its
doc comment.
* Update .changeset/world-module-scope-state.md
Co-authored-by: Peter Wielander <mittgfu@gmail.com>
Signed-off-by: Pranay Prakash <pranay.gp@gmail.com>
* fix(core): build the entrypoint's queue handler from getWorld()
Adopted from #3666 by @MintedKenny, which implements #3665 and could not run
CI as a fork PR. One line of behavior: `workflowEntrypoint`'s lazy handler
init calls `getWorld()` rather than `getWorldHandlers()`.
`getWorldHandlers()` owns a second, build-time-safe cache, so calling it from
the runtime route built a *second* World in the same process. That costs a
stateful World duplicate resources on every instance — world-postgres eagerly
constructs a `pg.Pool` (default `max: 10`) and a nested world-local World in
`createWorld()`, so self-hosted users have been paying for two of each — and,
for a bundled world package, the two Worlds are built by two different module
copies, which is the mechanism behind the WS transport regression the rest of
this branch contains.
The public `getWorldHandlers()` and its separate build-time cache are
unchanged; only the runtime route stops using it.
Kept from the original: the regression test asserting the factory runs exactly
once, and the api-reference wording (re-applied over #3704's list punctuation).
Not taken: renaming the `workflow.route.get_world_handlers` span. It is a
distinct span from the per-request `workflow.route.get_world` at the top of the
flow route, and reusing that name would collide with it in traces and in
`runtime-trace-mode.test.ts`; a comment records why the name outlived the call.
Co-authored-by: Kenneth <kenneth@standardforensics.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* fix: address AI review on the module-scope work
Two blocking findings, both real:
- **Cross-version state sharing** (`ws-transport.ts`). A process can hold two
*published versions* of `@workflow/world-vercel` (a transitive dependency
pinning an older `@workflow/core`, which depends on this package by exact
version). Both wrote to the same unversioned `Symbol.for` key, so one
version's write path could be handed a `WsEventsTransport` built by the
other's class and frame against a protocol it may not share — with no version
negotiation on the socket to catch it. `shapeVersion` cannot express this: the
container is stable, the hazard is its contents. The registry and the events
dispatcher recycler are now keyed by package version. The plain connection
pools stay unversioned; sharing those across copies is the point.
- **The documented pattern failed the rule this PR adds.** The custom-world docs
teach `store[StateKey] ??= …`, which the rule flagged as a field write. It now
recognizes state rooted at `globalThis`, following one alias hop, which is
also what `core/private.ts:23` and `next/src/index.ts:58` are already doing
correctly (core drops 26 findings to 22, next 7 to 6). The docs also now say
outright that `globalSingleton()` is the same thing, since AGENTS.md
prescribes it and the page did not mention it.
Rule precision, from the review's probes:
- `.mts`/`.cts` are scanned. `@workflow/world-testing` is authored in `.mts`, so
its entry in the sweep was passing vacuously — with the walk fixed it reports
a real finding, now annotated (it is a standalone `serve()` entry).
- Mutations in top-level statements no longer count. A table filled at module
evaluation is identical in every copy; divergence needs a later write.
- `static` class fields are collected, attributed to the class name.
- An *exported* binding initialized to an empty collection is a finding on its
own, which approximates the cross-file case the walk cannot resolve.
Six fixtures pin the new behavior. The rule's header now states what it does not
see, and AGENTS.md states where the sweep stops and why core is not gated yet.
Also tags `resetGlobalSingletonForTest` `@internal`.
* fix(lint): attribute a static-field write to the field, not the class
The static-field support added in the previous commit keyed `declared` on the
class name, so a class carrying more than one mutable static reported one
finding instead of one per field, and labelled the survivor with whichever
mutation was seen first. On a two-static fixture it reported
`static Registry.latch (`.set()`)`: the name of one field, the reason
belonging to the other, pointing the reader at the wrong line.
Key static fields `Class.field` and resolve a write to the same shape, via a
new `memberPath()` that takes the first two segments of a member chain and
tries that key before the bare root identifier. Two follow-ons fall out of
having the path:
- `this.field` inside a `static` member resolves to the class, which is the
ordinary way to write the mutation. `staticClassOf()` returns nothing for an
instance member, where `this` is an instance and the state is per-instance
rather than per-copy, and nothing inside a nested `function`, which rebinds
`this`.
- `state.count++` is now a finding, like the `state.count += 1` that
`assignment()` already reported.
Fixtures pin all four, including the instance-field case that must stay clean.
The four world packages still report zero, and the extracted `recordMutation()`
keeps the file at its previous two Biome complexity warnings.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* fix: make module duplication inert across every bundled package
`@workflow/core` is bundled into the host server build the same way the worlds
are, and always has been — the original repro measured three live copies in
every arm, including the pre-#3493 external one. One instance is not reachable:
layers cannot share a module, and core cannot be external because it *is*
workflow code (`runtime/start.ts:253` and nine methods in `runtime/run.ts` are
`'use step'`), so it must go through the SWC loader. The Next integration
already encodes that rule by removing workflow-bearing packages from
`serverExternalPackages`.
So the duplication stays and the hazard is removed instead, everywhere the
duplication can happen.
`@workflow/core` (22 findings to 0): warn-once latches in `constants.ts`,
`start.ts` and `telemetry.ts`; the source-map tracer cache; the VM script cache;
the QuickJS compiled-assets and baseline caches; the dev-server port cache (its
own comment already said "per process"); the text codecs; the zstd browser
decoder; and the `useStep` closure brand, where a function marked by one copy
was invisible to another.
The one with teeth was `step-single-flight.ts`: a per-copy map is not
single-flight. Two invocations reaching it through different layers would each
believe they were alone in the process and both run the step body, silently
degrading in-process dedup to the cross-process residual its own doc scopes out
to the ownership lease.
Also `@workflow/world` (a warn-once set, hand-rolled onto `globalThis` to keep
that package dependency-free), `@workflow/ai` (the lazy OTel API), and
`@workflow/nest` (bootstrap config in a module-level `let` and two static class
fields — configure one copy, read another, and the controller is unconfigured
for the life of the process).
Five sites are deliberately per-copy and now say why: state keyed on objects
that never cross copies (the barrier safety-net `WeakSet`, the QuickJS pending
byte `WeakMap`), the synchronously-scoped guest-code sink, and the OTel
diagnostic that reports what *this* copy sees.
The sweep now covers all of it. Packages with a single module graph stay out
(build-time code, the CLI, the o11y UI, the test runner) and AGENTS.md records
which and why.
Found while doing this: two static fields on one class collapsed into a single
entry in the rule, so `WorkflowModule.options` was invisible behind
`WorkflowModule.outDir`. Statics are now keyed `Class.field`.
* fix(world): suppress noAssignInExpressions on the globalThis idiom
The hand-rolled form trips Biome, as it does in `packages/core/src/private.ts`,
which carries the same suppression. Restructuring it into a helper function
instead would hide the state behind a call the module-scope rule cannot follow,
so the binding would stop being recognized as off-module and the package would
report a finding for correct code.
* fix: sweep every bundled package, and mark utils side-effect free
@shalabhc asked on review whether `@workflow/utils` needs this too. It does,
and so do three others: `utils`, `errors`, `serde` and `workflow` all end up in
the host application's server build and none were in the sweep. All four report
zero today, which is exactly the state `world-testing` appeared to be in before
the `.mts` walk was fixed and it turned out to have a real finding. Being clean
and being *checked* are different properties, and only the second one survives
the next contributor.
`sideEffects: false` on `@workflow/utils`: verified that every module in the
package only declares (no import-time work), so a bundler can now drop the
unused parts of the barrel instead of keeping all ~64 KB of it because three
packages import one 476-byte function.
---------
Signed-off-by: Pranay Prakash <pranay.gp@gmail.com>
Co-authored-by: Peter Wielander <mittgfu@gmail.com>
Co-authored-by: Kenneth <kenneth@standardforensics.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Peter Wielander <peter.wielander@vercel.com>
* Run the test suites CI was silently skipping
`turbo test` runs a package's tests only if that package declares a `test`
script, so a suite can sit in the repo for months without ever running. Four
were in that state: @workflow/world (13 files, 160 tests), @workflow/cli (5 /
51), @workflow/nitro (1 / 30), and two files under packages/core/e2e that no
workflow named.
Wire each one up, and add scripts/check-test-suites-wired.mjs plus a lint job
so the next unwired suite fails CI instead of going unnoticed.
Fixes#3731
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* Drop the changesets and rename the guard job
The PR only wires up existing suites and adds a CI check, so there is nothing
to release. Rename the job to match its `no-test-overrides` sibling.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
* docs: apply Vercel technical writing standards
Audit the complete documentation corpus, package READMEs, skills, and
source TSDoc/comments against the vercel-technical-writing skill and
style-rules.md. Normalize sentence-case headings without changing
published anchors, remove prose em dashes and filler wording, improve
active voice and self-contained phrasing, standardize product/brand
capitalization, American English, list punctuation, units, and code
fence languages, and preserve exact runtime strings/table placeholders.
All executable code is unchanged. Modified skills have their metadata
versions bumped.
* docs: extend writing audit to repository Markdown
Apply the same technical-writing rules to design documents, compiler
specifications, workbench guides, package changelogs, and the remaining
tracked Markdown outside the deployed docs corpus. Preserve historical
meaning, commands, output literals, table placeholders, and heading
anchors.
* docs: exclude generated package changelogs from audit
* fix(web-shared): use UTC tooltips for trace viewer markers
Replace the relative-time context card on hook and attribute ticks with the
shared tooltip, labeled `Hook received [UTC] …` and `Attribute set [UTC] …`.
Signed-off-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: Mitul Shah <mitulxshah@gmail.com>
* fix(web-shared): format marker helpers and drop invalid aria-label
Biome requires the kind filter ternary on one line, and aria-label is not
valid on the tooltip trigger span.
Signed-off-by: Cursor Agent <cursoragent@cursor.com>
* fix(web-shared): prefix marker context cards with event kind
Restore the relative-time context card on hook and attribute ticks, and
prefix the relative time with "Hook received" or "Attribute set".
Signed-off-by: Cursor Agent <cursoragent@cursor.com>
* refactor(web-shared): slim marker context-card prefix
Drop the label helper, type guard, and unused DefaultTimeText prefix path.
Keep kind on the marker, map copy at the tick, and prefix only the card heading.
Signed-off-by: Cursor Agent <cursoragent@cursor.com>
* refactor(web-shared): drop prefix comments and generic event-mark helper
Leave sortedEventMarks as a plain string filter. The prefix prop does not
need a JSDoc restatement.
Signed-off-by: Cursor Agent <cursoragent@cursor.com>
---------
Signed-off-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Instead of accessing private files from SvelteKit, we use @sveltejs/load-config to load the Svelte config (that package also knows about checking Vite config). The deadlock is avoided by having a module-level Set to see if we're currently recursing or not.
This is necessary for SvelteKit 3 since there the config lives exclusively in the vite config, and the previous logic did not handle that.
This also uncovered that we're needlessly rebuilding the generated files in sub builds/workers (SvelteKit, at least below 3, starts off secondary builds; and some things are done in workers), which a new file cache now checks.
Picked from stable branch PR #3474
Signed-off-by: Simon Holthausen <simon.holthausen@vercel.com>
Co-authored-by: Peter Wielander <mittgfu@gmail.com>
* fix(core): make step-argument serialization failures catchable in workflow code
A step whose arguments fail to serialize is now finalized by the
suspension handler as step_created + step_failed (mirroring a step-body
failure) instead of rejecting the whole suspension. The next replay —
forced in-process, since no step message is dispatched for the failed
step — rejects the step's promise with the SerializationError, so a
try/catch around the step call observes it. Uncaught, the error
propagates out of the workflow body and fails the run as a fatal
USER_ERROR immediately, instead of redelivering the orchestrator
message until max deliveries (49/48) as reported in production on v4.
* Serialize the step_failed error with the VM global; one-sentence changeset
Addresses review feedback: dehydrateStepError in
finalizeUnserializableStep now receives suspension.globalThis like every
other dehydration in this file. Error detection is realm-independent, so
the host-created SerializationError serializes identically, but VM-realm
values guest code threw into the cause chain are now detected by the
realm-sensitive reducers.
* Address review: QuickJS engine support, deferred-batch join, drain gate, placeholder marker, telemetry, docs
- QuickJS: dumpPendingOps now catches a step input's serialization
failure per-op, reframes it as a SerializationError with the same
framed message as dehydrateStepArguments, and surfaces it on the
pending op instead of failing the whole collection. The entrypoint's
dispatchPendingOps finalizes such steps as step_created (placeholder
input) + step_failed, excludes them from inline claims and queue
publishes, marks them handled, and raises the requeue signal so the
failure is observed even when the feed lags — mirroring the node:vm
engine, so both engines agree: catchable in workflow code, USER_ERROR
with the framed message when uncaught. Both step-argument e2e tests
now pass on WORKFLOW_VM=quickjs.
- runtime.ts: the failed-step replay path now joins
suspensionResult.deferredBatchWork before continuing, so a trailing
chunk commit or step-message publish rejection propagates instead of
being swallowed after ack; committed inline claims are documented as
deliberately handed to owned recovery.
- Terminal drain: finalization is gated on a stepDispatch target. The
drain caller has no replay to observe a finalization, so a completed
run no longer gains failed-step rows for an unawaited unserializable
step — the rethrown error is swallowed by the drain's catch,
preserving its pre-existing behavior.
- The placeholder input now carries a marker string ('[input
unavailable: step argument serialization failed]', shared via
runtime/unserializable-step.ts) so inspect/o11y don't render the
failed step as a genuine zero-argument call.
- New workflow.steps.failed_serialization span attribute on the
suspension span, so occurrence is measurable without log search.
- Docs: v5 serialization-failed error page documents where each
boundary's failure surfaces (catchable step failure vs run failure)
and the no-retry USER_ERROR semantics; foundations/errors-and-retries
gains a Serialization Failures section with the try/catch shape.
* Guard the finalization crash window; self-contained docs samples
- A crash or transient failure between finalization's two durable
writes leaves a lone placeholder step_created, and redelivery then
dispatches the step through normal crash recovery — previously
running user code with the placeholder arguments. The placeholder
now carries a structural flag on the input triple's top level (which
user code never controls, so no false positives), and the step
executor checks it after hydration: instead of running the body, it
throws the intended fatal SerializationError, completing the
interrupted finalization as step_failed. Applies to both engines
(they share the placeholder and the executor).
- Regression tests: executor fails a placeholder-input step without
running the body (and doesn't trip on a genuine argument equal to
the display marker); handleSuspension rejects for redelivery when
step_failed can't be written after step_created landed, leaving the
recoverable placeholder behind; mixed bad-step + large fan-out
returns the failure set alongside still-pending deferredBatchWork
whose rejection surfaces — the contract the runtime's failed-step
join (added previously) relies on.
- Docs: the two new code samples are now self-contained so the docs
code-sample typecheck passes.
* Pre-claim inline steps inside the suspension batch (born-running pairs)
Restacked onto main after #3025's squash-merge; folds in the review-round
changes to the flush loop (per-write requestId attribution on createBatch,
and the seeded/advancing slot-bump expectation, now shared with the
pre-claim ceiling).
Fold each lazy-inline step's deferred writes into the batched fan-out as an
adjacent [step_created, step_started] pair: the created row carries the input,
the started row is a bare ownership-stamped claim the server folds into one
born-running create. The whole scheduling turn commits as ONE durable write,
inline bodies start straight off that commit (in parallel with the VQS
publishes for backgrounded steps), and executeStep gains a pre-claimed mode
that runs or skips the body off the batch's per-event verdict - a pair 409 is
the same skipped outcome as losing the lazy claim. The lone-inline case keeps
the optimistic lazy path (a pair-only batch buys nothing over the single
claim). Also threads per-event computeInstanceId through the World batch
request, and folds the batch's committed slot ceiling into the inline slot
snapshot so terminal writes stop being answered with reports echoing the
batch's own events.
* Parallel chunk commits, per-chunk continuation, batch span attributes
Production trace of a 67-event fan-out showed the three batch chunks
POSTing back-to-back (~230ms each) with no bodies or queue messages until
all three settled (~670ms). Three changes:
- Chunks now POST concurrently. Slot assignment is the server's, so
parallel chunks race for slot ranges exactly like the pre-fold path's
parallel single writes did; entity conditions, not commit order, carry
correctness. The foreign-interleaving diagnostic is computed once over
the whole fold (committed span vs seed) instead of per chunk.
- Per-chunk continuation: each chunk's step-execution queue messages
publish the moment ITS creates are durable (in-flush, via stepDispatch,
same message shape and idempotency key as the caller's dispatch pass -
the affected steps are pre-reported in queuedStepCorrelationIds so the
caller skips them). Only the chunk carrying the inline pairs gates
handleSuspension's return (opt-in via allowDeferredBatchWork); trailing
chunk commits + all publishes ride result.deferredBatchWork, which the
runtime joins next to the dispatch join before it can ack - the
every-create-durable-before-ack contract is unchanged, the bodies just
start off the pair chunk instead of the slowest chunk.
- OTel: batch identity attributes (workflow.batch.size, per-type
workflow.batch.shape) now live on the world.events.createBatch span
(instrumentObject) instead of the http POST span, which keeps only
wire-level facts (transport, bytes) and no longer sets
workflow.event.type - that attribute names a single event write and
tagging a batch with its first event's type misclassifies traffic.
* Address review: settle deferred fold on failure, drop pair-batch retry
Three fixes from review of the deferred/parallel-chunk fold.
1. A pair-chunk rejection escaped `handleSuspension` while the trailing
chunks' commits and publishes were still in flight. `deferredBatchWork`
never reaches the caller once the handler throws, so nothing joined that
work — exactly the state `settlePhase` exists to prevent: a sibling create
landing after the rejection commits an event from the abandoned replay's
seeded sequence and races the caller's restart reload. The failure path now
settles `trailing` before rethrowing.
2. Every pair-carrying chunk gates the return, not just the first. Pairs sort
to the front and two rows per inline step fit inside one chunk, so this is
one commit today, but `findIndex` silently degraded if either cap moved: a
pair in an unawaited chunk yields no `inlineClaims` entry, the caller falls
back to a lazy `step_started`, and that races this same fold's in-flight
pair for the same step. constants.test.ts now pins the cap relationship.
3. A batch carrying a `step_started` is no longer retried in-process. The
born-running pair does converge to a 409, but the pre-claim caller reads a
pair 409 as "a concurrent writer owns this step" and skips the body — and
on a retry that is indistinguishable from "my own first attempt committed
the pair". Skipping there stranded a running step stamped with this
invocation's own message id until the ownership lease expired (860s), where
the single-POST path deliberately fails the delivery and recovers through
owned-recovery in seconds. Same reasoning `EVENT_RETRY_ELIGIBILITY` already
applies to `step_started`.
Also asserts `lazyStepInput` / `preclaimedStart` mutual exclusivity in
executeStep instead of only documenting it, and adds the changeset.
Tests: +1 suspension-handler (pair-chunk failure settles the trailing chunk
before escaping — fails without fix 1), +1 constants (cap relationship), +1
world-vercel (a born-running pair batch is single-attempt), and the existing
batch-retry test retargeted at an entity-conditioned batch. Full
@workflow/core unit suite 2178 green, @workflow/world-vercel 514 green,
typecheck green across core / world / world-vercel.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* Guard inline bodies against unhandledRejection; review follow-ups
The dispatch/deferred-batch joins now sit between the step promises'
creation and the `Promise.all` that reads them, so a body rejecting in that
window had no handler attached at the microtask checkpoint — an
unhandledRejection, fatal under Node's default --unhandled-rejections=throw.
A 412 fenced claim races exactly that window, and `deferredBatchWork` widens
it by a trailing-chunk round trip. Attach a no-op catch at creation, the same
way `dispatchesSettled` already does two lines up; the awaits below still
decide the outcome.
Review follow-ups:
- `workflow.batch.shape` is sorted by event type. Map iteration is first-seen
order, so a pre-claimed fold and a pure eager fold rendered the same
composition as different strings, which is not groupable as a dimension.
- A lost pre-claim reports StepSkipReason `running`, not `completed`. The
pair's 409 says the step already exists and its claim winner is executing;
the other skip site is a genuine terminal-state conflict, and tagging both
`completed` left the attribute unable to separate the two.
- `batchCommittedSlotCeiling`'s docstring now says the echo is only fully
suppressed for a single-chunk fold: on a multi-chunk fan-out an inline
terminal write issued before the trailing chunks land still names a
position below them and still draws a report.
- The defensive throw on a missing dehydrated input records where it lands —
the pair is already durable, so it fails with the step claimed and its body
unrun, recovered on redelivery via owned-recovery rather than failing
cleanly.
No regression test for the unhandledRejection: the existing
inlineClaimRejectionScenario runs both steps inline, so `dispatches` is empty
and the join resolves in a microtask — the window never opens and a test
there passes with or without the fix. Reproducing it needs a scenario with a
backgrounded step and a slow queue publish alongside the fenced claim.
Full @workflow/core unit suite 2178 green, @workflow/world-vercel 514 green,
typecheck and biome clean.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* Pin per-event computeInstanceId on the batch wire
Batch encoding is a separate path from the single-event POST, so the frame
meta had no coverage: the only assertion was at the World-call boundary.
Adds a wire-level test that a pre-claimed pair's step_started half carries
computeInstanceId in its frame meta and the step_created half does not.
Verified it fails when the threading in createWorkflowRunEventBatch is
removed.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* Guard the pre-claim path as inert on Worlds without createBatch
world-local and world-postgres do not implement createBatch, so the fold
never engages there — but the runtime passes ownerMessageId and
allowDeferredBatchWork unconditionally. The existing "keeps the single path
when the World lacks createBatch" test passed neither, so it never covered
the pre-claim path at all.
Assert the inertness with the params the runtime actually sends: no claims,
no deferred work, no slot ceiling, the lazy-inline step still carrying its
input, and no step_started reaching the world.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Peter Wielander <peter.wielander@vercel.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
A target answers HTTP well before its first run is picked up promptly,
for two reasons with one shape: a fresh Vercel deployment's queue
consumer takes a while to start delivering, and a local dev server pays
its first flow-route compile on the first queue delivery. The
run-pickup watchdog's telemetry shows the cost - stalls concentrated on
the suite's first test (addTenWorkflow), waitedMs pegged at the full
15s pickup budget, timestamps right at suite start; the sidecar
backends identify local-dev lanes as a dominant source alongside fresh
Vercel deployments. Each stall burns pickup budget inside a test,
drowns the infra telemetry in cold-start noise, and leaves the first
tests one stalled replacement away from failing.
warmDeployment() runs in the suite's beforeAll: it starts throwaway
probe runs, abandoning (best-effort cancelling) any still pending after
the pickup budget, until one is picked up or a total budget
(WORKFLOW_E2E_WARMUP_BUDGET_MS, default 120s) is spent. A warmup that
needed abandoned probes is recorded as a single cold-start-warmup infra
event - one per suite instead of per-test run-pickup-stall noise - and
an exhausted budget proceeds anyway: the per-test watchdog still guards
every start, and test failures carry run diagnostics a thrown warmup
would not.
Signed-off-by: Alex Langenfeld <alex.langenfeld@vercel.com>
The abort-fetch tests cancelled an in-flight fetch against external slow
endpoints (postman-echo, httpbin /delay/10, tried in order). Those
upstreams 5xx and return early from GH Actions runners often enough to
be a recurring flake class - the tests were measuring the public
internet instead of abort propagation - and heavier suite load (e.g.
re-enabling e2e concurrency, #2083) makes both upstreams flake at once.
fetchWithSignal now hosts its own slow endpoint: an in-process node:http
server on a loopback ephemeral port that holds each response open for
~30s. The subject is unchanged - a real in-flight HTTP fetch cancelled
mid-flight - with no external dependency. The 30s hold keeps regression
detection honest: broken abort propagation surfaces as natural
completion (ok: true) within the tests' 60s budgets.
A per-workbench /api/delay route was rejected earlier because it would
only exist on whichever workbench it was added to; the in-step server
travels with the workflow fixture to every app.
Signed-off-by: Alex Langenfeld <alex.langenfeld@vercel.com>
fib(6) spawns a 25-run tree whose ~24 concurrent parent polls saturate
the workflow scheduler past the test's 180s budget under a concurrent
suite (#2083 measured this as one of the three flake classes blocking
e2e concurrency re-enablement). fib(5)'s 15-run tree still exercises
the test's actual subject - recursive start() composition with parallel
children at every level - with 40% less peak load.
Signed-off-by: Alex Langenfeld <alex.langenfeld@vercel.com>
Where hasStepSourceMaps() reports maps unsupported, the tests asserted
their absence - but on some lanes (nuxt, nextjs-webpack) source maps
apply nondeterministically, so the negative assertion pinned that
nondeterminism as a flake (#2083 measured this as one of the three
flake classes blocking e2e concurrency re-enablement). A stack that
resolves to source where none was promised is an improvement, not a
failure; hasStepSourceMaps() remains the record to update when a lane
starts mapping reliably, verified by the positive assertions on the
lanes it does promise.
Signed-off-by: Alex Langenfeld <alex.langenfeld@vercel.com>
The dev HMR log line for a rebuild is written at classification time,
before the (potentially multi-second) rebuild runs. A reader counting
lines in a window has no way to tell 'quiet' from 'rebuild in flight
with queued events behind it', and the e2e HMR tests count lines in
exact windows: on CI, where rebuilds take 5-12s, a write from a
previous test case (or a teardown restore) can still be rebuilding when
the next window opens, and its legitimate rebuild lines land inside
that window - observed on main as the fuzz test failing both retry
attempts with 'expected 2 to be 1'.
The dev server now also logs 'workflow dev hmr: rebuild complete' when
a rebuild finishes processing (in a finally, so an erroring rebuild
cannot wedge readers; behind WORKFLOW_DEV_HMR_LOGS like every hmr
line). The e2e suite uses it to drain to quiescence - every started
rebuild completed, plus a short quiet window covering watcher latency
and the flush debounce - before taking a log cursor. Every cursor call
site writes only after taking its cursor, so draining there cannot
swallow lines a test means to count.
Signed-off-by: Alex Langenfeld <alex.langenfeld@vercel.com>
## Summary & Motivation
`step_started` now carries the invocation's request ID alongside the
compute instance ID, so observability can show both dimensions of where
an attempt ran. The two stay independent fields — world-vercel maps
`requestId` onto its analytics `vercelId`, which the compute instance ID
doesn't stand in for.
## Test Plan
Unit tests added for the stamping on all three step-start paths,
including when no request ID is available.
Signed-off-by: Alex Langenfeld <alex.langenfeld@vercel.com>
## Summary
- record `workflow.step.execute.duration` as an OpenTelemetry histogram
around the inner user-code step span
- rely on resource `service.name` for the service dimension and add only
a bounded status attribute
- cover the histogram name, unit, and attributes with a focused unit
test
## Validation
- `pnpm exec vitest run packages/core/src/telemetry-metrics.test.ts`
- `pnpm --filter @workflow/core typecheck`
---------
Co-authored-by: vercel[bot] <35613825+vercel[bot]@users.noreply.github.com>
## Summary & Motivation
Long live reads were ending silently at the server's 2-minute connection
cap: the max-duration abort reaches the client as a clean EOF on some
transport paths, and the reader read that as end-of-stream. On EOF it
now consults `streams.getInfo` and reconnects from the next chunk unless
the stream is done and every chunk up to the tail was delivered. A
failed metadata read trusts the EOF, so a transient blip can't fail a
healthy completion.
## Test Plan
Tests added for the reconnect, verified-completion, metadata-failure,
and reconnect-budget paths.
---------
Signed-off-by: Alex Langenfeld <alex.langenfeld@vercel.com>
Since we're moving to server side serialized ids, we dont need to
simulate the prior model.
## Summary
- remove the mint-ordered simulation mode and reservation API
- make commit-time log positions and lagging-prefix reads the only
simulator behavior
- simplify CI, scenarios, tests, and documentation to the single log
model
## Testing
- node --check on changed TypeScript files
- pnpm --filter @workflow/world-sim test (blocked: node_modules is
absent; vitest unavailable)
---------
Co-authored-by: vercel[bot] <35613825+vercel[bot]@users.noreply.github.com>
## createBatch: the client half of the v4 batch event write (per-event
results, no fence)
> **Note:** this PR was rebuilt from scratch. The previous revision
implemented the retired "v2 suspension fence" design
(`expectedRunVersion` / `batchId` / `logicalCreatedAt`, a
grammar-validated collect mode, world-postgres migration 0016,
world-local claim machinery). The server redesigned its endpoint in
place (vercel/workflow-server#646, merged and deployed) and this branch
now targets that contract on top of current `main` (specVersion 6 slot
identity). The old head is tagged `batch-client-v2-fence-design`; prior
review threads reference deleted code.
### The server contract this targets
`POST /api/v4/runs/:runId/events/batch` (workflow-server#646): an
**ordered** list of v4 frames — byte-identical to single-event POST
frames, **no batch-level meta** — committed in one DynamoDB transaction
per attempt, answered with HTTP 200 + `{ results }`: one entry per
frame, in request order. Each event reports what its own single POST
would have returned: `200` + the materialized entity, or the single-path
status/code (e.g. `409`/`conflict` for an event an earlier delivery
already applied). A transport retry of a committed batch converges to
all-409s with nothing written twice — idempotency comes from per-entity
conditions, not batch bookkeeping. Slot-identity runs only (specVersion
≥ 6 — what `world-vercel` stamps on every new run since #3389).
### What this revision ships
1. **`@workflow/world` — the spec addition.** `Storage['events']` gains
one optional method; **method presence is the capability declaration**
(no capability flag, no stub required):
```ts
createBatch?(
runId: string,
events: BatchEventRequest[],
params?: CreateEventBatchParams
): Promise<EventBatchResult>;
interface BatchEventRequest {
event: CreateEventRequest; // same discriminated union as the single create
occurredAt?: Date; // under slot identity: the source of the durable createdAt
}
type BatchEventItemResult = // one per submitted event, in request order
| { status: 200; event: Event; run?: WorkflowRun; step?: Step; wait?: Wait }
| { status: number; error: string; message: string };
interface EventBatchResult { results: BatchEventItemResult[] }
```
Contract: **ordered** (events land in the log in request order),
**per-event outcomes** (each event reports what its own single `create`
would have returned — success discriminated by `error === undefined`),
**idempotent on retry** (per-entity conditions make a retried committed
batch converge to per-event 409s). Worlds that don't implement it keep
the single-event path. `world-local` and `world-postgres` deliberately
do NOT implement it — batching a local/in-process write buys nothing
(this deletes the old revision's riskiest surface: the hand-written
postgres migration and the world-local claim machinery).
2. **`@workflow/world-vercel`** — the wire adapter: per-event frames
concatenated in order (reusing the single-frame encoder; each frame
carries its own `occurredAt`, which under slot identity is the source of
the durable `createdAt` — this natively closes the replay-clock question
the old `logicalCreatedAt` field existed for), CBOR `{ results }`
decoded against the **same per-type zod schemas as the single POST**,
loud `SCHEMA_VALIDATION` on any malformed response (wrong length,
invalid item), and the standard typed error mapping for request-level
failures.
3. **Retry policy** — a `batchIdempotent` override in the event-retry
eligibility machinery: the whole batch POST retries transient transport
failures/5xx (and waits out 429 `Retry-After` per #3504) regardless of
the contained event types, because per-event entity conditions make the
retry converge; the per-type non-retryability matrix guards *single*
posts (where e.g. a retried bare `step_started` would increment
`attempt`) and doesn't apply inside a batch.
Tests: 7 wire tests — frame encoding/ordering + **no fence fields on the
wire**, per-event result mapping (successes typed, failures passed
through), malformed-response failures (length mismatch, invalid item
body with index), typed request-level 400s, in-process 5xx retry,
empty-batch guard — plus 9 suspension-handler tests for the runtime
fold: ordering (steps then waits), per-event 409 tolerance, non-409
failure propagation, every gate exclusion (flag off / no `createBatch` /
pre-slot run / hook writes), 32-cap chunking, and lazy-inline exclusion.
Full `world-vercel` suite: 508 passed; full `@workflow/core` suite: 2126
passed.
### The runtime integration: batched suspension fan-out (ON by default)
The suspension handler folds a **clean fan-out** — the suspension's
eager `step_created` + `wait_created` writes — into `createBatch` calls
of at most **32 events**, and uses the batch endpoint **exactly when two
or more batchable eager events exist**: a lone eager event takes the
ordinary single write (same round trip, and it keeps the slot-snapshot +
bump-and-report the single path provides) (mirroring the server's
transaction budgets: 2 items/event against the 100-item cap, 768 KB
inline-byte budget; larger fan-outs commit in successive batches). The
gate requires: World implements `createBatch` ∧ run on slot identity
(specVersion ≥ 6) ∧ no attribute writes ∧ no hook writes ∧ no resilient
step dispatch. **Everything outside the gate keeps the single-event path
byte-for-byte**, and lazy-inline steps keep deferring their
`step_created` to the lazy start exactly as before.
Per-event semantics mirror the single path: a `409` is the same
already-exists tolerance as `EntityConflictError` (the conflicted step
is not marked owned); any other per-event failure fails the suspension
write the way a single-path rejection would. Slot bumps (the batch
endpoint has no bump-and-report) are tolerated and logged — the same
accepted exposure as a dropped truncated skipped-slot report on the
single path.
**On by default**, with the `WORKFLOW_TURBO`-shaped kill switch as the
operator escape hatch: **`WORKFLOW_BATCH_TRANSITIONS=0`** (or `false`)
disables batching and restores the exact prior one-write-per-event path.
Documented in the worlds configuration reference and the changelog
entry. Burn-in watch: the `event_batch`-tagged slot-conflict metrics and
DynamoDB throttle monitors on the server side.
### Docs
- New v5 changelog entry **`changelog/batched-event-writes`**
documenting the World spec addition (full `createBatch` signature +
contract — the signature block is compile-checked against
`@workflow/world` by the docs code-sample checker), the runtime fold,
and the follow-up.
- `configuration/worlds` gains the **`WORKFLOW_BATCH_TRANSITIONS`**
reference entry: default on, `=0`/`false` as the documented escape
hatch.
### Staged follow-up: the deferred sequential transition (the STSO win)
Hold `step_completed(N)` across the replay turn and commit
`[step_completed(N), step_created(N+1), step_started(N+1)]` as one batch
at the next lazy start (the server folds the pair born-running). This
needs the synthetic-completion replay machinery rebuilt against today's
runtime (parallel inline batches, turbo's run-ready barrier, optimistic
starts, slot bookkeeping) — it stays a separate PR so the SDK's most
sensitive replay path gets its own focused review. Its acceptance
criteria are already agreed: the runtime eligibility matrix as unit
tests, and an e2e that asserts ≥1 POST to `/events/batch` and **zero**
single-event POSTs for the batched transitions.
### Compatibility
- Old servers: no `/batch` route → 404/405 → callers fall back to
single-event posts (the runtime PRs will latch this per run).
- Pre-slot runs: request-level 400 (`batch-requires-slot-identity`) →
same fallback.
- No `WORKFLOW_SERVER_URL_OVERRIDE` pin this time — the server endpoint
is merged and deployed to production.
Refs: vercel/workflow-server#646 (endpoint), vercel/workflow-server#780
(unbatchable-types design space), #3389 (slot identity), #3504 (429
retry).
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
## Summary & Motivation
A full rebuild reads sources twice — once when the bundler consumes
them, once when the classifier baseline is refreshed from disk
afterwards — so an edit landing in between was absorbed into the
baseline without being built, and its queued watcher event classified as
a no-op. `pinBaselinesAcrossFullRebuild` pins baselines across the
rebuild instead: pre-build values for files present in both maps, and
the snapshots `classifyRebuild` read for the files that triggered the
rebuild, so routine duplicate watcher events still diff equal rather
than cascading into another full rebuild.
This is the largest ongoing CI flake — the nextjs-webpack local-dev e2e
HMR fuzz test writes a step definition right after triggering a
rediscovery and times out at 300s when the write lands mid-rebuild (~60
attempt-1 job failures in 10 days). For files the in-flight rebuild
discovered for the first time this narrows a create-then-edit race
rather than closing it; the initial-build startup window has the same
shape and is out of scope, documented at the call site.
## Test Plan
Unit tests cover the absorb race and the duplicate-event no-op; the
previously-flaky HMR fuzz e2e passed 3/3 locally against nextjs-webpack
where main reproduced the manifest drop, and a mid-rediscovery edit rig
went from 2/6 dropped on base to 0/6.
Signed-off-by: Alex Langenfeld <alex.langenfeld@vercel.com>
## Summary & Motivation
The e2e `start()` wrappers now poll the new run until it leaves
`pending`. A run still pending after `WORKFLOW_E2E_PICKUP_BUDGET_MS`
(default 15s) has executed no workflow code, so it is abandoned and
replaced in place and the test continues — one replacement, with the
CI-level retry still the backstop if that one stalls too.
Each replacement is recorded to an `e2e-infra-*.json` sidecar that every
e2e job uploads, and the aggregation script renders it as an "Infra
Events" section in the step summary and PR comment, so clustered
timestamps read as a backend blip rather than as unrelated flaky tests.
## Test Plan
Unit tests cover the pickup watchdog; a local nextjs-turbopack run
exercised both the clean path and, with a forced 1ms budget, the
replacement path end to end, and the aggregation script was smoke-tested
against synthetic sidecars in both modes.
Signed-off-by: Alex Langenfeld <alex.langenfeld@vercel.com>
Turns `packages/core/e2e/e2e.test.ts` into a cross-language conformance
suite and adds `workbench/python` as its first non-JavaScript subject.
---------
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
## Summary & Motivation
- **CRTT (chunk round-trip time)** — per-chunk write→read latency for a
paced stream, aggregated inside the reader step on the deployment (one
clock domain) into a fixed log-bin histogram plus index buckets and
mean-RTT profiles over stream progress and chunk size. Fills the gap
between SL (first chunk only) and SO (whole-stream throughput), where a
mid-stream delivery regression was invisible. It is deliberately a
*round*-trip name: the future production one-way write→read metric is
CTT, with its own skew caveats.
- **CDV (chunk delay variation)** — inter-arrival gap minus inter-write
gap per seq-adjacent pair, so each gap subtracts same-clock stamps and
the stat stays skew-free and measurable in production later. Reported as
each run's max positive value, since a 1-in-300 delivery stall dilutes
out of pooled percentiles.
- **Replay scenarios** — two real captured cadences (eve envelope
protocol via gpt-5.6-sol; raw gateway SSE via gpt-5.4-nano) replayed
through the same rig on an absolute open-loop schedule, so the workload
is measured rather than invented; the 2x speed multiplier is the only
chosen number, and matches how real fast-tier models behave (same chunk
sizes, compressed time). Each capture carries a semantic sha256 over
canonical `(offsetMs, bytes)` tuples so durabench's independent copy can
be checked for drift.
- **Streams table** — stream scenarios render in their own table with
writer/reader sustained rates, CRTT percentiles, and median worst stall.
No pass/fail targets yet: numbers and vs-main deltas only.
- **SL/SO report rows retired** — CRTT's seq-0 slice reproduces SL and
its aggregate reproduces SO's signal at ~100x the samples; write slip
stays as artifact-only data, the only guard for producer stalls that
neither CRTT nor CDV can see.
## Test Plan
- [x] Unit tests for the bucketing/merge/CDV helpers and the renderer;
the full benchmarks job ran green against real preview deployments, and
the first Streams numbers separated workload strain (eve 2x: read 173 <
write 181 c/s, CRTT p75 1278ms) from the transport floor (the paced
control and the 1x reality row both clean).
---------
Signed-off-by: Alex Langenfeld <alex.langenfeld@vercel.com>
Peter Wielander