* Split STSO by inline vs queue-hop steps, add distribution diffs vs main
The sequential-steps benchmark's STSO metric mixed two unrelated
phenomena: gaps between steps running back-to-back in the same warm
process, and gaps across an invocation boundary (queue dispatch, client
reinit, event-log replay), which cost ~10x more. The old step-index
windows (1-20 / 101-120 / 1001-1020) sampled 19 gaps each and captured
neither cleanly: whether a boundary happened to land inside a window
moved that window's P99 by hundreds of percent, which is most of the
run-to-run variance the benchmark comment was reporting.
The workflow now tags each step with whether it was the first step body
executed in its process ('queue-hop') or a later one in the same warm
process ('inline') via a process-global, so the split is ground truth
rather than inferred from step index or trace timestamps. STSO is
reported as two rows over *every* gap in the run instead of three
sampled windows. No targets on the new rows — the old ones described the
index-bucketed grouping.
computeStats now keeps the full sorted sample array alongside the
percentiles, and the comment renders a histogram + cumulative-time diff
against `main` under the table, one per STSO kind. Percentiles alone
hide how many samples moved and by how much, which is exactly where the
variance lives. Inline rows use a fixed 50ms bin width (the adaptive
width is coarse enough to hide structure inside that cluster); queue-hop
rows keep the adaptive width. Negative gaps (clock skew between two step
bodies' clocks) get their own bucket rather than being counted with the
slow tail.
Raw samples are stripped from the comment's embedded data block — ~1000
per run would exceed GitHub's comment size limit within a couple of
history entries — so the histogram renders for the current run only,
while collapsed history keeps its tables. Until this lands on `main` no
baseline has raw samples, so the section renders this run's distribution
as a single series.
Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com>
* Clarify what stripping raw samples from the data block does not affect
Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com>
* Drop the bucket tables; fold counts and deltas into the histogram bars
Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com>
* Collapse the STSO distribution section into a dropdown
Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com>
* Fix footer assertion after the dropdown wording change
Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com>
---------
Co-authored-by: vercel[bot] <35613825+vercel[bot]@users.noreply.github.com>
* Add 50, 100, 500 concurrent step benchmarks
Enable Promise.all and Promise.race benchmarks for 50, 100, and 500
concurrent steps (previously 100+ were skipped).
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* Add 50, 100, 500 sequential step benchmarks
Extends the sequential step benchmarks to test workflows with 50, 100,
and 500 sequential steps in addition to the existing 10 step test.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* Add full/quick benchmark suite toggle for CI
- Add BENCHMARK_FULL_SUITE env var to control which benchmarks run
- Quick suite (default for PRs): 10, 25, 50 step benchmarks
- Full suite (main branch, manual dispatch): adds 100, 500 step benchmarks
- Add workflow_dispatch input to manually trigger full suite from GitHub UI
- Skip 100+ sequential and concurrent step benchmarks by default
This keeps PR benchmarks fast while allowing full stress testing on demand.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* Add 25 sequential steps benchmark
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* Fix benchmark API calls to use binary format
PR #853 changed the workflow trigger API to expect binary data
(application/octet-stream) instead of JSON. The e2e tests were updated
but the benchmark file was missed, causing all benchmarks to fail
silently since Jan 28.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* Only run full benchmark suite on manual dispatch
Remove automatic full suite on main branch pushes - only run full suite
when manually triggered with full_suite=true.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
* Use proper OpenAI import
```
Error: ../example/workflows/5_hooks.ts(36,22): error TS2351: This expression is not constructable.
Type 'typeof import("/vercel/path0/node_modules/.pnpm/openai@6.1.0_ws@8.18.3_zod@4.1.11/node_modules/openai/index")' has no construct signatures.
```
* .
* perf: add 5s delay to benchmark steps to simulate real work
* perf: add realistic workloads to benchmark steps
Add realistic workloads to benchmark step functions:
- doWork() - 1 second delay to simulate real computation
- stressTestStep() - 1 second delay to simulate real computation
- genBenchStream() - generates ~5KB of data in 50 chunks
- transformStream() - uppercases stream content (renamed from doubleNumbers)
Add "Slurp Time" metric to stream benchmarks measuring time from
first byte to complete stream consumption, complementing TTFB.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
* fix(e2e): increase dev test timeouts for Windows
Windows file watching and rebuilding is slower than macOS/Linux,
causing the 10s timeouts to fail. Increased to 30s to accommodate.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
* fix(ci): treat community world failures as warnings, not errors
Community world tests/benchmarks are now non-blocking:
- Removed from has_failures check in both tests.yml and benchmarks.yml
- Added separate has_warnings output for community failures
- Split PR comment notices: ❌ for failures, ⚠️ for community warnings
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
* fix: apply PR review suggestions
- Fix chunk size calculation: chunkSize - 11 for ~100 bytes per chunk
- Remove redundant metadata check (always truthy)
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
---------
Co-authored-by: Claude <noreply@anthropic.com>
Add new benchmark workflows to reproduce reported issues with Promise.race
and Promise.all falling over when array sizes exceed a few hundred.
New workflows:
- promiseAllStressTestWorkflow(count) - Tests Promise.all with many concurrent steps
- promiseRaceStressTestLargeWorkflow(count) - Tests Promise.race with Map pattern
New benchmarks at 100, 500, and 1000 concurrent step scales for both
Promise.all and Promise.race patterns.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
* Add stream benchmarks and cnealup for benchmark code
* 10m delay for local world race
* potential fix for first bute"
* Improve sticky comment stuff
* local world: ignore controller close errors
* show missing data in benchmark comment
* nitrpicks
* Improve leaderboard
* Add benchmark comparisons against main
* changeset
Shalabh Chaturvedi