## Summary & Motivation
- **CRTT (chunk round-trip time)** — per-chunk write→read latency for a
paced stream, aggregated inside the reader step on the deployment (one
clock domain) into a fixed log-bin histogram plus index buckets and
mean-RTT profiles over stream progress and chunk size. Fills the gap
between SL (first chunk only) and SO (whole-stream throughput), where a
mid-stream delivery regression was invisible. It is deliberately a
*round*-trip name: the future production one-way write→read metric is
CTT, with its own skew caveats.
- **CDV (chunk delay variation)** — inter-arrival gap minus inter-write
gap per seq-adjacent pair, so each gap subtracts same-clock stamps and
the stat stays skew-free and measurable in production later. Reported as
each run's max positive value, since a 1-in-300 delivery stall dilutes
out of pooled percentiles.
- **Replay scenarios** — two real captured cadences (eve envelope
protocol via gpt-5.6-sol; raw gateway SSE via gpt-5.4-nano) replayed
through the same rig on an absolute open-loop schedule, so the workload
is measured rather than invented; the 2x speed multiplier is the only
chosen number, and matches how real fast-tier models behave (same chunk
sizes, compressed time). Each capture carries a semantic sha256 over
canonical `(offsetMs, bytes)` tuples so durabench's independent copy can
be checked for drift.
- **Streams table** — stream scenarios render in their own table with
writer/reader sustained rates, CRTT percentiles, and median worst stall.
No pass/fail targets yet: numbers and vs-main deltas only.
- **SL/SO report rows retired** — CRTT's seq-0 slice reproduces SL and
its aggregate reproduces SO's signal at ~100x the samples; write slip
stays as artifact-only data, the only guard for producer stalls that
neither CRTT nor CDV can see.
## Test Plan
- [x] Unit tests for the bucketing/merge/CDV helpers and the renderer;
the full benchmarks job ran green against real preview deployments, and
the first Streams numbers separated workload strain (eve 2x: read 173 <
write 181 c/s, CRTT p75 1278ms) from the transport floor (the paced
control and the 1x reality row both clean).
---------
Signed-off-by: Alex Langenfeld <alex.langenfeld@vercel.com>
* Split STSO by inline vs queue-hop steps, add distribution diffs vs main
The sequential-steps benchmark's STSO metric mixed two unrelated
phenomena: gaps between steps running back-to-back in the same warm
process, and gaps across an invocation boundary (queue dispatch, client
reinit, event-log replay), which cost ~10x more. The old step-index
windows (1-20 / 101-120 / 1001-1020) sampled 19 gaps each and captured
neither cleanly: whether a boundary happened to land inside a window
moved that window's P99 by hundreds of percent, which is most of the
run-to-run variance the benchmark comment was reporting.
The workflow now tags each step with whether it was the first step body
executed in its process ('queue-hop') or a later one in the same warm
process ('inline') via a process-global, so the split is ground truth
rather than inferred from step index or trace timestamps. STSO is
reported as two rows over *every* gap in the run instead of three
sampled windows. No targets on the new rows — the old ones described the
index-bucketed grouping.
computeStats now keeps the full sorted sample array alongside the
percentiles, and the comment renders a histogram + cumulative-time diff
against `main` under the table, one per STSO kind. Percentiles alone
hide how many samples moved and by how much, which is exactly where the
variance lives. Inline rows use a fixed 50ms bin width (the adaptive
width is coarse enough to hide structure inside that cluster); queue-hop
rows keep the adaptive width. Negative gaps (clock skew between two step
bodies' clocks) get their own bucket rather than being counted with the
slow tail.
Raw samples are stripped from the comment's embedded data block — ~1000
per run would exceed GitHub's comment size limit within a couple of
history entries — so the histogram renders for the current run only,
while collapsed history keeps its tables. Until this lands on `main` no
baseline has raw samples, so the section renders this run's distribution
as a single series.
Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com>
* Clarify what stripping raw samples from the data block does not affect
Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com>
* Drop the bucket tables; fold counts and deltas into the histogram bars
Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com>
* Collapse the STSO distribution section into a dropdown
Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com>
* Fix footer assertion after the dropdown wording change
Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com>
---------
Co-authored-by: vercel[bot] <35613825+vercel[bot]@users.noreply.github.com>
Alex Langenfeld