Pranay Prakash
5f0b845211
* feat(core,world): gzip-compress serialized payloads behind specVersion 5 Add a composable 'gzip' format prefix layer to the serialization pipeline (compress before encrypt: encr(gzip(devl))), cutting stored payload bytes by ~70-87% on real-world-style workloads. Compression is gated on run specVersion 5 (new SPEC_VERSION_SUPPORTS_COMPRESSION) and on target-deployment capabilities for cross-deployment writes; payloads under 1KB or that don't compress meaningfully are stored unchanged. Reads dispatch on the format prefix so both compressed and uncompressed data are always readable. WORKFLOW_DISABLE_COMPRESSION=1 disables writes. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * test(core): add CPU/perf compression benchmark + shared workloads Split the compression benchmark into reproducible size and CPU scripts sharing deterministic workloads (lib/workloads.mjs). The CPU benchmark measures serialize/deserialize overhead per payload, total CPU across thousands of events, and compares gzip levels/brotli/deflate. Documents how to run the size, CPU, and end-to-end (bench.bench.ts) benchmarks against local and Vercel in scripts/README.md. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat(world-vercel): advertise specVersion 5 to enable compression on Vercel Now that workflow-server declares spec-5 support (vercel/workflow-server#520), bump the Vercel world's advertised specVersion from 4 to 5 so new Vercel runs are stamped spec 5 and become eligible for gzip payload compression. Payloads stay opaque to the server (compression is client-side); spec 5 is a superset of spec 4, so initial run attributes still work. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat(core): emit OTel span attributes for compression impact Track gzip payload compression on both the serialize (write) and deserialize (read) paths via span attributes: workflow.serialization.{operation,compressed,uncompressed_bytes, stored_bytes,compression_ratio}. Sizes are measured at the compression boundary (pre-encryption), so they reflect compression's effect rather than the at-rest size. The compression codec stays pure — compress/decompress optionally populate a CompressionStats sink, threaded through CodecOptions to the mode serializers and read by the dehydrate/hydrate wrappers, which set attributes on the active span. Telemetry failures are swallowed so they can never break the serialize/deserialize data path. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat(core,web-shared): prefer zstd compression codec (gzip fallback) Switch the payload compression codec to zstd, which benchmarks 3–7× faster than gzip at an equal-or-better ratio on representative workloads (compression runs at every step boundary, so the write CPU is a per-step tax). zstd uses node:zlib (>= 22.15); gzip via the portable CompressionStream remains the fallback when zstd is unavailable, and WORKFLOW_COMPRESSION_CODEC=gzip forces it. Reads dispatch on the format prefix, so 'zstd' and 'gzip' payloads are both always decodable. zstd is Node-only (Web CompressionStream has no zstd), so the browser o11y read path registers a WASM-backed decoder (@tootallnate/zstd-wasm) via a new registerZstdDecoder hook; node:zlib handles Node-side reads (runtime replay, CLI, server o11y). A new workflow.serialization.codec span attribute reports which codec applied. gzip and zstd read support co-ship, so the existing specVersion-5 capability gate is unchanged. Verified end-to-end: spec-5 runs store zstd-prefixed payloads on disk and replay/complete correctly; the WASM decoder round-trips node:zlib zstd output. Benchmarks updated to compare zstd vs gzip. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Compression benchmarks
Reproducible benchmarks for the gzip payload compression feature
(specVersion 5, PR adding the gzip serialization format prefix). Two
dimensions are measured: storage size (bytes saved) and CPU cost
(time added to serialize/deserialize). All workloads are shared and
deterministic — see lib/workloads.mjs.
Build @workflow/core first so the scripts can import the compiled
serialization layer:
pnpm --filter @workflow/core build
cd packages/core
1. Storage size
node scripts/benchmark-compression-size.mjs
Prints the exact bytes the serialization layer hands to the World storage
backends (S3/DynamoDB refs for vercel, bytea columns for postgres, JSON
files for local), compression off vs on, per workload, plus a simulated
10-step AI-agent event-log total. Backends that base64-encode binary
(DynamoDB inline refs, world-local JSON) see ~33% larger absolute savings
than the raw numbers.
2. CPU cost
node scripts/benchmark-compression-cpu.mjs
Three sections:
- Per-payload serialize + deserialize cost through the real shipping
path (
step.serialize/step.deserialize, which use the WebCompressionStream('gzip')), off vs on, with throughput. - Stress — total serialization CPU to write + replay-read thousands of event payloads, modelling a long workflow.
- Algorithm comparison (
node:zlibsync) — gzip levels 1/6/9, brotli, deflate-raw — informational, to compare candidate codecs for a future format prefix (e.g. azsd1zstd codec). Not the shipping path.
Compression is a world-independent CPU cost added to the serialize/deserialize path. The world only changes the baseline you compare against: local (filesystem) is the fastest baseline so the relative impact is largest there; Vercel (network + AES encryption + S3) has the slowest baseline so the relative impact is smallest. The absolute microbenchmark numbers hold for every backend.
3. End-to-end runtime (local + vercel)
The end-to-end harness already in the repo drives the stress workflows in
workbench/example/workflows/97_bench.ts through a real World and records
per-run executionTimeMs (completedAt − createdAt) to
bench-timings-<app>-<backend>.json:
# Local world (nextjs-turbopack dev server on :3000)
cd workbench/nextjs-turbopack && WORKFLOW_PUBLIC_MANIFEST=1 pnpm dev &
pnpm bench:local # from repo root
# Full suite incl. 1000-step / 1000-concurrent / 500×10KB cases
BENCHMARK_FULL_SUITE=true pnpm bench:local
To measure the compression delta, run the harness twice and diff the
output JSON: once normally (compression on, specVersion 5) and once with
WORKFLOW_DISABLE_COMPRESSION=1 set on both the dev server and the
bench runner (compression off, everything else identical):
# compression OFF baseline
WORKFLOW_DISABLE_COMPRESSION=1 pnpm dev & # in the workbench
WORKFLOW_DISABLE_COMPRESSION=1 pnpm bench:local # from repo root
mv bench-timings-nextjs-turbopack-local.json bench-timings-...-off.json
For Vercel, the same harness targets a deployment when the Vercel env
vars from CLAUDE.md are set (WORKFLOW_VERCEL_ENV, VERCEL_DEPLOYMENT_ID,
WORKFLOW_VERCEL_AUTH_TOKEN, WORKFLOW_VERCEL_PROJECT, VERCEL_OIDC_TOKEN,
etc.); it then writes bench-timings-<app>-vercel.json. The
WORKFLOW_DISABLE_COMPRESSION=1 kill switch must be set on the deployment
(an env var on the Vercel project) for the off baseline, since compression
runs server-side in the step/workflow handlers there.