Files
Pranay Prakash 5f0b845211 RFC: compress serialized payload refs — zstd (gzip fallback), specVersion 5 (#2394)
* feat(core,world): gzip-compress serialized payloads behind specVersion 5

Add a composable 'gzip' format prefix layer to the serialization
pipeline (compress before encrypt: encr(gzip(devl))), cutting stored
payload bytes by ~70-87% on real-world-style workloads. Compression is
gated on run specVersion 5 (new SPEC_VERSION_SUPPORTS_COMPRESSION) and
on target-deployment capabilities for cross-deployment writes; payloads
under 1KB or that don't compress meaningfully are stored unchanged.
Reads dispatch on the format prefix so both compressed and uncompressed
data are always readable. WORKFLOW_DISABLE_COMPRESSION=1 disables
writes.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* test(core): add CPU/perf compression benchmark + shared workloads

Split the compression benchmark into reproducible size and CPU scripts
sharing deterministic workloads (lib/workloads.mjs). The CPU benchmark
measures serialize/deserialize overhead per payload, total CPU across
thousands of events, and compares gzip levels/brotli/deflate. Documents
how to run the size, CPU, and end-to-end (bench.bench.ts) benchmarks
against local and Vercel in scripts/README.md.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* feat(world-vercel): advertise specVersion 5 to enable compression on Vercel

Now that workflow-server declares spec-5 support (vercel/workflow-server#520),
bump the Vercel world's advertised specVersion from 4 to 5 so new Vercel runs
are stamped spec 5 and become eligible for gzip payload compression. Payloads
stay opaque to the server (compression is client-side); spec 5 is a superset of
spec 4, so initial run attributes still work.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* feat(core): emit OTel span attributes for compression impact

Track gzip payload compression on both the serialize (write) and
deserialize (read) paths via span attributes:
workflow.serialization.{operation,compressed,uncompressed_bytes,
stored_bytes,compression_ratio}. Sizes are measured at the compression
boundary (pre-encryption), so they reflect compression's effect rather
than the at-rest size.

The compression codec stays pure — compress/decompress optionally
populate a CompressionStats sink, threaded through CodecOptions to the
mode serializers and read by the dehydrate/hydrate wrappers, which set
attributes on the active span. Telemetry failures are swallowed so they
can never break the serialize/deserialize data path.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* feat(core,web-shared): prefer zstd compression codec (gzip fallback)

Switch the payload compression codec to zstd, which benchmarks 3–7×
faster than gzip at an equal-or-better ratio on representative workloads
(compression runs at every step boundary, so the write CPU is a per-step
tax). zstd uses node:zlib (>= 22.15); gzip via the portable
CompressionStream remains the fallback when zstd is unavailable, and
WORKFLOW_COMPRESSION_CODEC=gzip forces it. Reads dispatch on the format
prefix, so 'zstd' and 'gzip' payloads are both always decodable.

zstd is Node-only (Web CompressionStream has no zstd), so the browser
o11y read path registers a WASM-backed decoder (@tootallnate/zstd-wasm)
via a new registerZstdDecoder hook; node:zlib handles Node-side reads
(runtime replay, CLI, server o11y). A new workflow.serialization.codec
span attribute reports which codec applied. gzip and zstd read support
co-ship, so the existing specVersion-5 capability gate is unchanged.

Verified end-to-end: spec-5 runs store zstd-prefixed payloads on disk
and replay/complete correctly; the WASM decoder round-trips node:zlib
zstd output. Benchmarks updated to compare zstd vs gzip.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-06-16 23:27:47 +00:00

57 lines
1.9 KiB
TypeScript

/**
* Compatibility test for the browser zstd decode path: payloads written by
* the SDK's `node:zlib` zstd codec must decode via the `@tootallnate/zstd-wasm`
* decoder the web o11y uses. If these ever disagree, the dashboard can't read
* compressed runs — so this locks the cross-codec contract in.
*/
import { readFileSync } from 'node:fs';
import { createRequire } from 'node:module';
import zlib from 'node:zlib';
import { decompressBytes } from '@tootallnate/zstd-wasm';
import { beforeAll, describe, expect, it } from 'vitest';
const require = createRequire(import.meta.url);
let wasmModule: WebAssembly.Module;
beforeAll(async () => {
const wasmPath = require.resolve('@tootallnate/zstd-wasm/zstd.wasm');
wasmModule = await WebAssembly.compile(readFileSync(wasmPath));
});
function zstd(bytes: Uint8Array): Uint8Array {
return new Uint8Array(
zlib.zstdCompressSync(bytes, {
params: { [zlib.constants.ZSTD_c_compressionLevel]: 3 },
})
);
}
describe('zstd WASM decoder ↔ node:zlib zstd compatibility', () => {
it('decodes a payload compressed by the SDK codec', async () => {
const original = new TextEncoder().encode(
JSON.stringify({
// Repetitive + varied content, like a real serialized payload.
users: Array.from({ length: 300 }, (_, i) => ({
id: `user_${i}`,
email: `user.${i}@example.com`,
role: i % 3 === 0 ? 'admin' : 'member',
})),
})
);
const compressed = zstd(original);
expect(compressed.length).toBeLessThan(original.length);
const decoded = await decompressBytes(wasmModule, compressed);
expect(new Uint8Array(decoded)).toEqual(original);
});
it('round-trips an empty and a tiny payload', async () => {
for (const s of ['', '{}', 'x']) {
const original = new TextEncoder().encode(s);
const decoded = await decompressBytes(wasmModule, zstd(original));
expect(new Uint8Array(decoded)).toEqual(original);
}
});
});