2 Commits

Author	SHA1	Message	Date
Tobias Koppers	e648495f95	Replace lzzzz with lz4_flex (#98799) ... ## What? Replace the direct `lzzzz` dependency used by Turbopack persistence with `lz4_flex` 0.14. The migration covers persisted blocks, the SST inspection utility, and optional compressed-size diagnostics in the task backend. ## Why? This removes the direct C/liblz4-backed codec dependency from these paths in favor of a Rust implementation. Persistence caches are scoped to the Next.js and Turbopack version, so this does not require a compatibility path for caches written by earlier versions. ## How? Persistence continues to use raw LZ4 blocks and the existing compression selection and checksum layout. A large thread-local `CompressTable` is reused across blocks. Starting with the large table costs about 8 KiB more per compression thread than the small table, but improves compression speed and produces LZ4 streams that are faster for `lz4_flex` to decode. The compressor retains the existing caller-owned buffer API. It clears and reserves the output vector, lets `lz4_flex` write directly into the uninitialized backing allocation, and exposes only the initialized byte count returned by the compressor. This avoids both per-call output allocations and pre-zeroing the maximum output size. Zstandard behavior is unchanged. ### Benchmark vs `lzzzz` The final PR configuration was benchmarked against `lzzzz` 2.0.0 using the repository-pinned Rust nightly. The benchmark used source-code blocks sampled from this repository, pinned each process to one physical CPU, measured thread CPU time, and alternated codec order within every paired round. The table reports the median across four physical cores and 20 paired rounds per process (480 paired rounds total). Each decoder read the output produced by its own encoder, matching how a same-version persistence cache is written and later read. | Source corpus | Block size | Compression | Decompression | Compressed size | | --- | ---: | ---: | ---: | ---: | | Mixed | 12 KiB | **7.1% faster** | 9.9% slower | 0.08% larger | | Rust | 12 KiB | **5.5% faster** | 10.4% slower | 0.09% larger | | TypeScript | 12 KiB | **7.4% faster** | 7.4% slower | 0.23% larger | | Mixed | 16 KiB | **5.9% faster** | 8.8% slower | 0.17% smaller | | Rust | 16 KiB | **4.6% faster** | 10.0% slower | 0.12% smaller | | TypeScript | 16 KiB | **6.7% faster** | 6.1% slower | 0.02% smaller | For these 12–16 KiB source blocks, `lz4_flex` compression was **4.6–7.4% faster**, decompression was **6.1–10.4% slower**, and compressed size changed by **-0.17% to +0.23%**. The benchmark was stable enough to support those codec-level ranges: wall time divided by thread CPU time was 1.000073 at the median and 1.001883 at p99. The median difference between the two codec orders was 0.38 percentage points for compression and 1.45 points for decompression. A separate decoder-only comparison fed identical `lzzzz`-encoded source blocks to both decoders; there, `lz4_flex` was 17–21% slower. The smaller regression in the table above comes from the large Flex compression table producing a different valid LZ4 token stream. The PR already uses `lz4_flex`'s supported unsafe pointer implementation with validation; a local experiment that removed validation recovered only 5–6% and remained slower than liblz4, so this PR does not weaken decoder safety. These are codec microbenchmarks, not an end-to-end Turbopack speedup claim. ## Verification - `cargo fmt --all -- --check` - `cargo clippy -p turbo-persistence --all-targets -- -D warnings` - `cargo test -p turbo-persistence -- --test-threads=1` (110 passed) - `cargo check -p turbo-tasks-backend --features print_cache_item_size_with_compressed` - Pinned-nightly paired codec benchmark: 24 processes / 480 paired rounds <!-- NEXT_JS_LLM --> <!-- fleet 8f69f876-cb41-4fa2-ab60-dc8ccea88055 --> --------- Co-authored-by: vercel-fleet-prod[bot] <318278635+vercel-fleet-prod[bot]@users.noreply.github.com> Co-authored-by: Benjamin Woodruff ‮ <180404+bgw@users.noreply.github.com> Co-authored-by: Tobias Koppers <1365881+sokra@users.noreply.github.com>	2026-09-17 18:37:51 +00:00
Tobias Koppers	e44c5f192c	ci: check that Turbopack compiles for wasm (#97588) ... ### What? Adds a CI job that checks Turbopack compiles for `wasm32-wasip1-threads`, so wasm portability regressions are caught rather than rediscovered. ### Why? Everything under `#[cfg(target_family = "wasm")]` is invisible to host builds **and** to host clippy — it is only checked when you deliberately build for the target. Four separate defects in this stack were caught only that way (a wrong build-script condition, two bad imports, and a value that compiled but was wrong). Without a gate, the next one lands unnoticed. This also re-enables coverage that had been off for ~2 years: the old `test-next-napi-bindings-wasi` job was disabled with `if: false` pending napi-rs/napi-rs#2009, which closed in April 2024. ### How? Modelled on `rust-check` via `build_reusable.yml` (`needsRust`, `skipInstallBuild`, `skipNativeBuild`), so it does not pay for a JS build. Beyond `rustup target add` it needs two things: - **a WASI C toolchain**, because `lzzzz` (LZ4, via `turbo-persistence`) and `zstd-sys` have C build scripts. The SDK build is selected from `$RUNNER_ARCH` — `build_reusable.yml` defaults to an arm64 runner, and an x86_64 clang fails there with `Exec format error` — with a pinned sha256 per arch, unpacked under `$RUNNER_TEMP` so the workspace stays clean. - **emnapi**, because `next-napi-bindings`' build script calls `napi_build::setup()`, whose wasi path panics without `EMNAPI_LINK_DIR` — so it is required even for `cargo check`. It is installed into a scratch directory rather than the root `package.json`, because this job runs with `skipInstallBuild` and therefore never runs `pnpm install`. Two things worth recording, both of which cost a CI round trip to find: - pnpm is invoked from the repo root with `--dir`, not by `cd`-ing into the scratch directory: corepack resolves the pnpm version from the nearest `package.json`, and outside the repo it picks the latest pnpm (11.x), which cannot run on the pinned Node 20 (`ERR_VM_DYNAMIC_IMPORT_CALLBACK_MISSING`). - the job is added to `tests-pass`, whose `needs:` list is what actually blocks a PR — a job that runs but is absent from that list looks like coverage while blocking nothing. The `emnapi@2.0.0-alpha.4` pin is deliberate and is the one fragility here: the archive must define `emnapi_create_env` / `emnapi_delete_env`, which exists only in emnapi v2, still a prerelease. Move to the stable release once it ships. <!-- fleet b6d0486f-97c7-42a7-bdaf-3490774cdec3 --> Co-authored-by: vercel-fleet-prod[bot] <318278635+vercel-fleet-prod[bot]@users.noreply.github.com> Co-authored-by: Tobias Koppers <1365881+sokra@users.noreply.github.com>	2026-09-09 10:25:45 +02:00