Files
cloudflare__vinext/tests/precompress.test.ts
T
Nathan Nguyen 6da45a680a perf: build-time precompression + startup metadata cache for static serving (#641)
* perf: add build-time precompression for hashed static assets

Generate .br (brotli q11), .gz (gzip l9), and .zst (zstandard l19)
files alongside compressible assets in dist/client/assets/ during
vinext build. These are served directly by the production server,
eliminating per-request compression overhead for immutable build output.

Only targets assets/ (hashed, immutable). Public directory files still
use on-the-fly compression since they may change between deploys.

* perf: add startup metadata cache for zero-syscall static serving

StaticFileCache walks dist/client/ once at server boot and caches:
- File metadata (path, size, content-type, cache-control, etag)
- Pre-computed response headers per variant (original, br, gz, zst)
- In-memory buffers for small files (< 64KB) for res.end(buffer)
- Precompressed variant paths and sizes

Per-request serving is Map.get() + res.end(buffer) with zero filesystem
calls, zero object allocation, and zero header construction. Modeled
after sirv's production mode but with in-memory buffering for small
files which eliminates createReadStream fd overhead.

* perf: refactor tryServeStatic to async + cache + precompressed serving

Refactor tryServeStatic to use StaticFileCache for the hot path:
- Pre-computed response headers (zero object allocation per request)
- In-memory buffer serving for small precompressed files
- 304 Not Modified via ETag + If-None-Match
- HEAD request optimization (headers only, no body)
- Zstandard serving (zstd > br > gzip > original fallback chain)
- Async filesystem fallback for non-cached files (replaces blocking
  existsSync + statSync)
- Skip decodeURIComponent for clean URLs (no % in path)

Wire StaticFileCache.create() into both startAppRouterServer and
startPagesRouterServer at startup. Integrate precompressAssets() into
the vinext build pipeline.

CONTENT_TYPES is now a single source of truth exported from
static-file-cache.ts (was duplicated in prod-server.ts).

* docs: fix stale comments in precompress (mention .zst)

* fix: deduplicate buffer reads for HTML alias entries, fix stale JSDoc

* docs: fix stale comment in cli.ts (mention zstd)

* fix: address code review findings

- Add Vary: Accept-Encoding to original variant when compressed siblings exist
- Use pipeline() instead of .pipe() for stream error safety
- Remove legacy accessor fields from StaticFileEntry (all new code)
- Exclude Content-Type from 304 responses per RFC 9110
- Fix misleading Accept-Encoding comment in prod-server
- Batch stat() calls in walkFilesWithStats (chunks of 64)
- Bound precompression concurrency to availableParallelism (max 16)
- Gracefully skip zstd when zlib.zstdCompress unavailable (Node <22.15)
- Add --no-precompress CLI flag to skip build-time compression
- Drop brotli to q5 (fast, nearly same ratio as q11 at build time)
- Max out zstd at level 22 (build time, no reason to hold back)

* fix: address remaining review findings in prod-server

- Gate negotiateEncoding zstd on runtime availability (HAS_ZSTD)
- Add ZSTD_e_flush for streaming mode (progressive SSR decodability)
- Serve HEAD without body on slow path (compressed + uncompressed)
- Handle pipeline errors instead of silently swallowing them
- Lowercase Accept-Encoding in fast path per RFC 9110
- Fix stale comments: add zstd to module header and variant detection
- Add tests for slow path, HEAD, and malformed percent-encoded pathnames

* perf: move precompression to Vite plugin with edge auto-detection

Move build-time precompression from CLI post-step into a writeBundle
plugin hook (vinext:precompress). The plugin already knows the
deployment target — auto-skips for Cloudflare Workers and Nitro where
the CDN handles compression at the edge.

- Add precompress option to VinextOptions: 'auto' | boolean
- CLI --no-precompress forwards via VINEXT_NO_PRECOMPRESS env var
- Works for any build invocation (CLI, API, CI), not just vinext build

* docs: fix inaccurate comments in precompress and static-file-cache

- precompress: remove false idempotency claim (overwrites, not skips)
- static-file-cache: "frozen objects" → "reusable objects" (no freeze)
- static-file-cache: fix unrealistic compression ratio example

* fix: address code review findings

- Remove dead code: redundant .br/.gz/.zst extension checks already
  caught by COMPRESSIBLE_EXTENSIONS filter
- Fix unreachable pipeline error handlers: res.writeHead() was called
  before pipeline(), so res.headersSent was always true in the callback.
  Use res.destroy(err) to cleanly tear down the socket on stream errors.
- Read file content lazily inside compression chunks instead of loading
  all files upfront, keeping peak memory at O(CONCURRENCY * max_file_size)

* perf: tune compression levels and restore auto mode

- Drop zstd from level 22 to 8 (250x faster build, 7% larger output)
- Drop gzip from level 9 to 8 (marginal size difference, faster)
- Restore precompress: 'auto' | boolean config with auto as default
  (auto-enables for Node.js targets, skips for Cloudflare/Nitro)
- Add progress bar for precompression step (matches prerender UX)
- Clarify compression ratio output as brotli-specific

* fix: use filename hash for ETag on hashed assets

Hashed assets (assets/app-DqZc3R4n.js) now derive their ETag from the
content hash in the filename instead of mtime. More robust across
Docker builds, CI artifacts, and cp -r where mtime changes between
deploys even when content is identical.

- Use weak validator (W/) since the tag is shared across content-encoded
  variants (.br, .gz, .zst) which are byte-different (RFC 9110 §8.8.1)
- Fall back to mtime-based ETag when filename has no recognizable hash
  (custom assetFileNames, plugin-emitted assets, public/ files)

* fix: make precompress opt-in per maintainer feedback

Real-world benchmark shows 13.5% build-time regression on fumadocs
with auto mode (2.4s → 2.7s). Switch to explicit opt-in via
`precompress: true` or `--precompress` CLI flag until the feature
stabilises.

* fix: address code review findings

- ETag regex: use lastIndexOf('-') to extract only the final hash
  segment, avoiding capture of filename fragments in multi-hyphenated
  names (e.g. `before-pop-state-D3R3Ien6` now extracts `D3R3Ien6`)
- Progress bar: track processed count (including skipped files) so
  the bar always reaches 100%
- Tests: update ETag test for filename-hash behavior, add coverage
  for mtime fallback on non-hashed and unhashed-in-assets files

* chore: use type instead of interface per lint rule

* fix final precompression review comments

* perf: optimize precompress build path

- Overlap precompression with SSR build phase (async writeBundle + closeBundle await)
- Lower concurrency cap from 16 to 8 (reduces zlib queue contention)
- Fix progress counter race in chunked processing

* fix: address final review feedback

- Fix progress counter race in precompress (single onProgress call site
  after readFile, monotonic by construction)
- Add console.error before res.destroy in all pipeline error handlers
  for operational visibility
- Chunk buffer reads at 64 concurrent to bound FD usage on large projects
- Defer createCompressor until after HEAD check on slow path
- Use Math.floor for mtime ETag (matches send/Express convention)
- Document Rolldown hash length assumption

* fix: address final bonk review nits

- Move progress reporting to after chunk completion to avoid
  non-deterministic ordering within Promise.all
- Use Math.trunc instead of Math.floor for mtime-based ETags
  (semantically correct for truncating to whole seconds)

* address bonk review comments: counters post-write, Math.floor, Content-Length comment, warn log level

- precompress.ts: move filesCompressed++ and totalOriginalBytes+= to after
  writes complete so partial failures (e.g. ENOSPC) don't inflate totals
- precompress.ts: add comment clarifying that progress tracks all files
  including skipped ones below MIN_SIZE, unlike filesCompressed
- static-file-cache.ts: Math.trunc -> Math.floor for mtime-based ETags
  (matches send/Express convention; correct for pre-epoch mtimes)
- prod-server.ts: add comment explaining Content-Length is intentionally
  omitted on slow-path compressed responses (chunked transfer encoding)
- prod-server.ts: downgrade all three pipeline error logs from console.error
  to console.warn and add comments explaining the destroy-only pattern

* address bonk: slow-path ETag+304 parity, visible precompress error log

- prod-server.ts slow path: add mtime-based ETag (W/"size-mtime") and
  304 Not Modified support, matching fast-path cache behaviour; also
  propagate mtimeMs through resolveStaticFile/statIfFile
- index.ts: add console.error in precompress .catch() so build errors are
  visible even if the SSR closeBundle hook never fires

* address bonk: slow-path 304 Vary header, HAS_ZSTD comment

- prod-server.ts slow path: include Vary: Accept-Encoding in 304
  responses when the content type is compressible, matching fast-path
  notModifiedHeaders behaviour and preventing shared caches from
  collapsing compressed/uncompressed representations
- prod-server.ts fast path: add comment explaining HAS_ZSTD is not
  checked during precompressed variant selection — we're serving a
  pre-existing .zst file from disk, not invoking zstdCompress() at
  runtime; HAS_ZSTD only guards the slow-path on-the-fly compressor

* address bonk: clarify Vary/compress=false intent in slow-path 304 comment

Expand the 304 comment to explain that Vary: Accept-Encoding is
intentionally omitted when compress=false (matching fast-path behaviour
where compress=false skips all compressed variants), and note that
spreading undefined is a deliberate no-op per ES2018+.

* address bonk: readFile-before-stat comment, document VINEXT_PRECOMPRESS env var

- precompress.ts: add comment on MIN_SIZE early-return explaining the
  read-first vs stat-first tradeoff (one wasted read vs extra syscall
  per tiny file; read-first cheaper for typical hashed asset sets)
- index.ts: document VINEXT_PRECOMPRESS=1 env var in VinextOptions.precompress
  JSDoc alongside the existing --precompress CLI flag mention

* address bonk: filename-hash ETag on slow path, entry-level dedup in buffer pass

- static-file-cache.ts: export etagFromFilenameHash so prod-server can
  reuse the same filename-hash extraction logic
- prod-server.ts: slow path now uses etagFromFilenameHash for /assets/*
  files, falling back to mtime-based ETag for non-hashed files; this
  matches fast-path cache behaviour and makes ETags stable across deploys
- static-file-cache.ts buffer pass: deduplicate at the StaticFileEntry
  level (using a seenEntries Set) rather than at the variant path level;
  HTML aliases share the same entry by reference so the old approach
  iterated variants multiple times for sites with many HTML pages

* address bonk: fix etagFromFilenameHash arg, explicit re-run safety comment

- prod-server.ts: use resolved.path (not pathname) for etagFromFilenameHash
  so that ext and the hash extraction always come from the same file;
  pathname and resolved.path diverge after HTML fallback (e.g. a request
  for /assets/widget-abc123 resolving to widget-abc123.html would have
  extracted a hash from the wrong name)
- precompress.ts: add inline comment on the COMPRESSIBLE_EXTENSIONS filter
  making explicit that .br/.gz/.zst are intentionally absent, so
  precompressed variants from a previous run are never re-compressed

* address bonk: add slow-path 304 test coverage

---------

Co-authored-by: James <james@eli.cx>
2026-04-01 23:33:12 +01:00

196 lines
8.2 KiB
TypeScript

/**
* Tests for build-time precompression of hashed static assets.
*
* precompressAssets() runs after `vinext build` and generates .br (brotli) and
* .gz (gzip) files alongside compressible hashed assets in dist/client/assets/.
* This eliminates per-request compression overhead for immutable build output.
*/
import { describe, it, expect, beforeEach, afterEach } from "vite-plus/test";
import fsp from "node:fs/promises";
import fs from "node:fs";
import path from "node:path";
import os from "node:os";
import zlib from "node:zlib";
import { precompressAssets } from "../packages/vinext/src/build/precompress.js";
/** Write a file with repeated content to ensure it exceeds compression threshold. */
async function writeAsset(clientDir: string, relativePath: string, content: string): Promise<void> {
const fullPath = path.join(clientDir, relativePath);
await fsp.mkdir(path.dirname(fullPath), { recursive: true });
await fsp.writeFile(fullPath, content);
}
describe("precompressAssets", () => {
let clientDir: string;
beforeEach(async () => {
clientDir = path.join(
os.tmpdir(),
`vinext-precompress-${Date.now()}-${Math.random().toString(36).slice(2)}`,
);
await fsp.mkdir(clientDir, { recursive: true });
});
afterEach(async () => {
await fsp.rm(clientDir, { recursive: true, force: true });
});
it("generates .br and .gz files for compressible hashed assets", async () => {
const jsContent = "const x = 1;\n".repeat(200); // well above 1KB threshold
await writeAsset(clientDir, "assets/index-abc123.js", jsContent);
const result = await precompressAssets(clientDir);
// Both compressed variants should exist on disk
expect(fs.existsSync(path.join(clientDir, "assets/index-abc123.js.br"))).toBe(true);
expect(fs.existsSync(path.join(clientDir, "assets/index-abc123.js.gz"))).toBe(true);
// Result should report what was compressed
expect(result.filesCompressed).toBe(1);
});
it("brotli output decompresses to original content", async () => {
const jsContent = "export function hello() { return 'world'; }\n".repeat(100);
await writeAsset(clientDir, "assets/hello-def456.js", jsContent);
await precompressAssets(clientDir);
const brBuffer = await fsp.readFile(path.join(clientDir, "assets/hello-def456.js.br"));
const decompressed = zlib.brotliDecompressSync(brBuffer).toString();
expect(decompressed).toBe(jsContent);
});
it("gzip output decompresses to original content", async () => {
const cssContent = ".container { display: flex; }\n".repeat(100);
await writeAsset(clientDir, "assets/styles-789abc.css", cssContent);
await precompressAssets(clientDir);
const gzBuffer = await fsp.readFile(path.join(clientDir, "assets/styles-789abc.css.gz"));
const decompressed = zlib.gunzipSync(gzBuffer).toString();
expect(decompressed).toBe(cssContent);
});
it("skips files below the compression threshold", async () => {
// Tiny file — not worth compressing
await writeAsset(clientDir, "assets/tiny-aaa111.js", "const x = 1;");
const result = await precompressAssets(clientDir);
expect(fs.existsSync(path.join(clientDir, "assets/tiny-aaa111.js.br"))).toBe(false);
expect(fs.existsSync(path.join(clientDir, "assets/tiny-aaa111.js.gz"))).toBe(false);
expect(result.filesCompressed).toBe(0);
});
it("skips non-compressible file types (images, fonts)", async () => {
// PNG file (binary, already compressed)
const pngHeader = Buffer.alloc(2048, 0x89); // fake PNG data, above threshold
await writeAsset(clientDir, "assets/logo-bbb222.png", pngHeader.toString("binary"));
// WOFF2 font (already compressed)
const fontData = Buffer.alloc(2048, 0x77);
await writeAsset(clientDir, "assets/font-ccc333.woff2", fontData.toString("binary"));
const result = await precompressAssets(clientDir);
expect(fs.existsSync(path.join(clientDir, "assets/logo-bbb222.png.br"))).toBe(false);
expect(fs.existsSync(path.join(clientDir, "assets/font-ccc333.woff2.br"))).toBe(false);
expect(result.filesCompressed).toBe(0);
});
it("returns empty result for missing assets/ directory", async () => {
// clientDir exists but has no assets/ subdirectory
const result = await precompressAssets(clientDir);
expect(result.filesCompressed).toBe(0);
expect(result.totalOriginalBytes).toBe(0);
expect(result.totalBrotliBytes).toBe(0);
});
it("compresses CSS files alongside JS", async () => {
const jsContent = "function render() {}\n".repeat(200);
const cssContent = "body { margin: 0; }\n".repeat(200);
await writeAsset(clientDir, "assets/app-aaa111.js", jsContent);
await writeAsset(clientDir, "assets/app-bbb222.css", cssContent);
const result = await precompressAssets(clientDir);
expect(result.filesCompressed).toBe(2);
expect(fs.existsSync(path.join(clientDir, "assets/app-aaa111.js.br"))).toBe(true);
expect(fs.existsSync(path.join(clientDir, "assets/app-bbb222.css.br"))).toBe(true);
});
it("does not re-compress existing .br or .gz files", async () => {
const jsContent = "const x = 1;\n".repeat(200);
await writeAsset(clientDir, "assets/index-abc123.js", jsContent);
// Run twice — should not create .br.br or .gz.gz
await precompressAssets(clientDir);
await precompressAssets(clientDir);
expect(fs.existsSync(path.join(clientDir, "assets/index-abc123.js.br.br"))).toBe(false);
expect(fs.existsSync(path.join(clientDir, "assets/index-abc123.js.gz.gz"))).toBe(false);
});
it("handles nested directories under assets/", async () => {
const jsContent = "export default {}\n".repeat(200);
await writeAsset(clientDir, "assets/chunks/vendor-ddd444.js", jsContent);
const result = await precompressAssets(clientDir);
expect(result.filesCompressed).toBe(1);
expect(fs.existsSync(path.join(clientDir, "assets/chunks/vendor-ddd444.js.br"))).toBe(true);
expect(fs.existsSync(path.join(clientDir, "assets/chunks/vendor-ddd444.js.gz"))).toBe(true);
});
it("only compresses files inside assets/ not other client files", async () => {
const htmlContent = "<html><body>hello</body></html>\n".repeat(100);
const jsContent = "const x = 1;\n".repeat(200);
// This file is in client root, not in assets/ — should not be compressed
await writeAsset(clientDir, "index.html", htmlContent);
// This one is in assets/ — should be compressed
await writeAsset(clientDir, "assets/main-eee555.js", jsContent);
const result = await precompressAssets(clientDir);
expect(result.filesCompressed).toBe(1);
expect(fs.existsSync(path.join(clientDir, "index.html.br"))).toBe(false);
expect(fs.existsSync(path.join(clientDir, "assets/main-eee555.js.br"))).toBe(true);
});
it("generates .zst files alongside .br and .gz", async () => {
const jsContent = "const x = 1;\n".repeat(200);
await writeAsset(clientDir, "assets/zstd-aaa111.js", jsContent);
const result = await precompressAssets(clientDir);
expect(fs.existsSync(path.join(clientDir, "assets/zstd-aaa111.js.zst"))).toBe(true);
expect(fs.existsSync(path.join(clientDir, "assets/zstd-aaa111.js.br"))).toBe(true);
expect(fs.existsSync(path.join(clientDir, "assets/zstd-aaa111.js.gz"))).toBe(true);
expect(result.filesCompressed).toBe(1);
});
it("zstd output decompresses to original content", async () => {
const jsContent = "export function hello() { return 'world'; }\n".repeat(100);
await writeAsset(clientDir, "assets/hello-zstd.js", jsContent);
await precompressAssets(clientDir);
const zstdBuffer = await fsp.readFile(path.join(clientDir, "assets/hello-zstd.js.zst"));
const decompressed = zlib.zstdDecompressSync(zstdBuffer).toString();
expect(decompressed).toBe(jsContent);
});
it("reports total original and compressed byte sizes", async () => {
const jsContent = "const x = 1;\n".repeat(500);
await writeAsset(clientDir, "assets/big-fff666.js", jsContent);
const result = await precompressAssets(clientDir);
expect(result.totalOriginalBytes).toBe(jsContent.length);
// Compressed should be smaller than original for repetitive content
expect(result.totalBrotliBytes).toBeGreaterThan(0);
expect(result.totalBrotliBytes).toBeLessThan(result.totalOriginalBytes);
});
});