mirror of
https://github.com/cloudflare/vinext.git
synced 2026-09-14 19:04:59 +08:00
6da45a680a
* perf: add build-time precompression for hashed static assets
Generate .br (brotli q11), .gz (gzip l9), and .zst (zstandard l19)
files alongside compressible assets in dist/client/assets/ during
vinext build. These are served directly by the production server,
eliminating per-request compression overhead for immutable build output.
Only targets assets/ (hashed, immutable). Public directory files still
use on-the-fly compression since they may change between deploys.
* perf: add startup metadata cache for zero-syscall static serving
StaticFileCache walks dist/client/ once at server boot and caches:
- File metadata (path, size, content-type, cache-control, etag)
- Pre-computed response headers per variant (original, br, gz, zst)
- In-memory buffers for small files (< 64KB) for res.end(buffer)
- Precompressed variant paths and sizes
Per-request serving is Map.get() + res.end(buffer) with zero filesystem
calls, zero object allocation, and zero header construction. Modeled
after sirv's production mode but with in-memory buffering for small
files which eliminates createReadStream fd overhead.
* perf: refactor tryServeStatic to async + cache + precompressed serving
Refactor tryServeStatic to use StaticFileCache for the hot path:
- Pre-computed response headers (zero object allocation per request)
- In-memory buffer serving for small precompressed files
- 304 Not Modified via ETag + If-None-Match
- HEAD request optimization (headers only, no body)
- Zstandard serving (zstd > br > gzip > original fallback chain)
- Async filesystem fallback for non-cached files (replaces blocking
existsSync + statSync)
- Skip decodeURIComponent for clean URLs (no % in path)
Wire StaticFileCache.create() into both startAppRouterServer and
startPagesRouterServer at startup. Integrate precompressAssets() into
the vinext build pipeline.
CONTENT_TYPES is now a single source of truth exported from
static-file-cache.ts (was duplicated in prod-server.ts).
* docs: fix stale comments in precompress (mention .zst)
* fix: deduplicate buffer reads for HTML alias entries, fix stale JSDoc
* docs: fix stale comment in cli.ts (mention zstd)
* fix: address code review findings
- Add Vary: Accept-Encoding to original variant when compressed siblings exist
- Use pipeline() instead of .pipe() for stream error safety
- Remove legacy accessor fields from StaticFileEntry (all new code)
- Exclude Content-Type from 304 responses per RFC 9110
- Fix misleading Accept-Encoding comment in prod-server
- Batch stat() calls in walkFilesWithStats (chunks of 64)
- Bound precompression concurrency to availableParallelism (max 16)
- Gracefully skip zstd when zlib.zstdCompress unavailable (Node <22.15)
- Add --no-precompress CLI flag to skip build-time compression
- Drop brotli to q5 (fast, nearly same ratio as q11 at build time)
- Max out zstd at level 22 (build time, no reason to hold back)
* fix: address remaining review findings in prod-server
- Gate negotiateEncoding zstd on runtime availability (HAS_ZSTD)
- Add ZSTD_e_flush for streaming mode (progressive SSR decodability)
- Serve HEAD without body on slow path (compressed + uncompressed)
- Handle pipeline errors instead of silently swallowing them
- Lowercase Accept-Encoding in fast path per RFC 9110
- Fix stale comments: add zstd to module header and variant detection
- Add tests for slow path, HEAD, and malformed percent-encoded pathnames
* perf: move precompression to Vite plugin with edge auto-detection
Move build-time precompression from CLI post-step into a writeBundle
plugin hook (vinext:precompress). The plugin already knows the
deployment target — auto-skips for Cloudflare Workers and Nitro where
the CDN handles compression at the edge.
- Add precompress option to VinextOptions: 'auto' | boolean
- CLI --no-precompress forwards via VINEXT_NO_PRECOMPRESS env var
- Works for any build invocation (CLI, API, CI), not just vinext build
* docs: fix inaccurate comments in precompress and static-file-cache
- precompress: remove false idempotency claim (overwrites, not skips)
- static-file-cache: "frozen objects" → "reusable objects" (no freeze)
- static-file-cache: fix unrealistic compression ratio example
* fix: address code review findings
- Remove dead code: redundant .br/.gz/.zst extension checks already
caught by COMPRESSIBLE_EXTENSIONS filter
- Fix unreachable pipeline error handlers: res.writeHead() was called
before pipeline(), so res.headersSent was always true in the callback.
Use res.destroy(err) to cleanly tear down the socket on stream errors.
- Read file content lazily inside compression chunks instead of loading
all files upfront, keeping peak memory at O(CONCURRENCY * max_file_size)
* perf: tune compression levels and restore auto mode
- Drop zstd from level 22 to 8 (250x faster build, 7% larger output)
- Drop gzip from level 9 to 8 (marginal size difference, faster)
- Restore precompress: 'auto' | boolean config with auto as default
(auto-enables for Node.js targets, skips for Cloudflare/Nitro)
- Add progress bar for precompression step (matches prerender UX)
- Clarify compression ratio output as brotli-specific
* fix: use filename hash for ETag on hashed assets
Hashed assets (assets/app-DqZc3R4n.js) now derive their ETag from the
content hash in the filename instead of mtime. More robust across
Docker builds, CI artifacts, and cp -r where mtime changes between
deploys even when content is identical.
- Use weak validator (W/) since the tag is shared across content-encoded
variants (.br, .gz, .zst) which are byte-different (RFC 9110 §8.8.1)
- Fall back to mtime-based ETag when filename has no recognizable hash
(custom assetFileNames, plugin-emitted assets, public/ files)
* fix: make precompress opt-in per maintainer feedback
Real-world benchmark shows 13.5% build-time regression on fumadocs
with auto mode (2.4s → 2.7s). Switch to explicit opt-in via
`precompress: true` or `--precompress` CLI flag until the feature
stabilises.
* fix: address code review findings
- ETag regex: use lastIndexOf('-') to extract only the final hash
segment, avoiding capture of filename fragments in multi-hyphenated
names (e.g. `before-pop-state-D3R3Ien6` now extracts `D3R3Ien6`)
- Progress bar: track processed count (including skipped files) so
the bar always reaches 100%
- Tests: update ETag test for filename-hash behavior, add coverage
for mtime fallback on non-hashed and unhashed-in-assets files
* chore: use type instead of interface per lint rule
* fix final precompression review comments
* perf: optimize precompress build path
- Overlap precompression with SSR build phase (async writeBundle + closeBundle await)
- Lower concurrency cap from 16 to 8 (reduces zlib queue contention)
- Fix progress counter race in chunked processing
* fix: address final review feedback
- Fix progress counter race in precompress (single onProgress call site
after readFile, monotonic by construction)
- Add console.error before res.destroy in all pipeline error handlers
for operational visibility
- Chunk buffer reads at 64 concurrent to bound FD usage on large projects
- Defer createCompressor until after HEAD check on slow path
- Use Math.floor for mtime ETag (matches send/Express convention)
- Document Rolldown hash length assumption
* fix: address final bonk review nits
- Move progress reporting to after chunk completion to avoid
non-deterministic ordering within Promise.all
- Use Math.trunc instead of Math.floor for mtime-based ETags
(semantically correct for truncating to whole seconds)
* address bonk review comments: counters post-write, Math.floor, Content-Length comment, warn log level
- precompress.ts: move filesCompressed++ and totalOriginalBytes+= to after
writes complete so partial failures (e.g. ENOSPC) don't inflate totals
- precompress.ts: add comment clarifying that progress tracks all files
including skipped ones below MIN_SIZE, unlike filesCompressed
- static-file-cache.ts: Math.trunc -> Math.floor for mtime-based ETags
(matches send/Express convention; correct for pre-epoch mtimes)
- prod-server.ts: add comment explaining Content-Length is intentionally
omitted on slow-path compressed responses (chunked transfer encoding)
- prod-server.ts: downgrade all three pipeline error logs from console.error
to console.warn and add comments explaining the destroy-only pattern
* address bonk: slow-path ETag+304 parity, visible precompress error log
- prod-server.ts slow path: add mtime-based ETag (W/"size-mtime") and
304 Not Modified support, matching fast-path cache behaviour; also
propagate mtimeMs through resolveStaticFile/statIfFile
- index.ts: add console.error in precompress .catch() so build errors are
visible even if the SSR closeBundle hook never fires
* address bonk: slow-path 304 Vary header, HAS_ZSTD comment
- prod-server.ts slow path: include Vary: Accept-Encoding in 304
responses when the content type is compressible, matching fast-path
notModifiedHeaders behaviour and preventing shared caches from
collapsing compressed/uncompressed representations
- prod-server.ts fast path: add comment explaining HAS_ZSTD is not
checked during precompressed variant selection — we're serving a
pre-existing .zst file from disk, not invoking zstdCompress() at
runtime; HAS_ZSTD only guards the slow-path on-the-fly compressor
* address bonk: clarify Vary/compress=false intent in slow-path 304 comment
Expand the 304 comment to explain that Vary: Accept-Encoding is
intentionally omitted when compress=false (matching fast-path behaviour
where compress=false skips all compressed variants), and note that
spreading undefined is a deliberate no-op per ES2018+.
* address bonk: readFile-before-stat comment, document VINEXT_PRECOMPRESS env var
- precompress.ts: add comment on MIN_SIZE early-return explaining the
read-first vs stat-first tradeoff (one wasted read vs extra syscall
per tiny file; read-first cheaper for typical hashed asset sets)
- index.ts: document VINEXT_PRECOMPRESS=1 env var in VinextOptions.precompress
JSDoc alongside the existing --precompress CLI flag mention
* address bonk: filename-hash ETag on slow path, entry-level dedup in buffer pass
- static-file-cache.ts: export etagFromFilenameHash so prod-server can
reuse the same filename-hash extraction logic
- prod-server.ts: slow path now uses etagFromFilenameHash for /assets/*
files, falling back to mtime-based ETag for non-hashed files; this
matches fast-path cache behaviour and makes ETags stable across deploys
- static-file-cache.ts buffer pass: deduplicate at the StaticFileEntry
level (using a seenEntries Set) rather than at the variant path level;
HTML aliases share the same entry by reference so the old approach
iterated variants multiple times for sites with many HTML pages
* address bonk: fix etagFromFilenameHash arg, explicit re-run safety comment
- prod-server.ts: use resolved.path (not pathname) for etagFromFilenameHash
so that ext and the hash extraction always come from the same file;
pathname and resolved.path diverge after HTML fallback (e.g. a request
for /assets/widget-abc123 resolving to widget-abc123.html would have
extracted a hash from the wrong name)
- precompress.ts: add inline comment on the COMPRESSIBLE_EXTENSIONS filter
making explicit that .br/.gz/.zst are intentionally absent, so
precompressed variants from a previous run are never re-compressed
* address bonk: add slow-path 304 test coverage
---------
Co-authored-by: James <james@eli.cx>
196 lines
8.2 KiB
TypeScript
196 lines
8.2 KiB
TypeScript
/**
|
|
* Tests for build-time precompression of hashed static assets.
|
|
*
|
|
* precompressAssets() runs after `vinext build` and generates .br (brotli) and
|
|
* .gz (gzip) files alongside compressible hashed assets in dist/client/assets/.
|
|
* This eliminates per-request compression overhead for immutable build output.
|
|
*/
|
|
import { describe, it, expect, beforeEach, afterEach } from "vite-plus/test";
|
|
import fsp from "node:fs/promises";
|
|
import fs from "node:fs";
|
|
import path from "node:path";
|
|
import os from "node:os";
|
|
import zlib from "node:zlib";
|
|
import { precompressAssets } from "../packages/vinext/src/build/precompress.js";
|
|
|
|
/** Write a file with repeated content to ensure it exceeds compression threshold. */
|
|
async function writeAsset(clientDir: string, relativePath: string, content: string): Promise<void> {
|
|
const fullPath = path.join(clientDir, relativePath);
|
|
await fsp.mkdir(path.dirname(fullPath), { recursive: true });
|
|
await fsp.writeFile(fullPath, content);
|
|
}
|
|
|
|
describe("precompressAssets", () => {
|
|
let clientDir: string;
|
|
|
|
beforeEach(async () => {
|
|
clientDir = path.join(
|
|
os.tmpdir(),
|
|
`vinext-precompress-${Date.now()}-${Math.random().toString(36).slice(2)}`,
|
|
);
|
|
await fsp.mkdir(clientDir, { recursive: true });
|
|
});
|
|
|
|
afterEach(async () => {
|
|
await fsp.rm(clientDir, { recursive: true, force: true });
|
|
});
|
|
|
|
it("generates .br and .gz files for compressible hashed assets", async () => {
|
|
const jsContent = "const x = 1;\n".repeat(200); // well above 1KB threshold
|
|
await writeAsset(clientDir, "assets/index-abc123.js", jsContent);
|
|
|
|
const result = await precompressAssets(clientDir);
|
|
|
|
// Both compressed variants should exist on disk
|
|
expect(fs.existsSync(path.join(clientDir, "assets/index-abc123.js.br"))).toBe(true);
|
|
expect(fs.existsSync(path.join(clientDir, "assets/index-abc123.js.gz"))).toBe(true);
|
|
|
|
// Result should report what was compressed
|
|
expect(result.filesCompressed).toBe(1);
|
|
});
|
|
|
|
it("brotli output decompresses to original content", async () => {
|
|
const jsContent = "export function hello() { return 'world'; }\n".repeat(100);
|
|
await writeAsset(clientDir, "assets/hello-def456.js", jsContent);
|
|
|
|
await precompressAssets(clientDir);
|
|
|
|
const brBuffer = await fsp.readFile(path.join(clientDir, "assets/hello-def456.js.br"));
|
|
const decompressed = zlib.brotliDecompressSync(brBuffer).toString();
|
|
expect(decompressed).toBe(jsContent);
|
|
});
|
|
|
|
it("gzip output decompresses to original content", async () => {
|
|
const cssContent = ".container { display: flex; }\n".repeat(100);
|
|
await writeAsset(clientDir, "assets/styles-789abc.css", cssContent);
|
|
|
|
await precompressAssets(clientDir);
|
|
|
|
const gzBuffer = await fsp.readFile(path.join(clientDir, "assets/styles-789abc.css.gz"));
|
|
const decompressed = zlib.gunzipSync(gzBuffer).toString();
|
|
expect(decompressed).toBe(cssContent);
|
|
});
|
|
|
|
it("skips files below the compression threshold", async () => {
|
|
// Tiny file — not worth compressing
|
|
await writeAsset(clientDir, "assets/tiny-aaa111.js", "const x = 1;");
|
|
|
|
const result = await precompressAssets(clientDir);
|
|
|
|
expect(fs.existsSync(path.join(clientDir, "assets/tiny-aaa111.js.br"))).toBe(false);
|
|
expect(fs.existsSync(path.join(clientDir, "assets/tiny-aaa111.js.gz"))).toBe(false);
|
|
expect(result.filesCompressed).toBe(0);
|
|
});
|
|
|
|
it("skips non-compressible file types (images, fonts)", async () => {
|
|
// PNG file (binary, already compressed)
|
|
const pngHeader = Buffer.alloc(2048, 0x89); // fake PNG data, above threshold
|
|
await writeAsset(clientDir, "assets/logo-bbb222.png", pngHeader.toString("binary"));
|
|
|
|
// WOFF2 font (already compressed)
|
|
const fontData = Buffer.alloc(2048, 0x77);
|
|
await writeAsset(clientDir, "assets/font-ccc333.woff2", fontData.toString("binary"));
|
|
|
|
const result = await precompressAssets(clientDir);
|
|
|
|
expect(fs.existsSync(path.join(clientDir, "assets/logo-bbb222.png.br"))).toBe(false);
|
|
expect(fs.existsSync(path.join(clientDir, "assets/font-ccc333.woff2.br"))).toBe(false);
|
|
expect(result.filesCompressed).toBe(0);
|
|
});
|
|
|
|
it("returns empty result for missing assets/ directory", async () => {
|
|
// clientDir exists but has no assets/ subdirectory
|
|
const result = await precompressAssets(clientDir);
|
|
|
|
expect(result.filesCompressed).toBe(0);
|
|
expect(result.totalOriginalBytes).toBe(0);
|
|
expect(result.totalBrotliBytes).toBe(0);
|
|
});
|
|
|
|
it("compresses CSS files alongside JS", async () => {
|
|
const jsContent = "function render() {}\n".repeat(200);
|
|
const cssContent = "body { margin: 0; }\n".repeat(200);
|
|
await writeAsset(clientDir, "assets/app-aaa111.js", jsContent);
|
|
await writeAsset(clientDir, "assets/app-bbb222.css", cssContent);
|
|
|
|
const result = await precompressAssets(clientDir);
|
|
|
|
expect(result.filesCompressed).toBe(2);
|
|
expect(fs.existsSync(path.join(clientDir, "assets/app-aaa111.js.br"))).toBe(true);
|
|
expect(fs.existsSync(path.join(clientDir, "assets/app-bbb222.css.br"))).toBe(true);
|
|
});
|
|
|
|
it("does not re-compress existing .br or .gz files", async () => {
|
|
const jsContent = "const x = 1;\n".repeat(200);
|
|
await writeAsset(clientDir, "assets/index-abc123.js", jsContent);
|
|
|
|
// Run twice — should not create .br.br or .gz.gz
|
|
await precompressAssets(clientDir);
|
|
await precompressAssets(clientDir);
|
|
|
|
expect(fs.existsSync(path.join(clientDir, "assets/index-abc123.js.br.br"))).toBe(false);
|
|
expect(fs.existsSync(path.join(clientDir, "assets/index-abc123.js.gz.gz"))).toBe(false);
|
|
});
|
|
|
|
it("handles nested directories under assets/", async () => {
|
|
const jsContent = "export default {}\n".repeat(200);
|
|
await writeAsset(clientDir, "assets/chunks/vendor-ddd444.js", jsContent);
|
|
|
|
const result = await precompressAssets(clientDir);
|
|
|
|
expect(result.filesCompressed).toBe(1);
|
|
expect(fs.existsSync(path.join(clientDir, "assets/chunks/vendor-ddd444.js.br"))).toBe(true);
|
|
expect(fs.existsSync(path.join(clientDir, "assets/chunks/vendor-ddd444.js.gz"))).toBe(true);
|
|
});
|
|
|
|
it("only compresses files inside assets/ not other client files", async () => {
|
|
const htmlContent = "<html><body>hello</body></html>\n".repeat(100);
|
|
const jsContent = "const x = 1;\n".repeat(200);
|
|
// This file is in client root, not in assets/ — should not be compressed
|
|
await writeAsset(clientDir, "index.html", htmlContent);
|
|
// This one is in assets/ — should be compressed
|
|
await writeAsset(clientDir, "assets/main-eee555.js", jsContent);
|
|
|
|
const result = await precompressAssets(clientDir);
|
|
|
|
expect(result.filesCompressed).toBe(1);
|
|
expect(fs.existsSync(path.join(clientDir, "index.html.br"))).toBe(false);
|
|
expect(fs.existsSync(path.join(clientDir, "assets/main-eee555.js.br"))).toBe(true);
|
|
});
|
|
|
|
it("generates .zst files alongside .br and .gz", async () => {
|
|
const jsContent = "const x = 1;\n".repeat(200);
|
|
await writeAsset(clientDir, "assets/zstd-aaa111.js", jsContent);
|
|
|
|
const result = await precompressAssets(clientDir);
|
|
|
|
expect(fs.existsSync(path.join(clientDir, "assets/zstd-aaa111.js.zst"))).toBe(true);
|
|
expect(fs.existsSync(path.join(clientDir, "assets/zstd-aaa111.js.br"))).toBe(true);
|
|
expect(fs.existsSync(path.join(clientDir, "assets/zstd-aaa111.js.gz"))).toBe(true);
|
|
expect(result.filesCompressed).toBe(1);
|
|
});
|
|
|
|
it("zstd output decompresses to original content", async () => {
|
|
const jsContent = "export function hello() { return 'world'; }\n".repeat(100);
|
|
await writeAsset(clientDir, "assets/hello-zstd.js", jsContent);
|
|
|
|
await precompressAssets(clientDir);
|
|
|
|
const zstdBuffer = await fsp.readFile(path.join(clientDir, "assets/hello-zstd.js.zst"));
|
|
const decompressed = zlib.zstdDecompressSync(zstdBuffer).toString();
|
|
expect(decompressed).toBe(jsContent);
|
|
});
|
|
|
|
it("reports total original and compressed byte sizes", async () => {
|
|
const jsContent = "const x = 1;\n".repeat(500);
|
|
await writeAsset(clientDir, "assets/big-fff666.js", jsContent);
|
|
|
|
const result = await precompressAssets(clientDir);
|
|
|
|
expect(result.totalOriginalBytes).toBe(jsContent.length);
|
|
// Compressed should be smaller than original for repetitive content
|
|
expect(result.totalBrotliBytes).toBeGreaterThan(0);
|
|
expect(result.totalBrotliBytes).toBeLessThan(result.totalOriginalBytes);
|
|
});
|
|
});
|