Files
Vance Ingalls 450bb01018 perf(producer): native WebGPU shader-blend via Dawn (opt-in, Mac-Metal)
EXPERIMENTAL spike. Adds a Node-side WebGPU compositor backed by the
`webgpu` npm package (Dawn) and wires it into the existing
shader-transition worker as an opt-in alternative to the CPU blend.
Default OFF — set `HF_DAWN_WEBGPU=1` or pass `--gpu-shader-blend` to the
CLI to engage.

The hf#677 chain (PRs #756-#760) gets us 1.95x on Mac via a CPU shader
worker pool, ring-buffered pipelining, and a hybrid layered/parallel
capture path. The remaining ceiling is the per-pixel JS blend itself —
scalar f64 math in v8. On any host with a real GPU (Mac/Metal,
Linux/Vulkan, Windows/D3D), moving the blend to the GPU via Dawn
should drop blend wall time on a 854x480 rgb48le frame from
~150-910 ms (depending on shader complexity) to a few ms, projecting
3-5x end-to-end on top of the existing cascade — see the
`reference_5x_shader_perf_alternatives.md` memo (option B).

beginFrame is structurally Mac-unavailable
(crbug.com/40656275); native WebGPU is the next-best Mac-viable lever.

The Dawn binding (`webgpu@0.4.0`, dawn-gpu/node-webgpu, maintained by
Dawn upstream — Corentin Wallez, Kai Ninomiya) ships a `darwin-universal`
prebuilt that targets Apple Metal directly, plus Linux x64/arm64
Vulkan and Windows x64 D3D. Verified locally that the package loads on
Linux x64; adapter init returns null in the sandbox (no Vulkan driver)
and the worker transparently falls back to the existing CPU path.
Mac numbers must be measured on Vance's laptop — there's no GPU host
in CI.

- `packages/producer/src/services/shaderTransitionGpu.ts` (NEW)
  Node-side compositor: dynamic-imports `webgpu`, requests an adapter
  + device, compiles a WGSL compute shader per supported transition,
  manages per-size GPU resources, dispatches blends, reads back to
  `rgb48le`. Returns structured init failures rather than throwing —
  the caller can always fall back to CPU. `HF_DAWN_FORCE_FAIL=1`
  short-circuits init for testability of the fallback path.
- `packages/producer/src/services/shaderTransitionWorker.ts`
  Augmented: on the first message, if `HF_DAWN_WEBGPU=1`, dynamic-imports
  the GPU module and probes once. On success, supported shaders run on
  the GPU. Unsupported shaders (`glitch`, `domain-warp`, `swirl-vortex`,
  the rest) and any host without a GPU adapter fall through to the
  existing `TRANSITIONS[shader] ?? crossfade` CPU path with zero
  behavior change. Mid-render GPU failure disables the GPU path for the
  rest of the worker's life and falls back — the frame still completes.
- `packages/cli/src/commands/render.ts`
  New `--gpu-shader-blend` boolean flag (default false). When set, the
  CLI exports `HF_DAWN_WEBGPU=1` into `process.env` before any worker
  pool spawns. Env-var plumbing chosen over threading the flag through
  the orchestrator -> stage -> pool -> worker chain because env vars
  cross the `worker_threads` boundary unchanged.
- `packages/producer/build.mjs` + `packages/cli/tsup.config.ts`
  `webgpu` added to the `external` list in both bundlers. The Dawn
  binding ships a 70+ MB native `.dawn.node` binary per platform —
  it must be loaded from the user's node_modules at runtime, not
  inlined into the bundle.
- `packages/producer/package.json` + `packages/cli/package.json`
  `webgpu@^0.4.0` added to `optionalDependencies` on both. Optional
  because (a) it's a native binary that may fail to install on some
  hosts and (b) the feature is opt-in.
- `packages/producer/src/services/shaderTransitionGpu.test.ts` (NEW)
  3 vitest tests: `HF_DAWN_FORCE_FAIL` short-circuit, init never
  throws, and PSNR-vs-CPU >= 50 dB on hosts where the adapter is
  available (auto-skipped on Linux sandbox with a log line).

One representative shader (`crossfade`) is ported to WGSL end-to-end
as proof of correctness. The harness is shader-agnostic; porting more
is a mechanical add to `SHADERS_WGSL` in `shaderTransitionGpu.ts`.
Unsupported shaders transparently fall through to CPU even when the
flag is on.

GPU compute uses f32 + u16 storage; CPU canonical uses f64. Bit-exact
equality is not realistic. The default-off path keeps CI byte-equality
pins intact (the existing fixtures hit the CPU path unchanged). Fixtures
that exercise the GPU path must use a PSNR pin (>= 50 dB documented in
the new test). The fallback CPU path is bit-exact with the
canonical CPU implementation.

Bundled CLI worker smoke (854x480 single crossfade blend, see the
investigation log for the harness):

| config | wall | notes |
|---|---:|---|
| CPU baseline (`HF_DAWN_WEBGPU` unset) | 67 ms | reference |
| `HF_DAWN_WEBGPU=1` (fell back, no GPU) | 70 ms | +3 ms init+probe, then CPU |
| `HF_DAWN_WEBGPU=1` + `HF_DAWN_FORCE_FAIL=1` | 66 ms | fallback verified |

Both fallback runs produced byte-identical output to the CPU baseline
checksum — fallback fidelity confirmed.

**Mac numbers: Vance, please fill in.** Pull the branch, run a
shader-transition fixture both with and without `--gpu-shader-blend`,
and drop the wall-time pair in a comment.

Anywhere the GPU path can't run, the existing CPU path runs unchanged:

- `webgpu` package not installed (optional dep skipped on install) -> CPU
- `webgpu` loads but `requestAdapter()` returns null (no GPU) -> CPU
- WGSL pipeline compile fails -> CPU
- `--gpu-shader-blend` set but shader not in `SHADERS_WGSL` -> CPU
- Mid-render GPU failure -> log + disable GPU + finish on CPU

Failure reason is logged once per worker (not per frame). No crash path.

- Port the remaining 12 shaders to WGSL (mechanical: `flashThroughWhite`,
  `chromaticSplit`, `sdfIris`, `glitch`, `lightLeak`, `crossWarpMorph`,
  `whipPan`, `cinematicZoom`, `gravitationalLens`, `rippleWaves`,
  `swirlVortex`, `thermalDistortion`, `domainWarp`, `ridgedBurn`).
- Once Mac wall-time confirms the lever, decide on flip-default-on or
  keep opt-in.
- Update fixture pins for any fixture that needs to exercise the GPU
  path under CI (PSNR-only).
- A persistent device + pipeline cache shared across workers (currently
  per-worker init).

_PR drafted by Vai_

Co-Authored-By: Vai <vai@heygen.com>
2026-05-14 15:56:44 +00:00
..

@hyperframes/producer

Full HTML-to-video rendering pipeline: capture frames with Chrome's BeginFrame API, encode with FFmpeg, mix audio — all in one call.

Install

npm install @hyperframes/producer

Requirements: Node.js >= 22, Chrome/Chromium (auto-downloaded), FFmpeg

Usage

Render a video

import { createRenderJob, executeRenderJob } from "@hyperframes/producer";

const job = createRenderJob({
  inputPath: "./my-composition.html",
  outputPath: "./output.mp4",
  width: 1920,
  height: 1080,
  fps: 30,
});

const result = await executeRenderJob(job, (progress) => {
  console.log(`${Math.round(progress.percent * 100)}%`);
});

console.log(result.outputPath); // ./output.mp4

Run as an HTTP server

The producer can also run as a render server, accepting render requests over HTTP:

import { startServer } from "@hyperframes/producer";

await startServer({ port: 8080 });
// POST /render with a RenderConfig body

Configuration

RenderConfig controls the render pipeline:

Option Default Description
inputPath — Path to the HTML composition
outputPath — Output video file path (or directory, for format: "png-sequence")
width 1920 Frame width in pixels
height 1080 Frame height in pixels
fps 30 Frames per second (24, 30, or 60)
quality "standard" Encoder preset ("draft", "standard", "high")
format "mp4" Output container — "mp4", "webm", "mov", or "png-sequence". See Transparent Video Output below.

Transparent Video Output

The producer can render HTML compositions to formats that carry a true alpha channel — not chroma key. The same composition that renders an opaque MP4 renders a layerable overlay when you set format.

format Codec / pixel format Alpha Audio Use case
"mp4" (default) H.264 (yuv420p) or H.265 + HDR10 No AAC Streaming, social, default deliverable
"webm" VP9 + yuva420p True alpha Opus Web playback as overlay (<video> over background); supported in Chrome, Edge, Firefox
"mov" ProRes 4444 + yuva444p10le True alpha + 10-bit AAC Editor ingest (Premiere, Final Cut Pro, DaVinci Resolve)
"png-sequence" Numbered RGBA PNGs in a directory Lossless alpha Sidecar audio.aac After Effects / Nuke / Fusion, or pipelines that post-process frames before encoding

Example

import { createRenderJob, executeRenderJob } from "@hyperframes/producer";

const job = createRenderJob({
  inputPath: "./my-composition.html",
  outputPath: "./output.webm", // or a directory for "png-sequence"
  width: 1080,
  height: 1920,
  fps: 30,
  format: "webm", // "mp4" | "webm" | "mov" | "png-sequence"
});

await executeRenderJob(job);

What "transparent background" means here

The producer captures Chrome screenshots with the page background forced transparent (html, body, [data-composition-id] { background: transparent !important }) and the CDP default background override set to RGBA 0,0,0,0. The captured PNGs carry a real alpha channel and that channel is preserved end-to-end:

  • VP9 (webm) is encoded with -pix_fmt yuva420p, -auto-alt-ref 0, and alpha_mode=1 metadata.
  • ProRes 4444 (mov) is encoded with -pix_fmt yuva444p10le.
  • PNG sequences are written without re-encoding (zero-padded frame_NNNNNN.png).

This is not chroma keying. There is no green/blue background to remove and no "key" tolerance to tune — pixels that were transparent in the browser are transparent in the output.

Caveats

  • Linux + alpha forces screenshot capture. Chrome's BeginFrame compositor (the default deterministic capture path on Linux headless-shell) does not preserve alpha; the orchestrator falls back to Page.captureScreenshot, which is slower per frame. macOS and Windows already use screenshot mode by default, so they are unaffected.
  • HDR + alpha is not supported. Setting hdr: true together with an alpha-capable format logs a warning and falls back to SDR. Use format: "mp4" for HDR10 output.
  • png-sequence does not produce a single muxed file. When the composition contains audio elements, an audio.aac sidecar is written alongside the PNGs in outputPath.
  • Safari + WebM alpha is incomplete. For broad browser playback of an alpha video, ship format: "mov" to your editor and re-encode for the codec your distribution target supports.

Authoring transparent compositions

Don't paint a fullscreen background in your HTML. The default body background is overridden to transparent automatically — any body { background: ... }, #root { background: ... }, or [data-composition-id] { background: ... } rule is force-overridden during alpha rendering. Backgrounds on inner elements (cards, scenes, components) are kept.

How it works

  1. Serve — spins up a local file server for the HTML composition
  2. Capture — opens the page in headless Chrome, seeks frame-by-frame via HeadlessExperimental.beginFrame (or Page.captureScreenshot for transparent / non-Linux renders), captures screenshots
  3. Encode — pipes frames through FFmpeg (with GPU encoder detection and chunked concat). Skipped for format: "png-sequence".
  4. Mix — extracts <audio> elements and mixes them into the final video. For png-sequence, audio is written as an audio.aac sidecar.
  5. Finalize — applies faststart for streaming-friendly MP4 (no-op for WebM, MOV, and png-sequence)

Documentation

Full documentation: hyperframes.heygen.com/packages/producer