Vance Ingalls 450bb01018 perf(producer): native WebGPU shader-blend via Dawn (opt-in, Mac-Metal)
EXPERIMENTAL spike. Adds a Node-side WebGPU compositor backed by the
`webgpu` npm package (Dawn) and wires it into the existing
shader-transition worker as an opt-in alternative to the CPU blend.
Default OFF — set `HF_DAWN_WEBGPU=1` or pass `--gpu-shader-blend` to the
CLI to engage.

The hf#677 chain (PRs #756-#760) gets us 1.95x on Mac via a CPU shader
worker pool, ring-buffered pipelining, and a hybrid layered/parallel
capture path. The remaining ceiling is the per-pixel JS blend itself —
scalar f64 math in v8. On any host with a real GPU (Mac/Metal,
Linux/Vulkan, Windows/D3D), moving the blend to the GPU via Dawn
should drop blend wall time on a 854x480 rgb48le frame from
~150-910 ms (depending on shader complexity) to a few ms, projecting
3-5x end-to-end on top of the existing cascade — see the
`reference_5x_shader_perf_alternatives.md` memo (option B).

beginFrame is structurally Mac-unavailable
(crbug.com/40656275); native WebGPU is the next-best Mac-viable lever.

The Dawn binding (`webgpu@0.4.0`, dawn-gpu/node-webgpu, maintained by
Dawn upstream — Corentin Wallez, Kai Ninomiya) ships a `darwin-universal`
prebuilt that targets Apple Metal directly, plus Linux x64/arm64
Vulkan and Windows x64 D3D. Verified locally that the package loads on
Linux x64; adapter init returns null in the sandbox (no Vulkan driver)
and the worker transparently falls back to the existing CPU path.
Mac numbers must be measured on Vance's laptop — there's no GPU host
in CI.

- `packages/producer/src/services/shaderTransitionGpu.ts` (NEW)
  Node-side compositor: dynamic-imports `webgpu`, requests an adapter
  + device, compiles a WGSL compute shader per supported transition,
  manages per-size GPU resources, dispatches blends, reads back to
  `rgb48le`. Returns structured init failures rather than throwing —
  the caller can always fall back to CPU. `HF_DAWN_FORCE_FAIL=1`
  short-circuits init for testability of the fallback path.
- `packages/producer/src/services/shaderTransitionWorker.ts`
  Augmented: on the first message, if `HF_DAWN_WEBGPU=1`, dynamic-imports
  the GPU module and probes once. On success, supported shaders run on
  the GPU. Unsupported shaders (`glitch`, `domain-warp`, `swirl-vortex`,
  the rest) and any host without a GPU adapter fall through to the
  existing `TRANSITIONS[shader] ?? crossfade` CPU path with zero
  behavior change. Mid-render GPU failure disables the GPU path for the
  rest of the worker's life and falls back — the frame still completes.
- `packages/cli/src/commands/render.ts`
  New `--gpu-shader-blend` boolean flag (default false). When set, the
  CLI exports `HF_DAWN_WEBGPU=1` into `process.env` before any worker
  pool spawns. Env-var plumbing chosen over threading the flag through
  the orchestrator -> stage -> pool -> worker chain because env vars
  cross the `worker_threads` boundary unchanged.
- `packages/producer/build.mjs` + `packages/cli/tsup.config.ts`
  `webgpu` added to the `external` list in both bundlers. The Dawn
  binding ships a 70+ MB native `.dawn.node` binary per platform —
  it must be loaded from the user's node_modules at runtime, not
  inlined into the bundle.
- `packages/producer/package.json` + `packages/cli/package.json`
  `webgpu@^0.4.0` added to `optionalDependencies` on both. Optional
  because (a) it's a native binary that may fail to install on some
  hosts and (b) the feature is opt-in.
- `packages/producer/src/services/shaderTransitionGpu.test.ts` (NEW)
  3 vitest tests: `HF_DAWN_FORCE_FAIL` short-circuit, init never
  throws, and PSNR-vs-CPU >= 50 dB on hosts where the adapter is
  available (auto-skipped on Linux sandbox with a log line).

One representative shader (`crossfade`) is ported to WGSL end-to-end
as proof of correctness. The harness is shader-agnostic; porting more
is a mechanical add to `SHADERS_WGSL` in `shaderTransitionGpu.ts`.
Unsupported shaders transparently fall through to CPU even when the
flag is on.

GPU compute uses f32 + u16 storage; CPU canonical uses f64. Bit-exact
equality is not realistic. The default-off path keeps CI byte-equality
pins intact (the existing fixtures hit the CPU path unchanged). Fixtures
that exercise the GPU path must use a PSNR pin (>= 50 dB documented in
the new test). The fallback CPU path is bit-exact with the
canonical CPU implementation.

Bundled CLI worker smoke (854x480 single crossfade blend, see the
investigation log for the harness):

| config | wall | notes |
|---|---:|---|
| CPU baseline (`HF_DAWN_WEBGPU` unset) | 67 ms | reference |
| `HF_DAWN_WEBGPU=1` (fell back, no GPU) | 70 ms | +3 ms init+probe, then CPU |
| `HF_DAWN_WEBGPU=1` + `HF_DAWN_FORCE_FAIL=1` | 66 ms | fallback verified |

Both fallback runs produced byte-identical output to the CPU baseline
checksum — fallback fidelity confirmed.

**Mac numbers: Vance, please fill in.** Pull the branch, run a
shader-transition fixture both with and without `--gpu-shader-blend`,
and drop the wall-time pair in a comment.

Anywhere the GPU path can't run, the existing CPU path runs unchanged:

- `webgpu` package not installed (optional dep skipped on install) -> CPU
- `webgpu` loads but `requestAdapter()` returns null (no GPU) -> CPU
- WGSL pipeline compile fails -> CPU
- `--gpu-shader-blend` set but shader not in `SHADERS_WGSL` -> CPU
- Mid-render GPU failure -> log + disable GPU + finish on CPU

Failure reason is logged once per worker (not per frame). No crash path.

- Port the remaining 12 shaders to WGSL (mechanical: `flashThroughWhite`,
  `chromaticSplit`, `sdfIris`, `glitch`, `lightLeak`, `crossWarpMorph`,
  `whipPan`, `cinematicZoom`, `gravitationalLens`, `rippleWaves`,
  `swirlVortex`, `thermalDistortion`, `domainWarp`, `ridgedBurn`).
- Once Mac wall-time confirms the lever, decide on flip-default-on or
  keep opt-in.
- Update fixture pins for any fixture that needs to exercise the GPU
  path under CI (PSNR-only).
- A persistent device + pipeline cache shared across workers (currently
  per-worker init).

_PR drafted by Vai_

Co-Authored-By: Vai <vai@heygen.com>
2026-05-14 15:56:44 +00:00
2026-05-13 13:40:16 -07:00
2026-03-21 22:43:56 -07:00

HyperFrames

npm version npm downloads License Node.js Discord

Write HTML. Render video. Built for agents.

HyperFrames demo — HTML code on the left transforms into a rendered video on the right

Hyperframes is an open-source video rendering framework that lets you create, preview, and render HTML-based video compositions — with first-class support for AI agents.

Quick Start

Install the HyperFrames skills, then describe the video you want:

npx skills add heygen-com/hyperframes

This teaches your agent (Claude Code, Cursor, Gemini CLI, Codex) how to write correct compositions, GSAP timelines, Tailwind v4 browser-runtime styles, and first-party adapter animations. In Claude Code, the skills register as slash commands — invoke /hyperframes to author compositions, /hyperframes-cli for the dev-loop commands (init, lint, preview, render), /hyperframes-media for asset preprocessing (TTS, transcription, background removal), /tailwind for init --tailwind projects, /gsap for timeline animation help, or the adapter skills (/animejs, /css-animations, /lottie, /three, /waapi) when a composition uses those runtimes.

For Claude Design, open docs/guides/claude-design-hyperframes.md on GitHub and click the download button (↓) to save it, then attach the file to your Claude Design chat. It produces a valid first draft; refine in any AI coding agent. See the Claude Design guide.

For Codex specifically, the same skills are also exposed as an OpenAI Codex plugin — sparse-install just the plugin surface:

codex plugin marketplace add heygen-com/hyperframes --sparse .codex-plugin --sparse skills --sparse assets

For Claude Code, the repo also ships a Claude Code plugin manifest: test it locally with claude --plugin-dir .. The manifest intentionally omits skills because Claude Code auto-discovers the root skills/ directory by convention, and for marketplace submission use the title HyperFrames by HeyGen plus the black/white icon assets at assets/claude-code-icon-dark.svg and assets/claude-code-icon-light.svg for the two theme slots. For Cursor, the same skills are packaged as a Cursor plugin — install from the Cursor Marketplace, or sideload by cloning this repo and pointing Settings → Plugins → Load unpacked at the repo root.

Try it: example prompts

Copy any of these into your agent to get started. The /hyperframes prefix loads the skill context explicitly so you get correct output the first time.

Cold start — describe what you want:

Using /hyperframes, create a 10-second product intro with a fade-in title, a background video, and background music.

Warm start — turn existing context into a video:

Take a look at this GitHub repo https://github.com/heygen-com/hyperframes and explain its uses and architecture to me using /hyperframes.

Summarize the attached PDF into a 45-second pitch video using /hyperframes.

Turn this CSV into an animated bar chart race using /hyperframes.

Format-specific:

Make a 9:16 TikTok-style hook video about [topic] using /hyperframes, with bouncy captions synced to a TTS narration.

Iterate — talk to the agent like a video editor:

Make the title 2x bigger, swap to dark mode, and add a fade-out at the end.

Add a lower third at 0:03 with my name and title.

The agent handles scaffolding, animation, and rendering. See the prompting guide for more patterns.

Option 2: Start a project manually

npx hyperframes init my-video
cd my-video
npx hyperframes preview      # preview in browser (live reload)
npx hyperframes render       # render to MP4

hyperframes init installs skills automatically, so you can hand off to your AI agent at any point.

Requirements: Node.js >= 22, FFmpeg

Why Hyperframes?

  • HTML-native — compositions are HTML files with data attributes. No React, no proprietary DSL.
  • AI-first — agents already speak HTML. The CLI is non-interactive by default, designed for agent-driven workflows.
  • Deterministic rendering — same input = identical output. Built for automated pipelines.
  • Frame Adapter pattern — bring your own animation runtime (GSAP, Lottie, CSS, Three.js).

Hyperframes vs Remotion

Hyperframes is inspired by Remotion — we used Remotion at HeyGen in production, learned a ton from it, and kept attribution comments in the source for the patterns it pioneered (Chrome launch flags, image2pipe → FFmpeg streaming, frame buffering). Both tools drive headless Chrome and both are deterministic. They differ on one decision: what the primary author writes. Remotion's bet is React components; Hyperframes' bet is HTML.

Hyperframes Remotion
Authoring HTML + CSS + GSAP React components (TSX)
Build step None; index.html plays as-is Required (bundler)
Library-clock animations (GSAP, Anime.js, Motion One) Seekable, frame-accurate Plays at wall-clock during render
Arbitrary HTML / CSS passthrough Paste and animate Rewrite as JSX
Distributed rendering Single-machine today Lambda, production-ready

Licensing: fully open source vs source-available

Hyperframes is completely open source under Apache 2.0 — an OSI-approved license. Use it commercially at any scale, with no per-render fees, no seat caps, no company-size thresholds.

Remotion is source-available, not open source. The code is on GitHub under a custom Remotion License that requires a paid company license above small-team thresholds. It's a great product with a real team behind it — but if open-source licensing matters to you (OSI compliance, redistribution rights, no per-use fees), that's a first-order decision point.

Full write-up with benchmarks, an honest list of where each tool wins, and a GSAP side-by-side: Hyperframes vs Remotion guide.

How It Works

Define your video as HTML with data attributes:

<div id="stage" data-composition-id="my-video" data-start="0" data-width="1920" data-height="1080">
  <video
    id="clip-1"
    data-start="0"
    data-duration="5"
    data-track-index="0"
    src="intro.mp4"
    muted
    playsinline
  ></video>
  <img
    id="overlay"
    class="clip"
    data-start="2"
    data-duration="3"
    data-track-index="1"
    src="logo.png"
  />
  <audio
    id="bg-music"
    data-start="0"
    data-duration="9"
    data-track-index="2"
    data-volume="0.5"
    src="music.wav"
  ></audio>
</div>

Preview instantly in the browser. Render to MP4 locally or in Docker.

Catalog

50+ ready-to-use blocks and components — social overlays, shader transitions, data visualizations, and cinematic effects:

npx hyperframes add flash-through-white   # shader transition
npx hyperframes add instagram-follow      # social overlay
npx hyperframes add data-chart            # animated chart

Browse the full catalog at hyperframes.heygen.com/catalog.

Documentation

Full documentation at hyperframes.heygen.com/introduction — Quickstart | Guides | API Reference | Catalog

Packages

Package Description
hyperframes CLI — create, preview, lint, and render compositions
@hyperframes/core Types, parsers, generators, linter, runtime, frame adapters
@hyperframes/engine Seekable page-to-video capture engine (Puppeteer + FFmpeg)
@hyperframes/producer Full rendering pipeline (capture + encode + audio mix)
@hyperframes/studio Browser-based composition editor UI
@hyperframes/player Embeddable <hyperframes-player> web component
@hyperframes/shader-transitions WebGL shader transitions for compositions

Skills

HyperFrames ships skills that teach AI agents framework-specific patterns that generic docs don't cover.

npx skills add heygen-com/hyperframes
Skill What it teaches
hyperframes HTML composition authoring, captions, TTS, audio-reactive animation, transitions
hyperframes-cli Dev-loop CLI: init, lint, inspect, preview, render, doctor
hyperframes-media Asset preprocessing: tts (Kokoro), transcribe (Whisper), remove-background (u2net) — voice/model/codec selection
hyperframes-registry Block and component installation via hyperframes add
website-to-hyperframes Capture a URL and turn it into a video — full website-to-video pipeline
remotion-to-hyperframes Translate a Remotion (React) composition into a HyperFrames HTML composition
gsap GSAP timelines for HyperFrames: paused registration, deterministic seeking, easing, sequencing, performance
animejs Anime.js animations and timelines registered on window.__hfAnime for deterministic HyperFrames seeking
css-animations CSS keyframe animation patterns that HyperFrames can discover, pause, and seek
lottie lottie-web and dotLottie players registered on window.__hfLottie with local assets and paused playback
three Three.js scenes that render from HyperFrames hf-seek events and window.__hfThreeTime instead of wall-clock time
waapi Web Animations API element.animate() patterns seeked through document.getAnimations()

Contributing

See CONTRIBUTING.md for guidelines.

Cloning the repo

The repo uses Git LFS for golden regression-test baselines under packages/producer/tests/**/output.mp4 (~240 MB of .mp4 files). If you're cloning the full repo for development, install Git LFS first:

# macOS
brew install git-lfs

# Ubuntu/Debian
sudo apt install git-lfs

# Windows
winget install GitHub.GitLFS
# (or install Git for Windows, which bundles Git LFS as an optional component)

# Then (once, per machine)
git lfs install

If you hit git-lfs filter-process: command not found during git clone or npx skills add heygen-com/hyperframes, install Git LFS and retry. You can also skip LFS content if you only need the source files:

GIT_LFS_SKIP_SMUDGE=1 git clone https://github.com/heygen-com/hyperframes.git

License

Apache 2.0

S
Description
hyperframes-cli: Use the HyperFrames CLI development loop: init, add, catalog, capture, lint, check, snapshot, compare, grade-compare, preview, play, present, beats, keyframes,…; hyperframes: Mandatory entry point: read this first for any request to make, create, edit, animate, or render a video, animation, or motion graphic, including a promo,…; hyperframes-registry: Search, install, and wire registry blocks and components into HyperFrames compositions. Use BEFORE hand-building any named visua…
Readme 622 MiB
Languages
TypeScript 86.3%
JavaScript 9.2%
CSS 4%
Shell 0.3%
Python 0.2%