mirror of
https://github.com/heygen-com/hyperframes.git
synced 2026-09-14 18:01:20 +08:00
395fb9c084
## Problem HyperFrames already had `--gpu`, but that flag only controlled FFmpeg hardware encoding. The browser capture path still forced Chrome/WebGL through SwiftShader software GL via `--use-angle=swiftshader`, so WebGL-heavy local renders could leave the biggest bottleneck on the CPU path. That made the existing flag naming easy to misread: `--gpu` sounded like it accelerated the whole render, but it did not change the browser frame-capture backend. ## What this fixes - Enables host browser GPU acceleration automatically for local CLI renders. - Adds `--no-browser-gpu` as the local opt-out for software Chrome/WebGL capture. - Keeps `--browser-gpu` as an explicit local browser-GPU request. - Adds `browserGpuMode: "software" | "hardware"` to engine config, with `PRODUCER_BROWSER_GPU_MODE` env support for lower-level producer users. - Keeps Docker browser capture on the deterministic software path. - Maps hardware browser GPU mode to platform-native Chrome backends: - macOS: Metal-backed ANGLE - Windows: D3D11-backed ANGLE - Linux: EGL - Blocks explicit `--browser-gpu --docker` with a clear error because Docker browser GPU passthrough is not cross-platform. - Clarifies docs so `--gpu` means FFmpeg encoder GPU and browser GPU means Chrome/WebGL capture GPU. - Keeps encoder backend selection auto-detected from FFmpeg capabilities: - NVIDIA: NVENC - macOS: VideoToolbox - Linux: VAAPI - Intel: QSV ## Why two flags There are two separate GPU surfaces in the render pipeline: 1. Browser GPU controls Chrome frame capture. - Affects WebGL, canvas, CSS rendering, compositing, and screenshot capture inside the browser. - This is enabled automatically for local CLI renders. - Use `--no-browser-gpu` when you want the software browser baseline. 2. `--gpu` controls FFmpeg video encoding. - Affects the final encode step after frames have already been captured. - The concrete encoder is auto-detected from the host FFmpeg build and hardware. - It can be faster for some machines/codecs, but it is not equivalent to browser rendering acceleration. The controls stay independent because users may want: - `hyperframes render` for the fast local default with browser GPU capture. - `hyperframes render --no-browser-gpu` for the software-browser local baseline. - `hyperframes render --gpu` for browser GPU capture plus hardware FFmpeg encoding. - `hyperframes render --no-browser-gpu --gpu` for software browser capture plus hardware FFmpeg encoding. - `hyperframes render --docker` for deterministic browser capture. ## Why `--gpu` does not imply browser GPU Keeping `--gpu` scoped to FFmpeg encoding avoids a semantic break and keeps the risk profile explicit: - `--gpu` already means encoder acceleration. Expanding it to also change Chrome capture would silently alter behavior for users who only wanted hardware encoding. - Browser GPU and encoder GPU have different portability. Encoder GPU can work in Docker when the host exposes the right devices; browser GPU passthrough is not cross-platform, so this PR intentionally blocks explicit `--browser-gpu --docker`. - The Apple presentation benchmark shows why the controls should stay separate: browser GPU capture was the useful improvement, while macOS VideoToolbox via `--gpu` was slower and produced larger output for this `standard` H.264 run. If HyperFrames later wants a single umbrella acceleration control, it should be explicit, for example `--acceleration browser|encoder|all` or `--gpu=browser|encoder|all`, rather than changing the meaning of the existing boolean `--gpu`. ## Root cause `buildChromeArgs()` always injected `--use-gl=angle --use-angle=swiftshader`. `disableGpu` only appended `--disable-gpu`; it did not provide a hardware-GPU mode. That made the public `--gpu` flag look broader than it was, because render capture stayed software-backed even when encoder GPU was requested. ## Verification ### Local checks - `bun install` - `bun run build:hyperframes-runtime` - `bun run --filter @hyperframes/engine test src/config.test.ts src/services/browserManager.test.ts` - `bun run --filter @hyperframes/cli test src/utils/dockerRunArgs.test.ts src/commands/render.test.ts` - `bun run --filter @hyperframes/cli typecheck` - `bun run --filter @hyperframes/engine typecheck` - `bun run --filter @hyperframes/producer typecheck` - `cd packages/producer && bunx vitest run src/services/renderOrchestrator.test.ts` - `bunx oxlint packages/cli/src/commands/render.ts packages/cli/src/commands/render.test.ts packages/cli/src/utils/dockerRunArgs.ts packages/cli/src/utils/dockerRunArgs.test.ts packages/engine/src/config.ts packages/engine/src/config.test.ts packages/engine/src/services/browserManager.ts packages/engine/src/services/browserManager.test.ts packages/producer/src/services/renderOrchestrator.test.ts` - `bunx oxfmt --check ...` on changed source/docs files - `git diff --check` - `bun packages/cli/src/cli.ts render --help | rg -n "browser-gpu|no-browser-gpu|GPU"` - `bun packages/cli/src/cli.ts render packages/producer/tests/css-spinner-render-compat/src --output /tmp/hf-auto-browser-gpu-smoke.mp4 --workers 1 --quality draft --fps 24 --strict` - Render plan prints `GPU: browser GPU (auto)`. - `bun packages/cli/src/cli.ts render packages/producer/tests/css-spinner-render-compat/src --no-browser-gpu --output /tmp/hf-software-browser-gpu-smoke.mp4 --workers 1 --quality draft --fps 24 --strict` - Render plan does not print browser GPU. - `bun packages/cli/src/cli.ts render packages/producer/tests/css-spinner-render-compat/src --docker --browser-gpu --output /tmp/should-not-render.mp4` - Exits 1 with `Browser GPU is local-only`. - `buildDockerRunArgs()` regression coverage asserts Docker container args include `--no-browser-gpu`, preventing nested container renders from re-enabling browser GPU through the local CLI default. - `resolveBrowserGpuForCli()` regression coverage asserts `PRODUCER_BROWSER_GPU_MODE=software` opts out when no CLI browser-GPU flag is supplied, while explicit `--browser-gpu` / `--no-browser-gpu` still win. - `ffmpeg -v error -i /tmp/hf-auto-browser-gpu-smoke.mp4 -f null -` - `ffmpeg -v error -i /tmp/hf-software-browser-gpu-smoke.mp4 -f null -` - `ffprobe -v error -show_entries format=duration:stream=codec_name,width,height,r_frame_rate -of json /tmp/hf-browser-gpu-smoke.mp4` -> H.264, 1920x1080, 24fps, 5.0s ### Apple presentation benchmark Rendered `/Users/miguel07code/Downloads/apple-presentation.zip` as supplied after extracting to `/tmp/hf-apple-profile/apple-presentation`. Fixed settings: - 1920x1080 - 30fps - `standard` quality - 4240 frames - 141.32s duration - 8-worker cap; render auto-calibration used 6 capture workers - macOS host detected FFmpeg GPU encoder: `videotoolbox` | Mode | Equivalent flags after this PR | Wall time | vs software-browser baseline | Speed | Capture | Encode | Output | | --- | --- | ---: | ---: | ---: | ---: | ---: | ---: | | Software browser + CPU encode | `--no-browser-gpu` | 120.77s | baseline | 1.17x | 97.87s | 10.04s | 8.38MB | | Browser GPU + CPU encode | default local render | 70.10s | 42.0% faster | 2.02x | 50.72s | 9.91s | 8.39MB | | Software browser + encoder GPU | `--no-browser-gpu --gpu` | 133.16s | 10.3% slower | 1.06x | 103.58s | 18.31s | 25.43MB | | Browser GPU + encoder GPU | `--gpu` | 74.12s | 38.6% faster | 1.91x | 46.69s | 17.93s | 25.45MB | Result: browser GPU capture is the meaningful improvement for this WebGL/browser-capture-heavy presentation. VideoToolbox encoding was slower and produced larger files for this current `standard` H.264 path, so `--gpu` should stay separate and opt-in. Why `--gpu` plus browser GPU was slower than browser GPU alone: the combined run captured about 4.0s faster than browser GPU alone, but VideoToolbox encoding was about 8.0s slower than CPU x264 encoding, so the encode loss outweighed the capture gain. ### VideoToolbox flag check I also isolated the encode stage against the already-captured Apple frames to check whether macOS GPU encoding only needed special flags. `ffmpeg -h encoder=h264_videotoolbox` does not expose a CRF/CQ-style quality option like x264. It exposes bitrate-oriented and VideoToolbox-specific options such as `-b:v`, `-realtime`, `-profile`, `-coder`, `-prio_speed`, `-power_efficient`, and `-allow_sw`. That means our current `-q:v` mapping is not equivalent to x264 CRF and can produce very different bitrate/size behavior. Measured full-frame encode variants on this host: | VideoToolbox variant | Encode wall time | Output size | Bitrate | | --- | ---: | ---: | ---: | | Current `-q:v 64 -allow_sw 1` | 18.76s | 25.31MB | 1.43 Mbps | | Current without `-allow_sw 1` | 18.21s | 25.31MB | 1.43 Mbps | | `-b:v 500k -maxrate 750k -bufsize 1000k -profile high -coder cabac -realtime 1 -prio_speed 1 -power_efficient 0` | 20.58s | 7.42MB | 0.42 Mbps | | Same with `-b:v 1500k` | 20.84s | 16.70MB | 0.95 Mbps | | `-b:v 500k -profile baseline -coder cavlc -realtime 1 -prio_speed 1 -power_efficient 0` | 18.11s | 8.94MB | 0.51 Mbps | Conclusion: VideoToolbox can be made size/bitrate-predictable with explicit `--video-bitrate`, but the tested speed-oriented flags did not make it faster than CPU x264 wall time for this render. That reinforces keeping `--gpu` encoder acceleration explicit and separate from browser GPU capture. Artifacts from the local benchmark: - `/tmp/hf-apple-profile/results/cpu.mp4` - `/tmp/hf-apple-profile/results/browser-gpu.mp4` - `/tmp/hf-apple-profile/results/encoder-gpu.mp4` - `/tmp/hf-apple-profile/results/full-gpu.mp4` - `/tmp/hf-apple-profile/results/summary.json` All four benchmark MP4s completed `ffprobe` and full `ffmpeg -f null` decode checks. ### Pixel comparison Compared decoded MP4 output between software-browser and browser-GPU renders: - Apple presentation: - 4240 frames compared - 636 exact matching decoded frame hashes - 3604 different decoded frame hashes - Average PSNR: 57.79 dB - `css-spinner-render-compat` clean fixture: - 120 frames compared - 0 exact matching decoded frame hashes - Average PSNR: 61.57 dB Interpretation: browser GPU output is not strict hash/pixel-identical to the software-browser path after lossy H.264 encode, but the measured deltas are visually tiny. Above 50 dB PSNR is typically visually indistinguishable for normal video review. Use `--no-browser-gpu` or Docker when strict cross-run/cross-machine reproducibility matters more than local speed. ### Browser verification - Started HyperFrames Studio preview for `packages/producer/tests/css-spinner-render-compat/src`. - Used `agent-browser` to open `http://localhost:5191#project/src` and verify the composition loaded in Studio. - Screenshots: - `/tmp/hf-gpu-browser-proof/preview-loaded.png` - `/tmp/hf-gpu-browser-proof/preview-playing.png` - `/tmp/hf-gpu-browser-proof/preview-frame-60.png` - Agent-browser recordings: - `/tmp/hf-gpu-browser-proof/preview-playback.webm` - `/tmp/hf-gpu-browser-proof/preview-seek.webm` ## Notes - Browser GPU is enabled automatically for local CLI renders and disabled in Docker. - `--no-browser-gpu` is the opt-out for software Chrome/WebGL capture. - `--gpu` remains encoder-only and opt-in. - The Apple presentation zip has existing lint errors around unmanaged nested videos and imperative media `play()` calls. The benchmark still compares the same supplied source across modes, but it should not be treated as a clean deterministic-composition fixture.
243 lines
8.4 KiB
Plaintext
243 lines
8.4 KiB
Plaintext
---
|
|
title: Quickstart
|
|
description: "Create, preview, and render your first Hyperframes video in under two minutes."
|
|
---
|
|
|
|
Go from zero to a rendered MP4 — either by prompting your AI agent or by starting a project manually.
|
|
|
|
## Option 1: With an AI coding agent (recommended)
|
|
|
|
Install the HyperFrames skills, then describe the video you want:
|
|
|
|
```bash
|
|
npx skills add heygen-com/hyperframes
|
|
```
|
|
|
|
This teaches your agent (Claude Code, Cursor, Gemini CLI, Codex) how to write correct compositions and GSAP animations. In Claude Code the skills register as slash commands — `/hyperframes` for composition authoring, `/hyperframes-cli` for CLI commands, and `/gsap` for animation help. Invoking the slash command loads the skill context explicitly, which produces correct output the first time.
|
|
|
|
<Note>
|
|
Claude Design uses a different entry path. Open [`docs/guides/claude-design-hyperframes.md`](https://github.com/heygen-com/hyperframes/blob/main/docs/guides/claude-design-hyperframes.md) on GitHub, click the download button (↓) to save it, then attach to your Claude Design chat. It produces a valid first draft you can refine in any AI coding agent. See the [Claude Design guide](/guides/claude-design).
|
|
</Note>
|
|
|
|
### Try it: example prompts
|
|
|
|
Copy any of these into your agent to get started.
|
|
|
|
<CardGroup cols={1}>
|
|
<Card title="Cold start — describe what you want">
|
|
> Using `/hyperframes`, create a 10-second product intro with a fade-in title over a dark background and subtle background music.
|
|
</Card>
|
|
<Card title="Warm start — turn existing context into a video">
|
|
> Take a look at this GitHub repo https://github.com/heygen-com/hyperframes and explain its uses and architecture to me using `/hyperframes`.
|
|
|
|
> Summarize the attached PDF into a 45-second pitch video using `/hyperframes`.
|
|
|
|
> Turn this CSV into an animated bar chart race using `/hyperframes`.
|
|
</Card>
|
|
<Card title="Format-specific">
|
|
> Make a 9:16 TikTok-style hook video about [topic] using `/hyperframes`, with bouncy captions synced to a TTS narration.
|
|
</Card>
|
|
<Card title="Iterate — talk to the agent like a video editor">
|
|
> Make the title 2x bigger, swap to dark mode, and add a fade-out at the end.
|
|
|
|
> Add a lower third at 0:03 with my name and title.
|
|
</Card>
|
|
</CardGroup>
|
|
|
|
The agent handles scaffolding, animation, and rendering. See the [prompting guide](/guides/prompting) for more patterns.
|
|
|
|
<Tip>
|
|
Skills encode HyperFrames-specific patterns — like required `class="clip"` on timed elements, GSAP timeline registration, and `data-*` attribute semantics — that are not in generic web docs. Using skills produces correct compositions from the start.
|
|
</Tip>
|
|
|
|
## Option 2: Start a project manually
|
|
|
|
### Prerequisites
|
|
|
|
- **Node.js 22+** — runtime for the CLI and dev server
|
|
- **FFmpeg** — video encoding for local renders
|
|
|
|
<Accordion title="Install instructions">
|
|
<Steps>
|
|
<Step title="Install Node.js 22+">
|
|
Hyperframes requires Node.js 22 or later. Check your version:
|
|
|
|
```bash
|
|
node --version
|
|
```
|
|
|
|
```bash Expected output
|
|
v22.0.0 # or any version >= 22
|
|
```
|
|
</Step>
|
|
|
|
<Step title="Install FFmpeg">
|
|
FFmpeg is required for local video rendering (encoding captured frames into MP4).
|
|
|
|
<CodeGroup>
|
|
```bash macOS
|
|
brew install ffmpeg
|
|
```
|
|
```bash Ubuntu / Debian
|
|
sudo apt install ffmpeg
|
|
```
|
|
```bash Windows
|
|
# Download from https://ffmpeg.org/download.html
|
|
# or install via winget:
|
|
winget install ffmpeg
|
|
```
|
|
</CodeGroup>
|
|
|
|
Verify the installation:
|
|
|
|
```bash
|
|
ffmpeg -version
|
|
```
|
|
|
|
```bash Expected output
|
|
ffmpeg version 7.x ...
|
|
```
|
|
</Step>
|
|
</Steps>
|
|
</Accordion>
|
|
|
|
### Create your first video
|
|
|
|
<Steps>
|
|
<Step title="Scaffold the project">
|
|
```bash
|
|
npx hyperframes init my-video
|
|
cd my-video
|
|
```
|
|
|
|
This starts an interactive wizard that walks you through example selection and media import. To skip prompts (e.g. in CI or from an agent), use `--non-interactive`:
|
|
|
|
```bash
|
|
npx hyperframes init my-video --non-interactive --example blank
|
|
```
|
|
|
|
See [Examples](/examples) for all available examples.
|
|
|
|
This generates a project structure like:
|
|
|
|
<Tree>
|
|
<Tree.Folder name="my-video" defaultOpen>
|
|
<Tree.File name="meta.json" />
|
|
<Tree.File name="index.html" />
|
|
<Tree.Folder name="compositions" defaultOpen>
|
|
<Tree.File name="intro.html" />
|
|
<Tree.File name="captions.html" />
|
|
</Tree.Folder>
|
|
<Tree.Folder name="assets" defaultOpen>
|
|
<Tree.File name="video.mp4" />
|
|
</Tree.Folder>
|
|
</Tree.Folder>
|
|
</Tree>
|
|
|
|
| Path | Purpose |
|
|
|------|---------|
|
|
| `meta.json` | Project metadata (name, ID, creation date) |
|
|
| `index.html` | Root composition — your video's entry point |
|
|
| `compositions/` | Sub-compositions loaded via `data-composition-src` |
|
|
| `assets/` | Media files (video, audio, images) |
|
|
|
|
If you have a source video, pass it with `--video` for automatic transcription and captions:
|
|
|
|
```bash
|
|
npx hyperframes init my-video --example warm-grain --video ./intro.mp4
|
|
```
|
|
|
|
`hyperframes init` installs AI agent skills automatically, so you can hand off to your AI agent at any point.
|
|
</Step>
|
|
|
|
<Step title="Preview in the browser">
|
|
```bash
|
|
npx hyperframes preview
|
|
```
|
|
|
|
This starts the Hyperframes Studio and opens your composition in the browser. Edits to `index.html` reload automatically.
|
|
|
|
<Tip>
|
|
The dev server supports hot reload — save your HTML file and the preview updates instantly, no manual refresh needed.
|
|
</Tip>
|
|
</Step>
|
|
|
|
<Step title="Edit the composition">
|
|
Open the project with your AI coding agent (Claude Code, Cursor, etc.) — skills are installed automatically and your agent knows how to create and edit compositions.
|
|
|
|
Or edit `index.html` directly — here's a minimal composition:
|
|
|
|
```html index.html
|
|
<div id="root" data-composition-id="my-video"
|
|
data-start="0" data-width="1920" data-height="1080">
|
|
|
|
<!-- 1. Define a timed text clip on track 0 -->
|
|
<h1 id="title" class="clip"
|
|
data-start="0" data-duration="5" data-track-index="0"
|
|
style="font-size: 72px; color: white; text-align: center;
|
|
position: absolute; top: 50%; left: 50%;
|
|
transform: translate(-50%, -50%);">
|
|
Hello, Hyperframes!
|
|
</h1>
|
|
|
|
<!-- 2. Load GSAP for animation -->
|
|
<script src="https://cdn.jsdelivr.net/npm/gsap@3/dist/gsap.min.js"></script>
|
|
|
|
<!-- 3. Create a paused timeline and register it -->
|
|
<script>
|
|
const tl = gsap.timeline({ paused: true });
|
|
tl.from("#title", { opacity: 0, y: -50, duration: 1 }, 0);
|
|
window.__timelines = window.__timelines || {};
|
|
window.__timelines["my-video"] = tl;
|
|
</script>
|
|
</div>
|
|
```
|
|
|
|
Three rules to remember:
|
|
|
|
- **Root element** must have `data-composition-id`, `data-width`, and `data-height`
|
|
- **Timed elements** need `data-start`, `data-duration`, `data-track-index`, and `class="clip"`
|
|
- **GSAP timeline** must be created with `{ paused: true }` and registered on `window.__timelines`
|
|
</Step>
|
|
|
|
<Step title="Render to MP4">
|
|
```bash
|
|
npx hyperframes render --output output.mp4
|
|
```
|
|
|
|
```bash Expected output
|
|
✔ Capturing frames... 150/150
|
|
✔ Encoding MP4...
|
|
✔ output.mp4 (1920x1080, 5.0s, 30fps)
|
|
```
|
|
|
|
Your video is now at `output.mp4`. Open it with any media player.
|
|
</Step>
|
|
</Steps>
|
|
|
|
## Requirements summary
|
|
|
|
| Dependency | Required | Notes |
|
|
|-----------|----------|-------|
|
|
| **Node.js** 22+ | Yes | Runtime for CLI and dev server |
|
|
| **npm** or bun | Yes | Package manager |
|
|
| **FFmpeg** | Yes | Video encoding for local renders |
|
|
| **Docker** | No | Optional — for deterministic, reproducible renders |
|
|
|
|
## Next steps
|
|
|
|
<CardGroup cols={2}>
|
|
<Card title="Browse the Catalog" icon="grid-2" href="/catalog/blocks/data-chart">
|
|
50+ ready-to-use blocks — transitions, overlays, data visualizations, and effects
|
|
</Card>
|
|
<Card title="GSAP Animation" icon="wand-magic-sparkles" href="/guides/gsap-animation">
|
|
Add fade, slide, scale, and custom animations to your videos
|
|
</Card>
|
|
<Card title="Examples" icon="rocket" href="/examples">
|
|
Start from built-in examples like Warm Grain and Swiss Grid
|
|
</Card>
|
|
<Card title="Rendering" icon="film" href="/guides/rendering">
|
|
Explore render options: quality presets, Docker mode, and GPU acceleration
|
|
</Card>
|
|
</CardGroup>
|