The diagnostic logging block in executeRenderJob's HDR layer composite
path referenced an undeclared `hdrLayerStartTimes` map. The correct
variable, declared and populated earlier in the same function, is
`hdrVideoStartTimes`. The typo was introduced alongside the DOM-layer
masking work and broke the producer build/typecheck on CI.
Made-with: Cursor
SDR clips inside an HDR composition were rendering at full opacity even
when the user had animated their wrapper opacity (e.g. fade-in or
yoyo). Two bugs in the per-layer screenshot path conspired to drop the
GSAP-applied opacity on the floor:
1. removeDomLayerMask was unconditionally calling
`el.style.removeProperty("opacity")` on every wrapper after each
layer capture. applyDomLayerMask only ever sets `visibility`, so the
only inline opacity present is the value GSAP wrote. Stripping it
between layer captures means that on the next capture (at the same
timestamp), GSAP's `totalTime(t, false)` no-ops because the timeline
is already at that time — the opacity is never restored, and the
wrapper renders fully opaque.
2. injectVideoFramesBatch was reading the source <video>'s computed
opacity via `parseFloat(computedStyle.opacity) || 1` and copying it
onto the injected <img>. Because syncVideoFrameVisibility forces the
<video> to `opacity: 0 !important` to hide it during capture, the
computed value is always 0, which `|| 1` then silently flips to
full opacity. The <img> is a sibling of the <video> inside the same
wrapper, so it should inherit opacity from the wrapper directly
instead of having a value hard-set on it.
Fix both: drop the opacity removal in removeDomLayerMask, skip opacity
when copying visual properties from <video> to <img>, and explicitly
clear any stale inline opacity on the <img> so it inherits from the
wrapper that GSAP is animating.
Made-with: Cursor
The HDR layered compositor blits z-ordered layers over a shared canvas. DOM
layers used a full-page screenshot from `captureAlphaPng`, which captures
*every* painted pixel on the page — root background, sibling-scene content,
overlay UI elements that aren't part of the current layer. Those opaque
pixels were then blitted over the canvas, overwriting any HDR content
composited beneath in earlier layers.
The previous workaround toggled `display:none` on hide ids via
`hideVideoElements`/`showVideoElements`. That correctly hid native videos
but did nothing about the root composition's background or about overlay
elements that the layer grouping considered part of a different layer.
This commit replaces the workaround with a precise CSS mask installed
before each DOM screenshot:
1. `applyDomLayerMask` injects a stylesheet that hides every `body *` and
re-shows the layer's elements (and their descendants and their injected
`__render_frame_*` siblings) with `visibility: visible !important`. CSS
visibility is *not* multiplicative through descendants — a child with
`visibility: visible` overrides an ancestor's `visibility: hidden`, so
deeply nested layer content still paints even though every intermediate
ancestor is hidden by the mass-hide rule.
2. Non-layer data-start ids are inline-hidden with
`visibility: hidden !important`. Inline `!important` beats stylesheet
`!important`, so this overrides the show rule for elements that fall
under a show selector but should NOT paint — most importantly HDR
videos and other-layer SDR videos that live as descendants of `#root`.
3. `removeDomLayerMask` tears the stylesheet down and clears the inline
`visibility`/`opacity` properties so subsequent video frame injection
gets a clean slate.
Crucially the mask only sets `visibility`, never `opacity`. CSS opacity
*is* multiplicative — `opacity: 0` on `#root` would zero out every
descendant including layer videos, even with `visibility: visible`. We
also extend `initTransparentBackground` to force the composition root
(`[data-composition-id]`) transparent in addition to `html`/`body`,
because compositions almost always set `#root { background: ... }` and
that background paints across the whole viewport otherwise.
Both compositing paths use the new helpers:
- The per-layer DOM branch (`compositeToBuffer`) for normal frames.
- The transition path (single DOM screenshot per scene) so transition
frames also get a clean per-scene capture.
Adds extensive `KEEP_TEMP=1`-gated diagnostics to `compositeToBuffer`:
per-layer pixel-add accounting, dumps of every captured DOM PNG, and a
periodic raw `rgb48le` snapshot of the composite buffer. These were
essential to diagnosing the root-overwrite bug and stay zero-cost in
normal renders. Also stops the workDir / per-video frame-dir cleanup
when `KEEP_TEMP=1` so the dumps survive past frame N.
Made-with: Cursor
Adds:
- 8 new sampleRgb48le bilinear-interpolation tests covering boundary
pixels, sub-pixel weights, edge clamping, and odd-byte-offset Buffers.
- uint16-alignment-audit.test.ts documenting the alignment requirement
for Uint16Array views over Buffer slices vs. readUInt16LE/writeUInt16LE.
Background: ~105 hot-loop sites in shader transitions still use
readUInt16LE/writeUInt16LE. Switching to Uint16Array views would cut
overhead but requires guaranteed even byteOffsets — these tests document
the contract before any future refactor lands.
Commit c6b4619c ("feat(hdr): shader transitions, --hdr flag, and SDR
rendering fixes") accidentally removed two pieces of the deterministic
rendering pipeline:
1. The `VIRTUAL_TIME_SHIM` injected via `createFileServer.preHeadScripts`,
which freezes `Date.now()` and `requestAnimationFrame` so RAF-driven
animations advance only when `window.__hf.seek(t)` is called.
2. The `applyRenderModeHints` function and its post-`compileForRender`
call site, which auto-forces screenshot capture mode for compositions
the compiler flagged as needing it (RAF, iframes, etc.).
Without (1), RAF animations advanced by wall-clock between the main-loop
seek and the per-DOM-layer seek inside `compositeToBuffer`, producing the
sawtooth PSNR pattern on `raf-ball-render-compat` (high PSNR at integer
seconds, ~24 dB everywhere else). Without (2), `iframe-render-compat`
lost its automatic fallback to screenshot mode and the child-document
motion stopped being captured.
Both helpers are still produced by `htmlCompiler` and exercised by
`renderOrchestrator.test.ts` — the orchestrator just stopped calling
them. Restored:
- Re-import `VIRTUAL_TIME_SHIM` from `./fileServer.js`
- Pass `preHeadScripts: [VIRTUAL_TIME_SHIM]` to both `createFileServer`
call sites (probe + main render)
- Re-add `applyRenderModeHints` (matching the test expectations) and
call it immediately after `compileForRender`
- Persist `renderModeHints` in `summary.json` and the
"Compiled composition metadata" log line
Fixes the `iframe-render-compat` and `raf-ball-render-compat` regression
failures on `feat/hdr-layered-compositing`.
Made-with: Cursor
- Tighten the all-transitions smoke test thresholds: at progress=0 we now
require the center pixel R-channel > 35000 (was > 25000) and at
progress=1 < 15000 (was < 25000). The old midpoint of 25000 sat exactly
halfway between the test from-pixel (40000) and to-pixel (10000), so a
half-blended transition would silently pass.
- Add a runtime assertion in HyperShader.init() that every scene id
resolves to a DOM element with the .scene class. Without this, missing
ids silently no-op when textures + querySelectorAll(.scene) run later.
Addresses deferred review feedback from PR #268.
When the page is loaded by the Hyperframes engine, `init()` would happily set
up GL contexts and html2canvas captures even though the engine composites
every transition itself from `window.__hf.transitions` metadata. That work was
both wasted and harmful: scenes were left at `opacity: 0` (waiting for the
async capture to resolve) when the engine sampled the DOM, so
`queryElementStacking()` reported them invisible and the layered HDR
compositor produced black frames between transitions.
Engine mode is detected via `window.__HF_VIRTUAL_TIME__` (the shim the engine
injects). When present, `initEngineMode()` runs instead of the GL pipeline and
schedules deterministic opacity flips:
- `tl.set('#toId', { opacity: 1 }, T)` // crossfade start
- `tl.set('#fromId', { opacity: 0 }, T + d)` // crossfade end
`tl.set` (zero-duration tweens) is required instead of `tl.call`, because
`tl.call` only fires in the direction of playback. The engine's warmup loop
seeks forward through every transition start and then the main render loop
seeks back to t=0 — `tl.call`-set state would get stuck. `tl.set` tweens
revert cleanly on backward seeks.
The new path still calls the shared `registerTimeline()` helper so the timeline
shows up on `window.__timelines[compId]` for the engine to drive.
Made-with: Cursor
offsetWidth/offsetHeight are only defined on HTMLElement (not on
SVGElement, MathMLElement, etc.). Casting unconditionally can yield
undefined-by-runtime even though TypeScript thinks otherwise. Use an
instanceof check and fall back to the bounding rect dimensions when the
element is not an HTMLElement.
Addresses deferred review feedback from PR #290.
- Fall back to parentElement when offsetParent is null (position:fixed).
- Resolve transformOrigin percentages against offsetWidth/Height.
- Guard DOMMatrix with Number.isFinite + try/catch for malformed input.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Affine blit with bilinear interpolation for GSAP-animated HDR video.
Accumulated viewport matrix via DOMMatrix chain with transform-origin
handling. Effective opacity ancestor walk. Rounded-rectangle mask with
anti-aliased corners and percentage resolution.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Per-frame z-order analysis groups elements into DOM and HDR layers,
composited bottom-to-top. Adjacent DOM elements merge into single
screenshots. PQ (HDR10/smpte2084) support via sRGB-to-PQ LUT with
203-nit SDR reference white. queryElementStacking walks DOM for
effective z-index, groupIntoLayers splits on HDR/DOM boundaries.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The render bridge previously overwrote `window.__hf` wholesale at end-of-body,
clobbering anything written during user `<script>` execution. Libraries such as
`@hyperframes/shader-transitions` populate `window.__hf.transitions` inside
their `init()` call, so the engine never saw a transition map and HDR
compositing fell back to plain DOM layers.
Two changes restore the contract:
1. Inject HF_EARLY_STUB at the very start of `<head>` so `window.__hf` exists
before any other script runs. User libraries can now safely write to it
during initialisation.
2. HF_BRIDGE_SCRIPT now patches the existing `__hf` object (defining `duration`
as a getter and assigning `seek`) instead of replacing it, preserving any
fields the page already populated.
Adds a regression test that simulates the real injection order — early stub,
authored page script writing `__hf.transitions`, bridge script — and asserts
both `transitions` and the bridge-installed `seek`/`duration` are present
afterwards.
Made-with: Cursor
- blitRgb48leRegion: take canvasHeight as an explicit parameter so callers
with non-square canvases or other byte-stride conventions cannot silently
produce a wrong height. Updates all alphaBlit.test.ts call sites.
- renderOrchestrator: cache max frame index per pre-extracted HDR frame
directory and bounds-check videoFrameIndex before existsSync. Avoids
redundant filesystem syscalls when the requested time falls past the
last extracted frame for a given clip.
Made-with: Cursor
Use Object.assign to merge runtime bridge (seek/duration) with
existing window.__hf properties instead of overwriting them.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Without `master-display=...:max-cll=...` in -x265-params the encoder
writes a perfectly valid HEVC stream tagged BT.2020/PQ, but with no
HDR10 metadata SEI messages. Apple, YouTube, and HDR TVs see the file
as "unknown HDR" and tone-map it incorrectly (washed-out highlights,
clipped specular detail).
Changes:
- Add HdrMasteringMetadata + DEFAULT_HDR10_MASTERING (D65, P3-D65 mastering
primaries, 1000-nit max, 400-nit MaxFALL — the standard Hollywood HDR10
envelope).
- Extend getHdrEncoderColorParams to compose the full x265-params string
including range=full + master-display + max-cll for both PQ and HLG.
- Refactor streamingEncoder to call getHdrEncoderColorParams instead of
duplicating the colorprim/transfer/colormatrix string literal.
- Export buildStreamingArgs so the new streamingEncoder.test.ts can pin
down the exact CLI flags as a regression net.
Verified end-to-end with hdr-smoke.ts on hdr-pq + mixed-sdr-hdr fixtures:
ffprobe -show_frames now reports "Mastering display metadata" and
"Content light level metadata" SEI messages on both outputs.
- Replace inline HDR auto-detection in renderOrchestrator with the
analyzeCompositionHdr helper from @hyperframes/engine
- Remove the backward-compat re-export of isHdrColorSpace from
videoFrameExtractor; callers should import from @hyperframes/engine
- Use isHdrColorSpaceUtil consistently inside videoFrameExtractor
- Remove duplicate isHdrColorSpace tests from videoFrameExtractor.test.ts
(canonical tests live in utils/hdr.test.ts)
Addresses reviewer feedback on PR #265.
Made-with: Cursor
The engine package's hdrCapture.ts uses WebGPU APIs and references
@webgpu/types via a triple-slash directive, and both engine and
producer tsconfigs include @webgpu/types in compilerOptions.types.
However the package was never declared as a devDependency, so it
worked locally (transitive resolution) but failed in CI with
TS2688: Cannot find type definition file for '@webgpu/types'.
Add @webgpu/types ^0.1.69 as a devDependency to both packages and
refresh bun.lock so the CI Build and Typecheck jobs pass.
Made-with: Cursor
Restore .gitignore entries removed during branch creation (docs/images/,
examples/, .playwright-mcp/, skills/, captures/, etc.) and revert the
@webgpu/types devDep additions — the WebGPU path is experimental and
skipLibCheck:true means the reference directive is a no-op when the
types aren't installed.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Distinguish ERR_MODULE_NOT_FOUND from other errors in puppeteer dynamic
import so broken installs surface instead of silently falling back.
- Document detectTransfer's isHdrColorSpace precondition.
- Rename misleading chunkEncoder test name.
- Clarify streamingEncoder Buffer.from as HDR-only by code path.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
These fields were added to main after this branch was created. Without
them, the CLI's render command fails typecheck when passing crf/bitrate
options to createRenderJob.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Node streams hold a reference to the provided buffer and drain it
asynchronously. Without copying, callers that reuse buffers get
partially-overwritten data in the pipe.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Auto-detect HDR sources via ffprobe, switch to H.265 10-bit with
BT.2020/HLG metadata. HLG pass-through (no conversion), hvc1 codec
tag for Apple playback, 1-hour streaming timeout for HDR. WebGPU
float16 readback pipeline implemented and tested but gated behind
headed Chrome. HDR utilities with 15 unit tests.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
When a composition contains both HDR (bt2020) and SDR (bt709) video elements,
the SDR content appears washed out because Chrome clamps HDR frames to sRGB
during rendering. This adds automatic color space detection and normalization
during the video frame extraction phase — if any video is HDR, all SDR videos
are converted to HLG/BT.2020 (npl=600) via zscale before frame extraction.
- Add colorSpace (transfer, primaries, matrix) to VideoMetadata via ffprobe
- Add isHdrColorSpace() to detect bt2020/PQ/HLG content
- Add convertSdrToHdr() using zscale with 600 nit nominal peak luminance
- Restructure extractAllVideoFrames into 3 phases: resolve, normalize, extract
- 6 new unit tests for HDR detection across all color space variants
Same approach as HeyGen Rio's pipeline (heygen_rio.py#L568-585).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix(core): drive adapter seeks when composition has no GSAP timeline
renderSeek returned early when deps.getTimeline() was null, skipping the
onDeterministicSeek call that drives all frame adapters (CSS, WAAPI,
Lottie, Three.js). That meant compositions using any non-GSAP animation
primitive froze on their initial frame during capture.
Now we still quantize the seek time and fire onDeterministicSeek even
without a timeline, so each adapter gets a chance to advance.
GSAP compositions are unaffected — timeline-driven seek still takes the
same path it did before.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(producer): auto-fallback screenshot capture for raf and iframes
Co-Authored-By: Codex <codex@openai.com>
* test(producer): add render compatibility regression fixtures
Co-Authored-By: Codex <codex@openai.com>
* fix(core): scrub CSS animations via WAAPI currentTime
Co-Authored-By: Codex <codex@openai.com>
* test(producer): cover css keyframe renders
Co-Authored-By: Codex <codex@openai.com>
* fix(producer): propagate virtual time into iframe documents
Co-Authored-By: Codex <codex@openai.com>
* test(producer): refresh iframe docker golden
Co-Authored-By: Codex <codex@openai.com>
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-authored-by: Codex <codex@openai.com>
* docs(guides): add performance guide and preview-stutter troubleshooting
Adds a dedicated Performance guide covering preview-vs-render cost model,
expensive CSS patterns (backdrop-filter, filter, shadows), image sizing,
and how to diagnose slow compositions with Chrome DevTools.
Cross-links from troubleshooting (new "Preview stutters" accordion) and
common-mistakes (new "Oversized source images" and "Heavy backdrop-filter
stacks" accordions). Wires the new page into docs.json nav.
Also fixes a pre-commit format hook edge case: oxfmt would exit 2 when
the only staged files matching the format glob were all covered by
.prettierignore (e.g. docs-only changes). Add --no-error-on-unmatched-pattern
to the lefthook oxfmt invocation so docs-only commits are not blocked.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* docs: call out preview performance limits at the entry points
The preview command, studio package, and determinism concept pages all
frame preview as visually equivalent to render — correct for fidelity,
misleading for playback smoothness. A user who reads those pages and
then hits a paint-heavy composition has no way to know why preview
stutters, short of drilling into troubleshooting.
Adds short notes at each entry point linking out to the new Performance
guide, so users hit the "preview is hardware-bound, render isn't"
explanation wherever they land first.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
## Summary
Fixes#316 — `hyperframes skills` (and `npx skills add heygen-com/hyperframes`) fails with:
```
■ Failed to clone repository
fatal: active \`post-checkout\` hook found during \`git clone\`
└ Installation failed
```
## Root cause
Two layers stacked:
1. **Git 2.45+ refuses to execute hooks during `git clone` by default.** The opt-in is `GIT_CLONE_PROTECTION_ACTIVE=0` — the env-var name is intentionally explicit about the trade-off.
2. **Users who ran `git lfs install` globally have a post-checkout hook registered at `core.hooksPath`.** When the upstream `skills` CLI shells out to `git clone` to fetch a repo's `skills/` directory, git detects the user's LFS hook and aborts.
The check fires for **any repo**, regardless of whether the cloned repo uses LFS itself — it's protection against the user's own hooks, not the repo's content. Users who have git-lfs installed (very common) hit this for every clone the `skills` CLI does.
## The fix
`hyperframes skills` wraps `npx skills add`. The wrapper now sets `GIT_CLONE_PROTECTION_ACTIVE=0` on the spawned child's env via a single helper (`gitCloneFriendlyEnv`) with a docstring at the call site explaining exactly why. The rest of `process.env` is preserved — proxy settings, extra CA certs, locale, etc. stay untouched.
## What this fix doesn't do (deliberately)
This is the **code-path-we-own** fix. The deeper root cause is that the upstream `skills` CLI (vercel-labs/skills) should set this env var when it shells out to `git clone`. That would fix the bug for every user invoking `skills` directly — not just those who route through our wrapper. An upstream issue should be opened separately; not landing it as part of this PR.
## Users who call `npx skills add` directly
Documented in the new troubleshooting subsection: set the env var manually.
```bash
GIT_CLONE_PROTECTION_ACTIVE=0 npx skills add heygen-com/hyperframes
```
## Tests
`packages/cli/src/commands/skills.test.ts` — 2 cases:
- Every spawned child has `GIT_CLONE_PROTECTION_ACTIVE=0`
- The rest of `process.env` is preserved (not a wiped env)
Uses `vi.mock` on `node:child_process` because ESM doesn't allow live-module `vi.spyOn` on re-exported bindings.
## Docs
`docs/packages/cli.mdx` — new **Troubleshooting** subsection under the `skills` command. Explains both the automatic fix (`hyperframes skills` users are already covered) and the manual workaround (`npx skills add …` users).
## Closes
- #316
* fix(producer): external assets work on Windows (GH #321)
Two Unix-only assumptions in the external-asset pipeline caused every
absolute path on Windows to be rejected as "unsafe" at render time:
1. Containment checks used `child.startsWith(parent + "/")`. On Windows
the separator is `\`, so the predicate is always false unless the
paths are equal — every external asset tripped the safety guard in
`renderOrchestrator.ts`. The reporter saw:
[Render] Skipping external asset with unsafe path:
hf-ext/D:\coder\reactGin\hyperframes\reading\assets\segment_001.wav
Fix: use `path.relative()` through a shared helper
`isPathInside(child, parent)` that normalises separators per-platform
and correctly rejects siblings whose names start with the parent
(e.g. `/foo/bar-sibling` is NOT inside `/foo/bar`).
2. The external-asset key was built as `"hf-ext/" + absPath.replace(/^\//, "")`.
A Windows absolute path (`D:\coder\...`) became
`"hf-ext/D:\\coder\\..."` — and because Node's `path.join` treats a
drive-letter prefix as absolute, `join(compileDir, key)` silently
escaped `compileDir`. Fix: `toExternalAssetKey()` strips the drive
colon and normalises to forward slashes, producing
`hf-ext/D/coder/...` — a pure relative path that `path.join` cannot
promote to absolute on any OS.
Both helpers live in `packages/producer/src/utils/paths.ts` and are
exercised by 14 unit tests covering Unix paths, Windows drive-letter
paths, mixed separators, sibling-prefix confusion, and `..` traversal.
Docs: new "External assets" section in `docs/packages/producer.mdx`
describes detection, sanitised keys, and the cross-platform containment
invariant.
Closes#321.
* fix(producer): address review on #324 — UNC + integration test
Addresses the non-blocking observations from the PR #324 staff review
(https://github.com/heygen-com/hyperframes/pull/324#issuecomment):
1. UNC and extended-length Windows paths.
`toExternalAssetKey` now handles:
- `\\?\D:\very\long\path\clip.mp4` (extended-length) → `hf-ext/D/very/long/path/clip.mp4`
- `\\server\share\file.wav` (plain UNC) → `hf-ext/unc/server/share/file.wav`
- `\\?\UNC\server\share\file.wav` (extended-length UNC) → `hf-ext/unc/server/share/file.wav`
The UNC-collapsed form keeps the server boundary so two different
servers exposing the same share/file name cannot collide under one
relative key. Previously both edge cases silently produced keys with
stray `?` or `:` characters that downstream `isPathInside` rejected —
not a security hole, but a silent drop of user assets.
2. Short-circuit on already-sanitised input.
`toExternalAssetKey("hf-ext/…")` now returns its input unchanged
instead of prepending `hf-ext/` a second time. Makes the helper
genuinely idempotent, which is what the unit test claimed all along.
Renamed the test accordingly.
3. JSDoc caller contract.
`toExternalAssetKey` now documents that it expects canonicalised
input (`path.resolve`'d upstream) and does not strip `..`
components. `isPathInside` at copy time is still the defensive
backstop — called out explicitly in the doc so future callers read
the contract before the code.
4. End-to-end integration test.
`renderOrchestrator.test.ts` gains two seam tests that run the full
external-asset pipeline — build the sanitised key, populate an
`externalAssets` map, invoke `writeCompiledArtifacts`, and assert
both the success path (the file lands under `<compileDir>/hf-ext/…`)
and the escape-rejection path (a malicious `hf-ext/../../etc/passwd`
key does NOT materialise above `compileDir`). `writeCompiledArtifacts`
is exported for the test seam with a clear JSDoc disclaimer that
it's not part of the public API.
22 tests pass across `paths.test.ts` (17) and `renderOrchestrator.test.ts` (5).
Out of scope for this follow-up (tracked as follow-ups):
- Centralising every `startsWith("/")` absolute-path check into a
shared helper across htmlCompiler / audioExtractor / audioMixer /
videoFrameExtractor. Mentioned in the review; touches 5 files and
deserves its own PR.
- Windows CI runner.
The "FFmpeg not found" hint was hardcoded to `sudo apt install ffmpeg`
for any non-macOS platform — Windows users would see an apt command that
doesn't exist on their system, and Red Hat / Arch users got the wrong
package manager too.
`getFFmpegInstallHint()` already exists in browser/ffmpeg.ts (and is
already used by render.ts) and handles darwin / linux / win32 correctly.
Use it here too.
Also rewrite checkFFprobe:
- it previously used `which ffprobe` which is not available on Windows
(cmd uses `where`), so on Windows the check always reported "Not
found" even when ffprobe was on PATH
- run `ffprobe -version` directly instead, which works cross-platform
whenever ffprobe is resolvable on PATH, and surfaces the version
string in the same style as the FFmpeg check
HTML files in this repo are compositions (user-facing content and
registry templates), not the framework source. The framework itself is
TypeScript. Hide HTML from Linguist's detection so the language bar on
GitHub reflects what the repo actually implements.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Chrome's "Failed to load resource" message text does not include the failing
URL — it's only on msg.location().url. The previous filter in frameCapture.ts
only checked msg.text(), so every font 404 (e.g. Google Fonts <link> tags
in sandboxed render environments) fell through to the "[non-blocking]"
prefix instead of being suppressed.
Extract the classifier into isFontResourceError() and match against both
text and location.url, and extend the extension match to .ttf/.otf. Adds
a unit test covering the URL-in-location, URL-in-text, and non-font cases.
This is a targeted fix for the render-output noise that PR #311 attempted
to address by adding a ~120-entry SYSTEM_FONTS skip list. That approach
silently shadowed existing FONT_ALIASES (arial→inter, helvetica→inter,
courier new→jetbrains-mono, segoe ui→roboto, etc.) and changed render
output on Linux fleets that don't have those fonts installed. Fixing the
console-log filter here suppresses the noise without changing any font
resolution behavior.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Stacked on top of #307. Fixes three mobile UX bugs that made the studio unusable on iPhone Safari — discovered while testing the audio-ownership work from #307 on a physical device.
## Bugs fixed
### 1. Untappable Play button / bottom controls
`#root` was set to `100vh`. iOS Safari reports `100vh` as the **largest** viewport (toolbar hidden) and never shrinks it — so with the toolbar visible, the bottom of the layout sits under it. The Play button + timecode were fully occluded.
### 2. Scrubber not draggable by touch
The seek bar had only `onMouseDown`. Mouse events don't fire for touches on iOS Safari, so nothing responded. You could tap to jump but not drag.
### 3. Safari's horizontal swipe hijacked scrubber drags
Even when the seek bar caught `pointerdown`, `touch-action: manipulation` still let Safari consume horizontal edge-swipes for back-navigation — dragging the scrubber left was impossible.
## What changed
| File | Fix |
|---|---|
| `packages/studio/src/styles/studio.css` | `#root` → `height: 100dvh` with `100vh` fallback. Dynamic viewport height shrinks when the iOS toolbar is visible, so the bottom of `#root` lines up with the visible area. |
| `packages/studio/src/App.tsx` | Two `h-screen` containers → `h-full` so nested children fill the now-dynamic parent instead of asserting `100vh` and overflowing. |
| `packages/studio/index.html` | Added `viewport-fit=cover` so iOS exposes real `env(safe-area-inset-bottom)` values. |
| `packages/studio/src/player/components/PlayerControls.tsx` | Controls row gets `padding-bottom: calc(0.5rem + env(safe-area-inset-bottom))` so it clears the landscape home indicator. Scrubber replaced `onMouseDown` with `onPointerDown` + `setPointerCapture`, plus `touch-action: none` so Safari doesn't hijack horizontal swipes. Added `pointercancel` + window-level `pointerup` fallbacks. |
All desktop code paths are unchanged: `100dvh` falls back to `100vh`, `env(safe-area-inset-bottom)` is `0` off-iOS, Pointer Events subsume Mouse Events on desktop.
## Verified live
Via the `cloudflared` tunnel I ran during review on the factory-series-c-video project:
- iPhone Safari, portrait: Play button now fully visible and tappable. Bottom controls sit just above the URL bar.
- iPhone Safari, landscape: controls clear the home indicator.
- Finger-drag the scrubber left and right: tracks the touch smoothly, finger can leave the 6 px bar height without losing the drag.
- Desktop click-to-seek and click-drag: still work.
- Arrow-key seeking: still works.
## Stacked dependency
Base is `fix/player-audio-ownership-review` (PR #307). Once #307 merges, rebase this branch onto `main` — the changes are fully independent; the stacking is just to avoid waiting on the review for #307 before shipping pure UX wins.
## Test plan
- [x] `tsc --noEmit` on `packages/studio` — clean
- [x] `bun run --filter @hyperframes/studio build` — clean
- [x] Live repro on iPhone Safari via cloudflared tunnel: Play button tappable, scrubber drags with touch
- [ ] Android Chrome sanity pass before release (same Pointer Events code path, but worth eye-balling)
Follow-up to PR #298 addressing @jrusso1020's review. Each item below maps to a point in his comment.
## Significant
### 1\. Drift threshold 150 ms → 50 ms
_mirrorParentMediaTime_ was too loose for lip-synced talking-head content. ITU-R BT.1359 puts A/V perceptibility at ±45 ms; 150 ms sat well inside the "unacceptable" zone. Dropped to 50 ms, extracted as a static constant for clarity.
**Verified live on factory-series-c-video (agent-browser):** steady-state offset under parent ownership sampled five times over 400 ms = `[35.7, 33.5, 31.2, 27.2, 36.9]` ms — below the perceptibility floor. Before this PR the same measurement could drift up to 150 ms before correction.
### 2\. Dynamic sub-composition media proxies
Under parent ownership, a sub-composition that attaches a new `<audio data-start>` mid-playback was correctly silenced in the iframe (sticky `outputMuted`) but had no parent-frame counterpart to play → silent hole in the audio track.
Added a `MutationObserver` on the iframe body watching for `audio[data-start]` / `video[data-start]` additions. New elements are adopted through the same `_adoptIframeMedia` helper the initial scan uses, and if parent ownership is already active the new proxy gets its `currentTime` mirrored and `play()` called immediately (gated on `!this._paused`). Observer disconnects on iframe reload + component disconnect.
### 3\. `bridgeMuted` sticky in `syncRuntimeMedia`
The asymmetry James flagged: `outputMuted` was sticky per-tick, `bridgeMuted` was one-shot via `onSetMuted`. A sub-composition activating after a user mute would briefly play at author volume before the next bridge message.
`syncRuntimeMedia` now accepts `userMuted` and the per-clip loop uses a single combined `shouldMute` gate. One invariant, two inputs.
### 4\. Reset `_audioOwner` on iframe reload
The latch never cleared. On composition switch the player would stay in `parent` ownership against a fresh runtime that hadn't received `set-media-output-muted` and whose autoplay-blocked latch was clean — a brief double-audio window until the next `NotAllowedError` re-promoted (idempotently).
`_onIframeLoad` now resets `_audioOwner = "runtime"`, pauses any parent proxies, and disconnects the old MutationObserver before a fresh one attaches to the new document. If the player had been in `parent` ownership, a corresponding `audioownershipchange` event fires with `reason: "iframe-reload"`.
## Worth addressing
### 5\. Promotion → observable event + reason
Promotion was invisible. Added `CustomEvent("audioownershipchange", { detail: { owner, reason } })` fired on every owner transition. `reason` is either `"autoplay-blocked"` (promote → parent) or `"iframe-reload"` (reset → runtime). Gives host apps an SLO-ready signal for "% of sessions in parent ownership" without exposing internal state.
**Verified live:** dispatching a synthetic `media-autoplay-blocked` in the live studio produced `{ owner: "parent", reason: "autoplay-blocked" }` on the web component exactly once.
### 6\. Parent proxy play() rejection → `playbackerror` event
Previously swallowed silently. Now re-emitted as `CustomEvent("playbackerror", { detail: { source: "parent-proxy", error } })` so embedding apps can recover or fall back.
### 7\. Mobile verification on real hardware
Tested with a tunnel in a real iOS device.
## Test gaps (from review)
- `userMuted` stickiness (mirror of the existing `outputMuted` test).
- **OR invariant** between `outputMuted` and `userMuted` — explicit test that setting one false while the other is true keeps `el.muted === true`.
- **Contract pin:** `syncRuntimeMedia` fires `onAutoplayBlocked` on **every** rejection (no internal dedupe) — so a future refactor can't quietly move the latch and break the caller's posting logic.
- **Caller-side latch pattern:** a 5-rejection simulation with the init.ts-style wrapper posts exactly once.
- **`audioownershipchange`** **dispatch** on promotion + once per transition (no duplicate on idempotent re-promote).
- **Mid-playback promotion:** `_paused = false` at flip time fires `_playParentMedia` immediately.
- **`playbackerror`** **surface** on parent proxy rejection with the right `source` tag.
## Minor
- One-line comment on `_promoteToParentProxy` explaining the `postMessage` async race (the mute lands after ~one message-loop tick; the autoplay gate that triggered promotion keeps the iframe rejecting `play()` during that window, so the double-play bug doesn't reappear).
## What's good (from the review)
Kept as-is — noted for posterity:
- `muted` vs `volume` framing (orthogonal channels).
- Probing reality via `NotAllowedError` instead of `matchMedia('(pointer: coarse)')` / UA sniffing.
- Two orthogonal mute channels.
- Backwards compat (new actions / messages safely ignored by either side).
## Test results
- `packages/core/src/runtime/media.test.ts` — **42 tests pass** (+4 new: `userMuted` sticky, OR invariant, fires-every-rejection, caller-latch dedupe)
- `packages/core/src/runtime/bridge.test.ts` — **15 tests pass**
- `packages/player/src/hyperframes-player.test.ts` — **26 tests pass** (+3 new: `audioownershipchange` dispatch, mid-playback promotion, `playbackerror` surface)
- Typecheck green on `core` + `player`
- `tsup` build green on `core` / `player` / `cli`
- Live factory-series-c-video repro via agent-browser: runtime ownership still zero `volumechange` thrash, zero `PARENT.play()` calls; parent ownership measures 27–37 ms steady-state drift, well inside the 50 ms threshold.
## Test plan
- [x] Unit tests (83 total across touched files)
- [x] Typecheck clean
- [x] Build clean
- [x] Live studio repro on factory-series-c-video: runtime path unchanged, parent path drift tightened
- [x] `audioownershipchange` event fires with correct detail on synthetic autoplay block
- [x] Physical iOS / Android device verification (unchanged since #298)
Closes#309. Full credit to @gigadeniga for the diagnosis — the root cause + proposed fix in that issue are exactly what landed here.
## The bug
\`npx hyperframes preview\` failed deterministically on Crostini (ChromeOS Linux) with \`Ports 3002–3101 are all in use\`, even when nothing was actually listening on any of them.
## Why
\`testPortOnAllHosts\` ran four probes in parallel:
\`\`\`ts
const hosts = ["127.0.0.1", "0.0.0.0", "::1", "::"];
const results = await Promise.all(hosts.map((h) => isPortAvailableOnHost(port, h)));
\`\`\`
Each probe binds a socket and then calls \`server.close()\`. Close is async — the socket stays open until its callback fires on the next event-loop tick. While it's open, the wildcard binds (\`0.0.0.0\`, \`::\`) that include the loopback address race the still-open loopback socket and return \`EADDRINUSE\` spuriously. On Crostini this happens 100% of the time; other Linux configs hit it intermittently; macOS is less predictable. Net effect: every port in the 100-port scan range appears busy and the preview refuses to start.
Reproduces on any Linux box with the standalone snippet from the issue:
\`\`\`
127.0.0.1: OK
0.0.0.0: EADDRINUSE ← false positive
::1: OK
::: EADDRINUSE ← false positive
\`\`\`
## Fix
Serialize the probes. Each socket is fully closed before the next opens, eliminating the race window entirely.
\`\`\`ts
for (const host of hosts) {
const available = await isPortAvailableOnHost(port, host);
if (!available) return false;
}
return true;
\`\`\`
Kept the four-host check rather than collapsing to just \`0.0.0.0\` + \`::\` — the multi-host coverage is load-bearing for the devbox / SSH-forwarding case where a port is free on loopback but held on the wildcard. Sequentializing is the smaller, less-behaviourally-affecting fix.
## Regression tests
\`packages/cli/src/server/portUtils.test.ts\` — three cases binding real sockets, no mocks:
- **Returns true for a genuinely free port** — directly reproduces the Crostini bug; would fail on Linux against the parallel implementation.
- **Returns false when the port is occupied on \`0.0.0.0\`** — confirms the multi-host check still catches the devbox scenario.
- **Releases each probe socket before the next run** — two back-to-back calls for the same free port both return true, pinning the sequential contract against future refactors that might try to reparallelize for perf.
## Test plan
- [x] \`bunx vitest run packages/cli/src/server/portUtils.test.ts\` — 3/3 pass
- [x] Full CLI suite — 109/109 pass
- [x] \`tsc --noEmit\` clean
## Notes
- Independent of any version bump; ship whenever.
- Probing 4 hosts serially adds at most ~tens of milliseconds per port on the scan (binds are very fast on loopback). The worst-case cost shows up when the first port in the range is free — previously 1 parallel round-trip, now 4 sequential — and it's imperceptible (\`preview\` bind is a one-time startup cost, not a hot path).
Closes#304.
## Summary
The three `/schema/*.json` URLs baked into every Hyperframes project as `\$schema` references are 404ing on the live docs site — blocking editor autocomplete and validation.
- \`https://hyperframes.heygen.com/schema/hyperframes.json\` — **404** (missing entirely)
- \`https://hyperframes.heygen.com/schema/registry.json\` — **404** (only in npm package)
- \`https://hyperframes.heygen.com/schema/registry-item.json\` — **404** (only in npm package)
Mintlify serves top-level non-MDX dirs in \`docs/\` at \`/\<dir>/*\` (confirmed by \`docs/logo/*.svg\` → \`/logo/*.svg\`). This PR drops the three schemas into \`docs/schema/\` so the URLs resolve.
## What changed
| File | Role |
|---|---|
| \`docs/schema/hyperframes.json\` | **New.** Authored from the \`ProjectConfig\` type in \`packages/cli/src/utils/projectConfig.ts\`. |
| \`docs/schema/registry.json\` | Mirror of \`packages/core/schemas/registry.json\`. |
| \`docs/schema/registry-item.json\` | Mirror of \`packages/core/schemas/registry-item.json\`. |
| \`scripts/sync-schemas.ts\` | Keeps the registry mirrors in lockstep with their authoritative copies in \`packages/core/schemas/\`. \`--check\` mode fails the Docs workflow on drift. |
| \`.github/workflows/docs.yml\` | Runs \`tsx scripts/sync-schemas.ts --check\` on every PR touching docs or core schemas. |
| \`package.json\` | \`sync-schemas\` / \`sync-schemas:check\` npm scripts. |
## Why not make \`packages/core/schemas/\` authoritative for \`hyperframes.json\` too?
\`hyperframes.json\` is CLI config, not a core type. Keeping the schema in \`docs/\` avoids an artificial dependency between \`@hyperframes/core\` and \`@hyperframes/cli\`. If the two ever need to align, we can flip the direction then.
## Verification
- \`bun run sync-schemas:check\` → \`2/2 in sync\`.
- Ajv (draft 2020-12, in-process) validation against 9 cases:
- ✓ real factory-series-c-video config
- ✓ default shape from \`hyperframes init\`
- ✓ \`\$schema\` is optional
- ✓ missing registry → rejected
- ✓ missing paths.assets → rejected
- ✓ extra top-level key → rejected
- ✓ empty registry string → rejected
- ✓ empty block path → rejected
- ✓ missing paths entirely → rejected
## Test plan
- [x] \`tsx scripts/sync-schemas.ts --check\` passes locally
- [x] Schemas parse as valid JSON and validate real/default project configs
- [x] After merge: \`curl -sI https://hyperframes.heygen.com/schema/hyperframes.json\` returns 200 once Mintlify redeploys
- [x] Same check for \`/schema/registry.json\` and \`/schema/registry-item.json\`
- [x] VS Code autocomplete and error-highlighting work on \`hyperframes.json\` without extra config
## Notes
- The Docs workflow now triggers on \`packages/core/schemas/**\` and \`scripts/sync-schemas.ts\` in addition to \`docs/**\`, so a core-schemas change that forgets to run \`sync-schemas\` will fail CI instead of silently publishing stale docs.
- No runtime / API changes to any package; ship independent of a version bump.
Move all preview mp4/png/gif assets under docs/images/ out of the repo
and serve them from https://static.heygen.ai/hyperframes-oss/docs/images/
(backed by s3://heygen-public/hyperframes-oss/docs/images/, CloudFront).
Drops ~49MB from the working tree and, more importantly, ~49MB from every
future Mintlify build checkout. Combined with the (already-LFS-tracked)
producer snapshots, the remaining bloat in 'npx skills add heygen-com/
hyperframes' (see #300) is LFS smudge during clone — separate fix needed
in the skills CLI to pass GIT_LFS_SKIP_SMUDGE=1.
Changes:
- Delete docs/images/** (103 files, ~49MB). Files are uploaded to S3 already.
- Rewrite /images/* references in 44 MDX files, TemplateCard.jsx, and
catalog-index.json to absolute CDN URLs.
- Update README.md img src to CDN URL (renders correctly on GitHub).
- Add docs/images/ to .gitignore so regenerated previews aren't committed.
- Add scripts/upload-docs-images.sh to sync docs/images/ → S3 after running
the preview generators.
- Wire up bun run upload:docs-images and bun run generate:catalog-previews
scripts in package.json.
- Update generator script docstrings to point at the upload step.
External contributors can still regenerate previews locally (mintlify dev
reads the CDN URLs, so broken previews appear only for newly added items
pending a maintainer upload). Maintainers run:
bun run generate:catalog-previews --only <name>
bun run upload:docs-images
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Vance Ingalls