Files
heygen-com__hyperframes/packages/player/src/hyperframes-player.test.ts
Miguel Ángel c49181f1fa fix(player): address #298 review — tighter drift, dynamic proxies, ownership event (#307)
Follow-up to PR #298 addressing @jrusso1020's review. Each item below maps to a point in his comment.

## Significant

### 1\. Drift threshold 150 ms → 50 ms

_mirrorParentMediaTime_ was too loose for lip-synced talking-head content. ITU-R BT.1359 puts A/V perceptibility at ±45 ms; 150 ms sat well inside the "unacceptable" zone. Dropped to 50 ms, extracted as a static constant for clarity.

**Verified live on factory-series-c-video (agent-browser):** steady-state offset under parent ownership sampled five times over 400 ms = `[35.7, 33.5, 31.2, 27.2, 36.9]` ms — below the perceptibility floor. Before this PR the same measurement could drift up to 150 ms before correction.

### 2\. Dynamic sub-composition media proxies

Under parent ownership, a sub-composition that attaches a new `<audio data-start>` mid-playback was correctly silenced in the iframe (sticky `outputMuted`) but had no parent-frame counterpart to play → silent hole in the audio track.

Added a `MutationObserver` on the iframe body watching for `audio[data-start]` / `video[data-start]` additions. New elements are adopted through the same `_adoptIframeMedia` helper the initial scan uses, and if parent ownership is already active the new proxy gets its `currentTime` mirrored and `play()` called immediately (gated on `!this._paused`). Observer disconnects on iframe reload + component disconnect.

### 3\. `bridgeMuted` sticky in `syncRuntimeMedia`

The asymmetry James flagged: `outputMuted` was sticky per-tick, `bridgeMuted` was one-shot via `onSetMuted`. A sub-composition activating after a user mute would briefly play at author volume before the next bridge message.

`syncRuntimeMedia` now accepts `userMuted` and the per-clip loop uses a single combined `shouldMute` gate. One invariant, two inputs.

### 4\. Reset `_audioOwner` on iframe reload

The latch never cleared. On composition switch the player would stay in `parent` ownership against a fresh runtime that hadn't received `set-media-output-muted` and whose autoplay-blocked latch was clean — a brief double-audio window until the next `NotAllowedError` re-promoted (idempotently).

`_onIframeLoad` now resets `_audioOwner = "runtime"`, pauses any parent proxies, and disconnects the old MutationObserver before a fresh one attaches to the new document. If the player had been in `parent` ownership, a corresponding `audioownershipchange` event fires with `reason: "iframe-reload"`.

## Worth addressing

### 5\. Promotion → observable event + reason

Promotion was invisible. Added `CustomEvent("audioownershipchange", { detail: { owner, reason } })` fired on every owner transition. `reason` is either `"autoplay-blocked"` (promote → parent) or `"iframe-reload"` (reset → runtime). Gives host apps an SLO-ready signal for "% of sessions in parent ownership" without exposing internal state.

**Verified live:** dispatching a synthetic `media-autoplay-blocked` in the live studio produced `{ owner: "parent", reason: "autoplay-blocked" }` on the web component exactly once.

### 6\. Parent proxy play() rejection → `playbackerror` event

Previously swallowed silently. Now re-emitted as `CustomEvent("playbackerror", { detail: { source: "parent-proxy", error } })` so embedding apps can recover or fall back.

### 7\. Mobile verification on real hardware

Tested with a tunnel in a real iOS device.

## Test gaps (from review)

- `userMuted` stickiness (mirror of the existing `outputMuted` test).
- **OR invariant** between `outputMuted` and `userMuted` — explicit test that setting one false while the other is true keeps `el.muted === true`.
- **Contract pin:** `syncRuntimeMedia` fires `onAutoplayBlocked` on **every** rejection (no internal dedupe) — so a future refactor can't quietly move the latch and break the caller's posting logic.
- **Caller-side latch pattern:** a 5-rejection simulation with the init.ts-style wrapper posts exactly once.
- **`audioownershipchange`** **dispatch** on promotion + once per transition (no duplicate on idempotent re-promote).
- **Mid-playback promotion:** `_paused = false` at flip time fires `_playParentMedia` immediately.
- **`playbackerror`** **surface** on parent proxy rejection with the right `source` tag.

## Minor

- One-line comment on `_promoteToParentProxy` explaining the `postMessage` async race (the mute lands after ~one message-loop tick; the autoplay gate that triggered promotion keeps the iframe rejecting `play()` during that window, so the double-play bug doesn't reappear).

## What's good (from the review)

Kept as-is — noted for posterity:

- `muted` vs `volume` framing (orthogonal channels).
- Probing reality via `NotAllowedError` instead of `matchMedia('(pointer: coarse)')` / UA sniffing.
- Two orthogonal mute channels.
- Backwards compat (new actions / messages safely ignored by either side).

## Test results

- `packages/core/src/runtime/media.test.ts` — **42 tests pass** (+4 new: `userMuted` sticky, OR invariant, fires-every-rejection, caller-latch dedupe)
- `packages/core/src/runtime/bridge.test.ts` — **15 tests pass**
- `packages/player/src/hyperframes-player.test.ts` — **26 tests pass** (+3 new: `audioownershipchange` dispatch, mid-playback promotion, `playbackerror` surface)
- Typecheck green on `core` + `player`
- `tsup` build green on `core` / `player` / `cli`
- Live factory-series-c-video repro via agent-browser: runtime ownership still zero `volumechange` thrash, zero `PARENT.play()` calls; parent ownership measures 27–37 ms steady-state drift, well inside the 50 ms threshold.

## Test plan

- [x] Unit tests (83 total across touched files)
- [x] Typecheck clean
- [x] Build clean
- [x] Live studio repro on factory-series-c-video: runtime path unchanged, parent path drift tightened
- [x] `audioownershipchange` event fires with correct detail on synthetic autoplay block
- [x] Physical iOS / Android device verification (unchanged since #298)
2026-04-18 00:49:57 +02:00

325 lines
11 KiB
TypeScript

import { describe, it, expect, vi, beforeEach, afterEach } from "vitest";
import { formatTime, formatSpeed, SPEED_PRESETS } from "./controls.js";
// ── Controls unit tests ──
describe("SPEED_PRESETS", () => {
it("contains logarithmic speed steps", () => {
expect(SPEED_PRESETS).toEqual([0.25, 0.5, 1, 1.5, 2, 4]);
});
it("includes 1x as default speed", () => {
expect(SPEED_PRESETS).toContain(1);
});
});
describe("formatSpeed", () => {
it("formats integer speeds", () => {
expect(formatSpeed(1)).toBe("1x");
expect(formatSpeed(2)).toBe("2x");
expect(formatSpeed(4)).toBe("4x");
});
it("formats fractional speeds", () => {
expect(formatSpeed(0.25)).toBe("0.25x");
expect(formatSpeed(0.5)).toBe("0.5x");
expect(formatSpeed(1.5)).toBe("1.5x");
});
});
describe("formatTime", () => {
it("formats 0 seconds", () => {
expect(formatTime(0)).toBe("0:00");
});
it("formats seconds under a minute", () => {
expect(formatTime(45)).toBe("0:45");
});
it("formats exact minutes", () => {
expect(formatTime(120)).toBe("2:00");
});
it("formats minutes and seconds", () => {
expect(formatTime(95)).toBe("1:35");
});
it("pads seconds with leading zero", () => {
expect(formatTime(61)).toBe("1:01");
});
it("floors fractional seconds", () => {
expect(formatTime(3.7)).toBe("0:03");
});
it("handles negative input", () => {
expect(formatTime(-5)).toBe("0:00");
});
});
// ── Parent-frame audio proxies (ownership-based) ──
//
// Parent-frame audio/video copies are preloaded mirror proxies of the iframe's
// timed media. They exist as a fallback for environments that block iframe
// `.play()`. Under the default `runtime` audio ownership, the iframe drives
// audible playback and the proxies stay paused. Ownership flips to `parent`
// only when the runtime posts `media-autoplay-blocked` — then the proxies
// become the audible source and the iframe is silenced via bridge.
describe("HyperframesPlayer parent-frame media", () => {
type PlayerElement = HTMLElement & {
play: () => void;
pause: () => void;
seek: (t: number) => void;
_audioOwner?: "runtime" | "parent";
_promoteToParentProxy?: () => void;
};
let player: PlayerElement;
let mockAudio: {
src: string;
preload: string;
muted: boolean;
playbackRate: number;
currentTime: number;
paused: boolean;
play: ReturnType<typeof vi.fn>;
pause: ReturnType<typeof vi.fn>;
load: ReturnType<typeof vi.fn>;
};
beforeEach(async () => {
await import("./hyperframes-player.js");
mockAudio = {
src: "",
preload: "",
muted: false,
playbackRate: 1,
currentTime: 0,
paused: true,
play: vi.fn().mockResolvedValue(undefined),
pause: vi.fn(),
load: vi.fn(),
};
vi.spyOn(globalThis, "Audio").mockImplementation(
() => mockAudio as unknown as HTMLAudioElement,
);
player = document.createElement("hyperframes-player") as PlayerElement;
});
afterEach(() => {
player.remove();
vi.restoreAllMocks();
});
it("includes audio-src in observedAttributes", () => {
const Ctor = player.constructor as typeof HTMLElement & {
observedAttributes: string[];
};
expect(Ctor.observedAttributes).toContain("audio-src");
});
it("creates Audio and starts preloading when audio-src is set", () => {
player.setAttribute("audio-src", "https://cdn.example.com/narration.mp3");
document.body.appendChild(player);
expect(globalThis.Audio).toHaveBeenCalled();
expect(mockAudio.preload).toBe("auto");
expect(mockAudio.src).toBe("https://cdn.example.com/narration.mp3");
expect(mockAudio.load).toHaveBeenCalled();
});
it("syncs muted attribute to parent media", () => {
player.setAttribute("muted", "");
player.setAttribute("audio-src", "https://cdn.example.com/narration.mp3");
document.body.appendChild(player);
expect(mockAudio.muted).toBe(true);
});
it("syncs playback-rate to parent media", () => {
player.setAttribute("playback-rate", "1.5");
player.setAttribute("audio-src", "https://cdn.example.com/narration.mp3");
document.body.appendChild(player);
expect(mockAudio.playbackRate).toBe(1.5);
});
it("play() does NOT start parent-proxy under runtime ownership", () => {
// Default ownership is `runtime` — the iframe drives audible playback.
// If we also started parent proxies here, both would play and the user
// would hear doubled, slightly-offset audio (the original bug).
player.setAttribute("audio-src", "https://cdn.example.com/narration.mp3");
document.body.appendChild(player);
player.play();
expect(mockAudio.play).not.toHaveBeenCalled();
expect(player._audioOwner).toBe("runtime");
});
it("pause() does NOT touch parent-proxy under runtime ownership", () => {
player.setAttribute("audio-src", "https://cdn.example.com/narration.mp3");
document.body.appendChild(player);
player.pause();
expect(mockAudio.pause).not.toHaveBeenCalled();
});
it("seek() does NOT update parent currentTime under runtime ownership", () => {
// Under runtime ownership the iframe is authoritative for time; touching
// the proxy's currentTime would just trigger a re-buffer for no gain.
player.setAttribute("audio-src", "https://cdn.example.com/narration.mp3");
document.body.appendChild(player);
player.seek(12.5);
expect(mockAudio.currentTime).toBe(0);
});
it("after promotion to parent ownership: play/pause/seek drive parent proxy", () => {
// Simulates the runtime having posted `media-autoplay-blocked`. Post
// promotion: the web component owns audible output and fully drives
// the parent proxy.
player.setAttribute("audio-src", "https://cdn.example.com/narration.mp3");
document.body.appendChild(player);
player._promoteToParentProxy?.();
expect(player._audioOwner).toBe("parent");
player.play();
expect(mockAudio.play).toHaveBeenCalled();
player.seek(12.5);
expect(mockAudio.currentTime).toBe(12.5);
player.pause();
expect(mockAudio.pause).toHaveBeenCalled();
});
it("promotion is idempotent", () => {
player.setAttribute("audio-src", "https://cdn.example.com/narration.mp3");
document.body.appendChild(player);
player._promoteToParentProxy?.();
player._promoteToParentProxy?.();
player._promoteToParentProxy?.();
// Only one play() attempt is triggered by promotion itself (gated on
// `!this._paused`, which is true by default so it doesn't trigger at all).
// The test's meaning is: ownership stays `parent`, no thrash, no errors.
expect(player._audioOwner).toBe("parent");
});
it("dispatches audioownershipchange on promotion", () => {
player.setAttribute("audio-src", "https://cdn.example.com/narration.mp3");
document.body.appendChild(player);
const events: Array<{ owner: string; reason: string }> = [];
player.addEventListener("audioownershipchange", (e: Event) => {
const detail = (e as CustomEvent<{ owner: string; reason: string }>).detail;
events.push(detail);
});
player._promoteToParentProxy?.();
expect(events).toEqual([{ owner: "parent", reason: "autoplay-blocked" }]);
// Second promote is idempotent — no duplicate event.
player._promoteToParentProxy?.();
expect(events).toHaveLength(1);
});
it("promotion mid-playback plays parent proxy immediately", () => {
// Previously-missing coverage: if the user is already playing when
// the runtime reports autoplay-blocked, the proxy must start audible
// right away — not wait for the user to hit pause/play again.
player.setAttribute("audio-src", "https://cdn.example.com/narration.mp3");
document.body.appendChild(player);
player.play(); // `_paused = false`, owner still `runtime` → no parent play yet
expect(mockAudio.play).not.toHaveBeenCalled();
player._promoteToParentProxy?.();
expect(mockAudio.play).toHaveBeenCalled();
});
it("surfaces playbackerror when parent proxy play() rejects", async () => {
player.setAttribute("audio-src", "https://cdn.example.com/narration.mp3");
document.body.appendChild(player);
const rejection = Object.assign(new Error("blocked"), { name: "NotAllowedError" });
mockAudio.play = vi.fn().mockRejectedValueOnce(rejection);
const errors: unknown[] = [];
player.addEventListener("playbackerror", (e: Event) => {
errors.push((e as CustomEvent).detail);
});
player._promoteToParentProxy?.();
player.play();
// Promise rejection delivered on a microtask — flush.
await Promise.resolve();
await Promise.resolve();
expect(errors.length).toBeGreaterThan(0);
expect((errors[0] as { source: string }).source).toBe("parent-proxy");
});
it("playbackerror dedup: fires at most once per parent-ownership session", async () => {
// Under parent ownership with parent-also-blocked, every iframe
// paused→playing transition in the state loop re-invokes `_playParentMedia`.
// Without a latch, each rejection would re-fire `playbackerror`, spamming
// subscribers. Mirrors the runtime's `mediaAutoplayBlockedPosted` latch.
player.setAttribute("audio-src", "https://cdn.example.com/narration.mp3");
document.body.appendChild(player);
const rejection = Object.assign(new Error("blocked"), { name: "NotAllowedError" });
mockAudio.play = vi.fn().mockRejectedValue(rejection);
const errors: unknown[] = [];
player.addEventListener("playbackerror", (e: Event) => {
errors.push((e as CustomEvent).detail);
});
player._promoteToParentProxy?.();
player.play();
player.pause();
player.play();
player.pause();
player.play();
await Promise.resolve();
await Promise.resolve();
await Promise.resolve();
expect(errors).toHaveLength(1);
});
it("cleans up parent media on disconnect", () => {
player.setAttribute("audio-src", "https://cdn.example.com/narration.mp3");
document.body.appendChild(player);
player.remove();
expect(mockAudio.pause).toHaveBeenCalled();
expect(mockAudio.src).toBe("");
});
it("updates parent media when playback-rate changes after setup", () => {
player.setAttribute("audio-src", "https://cdn.example.com/narration.mp3");
document.body.appendChild(player);
player.setAttribute("playback-rate", "2");
expect(mockAudio.playbackRate).toBe(2);
});
it("updates parent media when muted toggles after setup", () => {
player.setAttribute("audio-src", "https://cdn.example.com/narration.mp3");
document.body.appendChild(player);
player.setAttribute("muted", "");
expect(mockAudio.muted).toBe(true);
player.removeAttribute("muted");
expect(mockAudio.muted).toBe(false);
});
});