mirror of
https://github.com/heygen-com/hyperframes.git
synced 2026-09-14 18:01:20 +08:00
c49181f1fa
Follow-up to PR #298 addressing @jrusso1020's review. Each item below maps to a point in his comment. ## Significant ### 1\. Drift threshold 150 ms → 50 ms _mirrorParentMediaTime_ was too loose for lip-synced talking-head content. ITU-R BT.1359 puts A/V perceptibility at ±45 ms; 150 ms sat well inside the "unacceptable" zone. Dropped to 50 ms, extracted as a static constant for clarity. **Verified live on factory-series-c-video (agent-browser):** steady-state offset under parent ownership sampled five times over 400 ms = `[35.7, 33.5, 31.2, 27.2, 36.9]` ms — below the perceptibility floor. Before this PR the same measurement could drift up to 150 ms before correction. ### 2\. Dynamic sub-composition media proxies Under parent ownership, a sub-composition that attaches a new `<audio data-start>` mid-playback was correctly silenced in the iframe (sticky `outputMuted`) but had no parent-frame counterpart to play → silent hole in the audio track. Added a `MutationObserver` on the iframe body watching for `audio[data-start]` / `video[data-start]` additions. New elements are adopted through the same `_adoptIframeMedia` helper the initial scan uses, and if parent ownership is already active the new proxy gets its `currentTime` mirrored and `play()` called immediately (gated on `!this._paused`). Observer disconnects on iframe reload + component disconnect. ### 3\. `bridgeMuted` sticky in `syncRuntimeMedia` The asymmetry James flagged: `outputMuted` was sticky per-tick, `bridgeMuted` was one-shot via `onSetMuted`. A sub-composition activating after a user mute would briefly play at author volume before the next bridge message. `syncRuntimeMedia` now accepts `userMuted` and the per-clip loop uses a single combined `shouldMute` gate. One invariant, two inputs. ### 4\. Reset `_audioOwner` on iframe reload The latch never cleared. On composition switch the player would stay in `parent` ownership against a fresh runtime that hadn't received `set-media-output-muted` and whose autoplay-blocked latch was clean — a brief double-audio window until the next `NotAllowedError` re-promoted (idempotently). `_onIframeLoad` now resets `_audioOwner = "runtime"`, pauses any parent proxies, and disconnects the old MutationObserver before a fresh one attaches to the new document. If the player had been in `parent` ownership, a corresponding `audioownershipchange` event fires with `reason: "iframe-reload"`. ## Worth addressing ### 5\. Promotion → observable event + reason Promotion was invisible. Added `CustomEvent("audioownershipchange", { detail: { owner, reason } })` fired on every owner transition. `reason` is either `"autoplay-blocked"` (promote → parent) or `"iframe-reload"` (reset → runtime). Gives host apps an SLO-ready signal for "% of sessions in parent ownership" without exposing internal state. **Verified live:** dispatching a synthetic `media-autoplay-blocked` in the live studio produced `{ owner: "parent", reason: "autoplay-blocked" }` on the web component exactly once. ### 6\. Parent proxy play() rejection → `playbackerror` event Previously swallowed silently. Now re-emitted as `CustomEvent("playbackerror", { detail: { source: "parent-proxy", error } })` so embedding apps can recover or fall back. ### 7\. Mobile verification on real hardware Tested with a tunnel in a real iOS device. ## Test gaps (from review) - `userMuted` stickiness (mirror of the existing `outputMuted` test). - **OR invariant** between `outputMuted` and `userMuted` — explicit test that setting one false while the other is true keeps `el.muted === true`. - **Contract pin:** `syncRuntimeMedia` fires `onAutoplayBlocked` on **every** rejection (no internal dedupe) — so a future refactor can't quietly move the latch and break the caller's posting logic. - **Caller-side latch pattern:** a 5-rejection simulation with the init.ts-style wrapper posts exactly once. - **`audioownershipchange`** **dispatch** on promotion + once per transition (no duplicate on idempotent re-promote). - **Mid-playback promotion:** `_paused = false` at flip time fires `_playParentMedia` immediately. - **`playbackerror`** **surface** on parent proxy rejection with the right `source` tag. ## Minor - One-line comment on `_promoteToParentProxy` explaining the `postMessage` async race (the mute lands after ~one message-loop tick; the autoplay gate that triggered promotion keeps the iframe rejecting `play()` during that window, so the double-play bug doesn't reappear). ## What's good (from the review) Kept as-is — noted for posterity: - `muted` vs `volume` framing (orthogonal channels). - Probing reality via `NotAllowedError` instead of `matchMedia('(pointer: coarse)')` / UA sniffing. - Two orthogonal mute channels. - Backwards compat (new actions / messages safely ignored by either side). ## Test results - `packages/core/src/runtime/media.test.ts` — **42 tests pass** (+4 new: `userMuted` sticky, OR invariant, fires-every-rejection, caller-latch dedupe) - `packages/core/src/runtime/bridge.test.ts` — **15 tests pass** - `packages/player/src/hyperframes-player.test.ts` — **26 tests pass** (+3 new: `audioownershipchange` dispatch, mid-playback promotion, `playbackerror` surface) - Typecheck green on `core` + `player` - `tsup` build green on `core` / `player` / `cli` - Live factory-series-c-video repro via agent-browser: runtime ownership still zero `volumechange` thrash, zero `PARENT.play()` calls; parent ownership measures 27–37 ms steady-state drift, well inside the 50 ms threshold. ## Test plan - [x] Unit tests (83 total across touched files) - [x] Typecheck clean - [x] Build clean - [x] Live studio repro on factory-series-c-video: runtime path unchanged, parent path drift tightened - [x] `audioownershipchange` event fires with correct detail on synthetic autoplay block - [x] Physical iOS / Android device verification (unchanged since #298)
325 lines
11 KiB
TypeScript
325 lines
11 KiB
TypeScript
import { describe, it, expect, vi, beforeEach, afterEach } from "vitest";
|
|
import { formatTime, formatSpeed, SPEED_PRESETS } from "./controls.js";
|
|
|
|
// ── Controls unit tests ──
|
|
|
|
describe("SPEED_PRESETS", () => {
|
|
it("contains logarithmic speed steps", () => {
|
|
expect(SPEED_PRESETS).toEqual([0.25, 0.5, 1, 1.5, 2, 4]);
|
|
});
|
|
|
|
it("includes 1x as default speed", () => {
|
|
expect(SPEED_PRESETS).toContain(1);
|
|
});
|
|
});
|
|
|
|
describe("formatSpeed", () => {
|
|
it("formats integer speeds", () => {
|
|
expect(formatSpeed(1)).toBe("1x");
|
|
expect(formatSpeed(2)).toBe("2x");
|
|
expect(formatSpeed(4)).toBe("4x");
|
|
});
|
|
|
|
it("formats fractional speeds", () => {
|
|
expect(formatSpeed(0.25)).toBe("0.25x");
|
|
expect(formatSpeed(0.5)).toBe("0.5x");
|
|
expect(formatSpeed(1.5)).toBe("1.5x");
|
|
});
|
|
});
|
|
|
|
describe("formatTime", () => {
|
|
it("formats 0 seconds", () => {
|
|
expect(formatTime(0)).toBe("0:00");
|
|
});
|
|
|
|
it("formats seconds under a minute", () => {
|
|
expect(formatTime(45)).toBe("0:45");
|
|
});
|
|
|
|
it("formats exact minutes", () => {
|
|
expect(formatTime(120)).toBe("2:00");
|
|
});
|
|
|
|
it("formats minutes and seconds", () => {
|
|
expect(formatTime(95)).toBe("1:35");
|
|
});
|
|
|
|
it("pads seconds with leading zero", () => {
|
|
expect(formatTime(61)).toBe("1:01");
|
|
});
|
|
|
|
it("floors fractional seconds", () => {
|
|
expect(formatTime(3.7)).toBe("0:03");
|
|
});
|
|
|
|
it("handles negative input", () => {
|
|
expect(formatTime(-5)).toBe("0:00");
|
|
});
|
|
});
|
|
|
|
// ── Parent-frame audio proxies (ownership-based) ──
|
|
//
|
|
// Parent-frame audio/video copies are preloaded mirror proxies of the iframe's
|
|
// timed media. They exist as a fallback for environments that block iframe
|
|
// `.play()`. Under the default `runtime` audio ownership, the iframe drives
|
|
// audible playback and the proxies stay paused. Ownership flips to `parent`
|
|
// only when the runtime posts `media-autoplay-blocked` — then the proxies
|
|
// become the audible source and the iframe is silenced via bridge.
|
|
|
|
describe("HyperframesPlayer parent-frame media", () => {
|
|
type PlayerElement = HTMLElement & {
|
|
play: () => void;
|
|
pause: () => void;
|
|
seek: (t: number) => void;
|
|
_audioOwner?: "runtime" | "parent";
|
|
_promoteToParentProxy?: () => void;
|
|
};
|
|
|
|
let player: PlayerElement;
|
|
let mockAudio: {
|
|
src: string;
|
|
preload: string;
|
|
muted: boolean;
|
|
playbackRate: number;
|
|
currentTime: number;
|
|
paused: boolean;
|
|
play: ReturnType<typeof vi.fn>;
|
|
pause: ReturnType<typeof vi.fn>;
|
|
load: ReturnType<typeof vi.fn>;
|
|
};
|
|
|
|
beforeEach(async () => {
|
|
await import("./hyperframes-player.js");
|
|
|
|
mockAudio = {
|
|
src: "",
|
|
preload: "",
|
|
muted: false,
|
|
playbackRate: 1,
|
|
currentTime: 0,
|
|
paused: true,
|
|
play: vi.fn().mockResolvedValue(undefined),
|
|
pause: vi.fn(),
|
|
load: vi.fn(),
|
|
};
|
|
|
|
vi.spyOn(globalThis, "Audio").mockImplementation(
|
|
() => mockAudio as unknown as HTMLAudioElement,
|
|
);
|
|
|
|
player = document.createElement("hyperframes-player") as PlayerElement;
|
|
});
|
|
|
|
afterEach(() => {
|
|
player.remove();
|
|
vi.restoreAllMocks();
|
|
});
|
|
|
|
it("includes audio-src in observedAttributes", () => {
|
|
const Ctor = player.constructor as typeof HTMLElement & {
|
|
observedAttributes: string[];
|
|
};
|
|
expect(Ctor.observedAttributes).toContain("audio-src");
|
|
});
|
|
|
|
it("creates Audio and starts preloading when audio-src is set", () => {
|
|
player.setAttribute("audio-src", "https://cdn.example.com/narration.mp3");
|
|
document.body.appendChild(player);
|
|
|
|
expect(globalThis.Audio).toHaveBeenCalled();
|
|
expect(mockAudio.preload).toBe("auto");
|
|
expect(mockAudio.src).toBe("https://cdn.example.com/narration.mp3");
|
|
expect(mockAudio.load).toHaveBeenCalled();
|
|
});
|
|
|
|
it("syncs muted attribute to parent media", () => {
|
|
player.setAttribute("muted", "");
|
|
player.setAttribute("audio-src", "https://cdn.example.com/narration.mp3");
|
|
document.body.appendChild(player);
|
|
|
|
expect(mockAudio.muted).toBe(true);
|
|
});
|
|
|
|
it("syncs playback-rate to parent media", () => {
|
|
player.setAttribute("playback-rate", "1.5");
|
|
player.setAttribute("audio-src", "https://cdn.example.com/narration.mp3");
|
|
document.body.appendChild(player);
|
|
|
|
expect(mockAudio.playbackRate).toBe(1.5);
|
|
});
|
|
|
|
it("play() does NOT start parent-proxy under runtime ownership", () => {
|
|
// Default ownership is `runtime` — the iframe drives audible playback.
|
|
// If we also started parent proxies here, both would play and the user
|
|
// would hear doubled, slightly-offset audio (the original bug).
|
|
player.setAttribute("audio-src", "https://cdn.example.com/narration.mp3");
|
|
document.body.appendChild(player);
|
|
|
|
player.play();
|
|
expect(mockAudio.play).not.toHaveBeenCalled();
|
|
expect(player._audioOwner).toBe("runtime");
|
|
});
|
|
|
|
it("pause() does NOT touch parent-proxy under runtime ownership", () => {
|
|
player.setAttribute("audio-src", "https://cdn.example.com/narration.mp3");
|
|
document.body.appendChild(player);
|
|
|
|
player.pause();
|
|
expect(mockAudio.pause).not.toHaveBeenCalled();
|
|
});
|
|
|
|
it("seek() does NOT update parent currentTime under runtime ownership", () => {
|
|
// Under runtime ownership the iframe is authoritative for time; touching
|
|
// the proxy's currentTime would just trigger a re-buffer for no gain.
|
|
player.setAttribute("audio-src", "https://cdn.example.com/narration.mp3");
|
|
document.body.appendChild(player);
|
|
|
|
player.seek(12.5);
|
|
expect(mockAudio.currentTime).toBe(0);
|
|
});
|
|
|
|
it("after promotion to parent ownership: play/pause/seek drive parent proxy", () => {
|
|
// Simulates the runtime having posted `media-autoplay-blocked`. Post
|
|
// promotion: the web component owns audible output and fully drives
|
|
// the parent proxy.
|
|
player.setAttribute("audio-src", "https://cdn.example.com/narration.mp3");
|
|
document.body.appendChild(player);
|
|
|
|
player._promoteToParentProxy?.();
|
|
expect(player._audioOwner).toBe("parent");
|
|
|
|
player.play();
|
|
expect(mockAudio.play).toHaveBeenCalled();
|
|
|
|
player.seek(12.5);
|
|
expect(mockAudio.currentTime).toBe(12.5);
|
|
|
|
player.pause();
|
|
expect(mockAudio.pause).toHaveBeenCalled();
|
|
});
|
|
|
|
it("promotion is idempotent", () => {
|
|
player.setAttribute("audio-src", "https://cdn.example.com/narration.mp3");
|
|
document.body.appendChild(player);
|
|
|
|
player._promoteToParentProxy?.();
|
|
player._promoteToParentProxy?.();
|
|
player._promoteToParentProxy?.();
|
|
// Only one play() attempt is triggered by promotion itself (gated on
|
|
// `!this._paused`, which is true by default so it doesn't trigger at all).
|
|
// The test's meaning is: ownership stays `parent`, no thrash, no errors.
|
|
expect(player._audioOwner).toBe("parent");
|
|
});
|
|
|
|
it("dispatches audioownershipchange on promotion", () => {
|
|
player.setAttribute("audio-src", "https://cdn.example.com/narration.mp3");
|
|
document.body.appendChild(player);
|
|
|
|
const events: Array<{ owner: string; reason: string }> = [];
|
|
player.addEventListener("audioownershipchange", (e: Event) => {
|
|
const detail = (e as CustomEvent<{ owner: string; reason: string }>).detail;
|
|
events.push(detail);
|
|
});
|
|
|
|
player._promoteToParentProxy?.();
|
|
expect(events).toEqual([{ owner: "parent", reason: "autoplay-blocked" }]);
|
|
|
|
// Second promote is idempotent — no duplicate event.
|
|
player._promoteToParentProxy?.();
|
|
expect(events).toHaveLength(1);
|
|
});
|
|
|
|
it("promotion mid-playback plays parent proxy immediately", () => {
|
|
// Previously-missing coverage: if the user is already playing when
|
|
// the runtime reports autoplay-blocked, the proxy must start audible
|
|
// right away — not wait for the user to hit pause/play again.
|
|
player.setAttribute("audio-src", "https://cdn.example.com/narration.mp3");
|
|
document.body.appendChild(player);
|
|
|
|
player.play(); // `_paused = false`, owner still `runtime` → no parent play yet
|
|
expect(mockAudio.play).not.toHaveBeenCalled();
|
|
|
|
player._promoteToParentProxy?.();
|
|
expect(mockAudio.play).toHaveBeenCalled();
|
|
});
|
|
|
|
it("surfaces playbackerror when parent proxy play() rejects", async () => {
|
|
player.setAttribute("audio-src", "https://cdn.example.com/narration.mp3");
|
|
document.body.appendChild(player);
|
|
|
|
const rejection = Object.assign(new Error("blocked"), { name: "NotAllowedError" });
|
|
mockAudio.play = vi.fn().mockRejectedValueOnce(rejection);
|
|
|
|
const errors: unknown[] = [];
|
|
player.addEventListener("playbackerror", (e: Event) => {
|
|
errors.push((e as CustomEvent).detail);
|
|
});
|
|
|
|
player._promoteToParentProxy?.();
|
|
player.play();
|
|
// Promise rejection delivered on a microtask — flush.
|
|
await Promise.resolve();
|
|
await Promise.resolve();
|
|
|
|
expect(errors.length).toBeGreaterThan(0);
|
|
expect((errors[0] as { source: string }).source).toBe("parent-proxy");
|
|
});
|
|
|
|
it("playbackerror dedup: fires at most once per parent-ownership session", async () => {
|
|
// Under parent ownership with parent-also-blocked, every iframe
|
|
// paused→playing transition in the state loop re-invokes `_playParentMedia`.
|
|
// Without a latch, each rejection would re-fire `playbackerror`, spamming
|
|
// subscribers. Mirrors the runtime's `mediaAutoplayBlockedPosted` latch.
|
|
player.setAttribute("audio-src", "https://cdn.example.com/narration.mp3");
|
|
document.body.appendChild(player);
|
|
|
|
const rejection = Object.assign(new Error("blocked"), { name: "NotAllowedError" });
|
|
mockAudio.play = vi.fn().mockRejectedValue(rejection);
|
|
|
|
const errors: unknown[] = [];
|
|
player.addEventListener("playbackerror", (e: Event) => {
|
|
errors.push((e as CustomEvent).detail);
|
|
});
|
|
|
|
player._promoteToParentProxy?.();
|
|
player.play();
|
|
player.pause();
|
|
player.play();
|
|
player.pause();
|
|
player.play();
|
|
await Promise.resolve();
|
|
await Promise.resolve();
|
|
await Promise.resolve();
|
|
|
|
expect(errors).toHaveLength(1);
|
|
});
|
|
|
|
it("cleans up parent media on disconnect", () => {
|
|
player.setAttribute("audio-src", "https://cdn.example.com/narration.mp3");
|
|
document.body.appendChild(player);
|
|
|
|
player.remove();
|
|
expect(mockAudio.pause).toHaveBeenCalled();
|
|
expect(mockAudio.src).toBe("");
|
|
});
|
|
|
|
it("updates parent media when playback-rate changes after setup", () => {
|
|
player.setAttribute("audio-src", "https://cdn.example.com/narration.mp3");
|
|
document.body.appendChild(player);
|
|
|
|
player.setAttribute("playback-rate", "2");
|
|
expect(mockAudio.playbackRate).toBe(2);
|
|
});
|
|
|
|
it("updates parent media when muted toggles after setup", () => {
|
|
player.setAttribute("audio-src", "https://cdn.example.com/narration.mp3");
|
|
document.body.appendChild(player);
|
|
|
|
player.setAttribute("muted", "");
|
|
expect(mockAudio.muted).toBe(true);
|
|
|
|
player.removeAttribute("muted");
|
|
expect(mockAudio.muted).toBe(false);
|
|
});
|
|
});
|