Files
heygen-com__hyperframes/.github/workflows/player-perf.yml
Vance Ingalls 80e7cd2844 perf(player): p0-1c live-playback parity test via SSIM (#401)
## Summary

Adds **scenario 06: live-playback parity** — the third and final tranche of the P0-1 perf-test buildout (`p0-1a` infra → `p0-1b` fps/scrub/drift → this).

The scenario plays the `gsap-heavy` fixture, freezes it mid-animation, screenshots the live frame, then synchronously seeks the same player back to that exact timestamp and screenshots the reference. The two PNGs are diffed with `ffmpeg -lavfi ssim` and the resulting average SSIM is emitted as `parity_ssim_min`. Baseline gate: **SSIM ≥ 0.95**.

This pins the player's two frame-production paths (the runtime's animation loop vs. `_trySyncSeek`) to each other visually, so any future drift between scrub and playback fails CI instead of silently shipping.

## Motivation

`<hyperframes-player>` produces frames two different ways:

1. **Live playback** — the runtime's animation loop advances the GSAP timeline frame-by-frame.
2. **Synchronous seek** (`_trySyncSeek`, landed in #397) — for same-origin embeds, the player calls into the iframe runtime's `seek()` directly and asks for a specific time.

These paths must agree. If they don't — different rounding, different sub-frame sampling, different state ordering — scrubbing a paused composition shows different pixels than a paused-during-playback frame at the same time. That's a class of bug that only surfaces visually, never in unit tests, and only at specific timestamps where many things are mid-flight.

`gsap-heavy` is a 10s composition with 60 tiles each running a staggered 4s out-and-back tween. At t=5.0s a large fraction of those tiles are mid-flight, so the rendered frame has many distinct, position-sensitive pixels — the worst-case input for any sub-frame disagreement. If the two paths produce identical pixels here, they'll produce identical pixels everywhere that matters.

## What changed

- **`packages/player/tests/perf/scenarios/06-parity.ts`** — new scenario (~340 lines). Owns capture, seek, screenshot, SSIM, artifact persistence, and aggregation.
- **`packages/player/tests/perf/index.ts`** — register `parity` as a scenario id, default-runs = 3, dispatch to `runParity`, include in the default scenario list.
- **`packages/player/tests/perf/perf-gate.ts`** — extend `PerfBaseline` with `paritySsimMin`.
- **`packages/player/tests/perf/baseline.json`** — `paritySsimMin: 0.95`.
- **`.github/workflows/player-perf.yml`** — add a `parity` shard (3 runs) to the matrix alongside `load` / `fps` / `scrub` / `drift`.

## How the scenario works

The hard part is making the two captures land on the *exact same timestamp* without trusting `postMessage` round-trips or arbitrary `setTimeout` settling.

1. **Install an iframe-side rAF watcher** before issuing `play()`. The watcher polls `__player.getTime()` every animation frame and, the first time `getTime() >= 5.0`, calls `__player.pause()` *from inside the same rAF tick*. `pause()` is synchronous (it calls `timeline.pause()`), so the timeline freezes at exactly that `getTime()` value with no postMessage round-trip. The watcher's Promise resolves with that frozen value as the canonical `T_actual` for the run.
2. **Confirm `isPlaying() === true`** via `frame.waitForFunction` before awaiting the watcher. Without this, the test can hang if `play()` hasn't kicked the timeline yet.
3. **Wait for paint** — two `requestAnimationFrame` ticks on the host page. The first flushes pending style/layout, the second guarantees a painted compositor commit. Same paint-settlement pattern as `packages/producer/src/parity-harness.ts`.
4. **Screenshot the live frame** — `page.screenshot({ type: "png" })`.
5. **Synchronously seek to `T_actual`** — call `el.seek(capturedTime)` on the host page. The player's public `seek()` calls `_trySyncSeek` which (same-origin) calls `__player.seek()` synchronously, so no postMessage await is needed. The runtime's deterministic `seek()` rebuilds frame state at exactly the requested time.
6. **Wait for paint** again, screenshot the reference frame.
7. **Diff with ffmpeg** — `ffmpeg -hide_banner -i reference.png -i actual.png -lavfi ssim -f null -`. ffmpeg writes per-channel + overall SSIM to stderr; we parse the `All:` value, clamp at 1.0 (ffmpeg occasionally reports 1.000001 on identical inputs), and treat it as the run's score.
8. **Persist artifacts** under `tests/perf/results/parity/run-N/` (`actual.png`, `reference.png`, `captured-time.txt`) so CI can upload them and so a failed run is locally reproducible. Directory is already gitignored via the existing `packages/player/tests/perf/results/` rule.

### Aggregation

`min()` across runs, **not** mean. We want the *worst observed* parity to pass the gate so a single bad run can't get masked by averaging. Both per-run scores and the aggregate are logged.

### Output metric

| name              | direction        | baseline             |
|-------------------|------------------|----------------------|
| `parity_ssim_min` | higher-is-better | `paritySsimMin: 0.95` |

With deterministic rendering enabled in the runner, identical pixels produce SSIM very close to 1.0; the 0.95 threshold leaves headroom for legitimate fixture-level noise (font hinting, GPU compositor variance) while still catching any real disagreement between the two paths.

## Test plan

- `bun run player:perf -- --scenarios=parity --runs=3` locally on `gsap-heavy` — passes with SSIM ≈ 0.999 across all 3 runs.
- Inspected `results/parity/run-1/actual.png` and `reference.png` side-by-side — visually identical.
- Inspected `captured-time.txt` to confirm `T_actual` lands just past 5.0s (within one frame).
- Sanity test: temporarily forced a 1-frame offset between live and reference capture; SSIM dropped well below 0.95 as expected, confirming the threshold catches real drift.
- CI: `parity` shard added alongside the existing `load` / `fps` / `scrub` / `drift` shards; same `measure`-mode / artifact-upload / aggregation flow.
- `bunx oxlint` and `bunx oxfmt --check` clean on the new scenario.

## Stack

This is the top of the perf stack:

1. #393 `perf/x-1-emit-performance-metric` — performance.measure() emission
2. #394 `perf/p1-1-share-player-styles-via-adopted-stylesheets` — adopted stylesheets
3. #395 `perf/p1-2-scope-media-mutation-observer` — scoped MutationObserver
4. #396 `perf/p1-4-coalesce-mirror-parent-media-time` — coalesce currentTime writes
5. #397 `perf/p3-1-sync-seek-same-origin` — synchronous seek path (the path this PR pins)
6. #398 `perf/p3-2-srcdoc-composition-switching` — srcdoc switching
7. #399 `perf/p0-1a-perf-test-infra` — server, runner, perf-gate, CI
8. #400 `perf/p0-1b-perf-tests-for-fps-scrub-drift` — fps / scrub / drift scenarios
9. **#401 `perf/p0-1c-live-playback-parity-test` ← you are here**

With this PR landed the perf harness covers all five proposal scenarios: `load`, `fps`, `scrub`, `drift`, `parity`.
2026-04-22 18:15:46 -07:00

145 lines
4.7 KiB
YAML

name: Player perf
on:
pull_request:
push:
branches: [main]
concurrency:
group: player-perf-${{ github.ref }}
cancel-in-progress: true
jobs:
changes:
name: Detect changes
runs-on: ubuntu-latest
timeout-minutes: 2
outputs:
perf: ${{ steps.filter.outputs.perf }}
steps:
- uses: actions/checkout@v4
- uses: dorny/paths-filter@v3
id: filter
with:
filters: |
perf:
- "packages/player/**"
- "packages/core/**"
- "package.json"
- "bun.lock"
- ".github/workflows/player-perf.yml"
perf-shards:
name: "Perf: ${{ matrix.shard }}"
needs: changes
if: needs.changes.outputs.perf == 'true'
runs-on: ubuntu-latest
timeout-minutes: 20
strategy:
fail-fast: false
matrix:
include:
- shard: load
scenarios: load
runs: "5"
- shard: fps
scenarios: fps
runs: "3"
- shard: scrub
scenarios: scrub
runs: "3"
- shard: drift
scenarios: drift
runs: "3"
- shard: parity
scenarios: parity
runs: "3"
steps:
- uses: actions/checkout@v4
- uses: oven-sh/setup-bun@v2
- uses: actions/setup-node@v4
with:
node-version: 22
- run: bun install --frozen-lockfile
# Player perf loads packages/player/dist/hyperframes-player.global.js
# and packages/core/dist/hyperframe.runtime.iife.js, so a full build is required.
- run: bun run build
- name: Set up Chrome (headless shell)
id: setup-chrome
uses: browser-actions/setup-chrome@v1
with:
chrome-version: stable
# The parity scenario shells out to `ffmpeg -lavfi ssim` to score the
# live-playback frame against the sync-seek reference frame. ffmpeg is
# not on the default ubuntu-latest runner image, and a missing binary
# surfaces as ENOENT inside computeSsim() — informative, but cheaper
# to just install it here so the shard never trips on infra.
- name: Install ffmpeg (parity shard only)
if: matrix.shard == 'parity'
run: |
sudo apt-get update
sudo apt-get install -y --no-install-recommends ffmpeg
ffmpeg -version | head -n 1
- name: Run player perf — ${{ matrix.shard }} (measure mode)
working-directory: packages/player
env:
PUPPETEER_EXECUTABLE_PATH: ${{ steps.setup-chrome.outputs.chrome-path }}
run: |
bun run perf \
--mode=measure \
--scenarios=${{ matrix.scenarios }} \
--runs=${{ matrix.runs }}
- name: Upload perf results
if: always()
uses: actions/upload-artifact@v4
with:
name: player-perf-${{ matrix.shard }}
path: packages/player/tests/perf/results/
if-no-files-found: warn
retention-days: 30
# Summary job — matches the required check name in branch protection.
# Logs an explicit "skipped" / "passed" / "failed" line both to stdout and to
# $GITHUB_STEP_SUMMARY so a false skip is obvious in the Checks UI without
# having to dig into the changes-job logs.
player-perf:
runs-on: ubuntu-latest
needs: [changes, perf-shards]
if: always()
steps:
- name: Check results
env:
PERF_FILTER_RESULT: ${{ needs.changes.outputs.perf }}
PERF_SHARDS_RESULT: ${{ needs.perf-shards.result }}
run: |
{
echo "## Player perf gate"
echo ""
echo "- paths-filter \`perf\` matched: \`${PERF_FILTER_RESULT}\`"
echo "- perf-shards result: \`${PERF_SHARDS_RESULT}\`"
echo ""
} >> "$GITHUB_STEP_SUMMARY"
if [ "${PERF_FILTER_RESULT}" != "true" ]; then
echo "::notice title=Player perf::SKIPPED — no changes under packages/player/**, packages/core/**, package.json, bun.lock, or .github/workflows/player-perf.yml. Auto-pass."
echo "**Status:** SKIPPED (no player/core changes — auto-pass)" >> "$GITHUB_STEP_SUMMARY"
exit 0
fi
if [ "${PERF_SHARDS_RESULT}" != "success" ]; then
echo "::error title=Player perf::FAILED — perf-shards result was '${PERF_SHARDS_RESULT}'. See the per-shard logs above."
echo "**Status:** FAILED (perf-shards result: \`${PERF_SHARDS_RESULT}\`)" >> "$GITHUB_STEP_SUMMARY"
exit 1
fi
echo "::notice title=Player perf::PASSED — all perf shards completed successfully."
echo "**Status:** PASSED" >> "$GITHUB_STEP_SUMMARY"