3 Commits

Author	SHA1	Message	Date
James Russo	b23b0751da	fix(player): parent-frame media playback for mobile (#266) ... * fix(player): parent-frame media playback for mobile Mobile browsers block media.play() inside iframes when the user gesture happened in the parent frame — postMessage doesn't transfer user activation (per the User Activation v2 spec). ## Problem The player renders compositions in a sandboxed iframe. When a user taps play in the parent frame, the player sends a postMessage to the iframe's runtime, which calls audio.play(). On mobile, this fails silently because the iframe has no user activation context. ## Solution The player now extracts ALL timed media elements (audio/video with data-start) from the iframe's DOM (same-origin access), creates parent-frame copies, and disables the iframe originals. On play(), parentMedia.play() runs synchronously in the gesture call stack, satisfying mobile autoplay policy. ### Generic media handling - Finds all `audio[data-start], video[data-start]` in the iframe - Creates a parent-frame copy for each (Audio or Video element) - Preserves data-start offsets for correct seek positioning - Strips data-start from iframe elements so the runtime ignores them - Falls back to iframe media for cross-origin iframes ### `audio-src` attribute Convenience for the common single-narration case. When set, the player starts preloading audio immediately — before the iframe loads. This eliminates the loading delay that caused jittery playback. ### No active sync Both parent media and the GSAP timeline are real-time systems. When started simultaneously, they naturally stay within ~10ms — no drift correction needed. Active sync with coarse granularity (50ms polling) caused MORE jitter than it prevented via repeated audio seeks. ## CI - Added unified `test` job replacing separate per-package test jobs - Added root `test` script: `bun run --filter '*' test` - New packages with test scripts are automatically included - Added happy-dom for player DOM tests ## Tests - 10 new tests for parent-frame media: preloading, play, pause, seek, muted/rate sync, cleanup, attribute changes - All 21 player tests pass Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix(shader-transitions): pass CI when no test files exist Add --passWithNoTests to vitest run so the unified test job doesn't fail on packages that have a test script but no test files yet. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix(cli): update tests for new id field and GSAP lint rule - normalize.test.ts: loadTranscript now assigns id fields (w0, w1, etc.) to SRT/VTT results and empty string for words-json passthrough - lintProject.test.ts: add GSAP CDN script to validHtml() fixture to satisfy the missing_gsap_script lint rule added in core Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix(cli): add missing data-start/data-duration to validHtml fixture The validHtml() test fixture was missing data-start and data-duration attributes, triggering the root_composition_missing_data_start and root_composition_missing_data_duration lint warnings. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix(ci): fetch LFS objects for producer test job Producer regression tests compare rendered output against reference MP4 files stored in git LFS. Without lfs: true, checkout fetches pointer files instead of actual videos, causing "moov atom not found" errors. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * ci: remove redundant test-producer job The regression workflow already runs the same 28 producer fixtures in a Docker container with prod-matching Chrome/fonts/ffmpeg, sharded across 8 parallel matrix jobs with 40-min timeouts. The CI test-producer job was a duplicate that ran on bare runners with worse determinism and a 15-min timeout too short for all fixtures. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>	2026-04-14 11:20:07 -07:00
Vance Ingalls	37404f23da	feat(whisper+captions): language detection, audio-reactive captions, multilingual defaults (#175) ... ## Summary **Whisper improvements:** - Auto-detect language and switch from `.en` to multilingual model when needed - Detect speech onset in WAV to strip hallucinated words before speech begins - Merge whisper-cpp token fragments: contractions (`didn` + `'t` → `didn't`), split capitals (`C` + `aught` → `Caught`), dropped-g (`shin` + `in'` → `shinin'`) - Interpolate zero-duration word clusters for reliable karaoke timing **Captions skill updates (folded from #176):** - Rewrite script-to-style mapping as an energy detection table (high → low) with mandatory animation requirements: karaoke baseline, 2+ highlight techniques, kinetic exits - Replace `tl.call()` per-frame audio-reactive pattern with group-level GSAP tweens — read peak bass/treble for each group's time range and modulate entrance intensity at build time - Add transcript quality check with automatic retry rules (>20% music tokens = retry with larger model) - Add caption word structure lint rule (`.caption-group` + `<span>`) for studio editor compatibility **Multilingual defaults (folded from #186):** - Default whisper model changed from `small.en` to `small` to prevent silent translation of non-English audio - Added non-negotiable language rule to captions skill ## Test plan - [ ] `pnpm test` passes (contraction merging, fragment merging, zero-duration interpolation, speech onset) - [ ] Transcribe non-English audio — verify it transcribes in original language, not translates - [ ] Skill files render correctly, cross-references resolve - [ ] `dynamic-techniques.md` audio-reactive section uses `tl.to()`/`tl.set()` only, no `tl.call()` loops 🤖 Generated with [Claude Code](https://claude.com/claude-code)	2026-04-02 00:08:41 -07:00
James Russo	2f99e33bbe	feat(cli,core): standalone transcribe command, transcript normalization, caption lint rules (#151) ... * feat(cli,core): add standalone transcribe command, transcript normalization, and caption lint rules - Add `hyperframes transcribe` command for transcribing audio/video and importing existing transcripts (SRT, VTT, OpenAI Whisper API JSON, whisper.cpp JSON) - Add transcript format normalizer (normalize.ts) with auto-detection and conversion to standard [{text, start, end}] word arrays - Upgrade default whisper model from base.en to small.en for better accuracy - Add --model and --language flags to both `transcribe` and `init` commands - Extract shared patchCaptionHtml() to eliminate duplication between init.ts and transcribe.ts (init.ts reduced by ~55 lines) - Add 3 caption lint rules: caption_exit_missing_hard_kill, caption_text_overflow_risk, caption_container_relative_position - Update captions skill with model guide, format docs, music guidance, text overflow prevention, caption exit guarantee pattern - Expand captions skill trigger to cover lyrics, karaoke, lyric videos Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs(cli): add transcribe command and --model/--language flags to CLI docs Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix(cli): fix blank template lint issues - blank/index.html: remove data-start from video (was nested in timed parent), add class="clip" for initial hidden state - blank/captions.html: add max-width + overflow:hidden to prevent text clipping, add tl.set hard kill after exit tween to prevent stuck captions Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs: add lint-after-edit rule to repo and project CLAUDE.md Agents must run `npx hyperframes lint` after editing compositions. Also expand captions skill description in project template. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * style: format _shared/CLAUDE.md Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>	2026-03-30 18:58:06 -07:00