mirror of
https://github.com/digitalsamba/claude-code-video-toolkit.git
synced 2026-09-18 19:41:13 +08:00
Conal Mullan
dfe2091695
Replaces the old "fresh ref upload + per-scene clone" pattern with the
proper Qwen3-TTS workflow: VoiceDesign generates a character voice from
a natural-language brief once, the resulting ref is cached, and all
subsequent scenes share a single voice_clone_prompt extracted from it —
Qwen's recommended pattern for consistent multi-scene narration.
Handler (Modal + RunPod, kept in lockstep):
- New "voice_design" mode (Qwen/Qwen3-TTS-12Hz-1.7B-VoiceDesign baked
into the image alongside CustomVoice + Base; ~19GB total)
- New "outputs": [...] response shape; legacy top-level keys still
populated for single-text calls (back-compat)
- Three modes: custom_voice / clone / voice_design
Clone path: extracts voice_clone_prompt ONCE per request and reuses it
across all utterances. Note: qwen-tts's batch path
(generate_voice_clone(text=[list])) hits a PyTorch tensor-aliasing bug
("written-to tensor refers to a single memory location") — we serialize
the calls per-text instead, while still sharing the pre-computed
prompt, which is what gives us the consistency win.
Tools:
- tools/qwen3_tts.py: new --design-instruct flag; generate_audio()
accepts list[str] for batch; design_instruct kwarg routes to
voice_design mode.
- tools/voiceover.py: --scene-dir auto-batches qwen3 clone mode (one
call, shared prompt); brand loader handles new clone.design block
with self-caching of the designed ref.
- tools/align_captions.py (new): transcribes each scene MP3 via
ElevenLabs Scribe, fuzzy-matches each cap() phrase to the word
stream, and patches sprint-config.ts in --apply mode.
- scratch/update_qwen3_template.py (new): one-shot helper to point the
RunPod template at a new image tag via saveTemplate GraphQL.
Deployment status:
- Modal: deployed and live
- RunPod: image built locally as ghcr.io/conalmullan/video-toolkit-
qwen3-tts:0.2.0-voicedesign (20.4GB); GHCR push was auth-denied
(PAT refresh needed). Push + scratch/update_qwen3_template.py to
promote when ready.
NOT YET REVIEWED OR TESTED ON RUNPOD. Modal smoke-test passed
(designed Lugh in 60s, batch-cloned 9 scenes successfully). Verify
RunPod parity once the image is pushed and the template updated.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>