mirror of
https://github.com/calesthio/OpenMontage.git
synced 2026-08-24 17:10:27 +08:00
Merge commit 'f72efb53d3cea7cce0943d5dc4e55e33e4e59af3' into codex/repair-pr-353
# Conflicts: # AGENT_GUIDE.md # docs/PROVIDERS.md
This commit is contained in:
calesthio
@@ -45,7 +45,7 @@ OpenMontage/
|
||||
│ ├── tool_registry.py # Auto-discovery singleton registry
|
||||
│ ├── cost_tracker.py # Budget governance (estimate → reserve → reconcile)
|
||||
│ ├── analysis/ # Transcription, scene detection, frame sampling, video understanding
|
||||
│ ├── audio/ # TTS (ElevenLabs, OpenAI, Piper), music gen, mixing, enhancement
|
||||
│ ├── audio/ # TTS (ElevenLabs, OpenAI, Piper, Azure, Google), music gen, mixing, enhancement
|
||||
│ ├── avatar/ # Talking head animation, lip sync
|
||||
│ ├── enhancement/ # Upscale, bg removal, face enhance/restore, color grading
|
||||
│ ├── graphics/ # Image gen (FLUX, GPT Image, Recraft, local diffusion), stock, diagrams, code snippets, math animation
|
||||
@@ -150,9 +150,9 @@ Selectors route based on: user preference when explicitly set, then scored ranki
|
||||
|
||||
### Tool Inventory by Category
|
||||
|
||||
**Analysis (4):** transcriber (WhisperX), scene_detect, frame_sampler, video_understand (CLIP/BLIP-2)
|
||||
**Analysis (5):** transcriber (WhisperX), azure_stt, scene_detect, frame_sampler, video_understand (CLIP/BLIP-2)
|
||||
|
||||
**Audio (8):** elevenlabs_tts, google_tts, openai_tts, piper_tts, tts_selector, music_gen, audio_mixer, audio_enhance
|
||||
**Audio (9):** elevenlabs_tts, google_tts, openai_tts, piper_tts, azure_tts, tts_selector, music_gen, audio_mixer, audio_enhance
|
||||
|
||||
**Avatar (2):** talking_head (SadTalker/MuseTalk), lip_sync (Wav2Lip)
|
||||
|
||||
@@ -383,6 +383,7 @@ All config is validated via Pydantic models in `lib/config_model.py`.
|
||||
| Variable | Used By | Purpose |
|
||||
|----------|---------|---------|
|
||||
| `ELEVENLABS_API_KEY` | elevenlabs_tts, music_gen | TTS, music, sound effects |
|
||||
| `AZURE_SPEECH_KEY` + `AZURE_SPEECH_REGION` | azure_stt, azure_tts | Azure AI Speech cloud transcription + neural TTS (one resource, both directions) |
|
||||
| `OPENAI_API_KEY` | openai_tts, openai_image | TTS fallback, GPT Image 2 |
|
||||
| `XAI_API_KEY` | grok_image, grok_video | Grok image editing/generation, Grok video generation |
|
||||
| `FAL_KEY` | flux_image, kling_video, veo_video, minimax_video, recraft_image | fal.ai hosted models (FLUX, Veo, Kling, MiniMax, Recraft) |
|
||||
|
||||
@@ -47,8 +47,8 @@ DOUBAO_SPEECH_API_KEY= # Volcengine Doubao Speech TTS (strong Mandarin nar
|
||||
DOUBAO_SPEECH_VOICE_TYPE= # Default Doubao speaker/voice type
|
||||
DASHSCOPE_API_KEY= # Alibaba DashScope (Qwen image gen, TTS, ASR with word timestamps)
|
||||
|
||||
# SPEECH-TO-TEXT (optional cloud transcription; local whisper is the default)
|
||||
AZURE_SPEECH_KEY= # Azure AI Speech — Fast Transcription (word-level timestamps)
|
||||
# AZURE AI SPEECH (optional cloud STT + TTS; one key unlocks both directions)
|
||||
AZURE_SPEECH_KEY= # Azure AI Speech — azure_stt (Fast Transcription) + azure_tts (neural narration)
|
||||
AZURE_SPEECH_REGION= # Speech resource region, e.g. eastus
|
||||
|
||||
# MULTI-MODEL GATEWAY (one key, 6+ tools)
|
||||
@@ -625,6 +625,64 @@ allowance). OpenMontage estimates cost from the transcribed audio duration. See
|
||||
|
||||
---
|
||||
|
||||
### Azure AI Speech — Text-to-Speech
|
||||
|
||||
> **Cloud neural narration.** Azure neural TTS delivers high-quality multilingual voices with SSML prosody control and express-as styles — same Speech resource as `azure_stt`, so one key/region unlocks both directions. Optional: the local `piper_tts` remains the default offline TTS path. When `AZURE_SPEECH_KEY` is set, the agent may prefer `azure_tts` for cloud narration.
|
||||
|
||||
**Tools unlocked:** `azure_tts`
|
||||
**Env vars:** `AZURE_SPEECH_KEY`, `AZURE_SPEECH_REGION` (or `AZURE_TTS_ENDPOINT`)
|
||||
|
||||
#### Setup
|
||||
|
||||
Identical to the STT setup above — the same Speech resource key and region work
|
||||
for both. If you already configured `azure_stt`, `azure_tts` is available now.
|
||||
|
||||
```bash
|
||||
AZURE_SPEECH_KEY=your-speech-resource-key
|
||||
AZURE_SPEECH_REGION=eastus
|
||||
# AZURE_TTS_ENDPOINT=https://<region>.tts.speech.microsoft.com # optional, overrides region
|
||||
```
|
||||
|
||||
Note: the TTS host (`<region>.tts.speech.microsoft.com`) differs from the STT
|
||||
endpoint, so the optional override var is `AZURE_TTS_ENDPOINT`, not
|
||||
`AZURE_SPEECH_ENDPOINT`.
|
||||
|
||||
#### API Notes
|
||||
|
||||
OpenMontage uses the synchronous REST v1 endpoint with an SSML body — no token
|
||||
exchange, Blob storage, or job polling:
|
||||
|
||||
```text
|
||||
POST https://{region}.tts.speech.microsoft.com/cognitiveservices/v1
|
||||
Ocp-Apim-Subscription-Key: ${AZURE_SPEECH_KEY}
|
||||
Content-Type: application/ssml+xml
|
||||
X-Microsoft-OutputFormat: audio-48khz-192kbitrate-mono-mp3
|
||||
```
|
||||
|
||||
Voice shortlist aliases: `andrew` (default — warm, confident), `brandon`
|
||||
(deeper), `ava` (bright female), `guy` (authoritative), `jenny` (friendly). Any
|
||||
Azure voice short name is accepted verbatim. See the `azure-text-to-speech`
|
||||
skill for SSML `rate`/`pitch`/`style` guidance.
|
||||
|
||||
#### What It Is Best For
|
||||
|
||||
- High-quality neural narration on existing Azure credentials
|
||||
- Calm, confident explainer / founder-register delivery
|
||||
- Multilingual narration via the *Multilingual* voice family
|
||||
- Deterministic re-renders (fixed voice + SSML → identical audio)
|
||||
|
||||
Not for: fully offline production (use `piper_tts`) or voice cloning (use
|
||||
`elevenlabs_tts`).
|
||||
|
||||
#### Pricing
|
||||
|
||||
Azure neural TTS Standard (S0) bills roughly **$16 per 1M characters** (a free
|
||||
F0 tier includes a limited monthly allowance). A 150-word narration segment
|
||||
costs about $0.015. OpenMontage estimates cost from character count. See
|
||||
[Azure AI Speech pricing](https://azure.microsoft.com/pricing/details/cognitive-services/speech-services/) for current rates.
|
||||
|
||||
---
|
||||
|
||||
### Google — TTS + Imagen + Music + Video (Shared Key)
|
||||
|
||||
> **One key, five tools.** Google Cloud TTS has 700+ voices in 50+ languages — the strongest localization option. `google_imagen` supports both Imagen 4 and Gemini 2.5 Flash Image, including projects without Imagen catalog access. Google Lyria generates high-quality background music. Gemini Omni Flash supports conversational video editing, and direct Veo generation covers premium short video clips.
|
||||
@@ -1166,6 +1224,7 @@ These tools require only FFmpeg or Python packages — no GPU, no API key.
|
||||
| **Pexels** | `PEXELS_API_KEY` | `pexels_image`, `pexels_video` | Free |
|
||||
| **Pixabay** | `PIXABAY_API_KEY` | `pixabay_image`, `pixabay_video` | Free |
|
||||
| **Piper** | — (install only) | `piper_tts` | Free |
|
||||
| **Azure AI Speech** | `AZURE_SPEECH_KEY` + `AZURE_SPEECH_REGION` | `azure_stt`, `azure_tts` | Free tier + paid |
|
||||
| **Google** | `GOOGLE_API_KEY` (or `GEMINI_API_KEY`) | `google_tts`, `google_imagen`, `google_music`, `gemini_omni_video`, `veo_video` | Free tier (TTS) + paid |
|
||||
| **ElevenLabs** | `ELEVENLABS_API_KEY` | `elevenlabs_tts`, `music_gen` | Free tier + paid |
|
||||
| **fish.audio** | `FISH_AUDIO_API_KEY` | `fish_audio_tts` | Free tier (s2.1-pro-free) + paid |
|
||||
@@ -1193,7 +1252,7 @@ How many providers cover each capability:
|
||||
|-----------|----------------|-----------------|--------------|
|
||||
| **Image Generation** | FLUX, Kling Official, Grok, Google Imagen, GPT Image 2, Recraft | Local Diffusion | Pexels, Pixabay (stock) |
|
||||
| **Video Generation** | Grok, Kling Official, Kling via fal.ai, Seedance via Volcengine Ark, Runway, Veo, Gemini Omni, Higgsfield, MiniMax, HeyGen, Tencent Hunyuan | WAN, Hunyuan, CogVideo, LTX | Pexels, Pixabay (stock) |
|
||||
| **Text-to-Speech** | ElevenLabs, fish.audio, Google TTS, Kling Official, OpenAI | Piper | Piper, Google free tier, ElevenLabs free tier, fish.audio s2.1-pro-free |
|
||||
| **Text-to-Speech** | Azure AI Speech, ElevenLabs, fish.audio, Google TTS, Kling Official, OpenAI | Piper | Piper, Google free tier, ElevenLabs free tier, Azure free tier, fish.audio s2.1-pro-free |
|
||||
| **Music Generation** | ElevenLabs, Suno, Google Lyria | — | ElevenLabs free tier |
|
||||
| **Post-Production** | — | FFmpeg (compose, stitch, trim, mix, enhance, grade) | All free |
|
||||
| **Analysis** | — | WhisperX, Scene Detect, Frame Sampler, CLIP/BLIP-2 | All free |
|
||||
|
||||
Reference in New Issue
Block a user