feat(tts): add Azure AI Speech as an optional cloud text-to-speech provider

Neural TTS via the synchronous REST v1 endpoint (SSML body, no token
exchange or job polling). Shares one Speech resource with azure_stt —
AZURE_SPEECH_KEY + AZURE_SPEECH_REGION unlock both directions; optional
AZURE_TTS_ENDPOINT overrides the TTS host (a different subdomain than
the STT endpoint). piper_tts remains the default offline path.

- tools/audio/azure_tts.py: azure_tts tool (capability=tts), voice
  shortlist aliases, SSML prosody/style, mp3/wav output, cost tracking
- tests/tools/test_azure_tts.py: contract, discovery, status, SSML,
  and mocked execute tests (21 tests, no live network)
- .agents/.claude skills: azure-text-to-speech Layer-3 skill
- docs: PROVIDERS.md section + tables, ARCHITECTURE.md inventories,
  AGENT_GUIDE.md + skills/INDEX.md rows, asset-director TTS cheatsheet,
  .env.example
This commit is contained in:
amartya-dev
2026-07-13 20:04:28 +05:30
parent f8d94632ea
commit 888d7b1e72
10 changed files with 831 additions and 12 deletions

View File

@@ -0,0 +1,110 @@
---
name: azure-text-to-speech
description: Generate neural narration audio using Azure AI Speech (REST text-to-speech). Use when synthesizing voiceovers or narration in OpenMontage. Optional cloud TTS provider — preferred when AZURE_SPEECH_KEY is configured; the local piper_tts remains the default offline path. Shares one Speech resource with azure_stt.
license: MIT
compatibility: Requires internet access and an Azure AI Speech resource (AZURE_SPEECH_KEY + AZURE_SPEECH_REGION).
metadata: {"openclaw": {"requires": {"env": ["AZURE_SPEECH_KEY", "AZURE_SPEECH_REGION"]}, "primaryEnv": "AZURE_SPEECH_KEY"}}
---
# Azure AI Speech — Text-to-Speech
Generate narration with **Azure neural TTS** — high-quality multilingual voices,
SSML prosody control, and express-as styles, served synchronously by the REST
`/cognitiveservices/v1` endpoint (no token exchange, Blob storage, or job
polling). In OpenMontage this is exposed through the `azure_tts` tool
(`capability=tts`, `provider=azure`). It is an **optional cloud TTS provider**
when `AZURE_SPEECH_KEY` is configured, prefer it for high-quality cloud
narration. The local `piper_tts` remains the **default offline path** and the
fallback when Azure is unavailable; `elevenlabs_tts` remains the choice for
voice cloning.
> Docs: [REST text to speech](https://learn.microsoft.com/azure/ai-services/speech-service/rest-text-to-speech) · [Voice gallery](https://speech.microsoft.com/portal/voicegallery)
## Setup
Same Speech resource as `azure_stt`**one key/region unlocks both directions**
(STT and TTS). Create a **Speech** resource in the
[Azure portal](https://portal.azure.com); copy the key and region from its
**Keys and Endpoint** page.
```bash
export AZURE_SPEECH_KEY=your_speech_resource_key
export AZURE_SPEECH_REGION=eastus # your resource's region
# export AZURE_TTS_ENDPOINT=https://... # optional: full custom TTS host
# (the TTS host is https://<region>.tts.speech.microsoft.com — a different
# subdomain than the STT endpoint, hence the separate override var)
```
`azure_tts` reports `AVAILABLE` once `AZURE_SPEECH_KEY` plus either
`AZURE_SPEECH_REGION` or `AZURE_TTS_ENDPOINT` are set.
## Using it in a pipeline
Route through `tts_selector` as usual (it auto-discovers `azure_tts`), or call
the provider tool directly when the user has approved Azure:
```python
from tools.tool_registry import registry
registry.discover()
tts = registry._tools["azure_tts"]
result = tts.execute({
"text": "Every design decision in this dashboard has a reason.",
"voice": "andrew", # alias or full Azure short name
"rate": "-4%", # slightly slower for narration
# "style": "narration-professional", # for voices that support styles
"output_path": "projects/my-video/assets/audio/seg_001.mp3",
"output_format": "mp3", # or "wav" (48kHz PCM) for mixing
})
```
If `azure_tts` is unavailable (no key) or errors, fall back per its declared
chain: `elevenlabs_tts``openai_tts``piper_tts`.
## Voice selection
Curated shortlist (aliases accepted by the `voice` param):
| Alias | Voice | Character |
|-------|-------|-----------|
| `andrew` | en-US-AndrewMultilingualNeural | warm, confident, conversational — the default; founder/explainer register |
| `brandon` | en-US-BrandonMultilingualNeural | deeper, measured |
| `ava` | en-US-AvaMultilingualNeural | confident, bright female |
| `guy` | en-US-GuyNeural | authoritative |
| `jenny` | en-US-JennyNeural | friendly, clear |
Any valid Azure voice short name may be passed verbatim (e.g.
`de-DE-KatjaNeural`); the *Multilingual* voices handle non-English text well —
set `locale` to match the text's language for correct SSML.
## Parameters that matter
- **`rate` / `pitch`** — SSML prosody. Narration usually reads best slightly
slowed (`"-4%"` to `"-8%"`); leave pitch at `"0%"` unless correcting a voice.
- **`style`** — express-as style for voices that support it
(`narration-professional`, `calm`, `newscast`). Unsupported styles are
silently ignored by Azure, so listen to a sample before batch runs.
- **`output_format`** — `mp3` (48kHz/192kbit) for delivery, `wav` (48kHz PCM)
when the segment feeds `audio_mixer` for further processing.
- Determinism: a fixed voice + SSML re-renders effectively identical audio —
safe to regenerate individual segments without re-recording the whole set.
## Cost
Azure neural TTS Standard tier bills roughly **$16 per 1M characters** (~$0.016
per 1k chars; a 150-word narration segment ≈ $0.015). The tool reports
per-call `cost_usd` for the cost tracker. See
[Azure AI Speech pricing](https://azure.microsoft.com/pricing/details/cognitive-services/speech-services/) for current rates.
## Limits & tips
- One `execute` call = one narration segment. Generate per script section (the
asset stage convention) rather than one giant paragraph — smaller segments
align cleanly to scene timings and are cheap to regenerate.
- The synchronous endpoint caps a request at 10 minutes of audio — far above
any segment OpenMontage generates.
- Text is XML-escaped automatically; do not pre-escape or wrap in SSML — pass
plain text plus the `rate`/`pitch`/`style` params.
- Verify quality: listen to the first generated segment before batch-running a
full script (voice/style fit is a creative decision — surface it at the
proposal stage per the Decision Communication Contract).

View File

@@ -0,0 +1,110 @@
---
name: azure-text-to-speech
description: Generate neural narration audio using Azure AI Speech (REST text-to-speech). Use when synthesizing voiceovers or narration in OpenMontage. Optional cloud TTS provider — preferred when AZURE_SPEECH_KEY is configured; the local piper_tts remains the default offline path. Shares one Speech resource with azure_stt.
license: MIT
compatibility: Requires internet access and an Azure AI Speech resource (AZURE_SPEECH_KEY + AZURE_SPEECH_REGION).
metadata: {"openclaw": {"requires": {"env": ["AZURE_SPEECH_KEY", "AZURE_SPEECH_REGION"]}, "primaryEnv": "AZURE_SPEECH_KEY"}}
---
# Azure AI Speech — Text-to-Speech
Generate narration with **Azure neural TTS** — high-quality multilingual voices,
SSML prosody control, and express-as styles, served synchronously by the REST
`/cognitiveservices/v1` endpoint (no token exchange, Blob storage, or job
polling). In OpenMontage this is exposed through the `azure_tts` tool
(`capability=tts`, `provider=azure`). It is an **optional cloud TTS provider**
when `AZURE_SPEECH_KEY` is configured, prefer it for high-quality cloud
narration. The local `piper_tts` remains the **default offline path** and the
fallback when Azure is unavailable; `elevenlabs_tts` remains the choice for
voice cloning.
> Docs: [REST text to speech](https://learn.microsoft.com/azure/ai-services/speech-service/rest-text-to-speech) · [Voice gallery](https://speech.microsoft.com/portal/voicegallery)
## Setup
Same Speech resource as `azure_stt`**one key/region unlocks both directions**
(STT and TTS). Create a **Speech** resource in the
[Azure portal](https://portal.azure.com); copy the key and region from its
**Keys and Endpoint** page.
```bash
export AZURE_SPEECH_KEY=your_speech_resource_key
export AZURE_SPEECH_REGION=eastus # your resource's region
# export AZURE_TTS_ENDPOINT=https://... # optional: full custom TTS host
# (the TTS host is https://<region>.tts.speech.microsoft.com — a different
# subdomain than the STT endpoint, hence the separate override var)
```
`azure_tts` reports `AVAILABLE` once `AZURE_SPEECH_KEY` plus either
`AZURE_SPEECH_REGION` or `AZURE_TTS_ENDPOINT` are set.
## Using it in a pipeline
Route through `tts_selector` as usual (it auto-discovers `azure_tts`), or call
the provider tool directly when the user has approved Azure:
```python
from tools.tool_registry import registry
registry.discover()
tts = registry._tools["azure_tts"]
result = tts.execute({
"text": "Every design decision in this dashboard has a reason.",
"voice": "andrew", # alias or full Azure short name
"rate": "-4%", # slightly slower for narration
# "style": "narration-professional", # for voices that support styles
"output_path": "projects/my-video/assets/audio/seg_001.mp3",
"output_format": "mp3", # or "wav" (48kHz PCM) for mixing
})
```
If `azure_tts` is unavailable (no key) or errors, fall back per its declared
chain: `elevenlabs_tts``openai_tts``piper_tts`.
## Voice selection
Curated shortlist (aliases accepted by the `voice` param):
| Alias | Voice | Character |
|-------|-------|-----------|
| `andrew` | en-US-AndrewMultilingualNeural | warm, confident, conversational — the default; founder/explainer register |
| `brandon` | en-US-BrandonMultilingualNeural | deeper, measured |
| `ava` | en-US-AvaMultilingualNeural | confident, bright female |
| `guy` | en-US-GuyNeural | authoritative |
| `jenny` | en-US-JennyNeural | friendly, clear |
Any valid Azure voice short name may be passed verbatim (e.g.
`de-DE-KatjaNeural`); the *Multilingual* voices handle non-English text well —
set `locale` to match the text's language for correct SSML.
## Parameters that matter
- **`rate` / `pitch`** — SSML prosody. Narration usually reads best slightly
slowed (`"-4%"` to `"-8%"`); leave pitch at `"0%"` unless correcting a voice.
- **`style`** — express-as style for voices that support it
(`narration-professional`, `calm`, `newscast`). Unsupported styles are
silently ignored by Azure, so listen to a sample before batch runs.
- **`output_format`** — `mp3` (48kHz/192kbit) for delivery, `wav` (48kHz PCM)
when the segment feeds `audio_mixer` for further processing.
- Determinism: a fixed voice + SSML re-renders effectively identical audio —
safe to regenerate individual segments without re-recording the whole set.
## Cost
Azure neural TTS Standard tier bills roughly **$16 per 1M characters** (~$0.016
per 1k chars; a 150-word narration segment ≈ $0.015). The tool reports
per-call `cost_usd` for the cost tracker. See
[Azure AI Speech pricing](https://azure.microsoft.com/pricing/details/cognitive-services/speech-services/) for current rates.
## Limits & tips
- One `execute` call = one narration segment. Generate per script section (the
asset stage convention) rather than one giant paragraph — smaller segments
align cleanly to scene timings and are cheap to regenerate.
- The synchronous endpoint caps a request at 10 minutes of audio — far above
any segment OpenMontage generates.
- Text is XML-escaped automatically; do not pre-escape or wrap in SSML — pass
plain text plus the `rate`/`pitch`/`style` params.
- Verify quality: listen to the first generated segment before batch-running a
full script (voice/style fit is a creative decision — surface it at the
proposal stage per the Decision Communication Contract).

View File

@@ -61,12 +61,13 @@ UNSPLASH_ACCESS_KEY= # Unsplash stock images (free developer key)
# --- Analysis ---
HF_TOKEN= # HuggingFace token — enables speaker diarization in transcriber
# Speech-to-text: optional Azure AI Speech (Fast Transcription). When set, the
# agent prefers azure_stt for cloud STT; the local faster-whisper transcriber
# remains the default offline path.
# Speech: optional Azure AI Speech. One key/region unlocks both directions —
# azure_stt (Fast Transcription cloud STT) and azure_tts (neural cloud TTS).
# The local faster-whisper transcriber / piper_tts remain the default offline paths.
AZURE_SPEECH_KEY= # Azure AI Speech resource key ('Keys and Endpoint' page)
AZURE_SPEECH_REGION= # Speech resource region, e.g. eastus
# AZURE_SPEECH_ENDPOINT= # Optional: full custom endpoint URL (overrides region)
# AZURE_SPEECH_ENDPOINT= # Optional: full custom STT endpoint URL (overrides region)
# AZURE_TTS_ENDPOINT= # Optional: full custom TTS host (e.g. https://<region>.tts.speech.microsoft.com)
# --- Avatar (local installs) ---
# WAV2LIP_PATH= # Path to cloned Wav2Lip repo (for lip sync)

View File

@@ -501,7 +501,7 @@ Three selector tools abstract multi-provider capabilities. **Selectors auto-disc
| Selector | Routes to | How it discovers |
|----------|-----------|-----------------|
| `tts_selector` | All tools with `capability="tts"` (ElevenLabs, Google TTS, OpenAI, Piper) | `registry.get_by_capability("tts")` |
| `tts_selector` | All tools with `capability="tts"` (ElevenLabs, Google TTS, OpenAI, Piper, Azure) | `registry.get_by_capability("tts")` |
| `image_selector` | All tools with `capability="image_generation"` (FLUX, Google Imagen, GPT Image, Recraft, etc.) | `registry.get_by_capability("image_generation")` |
| `video_selector` | All tools with `capability="video_generation"` | `registry.get_by_capability("video_generation")` |
@@ -677,7 +677,7 @@ The `.agents/skills/` directory is large. When you're not coming in through a to
| **Character animation** | `character-rigging`, `svg-character-animation`, `pose-library-design`, `canvas-procedural-animation`, `character-animation-qa` |
| **Image generation** | `bfl-api`, `flux-best-practices` |
| **Video generation** | `seedance-2-0` (preferred premium default — cinematic, trailer, multi-shot, synced audio, lip-sync), `gemini-omni` (conversational video editing, reference tags, timecoded beats), `ai-video-gen`, `ltx2` |
| **Audio** | `elevenlabs`, `music`, `sound-effects`, `acestep`, `text-to-speech`, `setup-api-key` |
| **Audio** | `elevenlabs`, `music`, `sound-effects`, `acestep`, `text-to-speech`, `azure-text-to-speech` (optional cloud TTS — tool `azure_tts`, same Speech key as `azure_stt`), `setup-api-key` |
| **Speech-to-text** | `speech-to-text` (whisper `transcriber` — default, offline), `azure-speech-to-text` (optional cloud STT — tool `azure_stt`, preferred when `AZURE_SPEECH_KEY` is set) |
| **Avatar / lip-sync** | `avatar-video`, `heygen`, `create-video`, `faceswap`, `video-translate`, `agents` |
| **Capture** | `playwright-recording` (browser flows), `ffmpeg` (post) |

View File

@@ -45,7 +45,7 @@ OpenMontage/
│ ├── tool_registry.py # Auto-discovery singleton registry
│ ├── cost_tracker.py # Budget governance (estimate → reserve → reconcile)
│ ├── analysis/ # Transcription, scene detection, frame sampling, video understanding
│ ├── audio/ # TTS (ElevenLabs, OpenAI, Piper), music gen, mixing, enhancement
│ ├── audio/ # TTS (ElevenLabs, OpenAI, Piper, Azure, Google), music gen, mixing, enhancement
│ ├── avatar/ # Talking head animation, lip sync
│ ├── enhancement/ # Upscale, bg removal, face enhance/restore, color grading
│ ├── graphics/ # Image gen (FLUX, GPT Image, Recraft, local diffusion), stock, diagrams, code snippets, math animation
@@ -150,9 +150,9 @@ Selectors route based on: user preference when explicitly set, then scored ranki
### Tool Inventory by Category
**Analysis (4):** transcriber (WhisperX), scene_detect, frame_sampler, video_understand (CLIP/BLIP-2)
**Analysis (5):** transcriber (WhisperX), azure_stt, scene_detect, frame_sampler, video_understand (CLIP/BLIP-2)
**Audio (8):** elevenlabs_tts, google_tts, openai_tts, piper_tts, tts_selector, music_gen, audio_mixer, audio_enhance
**Audio (9):** elevenlabs_tts, google_tts, openai_tts, piper_tts, azure_tts, tts_selector, music_gen, audio_mixer, audio_enhance
**Avatar (2):** talking_head (SadTalker/MuseTalk), lip_sync (Wav2Lip)
@@ -383,6 +383,7 @@ All config is validated via Pydantic models in `lib/config_model.py`.
| Variable | Used By | Purpose |
|----------|---------|---------|
| `ELEVENLABS_API_KEY` | elevenlabs_tts, music_gen | TTS, music, sound effects |
| `AZURE_SPEECH_KEY` + `AZURE_SPEECH_REGION` | azure_stt, azure_tts | Azure AI Speech cloud transcription + neural TTS (one resource, both directions) |
| `OPENAI_API_KEY` | openai_tts, openai_image | TTS fallback, GPT Image 2 |
| `XAI_API_KEY` | grok_image, grok_video | Grok image editing/generation, Grok video generation |
| `FAL_KEY` | flux_image, kling_video, veo_video, minimax_video, recraft_image | fal.ai hosted models (FLUX, Veo, Kling, MiniMax, Recraft) |

View File

@@ -44,8 +44,8 @@ DOUBAO_SPEECH_API_KEY= # Volcengine Doubao Speech TTS (strong Mandarin nar
DOUBAO_SPEECH_VOICE_TYPE= # Default Doubao speaker/voice type
DASHSCOPE_API_KEY= # Alibaba DashScope (Qwen image gen, TTS, ASR with word timestamps)
# SPEECH-TO-TEXT (optional cloud transcription; local whisper is the default)
AZURE_SPEECH_KEY= # Azure AI Speech — Fast Transcription (word-level timestamps)
# AZURE AI SPEECH (optional cloud STT + TTS; one key unlocks both directions)
AZURE_SPEECH_KEY= # Azure AI Speech — azure_stt (Fast Transcription) + azure_tts (neural narration)
AZURE_SPEECH_REGION= # Speech resource region, e.g. eastus
# MULTI-MODEL GATEWAY (one key, 6+ tools)
@@ -344,6 +344,64 @@ allowance). OpenMontage estimates cost from the transcribed audio duration. See
---
### Azure AI Speech — Text-to-Speech
> **Cloud neural narration.** Azure neural TTS delivers high-quality multilingual voices with SSML prosody control and express-as styles — same Speech resource as `azure_stt`, so one key/region unlocks both directions. Optional: the local `piper_tts` remains the default offline TTS path. When `AZURE_SPEECH_KEY` is set, the agent may prefer `azure_tts` for cloud narration.
**Tools unlocked:** `azure_tts`
**Env vars:** `AZURE_SPEECH_KEY`, `AZURE_SPEECH_REGION` (or `AZURE_TTS_ENDPOINT`)
#### Setup
Identical to the STT setup above — the same Speech resource key and region work
for both. If you already configured `azure_stt`, `azure_tts` is available now.
```bash
AZURE_SPEECH_KEY=your-speech-resource-key
AZURE_SPEECH_REGION=eastus
# AZURE_TTS_ENDPOINT=https://<region>.tts.speech.microsoft.com # optional, overrides region
```
Note: the TTS host (`<region>.tts.speech.microsoft.com`) differs from the STT
endpoint, so the optional override var is `AZURE_TTS_ENDPOINT`, not
`AZURE_SPEECH_ENDPOINT`.
#### API Notes
OpenMontage uses the synchronous REST v1 endpoint with an SSML body — no token
exchange, Blob storage, or job polling:
```text
POST https://{region}.tts.speech.microsoft.com/cognitiveservices/v1
Ocp-Apim-Subscription-Key: ${AZURE_SPEECH_KEY}
Content-Type: application/ssml+xml
X-Microsoft-OutputFormat: audio-48khz-192kbitrate-mono-mp3
```
Voice shortlist aliases: `andrew` (default — warm, confident), `brandon`
(deeper), `ava` (bright female), `guy` (authoritative), `jenny` (friendly). Any
Azure voice short name is accepted verbatim. See the `azure-text-to-speech`
skill for SSML `rate`/`pitch`/`style` guidance.
#### What It Is Best For
- High-quality neural narration on existing Azure credentials
- Calm, confident explainer / founder-register delivery
- Multilingual narration via the *Multilingual* voice family
- Deterministic re-renders (fixed voice + SSML → identical audio)
Not for: fully offline production (use `piper_tts`) or voice cloning (use
`elevenlabs_tts`).
#### Pricing
Azure neural TTS Standard (S0) bills roughly **$16 per 1M characters** (a free
F0 tier includes a limited monthly allowance). A 150-word narration segment
costs about $0.015. OpenMontage estimates cost from character count. See
[Azure AI Speech pricing](https://azure.microsoft.com/pricing/details/cognitive-services/speech-services/) for current rates.
---
### Google — TTS + Imagen + Music + Video (Shared Key)
> **One key, five tools.** Google Cloud TTS has 700+ voices in 50+ languages — the strongest localization option. Imagen 4 generates high-quality images. Google Lyria generates high-quality background music. Gemini Omni Flash supports conversational video editing, and direct Veo generation covers premium short video clips.
@@ -879,6 +937,7 @@ These tools require only FFmpeg or Python packages — no GPU, no API key.
| **Pexels** | `PEXELS_API_KEY` | `pexels_image`, `pexels_video` | Free |
| **Pixabay** | `PIXABAY_API_KEY` | `pixabay_image`, `pixabay_video` | Free |
| **Piper** | — (install only) | `piper_tts` | Free |
| **Azure AI Speech** | `AZURE_SPEECH_KEY` + `AZURE_SPEECH_REGION` | `azure_stt`, `azure_tts` | Free tier + paid |
| **Google** | `GOOGLE_API_KEY` (or `GEMINI_API_KEY`) | `google_tts`, `google_imagen`, `google_music`, `gemini_omni_video`, `veo_video` | Free tier (TTS) + paid |
| **ElevenLabs** | `ELEVENLABS_API_KEY` | `elevenlabs_tts`, `music_gen` | Free tier + paid |
| **fal.ai** | `FAL_KEY` | `flux_image`, `recraft_image`, `kling_video`, `veo_video`, `minimax_video` | Pay-as-you-go |
@@ -903,7 +962,7 @@ How many providers cover each capability:
|-----------|----------------|-----------------|--------------|
| **Image Generation** | FLUX, Kling Official, Grok, Google Imagen, GPT Image 2, Recraft | Local Diffusion | Pexels, Pixabay (stock) |
| **Video Generation** | Grok, Kling Official, Kling via fal.ai, Runway, Veo, Gemini Omni, Higgsfield, MiniMax, HeyGen | WAN, Hunyuan, CogVideo, LTX | Pexels, Pixabay (stock) |
| **Text-to-Speech** | ElevenLabs, Google TTS, Kling Official, OpenAI | Piper | Piper, Google free tier, ElevenLabs free tier |
| **Text-to-Speech** | Azure AI Speech, ElevenLabs, Google TTS, Kling Official, OpenAI | Piper | Piper, Google free tier, ElevenLabs free tier, Azure free tier |
| **Music Generation** | ElevenLabs, Suno, Google Lyria | — | ElevenLabs free tier |
| **Post-Production** | — | FFmpeg (compose, stitch, trim, mix, enhance, grade) | All free |
| **Analysis** | — | WhisperX, Scene Detect, Frame Sampler, CLIP/BLIP-2 | All free |

View File

@@ -83,6 +83,7 @@ Key capability families to look for in the output:
| HyperFrames | `core/hyperframes.md` | HTML/CSS/GSAP composition runtime — kinetic typography, music-to-video, product promos, website capture. Vendored at v0.7.17 (2026-06-27). | `hyperframes` (router) → `hyperframes-core` (contract), `hyperframes-creative` (palette/type/narration), `hyperframes-media` (TTS/BGM/SFX/captions), `hyperframes-animation` (all motion), `hyperframes-cli`, `hyperframes-registry`, `media-use`, `motion-graphics`, `music-to-video` (beats-driven), `website-to-video`, `remotion-to-hyperframes` (migration), `gsap-core`, `gsap-timeline` |
| WhisperX | `core/whisperx.md` | Transcription with word-level timestamps — default STT (offline, free) | `speech-to-text` |
| Azure STT | (tool: `azure_stt`) | Optional cloud speech-to-text, word-level timestamps — preferred when `AZURE_SPEECH_KEY` is set | `azure-speech-to-text` |
| Azure TTS | (tool: `azure_tts`) | Optional cloud neural narration (SSML prosody, express-as styles) — same Speech key as `azure_stt` | `azure-text-to-speech` |
| Subtitle Sync | `core/subtitle-sync.md` | Subtitle timing and alignment | `remotion-best-practices` |
| Color Grading | `core/color-grading.md` | FFmpeg color profiles, LUT workflow, accessibility | `ffmpeg` |

View File

@@ -93,6 +93,7 @@ For each script section:
- OpenAI: `instructions` only with `model: "gpt-4o-mini-tts"`; use `response_format` for output format
- Google TTS: `input_type: "ssml"` when using `<break>` tags, plus `speaking_rate` in `0.25..2.0` and `pitch` in `-20..20`
- ElevenLabs: `stability`, `similarity_boost`, `style`, `speed`, and `use_speaker_boost`
- Azure: voice aliases (`andrew`, `brandon`, `ava`, `guy`, `jenny`) or any Azure short name, SSML `rate` (e.g. `"-4%"`), `pitch` (e.g. `"+1st"`), and `style` (e.g. `"narration-professional"`) — see the `azure-text-to-speech` skill
7. Generate using `tts_selector` — it auto-routes to the best available TTS provider based on user preference and availability. Check the registry's `best_for` fields to understand each provider's strengths.
8. Record the applied `voice_performance` metadata on each narration asset
9. Verify the audio file exists and duration matches expected timing (±15%)

View File

@@ -0,0 +1,247 @@
"""Focused tests for the Azure AI Speech neural TTS tool.
No live API calls: the network layer is monkeypatched. Covers the tool
contract, registry discovery, status behavior, voice resolution, SSML
construction, and execute() guardrails.
"""
import sys
from pathlib import Path
import pytest
PROJECT_ROOT = Path(__file__).resolve().parent.parent.parent
sys.path.insert(0, str(PROJECT_ROOT))
from tools.base_tool import BaseTool, ToolStatus, ToolTier, ToolRuntime
from tools.tool_registry import ToolRegistry
from tools.audio.azure_tts import AzureTTS
FAKE_MP3 = b"\xff\xfb\x90\x00" + b"\x00" * 64
class _FakeResponse:
def __init__(self, content=FAKE_MP3, status_code=200, text=""):
self.content = content
self.status_code = status_code
self.text = text
@pytest.fixture
def azure_env(monkeypatch):
monkeypatch.setenv("AZURE_SPEECH_KEY", "fake-key")
monkeypatch.setenv("AZURE_SPEECH_REGION", "eastus")
monkeypatch.delenv("AZURE_TTS_ENDPOINT", raising=False)
# ---- Contract ----
class TestContract:
def test_inherits_base_tool(self):
assert issubclass(AzureTTS, BaseTool)
def test_identity(self):
t = AzureTTS()
assert t.name == "azure_tts"
assert t.capability == "tts"
assert t.provider == "azure"
assert t.runtime == ToolRuntime.API
assert t.tier == ToolTier.VOICE
assert t.fallback == "piper_tts"
assert "azure-text-to-speech" in t.agent_skills
assert len(t.capabilities) > 0
def test_get_info_valid(self):
info = AzureTTS().get_info()
assert info["name"] == "azure_tts"
assert info["capability"] == "tts"
assert "text" in info["input_schema"]["properties"]
def test_estimate_cost_by_characters(self):
t = AzureTTS()
# Standard tier ≈ $16 per 1M characters.
assert t.estimate_cost({"text": "x" * 1_000_000}) == pytest.approx(16.0)
assert t.estimate_cost({}) == 0.0
# ---- Registry discovery ----
class TestDiscovery:
def test_discoverable(self):
reg = ToolRegistry()
reg.discover("tools")
assert reg.get("azure_tts") is not None
def test_capability_routing(self):
reg = ToolRegistry()
reg.discover("tools")
names = [t.name for t in reg.get_by_capability("tts")]
assert "azure_tts" in names
# ---- Status behavior ----
class TestStatus:
def test_unavailable_without_env(self, monkeypatch):
monkeypatch.delenv("AZURE_SPEECH_KEY", raising=False)
monkeypatch.delenv("AZURE_SPEECH_REGION", raising=False)
monkeypatch.delenv("AZURE_TTS_ENDPOINT", raising=False)
assert AzureTTS().get_status() == ToolStatus.UNAVAILABLE
def test_available_with_key_and_region(self, azure_env):
assert AzureTTS().get_status() == ToolStatus.AVAILABLE
def test_available_with_key_and_endpoint(self, monkeypatch):
monkeypatch.setenv("AZURE_SPEECH_KEY", "fake-key")
monkeypatch.delenv("AZURE_SPEECH_REGION", raising=False)
monkeypatch.setenv("AZURE_TTS_ENDPOINT", "https://custom.tts.example.com")
assert AzureTTS().get_status() == ToolStatus.AVAILABLE
def test_key_alone_is_not_enough(self, monkeypatch):
monkeypatch.setenv("AZURE_SPEECH_KEY", "fake-key")
monkeypatch.delenv("AZURE_SPEECH_REGION", raising=False)
monkeypatch.delenv("AZURE_TTS_ENDPOINT", raising=False)
assert AzureTTS().get_status() == ToolStatus.UNAVAILABLE
# ---- Voice resolution + SSML construction (the risky logic) ----
class TestSSML:
def test_voice_alias_resolution(self):
t = AzureTTS()
assert t._resolve_voice({"voice": "andrew"}) == "en-US-AndrewMultilingualNeural"
assert t._resolve_voice({"voice": "JENNY"}) == "en-US-JennyNeural"
# full short names pass through untouched
assert t._resolve_voice({"voice": "de-DE-KatjaNeural"}) == "de-DE-KatjaNeural"
# default when omitted or blank
assert t._resolve_voice({}) == AzureTTS.DEFAULT_VOICE
assert t._resolve_voice({"voice": " "}) == AzureTTS.DEFAULT_VOICE
def test_ssml_prosody_and_voice(self):
t = AzureTTS()
ssml = t._build_ssml(
{"text": "Hello world", "rate": "-8%", "pitch": "+1st"},
"en-US-AndrewMultilingualNeural",
)
assert '<voice name="en-US-AndrewMultilingualNeural">' in ssml
assert '<prosody rate="-8%" pitch="+1st">Hello world</prosody>' in ssml
assert 'xml:lang="en-US"' in ssml
assert "<mstts:express-as" not in ssml # no style requested
def test_ssml_style_wrapping(self):
t = AzureTTS()
ssml = t._build_ssml(
{"text": "Hi", "style": "narration-professional"}, "en-US-JennyNeural"
)
assert '<mstts:express-as style="narration-professional">' in ssml
assert "</mstts:express-as>" in ssml
def test_ssml_escapes_xml(self):
t = AzureTTS()
ssml = t._build_ssml({"text": "Bread & <butter>"}, "en-US-GuyNeural")
assert "Bread &amp; &lt;butter&gt;" in ssml
assert "<butter>" not in ssml
def test_ssml_custom_locale(self):
t = AzureTTS()
ssml = t._build_ssml({"text": "Hallo", "locale": "de-DE"}, "de-DE-KatjaNeural")
assert 'xml:lang="de-DE"' in ssml
def test_host_prefers_explicit_endpoint(self, monkeypatch):
monkeypatch.setenv("AZURE_TTS_ENDPOINT", "https://custom.tts.example.com/")
assert AzureTTS()._host() == "https://custom.tts.example.com"
monkeypatch.delenv("AZURE_TTS_ENDPOINT")
monkeypatch.setenv("AZURE_SPEECH_REGION", "westeurope")
assert AzureTTS()._host() == "https://westeurope.tts.speech.microsoft.com"
# ---- execute() guardrails + mocked success ----
class TestExecute:
def test_missing_credentials(self, monkeypatch):
monkeypatch.delenv("AZURE_SPEECH_KEY", raising=False)
monkeypatch.delenv("AZURE_SPEECH_REGION", raising=False)
monkeypatch.delenv("AZURE_TTS_ENDPOINT", raising=False)
res = AzureTTS().execute({"text": "hello"})
assert not res.success
assert "not configured" in res.error.lower()
def test_success_path_mocked(self, azure_env, tmp_path, monkeypatch):
import requests
captured = {}
def fake_post(url, headers=None, data=None, timeout=None):
captured["url"] = url
captured["headers"] = headers
captured["body"] = data
return _FakeResponse()
monkeypatch.setattr(requests, "post", fake_post)
out = tmp_path / "narration.mp3"
res = AzureTTS().execute(
{"text": "Hello world", "voice": "andrew", "output_path": str(out)}
)
assert res.success
assert res.model == "azure-neural-tts:en-US-AndrewMultilingualNeural"
assert res.data["provider"] == "azure"
assert res.data["voice"] == "en-US-AndrewMultilingualNeural"
assert res.data["text_length"] == len("Hello world")
# cost is rounded to 4 decimals by estimate_cost
assert res.cost_usd == pytest.approx(round(11 * 16.0 / 1_000_000, 4))
# audio bytes written to the requested path
assert out.read_bytes() == FAKE_MP3
assert res.artifacts == [str(out)]
# correct endpoint, auth header, and output format used
assert captured["url"] == "https://eastus.tts.speech.microsoft.com/cognitiveservices/v1"
assert captured["headers"]["Ocp-Apim-Subscription-Key"] == "fake-key"
assert captured["headers"]["X-Microsoft-OutputFormat"] == "audio-48khz-192kbitrate-mono-mp3"
assert b"Hello world" in captured["body"]
def test_wav_output_format(self, azure_env, tmp_path, monkeypatch):
import requests
captured = {}
def fake_post(url, headers=None, data=None, timeout=None):
captured["headers"] = headers
return _FakeResponse(content=b"RIFF....")
monkeypatch.setattr(requests, "post", fake_post)
out = tmp_path / "narration.wav"
res = AzureTTS().execute(
{"text": "Hi", "output_format": "wav", "output_path": str(out)}
)
assert res.success
assert captured["headers"]["X-Microsoft-OutputFormat"] == "riff-48khz-16bit-mono-pcm"
assert out.exists()
def test_http_error_surfaced(self, azure_env, tmp_path, monkeypatch):
import requests
monkeypatch.setattr(
requests, "post",
lambda *a, **k: _FakeResponse(content=b"", status_code=401, text="Unauthorized"),
)
res = AzureTTS().execute(
{"text": "hello", "output_path": str(tmp_path / "x.mp3")}
)
assert not res.success
assert "401" in res.error
def test_request_exception_surfaced(self, azure_env, tmp_path, monkeypatch):
import requests
def boom(*a, **k):
raise requests.exceptions.ConnectionError("no route to host")
monkeypatch.setattr(requests, "post", boom)
res = AzureTTS().execute(
{"text": "hello", "output_path": str(tmp_path / "x.mp3")}
)
assert not res.success
assert "no route to host" in res.error

289
tools/audio/azure_tts.py Normal file
View File

@@ -0,0 +1,289 @@
"""Azure AI Speech text-to-speech provider tool.
Neural TTS served by Azure AI Speech via the REST v1 endpoint. This is an
optional cloud TTS provider; when ``AZURE_SPEECH_KEY`` + ``AZURE_SPEECH_REGION``
are configured the agent may prefer it for high-quality narration, while the
local ``piper_tts`` tool remains the default offline path.
Shares the same Speech resource credentials as the ``azure_stt`` transcription
tool (one key/region unlocks both directions). Uses the synchronous
``/cognitiveservices/v1`` endpoint with an SSML body — no token exchange, Blob
storage, or job polling required.
Docs: https://learn.microsoft.com/azure/ai-services/speech-service/rest-text-to-speech
"""
from __future__ import annotations
import os
import time
from pathlib import Path
from typing import Any
from xml.sax.saxutils import escape
from tools.base_tool import (
BaseTool,
Determinism,
ExecutionMode,
ResourceProfile,
RetryPolicy,
ToolResult,
ToolRuntime,
ToolStability,
ToolStatus,
ToolTier,
)
# Output format tokens keyed by container. Chosen for compositing quality.
_MP3_FORMAT = "audio-48khz-192kbitrate-mono-mp3"
_WAV_FORMAT = "riff-48khz-16bit-mono-pcm"
class AzureTTS(BaseTool):
name = "azure_tts"
version = "0.1.0"
tier = ToolTier.VOICE
capability = "tts"
provider = "azure"
stability = ToolStability.BETA
execution_mode = ExecutionMode.SYNC
# Azure neural TTS is effectively deterministic for a fixed voice + SSML.
determinism = Determinism.DETERMINISTIC
runtime = ToolRuntime.API
# Availability is decided by get_status() (env var check), mirroring the
# azure_stt and elevenlabs_tts provider tools — dependencies stays empty.
dependencies = []
install_instructions = (
"Set your Azure AI Speech credentials (same resource as azure_stt):\n"
" export AZURE_SPEECH_KEY=your_speech_resource_key\n"
" export AZURE_SPEECH_REGION=eastus # your Speech resource region\n"
"Create a Speech resource in the Azure portal "
"(https://portal.azure.com) — the key and region are on its "
"'Keys and Endpoint' page. Optionally set AZURE_TTS_ENDPOINT to a full "
"custom TTS host (e.g. https://<region>.tts.speech.microsoft.com)."
)
fallback = "piper_tts"
fallback_tools = ["elevenlabs_tts", "openai_tts", "piper_tts"]
agent_skills = ["azure-text-to-speech", "text-to-speech"]
capabilities = [
"text_to_speech",
"voice_selection",
"ssml_support",
"prosody_control",
]
supports = {
"voice_cloning": False,
"multilingual": True,
"offline": False,
"native_audio": True,
}
best_for = [
"high-quality neural narration on Azure credentials",
"calm, confident explainer / founder-register delivery",
"cloud TTS that shares one key with azure_stt",
]
not_good_for = [
"fully offline production (use piper_tts)",
"voice cloning (use elevenlabs_tts)",
]
# A small curated shortlist of expressive en-US neural voices. Any valid
# Azure voice short name may be passed via `voice`.
RECOMMENDED_VOICES = {
"andrew": "en-US-AndrewMultilingualNeural", # warm, confident, conversational (founder)
"brandon": "en-US-BrandonMultilingualNeural", # deeper, measured
"ava": "en-US-AvaMultilingualNeural", # confident, bright female
"guy": "en-US-GuyNeural", # authoritative
"jenny": "en-US-JennyNeural", # friendly, clear
}
DEFAULT_VOICE = "en-US-AndrewMultilingualNeural"
input_schema = {
"type": "object",
"required": ["text"],
"properties": {
"text": {"type": "string", "description": "Text to convert to speech"},
"voice": {
"type": "string",
"description": (
"Azure voice short name (e.g. 'en-US-AndrewMultilingualNeural') "
"or a shortlist alias: andrew, brandon, ava, guy, jenny. "
"Default: en-US-AndrewMultilingualNeural."
),
},
"rate": {
"type": "string",
"description": (
"SSML prosody rate, e.g. '-8%', '0%', '+5%', or 'slow'/'medium'. "
"Default '0%'."
),
"default": "0%",
},
"pitch": {
"type": "string",
"description": "SSML prosody pitch, e.g. '-2st', '0%', '+1st'. Default '0%'.",
"default": "0%",
},
"style": {
"type": "string",
"description": (
"Optional express-as style for voices that support it "
"(e.g. 'narration-professional', 'calm', 'newscast'). Omit for neutral."
),
},
"locale": {
"type": "string",
"default": "en-US",
"description": "BCP-47 locale for the SSML <speak> element.",
},
"output_path": {"type": "string"},
"output_format": {
"type": "string",
"enum": ["mp3", "wav"],
"default": "mp3",
"description": "Container: 48kHz 192kbit mp3 or 48kHz 16-bit PCM wav.",
},
},
}
output_schema = {
"type": "object",
"properties": {
"provider": {"type": "string"},
"voice": {"type": "string"},
"output": {"type": "string"},
"format": {"type": "string"},
"text_length": {"type": "integer"},
},
}
resource_profile = ResourceProfile(
cpu_cores=1, ram_mb=256, vram_mb=0, disk_mb=50, network_required=True
)
retry_policy = RetryPolicy(
max_retries=2,
retryable_errors=["ConnectionError", "Timeout", "429", "503"],
)
idempotency_key_fields = ["text", "voice", "rate", "pitch", "style", "output_format"]
side_effects = ["writes audio file to output_path", "sends text to Azure AI Speech"]
user_visible_verification = ["Listen to generated audio for natural speech quality"]
# Azure neural TTS Standard tier bills roughly $16 per 1M characters.
COST_PER_CHAR = 16.0 / 1_000_000
def get_status(self) -> ToolStatus:
if os.environ.get("AZURE_SPEECH_KEY") and (
os.environ.get("AZURE_SPEECH_REGION") or os.environ.get("AZURE_TTS_ENDPOINT")
):
return ToolStatus.AVAILABLE
return ToolStatus.UNAVAILABLE
def estimate_cost(self, inputs: dict[str, Any]) -> float:
return round(len(inputs.get("text", "")) * self.COST_PER_CHAR, 4)
def estimate_runtime(self, inputs: dict[str, Any]) -> float:
# Well under real-time for typical narration segments.
return 10.0
def _host(self) -> str:
endpoint = os.environ.get("AZURE_TTS_ENDPOINT")
if endpoint:
return endpoint.rstrip("/")
region = os.environ.get("AZURE_SPEECH_REGION", "").strip()
return f"https://{region}.tts.speech.microsoft.com"
def _resolve_voice(self, inputs: dict[str, Any]) -> str:
voice = (inputs.get("voice") or "").strip()
if not voice:
return self.DEFAULT_VOICE
return self.RECOMMENDED_VOICES.get(voice.lower(), voice)
def _build_ssml(self, inputs: dict[str, Any], voice: str) -> str:
locale = inputs.get("locale", "en-US")
rate = inputs.get("rate", "0%")
pitch = inputs.get("pitch", "0%")
style = inputs.get("style")
text = escape(inputs["text"])
inner = f'<prosody rate="{escape(rate)}" pitch="{escape(pitch)}">{text}</prosody>'
if style:
inner = (
f'<mstts:express-as style="{escape(style)}">{inner}</mstts:express-as>'
)
return (
f'<speak version="1.0" '
f'xmlns="http://www.w3.org/2001/10/synthesis" '
f'xmlns:mstts="https://www.w3.org/2001/mstts" '
f'xml:lang="{locale}">'
f'<voice name="{voice}">{inner}</voice></speak>'
)
def execute(self, inputs: dict[str, Any]) -> ToolResult:
api_key = os.environ.get("AZURE_SPEECH_KEY")
if not api_key or not (
os.environ.get("AZURE_SPEECH_REGION") or os.environ.get("AZURE_TTS_ENDPOINT")
):
return ToolResult(
success=False,
error="Azure Speech is not configured. " + self.install_instructions,
)
start = time.time()
try:
result = self._synthesize(inputs, api_key)
except Exception as exc:
return ToolResult(success=False, error=f"TTS generation failed: {exc}")
result.duration_seconds = round(time.time() - start, 2)
result.cost_usd = self.estimate_cost(inputs)
return result
def _synthesize(self, inputs: dict[str, Any], api_key: str) -> ToolResult:
import requests
voice = self._resolve_voice(inputs)
container = inputs.get("output_format", "mp3")
azure_format = _WAV_FORMAT if container == "wav" else _MP3_FORMAT
ext = "wav" if container == "wav" else "mp3"
ssml = self._build_ssml(inputs, voice)
url = f"{self._host()}/cognitiveservices/v1"
headers = {
"Ocp-Apim-Subscription-Key": api_key,
"Content-Type": "application/ssml+xml",
"X-Microsoft-OutputFormat": azure_format,
"User-Agent": "OpenMontage-azure-tts",
}
try:
response = requests.post(
url, headers=headers, data=ssml.encode("utf-8"), timeout=120
)
except requests.RequestException as exc:
return ToolResult(success=False, error=f"Azure TTS request failed: {exc}")
if response.status_code != 200:
detail = response.text[:500] if response.text else ""
return ToolResult(
success=False,
error=f"Azure TTS returned HTTP {response.status_code}: {detail}",
)
output_path = Path(inputs.get("output_path", f"tts_output.{ext}"))
output_path.parent.mkdir(parents=True, exist_ok=True)
output_path.write_bytes(response.content)
return ToolResult(
success=True,
data={
"provider": self.provider,
"voice": voice,
"text_length": len(inputs["text"]),
"output": str(output_path),
"format": azure_format,
},
artifacts=[str(output_path)],
model=f"azure-neural-tts:{voice}",
)