ragflow/internal/agent/audio/tts_design.md

# TTS engine — Phase 8b design decision

Status: **decision recorded, implementation pending**. The TTS
scaffold exists (internal/agent/audio/tts.go); this doc records
how a real engine should be wired in. The current scaffold's
shellSynthesizer uses an invented `--engine --text --voice --lang`
protocol that no real TTS binary matches — it must not be used
as-is.

## Context (what the Python side actually does)

The Python `agent/canvas.py:518-521` does NOT use `gtts` or
`edge-tts` shell-out. It looks up the tenant's default TTS model
via `get_tenant_default_model_by_type(self._tenant_id, LLMType.TTS)`
and creates an `LLMBundle(tenant, tts_model_config)`. The TTS
factory in `rag/llm/tts_model.py` dispatches to one of several
**HTTP-based** providers:

| Provider | Backend | Pure Go? |
|----------|---------|----------|
| FishAudioTTS | HTTP POST to `api.fish.audio/v1/tts` (msgpack body) | feasible |
| QwenTTS | DashScope SDK (over WebSocket) | heavy |
| OpenAITTS | HTTP POST to OpenAI-compatible `/audio/tts` | feasible |
| StepFunTTS | HTTP POST to vendor endpoint | feasible |
| RAGconTTS | LiteLLM proxy (HTTP) | feasible |
| XinferenceTTS | HTTP POST to xinference | feasible |
| TongyiTTS | DashScope SDK | heavy |

None of the production providers are gtts / edge-tts. The "gtts or
edge-tts shell-out" wording in the original plan was a placeholder
that didn't survive the v3.1 review; the production TTS layer is
HTTP / SDK all the way down.

## Options for the Go port

### A. Reimplement the HTTP clients in Go

Write Go HTTP clients for each provider (Fish / OpenAI / StepFun
/ Xinference / LiteLLM-proxy). Skip the DashScope SDK variants
for now (Qwen, Tongyi) — those are websocket-heavy and can
follow in a later phase.

Pros
: No Python dependency on the Go side.
: Lower latency, no subprocess overhead per call.
: Clean integration with the rest of the Go runtime.

Cons
: Five HTTP surfaces to maintain in lockstep with the Python
  ones. Every vendor release needs a Go update.
: The Python TTS layer is part of a multi-tenant
  `LLMBundle` abstraction (config, key rotation, retry
  policy) — reimplementing just the wire layer loses those
  cross-cutting concerns.

### B. Shell out to a Python subprocess that uses rag.llm.tts_model

Spawn `python3 -c "from rag.llm.tts_model import ...; ..."` and
pipe the audio bytes back.

Pros
: Reuses the verified Python TTS layer verbatim — including
  all providers, the LLMBundle config / key handling, and the
  retry / streaming logic.
: Plan §2.11.4-style "don't rewrite the vendor layer" applies
  here too: rag/llm/tts_model.py IS the vendor layer.
: One Python subprocess call covers all current providers.

Cons
: Per-call latency = Python interpreter startup + TTS module
  import + HTTP to vendor. ~hundreds of ms.
: Adds a Python dependency on the Go host.
: Stream chunks back from Python are awkward (binary audio).

### C. Hybrid — reimplement the simple HTTP ones, shell out for the rest

Reimplement OpenAI / Fish / Xinference / StepFun / LiteLLM-proxy
(5 providers, all straightforward HTTP). Shell out only for the
DashScope-SDK providers (Qwen, Tongyi) — those are websocket
clients whose cost / benefit doesn't justify a reimplementation
until the Go side has more DashScope users.

Pros
: Common case (OpenAI-compatible / LiteLLM proxy) has no
  Python dep.
: Avoids the worst of option B (latency + Python dep) for
  the providers most users actually deploy.

Cons
: Two code paths to maintain (Go HTTP + Python SDK).
: DashScope providers are exactly the ones several Chinese
  RAGFlow operators use — leaving them on the Python
  fallback is a real capability gap.

## Decision

**Option A (reimplement the HTTP clients in Go).** Reasoning:

1. All non-DashScope providers are straightforward HTTP POSTs
   with JSON / msgpack bodies. None of them need a streaming
   audio reader that's complex enough to justify a Python
   subprocess call.
2. The Go `rag/llm` package already has the same factory
   pattern for chat / embedding / rerank models. TTS slots
   into the same plumbing.
3. Option B's latency cost (~hundreds of ms for a Python
   interpreter + module import) is wasted — audio synthesis
   takes seconds anyway; the Python-startup overhead is
   noise compared to the network round-trip.
4. Option C's hybrid is the worst of both worlds: a new
   Go codebase that has to keep parity with a Python fallback
   for the DashScope providers.

For the DashScope SDK providers (Qwen, Tongyi), we accept
"not yet supported in Go" — the user can fall back to the
Python Canvas. Loud-fail via the existing
`ErrTTSEngineNotConfigured` pattern.

## Implementation sketch

1. New file `internal/rag/llm/tts_model.go` (Go side) with:
   - `TTSModel` interface: `Synthesize(ctx, text) (io.Reader, error)`
   - `FishTTS`, `OpenAITTS`, `XinferenceTTS`, `StepFunTTS`,
     `LiteLLMProxyTTS` implementations.
   - `TTSFactory` registry mirroring the Python one.
2. Wire `audio.Synthesizer` in the existing scaffold to
   delegate to `rag/llm/tts_model` (replacing the invented
   shell-out protocol in the current scaffold).
3. New file `internal/rag/llm/tts_model_test.go` with HTTP
   mock tests for each provider's request shape.

## What the existing scaffold needs to change

The current `shellSynthesizer.Synthesize` uses
`--engine --text --voice --lang` argv — no real TTS binary
matches that. Until option A lands, the existing scaffold
should be considered a placeholder. A subsequent commit
should either:
- (a) delete `shellSynthesizer` and `InstallShellSynthesizer`,
      leaving only the stub that returns
      `ErrTTSEngineNotConfigured`, or
- (b) replace them with a thin Python-subprocess client
      using the proven `rag.llm.tts_model` entry point (a
      safe interim — same pattern as CodeExec).

Option (a) is the safer default: it removes a footgun
(a non-functional shell-out) and lets the operator
discover the deferred state through the standard
"ErrTTSEngineNotConfigured" error. Option (b) is the
"ship a working path today" choice.

The choice between (a) and (b) is left to the implementer
of option A; until then, the scaffold's `shellSynthesizer`
is dead code.