mirror of
https://github.com/infiniflow/ragflow.git
synced 2026-07-16 20:57:21 +08:00
eeb59ec4f2
## What this PR does Adds support for Alibaba Cloud's hosted Fun-ASR-Flash snapshots to the existing Tongyi-Qianwen speech-to-text provider. - registers `fun-asr-flash-2026-06-15` as a speech-to-text model; - routes only `fun-asr-flash*` models to the documented workspace-native multimodal-generation endpoint; - supports local audio through size-checked data URIs as well as URL/data-URI inputs; - uses the documented SSE response mode for incremental streaming transcription; - closes the streamed HTTP response on completion, failure, or early consumer cancellation; - preserves the existing `dashscope.MultiModalConversation` path for all other Qwen audio models; - keeps RAGFlow's existing synchronous and streaming adapter interfaces. ## Why Fun-ASR-Flash does not use the legacy Qwen audio request shape currently used by `QWenSeq2txt`. Its synchronous API expects `input_audio` at: `/api/v1/services/aigc/multimodal-generation/generation` Without a narrowly scoped adapter path, the hosted model cannot be selected successfully through RAGFlow's Tongyi-Qianwen speech-to-text provider. Closes #16843. ## Compatibility The new behavior is gated by the `fun-asr-flash` model-name prefix. Existing Qwen audio models continue through the original code path unchanged. ## Validation - `pytest test/unit_test/rag/llm/test_sequence2txt_model.py`: 10 passed - Ruff check: passed - Ruff format check: passed - `llm_factories.json` validation: passed - Real hosted-API validation with WAV audio - Real RAGFlow upload/indexing validation with MP3 audio The unit tests cover the native Fun-ASR-Flash request, regression behavior for the legacy Qwen path, SSE streaming, and early response cleanup. ## Documentation - https://help.aliyun.com/document_detail/2979031.html - https://help.aliyun.com/document_detail/2869541.html ### Why a dedicated adapter path is necessary (official evidence) Alibaba Cloud's [Fun-ASR RESTful API reference](https://help.aliyun.com/en/model-studio/fun-asr-recorded-speech-recognition-http-api) makes the incompatibilities with RAGFlow's existing Qwen audio path explicit: | Adapter change | Official API requirement | Why the existing path is insufficient | | --- | --- | --- | | Call the workspace-native HTTP endpoint | The Fun-ASR-Flash synchronous section states that SDK calls are not supported and specifies `POST /api/v1/services/aigc/multimodal-generation/generation`. | The existing adapter calls `dashscope.MultiModalConversation`, so a direct HTTP path is required. | | Use the `input_audio` message shape | `input.messages`, `content`, `type: input_audio`, `input_audio`, and `input_audio.data` are documented as required for an audio request. | The existing Qwen path sends the legacy `audio` content shape, which does not match this API contract. | | Send `parameters.format` | The request schema marks `parameters` and `format` as **Required**, and says the value must match the actual audio format. | The legacy request has no Fun-ASR-Flash `parameters.format` field, so the adapter must derive and send it. | | Encode local files as Data URIs | `input_audio.data` accepts either a public URL or a Base64 Data URI; the reference gives the exact `data:{MIME_TYPE};base64,...` form. | RAGFlow supplies local file paths, which the remote API cannot read directly. | | Parse `output.text` | The documented non-streaming response returns the accumulated transcription in `output.text`. | The legacy Qwen response parser reads `output.choices[].message.content`, so a separate response parser is required. | | Enforce the Base64 input limit | The reference requires the Base64-encoded audio to remain within the 10 MB input limit. | The adapter checks encoded size before reading/sending local audio and directs oversized inputs to the existing public-URL path. | | Use SSE for streaming | The reference specifies `X-DashScope-SSE: enable` and documents intermediate and final SSE events. | The adapter parses those events instead of wrapping one blocking response as a synthetic stream. | | Release streamed responses | Streaming responses must be closed when iteration completes or stops early. | A `finally` cleanup releases the HTTP response on completion, errors, and consumer cancellation. | `sample_rate` is documented as **Optional**. The implementation omits it instead of declaring a fixed value that may not match remote or compressed audio. The [official speech-to-text model list](https://help.aliyun.com/en/model-studio/asr-model/) separately confirms that `fun-asr-flash-2026-06-15` is an offline HTTP model with a five-minute audio limit. --------- Signed-off-by: LauraGPT <LauraGPT@users.noreply.github.com> Co-authored-by: openhands <openhands@all-hands.dev> Co-authored-by: LauraGPT <LauraGPT@users.noreply.github.com>