mirror of
https://github.com/infiniflow/ragflow.git
synced 2026-07-26 02:13:29 +08:00
feat(stt): support Fun-ASR-Flash in Tongyi-Qianwen provider (#16844)
## What this PR does Adds support for Alibaba Cloud's hosted Fun-ASR-Flash snapshots to the existing Tongyi-Qianwen speech-to-text provider. - registers `fun-asr-flash-2026-06-15` as a speech-to-text model; - routes only `fun-asr-flash*` models to the documented workspace-native multimodal-generation endpoint; - supports local audio through size-checked data URIs as well as URL/data-URI inputs; - uses the documented SSE response mode for incremental streaming transcription; - closes the streamed HTTP response on completion, failure, or early consumer cancellation; - preserves the existing `dashscope.MultiModalConversation` path for all other Qwen audio models; - keeps RAGFlow's existing synchronous and streaming adapter interfaces. ## Why Fun-ASR-Flash does not use the legacy Qwen audio request shape currently used by `QWenSeq2txt`. Its synchronous API expects `input_audio` at: `/api/v1/services/aigc/multimodal-generation/generation` Without a narrowly scoped adapter path, the hosted model cannot be selected successfully through RAGFlow's Tongyi-Qianwen speech-to-text provider. Closes #16843. ## Compatibility The new behavior is gated by the `fun-asr-flash` model-name prefix. Existing Qwen audio models continue through the original code path unchanged. ## Validation - `pytest test/unit_test/rag/llm/test_sequence2txt_model.py`: 10 passed - Ruff check: passed - Ruff format check: passed - `llm_factories.json` validation: passed - Real hosted-API validation with WAV audio - Real RAGFlow upload/indexing validation with MP3 audio The unit tests cover the native Fun-ASR-Flash request, regression behavior for the legacy Qwen path, SSE streaming, and early response cleanup. ## Documentation - https://help.aliyun.com/document_detail/2979031.html - https://help.aliyun.com/document_detail/2869541.html ### Why a dedicated adapter path is necessary (official evidence) Alibaba Cloud's [Fun-ASR RESTful API reference](https://help.aliyun.com/en/model-studio/fun-asr-recorded-speech-recognition-http-api) makes the incompatibilities with RAGFlow's existing Qwen audio path explicit: | Adapter change | Official API requirement | Why the existing path is insufficient | | --- | --- | --- | | Call the workspace-native HTTP endpoint | The Fun-ASR-Flash synchronous section states that SDK calls are not supported and specifies `POST /api/v1/services/aigc/multimodal-generation/generation`. | The existing adapter calls `dashscope.MultiModalConversation`, so a direct HTTP path is required. | | Use the `input_audio` message shape | `input.messages`, `content`, `type: input_audio`, `input_audio`, and `input_audio.data` are documented as required for an audio request. | The existing Qwen path sends the legacy `audio` content shape, which does not match this API contract. | | Send `parameters.format` | The request schema marks `parameters` and `format` as **Required**, and says the value must match the actual audio format. | The legacy request has no Fun-ASR-Flash `parameters.format` field, so the adapter must derive and send it. | | Encode local files as Data URIs | `input_audio.data` accepts either a public URL or a Base64 Data URI; the reference gives the exact `data:{MIME_TYPE};base64,...` form. | RAGFlow supplies local file paths, which the remote API cannot read directly. | | Parse `output.text` | The documented non-streaming response returns the accumulated transcription in `output.text`. | The legacy Qwen response parser reads `output.choices[].message.content`, so a separate response parser is required. | | Enforce the Base64 input limit | The reference requires the Base64-encoded audio to remain within the 10 MB input limit. | The adapter checks encoded size before reading/sending local audio and directs oversized inputs to the existing public-URL path. | | Use SSE for streaming | The reference specifies `X-DashScope-SSE: enable` and documents intermediate and final SSE events. | The adapter parses those events instead of wrapping one blocking response as a synthetic stream. | | Release streamed responses | Streaming responses must be closed when iteration completes or stops early. | A `finally` cleanup releases the HTTP response on completion, errors, and consumer cancellation. | `sample_rate` is documented as **Optional**. The implementation omits it instead of declaring a fixed value that may not match remote or compressed audio. The [official speech-to-text model list](https://help.aliyun.com/en/model-studio/asr-model/) separately confirms that `fun-asr-flash-2026-06-15` is an offline HTTP model with a five-minute audio limit. --------- Signed-off-by: LauraGPT <LauraGPT@users.noreply.github.com> Co-authored-by: openhands <openhands@all-hands.dev> Co-authored-by: LauraGPT <LauraGPT@users.noreply.github.com>
This commit is contained in:
Zane
@@ -21,6 +21,7 @@ import re
|
||||
from abc import ABC
|
||||
import tempfile
|
||||
import logging
|
||||
from urllib.parse import urlparse
|
||||
|
||||
import requests
|
||||
from openai import OpenAI
|
||||
@@ -83,14 +84,34 @@ class FuturMixSeq2txt(GPTSeq2txt):
|
||||
|
||||
class QWenSeq2txt(Base):
|
||||
_FACTORY_NAME = "Tongyi-Qianwen"
|
||||
_FUN_ASR_FLASH_PREFIX = "fun-asr-flash"
|
||||
_FUN_ASR_BASE64_MAX_SIZE = 10 * 1024 * 1024
|
||||
_DASHSCOPE_API_BASE = "https://dashscope.aliyuncs.com/api/v1"
|
||||
_AUDIO_MIME_FORMATS = {
|
||||
"audio/mpeg": "mp3",
|
||||
"audio/mp3": "mp3",
|
||||
"audio/wav": "wav",
|
||||
"audio/wave": "wav",
|
||||
"audio/x-wav": "wav",
|
||||
}
|
||||
|
||||
def __init__(self, key, model_name="qwen-audio-asr", **kwargs):
|
||||
def __init__(self, key, model_name="qwen-audio-asr", base_url=None, **kwargs):
|
||||
import dashscope
|
||||
|
||||
dashscope.api_key = key
|
||||
self.api_key = key
|
||||
self.model_name = model_name
|
||||
self.base_url = (base_url or self._DASHSCOPE_API_BASE).rstrip("/")
|
||||
|
||||
def transcription(self, audio_path):
|
||||
# Fun-ASR-Flash uses DashScope's workspace-scoped native multimodal
|
||||
# endpoint and payload instead of MultiModalConversation.
|
||||
if self.model_name.startswith(self._FUN_ASR_FLASH_PREFIX):
|
||||
return self._transcribe_fun_asr_flash(audio_path)
|
||||
|
||||
return self._transcribe_qwen_audio(audio_path)
|
||||
|
||||
def _transcribe_qwen_audio(self, audio_path):
|
||||
import dashscope
|
||||
|
||||
if audio_path.startswith("http"):
|
||||
@@ -108,7 +129,126 @@ class QWenSeq2txt(Base):
|
||||
text = "**ERROR**: " + str(e)
|
||||
return text, num_tokens_from_string(text)
|
||||
|
||||
@classmethod
|
||||
def _fun_asr_audio_format(cls, audio_path):
|
||||
"""Derive the Fun-ASR audio format from a data URI, URL, or path."""
|
||||
|
||||
if audio_path.startswith("data:"):
|
||||
mime_type = audio_path[5:].split(";", 1)[0].lower()
|
||||
if not mime_type.startswith("audio/"):
|
||||
raise ValueError(f"Unsupported audio data URI MIME type: {mime_type or 'missing'}")
|
||||
audio_format = cls._AUDIO_MIME_FORMATS.get(mime_type, mime_type.split("/", 1)[1].removeprefix("x-"))
|
||||
else:
|
||||
path = urlparse(audio_path).path if audio_path.startswith(("http://", "https://")) else audio_path
|
||||
audio_format = os.path.splitext(path)[1].lower().lstrip(".")
|
||||
|
||||
if audio_format == "wave":
|
||||
audio_format = "wav"
|
||||
if not audio_format:
|
||||
raise ValueError("Cannot determine audio format; use a URL/path extension or an audio data URI MIME type")
|
||||
return audio_format
|
||||
|
||||
@classmethod
|
||||
def _validate_fun_asr_base64_size(cls, encoded_size):
|
||||
if encoded_size > cls._FUN_ASR_BASE64_MAX_SIZE:
|
||||
raise ValueError("Fun-ASR-Flash Base64 audio exceeds the 10 MB encoded-input limit; provide a publicly accessible URL (for example, OSS) instead")
|
||||
|
||||
def _fun_asr_flash_request(self, audio_path, *, stream=False):
|
||||
audio_format = self._fun_asr_audio_format(audio_path)
|
||||
|
||||
if audio_path.startswith(("http://", "https://")):
|
||||
audio_input = audio_path
|
||||
elif audio_path.startswith("data:"):
|
||||
_, separator, encoded_audio = audio_path.partition(",")
|
||||
if not separator:
|
||||
raise ValueError("Invalid audio data URI: missing Base64 payload")
|
||||
self._validate_fun_asr_base64_size(len(encoded_audio.encode("utf-8")))
|
||||
audio_input = audio_path
|
||||
else:
|
||||
file_size = os.path.getsize(audio_path)
|
||||
encoded_size = 4 * ((file_size + 2) // 3)
|
||||
self._validate_fun_asr_base64_size(encoded_size)
|
||||
mime_type = "audio/mpeg" if audio_format == "mp3" else f"audio/{audio_format}"
|
||||
with open(audio_path, "rb") as audio_file:
|
||||
audio_input = f"data:{mime_type};base64,{base64.b64encode(audio_file.read()).decode('utf-8')}"
|
||||
|
||||
api_base = self.base_url
|
||||
if api_base.endswith("/compatible-mode/v1"):
|
||||
api_base = api_base[: -len("/compatible-mode/v1")] + "/api/v1"
|
||||
url = f"{api_base}/services/aigc/multimodal-generation/generation"
|
||||
payload = {
|
||||
"model": self.model_name,
|
||||
"input": {
|
||||
"messages": [
|
||||
{
|
||||
"role": "user",
|
||||
"content": [{"type": "input_audio", "input_audio": {"data": audio_input}}],
|
||||
}
|
||||
]
|
||||
},
|
||||
# sample_rate is optional in the Fun-ASR-Flash API. Omitting it
|
||||
# avoids declaring incorrect metadata for remote or compressed audio.
|
||||
"parameters": {"format": audio_format},
|
||||
}
|
||||
headers = {
|
||||
"Authorization": f"Bearer {self.api_key}",
|
||||
"Content-Type": "application/json",
|
||||
"X-DashScope-SSE": "enable" if stream else "disable",
|
||||
}
|
||||
return url, headers, payload
|
||||
|
||||
def _transcribe_fun_asr_flash(self, audio_path):
|
||||
try:
|
||||
url, headers, payload = self._fun_asr_flash_request(audio_path)
|
||||
|
||||
response = requests.post(url, headers=headers, json=payload, timeout=60)
|
||||
response.raise_for_status()
|
||||
result = response.json()
|
||||
text = result.get("text") or result.get("output", {}).get("text")
|
||||
if not text:
|
||||
raise ValueError("Missing transcription text in Fun-ASR-Flash response")
|
||||
text = text.strip()
|
||||
return text, num_tokens_from_string(text)
|
||||
except Exception as e:
|
||||
logging.exception("Fun-ASR-Flash transcription failed for model %s", self.model_name)
|
||||
return "**ERROR**: " + str(e), 0
|
||||
|
||||
def _stream_fun_asr_flash(self, audio_path):
|
||||
response = None
|
||||
try:
|
||||
url, headers, payload = self._fun_asr_flash_request(audio_path, stream=True)
|
||||
response = requests.post(url, headers=headers, json=payload, timeout=60, stream=True)
|
||||
response.raise_for_status()
|
||||
|
||||
full = ""
|
||||
for line in response.iter_lines(decode_unicode=True):
|
||||
if not line or not line.startswith("data:"):
|
||||
continue
|
||||
event_data = line[5:].strip()
|
||||
if not event_data or event_data == "[DONE]":
|
||||
continue
|
||||
result = json.loads(event_data)
|
||||
text = result.get("text") or result.get("output", {}).get("text")
|
||||
if not text:
|
||||
continue
|
||||
full = text.strip()
|
||||
yield {"event": "delta", "text": full}
|
||||
|
||||
if not full:
|
||||
raise ValueError("Missing transcription text in Fun-ASR-Flash stream")
|
||||
yield {"event": "final", "text": full}
|
||||
except Exception as e:
|
||||
logging.exception("Fun-ASR-Flash streaming transcription failed for model %s", self.model_name)
|
||||
yield {"event": "error", "text": "**ERROR**: " + str(e)}
|
||||
finally:
|
||||
if response is not None:
|
||||
response.close()
|
||||
|
||||
def stream_transcription(self, audio_path):
|
||||
if self.model_name.startswith(self._FUN_ASR_FLASH_PREFIX):
|
||||
yield from self._stream_fun_asr_flash(audio_path)
|
||||
return
|
||||
|
||||
import dashscope
|
||||
|
||||
if audio_path.startswith("http"):
|
||||
|
||||
Reference in New Issue
Block a user