## What this PR does
Adds support for Alibaba Cloud's hosted Fun-ASR-Flash snapshots to the
existing Tongyi-Qianwen speech-to-text provider.
- registers `fun-asr-flash-2026-06-15` as a speech-to-text model;
- routes only `fun-asr-flash*` models to the documented workspace-native
multimodal-generation endpoint;
- supports local audio through size-checked data URIs as well as
URL/data-URI inputs;
- uses the documented SSE response mode for incremental streaming
transcription;
- closes the streamed HTTP response on completion, failure, or early
consumer cancellation;
- preserves the existing `dashscope.MultiModalConversation` path for all
other Qwen audio models;
- keeps RAGFlow's existing synchronous and streaming adapter interfaces.
## Why
Fun-ASR-Flash does not use the legacy Qwen audio request shape currently
used by `QWenSeq2txt`. Its synchronous API expects `input_audio` at:
`/api/v1/services/aigc/multimodal-generation/generation`
Without a narrowly scoped adapter path, the hosted model cannot be
selected successfully through RAGFlow's Tongyi-Qianwen speech-to-text
provider.
Closes#16843.
## Compatibility
The new behavior is gated by the `fun-asr-flash` model-name prefix.
Existing Qwen audio models continue through the original code path
unchanged.
## Validation
- `pytest test/unit_test/rag/llm/test_sequence2txt_model.py`: 10 passed
- Ruff check: passed
- Ruff format check: passed
- `llm_factories.json` validation: passed
- Real hosted-API validation with WAV audio
- Real RAGFlow upload/indexing validation with MP3 audio
The unit tests cover the native Fun-ASR-Flash request, regression
behavior for the legacy Qwen path, SSE streaming, and early response
cleanup.
## Documentation
- https://help.aliyun.com/document_detail/2979031.html
- https://help.aliyun.com/document_detail/2869541.html
### Why a dedicated adapter path is necessary (official evidence)
Alibaba Cloud's [Fun-ASR RESTful API
reference](https://help.aliyun.com/en/model-studio/fun-asr-recorded-speech-recognition-http-api)
makes the incompatibilities with RAGFlow's existing Qwen audio path
explicit:
| Adapter change | Official API requirement | Why the existing path is
insufficient |
| --- | --- | --- |
| Call the workspace-native HTTP endpoint | The Fun-ASR-Flash
synchronous section states that SDK calls are not supported and
specifies `POST /api/v1/services/aigc/multimodal-generation/generation`.
| The existing adapter calls `dashscope.MultiModalConversation`, so a
direct HTTP path is required. |
| Use the `input_audio` message shape | `input.messages`, `content`,
`type: input_audio`, `input_audio`, and `input_audio.data` are
documented as required for an audio request. | The existing Qwen path
sends the legacy `audio` content shape, which does not match this API
contract. |
| Send `parameters.format` | The request schema marks `parameters` and
`format` as **Required**, and says the value must match the actual audio
format. | The legacy request has no Fun-ASR-Flash `parameters.format`
field, so the adapter must derive and send it. |
| Encode local files as Data URIs | `input_audio.data` accepts either a
public URL or a Base64 Data URI; the reference gives the exact
`data:{MIME_TYPE};base64,...` form. | RAGFlow supplies local file paths,
which the remote API cannot read directly. |
| Parse `output.text` | The documented non-streaming response returns
the accumulated transcription in `output.text`. | The legacy Qwen
response parser reads `output.choices[].message.content`, so a separate
response parser is required. |
| Enforce the Base64 input limit | The reference requires the
Base64-encoded audio to remain within the 10 MB input limit. | The
adapter checks encoded size before reading/sending local audio and
directs oversized inputs to the existing public-URL path. |
| Use SSE for streaming | The reference specifies `X-DashScope-SSE:
enable` and documents intermediate and final SSE events. | The adapter
parses those events instead of wrapping one blocking response as a
synthetic stream. |
| Release streamed responses | Streaming responses must be closed when
iteration completes or stops early. | A `finally` cleanup releases the
HTTP response on completion, errors, and consumer cancellation. |
`sample_rate` is documented as **Optional**. The implementation omits it
instead of declaring a fixed value that may not match remote or
compressed audio.
The [official speech-to-text model
list](https://help.aliyun.com/en/model-studio/asr-model/) separately
confirms that `fun-asr-flash-2026-06-15` is an offline HTTP model with a
five-minute audio limit.
---------
Signed-off-by: LauraGPT <LauraGPT@users.noreply.github.com>
Co-authored-by: openhands <openhands@all-hands.dev>
Co-authored-by: LauraGPT <LauraGPT@users.noreply.github.com>
### Summary
Adds FunASR as a self-hosted speech-to-text provider through its
OpenAI-compatible `/v1/audio/transcriptions` endpoint.
This is a focused replacement for #15526 by @Rene0422 and relates to
#15448. The unrelated Markdown parser changes from the previous branch
are intentionally removed so this PR contains only the FunASR provider
integration.
- register FunASR as a `SPEECH2TEXT` factory;
- add `FunASRSeq2txt` with `sensevoice` and `http://localhost:8000/v1`
defaults, an optional API key, URL normalization, and inherited
transcription handling;
- wire FunASR into the current local-provider schema with a prefilled
local URL and official documentation link;
- discover the server's `/v1/models` dynamically and expose every
returned model as speech-to-text in the model picker;
- use RAGFlow's existing default provider icon fallback instead of
referencing a missing `funasr` asset;
- list FunASR in the supported-provider documentation;
- add focused backend and frontend regression tests.
### Validation
- focused backend pytest suite -> `7 passed`
- real CPU `funasr-server` + RAGFlow provider smoke test -> discovered
`fun-asr-nano`, `sensevoice`, and `paraformer`; transcribed a real WAV
as `我现在在录一段测试音频` (`10` tokens, `0.504s`)
- `ruff check` and `ruff format --check` on the changed Python files
- `python3 -m py_compile` on the provider and its test
- JSON parse and a semantic assertion for exactly one enabled FunASR
`SPEECH2TEXT` factory
- focused frontend Jest test -> `2 passed`
- ESLint and Prettier on all changed TypeScript files
- `npm run build` -> production build succeeded (`14,181` modules
transformed)
- `git diff --check`
### Deployment
Run FunASR separately and point the RAGFlow provider at it:
```bash
pip install funasr
funasr-server --device cuda --model sensevoice
```
The API key remains optional because the stock local server does not
require authentication. A key can still be supplied when the endpoint is
protected by a gateway.
---------
Signed-off-by: LauraGPT <LauraGPT@users.noreply.github.com>
Co-authored-by: LauraGPT <LauraGPT@users.noreply.github.com>
## Summary
Add support for **"New API"** as a model provider, enabling connection
to [New API](https://github.com/QuantumNous/new-api) /
[one-api](https://github.com/songquanpeng/one-api) compatible gateways
that aggregate multiple LLM backends behind a unified OpenAI-compatible
`/v1` endpoint.
### Features
- **All model types**: Chat, Embedding, Rerank, Image2Text, TTS,
Speech2Text
- **List Models discovery**: `NewAPI(OpenAIAPICompatible)` class in
`model_meta.py` queries the gateway's `/v1/models` to auto-discover
available models via the native `GET /api/v1/providers/<name>/models`
endpoint
- **Model parameter editing**: Pencil icon on each discovered model row
to edit `model_type`, `max_tokens`, and `features` (e.g. tool call
support) before submitting
- **Custom model addition**: "Add Custom Model" button at the bottom of
the List Models dropdown for models not returned by the API
- **Gear icon settings**: Enabled the Settings gear button on provider
instances to manage models on existing instances (viewMode)
- **viewMode credential passthrough**: Fixed List Models in viewMode —
merges `initialValues` credentials when `api_key`/`base_url` fields are
hidden by `hideWhenInstanceExists`
### Changes
**Backend** (8 files):
- `rag/llm/chat_model.py` — `NewAPIChat(Base)` class
- `rag/llm/embedding_model.py` — `NewAPIEmbed(OpenAIEmbed)` class (no
auto `/v1` append)
- `rag/llm/rerank_model.py` — `NewAPIRerank(Base)` class (uses `/rerank`
endpoint)
- `rag/llm/cv_model.py` — `NewAPICv(GptV4)` class
- `rag/llm/tts_model.py` — `NewAPITTS(OpenAITTS)` class
- `rag/llm/sequence2txt_model.py` — `NewAPISeq2txt(GPTSeq2txt)` class
- `rag/llm/model_meta.py` — `NewAPI(OpenAIAPICompatible)` class for List
Models discovery
- `conf/llm_factories.json` — New API factory entry with all model type
tags
**Frontend** (8 files + 1 new SVG):
- `web/src/assets/svg/llm/new-api.svg` — New API logo icon
- `web/src/constants/llm.ts` — `LLMFactory.NewAPI` enum + `IconMap`
entry
- `web/src/components/svg-icon.tsx` — `NewAPI` added to `svgIcons`
-
`web/src/pages/user-setting/setting-model/modal/provider-modal/field-config/local-llm-configs.ts`
— New API `buildLocalConfig`
-
`web/src/pages/user-setting/setting-model/modal/provider-modal/constants.ts`
— `LIST_MODEL_PROVIDERS` includes NewAPI
- `web/src/pages/user-setting/setting-model/components/used-model.tsx` —
Enable Settings gear button
-
`web/src/pages/user-setting/setting-model/modal/provider-modal/hooks/use-list-models-picker.ts`
— viewMode credential merge + model editing state/handlers
-
`web/src/pages/user-setting/setting-model/modal/provider-modal/hooks/use-list-models-options.tsx`
— Pencil edit icon per model row
-
`web/src/pages/user-setting/setting-model/modal/provider-modal/index.tsx`
— `AddCustomModelDialog` import + edit dialog rendering
**Note on Go implementation**: A Go model driver (`NewAPIModel`
delegating to `OpenAIModel`) has been prepared but is deferred until the
Go runtime is enabled in a future release (current v0.26.0 images use
`API_PROXY_SCHEME=python` and do not compile Go binaries). Will submit
as a follow-up PR.
## Related
- Depends on: #15996 (provider instance API improvements — server-side
credential lookup, idempotent `add_model`, security fixes — required for
viewMode gear icon and batch model submission)
## Test plan
- [ ] Add New API provider with api_key and base_url pointing to an
OpenAI-compatible gateway
- [ ] Click "List Models" — should discover and display available models
from `/v1/models`
- [ ] Click pencil icon on a model — should open edit dialog to change
model_type, max_tokens, features
- [ ] Select multiple models and click OK — should add all selected
models
- [ ] Click gear icon on the added instance — should open viewMode with
List Models working
- [ ] In viewMode, select new models including pre-existing ones, click
OK — should succeed (requires #15996)
- [ ] Verify all model types work: create a Chat assistant, Embedding
KB, Rerank setting
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Tim Wang <wanghualoong@users.noreply.github.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
### What problem does this PR solve?
Multiple `requests.post()` calls across the LLM integration layer lack a
`timeout` parameter. Without a timeout, a single unresponsive upstream
service can block the calling thread **indefinitely**, eventually
exhausting the thread pool and degrading the entire system.
This is a well-known issue — Python's `requests` library defaults to
`timeout=None` (infinite wait), and [the library docs explicitly
recommend](https://requests.readthedocs.io/en/latest/user/advanced/#timeouts)
always setting a timeout.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
### Change
Added `timeout` to all `requests.post()` calls missing it:
| File | Calls fixed | Timeout |
|------|-------------|---------|
| `rag/llm/rerank_model.py` | 9 | 30s |
| `rag/llm/embedding_model.py` | 8 | 30s |
| `rag/llm/cv_model.py` | 3 | 60s |
| `rag/llm/tts_model.py` | 2 | 60s |
| `rag/llm/sequence2txt_model.py` | 2 | 60s |
Embedding/rerank calls use 30s (lightweight API calls). Vision, TTS, and
audio transcription use 60s (heavier workloads with file uploads).
Note: other files in the codebase (e.g. `check_minio_alive`,
`check_ragflow_server_alive`) already use `timeout=10`, so this PR
brings the LLM layer in line with existing practice.
Signed-off-by: Ricardo-M-L <Sibyl_Hartmanbnb@webname.com>
Co-authored-by: Kevin Hu <kevinhu.sh@gmail.com>
## Summary
- Convert bare `open()` calls to `with` context managers or
`Path.read_text()`
- File handles leak if not properly closed, especially on exceptions
- Fixes in crypt.py, sequence2txt_model.py, term_weight.py,
deepdoc/vision/__init__.py
## Test plan
- [x] File operations work correctly with context managers
- [x] Resources properly cleaned up on exceptions
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
### What problem does this PR solve?
This PR aims to extend the list of possible providers. Adds new Provider
"RAGcon" within the Ollama Modal. It provides all model types except OCR
via Openai-compatible endpoints.
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
---------
Co-authored-by: Jakob <16180662+hauberj@users.noreply.github.com>
### What problem does this PR solve?
Update stepfun list.
Add TTS and Sequence2Text functionalities.
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
### What problem does this PR solve?
change:
new api /sequence2txt,
update QWenSeq2txt and ZhipuSeq2txt
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
### Related issues
#10078
### What problem does this PR solve?
Integrate DeerAPI provider.
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
- [x] Documentation Update
Co-authored-by: DeerAPI <tensor.null@gmail.com>
### What problem does this PR solve?
Rename the CometEmbed and CometSeq2txt classes to CometAPIEmbed and
CometAPISeq2txt, and correct supported_models.mdx.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
### Related issues
#10078
### What problem does this PR solve?
Integrate CometAPI provider.
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
- [x] Documentation Update
### What problem does this PR solve?
Dataflow support audio. And fix giteeAI's sequence2text model.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
- [x] New Feature (non-breaking change which adds functionality)
### What problem does this PR solve?
Refactor OpenAI to enable audio parsing.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
- [x] Refactoring
Updated constructors for base and derived classes in chat, embedding,
rerank, sequence2txt, and tts models to accept **kwargs. This change
improves extensibility and allows passing additional parameters without
breaking existing interfaces.
- [x] Bug Fix (non-breaking change which fixes an issue)
---------
Co-authored-by: IT: Sop.Son <sop.son@feavn.local>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
### What problem does this PR solve?
Add model provider DeepInfra. This model list comes from our community.
NOTE: most endpoints haven't been tested, but they should work as OpenAI
does.
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
### What problem does this PR solve?
https://github.com/infiniflow/ragflow/issues/6138
This PR is going to support vision llm for gpustack, modify url path
from `/v1-openai` to `/v1`
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
### What problem does this PR solve?
Add GPUStack as a new model provider.
[GPUStack](https://github.com/gpustack/gpustack) is an open-source GPU
cluster manager for running LLMs. Currently, locally deployed models in
GPUStack cannot integrate well with RAGFlow. GPUStack provides both
OpenAI compatible APIs (Models / Chat Completions / Embeddings /
Speech2Text / TTS) and other APIs like Rerank. We would like to use
GPUStack as a model provider in ragflow.
[GPUStack Docs](https://docs.gpustack.ai/latest/quickstart/)
Related issue: https://github.com/infiniflow/ragflow/issues/4064.
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
### Testing Instructions
1. Install GPUStack and deploy the `llama-3.2-1b-instruct` llm, `bge-m3`
text embedding model, `bge-reranker-v2-m3` rerank model,
`faster-whisper-medium` Speech-to-Text model, `cosyvoice-300m-sft` in
GPUStack.
2. Add provider in ragflow settings.
3. Testing in ragflow.
### What problem does this PR solve?
Fix keys of Xinference deployed models, especially has the same model
name with public hosted models.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
---------
Co-authored-by: 0000sir <0000sir@gmail.com>
Co-authored-by: Kevin Hu <kevinhu.sh@gmail.com>
### What problem does this PR solve?
add support for Tencent Cloud ASR
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
---------
Co-authored-by: Zhedong Cen <cenzhedong2@126.com>
Co-authored-by: Kevin Hu <kevinhu.sh@gmail.com>
### What problem does this PR solve?
Related source file is in Windows/DOS format, they are format to Unix
format.
### Type of change
- [x] Refactoring
Signed-off-by: Jin Hai <haijin.chn@gmail.com>
