conf/models/xinference.json

{
  "name": "xinference",
  "url_suffix": {
    "chat": "v1/chat/completions",
    "embedding": "v1/embeddings",
    "models": "v1/models",
    "rerank": "v1/rerank",
    "asr": "v1/audio/transcriptions",
    "tts": "v1/audio/speech"
  },
  "class": "local"
}
Go: implement provider: Xinference (#14938) ### What problem does this PR solve? Closes #14808. Adds a Go model driver for Xinference so self-hosted Xinference chat models can be used through the Go provider layer instead of falling through to the dummy driver. Xinference exposes an OpenAI-compatible API under `/v1`; the driver accepts either a root endpoint such as `http://127.0.0.1:9997` or an OpenAI-compatible endpoint such as `http://127.0.0.1:9997/v1` and normalizes it before calling chat or model-listing routes. ### What is changed? - Add `internal/entity/models/xinference.go` implementing `ModelDriver` for Xinference chat. - Route provider name `xinference` in `internal/entity/models/factory.go`. - Add `conf/models/xinference.json` as a local provider config. - Add focused unit tests in `internal/entity/models/xinference_test.go`. Initial method coverage: - `ChatWithMessages`: POST `/v1/chat/completions`. - `ChatStreamlyWithSender`: SSE streaming from `/v1/chat/completions`. - `ListModels`: GET `/v1/models`. - `CheckConnection`: lightweight `ListModels` probe. - Optional auth: send `Authorization: Bearer <api_key>` only when a non-empty key is configured, matching Xinference no-auth and auth-enabled deployments. - `Balance`, `Embed`, `Rerank`, ASR, TTS, and OCR return `no such method` for this initial chat-provider PR. ### Type of change - [x] New Feature (non-breaking change which adds functionality) - [x] Bug Fix (non-breaking change which fixes an issue) ### Tests - `go test -vet=off -run TestXinference -count=1 ./internal/entity/models/...` - `go test -vet=off -count=1 ./internal/entity/models/...` ### References - Xinference docs: https://inference.readthedocs.io/zh-cn/latest/index.html - OpenAI-compatible chat usage: https://inference.readthedocs.io/zh-cn/latest/getting_started/using_xinference.html - API key auth: https://inference.readthedocs.io/zh-cn/latest/user_guide/auth_system.html --------- Co-authored-by: Jin Hai <haijin.chn@gmail.com> 2026-05-18 21:10:13 -10:00			`{`
			`"name": "xinference",`
			`"url_suffix": {`
			`"chat": "v1/chat/completions",`
Go: implement Embed in Xinference driver (#14932) ## Summary - Replaces the `"no such method"` stub on `XinferenceModel.Embed` (`internal/entity/models/xinference.go`) with a real implementation against Xinference's OpenAI-compatible `/v1/embeddings` endpoint. - Adds the `"embedding": "v1/embeddings"` URL suffix to `conf/models/xinference.json`. - Mirrors the Python `XinferenceEmbed` class in `rag/llm/embedding_model.py:407` for payload shape (OpenAI-compatible `model + input` → `data[].index + data[].embedding`) and tolerates the same no-auth default Xinference deployments use. Authorization is only sent when a non-empty API key is configured, via the existing `setXinferenceAuth` helper. - Reuses the existing `normalizeXinferenceBaseURL` + `baseURLForRegion` helpers so both `http://127.0.0.1:9997` and `http://127.0.0.1:9997/v1` resolve to the same `/v1/embeddings` target without doubled `/v1`. - Validates response indices — duplicate, missing, or out-of-range `data[*].index` values fail with a clear error rather than silently producing misaligned vectors. - Returns `[]EmbeddingData` in original input order (placed by `Index`) so downstream callers can index positionally without re-sorting. - Forwards `EmbeddingConfig.Dimension` as `dimensions` when `> 0`, matching the OpenAI cluster pattern. Closes #14810 Co-authored-by: Jin Hai <haijin.chn@gmail.com> 2026-05-20 17:47:30 -10:00			`"embedding": "v1/embeddings",`
Go: implement Rerank in Xinference driver (#15032) ### What problem does this PR solve? Fixes #14816 The Xinference Go driver landed chat in #14938 and Embed is in review in #14932, but `Rerank` shipped as a stub that returns `"xinference, no such method"`. Tenants who launch a rerank model with `--model-type rerank` on their Xinference instance cannot route it through the Go API server. This PR fills the gap. Xinference exposes an OpenAI-compatible REST API. The rerank endpoint is at `POST <base>/v1/rerank` and accepts the Cohere-shaped body `{model, query, documents, top_n}`, returning `{results: [{index, relevance_score}]}` — the same wire shape used by the merged NVIDIA (#14778), Aliyun (#14676), Gitee (#14656), ZhipuAI (#14608), Novita (#15014), and LocalAI (#14813) Rerank implementations. Documented in [Xinference rerank docs](https://inference.readthedocs.io/en/v1.6.1/models/model_abilities/rerank.html); the [builtin rerank model catalog](https://inference.readthedocs.io/en/stable/models/builtin/rerank/) lists `bge-reranker-base`, `bge-reranker-large`, `bge-reranker-v2-m3`, and others. 2026-05-20 16:14:30 -10:00			`"models": "v1/models",`
fix(Go): rewrite chat, listmodels, embed for Ollama (#15213) ### What problem does this PR solve? IDK how to implement `Ollama` on #14580 but it's totally wrong. This is the rewrite version for `Ollama` Verified from CLI ``` # Embed RAGFlow(user)> embed text 'what is rag' 'who are you' with 'nomic-embed-text:latest@test12@ollama' dimension 1024; +-----------+-------+ \| dimension \| index \| +-----------+-------+ \| 768 \| 0 \| \| 768 \| 1 \| +-----------+-------+ # Chat RAGFlow(user)> think chat with 'qwen3:0.6b@test12@ollama' message 'who r u' Thinking: Okay, the user asked, "Who r u?" I need to respond appropriately. First, I should acknowledge their question. Since I'm an AI, I don't have a physical form, but I can confirm that I'm a large language model. I should keep the response friendly and offer help. Let me make sure I'm not making up any information and that the response is natural. Also, I should check for any typos and ensure clarity. Alright, that should cover it. Answer: I'm an AI language model, and I don't have a physical form. However, I can tell you that I'm designed to assist with questions and tasks. How can I help you today? Time: 2.914285 RAGFlow(user)> stream think chat with 'qwen3:0.6b@test12@ollama' message 'who r u' Thinking: , the user asked, "Who are you?" I need to respond appropriately. Since I'm an AI assistant, I should mention that I don't have a physical form or a mind. I should also clarify that I can help with various tasks like answering questions or providing information. It's important to keep the response friendly and informative while maintaining the correct tone. Answer: don't have a physical form or a mind, but I'm here to help with your questions or tasks! What can I do for you today? Time: 1.740047 # LisyModels RAGFlow(user)> list supported models from 'ollama' 'test12' +-------------------------+ \| model_name \| +-------------------------+ \| nomic-embed-text:latest \| \| qwen3:0.6b \| +-------------------------+ ``` ### Type of change - [x] Bug Fix (non-breaking change which fixes an issue) - [x] Refactoring 2026-05-25 18:55:03 +08:00			`"rerank": "v1/rerank",`
			`"asr": "v1/audio/transcriptions",`
			`"tts": "v1/audio/speech"`
Go: implement provider: Xinference (#14938) ### What problem does this PR solve? Closes #14808. Adds a Go model driver for Xinference so self-hosted Xinference chat models can be used through the Go provider layer instead of falling through to the dummy driver. Xinference exposes an OpenAI-compatible API under `/v1`; the driver accepts either a root endpoint such as `http://127.0.0.1:9997` or an OpenAI-compatible endpoint such as `http://127.0.0.1:9997/v1` and normalizes it before calling chat or model-listing routes. ### What is changed? - Add `internal/entity/models/xinference.go` implementing `ModelDriver` for Xinference chat. - Route provider name `xinference` in `internal/entity/models/factory.go`. - Add `conf/models/xinference.json` as a local provider config. - Add focused unit tests in `internal/entity/models/xinference_test.go`. Initial method coverage: - `ChatWithMessages`: POST `/v1/chat/completions`. - `ChatStreamlyWithSender`: SSE streaming from `/v1/chat/completions`. - `ListModels`: GET `/v1/models`. - `CheckConnection`: lightweight `ListModels` probe. - Optional auth: send `Authorization: Bearer <api_key>` only when a non-empty key is configured, matching Xinference no-auth and auth-enabled deployments. - `Balance`, `Embed`, `Rerank`, ASR, TTS, and OCR return `no such method` for this initial chat-provider PR. ### Type of change - [x] New Feature (non-breaking change which adds functionality) - [x] Bug Fix (non-breaking change which fixes an issue) ### Tests - `go test -vet=off -run TestXinference -count=1 ./internal/entity/models/...` - `go test -vet=off -count=1 ./internal/entity/models/...` ### References - Xinference docs: https://inference.readthedocs.io/zh-cn/latest/index.html - OpenAI-compatible chat usage: https://inference.readthedocs.io/zh-cn/latest/getting_started/using_xinference.html - API key auth: https://inference.readthedocs.io/zh-cn/latest/user_guide/auth_system.html --------- Co-authored-by: Jin Hai <haijin.chn@gmail.com> 2026-05-18 21:10:13 -10:00			`},`
			`"class": "local"`
			`}`