conf/models/vllm.json

{
  "name": "vllm",
  "url_suffix": {
    "chat": "chat/completions",
    "models": "models",
    "embedding": "embeddings",
    "rerank": "rerank"
  },
  "class": "local"
}
Update create model instance command (#14441) ### What problem does this PR solve? 1. support command: ``` RAGFlow(user)> create provider 'vllm' instance 'test' key 'test-key' url 'base-url' region 'abc'; SUCCESS RAGFlow(user)> list instances from 'vllm'; +----------+----------------------------------------+----------------------------------+--------------+----------------------------------+--------+ \| apiKey \| extra \| id \| instanceName \| providerID \| status \| +----------+----------------------------------------+----------------------------------+--------------+----------------------------------+--------+ \| test-key \| {"base_url":"base-url","region":"abc"} \| 40213c89430311f1a7cf38a74640adcc \| test \| b4d40e6142d311f1a4f938a74640adcc \| enable \| +----------+----------------------------------------+----------------------------------+--------------+----------------------------------+--------+ ``` 2. support add vllm model ``` RAGFlow(user)> add model 'Qwen/Qwen2-0.5B' to provider 'vllm' instance 'test' with tokens 131072 chat; SUCCESS ``` 3. add vllm chat ### Type of change - [x] New Feature (non-breaking change which adds functionality) - [x] Refactoring --------- Signed-off-by: Jin Hai <haijin.chn@gmail.com> 2026-04-29 17:05:08 +08:00			`{`
			`"name": "vllm",`
			`"url_suffix": {`
			`"chat": "chat/completions",`
Go: implement Encode (embeddings) in vLLM driver (#14688) ### What problem does this PR solve? The vLLM Go driver shipped with a stub \`Encode\` method that returned \`not implemented\`, even though vLLM is one of the most common production-grade self-hosted inference servers and exposes an OpenAI-compatible embeddings endpoint at \`/v1/embeddings\`. Users who self-host \`BAAI/bge-m3\`, \`Qwen3-Embedding-\`, \`NV-Embed-v2\`, or similar models on vLLM could not run an embedding call through the Go layer. The existing \`ListModels\` already discovers the loaded models, but the embedding path failed because \`Encode\` was a stub. ### What this PR includes - \`conf/models/vllm.json\`: add \`\"embedding\": \"embeddings\"\` under \`url_suffix\` so the driver can build the URL from config. - \`internal/entity/models/vllm.go\`: replace the \`Encode\` stub with a real implementation. Adds a small local response type that matches the OpenAI-compatible shape. No factory change. No interface change. ### How the driver works - Validate the model name. The API key is optional for self-hosted vLLM, so the Authorization header is only set when both \`apiConfig\` and \`ApiKey\` are non-nil and non-empty, the same pattern the recently merged CheckConnection PR (#14614) uses. - Resolve the region with a default fallback. Return a clear "missing base URL" error when the user has not configured the local access address yet. - Use a per-call \`context.WithTimeout(30s)\` and \`http.NewRequestWithContext\`, the same pattern the merged Aliyun Encode (#14647) and in-flight Ollama Encode (#14664) use. - Send \`{model, input: [texts]}\` in one request. - Parse \`data[].embedding\` and copy each slice into a \`[][]float64\` indexed by \`data[*].index\`, so the output order matches the input order. - Handle both \`float64\` and \`float32\` element types. - Empty input returns \`[][]float64{}\` with no HTTP call. - Length mismatch between input and result, out-of-range index, and any missing slot all return clear errors instead of silent zero vectors. ### Type of change - [x] New Feature (non-breaking change which adds functionality) ### How was this tested? - \`go build ./internal/entity/models/...\` in a clean go 1.25 image returns exit 0. - The full method set on \`VllmModel\` still matches the \`ModelDriver\` interface. - Pattern parity with the merged Aliyun Encode (#14647), the in-flight Ollama Encode (#14664), and the existing SiliconFlow Encode. Closes #14687 2026-05-11 06:09:17 +02:00			`"models": "models",`
Go: implement Rerank in vLLM driver (#14878) (#14880) ### What problem does this PR solve? Closes #14878. `VllmModel.Rerank()` in [internal/entity/models/vllm.go:551](internal/entity/models/vllm.go#L551) is currently a stub returning `nil, fmt.Errorf("%s, Rerank not implemented", z.Name())`, and [conf/models/vllm.json](conf/models/vllm.json) is missing a `rerank` entry in `url_suffix`. Chat (long-standing) and embeddings (#14688) already work, so rerank is the last missing leg of the retrieval pipeline for operators running everything on a single self-hosted vLLM server — today they have to point rerank at a different provider, which defeats the point of a fully local deployment. Upstream vLLM has supported a Jina/Cohere-compatible `POST /v1/rerank` endpoint since v0.7 ([vllm-project/vllm#12376](https://github.com/vllm-project/vllm/pull/12376)). The request/response shape is essentially identical to the NVIDIA driver landed in #14778, so this PR mirrors that structure with two vLLM-specific adjustments. This PR replaces the stub with a real implementation against vLLM's `/v1/rerank`: - `POST {baseURL}/rerank` - Request body: `{"model": "<modelName>", "query": "<query>", "documents": [...], "top_n": <int>}` — documents are a flat `[]string`, not wrapped as `{text: "..."}` like NVIDIA's `/ranking`. - Response body: `{"results": [{"index": int, "relevance_score": float}, ...]}` (Jina-compatible; the optional `document` field is ignored since callers reconstruct text via `Index`). - `Authorization: Bearer <ApiKey>` is set only when `APIConfig.ApiKey` is non-empty, matching the existing `Embed`/`ListModels` behaviour in this file. vLLM is a local driver and can be deployed without an API key. The return shape matches the existing `RerankResponse` contract used by the NVIDIA ([nvidia.go:461](internal/entity/models/nvidia.go#L461)), Aliyun ([aliyun.go:507](internal/entity/models/aliyun.go#L507)), and ZhipuAI ([zhipu-ai.go:554](internal/entity/models/zhipu-ai.go#L554)) drivers, i.e. `Data []RerankResult` carrying `{Index, RelevanceScore}` in the API's ranking order. Callers that need original-input order sort by `Index`. Behaviour requirements from the issue, all covered: 1. Empty `documents` → returns `&RerankResponse{}` without an HTTP call. 2. Missing `modelName` → `"model name is required"` validation error. 3. `rerankConfig.TopN` honored when `0 < TopN < len(documents)`; otherwise `top_n` defaults to `len(documents)` so callers get a score per input. 4. Non-200 responses return an error including upstream status and body (`"vLLM rerank API error: <status>, body: <body>"`). 5. Response `index` values are bounds-checked against `len(documents)`. Scope:* - [internal/entity/models/vllm.go](internal/entity/models/vllm.go) — replaces the `Rerank` stub at line 551 with a real implementation; adds `vllmRerankRequest`/`vllmRerankResponse` types for the slim subset of the payload we need. Region/baseURL resolution, 30s context timeout, conditional bearer header, and error wrapping all follow the existing patterns in this file. - [conf/models/vllm.json](conf/models/vllm.json) — adds `"rerank": "rerank"` to `url_suffix`, joined to the operator-configured vLLM base URL the same way the NVIDIA driver joins at [nvidia.go:485](internal/entity/models/nvidia.go#L485). - [internal/entity/models/vllm_rerank_test.go](internal/entity/models/vllm_rerank_test.go) — adds 7 `httptest`-backed tests mirroring `nvidia_rerank_test.go`: happy path (out-of-order ranking → Index preservation), `top_n` clamp to `RerankConfig.TopN`, empty-documents short-circuit, missing-model-name validation, HTTP error propagation, out-of-range index rejection, and a vLLM-specific `TestVllmRerankWithoutAPIKey` locking in the optional-auth behaviour that distinguishes this driver from NVIDIA. Out of scope: no interface change, no DDL, no frontend change. Chat, embeddings, and balance paths are untouched. No new user-facing docs required beyond the existing rerank model setup page — vLLM joins the list of providers whose rerank model can be selected once `/v1/rerank` is exposed by the server. ### Type of change - [x] New Feature (non-breaking change which adds functionality) 2026-05-14 22:27:22 -07:00			`"embedding": "embeddings",`
			`"rerank": "rerank"`
Update create model instance command (#14441) ### What problem does this PR solve? 1. support command: ``` RAGFlow(user)> create provider 'vllm' instance 'test' key 'test-key' url 'base-url' region 'abc'; SUCCESS RAGFlow(user)> list instances from 'vllm'; +----------+----------------------------------------+----------------------------------+--------------+----------------------------------+--------+ \| apiKey \| extra \| id \| instanceName \| providerID \| status \| +----------+----------------------------------------+----------------------------------+--------------+----------------------------------+--------+ \| test-key \| {"base_url":"base-url","region":"abc"} \| 40213c89430311f1a7cf38a74640adcc \| test \| b4d40e6142d311f1a4f938a74640adcc \| enable \| +----------+----------------------------------------+----------------------------------+--------------+----------------------------------+--------+ ``` 2. support add vllm model ``` RAGFlow(user)> add model 'Qwen/Qwen2-0.5B' to provider 'vllm' instance 'test' with tokens 131072 chat; SUCCESS ``` 3. add vllm chat ### Type of change - [x] New Feature (non-breaking change which adds functionality) - [x] Refactoring --------- Signed-off-by: Jin Hai <haijin.chn@gmail.com> 2026-04-29 17:05:08 +08:00			`},`
			`"class": "local"`
			`}`