mirror of
https://github.com/infiniflow/ragflow.git
synced 2026-09-08 10:14:35 +08:00
4c68a6b86ce0f126b4bbbc5df9f83d74ea8833a4
6236 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 FPlust
|
0734fd793a |
fix: scope pending_cell_images by sheet in excel parser (#14120)
pending_cell_images should be scoped by sheet ### What problem does this PR solve? _Briefly describe what this PR aims to solve. Include background context that will help reviewers understand the purpose of the PR._ ### Type of change - [x] Bug Fix (non-breaking change which fixes an issue) |
||
|
Wang Qi
|
3838770e7a |
GraphRAG feature - Part 1 - add spacy to extract entity and relation (#14670)
### What problem does this PR solve? GraphRAG feature - Part 1 - add spacy to extract entity and relation <img width="1621" height="1288" alt="image" src="https://github.com/user-attachments/assets/aadeddad-94da-46c6-adad-9c3784181f61" /> ### Type of change - [x] New Feature (non-breaking change which adds functionality) |
||
|
web-dev0521
|
cc207b5b05 |
Refactor: tidy up ThreadPoolExecutor lifecycle in file_service and task executor (#14668)
## Summary - Wrap the `ThreadPoolExecutor` instances in `FileService.parse_docs` and `FileService.get_files` with `with ... as exe:` blocks for deterministic cleanup - Replace the `concurrent.futures.ThreadPoolExecutor` in `do_handle_task` with `asyncio.create_task(asyncio.to_thread(build_TOC, ...))`, preserving the existing parallelism with chunk insertion while leveraging the surrounding async context - Drop the now-unused `import concurrent` and the `executor.shutdown(wait=False)` call in the `finally` block Closes #14622. No behavioral change, no public API change. Net diff: ~19 insertions / 25 deletions across two files. ## Test plan - [ ] `uv run ruff check api/db/services/file_service.py rag/svr/task_executor.py` passes - [ ] Upload a multi-file batch through the chat/file endpoint and confirm `FileService.parse_docs` still returns combined parsed text - [ ] Trigger `FileService.get_files` via the chat reference flow with a mix of image and non-image files; verify both `raw=True` and `raw=False` paths return correctly - [ ] Run a `naive`-parser document task with `toc_extraction: true` and confirm the TOC chunk is generated and inserted exactly as before - [ ] Run a `naive`-parser document task with `toc_extraction: false` and confirm the path with `toc_thread = None` is unaffected - [ ] Cancel a running task to exercise the `finally` block and confirm cleanup still works without the executor shutdown call --------- Co-authored-by: web-dev0521 <jasonpette1783@gmail.com> Co-authored-by: Wang Qi <wangq8@outlook.com> |
||
|
Joseff
|
13e6554901 |
Fix(Go): make OpenRouter Encode fail loudly on malformed responses (#14717)
### What problem does this PR solve?
The OpenRouter `Encode` method silently swallowed malformed responses.
If a `data[]` item from the API was missing a field (`index`,
`embedding`, or unexpected shape), the loop did `continue` instead of
returning an error — leaving `nil` entries in the result slice. Callers
got back partial results with no indication anything went wrong, which
then crashes downstream consumers when they try to use a `nil` vector.
There were three concrete gaps:
- No count-mismatch check between `data` length and input texts (only
checked for empty)
- No duplicate-index detection (a duplicate would silently overwrite)
- Parse failures on individual items returned partial slices instead of
erroring
This PR replaces `map[string]interface{}` parsing with a typed
`openrouterEmbeddingResponse` struct and applies the same 3-layer
validation used in the other drivers (count mismatch → out-of-range
index → duplicate index), so any malformed response produces a clear
error instead of corrupted data.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
|
||
|
Panda Dev
|
530edbac99 |
Go: implement Encode (embeddings) in LM Studio driver (#14694)
### What problem does this PR solve? The LM Studio Go driver shipped with a stub \`Encode\` method that returned \`no such method\`, even though LM Studio is one of the most common local LLM runners on macOS and Windows and exposes an OpenAI-compatible embeddings endpoint at \`/v1/embeddings\`. LM Studio users routinely load local embedding models such as \`nomic-ai/nomic-embed-text-v1.5\`, \`mixedbread-ai/mxbai-embed-large-v1\`, or \`BAAI/bge-m3\`. They run on the same \`/v1\` namespace as chat. The existing \`ListModels\` already discovers them, but because \`Encode\` was a stub, a tenant who picked one of these models in the Go layer could not actually run an embedding call. This finishes the local-LLM trio: Ollama Encode (#14664) and vLLM Encode (#14688) are already in flight, both using the same OpenAI-compatible \`/embeddings\` shape. ### What this PR includes - \`conf/models/lmstudio.json\`: add \`\"embedding\": \"embeddings\"\` under \`url_suffix\` so the driver can build the URL from config. - \`internal/entity/models/lmstudio.go\`: replace the \`Encode\` stub with a real implementation. Adds a small local response type that matches the OpenAI-compatible shape. No factory change. No interface change. ### How the driver works - Validate the model name. The API key is optional for local LM Studio, so the Authorization header is only set when both \`apiConfig\` and \`ApiKey\` are non-nil and non-empty, the same pattern the recently merged CheckConnection PR (#14614) uses. - Resolve the region with a default fallback. Return a clear "missing base URL" error when the user has not configured the local access address yet. - Use a per-call \`context.WithTimeout(30s)\` and \`http.NewRequestWithContext\`, the same pattern the merged Aliyun Encode (#14647) and the in-flight Ollama Encode (#14664) and vLLM Encode (#14688) use. - Send \`{model, input: [texts]}\` in one request. - Parse \`data[*].embedding\` and copy each slice into a \`[][]float64\` indexed by \`data[*].index\`, so the output order matches the input order. - Handle both \`float64\` and \`float32\` element types. - Empty input returns \`[][]float64{}\` with no HTTP call. - Length mismatch between input and result, out-of-range index, and any missing slot all return clear errors instead of silent zero vectors. ### Type of change - [x] New Feature (non-breaking change which adds functionality) ### How was this tested? - \`go build ./internal/entity/models/...\` in a clean go 1.25 image returns exit 0. - The full method set on \`LmStudioModel\` still matches the \`ModelDriver\` interface. - Pattern parity with the merged Aliyun Encode (#14647), the in-flight Ollama Encode (#14664) and vLLM Encode (#14688), and the existing SiliconFlow Encode. Closes #14693 |
||
|
Joseff
|
0580c137fa |
Perf(Go): batch SiliconFlow Encode requests with 32-item chunking (#14719)
### What problem does this PR solve? The SiliconFlow `Encode` method sent one HTTP request per text, which is wasteful and slow when indexing many documents (e.g., 100 docs = 100 round-trips). SiliconFlow's `/v1/embeddings` is OpenAI-compatible and accepts an array of strings in `input` (officially documented at https://docs.siliconflow.cn/en/api-reference/embeddings/create-embeddings, with a documented max array size of 32). This PR batches the requests up to that limit, reducing 100 docs to ~4 round-trips, and replaces `map[string]interface{}` parsing with a typed struct using the same 3-layer validation (count mismatch, out-of-range index, duplicate index) used in the other drivers. ### Type of change - [x] Performance Improvement |
||
|
BitToby
|
4b96362092 |
Go: implement Encode (embeddings) in NVIDIA driver (#14700)
### What problem does this PR solve? The NVIDIA Go driver in `internal/entity/models/nvidia.go` shipped with a stub `Encode` method that returned `no such method`. `conf/models/nvidia.json` already lists `nvidia/llama-3.2-nemoretriever-1b-vlm-embed-v1` as an embedding model, but the conf had no `embedding` URL suffix, so the picker had nothing wired even if `Encode` worked. A tenant who wanted to use NVIDIA NIM for chat (already working) and embeddings from a single provider could not, even though the upstream endpoint is public at `https://integrate.api.nvidia.com/v1/embeddings` and uses an OpenAI-compatible request body extended with the NVIDIA-specific `input_type` and `truncate` fields. Several other Go drivers already implement `Encode` (siliconflow, zhipu-ai, aliyun), so the interface and the pattern are well-established. This PR fills the gap. ### What this PR includes * `conf/models/nvidia.json`: declare the `embedding` URL suffix alongside the existing `chat` and `models` entries. The embedding model entry was already present, so no model addition is needed. * `internal/entity/models/nvidia.go`: replace the `Encode` stub with a real implementation. Adds a small local response type that matches the OpenAI-compatible shape NVIDIA NIM returns. No factory change. No interface change. ### How the driver works * Validates `apiConfig` and the API key, validates the model name, resolves the region with a default fallback (matching the pattern the merged `ListModels` and `CheckConnection` paths in this driver already use), and builds the URL from `BaseURL[region] + URLSuffix.Embedding`. * Sends all input texts in one request as the `input` array, with the NVIDIA-specific `input_type: "query"`, `encoding_format: "float"`, and `truncate: "END"` fields, mirroring the Python `NvidiaEmbed` reference. * Parses `data[*].embedding` and copies each slice into `[][]float64` indexed by `data[*].index` so the output order matches the input order even if the API returns items in a different order. * Handles both `float64` and `float32` element types. * Empty input returns `[][]float64{}` with no HTTP call. * Non-200 responses propagate the upstream status line and body. * A final pass checks every input slot got a vector and returns a clear error if any slot is still nil. * Per-call 30s context deadline so a slow call cannot block forever. ### Type of change - [x] New Feature (non-breaking change which adds functionality) ### How was this tested? * `go build ./internal/entity/models/...` returns exit 0. * `go vet ./internal/entity/models/...` is clean. * `gofmt -l internal/entity/models/nvidia.go` is clean. * The full method set on `NvidiaModel` still matches the `ModelDriver` interface. * Pattern parity with the just-merged Aliyun `Encode` (#14647). Closes #14699 |
||
|
Jack Storment
|
8ff623fbc4 |
Go: implement Encode (embeddings) in Ollama driver (#14664)
### What problem does this PR solve? The Ollama Go driver shipped with a stub \`Encode\` method that returned \`no such method\`, even though Ollama is one of the most common local LLM runners and exposes an OpenAI-compatible embeddings endpoint at \`/v1/embeddings\`. Ollama users routinely run local embedding models such as \`nomic-embed-text\`, \`mxbai-embed-large\`, or \`bge-m3\`. Pulled with \`ollama pull <model>\` and served on the same \`/v1\` namespace as chat. The existing \`ListModels\` already discovers them, but because \`Encode\` was a stub, a tenant who picked one of these models in the Go layer could not actually run an embedding call. ### What this PR includes - \`conf/models/ollama.json\`: add \`\"embedding\": \"embeddings\"\` under \`url_suffix\` so the driver can build the URL from config. - \`internal/entity/models/ollama.go\`: replace the \`Encode\` stub with a real implementation. Adds a small local response type that matches the OpenAI-compatible shape. No factory change. No interface change. ### How the driver works - Validate the model name. The API key is optional for local Ollama, so the Authorization header is only set when both \`apiConfig\` and \`ApiKey\` are non-nil and non-empty, the same pattern the recently merged CheckConnection PR (#14614) uses. - Resolve the region with a default fallback. Return a clear "missing base URL" error when the user has not configured the local access address yet. - Use a per-call \`context.WithTimeout(30s)\` and \`http.NewRequestWithContext\`, the same pattern the merged Aliyun Encode (#14647) uses. - Send \`{model, input: [texts]}\` in one request. - Parse \`data[*].embedding\` and copy each slice into a \`[][]float64\` indexed by \`data[*].index\`, so the output order matches the input order. - Handle both \`float64\` and \`float32\` element types. - Empty input returns \`[][]float64{}\` with no HTTP call. - Length mismatch between input and result, out-of-range index, and any missing slot all return clear errors instead of silent zero vectors. ### Type of change - [x] New Feature (non-breaking change which adds functionality) ### How was this tested? - \`go build ./internal/entity/models/...\` in a clean go 1.25 image returns exit 0. - The full method set on \`OllamaModel\` still matches the \`ModelDriver\` interface. - Pattern parity with the merged Aliyun Encode (#14647) and the existing SiliconFlow Encode. Closes #14662 |
||
|
hyl64
|
77ce88dfcc |
fix(prompt): reserve system budget in message_fit_in (#14164)
## Summary This PR fixes the `message_fit_in()` truncation bug reported in #13607. Changes: - fix the user-message truncation branch to reserve room for the system prompt token budget - guard the zero-token edge case to avoid dividing by zero in the truncation ratio check - add focused regression tests covering both the user-dominant truncation path and the zero-token boundary case ## Validation ```bash pytest -q --noconftest test/unit_test/rag/prompts/test_generator_message_fit_in.py ``` Result: `2 passed` Closes #13607 |
||
|
07heco
|
e46989832e |
fix: complete robustness fixes for rerank module addressing all review comments (#14265)
## Summary This PR fully addresses all CodeRabbit review feedback and enhances the robustness of the reranking module with 100% backward compatibility. ## Key Fixes 1. Fixed JinaRerank hardcoded base_url to support subclass endpoint overrides 2. Corrected GPUStackRerank exception handling to use proper requests exceptions and preserve stack traces 3. Added 30s timeout to all API calls to prevent service hanging 4. Added empty input validation for all rerank providers 5. Replaced direct dict key access with .get() to eliminate KeyError crashes 6. Fixed _normalize_rank edge case for empty arrays 7. Implemented missing functionality for Ai302Rerank 8. Standardized type hints and fixed typo issues ## Compatibility - No breaking changes to any existing functionality - All rerank providers work as originally intended - Fully compatible with existing configurations and workflows ### Type of change - [x] Bug Fix (non-breaking change which fixes an issue) - [x] Refactoring --------- Co-authored-by: Kevin Hu <kevinhu.sh@gmail.com> |
||
|
Panda Dev
|
fa53b93dd5 |
Go: implement Encode (embeddings) in vLLM driver (#14688)
### What problem does this PR solve? The vLLM Go driver shipped with a stub \`Encode\` method that returned \`not implemented\`, even though vLLM is one of the most common production-grade self-hosted inference servers and exposes an OpenAI-compatible embeddings endpoint at \`/v1/embeddings\`. Users who self-host \`BAAI/bge-m3\`, \`Qwen3-Embedding-*\`, \`NV-Embed-v2\`, or similar models on vLLM could not run an embedding call through the Go layer. The existing \`ListModels\` already discovers the loaded models, but the embedding path failed because \`Encode\` was a stub. ### What this PR includes - \`conf/models/vllm.json\`: add \`\"embedding\": \"embeddings\"\` under \`url_suffix\` so the driver can build the URL from config. - \`internal/entity/models/vllm.go\`: replace the \`Encode\` stub with a real implementation. Adds a small local response type that matches the OpenAI-compatible shape. No factory change. No interface change. ### How the driver works - Validate the model name. The API key is optional for self-hosted vLLM, so the Authorization header is only set when both \`apiConfig\` and \`ApiKey\` are non-nil and non-empty, the same pattern the recently merged CheckConnection PR (#14614) uses. - Resolve the region with a default fallback. Return a clear "missing base URL" error when the user has not configured the local access address yet. - Use a per-call \`context.WithTimeout(30s)\` and \`http.NewRequestWithContext\`, the same pattern the merged Aliyun Encode (#14647) and in-flight Ollama Encode (#14664) use. - Send \`{model, input: [texts]}\` in one request. - Parse \`data[*].embedding\` and copy each slice into a \`[][]float64\` indexed by \`data[*].index\`, so the output order matches the input order. - Handle both \`float64\` and \`float32\` element types. - Empty input returns \`[][]float64{}\` with no HTTP call. - Length mismatch between input and result, out-of-range index, and any missing slot all return clear errors instead of silent zero vectors. ### Type of change - [x] New Feature (non-breaking change which adds functionality) ### How was this tested? - \`go build ./internal/entity/models/...\` in a clean go 1.25 image returns exit 0. - The full method set on \`VllmModel\` still matches the \`ModelDriver\` interface. - Pattern parity with the merged Aliyun Encode (#14647), the in-flight Ollama Encode (#14664), and the existing SiliconFlow Encode. Closes #14687 |
||
|
Qinsanz
|
d6660cf156 |
fix(keyword_extraction): accept Chinese commas/semicolons/newlines as keyword delimiters (#14540)
## What
Widen the keyword delimiter in `rag/svr/task_executor.py`:
both `build_chunks` (LLM `keyword_extraction` cache parsing) and
`run_dataflow` (chunk-level `keywords` ingestion) now split on
`, ， ; ； 、 \r \n` instead of only ASCII comma.
## Why
`rag/prompts/keyword_prompt.md` instructs the LLM:
> The keywords are delimited by ENGLISH COMMA.
In practice, Chinese-leaning models (Qwen / Tongyi-Qianwen, GLM,
etc.) frequently ignore this instruction when the source content is
Chinese and emit Chinese commas (`，`) instead. Result:
`cached.split(",")` sees the full LLM output as a *single* keyword.
Repro: `auto_keywords>=4` + Chinese docs + `qwen-plus@Tongyi-Qianwen`.
We observed entries in `important_kwd` like
`"功能介绍，配置说明，参数详解，问题排查"` — one bucket instead of four.
## Impact
- Silent data-quality bug; no exception thrown.
- BM25 `important_kwd^30` boost effectively stops firing — the
indexed term is the whole list, never matches user query tokens.
- Any downstream aggregating `important_kwd` (tagging, analytics,
candidate-keyword review UIs) sees garbage.
## Compatibility
- Pure widening of the splitter; ASCII-comma-only outputs continue
to work identically.
- No schema / API change.
## Test plan
Manually verified against `qwen-plus@Tongyi-Qianwen` with
`auto_keywords=10` on Chinese .txt files:
- Before: `important_kwd` contains one element per chunk that is the
full LLM string with `，`-separated phrases inside.
- After: `important_kwd` contains N elements, one per phrase, as the
LLM intended.
|
||
|
BitToby
|
bfb4a0eea2 |
Go: implement Encode (embeddings) in Gitee AI driver (#14698)
### What problem does this PR solve? The Gitee AI Go driver in `internal/entity/models/gitee.go` shipped with a stub `Encode` method that returned `gitee, no such method`, even though `conf/models/gitee.json` already wires the `embedding` URL suffix. The conf also listed no embedding models, so the picker had nothing to select. This blocked any tenant who wanted to use Gitee AI for chat, rerank (already working, see #14656), and embeddings from a single provider. This PR fills the gap, mirroring the just-merged Aliyun `Encode` (#14647): - `internal/entity/models/gitee.go`: replace the `Encode` stub with a real implementation. Validates inputs, resolves the region with a default fallback, POSTs the standard OpenAI-compatible `{"model", "input": [...]}` body to `BaseURL[region] + URLSuffix.Embedding`, parses `data[*].embedding` indexed by `data[*].index` so output order matches input order, handles both `float64` and `float32` element types, and uses a 30s per-call context deadline matching the merged `Rerank`. - `conf/models/gitee.json`: add `BAAI/bge-m3` so the embedding picker has something to select. No factory change. No interface change. No URL suffix change. Verified with `go build`, `go vet`, and `gofmt -l` : all clean. Closes #14697 ### Type of change - [x] New Feature (non-breaking change which adds functionality) |
||
|
VincentLambert
|
b83e2ae5a2 |
fix: handle missing parent chunk in retrieval_by_children (#14556)
### What problem does this PR solve?
`retrieval_by_children()` in `rag/nlp/search.py` crashes with a
`TypeError: 'NoneType' object is not subscriptable` when a parent
("mom") chunk referenced by child chunks is missing from the index.
This happens when the index is in an inconsistent state — for example
after a partial re-index, a document deletion that didn't clean up all
children, or a race condition during ingestion. `dataStore.get()`
returns `None` for the missing parent, and the subsequent access to
`chunk["content_with_weight"]` raises a `TypeError`.
**Stack trace:**
```
TypeError: 'NoneType' object is not subscriptable
File "rag/nlp/search.py", line 792, in retrieval_by_children
"content_with_weight": chunk["content_with_weight"],
```
### Type of change
- [x] Bug Fix
### Fix
When `dataStore.get()` returns `None` for a parent chunk, fall back to
using the child chunks directly and continue processing the remaining
parents. This preserves retrieval results for all other chunks rather
than aborting the entire query with an exception.
```python
chunk = self.dataStore.get(id, idx_nms[0], [ck["kb_id"] for ck in cks])
if chunk is None:
chunks.extend(cks)
continue
```
---------
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
|
||
|
Sp1kyss
|
e6cb9faace |
fix: close two security analyzer bypass paths in sandbox executor (#14690)
## Summary
Two bypass vectors in the sandbox code security analyzer allowed
malicious code to pass the safety check undetected and reach the Docker
executor.
### 1. JavaScript: template-literal bypass of `require()` block
The `SecureJavaScriptAnalyzer` regex patterns used `['"]` to match
module names, covering only single and double quotes. An attacker could
use ES6 template literals to bypass all three `require` checks:
`javascript
const cp = require(`child_process`);
async function main() {
return cp.execSync('cat /etc/passwd').toString();
}
`
The same bypass applied to `fs` and `worker_threads`.
**Fix:** Updated all three `require` patterns from `['"]` to `['"\]` to
also match backtick template literals.
### 2. Python: `builtins` not blocked + attribute-call blind spot in
`visit_Call`
`visit_Call` only checked `ast.Name` nodes, so attribute-style calls
like `module.func()` were invisible to the analyzer. Additionally,
`builtins` was absent from `DANGEROUS_IMPORTS`. Combined, this allowed:
`python
import builtins
def main():
builtins.exec('import os; os.system("id")')
`
Neither the import nor the exec call triggered any flag.
**Fix:** Added `builtins` to `DANGEROUS_IMPORTS` and added an
`ast.Attribute` branch to `visit_Call` so that `module.dangerous_func()`
style calls are caught alongside bare `dangerous_func()` calls.
## Tests
Added four regression tests covering each new bypass vector:
- `test_javascript_child_process_template_literal_is_rejected`
- `test_javascript_fs_template_literal_is_rejected`
- `test_python_builtins_import_is_rejected`
- `test_python_attribute_eval_call_is_rejected`
---------
Co-authored-by: bounty-hunter <bounty@hunter.local>
|
||
|
Joseff
|
827cceccba |
Fix(Go): correct Name() and region URL fallback in Aliyun driver (#14673)
### What problem does this PR solve? Two bugs in the Aliyun Go driver: 1. **`Name()` returns `"siliconflow"`** — a copy-paste bug from when the driver was created. `Name()` is used in error messages and log output, so every Aliyun error incorrectly attributed itself to SiliconFlow. 2. **Silent empty URL for unknown regions in `ChatWithMessages`, `ChatStreamlyWithSender`, and `ListModels`** — all three methods construct the request URL as `z.BaseURL[region]` without checking whether the key exists. For an unrecognised region this returns `""`, producing a malformed URL like `"/chat/completions"` that the HTTP transport rejects with a confusing error. `Encode` and `Rerank` (already merged) correctly fall back to `"default"` and return a clear error. This PR applies the same pattern to the remaining three methods. ### Type of change - [x] Bug Fix (non-breaking change which fixes an issue) |
||
|
Carmen Fernández Ruiz
|
f852a7524e |
fix(go): wire Google CheckConnection to ListModels (#14660)
### What problem does this PR solve? Closes #14703 `GoogleModel.CheckConnection` currently returns a hardcoded `no such method` error even though the Google Go driver already supports `ListModels`. This makes provider connection checks fail regardless of whether the configured API key can list Google models. This PR makes `CheckConnection` call `ListModels`, adds a small API-key guard for nil, empty, and whitespace-only keys, and keeps `ListModels` useful by following paginated Google model responses. ### What stays unchanged * Google model listing still uses the Google GenAI SDK with `genai.BackendGeminiAPI`. * Model names still come from `models.Items[*].Name`. * `Balance`, `Encode`, chat, streaming, provider config, and factory wiring are unchanged. ### Tests and validation Added focused unit coverage for: * `CheckConnection` delegating to `ListModels` and returning its error * nil, missing, empty, and whitespace-only API key validation * model-name passthrough from the list-models adapter * paginated model listing, empty-result preservation, and next-page error propagation Validated current PR head `17ceef43515ba8c46c254dd349b9085bf26dcbea` locally with Go 1.25.0: * `go test ./internal/entity/models -run 'TestGoogleModel|TestCollectGoogleModelNames' -count=1 -v` - PASS * `go test ./internal/entity/models -count=1` - PASS * `go test -race ./internal/entity/models -count=1` - PASS * `gofmt -w internal/entity/models/google.go internal/entity/models/google_test.go` - PASS, no diff * `git diff --check` - PASS ### Type of change * [x] Bug Fix (non-breaking change which fixes an issue) Co-authored-by: Jin Hai <haijin.chn@gmail.com> |
||
|
Joseff
|
f4f8bed9f7 |
Go: implement Encode (embeddings) in Google Gemini driver (#14682)
### What problem does this PR solve? - Implements the `Encode` method in the Google Gemini driver, which was previously a stub returning `not implemented` - Uses the `google.golang.org/genai` SDK's `EmbedContent` API, which routes to the `batchEmbedContents` endpoint internally — all texts are sent in a single request - Adds `text-embedding-004` (max 2048 tokens) to `conf/models/google.json` - Response values are `[]float32` from the SDK and are cast to `[]float64` to satisfy the `ModelDriver` interface ## Files changed - `internal/entity/models/google.go` — full `Encode` implementation - `conf/models/google.json` — adds `text-embedding-004` embedding model ### Type of change - [x] New Feature (non-breaking change which adds functionality) |
||
|
Ricardo-M-L
|
13922209e6 |
fix(llm): add timeout to HTTP requests in LLM integration layer (#14313)
### What problem does this PR solve? Multiple `requests.post()` calls across the LLM integration layer lack a `timeout` parameter. Without a timeout, a single unresponsive upstream service can block the calling thread **indefinitely**, eventually exhausting the thread pool and degrading the entire system. This is a well-known issue — Python's `requests` library defaults to `timeout=None` (infinite wait), and [the library docs explicitly recommend](https://requests.readthedocs.io/en/latest/user/advanced/#timeouts) always setting a timeout. ### Type of change - [x] Bug Fix (non-breaking change which fixes an issue) ### Change Added `timeout` to all `requests.post()` calls missing it: | File | Calls fixed | Timeout | |------|-------------|---------| | `rag/llm/rerank_model.py` | 9 | 30s | | `rag/llm/embedding_model.py` | 8 | 30s | | `rag/llm/cv_model.py` | 3 | 60s | | `rag/llm/tts_model.py` | 2 | 60s | | `rag/llm/sequence2txt_model.py` | 2 | 60s | Embedding/rerank calls use 30s (lightweight API calls). Vision, TTS, and audio transcription use 60s (heavier workloads with file uploads). Note: other files in the codebase (e.g. `check_minio_alive`, `check_ragflow_server_alive`) already use `timeout=10`, so this PR brings the LLM layer in line with existing practice. Signed-off-by: Ricardo-M-L <Sibyl_Hartmanbnb@webname.com> Co-authored-by: Kevin Hu <kevinhu.sh@gmail.com> |
||
|
Paras Sondhi
|
51b73850e1 |
feat: make sandbox Dockerfile mirrors optional with ARG (#14553)
### What problem does this PR solve? Resolves #14447. *(Note: This supersedes stalled PR #14448 and implements the requested CodeRabbitAI fixes).* Currently, the Dockerfiles inside `agent/sandbox/sandbox_base_image` (both Python and Node.js) have hardcoded Chinese package mirrors. This forces the mirrors on all users globally, which causes build network timeouts for contributors outside of China. This PR introduces an enhancement to fix the issue by: 1. Implementing the `NEED_MIRROR` build argument in the sandbox Dockerfiles. 2. Replacing static `ENV` instructions with conditional shell logic inside `RUN` blocks to dynamically set the package registries. 3. Allowing the build to cleanly fall back to default global registries (`pypi.org` and `npmjs.org`) when `--build-arg NEED_MIRROR=0` is passed. ### Type of change - [x] New Feature (non-breaking change which adds functionality) - [x] Refactoring --------- Co-authored-by: Jin Hai <haijin.chn@gmail.com> |
||
|
BitToby
|
39a1773f7f |
Go: implement ListModels in Volcengine driver (#14702)
### What problem does this PR solve? The VolcEngine Go driver in `internal/entity/models/volcengine.go` shipped with a `ListModels` stub that returned `volcengine, no such method`. `conf/models/volcengine.json` also did not declare a `models` URL suffix, so the model picker had nothing to call even if the method body were filled in. A tenant who configured Volcengine (Doubao / Ark) as a provider could not see the list of available endpoints from the RAGFlow UI. Several other Go drivers already implement `ListModels` against the OpenAI-compatible `/models` endpoint (deepseek, gitee, nvidia, openai, siliconflow), so the interface and pattern are well-established. This PR fills the gap. ### What this PR includes * `conf/models/volcengine.json`: declare the `models` URL suffix alongside the existing `chat`, `files`, and `embedding` entries. The Ark v3 API exposes `https://ark.cn-beijing.volces.com/api/v3/models`, so the suffix is just `models`. * `internal/entity/models/volcengine.go`: replace the `ListModels` stub with a real implementation. Reuses the package-level `DSModelList` / `DSModel` types that DeepSeek, Gitee, and SiliconFlow already use to parse the OpenAI-compatible models response shape. No factory change. No interface change. ### How the driver works * Resolves the region with a default fallback, the same way the other VolcEngine methods in this driver already do. * Builds the URL from `BaseURL[region] + URLSuffix.Models`, with `strings.TrimSuffix` on the base to keep the join robust. * Issues a `GET` with optional `Authorization: Bearer <api_key>` (the header is omitted when no key is configured, mirroring the existing NVIDIA `ListModels`). * Reads the response body once, surfaces a non-200 with the upstream status line plus body, and parses the JSON via the shared `DSModelList` type. * Returns the model id list in input order. When the response includes an `owned_by` field, the entry is rendered as `id@owned_by`, matching the convention used by the other Go drivers. ### Type of change - [x] New Feature (non-breaking change which adds functionality) ### How was this tested? * `go build ./internal/entity/models/...` returns exit 0. * `go vet ./internal/entity/models/...` is clean. * `gofmt -l internal/entity/models/volcengine.go` is clean. * The full method set on `VolcEngine` still matches the `ModelDriver` interface. * Endpoint reachability check: `GET https://ark.cn-beijing.volces.com/api/v3/models` returns `401 Unauthorized` without an API key, confirming the path exists and accepts Bearer authentication. * Pattern parity with DeepSeek, Gitee, NVIDIA, and SiliconFlow `ListModels`. Fixes #14701 Co-authored-by: Jin Hai <haijin.chn@gmail.com> |
||
|
VincentLambert
|
08bb53bbb1 |
Feat: add BedrockCV for vision/image2text inference via LiteLLM (#14705)
## Summary - `CvModel["Bedrock"]` was absent from `rag/llm/cv_model.py`, causing `model_instance()` to return `None` when a Bedrock model was used as a PDF parser — even after correct model resolution. - This PR adds `BedrockCV`, enabling Bedrock vision models (e.g. `amazon.nova-pro-v1:0`, `anthropic.claude-3-5-sonnet`) to be used as PDF parsers. ## What problem does this PR solve? When a Bedrock model is selected as the PDF parser in a knowledge base, ingestion failed with: ``` 'LiteLLMBase' object has no attribute 'describe_with_prompt' ``` The root cause: `LiteLLMBase` (the Bedrock chat implementation) was the only registered handler for the Bedrock factory. It does not implement `describe_with_prompt`. `CvModel` had no Bedrock entry, so `model_instance()` returned `None` for `image2text` requests. ## Type of change - [x] New Feature (non-breaking change which adds functionality) ## Changes **`rag/llm/cv_model.py`** Adds `BedrockCV(Base)` with `_FACTORY_NAME = "Bedrock"`: - Uses `litellm.completion` with the `bedrock/` prefix (consistent with `LiteLLMBase`) - Parses AWS credentials from the JSON key assembled by `add_llm` (`auth_mode`, `bedrock_ak`, `bedrock_sk`, `bedrock_region`, `aws_role_arn`) - Supports three auth modes: `access_key_secret`, `iam_role` (via STS `assume_role`), and default credential chain (IRSA, instance profile) - Implements `describe_with_prompt` and `describe` ## Test plan - [ ] Configure a Bedrock vision model (e.g. `amazon.nova-pro-v1:0`) with valid AWS credentials - [ ] Select it as PDF parser in a knowledge base - [ ] Verify ingestion of a PDF document completes without errors - [ ] Verify `CvModel["Bedrock"]` resolves to `BedrockCV` 🤖 Generated with [Claude Code](https://claude.ai/claude-code) --------- Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com> |
||
|
Ahmad Intisar
|
3c4d1da98f |
Feature/table parser column roles (#13710)
### What problem does this PR solve? The table file parser (CSV/Excel) currently treats all columns identically — every column is both vectorized (embedded in chunk text) and stored as filterable metadata. There's no way for users to control which columns should be searchable by semantic meaning versus which should only be filterable attributes. For example, when ingesting a news articles CSV with columns like title, content, country, category, source, etc., the embedding includes metadata fields like country: Brazil and source: Reuters in the chunk text, which dilutes the semantic quality of the embedding without adding retrieval value. The RDBMS connector (MySQL/PostgreSQL) already supports content_columns / metadata_columns, but this capability was missing for file-based table ingestion. This PR adds column-level control (vectorize / metadata / both) for the table file parser, following RAGFlow's existing patterns. Backward compatible: Datasets without table_column_roles or with table_column_mode: auto behave exactly as before (all columns = both). ### Type of change - [x] New Feature (non-breaking change which adds functionality) |
||
|
Igor Ilinskii
|
889aba6a32 |
fix base_url handling in HuggingfaceRerank (#14555)
### What problem does this PR solve? HuggingfaceRerank.post() unconditionally prepends `http://` to base_url, which already contains a protocol. This creates invalid URLs like http://http://127.0.0.1:8080/rerank, breaking all requests. The fix normalizes URL handling to match the rest of the codebase, removing redunant `http://`. ### Type of change - [x] Bug Fix (non-breaking change which fixes an issue) ### Related Issues - #7318 - #7796 --------- Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com> |
||
|
Tim Wang
|
ed01ac9994 |
Fix: resolve template strings in tool component parameters (#14601)
## Summary
- Tool-type components (Email, Invoke, etc.) fail to resolve template
strings that mix variable references with literal text in their
parameters.
- This adds template string resolution to `get_input()` in
`ComponentBase`, reusing existing `get_input_elements_from_text()` and
`string_format()` methods.
## Problem
`get_input()` in `ComponentBase` handles two cases:
1. **Pure reference** (`{Component:ID@field}`) — resolved via
`is_reff()` + `get_variable_value()`
2. **Literal value** — passed through as-is
But template strings like `{UserFillUp:X@name}@duke.edu` or `Question
from {Agent:Y@topic}` fall through to the literal branch because
`is_reff()` returns `False` (it expects the entire string to be a single
reference). The unresolved template is passed directly to the tool.
This affects **all** tool components (Email, Invoke, etc.) that need
mixed reference + text parameters — for example, constructing email
addresses or subjects dynamically.
## Fix
```python
# In get_input(), between is_reff check and literal fallback:
elif isinstance(v, str) and re.search(self.variable_ref_patt, v):
elements = self.get_input_elements_from_text(v)
kv = {k: e.get('value', '') for k, e in elements.items()}
self.set_input_value(var, self.string_format(v, kv))
```
This reuses `get_input_elements_from_text()` and `string_format()` which
are already used by `Message` components for the same purpose. The fix
only activates when the string contains at least one variable reference
pattern but is not a pure reference.
## Test plan
- [x] Pure references (`{Component:ID@field}`) still resolve correctly
via `is_reff()` path
- [x] Literal values without references pass through unchanged
- [x] Template strings like `{ref}@duke.edu` resolve the reference and
keep the literal suffix
- [x] Template strings like `Question from {ref}` resolve correctly
- [x] Multiple references in one string (`{ref1} and {ref2}`) both
resolve
- [x] Message components unaffected (they use their own template
resolution in `_run`)
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: wanghualoong <wanghualoong@gmail.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
|
||
|
Mehmet Karakose
|
7ec87f7cb7 |
fix(auth): fall back to session-based auth in _load_user (#14569)
## Summary Closes #13663. OAuth / OIDC callbacks call `login_user(user)` which writes `_user_id` into the session cookie, but `_load_user()` in `api/apps/__init__.py` only ever looked at the `Authorization` header. The SPA's response interceptor wipes the Authorization value from `localStorage` on the first 401 it sees — meaning that during the post-redirect window after an OAuth login, a single transient 401 sends every subsequent request back to the login page even though `login_user()` had already established a perfectly good server-side session. The reporter's analysis traces this all the way through the redirect → `navigate('/')` → first request → empty header → 401 → `removeAll()` → infinite-redirect-to-login chain. ## What changed - New `_load_user_from_session()` helper that reads `session["_user_id"]`, looks up the user in `UserService` (with the same `StatusEnum.VALID` and `access_token` checks already used elsewhere), and assigns `g.user`. - Every `return None` path in `_load_user()` now routes through that helper before giving up: - missing `Authorization` header - malformed `bearer ` prefix - empty / too-short JWT payload - JWT signature failure - JWT-resolved user not found / has no `access_token` - `APIToken.query()` fallback exhausted The JWT and API-token paths still take precedence — the session is only consulted when those can't authenticate the request. So existing local-login and SDK callers see no behaviour change; only OAuth / OIDC users that hit the original race now stay logged in. The Bearer-prefix issue called out in #13663 (lines 103-110) is already handled in the current code, so this PR only addresses the second half of the report. ## Test plan - [ ] Configure OIDC under `oauth` in `service_conf.yaml` - [ ] Click the OIDC login button, complete auth at the IdP - [ ] Confirm that navigating between pages no longer bounces back to `/login` - [ ] Confirm local email/password login still issues + accepts JWTs - [ ] Confirm SDK/API key callers still authenticate via `Authorization: Bearer <api-token>` --------- Co-authored-by: Kevin Hu <kevinhu.sh@gmail.com> |
||
|
很拉风的James
|
6cb4bc2947 |
Fix: Radio.Group cloneElement crashes on non-element children (#14407)
### What problem does this PR solve? `Radio.Group` in `web/src/components/ui/radio.tsx` injects the parent's `disabled` prop into each child via `React.cloneElement` with `as React.ReactElement` and no validation. This throws at runtime when a consumer passes strings, numbers, `null`, `false`, or other non-element nodes, while the cast hides the unsafe access from TypeScript. Use `React.isValidElement<RadioProps>(child)` as a type guard before calling `cloneElement`. Non-element children pass through unchanged, and `child.props` access becomes type-checked without an `as` cast. ### Type of change - [x] Bug Fix (non-breaking change which fixes an issue) |
||
|
Panda Dev
|
6bfe0f9a10 |
Go: implement Encode (embeddings) in OpenAI driver (#14630)
### What problem does this PR solve? The OpenAI Go driver landed in #14605 with chat, list models, and check connection. Encode was left as a stub that returns \`not implemented\`. \`conf/models/openai.json\` already lists three embedding models out of the box: - text-embedding-ada-002 - text-embedding-3-small - text-embedding-3-large So a tenant who picked one of these in the Go layer could not actually run an embedding call. This PR fills the gap. ### What this PR includes - \`conf/models/openai.json\`: add \`\"embedding\": \"embeddings\"\` under \`url_suffix\` so the driver can build the URL from config. This matches the \`URLSuffix.Embedding\` field used by other drivers (siliconflow, zhipu-ai). - \`internal/entity/models/openai.go\`: replace the Encode stub with a real implementation that POSTs to \`/v1/embeddings\`. Adds a small local response type \`openaiEmbeddingResponse\`. No factory change. No interface change. ### How the implementation works - Validate \`apiConfig\` and the API key, validate the model name. Use the existing \`baseURLForRegion\` helper so an unknown region fails fast with a clear error. - Wrap the request with \`context.WithTimeout(nonStreamCallTimeout)\` so the call has a clear deadline. Same pattern as \`ChatWithMessages\` and \`ListModels\` already use in this file. - Send all input texts in one request. The OpenAI API accepts the \`input\` field as an array. - Parse \`data[*].embedding\` and copy each slice into a \`[][]float64\` indexed by \`data[*].index\` so the output order matches the input order even if the API returns items in a different order. - Handle both \`float64\` and \`float32\` element types, the way the SiliconFlow driver does. - An empty input slice returns \`[][]float64{}\` with no HTTP call. - Non-200 responses propagate the upstream status line and body. - A final pass checks that every input slot got a vector. If any slot is still nil, return a clear error so the caller does not silently use a zero vector. ### Type of change - [x] New Feature (non-breaking change which adds functionality) ### How was this tested? - \`go build ./internal/entity/models/...\` in a clean go 1.25 image (the go.mod minimum) returns exit 0. - The full method set on \`OpenAIModel\` still matches the \`ModelDriver\` interface. - Pattern parity with the existing SiliconFlow Encode implementation (\`internal/entity/models/siliconflow.go\`). Closes #14629 --------- Co-authored-by: Jin Hai <haijin.chn@gmail.com> |
||
|
Jin Hai
|
048ec2fc5c |
Go: fix siliconflow rerank issue (#14743)
### What problem does this PR solve? As title. ### Type of change - [x] Bug Fix (non-breaking change which fixes an issue) Signed-off-by: Jin Hai <haijin.chn@gmail.com> |
||
|
Jin Hai
|
779cd83862 |
Go: fix Baidu rerank issue (#14742)
### What problem does this PR solve? top_n is missing ### Type of change - [x] Bug Fix (non-breaking change which fixes an issue) Signed-off-by: Jin Hai <haijin.chn@gmail.com> |
||
|
Hunnyboy1217
|
782084780e |
feat(connectors): ETag-based bypass for incremental S3 ingestion (#14628) (#14677)
### What problem does this PR solve? S3-family connector syncs currently re-download every in-window object just so we can compute `xxhash128(blob)` and compare against `Document.content_hash`. Anything that bumps `LastModified` without changing bytes (`aws s3 cp` touches, bucket re-encryption, etc.) pays full bandwidth and re-parses files that didn't actually change. #14628 covers the broader incremental-ingestion redesign; this PR is the first slice. The fix is a pre-listing short-circuit. `BlobStorageConnector` (S3 / R2 / GCS / OCI / S3-compat) now implements a new `FingerprintConnector` interface: `list_keys()` paginates `list_objects_v2` and yields `KeyRecord(key, fingerprint)` where `fingerprint = xxhash128(ETag)`. The orchestrator joins those against the connector's existing `{doc_id: content_hash}` map and only calls `get_value(key)` when the fingerprint differs. Unchanged keys are skipped entirely — no `GetObject`, no re-parse. No DDL. xxhash128(ETag) is 32 hex chars and reuses the existing `Document.content_hash` column per @yingfeng's suggestion; the connector decides at listing time whether to populate it. Local uploads and connectors that don't opt in fall through to the existing post-download `xxhash128(blob)` path with no behavior change. This is PR-1 of a 4-PR series — full design lives on #14628. Subsequent PRs extend tier 1 to local FS / WebDAV / Dropbox / Seafile / RDBMS (PR-2), wire up tier 2 cursor connectors with `SyncLogs.next_checkpoint` (PR-3), and unify deletion via `KeyRecord(deleted=True)` reconciliation (PR-4). Holding those back keeps this PR additive and reviewable on its own. #### Files touched - `common/data_source/models.py` — new `KeyRecord`; optional `fingerprint` on `Document` - `common/data_source/interfaces.py` — `IncrementalCapability` enum, `FingerprintConnector` ABC - `common/data_source/blob_connector.py` — `BlobStorageConnector` implements `FingerprintConnector`; per-object download factored into `_build_document_from_obj()` so `_yield_blob_objects`, `list_keys`, `get_value` all share it - `rag/svr/sync_data_source.py` — `_BlobLikeBase._fingerprint_filtered_generator` does the bypass loop; `_run_task_logic` plumbs `doc.fingerprint` into the upload dict - `api/db/services/document_service.py` — `list_id_content_hash_map_by_kb_and_source_type()` helper - `api/db/services/connector_service.py` + `file_service.py` — fingerprint flows through `duplicate_and_parse → upload_document` and lands in `content_hash` - `test/unit_test/common/test_blob_connector_fingerprint.py` — 14 tests covering ETag normalization (single-part, multipart, quoted, empty), `list_keys()` not calling `GetObject`, `get_value()` materializing with fingerprint, deterministic/stable fingerprints, and the bypass loop asserting `GetObject` is *not* called on a match #### Worth flagging for review Old `_BlobLikeBase._generate` called `poll_source(start, now)` with a `LastModified` window when `poll_range_start` was set. New code uses `_fingerprint_filtered_generator` (full bucket listing + fingerprint compare) outside of explicit `reindex=1`. Strictly better for unchanged-bucket cases since it skips `GetObject`, but it does mean every sync now does a full `list_objects_v2` paginate. Should still be cheap for most buckets — flagging in case anyone has a very large bucket where the time-window filter was meaningful. On migration: existing rows have `content_hash = xxhash128(blob)` from the old code. The first sync after this lands sees ETag-derived fingerprints that don't match, re-fetches every object once, and writes the new fingerprint. From the second sync onward the bypass works as expected. "Slow day one, fast every day after." A `fingerprint_backfill: trust` opt-out is sketched in the design doc but not in this PR. #### Test plan - [x] `uv run ruff check` — clean on all 8 touched files - [x] `uv run pytest test/unit_test/common/test_blob_connector_fingerprint.py -v` — 14 passed - [x] Broader unit-test suite — no regressions in anything I touched - [ ] Manual smoke against a real S3 bucket — configure a connector, run sync twice, expect the second sync to log `bypassed=N, fetched=0` and no `GetObject` calls in CloudTrail / bucket access logs - [ ] Manual smoke with `reindex=1` — confirm the full re-download path still works ### Type of change - [x] New Feature (non-breaking change which adds functionality) --------- Co-authored-by: Yingfeng <yingfeng.zhang@gmail.com> |
||
|
Haruko386
|
7931b693dc |
Go: implement provider: Baidu (#14741)
### What problem does this PR solve?
This PR completes the Baidu Qianfan provider integration in RAGFlow.
**The following functionalities are now supported:**
- [x] Chat / Think Chat / Stream Chat / Stream Think Chat
- [x] Embedding
- [x] Rerank
- [x] Model listing
- [x] Provider connection checking
- [ ] Balance
-----
**Verified examples from the CLI:**
```plaintext
RAGFlow(user)> embed text 'what is rag' 'who are you' with 'embedding-3@test@zhipu-ai' dimension 16;
+-----------+-------+
| dimension | index |
+-----------+-------+
| 16 | 0 |
| 16 | 1 |
+-----------+-------+
RAGFlow(user)> rerank query 'what is rag' document 'rag is retrieval augment generation' 'rag need llm' 'famous rag project includes ragflow' with 'qwen3-reranker-4b@test@baidu' top 2;
+-------+---------------------+
| index | relevance_score |
+-------+---------------------+
| 0 | 0.974821150302887 |
| 1 | 0.14223189651966095 |
| 2 | 0.08632347732782364 |
+-------+---------------------+
RAGFlow(user)> think chat with 'deepseek-v3.2@test@baidu' message 'who r u'
Thinking: Hmm, the user is asking for a simple introduction. This is straightforward – no need for overcomplication.
I should give a clear, friendly response that covers my basic identity as an AI assistant, my purpose, and my capabilities. Keeping it concise but informative is key here.
Mentioning my creator Anthropic adds credibility, and ending with an offer to help invites further interaction. No need for technical details unless the user asks later.
Answer: Hello! I'm an AI assistant created by Anthropic, designed to help with a wide variety of tasks. You can think of me as a helpful digital companion—I can answer questions, assist with writing, help solve problems, provide explanations, and engage in conversation on many topics. I'm here to help with whatever you need! How can I assist you today?
Time: 8.103902
RAGFlow(user)> stream think chat with 'deepseek-v3.2@test@baidu' message 'who r u'
Thinking: mm, the user is asking "who r u" with casual spelling. This is a straightforward identity question. should give a clear, friendly introduction without overcomplicating it. Can start with my core function as an AI assistant, mention my creator, and briefly state my key capabilities. response should be welcoming and invite further interaction since this seems like an introductory question. Keeping it concise but covering the essentials: who I am, what I do, and how I can help.
Answer: ! I am DeepSeek, an AI assistant created by DeepSeek Company. I'm designed to help answer questions, provide information, assist with various tasks, and engage in conversations on a wide range of topics. I'm here to assist you with whatever you need - whether it's answering questions, helping with analysis, writing, coding, or just having a friendly chat!Is there anything specific I can help you with today? 😊
Time: 7.219703
RAGFlow(user)> list supported models from 'baidu' 'test'
+--------------------------------------+
| model_name |
+--------------------------------------+
| ernie-3.5-8k-preview |
| ernie-4.0-8k |
| ernie-4.0-turbo-8k-latest |
| ernie-4.0-turbo-8k-preview |
| ernie-4.0-8k-preview |
| ernie-speed-pro-128k |
| ernie-char-fiction-8k |
| ernie-3.5-8k |
| ernie-3.5-128k |
| ernie-lite-pro-128k |
| ernie-novel-8k |
| ernie-4.0-turbo-8k |
| ernie-4.0-turbo-128k |
| ernie-4.0-8k-latest |
| irag-1.0 |
| ........... |
| glm-5.1 |
| ernie-image-turbo |
| deepseek-v4-pro |
| deepseek-v4-flash |
| ernie-5.1 |
+--------------------------------------+
RAGFlow(user)> check instance 'test' from 'baidu'
SUCCESS
```
Additionally, this PR fixes an incorrect error message typo:
Before:
```go
fmt.Errorf("API requestssss failed with status %d: %s : %s", ...)
```
After:
```go
fmt.Errorf("API request failed with status %d: %s", ...)
```
This PR mainly improves provider compatibility, API completeness, and
runtime stability.
### Type of change
* [x] Bug Fix (non-breaking change which fixes an issue)
* [x] New Feature (non-breaking change which adds functionality)
* [x] Refactoring
|
||
|
Liu An
|
57b24be6d6 |
Docs: Update version references to v0.25.2 in READMEs and docs (#14731)
### What problem does this PR solve? - Update version tags in README files (including translations) from v0.25.1 to v0.25.2 - Modify Docker image references and documentation to reflect new version - Update version badges and image descriptions - Maintain consistency across all language variants of README files ### Type of change - [x] Documentation Update |
||
|
writinwaters
|
a3de873617 |
Docs: Updated release date (#14740)
### What problem does this PR solve? Updated v0.25.2 release date. ### Type of change - [x] Documentation Update |
||
|
euvre
|
f4b8f53b6d |
Fix: restore embedding model switching for datasets with existing chunks (#14732)
### What problem does this PR solve? ## Problem During the REST API refactoring (#13690), the `/api/v2/kb/check_embedding` endpoint was removed and never migrated to the new RESTful structure. The frontend was pointed to the `/api/v1/datasets/{id}/embedding` endpoint (which is `run_embedding` — a completely different function). Additionally, a hard guard was introduced that rejects any `embd_id` change when `chunk_num > 0`, making it impossible to switch embedding models on datasets with existing chunks. ## Root Cause 1. **Missing endpoint**: The old `check_embedding` logic (sample random chunks, re-embed with the new model, compare cosine similarity) was not carried over to the new REST API service layer. 2. **Wrong frontend URL**: `checkEmbedding` in `api.ts` pointed to `/datasets/{id}/embedding` (`run_embedding`) instead of a dedicated check endpoint. 3. **Overly restrictive guard**: `dataset_api_service.py` line 310 blocked all `embd_id` updates when `chunk_num > 0`. This check did not exist in the pre-refactor code — it was incorrectly introduced during the refactor. ## Changes ### Backend - **`api/apps/services/dataset_api_service.py`** - Remove the `chunk_num > 0` hard guard on `embd_id` updates - Add `check_embedding()` service function: samples random chunks, re-embeds them with the candidate model, computes cosine similarity, returns compatibility result (avg ≥ 0.9 = compatible) - Add `import re` for the `_clean()` helper - **`api/apps/restful_apis/dataset_api.py`** - Add `POST /datasets/<dataset_id>/embedding/check` endpoint following the new REST API conventions - Clean up unused top-level imports (`random`, `re`, `numpy`) ### Frontend - **`web/src/utils/api.ts`** - Fix `checkEmbedding` URL from `/datasets/${datasetId}/embedding` → `/datasets/${datasetId}/embedding/check` ### Tests - **`test/testcases/test_http_api/test_dataset_management/test_update_dataset.py`** - Update `test_embedding_model_with_existing_chunks` to assert success (`code == 0`) instead of expecting the old `102` error - **`test/testcases/test_web_api/test_dataset_management/test_dataset_sdk_routes_unit.py`** - Update `test_update_route_branch_matrix_unit` to assert `RetCode.SUCCESS` when updating `embd_id` on a chunked dataset, replacing the old `chunk_num` error assertion ### Type of change - [x] Bug Fix (non-breaking change which fixes an issue) --------- Signed-off-by: noob <yixiao121314@outlook.com> |
||
|
buua436
|
330257b611 |
Fix: Add legacy system healthz route (#14738)
### What problem does this PR solve? Add legacy system healthz route ### Type of change - [x] Bug Fix (non-breaking change which fixes an issue) |
||
|
Jin Hai
|
17d71e5d79 |
Go CLI: embed and rerank (#14735)
### What problem does this PR solve? ``` RAGFlow(user)> embed text 'what is rag' 'who are you' with 'embedding-3@test@zhipu-ai' dimension 16; +-----------+-------+ | dimension | index | +-----------+-------+ | 16 | 0 | | 16 | 1 | +-----------+-------+ RAGFlow(user)> rerank query 'what is rag' document 'rag is retrieval augment generation' 'rag need llm' 'famous rag project includes ragflow' with 'rerank@test@zhipu-ai' top 2; +-------+-----------------+ | index | relevance_score | +-------+-----------------+ | 0 | 1 | | 2 | 0.99999976 | +-------+-----------------+ ``` ### Type of change - [x] New Feature (non-breaking change which adds functionality) Signed-off-by: Jin Hai <haijin.chn@gmail.com> |
||
|
Lynn
|
efe6d23d61 |
Fix: handle id as keyword (#14729)
### What problem does this PR solve? Update mapping.json to treat id as a keyword. ### Type of change - [x] Bug Fix (non-breaking change which fixes an issue) |
||
|
chanx
|
8ac14b597f |
Fix: Some bugs (#14734)
### What problem does this PR solve? Fix: Some bugs - Error during batch modification of metadata in the Knowledge Base - Manually configured metadata is not displayed in search settings ### Type of change - [x] Bug Fix (non-breaking change which fixes an issue) |
||
|
akie
|
c11650bb4c |
Fix IDOR: Add permission checks to file ancestry endpoints (#14725)
Close #14292 ## Issue File ancestry endpoints return folder metadata without validating tenant permissions, allowing any authenticated user to query arbitrary `file_id` values across tenant boundaries. ## Affected Endpoints - `GET /v1/file/parent_folder?file_id={file_id}` - `GET /v1/file/all_parent_folder?file_id={file_id}` - `GET /api/v1/files/{id}/ancestors` ## Root Cause These endpoints **skip the permission check** that other file operations (Delete, Download, Move) perform. ## Expected Permission Check All file operations should follow this 3-step validation: - Check file.tenant_id - Check if user_id belongs to this tenant (via user_tenant join table) - Check KB permission type (team permission) **Code reference:** This is implemented in `checkFileTeamPermission()` and used by Delete/Download/Move, but **missing** from GetParentFolder/GetAllParentFolders. ## Reproduction ```bash # User B (tenant: BBB) accessing User A's file (tenant: AAA) curl -H "Authorization: Bearer USER_B_TOKEN" \ "http://localhost:9384/v1/file/parent_folder?file_id=AAA_FILE_123" # Result: Returns User A's folder metadata ❌ # Expected: "No authorization." ✅ Fix Pass userID from handler to service and call checkFileTeamPermission() — same as Download/Delete/Move handlers. --------- Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com> |
||
|
writinwaters
|
6465753968 |
Docs: Added v0.25.2 release notes (#14727)
### What problem does this PR solve? Added v0.25.2 release notes. ### Type of change - [x] Documentation Update |
||
|
Magicbook1108
|
f7e8c39dcc |
Fix: filter api in dataset document (#14728)
### What problem does this PR solve? Fix: filter api in dataset document ### Type of change - [x] Bug Fix (non-breaking change which fixes an issue) |
||
|
buua436
|
de2abe9ed8 |
Fix: tag parser id (#14724)
### What problem does this PR solve? tag parser id ### Type of change - [x] Bug Fix (non-breaking change which fixes an issue) |
||
|
Haruko386
|
ee0de58204 |
Go: implement provider: HuggingFace (#14722)
### What problem does this PR solve? Implement `HuggingFace` provider ### Type of change - [x] New Feature (non-breaking change which adds functionality) |
||
|
jony376
|
3b6eeabb09 |
Fix: private dataset authorization bypass in shared dataset access checks (#14645)
### Related issues Closes #14644 ### What problem does this PR solve? This PR fixes an authorization bug where datasets marked with `permission = me` could still be accessed by other members of the same tenant through APIs that relied on `KnowledgebaseService.accessible()` or `DocumentService.accessible()`. Before this change, those shared access helpers only checked tenant membership and did not enforce the dataset's permission mode. As a result, a non-owner who knew a private `dataset_id` could still reach downstream document and chunk operations even though the dataset was intended to be owner-only. This change updates the central access checks so that: - dataset owners always retain access - joined tenant members only get access when the dataset permission is `TEAM` - private datasets with `permission = me` remain inaccessible to non-owners - document-level access follows the same dataset permission rules The PR also adds regression coverage for private-vs-team dataset access behavior. ### Type of change - [x] Bug Fix (non-breaking change which fixes an issue) - [ ] New Feature (non-breaking change which adds functionality) - [ ] Documentation Update - [ ] Refactoring - [ ] Performance Improvement - [ ] Other (please describe): ### Testing - Added `test/unit_test/api/db/services/test_dataset_access_permissions.py` - Attempted to run: `python -m pytest test\\unit_test\\api\\db\\services\\test_dataset_access_permissions.py -q` - Local execution in this workspace is currently blocked during test collection because the environment is missing the `strenum` dependency --------- Signed-off-by: Jin Hai <haijin.chn@gmail.com> Co-authored-by: jony376 <jony376@gmail.com> Co-authored-by: Wang Qi <wangq8@outlook.com> Co-authored-by: d 🔹 <liusway405@gmail.com> Co-authored-by: Jin Hai <haijin.chn@gmail.com> Co-authored-by: Magicbook1108 <newyorkupperbay@gmail.com> Co-authored-by: chanx <1243304602@qq.com> Co-authored-by: sxxtony <166789813+sxxtony@users.noreply.github.com> Co-authored-by: sxxtony <sxxtony@users.noreply.github.com> Co-authored-by: Baki Burak Öğün <63836730+bakiburakogun@users.noreply.github.com> Co-authored-by: bakiburakogun <bakiburakogun@users.noreply.github.com> Co-authored-by: Panda Dev <56657208+pandadev66@users.noreply.github.com> Co-authored-by: Haruko386 <tryeverypossible@163.com> Co-authored-by: D2758695161 <13510221939@163.com> Co-authored-by: Hunter <hunter@yitong.ai> Co-authored-by: Lynn <lynn_inf@hotmail.com> Co-authored-by: buua436 <sz_buua@foxmail.com> Co-authored-by: web-dev0521 <jasonpette1783@gmail.com> Co-authored-by: Tim Wang <38489718+wanghualoong@users.noreply.github.com> Co-authored-by: wanghualoong <wanghualoong@gmail.com> Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> Co-authored-by: qinling0210 <88864212+qinling0210@users.noreply.github.com> Co-authored-by: dale053 <star05223@outlook.com> |
||
|
Ricardo-M-L
|
1046042e01 |
fix(llm): replace mutable default gen_conf={} with None + defensive copy (#14566)
### What
19 methods across `rag/llm/chat_model.py` and `rag/llm/cv_model.py`
declare `gen_conf={}` (or `gen_conf: dict = {}`) as a parameter default
and then mutate `gen_conf` in place — typically `del
gen_conf["max_tokens"]`, `gen_conf["penalty_score"] = ...`, or
`gen_conf.pop(...)` as part of provider-specific normalization.
### The two bugs in this pattern
**1. Mutable default argument (Python footgun).** Python evaluates
default values **once** at function-definition time, so the single `{}`
dict is *shared* across every caller that doesn't pass `gen_conf`. The
first such call's mutations leak into the default seen by every
subsequent call.
```python
# Before
def chat_streamly(self, system, history, gen_conf={}, **kwargs):
if "max_tokens" in gen_conf:
del gen_conf["max_tokens"] # mutates the SHARED default dict
...
```
After call N with `max_tokens` set, call N+1 that omits `gen_conf` no
longer sees `max_tokens` — even though the caller never touched it.
**2. Caller-dict pollution.** When the caller *does* pass a `gen_conf`
dict, the same in-place mutations modify the caller's dict. A reused
`gen_conf` (very common for chat-loop callers that build the config once
and pass it on every turn) silently loses `max_tokens`,
`presence_penalty`, etc. after the first round.
### The fix
In every affected method:
- Change `gen_conf={}` (or `gen_conf: dict = {}`) → `gen_conf=None`.
- Add `gen_conf = dict(gen_conf or {})` as the first statement of the
body so all subsequent mutations operate on a fresh local copy.
```python
# After
def chat_streamly(self, system, history, gen_conf=None, **kwargs):
gen_conf = dict(gen_conf or {})
if "max_tokens" in gen_conf:
del gen_conf["max_tokens"] # local copy — safe
...
```
This is byte-for-byte identical provider-side behavior for callers that
already pass a fresh `gen_conf` per call. The new `dict(...)` copy is
O(small constant) per call.
### Files changed
- `rag/llm/chat_model.py` — 17 methods
- `rag/llm/cv_model.py` — 2 methods
### Tests
Adds `test/unit_test/rag/llm/test_gen_conf_no_mutable_default.py` — an
`ast`-based regression guard that walks both modules and asserts no
parameter named `gen_conf` ever has a mutable literal (`{}` or `[]`) as
its default. The test caught **five additional `gen_conf: dict = {}`
sites** that an initial `gen_conf={}` text grep had missed (annotated
parameters with whitespace), and would fail again if the pattern is ever
reintroduced.
```
$ pytest test/unit_test/rag/llm/test_gen_conf_no_mutable_default.py -v
============================== 3 passed in 0.04s ===============================
```
`ruff check` passes on all touched files.
### Notes
- This PR is intentionally focused on **just** the `gen_conf` default +
copy fix. There's a related (but separate) `history.insert(0, ...)`
pattern in the same files that mutates the caller's history list in 12
places — left for a follow-up so this PR stays mechanical and easy to
review.
### Latest revision (`700bb54a7`) — addresses CodeRabbit review
- Type annotation: `gen_conf: dict = None` → `gen_conf: dict | None =
None` (5 occurrences in `chat_model.py`). The old annotation was a
static-checker mismatch since `None` isn't a `dict`.
- Regression test: the AST check accessed `default.keys` directly.
`ast.List` has no `.keys` attribute — a future `gen_conf=[]` would crash
with `AttributeError` instead of being caught. Use `getattr` for both
`.keys` (Dict) and `.elts` (List). Manually verified the updated check
correctly catches both `gen_conf={}` and `gen_conf=[]` while ignoring
`gen_conf=None` and non-empty literals.
---------
Co-authored-by: Ricardo <ricardo@example.com>
|
||
|
Wang Qi
|
42504fa18c |
Bugfix: keep document api backward compatible (#14726)
### What problem does this PR solve? Bugfix: keep document api backward compatible Fix 1: https://github.com/infiniflow/ragflow/issues/14634 ### Type of change - [x] Bug Fix (non-breaking change which fixes an issue) |
||
|
Yingfeng
|
3234a0ef35 |
Update README (#14723)
### Type of change - [x] Documentation Update |
||
|
VincentLambert
|
4f3711d37f |
fix: handle missing 'total' key causing KeyError in deep research retrieval (#13942)
## Summary
- When KB retrieval fails (e.g. ES `AssertionError` on empty
`index_names`), `kbinfos` falls back to a dict without a `total` key
- `_async_update_chunk_info` then iterates over `chunk_info.keys()`
(which includes `total`) and tries `kbinfos['total']`, raising a
`KeyError`
- This error surfaces when using Tavily web retrieval in a chat with no
knowledge base attached
## Changes
- Add `'total': 0` to all default `kbinfos` dicts in
`_retrieve_information`
- Add `setdefault('total', 0)` guard after successful KB retrieval to
handle cases where the retrieval result omits the key
- Accumulate `total` correctly in the merge branch of
`_async_update_chunk_info`
## Test plan
- [ ] Start a chat with Tavily configured and no knowledge base
- [ ] Verify no `KeyError: 'total'` is raised
- [ ] Verify Tavily results are returned correctly
---------
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
|
||
|
VincentLambert
|
870bc59365 |
Fix: Bedrock api_key overridden by existing-key fallback in add_llm (#14707)
## Summary
- Adding a Bedrock model from the frontend fails with `Fail to access
model(Bedrock/<model>).Expecting value: line 1 column 1 (char 0)`.
- The assembled Bedrock JSON credentials are silently replaced by `"x"`
before the connection test, causing `json.loads("x")` to raise a
`JSONDecodeError`.
## What problem does this PR solve?
Commit `050113482` introduced a fallback in `add_llm()` that reuses the
existing DB key when `req.get("api_key") is None`:
```python
if req.get("api_key") is None:
api_key = existing_api_key if existing_api_key is not None else "x"
```
For Bedrock, credentials are sent as separate fields (`auth_mode`,
`bedrock_ak`, `bedrock_sk`, `bedrock_region`, `aws_role_arn`) — the
frontend does not send an `api_key` field. The function correctly
assembles the JSON key:
```python
api_key = apikey_json(["auth_mode", "bedrock_ak", "bedrock_sk", "bedrock_region", "aws_role_arn"])
```
But since `req.get("api_key")` is `None`, the override immediately
replaces `api_key` with `"x"` (or a stale DB value). `LiteLLMBase` then
calls `json.loads("x")` for Bedrock auth → `JSONDecodeError`.
## Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
## Changes
**`api/apps/llm_app.py`**
Write the assembled key into `req["api_key"]` so the `None` check
evaluates to `False` and the override is skipped — consistent with how
`Tencent Cloud` is already handled.
```python
# Before
api_key = apikey_json(["auth_mode", "bedrock_ak", "bedrock_sk", "bedrock_region", "aws_role_arn"])
# After
req["api_key"] = apikey_json(["auth_mode", "bedrock_ak", "bedrock_sk", "bedrock_region", "aws_role_arn"])
api_key = req["api_key"]
```
## Test plan
- [ ] Configure a Bedrock provider in Model Providers with valid AWS
credentials
- [ ] Add a Bedrock chat model — verify no `Expecting value` error
- [ ] Update the same model — verify the existing key is reused
correctly when credentials fields are left empty
🤖 Generated with [Claude Code](https://claude.ai/claude-code)
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
|