### Summary
Add object as a begin-node parameter type with JSON editor UI, webhook
schema support, and backend parsing in UserFillUp.
Co-authored-by: Cursor <cursoragent@cursor.com>
### Summary
The Go backend never seeded the `canvas_template` table, so the "Create
agent from template" page was blank when the frontend proxies to the Go
API (`API_PROXY_SCHEME=go`). This PR adds `SeedCanvasTemplates()` in
`internal/dao`, invoked from `InitDB()` after migrations, which loads
`agent/templates/*.json` and mirrors Python's `add_graph_templates()`
behavior.
Changes:
- Add `internal/dao/canvas_template_seed.go` to parse and upsert
built-in templates.
- Call `SeedCanvasTemplates()` in `InitDB()`.
- Add `CanvasTypes` (`JSONSlice`) to `entity.CanvasTemplate` so the
frontend can filter/group by category.
- Skip seeding gracefully when the templates directory is absent.
This fixes the blank template catalogue in Go mode.
## Summary
Adds the missing input form metadata for the Go BGPT canvas component.
## Root Cause
The standalone BGPT component was registered in Go, but it did not
implement GetInputForm(). During component trial run, the backend asks
the component for its input_form. Since BGPT had none, the API returned:
component has no input_form: BGPT:<node_id>
Python BGPT already exposes the query input form, so the Go component
needed the same contract.
## Change
Added GetInputForm() to the Go BGPT component with a single query line
input.
Added test coverage to ensure BGPT exposes the input form.
## Validation
Backend:
bash build.sh --test -run TestBGPT ./internal/agent/component
<img width="1369" height="1184" alt="image"
src="https://github.com/user-attachments/assets/f99e4a81-2359-42e5-80bb-dcc4e6a63fea"
/>
<img width="1736" height="1152" alt="image"
src="https://github.com/user-attachments/assets/c11240a5-2c42-4d08-88e3-c6dfbf49eedb"
/>
## Summary
Fixes a page crash when opening the BGPT node configuration in the
canvas.
## Root Cause
BGPT was using the tool-form watcher call pattern in a normal canvas
component form.
Tool forms use:
useWatchFormChange(form)
Canvas component forms use:
useWatchFormChange(node?.id, form)
Tool is not equal to component. The BGPT canvas component imported the
component-level hook but called it like a tool-form hook, so the form
argument became undefined and React Hook Form tried to read control from
a null context.
## Change
Updated the BGPT canvas form to pass the node id and form instance
correctly.
## Validation
Ran ESLint for the changed file:
npx eslint src/pages/agent/form/bgpt-form/index.tsx
<img width="1369" height="1184" alt="image"
src="https://github.com/user-attachments/assets/a40c5202-7394-4f26-9da2-08329dcc7fbf"
/>
## Summary
- Add language-aware Snowball stemmer to `RagTokenizer` supporting 16
languages (Dutch, German, French, Spanish, etc.)
- Thread the KB `language` parameter through the full tokenization
pipeline (14 parser modules + task executor)
- Add Dutch to the frontend language lists and cross-language form
## Problem
RAGFlow uses the English Porter stemmer + WordNet lemmatizer for **all**
BM25 tokenization, regardless of the knowledge base language setting.
This produces incorrect stems for non-English text. For example:
| Dutch word | Dutch stemmer | English Porter |
|---|---|---|
| documenten | document | documenten (unchanged!) |
| gebruikers | gebruiker | gebruik (over-stemmed) |
| instellingen | instell | instellingen (unchanged!) |
This degrades BM25 recall for any non-English knowledge base.
## Solution
NLTK already ships Snowball stemmers for 16 languages. This PR:
1. **`rag/nlp/rag_tokenizer.py`**: Overrides `tokenize()` with
`set_language()` and `_normalize_token()` that selects the correct NLTK
Snowball stemmer. Falls back to Porter for unmapped languages (Chinese,
Japanese, Korean, etc. — these use character-based tokenization anyway).
2. **`rag/nlp/__init__.py`** + **14 `rag/app/*.py` parsers** +
**`rag/svr/task_executor.py`**: Threads the `language` parameter through
`tokenize()`, `tokenize_chunks()`, `tokenize_table()`, and all callers.
3. **Frontend**: Adds Dutch (`Nederlands`) to `LanguageList`,
`LanguageMap`, `LanguageAbbreviationMap`, `LanguageTranslationMap`,
cross-language form field, and `en.ts` locale.
## Backward Compatibility
- Default language is `"English"`, preserving existing behavior for all
current users
- Languages without a Snowball stemmer mapping fall back to Porter (no
change)
- No new dependencies — NLTK Snowball is already bundled
### Motivation
This PR evolves the harness from a pure execution runtime into an
**observable, replayable agent evaluation platform**. The current
`harness/graph` checkpoint mechanism is insufficient for true
event-sourced introspection—we need append-only event logs capturing
every tool call, state transition, memory write, and approval decision,
enabling deterministic replay, fork/diff, postmortem analysis, and
time-travel debugging.
### Key Design Goals
1. **Event-Sourced Execution Model**
Replace coarse checkpoints with granular, append-only event logs. Every
operation becomes a durable event: tool invocation, state mutation,
memory update, human approval. This unlocks deterministic replay,
branching execution histories, and regression datasets derived directly
from production failures.
2. **First-Class Replay & Evaluation Loop**
Replay is not an afterthought—it is a core primitive. A single live run
seeds an offline corpus that supports: repeated playback, model
substitution, tool result mocking, and strategy comparison. The harness
graduates from "executor" to "continuous evaluation platform" where
failed production traces convert directly into offline regression
suites.
3. **Operational Observability**
Beyond raw traces, expose metrics that prove stability over time:
- Tool success / failure rates
- Approval latency distributions
- Retry frequencies
- Checkpoint restore reliability
- Memory retrieval quality
- Cost per completed task
- Fork replay pass rates
The underlying thesis: the bottleneck for most agent systems is not
execution capability, but the inability to **demonstrate continuous,
measurable improvement**.
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
## Summary
Closes#15483.
Default workflow/session agent completions to non-streaming when
`stream` is omitted.
## Changes
- `api/apps/restful_apis/agent_api.py`: `req.get("stream", False)` on
workflow paths.
## Test plan
- [ ] POST workflow completion without `stream`; assert JSON response.
Ports remaining Go parser wiring and PDF backends, adds tenant-aware VLM
dispatch, aligns post-processing with Python, and adds end-to-end
pipeline coverage with a generated six-page PDF.
### Summary
Keep `data` as the uploaded document array when dataset document upload
partially succeeds.
This matches the Python API behavior and allows parse-on-creation to run
for successfully uploaded files when other files in the same folder are
unsupported.
## Summary
- Add Go dynamic input form support for ExeSQL and Browser components.
- Align their input form metadata with the Python implementation.
- Add regression tests for `/components/:component_id/input-form`.
## Summary
Debugging YahooFinance component in agent canvas returns "unknown
component" and "no input_form".
YahooFinance was only registered as an eino tool, not as a runtime
component. The component factory only searches the runtime registry.
- `universe_a_wrappers.go`: add `yahooFinanceComponent` wrapper
delegating to `agenttool.YahooFinanceTool` with `GetInputForm()`
- `fixture_stubs.go`: register `"YahooFinance"` component
## TEST
`go build` and `go test ./internal/agent/component/...` all pass.
## What this fixes
Closes#16400.
`get_data_openai()` currently returns `created: null` when callers do
not pass a timestamp, and it replaces explicit timestamp values with the
current time. This makes non-streaming OpenAI-compatible responses
inconsistent with the expected integer `created` timestamp field.
## Change
- Preserve explicit `created` values when provided.
- Default non-streaming responses to `int(time.time())` when `created`
is not provided.
- Add focused unit coverage for default timestamps, explicit timestamps,
and unchanged streaming chunk shape.
## Verification
- `./.venv/bin/python -m pytest
test/unit_test/api/utils/test_api_utils.py -q`
- `python3 -m py_compile api/utils/api_utils.py
test/unit_test/api/utils/test_api_utils.py`
- `uvx ruff check api/utils/api_utils.py
test/unit_test/api/utils/test_api_utils.py`
---------
Co-authored-by: Harsh Kashyap <harshkashyap@Harshs-MacBook-Pro.local>
### What problem does this PR solve?
The Go chunk pipeline's `PostprocessOperator` `filter` stage
(`internal/ingestion/chunk/postprocess.go`) only filtered by length
(`min_length`/`max_length`). It could not drop empty/whitespace-only
chunks or duplicate chunks — both standard RAG post-processing steps
(blank chunks shouldn't be indexed; identical chunks waste embedding
compute and add redundant retrieval results).
This adds two optional, default-off booleans to the `filter` config:
- `drop_empty` — drop chunks whose content is empty or whitespace-only.
- `drop_duplicates` — drop chunks whose exact content already appeared
(order-preserving; the first occurrence is kept).
They compose with the existing length bounds and are reflected in
`String()` for plan explainability. Also adds the first unit tests for
the postprocess filter (length bounds, drop_empty, drop_duplicates,
combined, exact-content matching, and config parsing).
Validation: `gofmt` clean, `go vet ./internal/ingestion/chunk/` clean,
`go build` ok, `go test ./internal/ingestion/chunk/` — all tests pass.
Closes#16048
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
Co-authored-by: Ling Qin <qinling0210@163.com>
### What problem does this PR solve?
The Go ingestion chunk pipeline's `SplitOperator`
(`internal/ingestion/chunk/split.go`) supported only `sentence`, `char`,
and `paragraph` strategies, but not **fixed-size (length) chunking with
overlap** — the canonical RAG strategy for bounding chunk length while
preserving cross-boundary context.
This adds a `length` strategy alongside the existing ones, configurable
via DSL `params`:
- `chunk_size` — target window size in **runes** (rune-aware:
multi-byte/CJK text is windowed by character, never split mid-rune).
- `overlap` — runes carried from the end of each window into the next.
The window advances by `chunk_size - overlap`. `chunk_size` falls back
to a default (256) when unset/non-positive, and `overlap` is clamped to
`[0, chunk_size-1]` so the window always advances and the operation
terminates. Implementation follows the existing
`splitByChar`/`splitByParagraph` pattern and reuses `DetectLanguage` for
chunk metadata.
It also adds `split_test.go` — the first unit tests for the `chunk`
package — covering basic windowing, overlap, overlap
clamping/termination, rune-awareness (CJK), default sizing, no-overlap
reconstruction, empty input, and DSL param parsing.
Validation: `gofmt` clean, `go vet ./internal/ingestion/chunk/` clean,
`go build` ok, `go test ./internal/ingestion/chunk/` — all tests pass.
Closes#16046
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
Co-authored-by: bittoby <218712309+bittoby@users.noreply.github.com>
## Summary
- derive Go Agent debug input forms from prompt variable references
instead of Agent meta fields
- seed `sys.*` debug params into `CanvasState.Sys` so single-component
debug resolves prompt variables like Python
- restore Agent test-run parity for form rendering and debug execution
## Tests
- `go test ./internal/agent/component -run
'TestAgent_(GetInputForm_UsesPromptReferences|GetInputForm_DeduplicatesPromptReferences|Meta_DefaultsToEmpty|Reset_NoTools)$'`
- `go test ./internal/handler -run
'Test(DebugComponent_SeedsSysInputsIntoCanvasState|DebugComponent_HappyPath_Begin|GetComponentInputForm_HappyPath)$'`
AFTER:
<img width="669" height="456" alt="image"
src="https://github.com/user-attachments/assets/4fd86559-aafc-4027-91ae-6e666137ee1b"
/>
## Related issues
Closes#15375
### What problem does this PR solve?
`GET /api/v1/messages` and `GET /api/v1/messages/search` accepted
unbounded `limit` / `top_n` query parameters while other REST list
endpoints enforce `REST_API_MAX_PAGE_SIZE` (100) via
`validate_rest_api_page_size()`. Oversized values can trigger expensive
memory index queries and large result sets (DoS risk).
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
### Changes
| File | Change |
|------|--------|
| `api/apps/restful_apis/memory_api.py` | Cap `limit` and `top_n` with
`validate_rest_api_page_size`; return argument error when exceeded |
|
`test/testcases/test_web_api/test_message_app/test_message_routes_unit.py`
| Regression tests for oversized `limit` / `top_n` |
### Test plan
- [x] Unit tests added
- [ ] `pytest
test/testcases/test_web_api/test_message_app/test_message_routes_unit.py`
Co-authored-by: Cursor <cursoragent@cursor.com>
Follow-up from #13896Fixes#13840
### What problem does this PR solve?
In #13896, only the docker-compose-base.yml was adjusted. However, in
the Helm chart, the unmaintained minio/minio image is still referenced.
This PR syncs the Helm chart with the docker compose setup again.
I also added a line to AGENTS.md, so agents should know to do this
automatically in the future.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
- [ ] New Feature (non-breaking change which adds functionality)
- [x] Documentation Update
- [ ] Refactoring
- [ ] Performance Improvement
- [ ] Other (please describe):
Co-authored-by: Zhichang Yu <yuzhichang@gmail.com>
## Summary
Add **per-node thinking mode control** for LLM components in RAGFlow
Agent canvas, supporting Qwen3/Qwen3.6/B200M thinking-capable models.
Users can now independently configure thinking mode
(thinking/non-thinking) for each LLM node via the existing UI dropdown.
## Motivation
When Qwen3.6-27B (and other thinking-capable models like Qwen3-32B,
B200M) are used in RAGFlow Agent nodes, different nodes need different
thinking behavior:
- **Reasoning nodes** (complex analysis, math, coding): thinking mode ON
- **Simple nodes** (direct Q&A, intent classification): thinking mode
OFF
The web UI already has a `thinking` dropdown (default/enabled/disabled)
in LLM settings, and the LLM backend `_apply_model_family_policies()`
already supports `enable_thinking`. **The missing link was `gen_conf()`
not forwarding the parameter.**
This 3-line fix completes the chain.
## Changes
**`agent/component/llm.py`** — `LLMParam.gen_conf()`:
```python
if hasattr(self, "thinking") and self.thinking and self.thinking != "default":
conf["thinking"] = self.thinking
```
## End-to-end flow
```
UI dropdown: default / enabled / disabled
→ DSL: {"thinking": "enabled"}
→ LLMParam.thinking = "enabled"
→ gen_conf() returns {"thinking": "enabled"}
→ _apply_model_family_policies()
→ extra_body {enable_thinking: true}
→ Model API call with thinking ON
```
## Backward compatibility
- **Fully backward compatible** — only 3 lines added, nothing changed
- When `thinking` is "default" or not set, existing behavior is
preserved
- Qwen3 models default to `enable_thinking: false` (non-thinking),
unchanged
## Related issues
- Closes#16321 (thinking content leaks in non-streaming agent API
responses)
- Closes#13957 (how to view model reasoning process in agent API)
## Testing
- Verified in
`test/unit_test/rag/llm/test_chat_model_thinking_policy.py` that
thinking policy already tested for Qwen3 models
- The 3-line change passes through existing tested code path
(`_apply_model_family_policies`)
Co-authored-by: Hermes Agent <hermes-agent@agent.local>
## Summary
Adds ownership/access checks before updating or deleting documents,
setting document metadata, and reading file contents from storage. Also
adds tests for authorized and unauthorized access paths.
## Summary
Fix Elasticsearch-backed skill search by mapping skill search fields to
their indexed token fields.
`name`, `tags`, `description`, and `content` are stored for display but
are not searchable in the skill ES mapping. Search queries now target
`name_tks`, `tags_tks`, `description_tks`, and `content_tks`.
## Testing
- Ran Go unit tests:
```bash
/usr/local/go/bin/go test -count=1 ./internal/engine/elasticsearch
```
- Frontend verification:
1. Open /files/skills.
2. Enter a skill space.
3. Reindex the skill space if existing skills were created before this
fix.
4. Search by skill name or description keyword.
5. Confirm matching skills are returned.
### Summary
This PR fixes two issues that prevented the Agent component's
single-component debug/test run from working under the Go backend:
1. **Dynamic input_form generation**: Some components (e.g. `Agent`) do
not store a static `input_form` in the DSL. The Go handler now falls
back to the runtime component's `GetInputForm()` method, matching
Python's `Canvas.get_component_input_form` behavior. This resolves the
frontend 102 error: `component has no input_form`.
2. **Tenant ID injection for debug**: Single-component debug runs use a
fresh `CanvasState` that previously lacked `tenant_id`.
`AgentComponent.Invoke` resolves LLM credentials via the tenant tables,
so the debug run failed with `api key is required`. The handler now
seeds `state.Sys["tenant_id"]` with the authenticated user's ID,
mirroring Python's `@add_tenant_id_to_kwargs` decorator.
### Changes
- `internal/handler/agent_component.go`:
- Added `componentInputForm` helper that first reads the static
`input_form` and, if missing, instantiates the component and calls
`GetInputForm()`.
- In `DebugComponent`, set `debugState.Sys["tenant_id"] = user.ID`
before invoking the component.
### Summary
The Go backend Agent component was not returning artifacts produced by
the CodeExec tool. While the Python agent collects the "`_ARTIFACTS`"
envelope from tool responses and appends artifact markdown to the final
content, the Go agent only returned the assistant text, so generated
images were missing from the chat output.
### Changes
- Wire `react.WithMessageFuture()` in `runEinoReActAgent` and store the
resulting `MessageFuture` in the invocation context.
- After the ReAct loop finishes, drain the future and extract
``_ARTIFACTS`` entries from every tool response message.
- Support reading the tool payload from both `msg.Content` and
`msg.UserInputMultiContent` to match eino's tool contract.
- De-duplicate artifacts by URL and render images as `!` and other files
as download links.
- Add `agent_artifact_test.go` with a regression test that simulates a
CodeExec-style tool response carrying an image artifact and verifies it
is collected and formatted.
### Verification
- `go test ./internal/agent/component/... -run
TestAgent_ReActAgent_CollectsArtifactsFromCodeExecTool` passes.
- `go test ./internal/agent/component/... -count=1` compiles; the only
failure is an unrelated DNS-pinning timeout test
(`TestInvoke_ProxyDNSPin`).
- `gofmt` clean for modified files.
### Related
Fixes the behavior shown in the screenshot where the Go agent ignored
the CodeExec-generated PNG artifact.
## Problem
`raptor.py` computes `n_neighbors = int((len(embeddings) - 1) ** 0.8)`
and
passes it to `umap.UMAP(...)`. In a dataset-scope RAPTOR build the first
layer's `embeddings` is the entire KB's chunk set, so this is
effectively
unbounded: ~93k chunks → n_neighbors ≈ 9,446.
UMAP's k-NN graph is `N × n_neighbors`; at these values the raw neighbor
arrays alone are ~14 GB (93k × 9446 × 16 B), and the symmetrized fuzzy
simplicial set + spectral init push peak well past 30 GB. The task
executor is OOM-killed inside `fit_transform` before any clustering runs
—
the log shows "Task has been received" with no "Cluster one layer" line
—
after which the unacked task re-queues and OOMs again in a loop.
The line above already flags this: `# Degrade too much ??`.
## Fix
Cap `n_neighbors` at 100. UMAP's neighborhood size has strongly
diminishing returns well below this (default 15; a few dozen already
captures global structure), so the ceiling preserves — likely improves —
cluster quality while bounding memory to O(N). Mirrors the existing
`n_components=min(12, len(embeddings) - 2)` clamp two lines down.
```diff
- n_neighbors = int((len(embeddings) - 1) ** 0.8)
+ n_neighbors = min(int((len(embeddings) - 1) ** 0.8), 100)
```
## Repro
Dataset-scope RAPTOR over a KB with ~90k+ chunks on a box with <~64 GB
available: executor OOM-killed in the first-layer UMAP `fit_transform`.
With the cap, first-layer UMAP peaks in low single-digit GB and the
build
proceeds to completion.
## Scope
Only affects large dataset-scope builds; file-scope RAPTOR already had
n_neighbors well under 100. No behavior change beyond the ceiling.
### Summary
1. Move common functions to format.go
2. modify show name spaces to _
3. move _order _columns column sort group;
4. add dao empty enterprise file
## Summary
While fixing #16467 (IterationItem crash on `@` in user-defined output
keys), an audit of `agent/**/*.py` revealed **three additional sites**
with the same vulnerability. This PR hardens all of them with
`maxsplit=1` and adds regression tests.
This is **defense-in-depth hardening**, not a behavior change. The
current `variable_ref_patt` regex constrains `var_nm` to
`[A-Za-z0-9_.-]+`, so single-`@` templates resolve exactly as before.
The `maxsplit=1` only kicks in if the trailing side itself contains `@`
— currently unreachable from the public DSL surface, but trivially
exploitable the moment a user-defined output key happens to contain `@`
(e.g. `user@email`) or the regex is ever relaxed.
> **Note on issue scope**: The primary fix for #16467 (the
`list_tenant_added_models` `ValueError` crash on `@` in model names) is
in PR #16468. This PR is a **follow-up hardening sweep** of the same
vulnerability class found in `agent/` during that audit; it does not
duplicate or replace #16468.
## Sites hardened
| File | Line | Method |
|------|------|--------|
| `agent/canvas.py` | 206 | `Graph.get_variable_value` |
| `agent/canvas.py` | 256 | `Graph.set_variable_value` |
| `agent/component/base.py` | 533 |
`ComponentBase.get_input_elements_from_text` |
| `agent/component/iterationitem.py` | 88 |
`IterationItem.output_collation` |
All now use `split("@", 1)` with an inline comment explaining the
rationale. The trailing side keeps any embedded `@`.
## Sites already safe (audited but left alone)
| File | Reason safe |
|------|------------|
| `agent/canvas.py:708` (`is_reff`) | Pre-checks `len(arr) != 2` |
| `agent/component/categorize.py` | Uses `rsplit` |
| `agent/component/iteration.py` | Pre-validates via regex |
| Other call sites | `rsplit` or regex pre-validation |
## Regression tests
9 new tests across 2 files, all `pytest.mark.p2`:
| File | Tests |
|------|-------|
| `test/unit_test/agent/test_canvas_at_split.py` | 6 —
`get_variable_value`, `set_variable_value`, round-trip, single-`@`,
missing-component |
| `test/unit_test/agent/component/test_iterationitem_at_split.py` | 3 —
`output_collation` with `@` in var, single-`@`, non-matching cid |
Each test was **verified to fail with `ValueError: too many values to
unpack (expected 2)`** when the corresponding fix is temporarily
reverted, confirming the tests actually catch the bug rather than just
exercising the happy path.
## Test results
```
9 passed in 0.04s
```
Full agent unit suite also clean (38 passed, 3 skipped; 6 unrelated
pre-existing collection errors from missing `peewee`/`requests` in local
venv — not caused by this PR).
## Related
- Issue: #16467
- Primary fix PR: #16468 (closes the issue)
- This PR: defense-in-depth follow-up, intentionally non-blocking on
#16467
---------
Co-authored-by: skbs-eng <skbs-eng@users.noreply.github.com>
### What problem does this PR solve?
Closes#16414.
The **Crawler** agent tool (`agent/tools/crawler.py`) was never ported
to the modern `ToolBase`/`_invoke` interface during the agent module
redesign, so it was broken in three independent ways:
1. **Crashed on construction.** `CrawlerParam` extends `ToolParamBase`,
whose `__init__` reads `self.meta["parameters"]`, but `CrawlerParam`
defined no `meta`. Constructing it raised `AttributeError:
'CrawlerParam' object has no attribute 'meta'`. Because
`agent/canvas.py` instantiates `component_class(component_name +
"Param")()` while loading a canvas, **any agent containing a Crawler
node failed to load.**
2. **`_invoke` missing.** It extends `ToolBase` (whose `invoke()`
dispatches to `self._invoke`) but only implemented the legacy `_run`, so
`_invoke` resolved to `ComponentBase._invoke` → `NotImplementedError`.
3. **`be_output` removed.** `_run` called `Crawler.be_output(...)`,
which no longer exists on the base classes.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
### Changes
- Add a `ToolMeta` to `CrawlerParam` (defined before
`super().__init__()`, matching every other ported tool such as
`ArXivParam`/`TavilyExtractParam`) advertising a required `query`
parameter — the URL to crawl, default `{sys.query}`, consistent with the
`{sys.query}` convention shared by the other tools.
- Replace the legacy `_run`/`be_output` with `_invoke`/`set_output`,
writing the extracted page content to `formalized_content` (errors
surfaced via `_ERROR`), consistent with the other tools.
- Preserve the existing SSRF guard (`assert_url_is_safe` +
`pin_dns_global`).
- Add regression tests
(`test/unit_test/agent/component/test_crawler.py`) covering param
construction, validation, and the tool descriptor.
Same class of defect as #16329 (DeepL). Backend-only; no frontend
changes.
---------
Co-authored-by: Zhichang Yu <yuzhichang@gmail.com>
## Summary
Use `DocumentService.RemoveDocumentKeepFile` when deleting files that
are linked to documents.
## Change
- inject `DocumentService` into `FileService`
- replace direct document deletion in `deleteSingleFile`
- remove the obsolete file-local engine deletion helper
## Result
Deleting a file now cleans up linked documents through the same service
path used elsewhere, keeping KB counters and document engine cleanup
consistent.
### What problem does this PR solve?
Closes#16416.
The **AkShare** agent tool (`agent/tools/akshare.py`) was never ported
to the modern `ToolBase`/`_invoke` interface during the agent module
redesign and was still written against the removed legacy
`_run`/`be_output` API, so it was non-functional:
1. **Adding it to an Agent raised `AttributeError`.** `AkShare` extended
`ComponentBase` (not `ToolBase`) and `AkShareParam` defined no `meta`,
so it had no `get_meta()`. `agent/component/agent_with_tools.py` builds
each tool's function descriptor via `cpn.get_meta()`, so constructing an
Agent that includes the AkShare tool raised `AttributeError: 'AkShare'
object has no attribute 'get_meta'`.
2. **It could never run.** `invoke()` dispatches to `self._invoke`, but
`AkShare` only implemented the legacy `_run`, so `_invoke` fell through
to `ComponentBase._invoke` → `NotImplementedError`. `_run` also called
`be_output(...)`, which no longer exists on the base classes.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
### Changes
- Port `AkShareParam` to `ToolParamBase` with a `ToolMeta` (defined
before `super().__init__()`, matching `ArXivParam`/`TavilyExtractParam`)
exposing a required `query` parameter — the stock symbol to look up,
default `{sys.query}`. `query` matches the `{sys.query}` convention
shared by the other tools.
- Rewrite the component with `_invoke`/`set_output("formalized_content",
...)` (errors surfaced via `_ERROR`), keeping `top_n` and importing
`akshare` lazily.
- Add regression tests
(`test/unit_test/agent/component/test_akshare.py`) covering param
construction, validation, and the tool descriptor.
Same class of defect as #16329 (DeepL) and #16414 (Crawler).
Backend-only; no frontend changes.
---------
Co-authored-by: Zhichang Yu <yuzhichang@gmail.com>
### Summary
Plan to start api_server, admin_server and ingestor in one binary:
- ./ragflow_main --admin
- ./ragflow_main --api
- ./ragflow_main --ingestor
---------
Signed-off-by: Jin Hai <haijin.chn@gmail.com>
## Summary
- Decrement document and knowledgebase chunk counts after chunks are
deleted
- Keep token counts unchanged because deleted chunk token totals are not
available
- Add tests for stats update, zero-delete behavior, error handling, and
transaction rollback
### Summary
1. env 'MINIO_PORT' is used for MINIO external access, which shouldn't
be used in Go config.
2. After RAGFlow 1.0 release, MINIO_PORT will be used for docker compose
internal usage. new ENV MINIO_EXTERNAL_PORT will be used for external
access.
Signed-off-by: Jin Hai <haijin.chn@gmail.com>
### Summary
Per discussion with @yuzhichang , disable agent test firstly.
https://github.com/infiniflow/ragflow/actions/runs/28562749273/job/84704079689?pr=16521
[0.094ms] [rows:0] SELECT * FROM `tenant_model_instance` WHERE
provider_id = "provider-1" AND instance_name = "default" ORDER BY
`tenant_model_instance`.`id` LIMIT 1
--- FAIL: TestInvoke_ProxyDNSPin (2.00s)
invoke_test.go:375: dial error = Invoke: do: Get "http://8.8.8.8/api":
context deadline exceeded; want pinned proxy IP 192.88.99.1:9999
(connection-refused is acceptable; an absent IP means the dialer fell
through to the default resolver and the pinning regression went
undetected)
2026/07/02 14:34:58
/home/infiniflow/runners_work/tower01-9dd627fd9c44/ragflow/ragflow/internal/dao/kb.go:79
record not found
[0.358ms] [rows:0] SELECT * FROM `knowledgebase` WHERE id = "da1" AND
status = "1" ORDER BY `knowledgebase`.`id` LIMIT 1
2026/07/02 14:34:58
/home/infiniflow/runners_work/tower01-9dd627fd9c44/ragflow/ragflow/internal/dao/kb.go:79
record not found
[0.283ms] [rows:0] SELECT * FROM `knowledgebase` WHERE id = "da1" AND
status = "1" ORDER BY `knowledgebase`.`id` LIMIT 1
2026/07/02 14:34:58
/home/infiniflow/runners_work/tower01-9dd627fd9c44/ragflow/ragflow/internal/dao/kb.go:79
record not found
[0.523ms] [rows:0] SELECT * FROM `knowledgebase` WHERE id = "da1" AND
status = "1" ORDER BY `knowledgebase`.`id` LIMIT 1
2026/07/02 14:34:58 ExpectPing will have no effect as monitoring pings
is disabled. Use MonitorPingsOption to enable.
FAIL
FAIL ragflow/internal/agent/component 2.759s
ok ragflow/internal/agent/component/io 0.026s
### What problem does this PR solve?
Part of #15240 (rewriting the RAGFlow API server in Go).
Implements the two public bot endpoints from
`api/apps/restful_apis/bot_api.py`:
- **`GET /api/v1/chatbots/<dialog_id>/info`** (`chatbots_inputs`) —
returns `{title, avatar, prologue, has_tavily_key}` for a dialog the
authenticated tenant owns (tenant match + `status == VALID`), otherwise
`"Authentication error: no access to this chatbot!"`.
- **`GET /api/v1/searchbots/detail`** (`detail_share_embedded`) —
returns search-app detail for a `search_id` the tenant can access.
Permission is checked across the tenant's joined tenants; denial returns
`"Has no permission for this operation."` (operating error, `data:
false`) and a missing app returns `"Can't find this Search App!"`.
Both endpoints authenticate with an SDK **beta token** (`Authorization:
Bearer <beta>`) rather than a session — the token is resolved to a
tenant via `APIToken.query(beta=token)`, backed by a new
`APITokenDAO.GetByBeta`. Because they perform their own token-based
auth, the routes are registered on the unauthenticated route group
(mirroring the Python blueprint, which has no `@login_required`).
Both live in a new `internal/handler/bot.go` + `internal/service/bot.go`
since they share the same source module. Handler unit tests cover the
auth, success, and error-mapping paths.
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
---------
Co-authored-by: Claude Code <claude@anthropic.com>
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-authored-by: Ling Qin <qinling0210@163.com>
## Summary
Fix error messages in `build.sh` and add documentation in
`internal/development.md` for downloading native static libraries
(pdfium, pdf_oxide, office_oxide).
## Changes
- `build.sh`: change error hint from `uv run download_deps.py` to `uv
run ragflow_deps/download_deps.py` (correct path from project root)
- `internal/development.md`: add section 2.1 documenting how to download
native libs and install lld
## Summary
- use the project-standard 32-character ID generator when creating
shared chatbot sessions
- fix MySQL insert failures caused by writing 36-character UUID strings
into `api_4_conversation.id`
### Summary
Adopt sync.WaitGroup.Go (Go 1.25) to simplify tracked goroutine
spawning. This replaces the error-prone trio of wg.Add(1), go func(),
and defer wg.Done() with a single, self-contained call.
More info: https://github.com/golang/go/issues/63796
Signed-off-by: grandpig <grandpig@outlook.com>
### Summary
This PR fixes a Go backend bug where updating agent settings, such as
description, could clear the agent DSL.
Root cause:
PUT /api/v1/agents/:canvas_id only bound the dsl field in Go. When the
frontend submitted settings without dsl, the service still updated the
canvas with an empty DSL value.
Changes:
- Treat agent updates as partial patches.
- Preserve existing DSL when dsl is not present in the request.
- Update only specified user_canvas fields instead of saving the full
row.
- Add a regression test for settings updates preserving DSL.
Test:
`go test ./internal/service ./internal/handler`
Co-authored-by: Zhichang Yu <yuzhichang@gmail.com>
## Summary
- align the Go `/api/v1/thumbnails` endpoint with the frontend request
format for repeated `doc_ids`
- return thumbnail mappings for multiple documents instead of failing on
a single missing document
- preserve Python-compatible thumbnail formatting, including base64
thumbnail passthrough
### Summary
Fixes a bug in the Go chunk list handler where the available` query
parser rejected `false` and `0` even though they were documented as
supported values.`
This caused requests from the "Disabled" chunk filter to return HTTP 400
and broke the chunk list page when filtering disabled chunks.
### Summary
As title
This PR fixes dataset index task creation failing with unsupported data
type: entity.JSONMap when loading document chunking config.
#### issues:
```
2026/06/30 15:19:40 /home/infiniflow/Documents/development/ragflow/internal/dao/document.go:162
[error] unsupported data type: ragflow/internal/entity.JSONMap
```
#### Changes:
+ Adds the missing GORM type:longtext tag to ParserConfig in
DocumentDAO.GetChunkingConfig.
+ Adds a DAO regression test covering GetChunkingConfig joins across
document, knowledgebase, and tenant while scanning parser_config.
### Summary
As title
Before, it return `update success` but never insert or update any
metadata
fixed:
```go
_, err = s.docEngine.InsertMetadata(nil, []map[string]interface{}{
{
"id": docID,
"kb_id": doc.KbID,
"meta_fields": meta,
},
}, tenantID)
```
### What problem does this PR solve?
Closes#12962
MCPToolCallSessions created during agent execution (in `Agent.__init__`)
are never explicitly closed. Each session starts its own event loop
thread and opens an SSE/HTTP connection to the MCP server. When the
canvas goes out of scope, these threads and connections remain alive
indefinitely, accumulating over time and causing resource exhaustion
after prolonged use.
### Solution
1. Add a `Graph.close()` method that iterates all components, finds
MCPToolCallSessions held by Agent tools, and calls `close_sync()` on
each to properly shut down the event loop, thread, and connection.
2. Call `canvas.close()` in `finally` blocks after `canvas.run()`
completes in `canvas_service.py` and `canvas_app.py`.
3. Move MCP session cleanup to `finally` blocks in `test_tool` endpoint
(`mcp_server_app.py`) and `get_mcp_tools` (`api_utils.py`) to ensure
sessions are closed even on exceptions.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
---------
Co-authored-by: conflict-resolver <conflict-resolver@local>
Co-authored-by: Zhichang Yu <yuzhichang@gmail.com>
Replace fragile wall-clock timeout assertions with semantic checks for
deadline errors, retry suppression, and event ordering. Keep only
lower-bound timing checks where they prove backoff behavior. This
reduces CPU-load flakes without weakening regression coverage.
### Summary
This PR fixes incorrect dataset document counters in the Go service.
Several document creation paths inserted document records directly
through documentDAO.Create, bypassing the shared InsertDocument logic
that increments knowledgebase.doc_num. As a result, datasets could
contain documents while doc_num remained 0.
### Summary
```
/api/v1/chats/<chat_id>/sessions/<session_id>/messages/<msg_id> DELETE
/api/v1/chats/<chat_id>/sessions/<session_id>/messages/<msg_id>/feedback PUT
```
Migrates the chat session message delete and feedback APIs to the Go
server, matching the Python behavior for authorization, session
ownership checks, message/reference updates, and feedback validation.
### What problem does this PR solve?
_Briefly describe what this PR aims to solve. Include background context
that will help reviewers understand the purpose of the PR._
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
- [x] New Feature (non-breaking change which adds functionality)
- [x] Other (please describe):
## Summary
Agent (Canvas) runs previously did not surface token usage in the SSE
stream, and RAGFlow's own Langfuse generations for agent runs were
missing the prompt/completion split and the session/user correlation.
This made it impossible for an external caller (or Langfuse) to
reconcile an agent turn's cost with the upstream provider (e.g.
OpenRouter), because a single turn can issue several distinct LLM calls
(query rewriting / cross-language translation, multi-round tool
reasoning, nested sub-agents, and the final answer).
This PR introduces a per-run token usage sink so that **every** LLM call
in a run is aggregated and reported once, and enriches Langfuse
generations with the prompt/completion split plus session/user
attributes.
## What changes
### 1. Per-run token usage sink (`common/token_utils.py`)
- Adds two `contextvars`: `token_usage_sink` (a mutable per-run
accumulator) and `langfuse_run_attrs` (session_id/user_id for the run).
- Adds `record_run_token_usage(...)` (thread-safe via a lock, because
`thread_pool_exec` copies the context into worker threads that share the
sink dict) and `usage_from_response(...)` which extracts a
`{prompt_tokens, completion_tokens, total_tokens}` split from
OpenAI/OpenRouter-style responses.
### 2. Provider layer captures the prompt/completion split
(`rag/llm/chat_model.py`)
- `LiteLLMBase` and `Base` now store `self.last_usage`
(prompt/completion/total) for the most recent chat call, in both the
plain and tool-calling paths.
- Streaming requests set `stream_options.include_usage = True` (LiteLLM
path) so the authoritative usage arrives on the final chunk; this is
read even on the usage-only chunk that carries no `choices`.
- Fixes a multi-round accounting bug in `*_with_tools`: token totals
were **overwritten** by each round (`total_tokens = tol`) instead of
accumulated, undercounting multi-round tool conversations. Each round is
now committed to a running aggregate.
### 3. LLMBundle reports usage once, per call
(`api/db/services/llm_service.py`)
- New `_report_usage(total_tokens)` records the call's usage into the
active run sink and returns the prompt/completion/total split for
Langfuse. The split is only used when it is consistent with the
authoritative total; otherwise only the total is reported.
- All three chat entry points (`async_chat`, `async_chat_streamly`,
`async_chat_streamly_delta`) now emit `usage_details` with
`input`/`output`/`total` instead of total-only.
- `_start_langfuse_observation` now applies `session_id`/`user_id` from
the per-run context (`langfuse_run_attrs`) so agent-run generations are
correctly grouped, even though agent LLMBundles are constructed without
those attributes.
### 4. Canvas installs the sink and emits the aggregate
(`agent/canvas.py`)
- `Canvas.run()` installs a fresh `token_usage_sink` and
`langfuse_run_attrs` (from `user_id`/`session_id`) at the start of every
turn.
- `message_end` now includes an aggregated `usage` object:
`{prompt_tokens, completion_tokens, total_tokens, calls}` covering all
LLM calls in the run.
### 5. Pass session id into the run
(`api/db/services/canvas_service.py`)
- `completion()` forwards `session_id` to `Canvas.run()` for Langfuse
session correlation.
## Why a context variable
LLM calls in an agent run originate from many places that each build
their own `LLMBundle` (e.g. `cross_languages`/`keyword_extraction`
helpers, the Agent component, and nested sub-agents invoked as tools). A
run-scoped context variable is the only non-invasive chokepoint that
captures all of them exactly once, including nested agents (which run in
the same async context) and thread-pool tools (the executor copies the
context).
## Behavior / compatibility
- No public API or wire-format removal: `message_end` gains an
additional optional `usage` field; existing consumers are unaffected.
- When a provider does not return authoritative usage, behavior falls
back to the previous token estimate (total only, no split).
- Non-agent flows (Dataflow `Pipeline`, sync `Graph.run`) are untouched.
## Testing
- [x] Simple agent answer: `message_end.usage.total_tokens` matches
provider usage.
- [x] Agent with cross-language retrieval: aggregate equals the sum of
both provider calls.
- [x] Tool-calling agent (multi-round): total accumulates across rounds.
- [x] Nested agent (agent-as-tool): sub-agent tokens included in the
parent run total.
- [x] Langfuse: agent generations show input/output split and are
grouped by session/user.
---------
Co-authored-by: yzc <yuzhichang@gmail.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
## Problem
When building or updating a knowledge graph with a large number of
entities and edges, `set_graph()` in `rag/graphrag/utils.py` creates one
`asyncio` task per entity and one per edge, each calling
`embd_mdl.encode([single_name])` — a single-item HTTP request to the
embedding server.
For a graph with 17,000+ nodes and edges (real case reported in #16205),
this generates **34,000+ individual embedding API round-trips** instead
of ~266 batched calls at the default `_INSERT_BULK_SIZE=128`. The
asyncio gather over thousands of tasks makes the embedding server the
bottleneck; under load, a single slow/failed call aborts all remaining
tasks, causing the pipeline to stall and never complete.
Closes#16205. Related: #15921.
## Root Cause
```python
# Before (in set_graph, node loop):
tasks = [asyncio.create_task(graph_node_to_chunk(n, ...)) for n in nodes]
# Each task calls embd_mdl.encode([single_name]) — 1 HTTP call per node
```
`graph_node_to_chunk` checks the embed cache first, but the cache is
cold on first build, so every task makes a live API call.
## Fix
Pre-warm the embedding cache with batched calls before spawning tasks.
Each batch pre-warm calls `embd_mdl.encode(batch_of_128)` once,
populating the cache. Then every individual task hits the cache and
makes zero embedding API calls.
- Only encodes names not already in cache (no-op on warm cache / small
incremental updates)
- Uses existing project idioms: `thread_pool_exec`, `chat_limiter`,
`_INSERT_BULK_SIZE`, `get_embed_cache`, `set_embed_cache`
- Mirrors the `ENABLE_TIMEOUT_ASSERTION` timeout pattern from
`graph_node_to_chunk`
- Zero behavior change: per-task encode logic remains as a correct
fallback
## Result
| Graph size | Before | After |
|---|---|---|
| 17,576 edges | ~17,576 embedding calls → stall | ~138 batched calls |
| 17,509 nodes | ~17,509 embedding calls → stall | ~137 batched calls |
| **Total** | **~35,000 calls** | **~275 calls** |
---------
Co-authored-by: Oti_B <oti@mac.speedport.ip>
### Summary
The addition of the Context method to Go's testing.T provides
significant improvements for writing concurrent tests. It allows better
management of goroutines, ensuring they properly exit and preventing
issues like deadlocks and unfinished processes.
By using Context, errors and cancellations can be handled more
effectively, making tests more robust and easier to reason about. This
change also enables tighter integration between tests and the
application code, especially for systems that span multiple concurrent
components. Overall, it simplifies test code and enhances test stability
and maintainability.
More info: [golang/go#18368](https://github.com/golang/go/issues/18368)
Signed-off-by: blackflytech <blackflytech@outlook.com>
### Summary
`truncateText` in the `reduction` and `summarization` middlewares
truncates with `s[:maxLen]`, which slices by byte. When `maxLen` lands
inside a multi-byte character (common with CJK or other non-ASCII
content flowing through the agent), the string is cut mid-rune and the
tail byte(s) become invalid UTF-8. That broken text then goes into the
reduced context / summary prompt.
`TruncateToolResult` in the same `reduction` package already avoids this
by slicing on a rune boundary and even notes it in a comment. This PR
makes the two `truncateText` helpers do the same, so they stay
consistent with the existing helper.
Both functions keep their existing output shape (summarization still
appends `...`). Added a small unit test in each package covering ASCII
truncation and a CJK string, asserting the result stays valid UTF-8.
## Summary
TuShare required non-empty upstream input but filtered fetched news with
the static `keyword` param (default empty string), so agent-provided
keywords were ignored.
Use `self._param.keyword or ans` when filtering, matching how AkShare
uses upstream input for its query.
Fixes#16360
## Test plan
- [x] `test_tushare_filters_with_upstream_keyword_when_param_empty`
mocks the API and asserts only rows matching the upstream keyword are
returned
---------
Co-authored-by: yzc <yuzhichang@gmail.com>
Co-authored-by: Harsh Kashyap <harshkashyap@Harshs-MacBook-Pro.local>
### What problem does this PR solve?
Closes#16418.
`scholarly.search_pubs(...)` returns a **lazy generator**, but
`agent/tools/googlescholar.py` treated it as a re-iterable, bounded
list:
```python
scholar_client = scholarly.search_pubs(kwargs["query"], ...) # lazy generator
self._retrieve_chunks(scholar_client, ...) # (1) iterates -> exhausts it
self.set_output("json", list(scholar_client)) # (2) already empty -> []
```
1. **`json` output was always empty.** `_retrieve_chunks` iterates
`scholar_client`, exhausting the generator; `list(scholar_client)` then
returns `[]`.
2. **`top_n` was never applied.** Unlike `ArXiv`
(`max_results=self._param.top_n`), the unbounded generator was passed
straight to `_retrieve_chunks`, which has no internal limit — so the
tool kept paginating well past Top N (until an error, rate-limit/block,
or `COMPONENT_EXEC_TIMEOUT`).
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
### Changes
- Materialize at most `top_n` results once with `itertools.islice`, and
reuse that list for both `_retrieve_chunks` and the `json` output.
- Add regression tests
(`test/unit_test/agent/component/test_googlescholar.py`, stubbing
`scholarly.search_pubs`) covering the `top_n` bound, the non-empty
`json` output, and the empty-query short-circuit.
Verified: against `main` the new tests fail with `assert 30 == 5` (top_n
ignored) and `assert 0 == 5` (empty json); with this fix all pass.
Backend-only.
---------
Co-authored-by: Zhichang Yu <yuzhichang@gmail.com>
## Summary
Add two new MCP tools to the RAGFlow MCP server:
1. **ragflow_list_datasets** - List all accessible datasets with IDs,
names, descriptions
2. **ragflow_list_chats** - List all accessible chat assistants with
IDs, names, descriptions
### Implementation
- Added `list_chats()` method to `RAGFlowConnector`
- Registered both tools in `list_tools()` and `call_tool()`
- Follows existing `ragflow_retrieval` pattern for error handling
### Usage via langchain-mcp-adapters
---------
Co-authored-by: saltsalt123 <saltsalt123@users.noreply.github.com>
Co-authored-by: yzc <yuzhichang@gmail.com>
### What problem does this PR solve?
Closes https://github.com/infiniflow/ragflow/issues/14571.
Adds CAJAL as a first-class local scientific-writing option in RAGFlow:
- registers `agnuxo/cajal-4b-p2pclaw` as a known Ollama chat model with
a 32K context setting
- adds a built-in “CAJAL scientific paper agent” template under the
existing agent template catalog
- preconfigures the agent for grounded scientific writing: retrieval
first, citation traceability, LaTeX-ready output, and explicit
limitations when evidence is missing
- adds unit coverage to ensure the template normalizes through RAGFlow’s
production template loader, keeps graph form data in sync, and exposes
the Ollama model option
Behavior/evidence gathered for the requested model:
- Hugging Face model metadata for `Agnuxo/CAJAL-4B-P2PCLAW` reports
`pipeline_tag=text-generation` and tags including `gguf`, `llama.cpp`,
`vllm`, `scientific-research`, `papers`, `academic-writing`, `latex`,
and `license:apache-2.0`.
- The model card documents CAJAL as a 4B scientific paper generation
model with 32K context, local inference, LaTeX/citation specialization,
and CPU-only support around 5 tok/s on Ryzen 7 5800X.
- Local CPU generation could not be completed on this machine because
the advertised Ollama model name is not currently resolvable from
Ollama’s registry: both
`https://registry.ollama.ai/v2/agnuxo/cajal-4b-p2pclaw/manifests/latest`
and
`https://registry.ollama.ai/v2/library/agnuxo/cajal-4b-p2pclaw/manifests/latest`
returned `404 Not Found`; the Hugging Face repo tree currently exposes
an 8.4 GB `model.safetensors` but no GGUF artifact in `main`. The
template therefore targets the documented Ollama model name for users
who have the local CAJAL deployment/model file available.
Verification run locally:
```bash
python3 -m pytest test/test_cajal_template_unit.py -q
# 3 passed in 0.34s
python3 - <<'PY'
import json, glob
for f in sorted(glob.glob('agent/templates/*.json') + ['conf/llm_factories.json']):
with open(f, encoding='utf-8') as fp: json.load(fp)
print('json_ok')
PY
# json_ok
python3 -m ruff check test/test_cajal_template_unit.py
# All checks passed!
git diff --check
```
`uv run pytest
test/testcases/test_web_api/test_agent_app/test_cajal_template_unit.py
-q` was also attempted first, but dependency setup failed before test
collection while building `ormsgpack==1.5.0` from uv with a package
metadata parse error. Clearing uv’s `ormsgpack` cache and retrying
reproduced the same build failure, so the focused unit test was run with
the system Python environment instead.
### Type of change
- [ ] Bug Fix (non-breaking change which fixes an issue)
- [x] New Feature (non-breaking change which adds functionality)
- [ ] Documentation Update
- [ ] Refactoring
- [ ] Performance Improvement
- [ ] Other (please describe):
---------
Co-authored-by: sxxtony <sxxtony@users.noreply.github.com>
Co-authored-by: yzc <yzc@users.noreply.github.com>
Co-authored-by: Zhichang Yu <yuzhichang@gmail.com>
…rialization
Agent components (llm.py, agent_with_tools.py, message.py) store
functools.partial objects as deferred streaming handles in their output
slots. When the canvas state gets serialized for SSE events, Redis
commits, or logging, these partials — plus non-copyable objects like
Langfuse clients — crash json.dumps and deepcopy.
Changes:
- canvas_app.py: add default=str to json.dumps for SSE event
serialization (lines 238, 296)
- canvas.py: wrap deepcopy calls in try/except to handle non-copyable
objects (Langfuse clients, etc.), add default=str to final json.dumps
- base.py: add default=str to ComponentParamBase.__str__ to handle
non-serializable objects in component parameters
Closes#14229
### What problem does this PR solve?
_Briefly describe what this PR aims to solve. Include background context
that will help reviewers understand the purpose of the PR._
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
- [ ] New Feature (non-breaking change which adds functionality)
- [ ] Documentation Update
- [ ] Refactoring
- [ ] Performance Improvement
- [ ] Other (please describe):
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-authored-by: yzc <yuzhichang@gmail.com>
## Summary
- Treat `max_tokens=0` as unset (`or 8192`) when building model context
budgets, fixing agents that silently zeroed prompts when a vLLM model
had `max_tokens: 0` in tenant config
- Replace trailing same-role canvas history in `LLM._sys_prompt_and_msg`
instead of skipping the current user prompt
- Add `LLM.fit_messages()` validation after `message_fit_in` on agent
paths so empty user content fails fast with a clear error instead of
reaching vLLM
Fixes#16411
## Root cause
Agent canvas workflow called `message_fit_in` with `int(max_length *
0.97)`. When `max_length` was `0`, both system and user content were
trimmed to empty strings. The `[HISTORY STREAMLY]` log showing only
`{"role":"user","content":""}` matches this. A secondary bug skipped
appending the formatted user prompt when history ended with a `user`
role message.
## Test plan
- [x] Added `test/unit_test/agent/component/test_llm_prompt.py` for
role-replace, validation, and zero-budget fitting
- [x] Added
`test_message_fit_in_zero_budget_preserves_non_empty_messages` in
`test_generator_message_fit_in.py`
- [ ] CI unit tests
- [ ] Manual: agent canvas `begin → Retrieval → Agent → Message` with
vLLM Qwen3; confirm user message reaches LLM
Made with [Cursor](https://cursor.com)
---------
Co-authored-by: Taranum Wasu <taranumwasu@Taranums-MacBook-Pro.local>
Co-authored-by: Cursor <cursoragent@cursor.com>
Recovery PR for #16173 after the fork branch was accidentally reset
during rewrite-cleanup.
Cherry-picked onto current `main`:
- fix(common/time_utils): correct fallback timestamp and ISO-8601
normalization
- fix(common/time_utils): preserve zero timestamps and mark regression
tests
- test(common/time_utils): make fallback assertions deterministic
Supersedes closed#16173 — same branch
`Harsh23Kashyap/fix/time-utils-edgecases`, rebuilt per @yuzhichang
recovery steps in
https://github.com/infiniflow/ragflow/pull/16173#issuecomment-4829663835
---------
Co-authored-by: Harsh Kashyap <harshkashyap@Harshs-MacBook-Pro.local>
Co-authored-by: Cursor <cursoragent@cursor.com>
## Summary
Two changes to make Go build \& run independent of native libraries
(office_oxide, pdfium, pdf_oxide).
## 1. Make native libraries optional (build.sh + Go source)
## 2. Roll back tests.yml CI changes from PR #16391
This PR migrates the Box web OAuth flow from Python to Go for:
- POST /api/v1/connectors/box/oauth/web/start
- GET /api/v1/connectors/box/oauth/web/callback
- POST /api/v1/connectors/box/oauth/web/result
### What problem does this PR solve?
GET /api/v1/agents (list_agents) already supports filtering by
canvas_category, keywords, tags, and owner_ids, but it does not support
canvas_type — even though canvas_type is a persisted field on UserCanvas
and is already accepted on agent create/update APIs.
This gap causes two issues:
Filtering — clients cannot list agents by business category (e.g.
Marketing, Agent, Ingestion Pipeline) without fetching all agents and
filtering client-side.
Response payload — list_agents did not return canvas_type in each canvas
item, so consumers had to call GET /api/v1/agents/{id} per agent to read
it.
This PR adds optional canvas_type query parameter support and includes
canvas_type in the list response.
### Type of change
- [√] Bug Fix (non-breaking change which fixes an issue)
- [ ] New Feature (non-breaking change which adds functionality)
- [ ] Documentation Update
- [ ] Refactoring
- [ ] Performance Improvement
- [ ] Other (please describe):
## Summary
Fixes#15169 — `POST /api/v1/agents/chat/completions` returned
`data: {}` with no `session_id` when the agent produced no events
(e.g. the reporter's payload sent `"query": ""`).
## Root cause
For `{"agent_id": "...", "query": "", "stream": false}`:
1. No `session_id` in the request → new-session branch at
`agent_api.py:1278`.
2. `session_id = get_uuid()` at `agent_api.py:1294`.
3. Falls into `_run_workflow_session`.
4. `canvas.run(query="")` produces no events, so `final_ans`
stays `{}`.
5. Non-streaming path then hit:
```python
if not final_ans:
await commit_runtime_replica()
return get_result(data={})
```
`session_id` was allocated but silently dropped on the way out.
The streaming path had the same shape (only a bare `[DONE]` was
yielded — no SSE event carrying `session_id`). The
session-continuation path at `agent_api.py:1463` had the same bug
for callers that passed `session_id` and got `{}` back.
The successful (non-empty) paths were fine because every canvas
event has `ans["session_id"] = session_id` attached before being
yielded / captured into `final_ans` (see
`agent_api.py:255` and `:303`).
## Fix
Three minimal changes, all in
`api/apps/restful_apis/agent_api.py`:
1. **`_run_workflow_session` (non-streaming)**:
`return get_result(data={"session_id": session_id})` instead of
`data={}`.
2. **`_run_workflow_session` (SSE)**: if the canvas loop emits no
events, yield one
`data:{"session_id": "...", "data": {}}` event before
`[DONE]`, so the client receives the id over the wire.
3. **`agent_chat_completion` session-continuation**: echo the
caller-supplied `session_id` back in the empty-events case
instead of `{}`.
No change needed on the happy paths — they already attach
`session_id` to every event.
## Test plan
- [ ] Repro from the issue: `POST /api/v1/agents/chat/completions`
with `{"agent_id": "<id>", "query": "", "stream": false}`.
Response `data` should now contain `session_id`.
- [ ] Same payload with `"stream": true`. SSE stream should
contain one event with `session_id` before `data:[DONE]`.
- [ ] Same shape but with a real, non-empty `"query"` (new
session). Response should be unchanged from before — every
event still carries `session_id`, final response still
includes it on `final_ans`.
- [ ] Pass an existing `session_id` plus `"query": ""`. Response
should echo that `session_id` back instead of `{}`.
- [ ] Pass an existing `session_id` plus a normal query. Response
should be unchanged from before.
- [ ] `openai-compatible: true` path is untouched — sanity-check
it still works.
- [ ] Run `uv run pytest` to make sure no existing tests regress.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
- [ ] New Feature (non-breaking change which adds functionality)
- [ ] Documentation Update
- [ ] Refactoring
- [ ] Performance Improvement
- [ ] Other (please describe):
## Summary
Fixes#15215 — attachments uploaded to an agent were not reaching the
LLM.
When a user uploads a file in an agent chat, `canvas.run` parses it into
the `sys.files` global (text content for documents, `data:image/...`
URIs
for images — see `agent/canvas.py:752-768`). But the LLM/Agent
component's
`_prepare_prompt_variables` only substitutes variables the user's prompt
template explicitly references via `{var}` placeholders. The default
prompt is `[{"role": "user", "content": "{sys.query}"}]` with no
`{sys.files}`, so the parsed attachment content never reaches the model.
In the reporter's logs, this is why the agent saw only the bare query
`附件 摘要 attachment summary` and went searching the dataset instead of
reading the uploaded PDF.
## Fix
`agent/component/llm.py` — added `_collect_sys_files()` and an
auto-injection step in `_prepare_prompt_variables`:
- If `sys.files` is non-empty **and** neither `sys_prompt` nor any entry
in `prompts` already contains `{sys.files}` (no double-injection),
split the entries into text vs. `data:image/...` URIs.
- Image URIs are merged into `self.imgs`, which the existing logic uses
to switch the chat model to `IMAGE2TEXT` and pass `images=...` to
`async_chat`.
- Text content is appended to the last `user` role message in `msg`,
mirroring how `dialog_service.async_chat_solo` handles attachments for
the non-agent chat path (`api/db/services/dialog_service.py:318-321`).
Both `LLM._invoke_async` and `Agent._invoke_async` (tool-using) go
through `_prepare_prompt_variables`, so plain LLM nodes and Agent nodes
are fixed in both streaming and non-streaming paths.
## Test plan
- [ ] Upload a PDF attachment to an agent with the default `{sys.query}`
prompt and ask "summarize the attachment" — the model should answer
from the file content rather than searching the knowledge base.
- [ ] Upload an image attachment to an agent and ask about its contents
—
the model should switch to the vision-capable LLM and answer from
the image.
- [ ] Verify that an agent whose prompt **does** include `{sys.files}`
still works and does **not** include the file content twice.
- [ ] Verify that an agent run with no attachments behaves unchanged.
- [ ] Run `uv run pytest` to make sure no existing tests regress.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
- [ ] New Feature (non-breaking change which adds functionality)
- [ ] Documentation Update
- [ ] Refactoring
- [ ] Performance Improvement
- [ ] Other (please describe):
---------
Co-authored-by: yzc <yuzhichang@gmail.com>
### Summary
```
RAGFlow(admin)> show users plan summary;
+---------+----------------------------------------------------------------+
| field | value |
+---------+----------------------------------------------------------------+
| command | show_users_plan_summary |
| error | 'Show users plan summary' is implemented in enterprise edition |
+---------+----------------------------------------------------------------+
```
Signed-off-by: Jin Hai <haijin.chn@gmail.com>
### What problem does this PR solve?
This PR adds Google BigQuery as a first-class data source connector in
RAGFlow.
It enables users to ingest and sync BigQuery data using the same
row-to-document model used by relational database connectors: selected
content columns become document text, metadata columns become document
metadata, an optional ID column provides stable document IDs, and an
optional timestamp column enables cursor-based incremental sync.
The connector supports service-account JSON credentials, table mode,
custom query mode, GoogleSQL queries, cursor-based incremental sync,
deleted-row pruning support, configurable query limits such as
`maximum_bytes_billed`, dry-run validation, batch loading, stable
document IDs, and BigQuery-aware value serialization.
### Summary
As title
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
- [x] Bug Fix (non-breaking change which fixes an issue)
### Summary
```
/api/v1/chats/<chat_id>/sessions/<session_id>/messages/<msg_id> DELETE
/api/v1/chats/<chat_id>/sessions/<session_id>/messages/<msg_id>/feedback PUT
```
Migrates the chat session message delete and feedback APIs to the Go
server, matching the Python behavior for authorization, session
ownership checks, message/reference updates, and feedback validation.
### Testing
- `/usr/local/go/bin/go test ./internal/service ./internal/handler`
- Verified through the frontend page for deleting chat messages and
updating message feedback
Fix the reference index used when deleting a chat message pair.
Each user/assistant message pair shares one reference entry, while the
first assistant prologue has no reference. Using `i // 2` correctly
removes the reference for the deleted pair and avoids deleting the
previous turn's reference.
### What problem does this PR solve?
As title
main fix:
```go
if _, ok := req["meta_data_filter"]; !ok || req["meta_data_filter"] == nil {
req["meta_data_filter"] = map[string]interface{}{}
}
```
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
- [x] Refactoring
### What problem does this PR solve?
Package refactor and PDF post process.
### Type of change
- [x] Refactoring
---------
Co-authored-by: Claude <noreply@anthropic.com>
### Summary
fix: user-setting modal fixes and DOMPurify cleanup
- HighlightMarkdown: drop post-process DOMPurify pass (ineffective after
preprocessLaTeX; Coderabbit CRITICAL
#3486038798)
- SettingTeam: add invite-only-registered-users hint to add-user modal
- SettingModel: reset provider loading state when add-provider modal
closes
- MCP edit dialog: set maskClosable=false to prevent accidental
dismissal
- Form: switch FormDescription color from text-muted-foreground to
text-text-disabled
## What problem does this PR solve?
DOCX parsing could crash when a paragraph used a `Heading`-prefixed
style without a trailing numeric level, such as `Heading`, `Heading1`,
or `Heading Title`.
`docx_question_level()` assumed every heading style looked like `Heading
N` and called `int(p.style.name.split(" ")[-1])`. For non-numbered
heading styles, that raises `ValueError` and breaks Manual, Q&A, and
Laws chunking.
This PR parses heading levels safely and falls back to level 1 for
Heading-prefixed styles without an explicit numeric suffix.
Closes#16163.
## Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
- [x] Test Case (non-breaking change which adds test coverage)
### What problem does this PR solve?
1. Add CREATE and DROP DATASET / MEMORY / AGENT / SEARCH / CHAT.
2. Add option to build.sh to strip RAGFlow binary.
### Type of change
- [x] Refactoring
---------
Signed-off-by: Jin Hai <haijin.chn@gmail.com>
Summary
Add module-level debug logging to track Agent canvas execution flow
(Closes#9306), enabling developers to diagnose component invocation,
input/output states, and variable resolution without modifying
production code.
Also fix related bugs in message.py: re.sub backreference issue and
unawaited _save_to_memory coroutine causing silent memory save failures.
Changes
agent/canvas.py: log workflow start, component invocation, and component
completion
agent/component/agent_with_tools.py: log Agent parameter resolution and
LLM invocation path; standardize json.dumps usage
agent/component/base.py: log get_input() variable resolution branches
agent/component/message.py: fix re.sub backreference issue; properly
await _save_to_memory coroutine
Design
Uses module-level loggers (logging.getLogger(__name__)) to support
selective debugging: LOG_LEVELS=agent=DEBUG
Zero performance impact in production (INFO level by default)
Works with existing PUT /system/config/log API for runtime level changes
Closes#9306
Note: While adding debug logging, I discovered and fixed two related
bugs in message.py:
- re.sub replacement value was interpreted as regex backreference
instead of literal string
- _save_to_memory coroutine was not properly awaited, causing silent
failures
---------
Co-authored-by: wills <willsgao@163.com>
closes#14769
### What problem does this PR solve?
_Briefly describe what this PR aims to solve. Include background context
that will help reviewers understand the purpose of the PR._
### Type of change
- [ ] Bug Fix (non-breaking change which fixes an issue)
- [x] New Feature (non-breaking change which adds functionality)
- [ ] Documentation Update
- [ ] Refactoring
- [ ] Performance Improvement
- [ ] Other (please describe):
---------
Co-authored-by: Zhichang Yu <yuzhichang@gmail.com>
### What problem does this PR solve?
Fixes a workflow editor bug where deleting an Iteration Box could leave
orphan child nodes and dangling edges in client state. Those stale
references could be exported with the workflow and later cause rendering
errors, broken connections, and unstable editing behavior.
### Root Cause
Iteration deletion logic only removed the container, its direct
children, and some internal edges. It did not consistently remove the
full descendant subtree or all edges connected to deleted child nodes,
and the keyboard delete path was not expanded to include Iteration
descendants.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
- [ ] New Feature (non-breaking change which adds functionality)
- [ ] Documentation Update
- [ ] Refactoring
- [ ] Performance Improvement
- [ ] Other (please describe):
---------
Co-authored-by: Zhichang Yu <yuzhichang@gmail.com>
## Related issues
Closes#15144
### What problem does this PR solve?
`POST /api/v1/agents/rerun` loaded a pipeline operation log by UUID via
`PipelineOperationLogService.get_documents_info` with no authorization,
then wiped chunks, reset document counters, deleted tasks, and re-queued
dataflow for the victim document.
Any authenticated user who knew a victim's pipeline log id could disrupt
parsing on documents they did not own.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
- [ ] New Feature (non-breaking change which adds functionality)
- [ ] Documentation Update
- [ ] Refactoring
- [ ] Performance Improvement
- [ ] Other (please describe):
### Changes
| File | Change |
|------|--------|
| `api/apps/restful_apis/agent_api.py` | Call
`DocumentService.accessible(doc["id"], tenant_id)` before destructive
rerun operations; deny with generic `"Document not found."` |
|
`test/unit_test/api/apps/restful_apis/test_rerun_agent_authorization.py`
| Unit tests: cross-tenant log rejected, missing/unauthorized same
message, authorized rerun proceeds |
### Security notes
- **CWE-639:** Closes cross-tenant pipeline rerun / chunk wipe via
leaked log UUID.
- `tenant_id` from `@add_tenant_id_to_kwargs` is `current_user.id`;
`DocumentService.accessible` covers team-shared KBs.
### Test plan
- [ ] `pytest
test/unit_test/api/apps/restful_apis/test_rerun_agent_authorization.py`
- [ ] Manual: attacker cannot rerun victim pipeline log id
```bash
cd ragflow
uv run pytest test/unit_test/api/apps/restful_apis/test_rerun_agent_authorization.py -q
```
---------
Co-authored-by: Zhichang Yu <yuzhichang@gmail.com>
## Summary
- **Backend**: `_iter_session_completion_events` in `agent_api.py` was
filtering out `user_inputs` and `workflow_finished` SSE events, causing
agents with UserFillUp components to silently fail in explore mode — the
interactive form never appeared, while the same agent worked correctly
in run (editor) mode.
- **Frontend**: `SessionChat` component in explore mode was missing
`DebugContent` children rendering inside `MessageItem`, so even if the
backend forwarded the events, the form UI would not render. Added
`DebugContent`, `MarkdownContent`, `useAwaitCompentData` hook, and
input-disabling logic to match the run mode's `chat/box.tsx` behavior.
## What was changed
### Backend (`api/apps/restful_apis/agent_api.py`)
- Line 266: Added `"user_inputs"` and `"workflow_finished"` to the
allowed event filter in `_iter_session_completion_events`
### Frontend (`web/src/pages/agent/explore/components/session-chat.tsx`)
- Added imports: `DebugContent`, `MarkdownContent`,
`useAwaitCompentData`, `useParams`
- Added `sendFormMessage` from `useSendSessionMessage()` hook
- Added `useAwaitCompentData` hook for form state management
- Added `DebugContent` as `MessageItem` children for the latest
assistant message (renders UserFillUp form)
- Added `MarkdownContent` + submitted values display for previous
assistant messages
- Updated `NextMessageInput` disabled states to respect `isWaitting`
(form submission in progress)
## Test plan
- [x] Agent with UserFillUp component (e.g., email draft with
send/edit/cancel options) shows interactive form in **explore mode**
- [x] Same agent continues to work correctly in **run (editor) mode**
- [x] Form submission sends data back to the agent and workflow
continues
- [x] Input field is disabled while waiting for form submission
- [ ] Agents without UserFillUp components are unaffected in explore
mode
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Co-authored-by: Zhichang Yu <yuzhichang@gmail.com>
Fixes#16168
## Summary
- Add session-scoped authorization for `GET
/api/v1/documents/artifact/<filename>`
- Allow download only when the artifact filename appears in the caller's
`api_4_conversation` message and
`UserCanvasService.accessible(dialog_id, user_id)` passes
- Deny with generic `"Artifact not found."` before storage access (no
cross-user enumeration)
- Return 4xx when the blob is missing (existing behavior preserved)
## Approach
Sandbox artifacts are runtime CodeExec outputs, not KB documents — this
uses the same session gate pattern as `agent_chat_completion`, not
`DocumentService.accessible`.
## Test plan
- [x] Unit: denied when filename not referenced in user sessions
- [x] Unit: denied when agent canvas is not accessible
- [x] Unit: authorized user receives bytes; missing blob returns
`"Artifact not found."`
- [ ] `pytest
test/testcases/test_web_api/test_document_app/test_document_metadata.py
-k get_artifact`
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Zhichang Yu <yuzhichang@gmail.com>
## Summary
Closes#15425. The agent **Invoke** (HTTP Request) component now calls
`assert_url_is_safe` and `pin_dns` before `requests.*`, matching Crawler
and SearXNG.
## Changes
- `agent/component/invoke.py`: SSRF guard + DNS pinning on outbound
requests.
- `test_invoke_component_unit.py`: unit test blocks loopback URL without
calling `requests.get`.
## Test plan
- [x] `pytest
test/testcases/test_web_api/test_canvas_app/test_invoke_component_unit.py::test_invoke_blocks_loopback_url_with_ssrf_guard`
(requires project test env / `ZHIPU_AI_API_KEY` in CI)
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Zhichang Yu <yuzhichang@gmail.com>
### What problem does this PR solve?
Fixes#14882
Agent webhook execution currently fails open when the saved webhook
`security` block is missing/empty, or when `auth_type` is set to `none`.
This allows unauthenticated webhook invocation without an explicit
operator opt-in.
This PR makes anonymous webhook access explicit:
- Rejects missing or empty webhook security config.
- Requires `allow_anonymous: true` when `auth_type` is `none`.
- Preserves explicit anonymous webhooks by having the frontend serialize
`allow_anonymous: true` when the user selects `None` auth.
- Updates webhook unit tests to cover both denied implicit-anonymous
configs and allowed explicit-anonymous configs.
### Type of change
- [x] Bug Fix
- [x] Security hardening
- [x] Test
### Tests
- [x] `ZHIPU_AI_API_KEY=dummy uv run python -m pytest
--confcutdir=test/testcases/test_web_api/test_agent_app
test/testcases/test_web_api/test_agent_app/test_agents_webhook_unit.py`
- [x] `uv run ruff check api/apps/restful_apis/agent_api.py
test/testcases/test_web_api/test_agent_app/test_agents_webhook_unit.py`
- [x] `npm exec eslint src/pages/agent/utils.ts
src/pages/agent/form/begin-form/schema.ts`
---------
Co-authored-by: Zhichang Yu <yuzhichang@gmail.com>
### What problem does this PR solve?
Fixes#15456.
The SDK agent-bot routes `POST /api/v1/agentbots/<agent_id>/completions`
and `GET /api/v1/agentbots/<agent_id>/inputs`
(`api/apps/restful_apis/bot_api.py`) authenticate the caller with a beta
API token — which only yields the caller's `tenant_id` — but then load
and run the agent named in the URL **without verifying the agent belongs
to the caller's tenant**. `UserCanvasService.get_agent_dsl_with_release`
even accepts a `tenant_id` it never uses, and `begin_inputs` calls
`get_by_id` directly. Any holder of a single valid beta token could
therefore run another tenant's agent (leaking its DSL/prompts/tool
config) or read another tenant's agent metadata and begin input form,
just by substituting a victim `agent_id`.
This PR adds the project's existing ownership gate,
`UserCanvasService.accessible(agent_id, tenant_id)`, to both endpoints
right after token authentication — mirroring the checks already enforced
on the equivalent first-party routes in
`api/apps/restful_apis/agent_api.py` (lines 75/578/775) and on the
sibling `chatbot_completions` / `create_agent_session` /
`delete_agent_session` handlers in the same file. On failure it returns
the same `Can't find agent by ID: <id>` message already used by
`begin_inputs`, so it does not reveal whether an `agent_id` exists in
another tenant.
Added a regression test
(`test/unit_test/api/apps/restful_apis/test_agentbots_access_control.py`,
following the existing stubbed-loader pattern from
`test_get_agent_session.py`) asserting that an inaccessible `agent_id`
is rejected before the agent is loaded (`begin_inputs`) or executed
(`completions`), and that an accessible agent still proceeds.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
- [ ] New Feature (non-breaking change which adds functionality)
- [ ] Documentation Update
- [ ] Refactoring
- [ ] Performance Improvement
- [ ] Other (please describe):
---------
Co-authored-by: Zhichang Yu <yuzhichang@gmail.com>
## Summary
Fixes#14985 — clicking the **Thinking** button in a shared/embedded
chat returns 401 and bounces the user to the login page, even though
the same share page can chat with the agent just fine.
## Root cause
In shared chat, `useGetSharedChatSearchParams` binds `conversationId`
to the URL's `shared_id` query param — which is the **beta APIToken**,
not the real agent id. That `conversationId` propagates through the
component tree:
```tsx
<WorkFlowTimeline canvasId={conversationId}>
→ useFetchMessageTrace(canvasId)
→ GET /api/v1/agents/<sharedId>/logs/<messageId>
```
But `/agents/<agent_id>/logs/<message_id>` is decorated with
`@login_required` (`api/apps/restful_apis/agent_api.py:842-846`).
The share page only holds the beta token — there is no session JWT
— so the request 401s and quart-auth redirects to the login page.
The reporter's server log matches exactly:
```
load_user from jwt got exception No b'.' found in value
load_user: No APIToken found for token=ULG10SWG3E...
Unauthorized request (quart_auth)
GET /api/v1/agents/394013f8d42211f0bad6123fa55e8ed9/logs/96fd72e2-... 1.1 401
```
The `394013f8...` segment in the URL is the `shared_id` (beta
token), not an actual agent id. `_load_user` already accepts the
regular `APIToken.token` field, but not `APIToken.beta`, by design
— beta is a much weaker share-link credential than a personal API
key.
The sibling endpoints `/agentbots/<id>/completions` and
`/agentbots/<id>/inputs` already use the right auth pattern for
this scope (beta-token via `_get_sdk_authorization_token` →
`APIToken.query(beta=token)`). Trace just didn't have a parallel.
## Fix
### Backend (`api/apps/restful_apis/bot_api.py`)
Added a beta-token sibling endpoint:
```
GET /api/v1/agentbots/<shared_id>/logs/<message_id>
```
- Same auth shape as the existing `agentbots` endpoints.
- The `<shared_id>` path segment is a client-supplied label only.
The real `agent_id` used to build the Redis key
(`<agent_id>-<message_id>-logs`) is taken from
`APIToken.dialog_id` on the looked-up token, so the endpoint
never trusts client-supplied identifiers for the data lookup.
- Returns the same `{data: ...}` shape as the existing
`/agents/<id>/logs/<message_id>` endpoint, so the frontend
doesn't need to reshape the response.
### Frontend
- `web/src/utils/api.ts`: added `sharedTrace(sharedId, messageId)`
URL builder.
- `web/src/services/agent-service.ts`: added
`fetchSharedTrace({ shared_id, message_id })`.
- `web/src/hooks/use-agent-request.ts`: `useFetchMessageTrace`
takes an optional `isShare` argument. When set, it calls
`fetchSharedTrace`; `isShare` is also folded into the
`queryKey` so the two modes never share cached results.
- `web/src/pages/agent/log-sheet/workflow-timeline.tsx`:
forwards the already-existing `isShare` prop into the hook.
All other existing call sites of `useFetchMessageTrace` (webhook
timeline, pipeline log, dataflow result) pass no `isShare`
argument → undefined → falsy → unchanged behavior.
## Test plan
- [ ] In the regular Agent UI (logged-in user): open the trace /
log sheet for any message and click into "Thinking" — the
timeline should still load via `/agents/<id>/logs/<msg>`,
same as before.
- [ ] From the Agent page, click **Chat in new tab** to open
`/chat/share?shared_id=<token>&from=agent`. Send a message,
wait for a response, then click **Thinking** on the
assistant turn. The trace panel should load instead of
redirecting to the login page.
- [ ] Same flow but with the agent embedded in an iframe ("Embed
into webpage") — confirm there is no login redirect.
- [ ] In DevTools → Network, confirm the share-chat trace request
goes to `/api/v1/agentbots/<sharedId>/logs/<msgId>` and
returns 200 with the same JSON shape as the logged-in path.
- [ ] Confirm the chat completions, inputs, and upload flows in
the share page still work — they were not touched.
- [ ] Send a bogus / expired beta token to the new endpoint and
confirm it returns the standard "Authentication error: API
key is invalid!" response (no traceback, no 500).
- [ ] Run `uv run pytest` to make sure no existing tests regress.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
- [ ] New Feature (non-breaking change which adds functionality)
- [ ] Documentation Update
- [ ] Refactoring
- [ ] Performance Improvement
- [ ] Other (please describe):
---------
Co-authored-by: Zhichang Yu <yuzhichang@gmail.com>
## Related issues
Closes#15128
### What problem does this PR solve?
`GET` and `DELETE` `/api/v1/agents/<agent_id>/sessions/<session_id>`
verified canvas access for `agent_id` in the URL but loaded/deleted
sessions only by `session_id`, without checking `conv.dialog_id ==
agent_id`.
Any user with access to **any** agent could read or delete another
agent's `API4Conversation` session (messages, references, DSL, etc.)
when they knew the session UUID.
Agent completions in the same file already enforce this binding; chat
sessions do too — these two routes were inconsistent.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
- [ ] New Feature (non-breaking change which adds functionality)
- [ ] Documentation Update
- [ ] Refactoring
- [ ] Performance Improvement
- [ ] Other (please describe):
### Changes
| File | Change |
|------|--------|
| `api/apps/restful_apis/agent_api.py` | Require `conv.dialog_id ==
agent_id` in `get_agent_session` and `delete_agent_session_item`; return
generic `"Session not found!"` on mismatch |
| `test/unit_test/api/apps/restful_apis/test_get_agent_session.py` | Add
IDOR regression tests for GET/DELETE; fix success fixture to include
`dialog_id`; track `delete_by_id` calls |
### Test plan
- [x] Unit tests added for GET/DELETE IDOR and success paths
- [ ] `pytest
test/unit_test/api/apps/restful_apis/test_get_agent_session.py`
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Zhichang Yu <yuzhichang@gmail.com>
## Summary
Fixes the agent `Switch` component matching an **empty/all-skipped
condition** unconditionally because `all([]) is True`.
## Root cause
`res` only accumulates for items with a non-empty `cpn_id` (blank ones
`continue`). For a condition with empty `items` (or all-blank `cpn_id`),
`res == []`, and `if all(res):` is `True`, so the Switch routes to that
condition's `to` target before reaching the else/`end_cpn_ids` branch.
## Fix
```diff
- if all(res):
+ if res and all(res):
```
An empty result set no longer counts as a match; genuinely-satisfied
"and" conditions still route (the real `all(res)` path is preserved).
## Files changed
- `agent/component/switch.py`
- `test/unit_test/agent/component/test_switch_empty_condition.py` (new)
## Verification
- `ruff check` / `ruff format --check` — clean
- Added unit tests (mirroring the existing `_FakeCanvas` component-test
pattern): an empty/all-skipped "and" condition now falls through to
`end_cpn_ids`; a genuinely-satisfied "and" condition still routes to its
target.
- Local full pytest not run (heavy RAG deps); CI validates.
## Note
Implemented with LLM assistance (model: claude-opus-4-8).
Closes#15643
---------
Co-authored-by: seekmistar01 <seekmistar01@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Co-authored-by: Zhichang Yu <yuzhichang@gmail.com>
### What problem does this PR solve?
Closes#15608.
The ExeSQL agent tool (`agent/tools/exesql.py`) opens database
connections to a node-author-controlled host/port with no SSRF
validation. The sibling `test_db_connection` endpoint already validates
the host via `common.ssrf_guard.assert_host_is_safe` (added by PR
#14860), but the tool that actually performs the connection at agent run
time was left unguarded — so the guard is bypassed simply by running the
agent. An agent author can point the host at `127.0.0.1`,
`169.254.169.254` (cloud metadata), or any internal RFC1918 host/port,
turning ExeSQL into an internal port-scanner / metadata-fetch primitive.
### Fix
Mirror the accepted endpoint guard: validate (and resolve) the host
once, before the `db_type` dispatch, and connect to the validated public
IP so a later DNS change cannot rebind the host to an internal address.
- Add `from common.ssrf_guard import assert_host_is_safe`.
- `safe_host = assert_host_is_safe(self._param.host)` before the
dispatch (rejects loopback, link-local/metadata, RFC1918, and
unresolvable hosts).
- Substitute the validated IP into all 6 driver branches: mysql/mariadb,
oceanbase, postgres, mssql, trino, IBM DB2.
Adds `test/unit_test/agent/tools/test_exesql_ssrf.py` covering loopback,
link-local/metadata, RFC1918, and empty-host rejection (before any
connection), plus an allowed host dialing the validated IP.
### Validation
- `python3 -m py_compile agent/tools/exesql.py`
- `ruff check agent/tools/exesql.py
test/unit_test/agent/tools/test_exesql_ssrf.py`
- `pytest test/unit_test/agent/tools/test_exesql_ssrf.py` — 5 passed
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
---------
Co-authored-by: Zhichang Yu <yuzhichang@gmail.com>
### What problem does this PR solve?
This PR adds an Agent LLM setting to control thinking mode for official
providers that expose a thinking switch.
Related to #12842.
Closes#15445.
Some providers expose thinking controls through provider-specific
request fields, but Agent LLM settings did not have a unified option for
users to enable or disable thinking mode.
This PR adds a `Thinking` selector with:
- System default
- Enabled
- Disabled
<img width="452" height="278" alt="8566b0b4-0546-4c8a-913d-f9bbd38319f6"
src="https://github.com/user-attachments/assets/25b497f7-1ba0-4bfe-940d-6fe79287d6ab"
/>
<img width="471" height="971" alt="8a0a6bee-f45f-48d5-bd83-17af260de3db"
src="https://github.com/user-attachments/assets/41ad43c1-5087-48f1-bf37-f2ca14c2be2f"
/>
Initial support is limited to the verified official providers:
- Qwen / DashScope: `enable_thinking`
- Kimi / Moonshot: `thinking.type`
- GLM / ZHIPU-AI: `thinking.type`
For LiteLLM-based providers, provider-specific fields are forwarded
through `extra_body` before `drop_params` filtering so the request
parameters are preserved.
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
---------
Co-authored-by: jiashi <jiashi19@outlook.com>
Co-authored-by: Zhichang Yu <yuzhichang@gmail.com>
## Summary
#16332 fixed the missing `return` in DeepL's except branch, but
`ComponentBase.be_output` was removed during the agent refactor (#9113)
while several components still call it. DeepL (and other tools) would
raise `AttributeError` before any error message could be returned.
- Restore `ComponentBase.be_output` as `pd.DataFrame([{"content": v}])`
(same as pre-refactor behavior)
- Add regression test that `_run` returns the `**Error**:` message when
translation fails
Related to #16329
## Test plan
- [x] `test_run_returns_error_on_translation_failure`
- [x] Existing `test_deepl.py` check() tests still pass
---------
Co-authored-by: Harsh Kashyap <harshkashyap@Harshs-MacBook-Pro.local>
Co-authored-by: Zhichang Yu <yuzhichang@gmail.com>
### What problem does this PR solve?
Closes#15435
Several agent tools call external HTTP APIs through `requests` with no
request timeout. When an upstream host accepts the connection but never
responds (a slow or overloaded API, a half open connection, a stuck load
balancer), the call blocks forever. These tools run inside agent canvas
execution, so a single stalled socket freezes the entire agent run with
no recovery.
Ten call sites were affected:
- `agent/tools/qweather.py` (4 calls)
- `agent/tools/jin10.py` (4 calls)
- `agent/tools/tushare.py` (1 call)
- `agent/tools/github.py` (1 call)
The `github.py` tool already carried the `@timeout` decorator from
`common/connection_utils.py`, but that does not protect against this
case. In the default configuration the decorator waits on its result
queue with no timeout, and a daemon thread blocked inside a socket read
cannot be killed, so the run still hangs. The per request timeout added
here is what actually bounds the call.
This is the same bug class as the merged Go stream timeout fix,
surfacing in the Python tool layer.
Changes:
- Pass `timeout=DEFAULT_TIMEOUT` on all 10 calls, reusing the existing
shared constant in `common/http_client.py` (configurable via
`HTTP_CLIENT_TIMEOUT`) so there is one source of truth rather than
scattered literals.
- Add an AST based unit test at
`test/unit_test/agent/tools/test_http_timeout.py` that scans every tool
module and fails if any `requests` or `httpx` request call omits a
`timeout`, guarding current and future call sites.
Verification:
- Reproduced the indefinite block against a stalling local server, and
confirmed that adding a timeout raises `ReadTimeout` promptly.
- Confirmed the `@timeout` decorator does not interrupt a blocked no
timeout request in its default configuration.
- The new test flags exactly the 10 original call sites on the pre fix
code and passes (22 modules) after the fix.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
- [ ] New Feature (non-breaking change which adds functionality)
- [ ] Documentation Update
- [ ] Refactoring
- [ ] Performance Improvement
- [ ] Other (please describe):
---------
Co-authored-by: Zhichang Yu <yuzhichang@gmail.com>
### What problem does this PR solve?
Closes#14773.
Today, Pipeline (`rag/flow/`) chunking strategies only run as part of a
dataset ingestion that always embeds and indexes the result. There is no
way to drive Pipeline-style chunking from an Agent workflow without
paying that vectorization/persistence cost.
This PR adds a single new Agent component, `PipelineChunker`, that:
- Takes one or more file references (from `Begin` / `UserFillUp`
uploads) as input.
- Runs the existing `rag.app.*` chunking strategies (`naive`, `paper`,
`qa`, `manual`, `book`, `presentation`, `laws`, `table`, `one`, `email`,
`picture`, `audio`, `resume`, `tag`) against each file.
- Emits the resulting chunks as `chunks: list[str]` and `chunks_full:
list[dict]` for downstream Agent nodes.
- Performs **no embedding and no persistence** — chunks live only in
canvas variables for the duration of the run, exactly as requested in
the issue.
The component is auto-discovered by `agent/component/__init__.py`; no
registry edits required. Chunker functions are imported lazily so the
component itself does not pull `deepdoc` / OCR / VLM at
component-discovery time. File resolution mirrors the existing
`ExcelProcessor` convention.
Out of scope for this PR (potential follow-ups):
- Vectorization / KB persistence (explicit ask in the issue).
- Frontend canvas UI for the new component.
- Bridging to the newer Pydantic-based `rag/flow/chunker/TokenChunker`
(consumes a parser node's structured output rather than a raw file — a
separate, larger feature).
### Type of change
- [ ] Bug Fix (non-breaking change which fixes an issue)
- [x] New Feature (non-breaking change which adds functionality)
- [ ] Documentation Update
- [ ] Refactoring
- [ ] Performance Improvement
- [ ] Other (please describe):
---
## Files changed
- `agent/component/pipeline_chunker.py` — new component (~180 lines)
- `test/unit_test/agent/test_pipeline_chunker.py` — unit tests (~120
lines)
## Test plan
- [x] `ruff check` on changed files — clean.
- [x] `ruff format` applied to the new component file.
- [x] `python -m py_compile` on both new files — both compile.
- [x] New unit test file carries `pytestmark = pytest.mark.p2` so it
runs under marker-filtered CI.
- [x] Every new function, method, and class has a docstring (CodeRabbit
80% docstring-coverage gate).
- [x] `python -m pytest test/unit_test/agent/test_pipeline_chunker.py -x
-q` — **7 passed in 1.95s** locally. Tests stub
`api.db.services.file_service` and `rag.app.*` so they exercise the
parameter validation and parser-id lookup table without requiring the
full backend / model stack.
## Manual integration plan (post-merge)
1. Drop the component into an Agent canvas after a `Begin` node with a
file input.
2. Set `parser_id = "naive"` (or any other strategy) and reference the
file input in `inputs`.
3. Wire the `chunks` output into a downstream `LLM` / `Message` /
`Iteration` node — chunks are available as plain text without any
embedding or KB write.
Co-authored-by: John Baillie <johnbaillie2007@gmail.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-authored-by: Zhichang Yu <yuzhichang@gmail.com>
## Summary
- harden reopened advisory fixes across REST connector, invoke, document
downloads, and markdown rendering
- add targeted regression coverage for redirect-safe SSRF handling,
invoke SSRF checks, document access control, and markdown sanitization
- verify each referenced GHSA against the original GitHub advisory text
and align the closed-advisory plan with the implemented remediation
## What changed
- add tenant access checks to document download endpoints to avoid
cross-tenant document disclosure
- add per-hop SSRF validation, DNS pinning, redirect handling, and
redirect limits to the REST API connector
- ensure invoke requests validate and pin the resolved host and never
follow redirects implicitly
- keep the generic rate-limited request path wrapped, not just GET and
POST helpers
- sanitize markdown HTML before rendering in the highlight markdown
component
## Validation
- `cd web && npm test -- --runInBand
src/components/highlight-markdown/__tests__/index.test.tsx`
- `.venv/bin/python -m pytest -q
test/unit_test/data_source/test_rest_api_connector.py`
- targeted `test/testcases/test_web_api/...` unit additions were
reviewed, but the suite cannot be executed end-to-end in this
environment because parent `test/testcases/conftest.py` requires a local
service on `127.0.0.1:9380`
## Notes
- all GHSA entries referenced by the plan were checked against the
original GitHub advisory text, not sampled
- the closed-advisory plan document was updated locally during review,
but is intentionally not included in this PR
## Problem
The CodeQL Go analysis was failing on the entire codebase with:
fatal error: office_oxide.h: No such file or directory
because six ingestion parser files (`doc`, `docx`, `ppt`, `pptx`, `xls`,
`xlsx`) import `github.com/yfedoseev/office_oxide/go`, a CGO binding to
a Rust library. The CodeQL runner image doesn't ship the
`office_oxide.h` native header, so the Go AST build aborts before CodeQL
can analyze anything.
This means **no Go-language alerts have been re-evaluated** since the
suppression comments were added in #16407 and #16408. The most recent
CodeQL run fixed 51 alerts (all Python/JS), but every Go alert stayed
open, including ones in files that have nothing to do with office_oxide.
## Fix
Add a `.github/codeql/codeql-config.yml` that uses `paths-ignore` to
skip the six parser files. The rest of the Go tree is pure Go (no CGO)
and analyzes cleanly.
The parser files are also excluded from local `go test` / `go build`
when the office_oxide C library isn't installed, so this brings CodeQL
in line with the existing toolchain.
## Expected outcome
After this PR merges, the next CodeQL run on main will:
1. Complete successfully (Go analysis no longer aborts)
2. Re-evaluate the alerts in the remaining files
3. Match the existing `// codeql[go/...] suppression comments` added in
#16407 and #16408
4. Close those alerts
This should drop the open-alert count from 44 to near zero (the 6 Python
clear-text-logging and 1 JS prototype-pollution alerts that were added
in #16408 will also be re-evaluated).
## Why not just install office_oxide in the CodeQL runner?
- The `office_oxide` Go binding is a 3rd-party module
(`github.com/yfedoseev/office_oxide/go`) with CGO that pulls in a Rust
crate
- The CodeQL runner uses a stock Go toolchain that doesn't include the C
library
- Installing it would require modifying the GitHub-managed CodeQL
workflow, which is owned by GitHub and not easily customizable
- The parsers are also unimplemented stubs (each `Parse` function logs
the filename and returns `nil` after my earlier clear-text-logging fix),
so they have no security-relevant code to scan anyway
🤖 Generated with [Claude Code](https://claude.com/claude-code)
## Summary
After #16407 merged, 44 of the original 93 CodeQL alerts were still open
on the default branch. This PR closes the remaining ones by:
1. **Moving 32 existing `// codeql[...]` directives** so they sit on the
line **immediately before** the suppressed statement. The original
multi-line suppression blocks had the directive as the first line, with
the rationale on subsequent lines. After line shifts (refactors, linter
reformat), the directive ended up several lines above the alert location
— CodeQL only recognizes the suppression when it appears on the line
directly above. (32 alerts across 27 files.)
2. **Adding 9 new `// codeql[...]` suppressions** for alerts that had no
suppression in the preceding lines at all — mostly real-fixes that
CodeQL conservatively still flags (filepath.Base, bounded slice sizes,
model-identifier strings, the MD5-legacy-migration lookup in
`conversation_service.py`).
## Files changed
- `api/db/services/conversation_service.py` — add
`py/weak-sensitive-data-hashing` suppression (MD5 for backward-compat
legacy row lookup; not used for auth)
- `api/db/services/llm_service.py` — 3×
`py/clear-text-logging-sensitive-data` suppressions on the lines that
log `llm_name` in warnings/info
- `common/misc_utils.py` — 2× `py/clear-text-logging-sensitive-data`
suppressions on the redacted `current_url` log sites
- `internal/agent/component/invoke.go` — moved existing
`go/request-forgery` directive
- `internal/agent/sandbox/ssh.go` — moved existing
`go/command-injection` directive
- `internal/agent/tool/retrieval_service.go` — added
`go/uncontrolled-allocation-size` suppression (`topN` is bounded to 1024
above)
- `internal/cli/common_command.go` — moved 2×
`go/disabled-certificate-check` directives
- `internal/cli/user_command.go` — added `go/clear-text-logging`
suppression (filepath.Base already strips user-identifying path)
- `internal/dao/pipeline_operation_log.go` — moved 2× `go/sql-injection`
directives
- `internal/dao/user_canvas.go` — added `go/sql-injection` suppression
in `GetList` (the new `userCanvasOrderClause` call path)
- `internal/engine/infinity/chunk.go` — moved existing
`go/unsafe-quoting` directive
- `internal/entity/models/*` — moved `go/path-injection` directives (15
files)
- `internal/handler/oauth_login.go` — moved existing
`go/cookie-httponly-not-set` directive
- `internal/handler/tenant.go` — moved existing `go/path-injection`
directive
- `internal/service/deep_researcher.go` — moved existing
`go/unsafe-quoting` directive
- `internal/service/dataset.go` — added
`go/uncontrolled-allocation-size` suppression (`n` bounded to 1024
above)
- `internal/service/file.go` — moved existing `go/request-forgery`
directive
- `internal/service/langfuse.go` — moved 2× `go/request-forgery`
directives
- `internal/utility/mcp_client.go` — moved 3× `go/request-forgery`
directives
- `internal/utility/smtp.go` — moved existing `go/email-injection`
directive
- `rag/prompts/generator.py` — added
`py/clear-text-logging-sensitive-data` suppression
- `web/.../use-provider-fields.tsx` — added
`js/prototype-pollution-utility` suppression (FORBIDDEN_KEYS guard is on
the line above)
## Why the previous PR left alerts open
`// codeql[query-id] explanation` must be on the line **immediately
before** the suppressed statement per the [GitHub CodeQL suppression
spec](https://docs.github.com/en/code-security/code-scanning/automatically-scanning-your-code-for-vulnerabilities-and-errors/customizing-code-scanning-with-codeql/suppressing-code-scanning-alerts).
The original suppression blocks were 4-5 lines, with the directive as
the **first** line. After linter reformat / line shifts, the directive
ended up too far above the actual alert line to be recognized. The fix
is to put the directive on the line directly above the suppressed
statement, with the rationale above it.
## Test plan
- All 9 modified Python files `ast.parse` clean
- All 4 modified Go files `gofmt` clean
- 36/44 expected alert suppressions in place
- 8 remaining CodeQL alerts are the originals (#3485851828, #3485851831,
#3485869759, #3485869766, #3485869768, #3485869771, #3485885962,
#3485895527) which were resolved by the corresponding commit comments;
these should close on the next scan when the suppression comments match
the alert lines.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
## Summary
Ports five Python agent APIs to Go under the v1 Gin router:
- `GET /api/v1/agents/attachments/<attachment_id>/download`
- `POST /api/v1/chatbots/<dialog_id>/completions` (SSE)
- `GET /api/v1/chatbots/<dialog_id>/info`
- `POST /api/v1/agentbots/<agent_id>/completions` (SSE)
- `GET /api/v1/agentbots/<agent_id>/inputs`
Mirrors the existing Python wire shape (`{code, message,
data:{answer,reference,...}}` per Python `canvas_service.completion`) so
the iframe SDK and existing JS widgets keep working.
## Behavioural parity with Python
| # | Concern | How it's met |
|---|---------|--------------|
| R0 | Bot routes must not require regular user session | Routes mount
on `apiNoAuth` (router.go:198-202), with `BetaAuthMiddleware` only |
| R3 | Two SSE formats in Go drift | F2: `AgentChatCompletions` and
`AgentbotCompletion` share `service.WriteChatbotRunEvent` |
| R7 | `GetBySessionID` returns `(nil, nil)` on miss | Defensive
nil-check before `session.UserID != tenantID` |
| R8 | Begin component name vs ID | `FindBeginComponentID` resolves name
→ ID first, then `ExtractComponentInputForm(dsl, beginID)` |
| R9 | Defensive PromptConfig parsing | `stringFromMap` helper used for
`prologue` and `tavily_api_key` |
| R10 | `BetaAuthMiddleware` Bearer-prefix pre-filter | Removed —
`GetUserByToken` is called unconditionally, falls back to
`GetUserByBetaAPIToken` |
| F8 | Multi-turn chatbot history | `ChatbotCompletion` reads prior
turns from `session.Message`, appends user turn, calls LLM, persists new
pair via new `API4ConversationDAO.Update` |
| F9 | UUID gate stricter than plan | Removed — only `filepath.Base` +
CR/LF/quote header sanitization remains |
| H2 | Defence-in-depth IDOR | `AgentbotCompletion` calls `loadCanvas`
before delegating to `RunAgent` |
| M2 | SSE error leakage | `WriteChatbotFrame` emits generic `"an
internal error occurred"`; real error logged via `common.Error` |
## Verification
```bash
$ go vet ./... # clean (only pre-existing issues)
$ go build ./... # success
$ go test ./internal/handler/ ./internal/service/ ./internal/agent/dsl/ ./internal/common/ ./internal/dao/
ok ragflow/internal/handler 0.617s
ok ragflow/internal/service 1.729s
ok ragflow/internal/agent/dsl 0.008s
ok ragflow/internal/common 0.087s
ok ragflow/internal/dao 0.083s
```
1199 tests pass across 5 packages.
## Known follow-ups (out of scope for this PR)
- **F1**: token-level streaming in `ChatbotCompletion` (currently emits
one frame per turn)
- **F3**: per-route `auth_types` attribute in Go (currently applied via
route group middleware)
---------
Co-authored-by: Claude <noreply@anthropic.com>
## Summary
- Added Go API route `PUT /api/v1/chats/:chat_id` to align with Python
`PUT /api/v1/chats/<chat_id>` chat update behavior.
- Added Go API route `PATCH /api/v1/chats/:chat_id` to align with Python
`PATCH /api/v1/chats/<chat_id>` partial chat update behavior.
- Added matching handler and service logic for owner checks, tenant
validation, persisted-field filtering, read-only field filtering,
`dataset_ids` to `kb_ids` conversion, and PATCH shallow merge semantics
for `prompt_config` and `llm_setting`.
### What problem does this PR solve?
RAGFlow(api/default)> show admin server;
RAGFlow(api/default)> show api server 'default';
RAGFlow(admin)> show admin server;
RAGFlow(admin)> show api server 'default';
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
---------
Signed-off-by: Jin Hai <haijin.chn@gmail.com>
## Summary
Add support for **"New API"** as a model provider, enabling connection
to [New API](https://github.com/QuantumNous/new-api) /
[one-api](https://github.com/songquanpeng/one-api) compatible gateways
that aggregate multiple LLM backends behind a unified OpenAI-compatible
`/v1` endpoint.
### Features
- **All model types**: Chat, Embedding, Rerank, Image2Text, TTS,
Speech2Text
- **List Models discovery**: `NewAPI(OpenAIAPICompatible)` class in
`model_meta.py` queries the gateway's `/v1/models` to auto-discover
available models via the native `GET /api/v1/providers/<name>/models`
endpoint
- **Model parameter editing**: Pencil icon on each discovered model row
to edit `model_type`, `max_tokens`, and `features` (e.g. tool call
support) before submitting
- **Custom model addition**: "Add Custom Model" button at the bottom of
the List Models dropdown for models not returned by the API
- **Gear icon settings**: Enabled the Settings gear button on provider
instances to manage models on existing instances (viewMode)
- **viewMode credential passthrough**: Fixed List Models in viewMode —
merges `initialValues` credentials when `api_key`/`base_url` fields are
hidden by `hideWhenInstanceExists`
### Changes
**Backend** (8 files):
- `rag/llm/chat_model.py` — `NewAPIChat(Base)` class
- `rag/llm/embedding_model.py` — `NewAPIEmbed(OpenAIEmbed)` class (no
auto `/v1` append)
- `rag/llm/rerank_model.py` — `NewAPIRerank(Base)` class (uses `/rerank`
endpoint)
- `rag/llm/cv_model.py` — `NewAPICv(GptV4)` class
- `rag/llm/tts_model.py` — `NewAPITTS(OpenAITTS)` class
- `rag/llm/sequence2txt_model.py` — `NewAPISeq2txt(GPTSeq2txt)` class
- `rag/llm/model_meta.py` — `NewAPI(OpenAIAPICompatible)` class for List
Models discovery
- `conf/llm_factories.json` — New API factory entry with all model type
tags
**Frontend** (8 files + 1 new SVG):
- `web/src/assets/svg/llm/new-api.svg` — New API logo icon
- `web/src/constants/llm.ts` — `LLMFactory.NewAPI` enum + `IconMap`
entry
- `web/src/components/svg-icon.tsx` — `NewAPI` added to `svgIcons`
-
`web/src/pages/user-setting/setting-model/modal/provider-modal/field-config/local-llm-configs.ts`
— New API `buildLocalConfig`
-
`web/src/pages/user-setting/setting-model/modal/provider-modal/constants.ts`
— `LIST_MODEL_PROVIDERS` includes NewAPI
- `web/src/pages/user-setting/setting-model/components/used-model.tsx` —
Enable Settings gear button
-
`web/src/pages/user-setting/setting-model/modal/provider-modal/hooks/use-list-models-picker.ts`
— viewMode credential merge + model editing state/handlers
-
`web/src/pages/user-setting/setting-model/modal/provider-modal/hooks/use-list-models-options.tsx`
— Pencil edit icon per model row
-
`web/src/pages/user-setting/setting-model/modal/provider-modal/index.tsx`
— `AddCustomModelDialog` import + edit dialog rendering
**Note on Go implementation**: A Go model driver (`NewAPIModel`
delegating to `OpenAIModel`) has been prepared but is deferred until the
Go runtime is enabled in a future release (current v0.26.0 images use
`API_PROXY_SCHEME=python` and do not compile Go binaries). Will submit
as a follow-up PR.
## Related
- Depends on: #15996 (provider instance API improvements — server-side
credential lookup, idempotent `add_model`, security fixes — required for
viewMode gear icon and batch model submission)
## Test plan
- [ ] Add New API provider with api_key and base_url pointing to an
OpenAI-compatible gateway
- [ ] Click "List Models" — should discover and display available models
from `/v1/models`
- [ ] Click pencil icon on a model — should open edit dialog to change
model_type, max_tokens, features
- [ ] Select multiple models and click OK — should add all selected
models
- [ ] Click gear icon on the added instance — should open viewMode with
List Models working
- [ ] In viewMode, select new models including pre-existing ones, click
OK — should succeed (requires #15996)
- [ ] Verify all model types work: create a Chat assistant, Embedding
KB, Rerank setting
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Tim Wang <wanghualoong@users.noreply.github.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
### What problem does this PR solve?
```
RAGFlow(api/default)> show model 'WiseDiag-Z1 Think';
RAGFlow(api/default)> list models;
RAGFlow(admin)> show model 'WiseDiag-Z1 Think';
RAGFlow(admin)> list models;
```
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
Signed-off-by: Jin Hai <haijin.chn@gmail.com>
### What problem does this PR solve?
```
RAGFlow(api/default)> show var 'mail.port';
+-----------+-----------+--------------+-------+
| data_type | name | setting_type | value |
+-----------+-----------+--------------+-------+
| integer | mail.port | config | 30 |
+-----------+-----------+--------------+-------+
```
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
---------
Signed-off-by: Jin Hai <haijin.chn@gmail.com>
### What problem does this PR solve?
```
RAGFlow(api/default)> show provider 'zhipu-ai'
RAGFlow(api/default)> show provider 'zhipu-ai' instance 'test';
RAGFlow(api/default)> show provider 'zhipu-ai' instance 'test' balance;
RAGFlow(api/default)> show provider 'zhipu-ai' model 'glm-4.5';
```
### Type of change
- [x] Refactoring
---------
Signed-off-by: Jin Hai <haijin.chn@gmail.com>
## Summary
- Align Langfuse API key set/get/delete behavior with the Python
implementation.
- Improve DAO handling for Langfuse credential save/delete flows.
- Add tests for Langfuse service error handling and API key lifecycle
behavior.
### What problem does this PR solve?
As title
/api/v1/connectors/<connector_id> PATCH was implemented in #15512
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
- [x] Refactoring
### What problem does this PR solve?
As title:
implement:
```
/api/v1/messages/search GET
/api/v1/messages GET
/api/v1/messages/<memory_id>:<message_id>/content GET
/api/v1/memories/<memory_id>/config GET
/api/v1/messages/<memory_id>:<message_id> PUT
```
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
### Bug
`RAGFlowHtmlParser.chunk_block()` splits an oversized block by slicing
the **tokenized** string and storing the joined tokens:
```python
tks_str = rag_tokenizer.tokenize(block)
...
tokens = tks_str.split(" ")
while start < len(tokens):
chunks.append(" ".join(tokens[start:start + chunk_token_num])) # tokenized form, not source
```
On the default (Elasticsearch) backend `rag_tokenizer.tokenize`
transforms text: it lowercases/stems Latin words and inserts spaces
between CJK characters. So any text block longer than `chunk_token_num`
is stored as garbled, lowercased, space-segmented text instead of the
source content. The small-block branch correctly stores the original
`block`, so only oversized blocks are corrupted. Affects HTML and EPUB
ingestion (both go through `chunk_block`), degrading retrieved chunks
and the answers generated from them.
### Real tokenizer behavior (infinity-sdk 0.7.0, ES backend)
```
tokenize("Hello World FOO Bar Baz Qux Jumps") -> "hello world foo bar baz qux jump" # lowercased + stemmed
tokenize("你好世界这是一个测试") -> "你好世界 这 是 一个 测试" # spaces inserted
```
### Fix
Split the **original** text: break it into atoms (whitespace-delimited
runs for space-separated scripts, per-character for spaceless scripts
such as Chinese) and pack them into pieces of at most `chunk_token_num`
tokens. This preserves the source characters and still splits scripts
that have no whitespace — a plain whitespace split would leave CJK as
one un-splittable chunk.
### Proof (real tokenizer, before/after)
Running the old vs new split against the real `infinity.rag_tokenizer`:
```
ENGLISH "Hello World FOO Bar Baz Qux Lazy Dogs" (chunk_token_num=4)
OLD: ['hello world foo bar', 'baz qux jump over', 'lazi dog'] # lowercased + stemmed
NEW: ['Hello World FOO Bar ', 'Baz Qux Jumps Over ', 'Lazy Dogs'] # preserved; each <= 4 tokens
NEW preserves text exactly: True
CHINESE "你好世界这是一个测试用例需要被切分成多个块" (chunk_token_num=3)
OLD: ['你好世界 这 是', '一个 测试用例 需要', ...] # spurious spaces
NEW: ['你好世', '界这是', '一个测', ...] # preserved; each <= 3 tokens
NEW preserves text exactly: True
```
### Tests
Added `test/unit_test/deepdoc/parser/test_html_parser.py` (English +
Chinese oversized blocks, plus small-block merge). Before the fix the
two oversized tests fail (English shows lowercasing, Chinese shows
inserted spaces); after the fix all pass. `ruff check` clean.
### What problem does this PR solve?
`DeepLParam.check()` validated `self.top_n`, but DeepL has no such
parameter (it is not defined on the param class or its base), so
`check()` always raised `AttributeError` and a DeepL component could
never pass validation. Removed the bogus `top_n` check.
Also fixed the `_run` except branch, which computed
`be_output("**Error**...")` but never returned it, silently dropping the
error message.
Closes#16329
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
- [x] Add test cases
### Testing
Added `test/unit_test/agent/component/test_deepl.py` covering
`DeepLParam.check()` with valid defaults and rejection of invalid
source/target languages.
### What problem does this PR solve?
Fixes the PubMed tool always emitting `Authors: Unknown Authors`. The
`safe_find` closure in `_format_pubmed_content` was hardcoded to search
from the article root, so the per-author `LastName`/`ForeName` lookups
never matched.
`safe_find` now accepts an optional `base` node (defaults to `child`,
preserving the existing field lookups), and the author loop passes the
current `<Author>` element.
Closes#16328
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
- [x] Add test cases
### Testing
Added `test/testcases/test_web_api/test_canvas_app/test_pubmed_unit.py`
covering per-author parsing, intact title/journal/DOI fields, and the
no-authors fallback.
Before: `Authors: Unknown Authors`
After: `Authors: Furqan Khan, Jane Smith`
## Summary
Add the Go implementation of `POST
/api/v1/datasets/{dataset_id}/documents/{document_id}/chunks`.
This wires the full create-chunk path in Go:
- router and handler registration
- request/response structs
- chunk creation service logic
- embedding generation
- chunk insert into doc engine
- chunk/token counter increment
- `tag_feas` validation
- `image_base64` decoding and chunk image storage/merge
- unit tests for handler and service
## Testing
Unit tests:
- `/usr/local/go/bin/go test ./internal/handler`
- `/usr/local/go/bin/go test ./internal/service/chunk`
- `/usr/local/go/bin/go test ./internal/service`
- `/usr/local/go/bin/go test ./...`
All passed locally.
Manual curl checks:
- basic text chunk: Go passed
- chunk with `important_keywords` / `questions` / `tag_kwd` /
`tag_feas`: Go passed
- blank content validation: Go matched expected `code=102`
- invalid `image_base64` validation: Go matched expected `code=102`
- image upload and repeated image upload / merge path: Go passed twice
### What problem does this PR solve?
```
RAGFlow(api/default)> list dataset 'ccc' files;
Total: 1
```
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
Signed-off-by: Jin Hai <haijin.chn@gmail.com>
## Summary
Migrated the dataset document upload API (`POST
/api/v1/datasets/:dataset_id/documents`) from Python to the Go backend.
It supports local file uploads (`type=local`), web page ingestion
(`type=web`), and empty document creation (`type=empty`).
## Changes
- **Router**: Registered `POST /api/v1/datasets/:dataset_id/documents`
route.
- **Handler**: Implemented `UploadDocuments` handler and its routing
functions (`uploadLocalDocuments`, `uploadWebDocument`,
`uploadEmptyDocument`).
- **Service**: Implemented `UploadLocalDocuments`, `UploadWebDocument`,
and `UploadEmptyDocument` in `DocumentService`.
- **Refactoring**: Moved permission checking logic to a shared helper
for reuse in file and document services.
- **Tests**: Added comprehensive unit tests for the new handler and
service upload paths.
## Verification
Ran and passed the test suite for service and handler packages:
- `go test ./internal/service`
- `go test ./internal/handler`
### What problem does this PR solve?
- added the new dataset search route and handler
- reused the existing shared SearchDatasets service by adapting
single-dataset requests into dataset_ids=[dataset_id]
- aligned handler error responses with Python behavior for argument/data
errors
- aligned key service error messages such as invalid search_id and mixed
embedding models
- added focused handler and service tests for request mapping and error
behavior
### Tests:
`/usr/local/go/bin/go test ./internal/service -run
'TestSearchDatasetRequestToSearchDatasetsRequest|TestDatasetServiceSearchDatasets'`
`/usr/local/go/bin/go test ./internal/handler -run
'TestDatasetsHandlerSearchDataset'`
## Problem
The Wikipedia tool silently swallows all exceptions with `except
Exception: pass`, making it impossible to debug failures when fetching
Wikipedia pages.
## Fix
Replace the bare `except Exception: pass` with specific exception
handling:
- `DisambiguationError`: log available options
- `PageError`: log page not found
- `Exception`: log unexpected errors with full traceback
Co-authored-by: wills <willsgao@163.com>
Co-authored-by: Zhichang Yu <yuzhichang@gmail.com>
Adds Keenable as a web search tool in the agent, alongside the existing
Tavily/DuckDuckGo/SearXNG/Google tools.
The main difference from the other search tools is that it doesn't
require an
API key. By default it uses Keenable's keyless public endpoint, so it
works out
of the box. Providing a key (in the tool config) switches to the
authenticated
endpoint and lifts the rate limits.
### Changes
- Backend: `agent/tools/keenable.py` — `KeenableSearch`, follows the
Tavily/DuckDuckGo tool shape (results go through `_retrieve_chunks`).
Auto-registered by `agent/tools/__init__.py`.
- Frontend: wired into the agent builder — operator + icon, config form
(optional API key, search mode, site filter, top N), the search tool
menu,
and the existing api_key export sanitizer.
### Config
- API key: optional. Blank = keyless free tier; set it to lift limits /
enable
`realtime` mode.
- `site`: restrict to a single domain.
- `mode`: `pro` (default) or `realtime`.
### Notes
`KEENABLE_API_URL` can override the API base (HTTPS enforced; defaults
to
`https://api.keenable.ai`). The tool only sends the query (no URL
fetch), so
there's no SSRF surface. Verified the frontend with `vite build` and the
backend search path against the public endpoint.
### What problem does this PR solve?
- OpenTelemetry integration
- Checkpoint conformance tests
- State inspector API
- Callbacks
- A series of fault injection tests
- Pregel integration tests
### Type of change
- [x] Refactoring
### What problem does this PR solve?
```
RAGFlow(api/default)> set key 'ragflow-JgnarFSCUiV99oOvvMDei7ZzZg1cVlqGd1AMHrHeKE4';
SUCCESS
RAGFlow(api/default)> unset key;
SUCCESS
RAGFlow(api/default)> list provider 'zhipu-ai' instances;
RAGFlow(api/default)> list providers;
RAGFlow(api/default)> list available providers;
RAGFlow(api/default)> list provider 'zhipu-ai' instance 'test' models;
```
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
---------
Signed-off-by: Jin Hai <haijin.chn@gmail.com>
## Summary
- add `GET /api/v1/datasets/:dataset_id/tags`
- add `PUT /api/v1/datasets/:dataset_id/tags`
- implement dataset tag listing and rename flow
- align rename tag validation and response shape with the Python API
- add handler and service tests for dataset tags
## Routes
- `GET /api/v1/datasets/:dataset_id/tags`
- `PUT /api/v1/datasets/:dataset_id/tags`
## Test
- Run specific tests for dataset tags:
```
go test -v ./internal/service ./internal/handler -run 'TestDatasetServiceListTags|TestDatasetServiceRenameTag|TestDatasetsHandlerListTags|TestDatasetsHandlerRenameTag'
```
- Run all tests for service and handler to verify no regressions:
```
go test ./internal/service ./internal/handler
```
- use curl cmd to test
## Summary
Align the Go implementations of these APIs with the Python behavior:
- `POST /api/v1/datasets/:dataset_id/metadata/update`
- `PATCH /api/v1/datasets/:dataset_id/documents/metadatas`
- `POST /api/v1/documents/upload`
## What changed
- Added the Go routes and handlers for the 3 APIs.
- Aligned batch document metadata updates with Python semantics:
- support `match` in update items
- support list append / replace behavior
- support deleting specific list values
- remove metadata entirely when it becomes empty
- create metadata for documents that previously had none when updates
apply
- count `updated` only when a document actually changes
- Aligned `documents/upload` file uploads with Python-style
`upload_info` behavior:
- store upload-info blobs in the per-user downloads bucket
- return lightweight upload descriptors instead of normal
file-management responses
- Improved URL upload behavior:
- SSRF-guarded fetch with redirect validation
- redirect limit aligned to Python behavior
- normalize filename and MIME type
- add `.pdf` when the fetched content is PDF
- normalize HTML content into readable text instead of storing raw HTML
shells
## Validation
### Unit tests
Passed:
- `go test ./internal/service`
- `go test ./internal/handler`
Also verified targeted cases for:
- batch metadata update semantics
- upload_info URL handling
- upload_info download bucket behavior
### curl checks
Verified the new Go endpoints with `curl` and compared the response
shape and behavior with Python for:
- `POST /api/v1/datasets/{dataset_id}/metadata/update`
- `PATCH /api/v1/datasets/{dataset_id}/documents/metadatas`
- `POST /api/v1/documents/upload`
The Go responses were checked against Python for:
- argument validation
- success response shape
- metadata update results
- upload_info result structure
- file vs URL input handling
### Description
Migrates the datasets tags aggregation API `GET
/api/v1/datasets/tags/aggregation` from Python to Go.
### Changes
- Registered the `GET /api/v1/datasets/tags/aggregation` route.
- Implemented `AggregateTags` in datasets `handler` and `service`.
- Added handler and service `unit tests`.
### Test Verification
- Verified by comparing results between Python (9380) and Go (9384)
services.
- Tested scenarios: single dataset, multiple datasets, empty parameters,
and unauthorized/invalid IDs.
- All tests and Go `unit tests` passed.
### What problem does this PR solve?
```
RAGFlow(api/default)> add admin host '127.0.0.1:9383';
SUCCESS
RAGFlow(api/default)> use admin;
SUCCESS
RAGFlow(admin)> delete api 'default';
SUCCESS
RAGFlow(admin)> delete api 'default';
CLI error: api server: default not found
RAGFlow(admin)> add api 'default' host '127.0.0.1:9384';
SUCCESS
RAGFlow(admin)> use api 'default';
SUCCESS
RAGFlow(api/default)> delete admin
SUCCESS
RAGFlow(api/default)> delete admin;
CLI error: admin server not exists
RAGFlow(api/default)> list api server;
+------------+---------------+-----------------+---------+
| api_server | api_server_ip | api_server_port | auth |
+------------+---------------+-----------------+---------+
| default | 127.0.0.1 | 9384 | no auth |
+------------+---------------+-----------------+---------+
RAGFlow(api/default)> add admin host '127.0.0.1:9383';
SUCCESS
RAGFlow(api/default)> show admin server;
+-------------------+-----------+
| field | value |
+-------------------+-----------+
| admin_server_ip | 127.0.0.1 |
| admin_server_port | 9383 |
| auth | no auth |
+-------------------+-----------+
```
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
Signed-off-by: Jin Hai <haijin.chn@gmail.com>
### What problem does this PR solve?
When I run RAGFlow_server.py:
```
2026-06-24 10:27:01,938 ERROR 3413485 fetch task exception
Traceback (most recent call last):
File "/home/infiniflow/Documents/development/ragflow/api/db/services/document_service.py", line 948, in _sync_progress
if t.progress_msg.strip():
^^^^^^^^^^^^^^^^^^^^
AttributeError: 'NoneType' object has no attribute 'strip'
```
fixed:
```python
if t.progress_msg.strip():
# fix:
if (t.progress_msg or "").strip():
```
Fix crash in `_sync_progress` when `progress_msg` is `None`.
#### Root Cause
`progress_msg` from task records can be `None`, causing:
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
### What problem does this PR solve?
This PR follows up on
[#15863](https://github.com/infiniflow/ragflow/pull/15863) (Korean i18n)
with translation refinements and i18n coverage for hardcoded strings
found in the UI.
- Refine awkward Korean phrasing (e.g. 'Chunk 만들기' → 'Chunk 생성', '유형' →
'타입', etc.)
- Apply i18n to hardcoded strings in `message-item`,
`next-message-item`, `multi-select`, `chat-prompt-engine`, and various
filter hooks
- Rename `use-selelct-filters.ts` → `use-select-filters.ts` (typo fix)
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
### What problem does this PR solve?
Fix release
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
---------
Signed-off-by: Jin Hai <haijin.chn@gmail.com>
## Summary
This PR fixes two issues discovered during testing of the PaddleOCR
async API refactoring:
### 1. PP-OCRv6 returns `ocrResults` instead of `layoutParsingResults`
Models like PP-OCRv6 are pure text recognition models that return
results in `ocrResults.prunedResult.rec_texts` format rather than the
`layoutParsingResults.prunedResult.parsing_res_list` format used by
layout-aware models (PaddleOCR-VL series).
**Changes:**
- `deepdoc/parser/paddleocr_parser.py`: Extract `ocrResults` alongside
`layoutParsingResults` in `_send_request()`, add fallback logic in
`_transfer_to_sections()` and `parse_image()`
- `internal/entity/models/paddleocr.go`: Add `ocrResults` struct and
fallback extraction in Go OCR handler
### 2. Image parsing not integrated into picture chunker
The `parse_image()` method existed in PaddleOCRParser but was never
called from `rag/app/picture.py` (the module that handles image file
uploads). Users configuring PaddleOCR as their layout recognizer would
still get local deepdoc OCR for images.
**Changes:**
- `rag/app/picture.py`: When `layout_recognize` is set to PaddleOCR, use
`PaddleOCROcrModel.parse_image()` instead of local OCR. Falls back
gracefully to local OCR on failure.
## Testing
Verified end-to-end in Docker:
- PaddleOCR-VL-1.6 PDF parsing: ✅ (10 text blocks with bbox)
- PaddleOCR-VL-1.6 image parsing: ✅ (219 chars)
- PP-OCRv6 PDF parsing with ocrResults fallback: ✅ (10 text blocks)
- PP-OCRv6 image parsing with ocrResults fallback: ✅ (136 chars)
## Related PRs
- #15967 (merged) - PaddleOCR async Job API refactoring + new models
- #16086 (merged) - PaddleOCR image parsing support
### What problem does this PR solve?
`_get_optimal_clusters` in `rag/raptor.py` had two edge-case issues in
GMM cluster-count selection:
1. It used `np.arange(1, max_clusters)`, which never evaluates the
upper-bound candidate (`max_clusters`).
2. When effective `max_clusters` becomes `1`, the candidate list was
empty and `argmin` crashed.
This PR makes candidate evaluation inclusive (`1..max_clusters`) and
guards the single-cluster case by returning `1` directly.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
### Validation
- `pytest test/unit_test/rag/test_raptor_psi_tree_builder.py
--config-file pyproject.toml -q`
- `ruff check rag/raptor.py
test/unit_test/rag/test_raptor_psi_tree_builder.py`
### Tests added
- Regression test for `max_cluster == 1` path (no crash, returns 1)
- Regression test verifying upper-bound candidate is evaluated and can
be selected
_AI-assistance disclosure: parts of this change (bug triage and test
scaffolding) were drafted with AI assistance and fully reviewed and
verified by me._
---------
Co-authored-by: Harsh Kashyap <harshkashyap@Harshs-MacBook-Pro.local>
Co-authored-by: Cursor <cursoragent@cursor.com>
### What problem does this PR solve?
- Tools management
- Pregel engine wrapper for better usage
- UT race
- Coding style
### Type of change
- [x] Refactoring
### What problem does this PR solve?
```
RAGFlow(admin)> show model 'abc';
+------------+----------------------------------------------------------------+
| field | value |
+------------+----------------------------------------------------------------+
| command | get_model_by_model_name |
| error | 'get model by model name' is implemented in enterprise edition |
| model_name | abc |
+------------+----------------------------------------------------------------+
RAGFlow(admin)> list models;
+-----------------+--------------------------------------------------------+
| command | error |
+-----------------+--------------------------------------------------------+
| list_all_models | 'list all models' is implemented in enterprise edition |
+-----------------+--------------------------------------------------------+
```
### Type of change
- [x] Refactoring
Signed-off-by: Jin Hai <haijin.chn@gmail.com>
### What problem does this PR solve?
Implement:
1. `/api/v1/datasets/<dataset_id>/documents/<document_id>/chunks GET`
2.
`/api/v1/datasets/<dataset_id>/documents/<document_id>/chunks/<chunk_id>
PATCH`
3. `/api/v1/datasets/<dataset_id>/documents/<document_id>/chunks PATCH`
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
### What problem does this PR solve?
Provider default public key for CLI
### Type of change
- [x] Refactoring
Signed-off-by: Jin Hai <haijin.chn@gmail.com>
Refactor the Go agent port's logging so every log line — gin access,
agent canvas events, harness warnings, fatal boot errors — flows through
a single common.Logger (zap) backed by a rotated file, with structured
fields, level filtering, and configurable rotation.
---------
Co-authored-by: Claude <noreply@anthropic.com>
## Problem
`thread_pool_exec()` dispatches work via `loop.run_in_executor()`, which
submits the callable with a plain `executor.submit(func, *args)` and
does **not** copy the caller's `contextvars.Context`. So a `ContextVar`
set in the async caller is not visible inside the function running in
the worker thread.
This differs from `asyncio.to_thread()`, which runs the callable inside
a copied context. `run_in_executor()` has never propagated context
(verified on Python 3.12 and 3.13) — so this is a pre-existing gap in
the helper, **not** a regression or a Python-version compatibility
issue.
Concretely, any code that sets a `ContextVar` in async code and reads it
inside a function dispatched via `thread_pool_exec` (request tracing,
per-task state, Langfuse trace propagation, etc.) silently loses that
context.
## Fix
Copy the current context before submitting and run the callable inside
it with `ctx.run()`, matching what `asyncio.to_thread()` does:
```python
async def thread_pool_exec(func, *args, **kwargs):
loop = asyncio.get_running_loop()
ctx = contextvars.copy_context()
if kwargs:
inner = functools.partial(func, *args, **kwargs)
return await loop.run_in_executor(_thread_pool_executor(), ctx.run, inner)
return await loop.run_in_executor(_thread_pool_executor(), ctx.run, func, *args)
```
This explicitly **adds** ContextVar propagation to the helper (it does
not restore any prior behavior). Backward-compatible.
## Tests
`TestThreadPoolExec` covers propagation, the kwargs path, per-call
isolation and the unset-default case.
> Note: the branch name still contains `python313` for historical
reasons; the change is unrelated to any Python version.
### What problem does this PR solve?
Updated the release workflow to install SIMDe headers into the MSYS2
toolchain include directory. Adjusted CMake flags to remove references
to the previous SIMDE_INCLUDE_DIR.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
## Summary
Stabilizes the Go unit-test surface so the test suite can run reliably
in CI and locally via \`bash build.sh --test\`.
## Verification
\`\`\`bash
bash build.sh --test -- -count=10 -run TestWithCancel_SequentialAgent
./internal/harness/core/
bash build.sh --test -- -count=5 -run TestSiliconflowChatExtracts
./internal/entity/models/
bash build.sh --test # full suite
\`\`\`
All previously failing packages (\`admin\`, \`cli\`, \`handler\`,
\`parser\`,
\`router\`, \`service\`, \`service/chunk\`) now build and test
successfully.
\`TestWithCancel_SequentialAgent\` passes 10/10 (was flaky). SiliconFlow
reasoning test passes after switching the assertion to the SiliconFlow
wire
format.
---------
Co-authored-by: Claude <noreply@anthropic.com>
## Summary
Several keys added in recent releases were missing from the French
(`fr.ts`) locale file.
- **`top`** — missing in both the common section and the dataset section
- **Chat channels** — all UI strings for the new chat channels feature
(`chatChannels`, `chatChannelDesc.*`, `connectDialog`, `notConnected`,
etc.)
- **Username validation** — `usernameMaxLength`,
`usernameInvalidCharacters`
- **Model editing** — `editCustomModelTitle`
## Changes
- `web/src/locales/fr.ts` — 47 lines added, no other files touched
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
### What problem does this PR solve?
```
RAGFlow(admin)> show role 'user' default models;
+--------------------------+-----------------------------------------------------------------+-----------+
| command | error | role_name |
+--------------------------+-----------------------------------------------------------------+-----------+
| show_role_default_models | 'show role default models' is implemented in enterprise edition | user |
+--------------------------+-----------------------------------------------------------------+-----------+
RAGFlow(admin)> set role 'user' default chat 'glm4.5@test@zhipu-ai';
+------------+---------------------------------------------------------------+
| field | value |
+------------+---------------------------------------------------------------+
| model_id | |
| model_type | chat |
| role_name | user |
| command | set_role_default_model |
| error | 'set role default model' is implemented in enterprise edition |
+------------+---------------------------------------------------------------+
RAGFlow(admin)> reset role 'user' default chat;
+------------+-----------------------------------------------------------------+
| field | value |
+------------+-----------------------------------------------------------------+
| command | reset_role_default_model |
| error | 'reset role default model' is implemented in enterprise edition |
| model_type | chat |
| role_name | user |
+------------+-----------------------------------------------------------------+
```
---------
Signed-off-by: Jin Hai <haijin.chn@gmail.com>
## Summary
- add public Go route for `/api/v1/searchbots/detail`
- implement beta-token auth flow for shared search access
- add tenant-based access check for shared search apps
- add joined search detail query for the share response
- align Go response shape with the current Python runtime behavior
- add DAO / service / handler tests for the new endpoint
close#16132
## Summary
This PR completes the Go-side merge and cleanup for chat channel APIs,
including handler/service wiring, route registration, and test coverage.
Implemented and aligned 5 chat channel APIs:
```
- POST `/api/v1/chat-channels`
- GET `/api/v1/chat-channels`
- GET `/api/v1/chat-channels/:channel_id`
- PATCH `/api/v1/chat-channels/:channel_id`
- DELETE `/api/v1/chat-channels/:channel_id`
```
Co-authored-by: Haruko386 <tryeverypossible@163.com>
### What problem does this PR solve?
```
fixed:
RAGFlow(api/default)> use model 'minimax-m2.5@test@minimax'
SUCCESS
RAGFlow(api/default)> chat message 'who r u'
Answer: Hey! I'm MiniMax-M2.5, an AI assistant here to help you with questions, tasks, or whatever you need. What can I do for you?
Time: 1.727263
```
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
### What problem does this PR solve?
Removed references to 'simde' from the package lists and updated paths
for compiler detection and CMake configuration to ensure proper handling
of Windows executables.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
### What problem does this PR solve?
As title.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
---------
Signed-off-by: Jin Hai <haijin.chn@gmail.com>
Closes#15139.
The "+ Add" flow in the Set/Edit Metadata modal posted updates with an
empty value, so backend saves were silent no-ops and the document's "X
fields" count stayed at 0 despite a "Success" toast.
The value `<Input>` updates `tempValues` synchronously per keystroke but
only writes through to `metaData.values` on blur (via
`handleValueBlur`). When the user clicks the nested modal's Confirm
button without first blurring, the click handler races the blur and
`handleSave` closes over the pre-blur `metaData.values` — still the
initial `['']`. `addUpdateValue` then queues an empty-string update; the
auto-fire save sends it, and after `resetOperations()` the outer Save
button posts `updates: []`.
Read from `tempValues` instead so the queued update carries the typed
value.
Regression test in `tests/use-manage-values-modal.test.ts` asserts that
`handleSave` passes the typed value (not the pre-blur empty string) to
`addUpdateValue` in the add-new code path.
### What problem does this PR solve?
Updated rust_target and added simde support for Windows builds. Modified
CMake commands to include simde and adjusted paths for compilers.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
### What problem does this PR solve?
Updated MSYS2 package list for Windows builds and added Rust target
specifications. Modified build steps for office_oxide and rag tokenizer
libraries to improve compatibility and streamline the build process.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
## What
Fixes#16008 — tables contained in a DOCX are silently dropped when the
document is parsed with the **laws** chunking method.
## Root cause
`Docx.__call__` in `rag/app/laws.py` iterated `self.doc.paragraphs`,
which only yields paragraph elements. Tables are separate `tbl` blocks
in the document body, so they were never visited and were lost from the
output. (The `naive` parser already handles tables by iterating the
document body.)
## Changes
- Iterate `self.doc._element.body` so tables are visited in document
order alongside paragraphs.
- Add a `__table_to_html` helper that renders each table to HTML,
including merged-cell `colspan` detection (mirrors the `naive` parser's
logic).
- Inject each table into the section tree with a sentinel level deeper
than any heading, so `Node.build_tree` merges it into its **enclosing
section** — keeping the chapter/article title path as retrieval context
rather than producing an orphaned chunk.
- Guard the `h2_level` computation against an empty heading set, so a
tables-only or empty DOCX no longer raises `IndexError`.
This keeps the laws parser's hierarchical chunking **and** adds table
extraction, so users no longer have to choose between losing structure
(naive) or losing tables (laws).
## Tests
Adds `test/unit_test/rag/test_laws_docx_tables.py` covering:
- table content is preserved and carries its section title path,
- merged adjacent cells collapse to `colspan`,
- tables-only document does not crash,
- empty document returns `[]`.
All four pass; `ruff check` / `ruff format` are clean.
### What problem does this PR solve?
- list configs;
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
---------
Signed-off-by: Jin Hai <haijin.chn@gmail.com>
### What problem does this PR solve?
- list resources
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
- [x] Refactoring
---------
Signed-off-by: Jin Hai <haijin.chn@gmail.com>
### What problem does this PR solve?
1. Remove unused file
2. Remove duplicate models
3. Resort the function order
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
- [x] Refactoring
Signed-off-by: Jin Hai <haijin.chn@gmail.com>
### What does this PR do?
This PR adds a new DingTalk chat channel integration and hardens the
inbound callback path.
### Summary
- Adds DingTalk as a selectable chat channel in the UI and backend
channel registry.
- Adds the DingTalk chat channel icon asset.
- Acknowledges DingTalk Stream callbacks and deduplicates repeated
inbound messages to avoid duplicate replies.
### What problem does this PR solve?
Implement OpenAI chat completions in GO
POST /api/v1/openai/<chat_id>/chat/completions
OpenAI chat cli: internal/development.md
### Type of change
- [x] Refactoring
### What problem does this PR solve?
1. add modelID for delete_model and update_status
2. fix the bug when update-status delete model
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
- [x] New Feature (non-breaking change which adds functionality)
### What problem does this PR solve?
RAGFlow(admin)> show tasks summary;
+---------+-----------------------------------------------------------------+
| field | value |
+---------+-----------------------------------------------------------------+
| command | show_users_quota_summary |
| error | 'Show users quota summary' is implemented in enterprise
edition |
+---------+-----------------------------------------------------------------+
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
Signed-off-by: Jin Hai <haijin.chn@gmail.com>
The pipeline file log detail hook (`useFetchPipelineFileLogDetail`) was
calling the legacy `kbService.getPipelineDetail({ log_id })` endpoint,
which does not match the current RESTful API contract. The backend now
expects both `datasetId` and `logId` to construct the correct URL (`GET
/api/v1/datasets/{datasetId}/ingestions/{logId}`).
### What problem does this PR solve?
Prepare for enterprise command
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
---------
Signed-off-by: Jin Hai <haijin.chn@gmail.com>
### What problem does this PR solve?
As title
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
- [x] New Feature (non-breaking change which adds functionality)
## Summary
Fixes#15487 — lone markdown headers are no longer isolated as empty
chunks when a custom `delimiter` is set.
- Merge consecutive lone headers before attaching to the following prose
body
- Skip code fences, tables, lists, and blockquotes via
`_is_attachable_body()`
- Unit tests include the `# Title / ## Intro / Body` regression from
CodeRabbit review
## Validation
- `pytest test/unit_test/deepdoc/parser/test_markdown_parser.py` (11
passed locally)
Closes#15487
### What problem does this PR solve?
- Migrated MCP server detail and export (download) API from Python to
Go.
- Registered route: `GET /api/v1/mcp/servers/:mcp_id` (supporting
`?mode=download` query parameter).
### What problem does this PR solve?
This PR implements the Go backend counterpart for the document partial
update API:
`PATCH /api/v1/datasets/:dataset_id/documents/:document_id`
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
- [x] Refactoring
### What problem does this PR solve?
This PR improves code readability in the CLI parser by renaming the loop
index `i` to `modelIndex`. It also renames the loop label `A` to
`optionsLoop` to align with standard Go naming conventions.
### Type of change
- [x] Refactoring
### What problem does this PR solve?
- Update version tags in README files (including translations) from
v0.26.0 to v0.26.1
- Modify Docker image references and documentation to reflect new
version
- Update version badges and image descriptions
- Maintain consistency across all language variants of README files
### Type of change
- [x] Documentation Update
### What problem does this PR solve?
Fixes two bugs in the OpenRouter streaming chat request builder
(`internal/entity/models/openrouter.go`, `ChatStreamlyWithSender`):
1. **qwen/glm models streamed to a broken URL.** The code routed any
`qwen`/`glm` model to
`URLSuffix.AsyncChat`, but `conf/models/openrouter.json` defines no
`async_chat` suffix
(empty), so the request was POSTed to `<base>/` instead of
`<base>/chat/completions` —
breaking streaming for every qwen/glm model. The non-stream path has no
such branch.
Fix: all models use the standard `Chat` suffix, consistent with the
non-stream path.
2. **Streaming reasoning was never enabled.** The request set reasoning
via a non-standard
`thinking` key, which OpenRouter ignores. OpenRouter's API — and this
provider's own
non-stream request (line ~110) and its streamed `delta.reasoning` parser
(line ~311) —
use the `reasoning` object. Fix: send `reasoning: {"enabled":
<thinking>}` (and
`{"effort": ...}` when set, taking precedence as in the non-stream
path).
Closes#16110
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
### What problem does this PR solve?
Add chat model factory for Xiaomi model.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
### What problem does this PR solve?
Adds qqbot as a built-in chat channel so it can be discovered and
started by the channel bootstrapper and shown in the chat channel
settings UI.
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
### What problem does this PR solve?
Part of #15853 (provider model-list refactor).
Refactors **Ollama** `ListModels` onto the shared `ParseListModel`
pattern and fixes two correctness issues:
- **Endpoint:** switch the models suffix from `api/ps` (only
currently-running models) to `api/tags` (all installed models) — the
latter is what a model picker should show.
- **Parsing:** Ollama returns `{"models":[{"name","model"}]}`, a
non-OpenAI shape. Decode it into a typed struct, map the names into
`ModelList`, then enrich through `ParseListModel`. This removes the
previous unchecked type assertions (`result["models"].([]interface{})` /
`.(map[string]interface{})` / `.(string)`) that **panicked** when the
body was missing the `models` array or any field, and adds a fallback to
the `model` field when `name` is blank.
- Drops the no-op GET request body and a dead base-URL reassignment.
#### Drive-by fix
Shared gitee_test.go `DSModelList` -> `ModelList` compile fix (renamed
in #15900) so the models test package builds; auto-resolves against the
sibling #15853 PRs.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
- [x] Refactoring
### What problem does this PR solve?
Fix: The pipeline created from the template fails immediately upon
execution.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
### What problem does this PR solve?
fix: misc frontend fixes for agent log, login, search settings
- agent-log: restore server-side pagination on export and search;
replace hardcoded labels with i18n keys; switch container to
text-text-primary
- login: validate register nickname against NICKNAME_PATTERN with
reusable setting i18n
- next-search: align llm_setting schema with chat (LlmSettingFieldSchema
+ LLMIdFormField nested, LlmSettingEnabledSchema at form
root) so the slider Switch reads the correct path; strip *Enabled flags
before submit to avoid backend "Unrecognized field name"
errors
- locales: add common.reset (zh/en)
- skills/go-naming: fix relative link to rules/named.md
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
### What problem does this PR solve?
Fix: The .docx file is not displaying fully; the hierarchy of the
pipeline created from the template is missing.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
### What problem does this PR solve?
Fix chat channel message routing to use the connected `chat_id`, and
make the Feishu websocket client bind to the thread-local event loop.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
### What problem does this PR solve?
Show Telegram in the chat channel picker alongside the existing Discord
and Feishu entries.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
### What problem does this PR solve?
Use a 95% max_length threshold before truncating embedding inputs, which
reduces the chance of provider-side invalid-parameter errors on
near-limit chunks.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
Replaces the Python agent canvas runtime with a Go implementation that
runs inside `cmd/server_main`.
The canvas compiles into an eino Workflow that pauses on wait-for-user
via native Interrupt/Resume (no sentinel flag) and resumes from a
Redis-backed CheckPointStore.
All 21 Python agent components and ~35 tools are ported with functional
parity.
Sandbox providers now read their JSON config from the admin-panel
system_settings table with env fallback.
234 files / +35,413 / -6,111. All Go files are gofmt-clean (CI gate
added); drops the v2 DSL E2E step and the gap-analysis plan (both
redundant after the port ships).
## Type of change
- [x] Refactoring
- [x] New feature
- [x] Bug fix
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude <noreply@anthropic.com>
### What problem does this PR solve?
The `get_ingestion_log` endpoint (both Python
`dataset_api_service.get_ingestion_log` and Go
`DatasetService.GetIngestionLog`) was returning only the
**dataset-level** field set, which omits critical fields such as `dsl`,
`document_id`, `parser_id`, `document_name`, `pipeline_id`, etc.
This caused the front-end **dataflow-result page** to be unable to
render the pipeline timeline and chunks when viewing a single ingestion
log, regardless of whether the log was a dataset-level operation
(graph/raptor/mindmap) or a per-file parse.
### Background
`PipelineOperationLogService` provides two field sets:
| Method | Fields |
|---|---|
| `get_dataset_logs_fields` | Minimal set (progress, status, timestamps,
etc.) |
| `get_file_logs_fields` | Superset — includes `document_id`, `dsl`,
`parser_id`, `document_name`, `pipeline_id`, … |
When listing logs, the API correctly distinguishes dataset-level vs
file-level logs and uses the appropriate converter. However, when
**fetching a single log by ID**, both the Python and Go implementations
were hardcoded to the dataset-level set, dropping the extra fields that
the front-end needs.
### What problem does this PR solve?
Part of #15853 (provider model-list refactor). Final two providers.
- **voyage:** Voyage AI exposes no live model-list endpoint — its public
API only has `/v1/embeddings` and `/v1/rerank` — so the previous
`ListModels` was a `no such method` stub. Replace it with a
static-catalog listing sourced from the loaded provider definition,
carrying each model's `max_tokens`, `model_types`, and embedding
`dimensions`. `list models from voyage` now returns the 13-model catalog
instead of erroring.
- **fishaudio:** route the existing `/model` voice listing through the
shared `ParseListModel` helper for consistency; keep the human-readable
`title` as the model name and fall back to `_id` when a title is blank.
#### Drive-by fix
Shared gitee_test.go `DSModelList` -> `ModelList` compile fix (renamed
in #15900); auto-resolves against the sibling #15853 PRs.
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
- [x] Refactoring
Co-authored-by: Haruko386 <tryeverypossible@163.com>
# Summary
- The culprit is commit b4c8711d5 / PR #15415 (fix: upgrade crawl4ai to
0.8.0).
- That upgrade brought in unclecode-litellm, which installs the same
top-level litellm namespace as upstream litellm.
- The crash happens when files from one LiteLLM distribution are mixed
with files from the other: custom_guardrail.py expects
GuardrailTracingDetail, but types/utils.py can come from the older
conflicting package.
## Changes
1. **Entity (`internal/entity/chat_channel.go`)**:
- Implemented `ChatChannel` struct mapping the `chat_channel` database
table.
- Declared `ChatChannelListResponse` as a DTO to filter out sensitive
credentials (`config` field) and fetch the associated `dialog_name` via
left join.
2. **GORM Migration (`internal/dao/database.go`)**:
- Registered `&entity.ChatChannel{}` in the `dataModels` array inside
`InitDB()` to enable safe GORM schema synchronization.
3. **DAO (`internal/dao/chat_channel.go`)**:
- Implemented `ChatChannelDAO` wrapping GORM CRUD methods (`Create`,
`GetByID`, `UpdateByID`, `DeleteByID`).
- Implemented `ListByTenantID` performing a `LEFT JOIN` on the `dialog`
table to retrieve `dialog_name` while excluding `config` values to avoid
credential leaks.
4. **Test (`internal/dao/chat_channel_test.go`)**:
- Added integration unit tests testing the full CRUD lifecycle and GORM
left-join mapping list querying.
### What problem does this PR solve?
Fix: Importing the MCP dialog causes duplicate submissions.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
The parser pods suffer from OOM kills when processing large PDF
documents. The root cause is in api/db/services/task_service.py: when
layout_recognize is not DeepDOC (e.g. Plain Text), page_size was set to
MAXIMUM_TASK_PAGE_NUMBER (100 million), causing the entire PDF to be
processed as a single task with all pages loaded into memory
simultaneously.
This PR fixes the issue by paginating non-DeepDOC PDF parsing tasks the
same way DeepDOC already does.
### What problem does this PR solve?
Syncs the /api/v1/chat/completions docs with the current behavior,
including the new legacy streaming mode.
### Type of change
- [x] Documentation Update
### What problem does this PR solve?
The parser pods suffer from OOM kills when processing large PDF
documents. The root cause is in api/db/services/task_service.py: when
layout_recognize is not DeepDOC (e.g. Plain Text), page_size was set to
MAXIMUM_TASK_PAGE_NUMBER (100 million), causing the entire PDF to be
processed as a single task with all pages loaded into memory
simultaneously.
This PR fixes the issue by paginating non-DeepDOC PDF parsing tasks the
same way DeepDOC already does.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
- [ ] New Feature (non-breaking change which adds functionality)
- [ ] Documentation Update
- [ ] Refactoring
- [x] Performance Improvement
- [ ] Other (please describe):
## Summary
Add image parsing capability to PaddleOCR integration, building on top
of #15967 (async Job API migration).
## Changes
### `deepdoc/parser/paddleocr_parser.py`
- Add `parse_image()` method that uses the same async Job API flow as
`parse_pdf()`
- Extracts text from `layoutParsingResults` → `prunedResult` →
`parsing_res_list`
- Returns concatenated block content as a single string
### `rag/llm/ocr_model.py`
- Add `parse_image()` wrapper to `PaddleOCROcrModel` with availability
check and logging
## Relationship to other PRs
- **Depends on**: #15967 (async Job API migration) — this PR is based on
that branch
- **Replaces**: #14826 (original image processing PR based on old sync
API)
## Notes
This PR uses `base_url` and the async Job API (submit → poll → fetch)
consistent with #15967, rather than the old `api_url` + sync POST
pattern from #14826.
## Summary
Migrate PaddleOCR integration from the deprecated synchronous HTTP API
to the new asynchronous Job API (`submit → poll → fetch`), aligning with
PaddleOCR 3.6.0+ architecture.
## Changes
### Python (`deepdoc/parser/paddleocr_parser.py`)
- Replace synchronous `requests.post()` with async Job API flow (submit
→ poll → fetch)
- Authentication: `token {token}` → `Bearer {token}`
- File transfer: base64 JSON body → multipart file upload
- Polling: exponential backoff (initial 3s, ×1.5, max 15s, timeout
controlled by `request_timeout`)
- Result: fetch full JSONL from result URL, preserving `prunedResult`
with bbox info for crop functionality
- Rename `api_url` → `base_url` (backward compatible: `api_url` still
accepted as fallback)
### Python (`rag/llm/ocr_model.py`)
- Prefer `paddleocr_base_url` / `PADDLEOCR_BASE_URL`, fallback to
`paddleocr_api_url` / `PADDLEOCR_API_URL`
### Go (`internal/entity/models/paddleocr.go`)
- Add `Client-Platform: ragflow` header to submit and poll requests
- Change polling from fixed 3s to exponential backoff (initial 3s, ×1.5,
max 15s)
### Python (`common/constants.py`)
- Add `PADDLEOCR_BASE_URL` to env keys and default config
## Backward Compatibility
- Old env var `PADDLEOCR_API_URL` still works (used as fallback)
- Frontend field `paddleocr_api_url` still works (backend reads it as
fallback)
- No user-facing configuration changes required for existing setups
## Why not use the `paddleocr` SDK package directly?
RAGFlow's `_transfer_to_sections()` relies on `prunedResult` (containing
`block_bbox`, `block_label`, `parsing_res_list`) from the raw API
response for PDF crop functionality. The SDK's public `parse_document()`
API only returns `DocParsingResult` with `markdown_text`, discarding the
bbox data. Therefore we implement the async Job API flow directly via
HTTP, following the same logic as the SDK internally.
### What problem does this PR solve?
Merge password related functions
### Type of change
- [x] Refactoring
Signed-off-by: Jin Hai <haijin.chn@gmail.com>
## Summary
- The `ChatChannel` DB column was renamed from `dialog_id` to `chat_id`
via a migration (added in a prior commit).
- Aligns the REST API layer (`chat_channel_api.py`,
`chat_channel_service.py`) to use `chat_id` consistently.
- Updates the frontend (`interface.ts`, `hooks.ts`,
`connect-dialog-modal.tsx`, `added-channel-card.tsx`) to read/write
`chat_id` instead of `dialog_id`.
- The joined `dialog_name` alias in the list query is unchanged (backend
still returns it under that name).
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
### What problem does this PR solve?
`rag/app/naive.py` `Markdown.load_images_from_urls` fetched image URLs
parsed
straight out of an untrusted uploaded markdown document via a raw
`requests.get`,
with no SSRF validation. Markdown chunking always reaches this path
(`return_section_images=True`), so any authenticated user who uploads a
`.md`/`.markdown`/`.mdx` file to a knowledge base could make the server
issue
requests to internal services or cloud-metadata endpoints, e.g.
``. The `image/`
Content-Type
check only gates decoding — the outbound request (the SSRF) always
fires.
This was the one user-controlled fetch site missed by the project's
existing
SSRF-hardening (`common/ssrf_guard.py`, already applied to the crawler,
SearXNG,
RSS connector, MCP/document APIs, and OAuth avatar download).
The fix validates and DNS-pins every hop with
`common.ssrf_guard.assert_url_is_safe`
before connecting, and follows redirects manually so each redirect
target is
re-validated (closing the DNS-rebinding / redirect-bypass window),
mirroring
`common/data_source/rss_connector.py`. Blocked URLs are skipped and
logged like
any other unreachable image, so legitimate public images are unaffected.
Adds a
regression test at `test/unit_test/rag/app/test_markdown_image_ssrf.py`.
Closes#15437
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
---------
Co-authored-by: Ubuntu <ubuntu@ubuntu-2204.linuxvmimages.local>
Co-authored-by: galuis116 <galuis116@users.noreply.github.com>
### What problem does this PR solve?
fix: remove unnecessary 'asChild' prop from FilterButton component
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
### What problem does this PR solve?
fix: remove unnecessary div in profile page layout
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
### What problem does this PR solve?
fix: update channelTemplates to filter for Discord and Lark only
- Fixed a display issue with chunks during pipeline parsing.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
### What problem does this PR solve?
fix: update localization keys for image2text and add ocr option
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
### What problem does this PR solve?
Fix DB migration issue.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
---------
Signed-off-by: Jin Hai <haijin.chn@gmail.com>
### What problem does this PR solve?
Fix:
- Pass session_id to langfuse.
- Get correct status for add model_type.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
### What problem does this PR solve?
This PR fixes Go admin server startup failure caused by duplicate model
aliases in conf/all_models.json.
The model provider loader builds a global lookup table from both model
name and alias values. Some aliases duplicated another model's name or
another
alias, for example amazon.titan-embed-text-v1, which caused startup to
fail with a duplicate alias error. This PR removes conflicting duplicate
aliases
while keeping all model definitions intact.
### What problem does this PR solve?
As title
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
- [x] Documentation Update
### What problem does this PR solve?
Fix register user
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
---------
Signed-off-by: Jin Hai <haijin.chn@gmail.com>
### What problem does this PR solve?
TOC chunks now include a toc field so the agent pipeline logs expose the
data the frontend expects.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
### What problem does this PR solve?
## What
- Add `group` class to wrapper div to enable hover state coordination
- Apply hover styles to checkbox via group-hover
- Make FormLabel clickable via onClick toggle + cursor-pointer
- Fix label color logic: disabled vs primary state
## Why
The "Remember me" label was not clickable and had no hover feedback,
making the UX inconsistent with standard checkbox behavior.
## How to test
0. Go to the demo video before/after attached below
1. Go to the login page
2. Click directly on the "Remember me" label → should toggle the
checkbox
3. Hover over the checkbox area → should show hover styles
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
## Before
https://github.com/user-attachments/assets/bd47d45c-09ea-437f-bd98-3397ce040c1e
## After
https://github.com/user-attachments/assets/45c65d1a-bec7-4ad6-8f1c-d149f7296f8f
### What problem does this PR solve?
This PR enhances the CLI parser to support dimension configurations for
custom embedding models. Users can now specify the maximum dimension and
other supported dimensions directly after the embedding keyword.
```
add model 'x1 x2 x3 x4 x5' to provider 'vllm' instance 'test' with
tokens 1024 chat think vision,
token 2048 chat,
token 1024 think vision,
token 0 embedding 2048 64 1024 2048,
token 0 embedding 2048;
```
- The first integer following embedding represents the max_dimension.
- Any subsequent integers represent specific alternative dimensions.
- If no subsequent integers are provided, dimensions defaults to empty,
indicating all sizes under max_dimension are supported.
### Description
Currently, when setting tenant default models (e.g., chat, embedding,
rerank), the API only accepts the composite name
(`model_name@model_instance@model_provider`). However, some integrations
and front-end features prefer using the database `model_id` (UUID)
directly.
This PR adds support for `model_id` in default model configuration:
1. **Request Binding**: Added `model_id` (optional field) to the request
body schema in the handler.
2. **Database Lookup**: If `model_id` is supplied, the service queries
the database to resolve the respective provider, instance, and model
names.
3. **Security Validation**: Verified that the provider associated with
the resolved `model_id` belongs to the requesting tenant.
4. **Unit Tests**: Added `TestSetTenantDefaultModels_WithModelID` to
verify DB ID resolution and tenant mapping.
### What problem does this PR solve?
core module for agent layer built on top of graph engine #16039
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
### What problem does this PR solve?
Fixes the sandbox config API method mismatch so the frontend and backend
use the same HTTP verb.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
### What problem does this PR solve?
Adds a legacy mode for /chat/completions that restores v0.23.0-style
output by converting start_to_think/end_to_think back into raw
<think></think> markers and streaming cumulative answer text.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
### What problem does this PR solve?
go-version of Pregel-based BSP engine
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
### What problem does this PR solve?
Feat:
- Allow upsert model_type for instance model
Fix:
- Allow create instance with duplicate api_key
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
- [x] New Feature (non-breaking change which adds functionality)
### What problem does this PR solve?
Feat: Move less important chat settings into a collapsible panel.
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
### What problem does this PR solve?
As title.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
Signed-off-by: Jin Hai <haijin.chn@gmail.com>
### What problem does this PR solve?
```
RAGFlow(api/default)> parse file 'test.html';
Parsing HTML file: test.html
<html>
......
```
Signed-off-by: Jin Hai <haijin.chn@gmail.com>
Guard the agent-attachment download against a missing or empty storage blob so the caller gets a structured 4xx (`Document not found!`) instead of an HTTP 500. Same bug class as #15365 on document preview.
Resolve#15502
### What problem does this PR solve?
```
RAGFlow(api/default)> parse file 'README.md';
Parsing Markdown file: README.md
--- AST tree:
HTMLBlock '<div align="center">\n<a href="https:…'
```
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
Signed-off-by: Jin Hai <haijin.chn@gmail.com>
### What problem does this PR solve?
Allow S3-compatible data source region fields to accept custom values
while preserving search-and-select behavior.
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
### What problem does this PR solve?
| # | Method | Endpoint | Description | Git Equivalent |
|---|--------|----------|-------------|----------------|
| 1 | `POST` | `/api/v1/{prefix}/{folder_id}/commits` | Create a
snapshot commit with file changes (add/modify/delete/rename) | `git add`
+ `git commit` |
| 2 | `GET` | `/api/v1/{prefix}/{folder_id}/commits` | List commit
history (paginated) | `git log` |
| 3 | `GET` | `/api/v1/{prefix}/{folder_id}/commits/{commit_id}` | Get
commit detail with file changes | `git show` |
| 4 | `GET` | `/api/v1/{prefix}/{folder_id}/commits/{commit_id}/files` |
List file changes in a commit | `git show --name-status` |
| 5 | `GET` |
`/api/v1/{prefix}/{folder_id}/commits/diff?from=...&to=...` | Compare
two commits and return differences | `git diff` |
| 6 | `GET` | `/api/v1/{prefix}/{folder_id}/changes` | Get uncommitted
changes (add/modify/delete) | `git status` |
| 7 | `GET` | `/api/v1/{prefix}/{folder_id}/commits/{commit_id}/tree` |
Get the folder tree snapshot at commit time | `git ls-tree` |
| 8 | `GET` |
`/api/v1/{prefix}/{folder_id}/commits/{commit_id}/files/{file_id}/content`
| Get a file's content as it existed in a specific commit | `git show
HEAD:file` |
| 9 | `GET` | `/api/v1/{prefix}/{file_id}/versions` | Get version
history for a specific file across all commits | `git log -- file` |
Where `{prefix}/{id}` can be:
- `folders/{folder_id}` — direct folder access
- `workspaces/{workspace_id}` — alias of `folders/{folder_id}`
- `datasets/{dataset_id}` — resolves to the dataset's folder
- `memories/{memory_id}` — resolves to the memory's folder
- `skills/{skill_id}` — resolves to the skill's folder
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
- [x] Documentation Update
After upgrading to v0.26.0, the Ollama provider returns an empty model
list because the Go rewrite uses `/api/ps` (only running models) instead
of `/api/tags` (all installed models). This PR changes the endpoint to
`/api/tags`, restoring the ability to list and add Ollama models.
Closes#16000
### What problem does this PR solve?
Not not only model_name@instance_name@provider_name is acceptable, but
also model_id is acceptable.
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
---------
Signed-off-by: Jin Hai <haijin.chn@gmail.com>
### What problem does this PR solve?
Add parser config
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
Signed-off-by: Jin Hai <haijin.chn@gmail.com>
### What problem does this PR solve?
Now we can parse 'pptx', 'ppt', 'doc', 'xls', 'xlsx'
```
RAGFlow(api/default)> parse file 'test.pptx';
Parsing PPTX file: test.pptx
Document format: pptx
```
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
Signed-off-by: Jin Hai <haijin.chn@gmail.com>
### What problem does this PR solve?
Fixes#15840.
The Go HTTP server sets `WriteTimeout: 120s`, which also applies to
long-lived SSE responses. Existing Go streaming handlers did not clear
the per-response write deadline, so streams that run longer than the
server timeout can be terminated mid-response.
This PR adds a small handler helper that clears the response write
deadline for SSE requests and calls it only in existing Go streaming
branches:
- conversation completion streaming
- provider chat streaming
- provider transcription streaming
- provider speech streaming
The global server `WriteTimeout` remains unchanged for non-streaming
requests.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
### Test plan
- `/root/go/bin/go test ./internal/handler -run
TestDisableWriteDeadlineForSSEAllowsLongLivedStream -count=1`
- `/root/go/bin/go test ./internal/handler -count=1`
### What problem does this PR solve?
As title.
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
---------
Signed-off-by: Jin Hai <haijin.chn@gmail.com>
### What problem does this PR solve?
#15844
Adds a **Chat channels** capability so a RAGFlow assistant (Dialog) can
be exposed as a bot on external messaging platforms (Feishu/Lark,
Discord, Telegram, Slack, WeCom, LINE, etc.). An admin configures a bot
in the UI, connects it to an assistant, and inbound messages are
answered from that assistant's knowledge base — replies are delivered
back on the channel.
**Feishu/Lark is implemented and tested end-to-end.** Discord, Telegram,
LINE, and WeCom are scaffolded against the same interface; the remaining
listed channels are tracked as follow-ups.
### Design
**Backend**
- New `chat_channel` table (`tenant_id`, `name`, `channel`, `config`
JSON holding `{credential: {...}}`, `dialog_id`, `status`) +
`ChatChannelService` and RESTful CRUD under `/api/v1/chat_channels`.
- Channel framework under `api/channels/`: a `core` registry +
per-channel packages that self-register a builder and implement a common
`Channel` interface (`start`/`stop`/`send` + inbound normalization) over
`IncomingMessage`/`OutgoingMessage`.
- Embedded **reconcile loop** in `ragflow_server`
(`api/channels/bootstrap.py`): loads enabled bots, and
starts/stops/restarts them as rows change (no server restart needed).
Inbound messages run the connected dialog via the non-streaming
completion path, keeping per-end-user conversation history.
- Missing optional channel SDKs degrade gracefully (channel skipped with
a warning; others unaffected). Channel-level errors are logged, not
crashed.
- Feishu's WebSocket client runs in a dedicated thread with its own
event loop to avoid cross-loop/contextvars conflicts with the channel
runtime.
**Frontend**
- **Settings → Chat channels** panel: available-channels grid +
configured-bots list with add/edit/delete and a **Connect assistant**
popup that binds a bot to a dialog.
- Brand icons via simple-icons / reused shared data-source assets, with
colored fallbacks for brands not available.
- Route, sidebar entry, i18n (en/zh), and a top-nav segment-boundary fix
so the settings page no longer highlights the Chat tab.
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
### Notes
- DB: new `chat_channel` table is auto-created; `chat_channel.dialog_id`
is also covered by a `migrate_db` `alter_db_add_column` for existing
installs.
- Channel SDKs (`lark-oapi`, `discord.py`, `python-telegram-bot`,
`line-bot-sdk`, `wechatpy`, `aiohttp`) added to dependencies.
- Screenshots / per-channel credential docs to follow.
<img width="1338" height="1290" alt="Image"
src="https://github.com/user-attachments/assets/042cb2f9-0dad-4e6a-bcf7-43ced4bbd704"
/>
<img width="1344" height="738" alt="Image"
src="https://github.com/user-attachments/assets/373cd08e-ec40-4c67-9c51-4d948b1ba617"
/>
<img width="672" height="887" alt="Image"
src="https://github.com/user-attachments/assets/5a34953f-a9a3-4c1e-869e-5eff0dc64c84"
/>
---------
### What problem does this PR solve?
Fix: chat/agent -- Default avatar is not displaying correctly.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
### What problem does this PR solve?
Refs #15743
Some Go API handlers return raw `err.Error()` strings in
`CodeServerError` responses. Those errors can include internal backend
details such as database, storage, search engine, or host information.
This PR adds a small shared `jsonInternalError` helper for handler-level
internal failures. The helper logs the raw error server-side with
request method/path context, then returns the existing generic
`common.CodeServerError.Message()` to API clients.
This first slice migrates the existing `jsonError(c,
common.CodeServerError, err.Error())` production call sites in agent,
dataset graph, file, and system handlers. It intentionally does not
close the full issue because direct `c.JSON` error responses in other
handlers remain for follow-up PRs.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
- [ ] New Feature (non-breaking change which adds functionality)
- [ ] Documentation Update
- [ ] Refactoring
- [ ] Performance Improvement
- [ ] Other (please describe):
### Tests
- `/root/go/bin/go test ./internal/handler -count=1`
---------
Co-authored-by: Yingfeng <yingfeng.zhang@gmail.com>
### What problem does this PR solve?
The Profile **Name** field currently lacks application-level validation
and allows users to save excessively long names and unsupported special
characters.
While the database enforces a maximum length of 100 characters, neither
the frontend nor backend validates nickname format before persistence.
This can result in inconsistent user data, poor user experience, and UI
layout issues when long names wrap across multiple lines.
This PR introduces consistent frontend and backend validation for
profile names, enforces length and character constraints, provides clear
validation feedback, and prevents invalid values from being saved.
Fixes#15693
### Type of change
* [x] Bug Fix (non-breaking change which fixes an issue)
## Summary
This PR passes `session_id` into Langfuse trace observations so
multi-turn chat messages can be grouped under the same session in
Langfuse.
Changes include:
- Propagate `session_id` from chat/session APIs into
`dialog_service.async_chat`.
- Pass `session_id` into Langfuse `start_observation(...)`.
- Share Langfuse `trace_context` with chat, embedding, rerank, and TTS
model bundles where applicable.
- Add unit coverage to verify Langfuse observations receive
`session_id`.
- Update affected test stubs for the new optional Langfuse context
arguments.
## Related Issue
Closes: #15636
## Change Type
- [x] Feature
- [x] Bug fix
- [x] Test
- [ ] Refactor
- [ ] Documentation
- [ ] Breaking change
## Real Behavior Proof
Before this change:
- Langfuse observations were created without `session_id`.
- Multi-turn chat traces could not be grouped by session in Langfuse.
After this change:
- Chat/session flows pass `session_id` into `async_chat`.
- Langfuse observations include `session_id`.
- Related model bundles receive shared trace context and session
metadata.
Validation result:
```bash
uv run python -m py_compile \
api/db/services/tenant_llm_service.py \
api/db/services/llm_service.py \
api/db/services/dialog_service.py \
api/db/services/conversation_service.py \
api/apps/restful_apis/chat_api.py \
test/unit_test/api/db/services/test_dialog_service_final_answer.py \
test/unit_test/api/db/services/test_dialog_service_use_sql_source_columns.py
```
Passed.
```bash
uv run pytest \
test/unit_test/api/db/services/test_dialog_service_final_answer.py \
test/unit_test/api/db/services/test_dialog_service_use_sql_source_columns.py -q
```
Result:
```text
11 passed in 16.89s
```
```bash
git diff --check
```
Passed.
## Checklist
- [x] Analyzed the issue requirement.
- [x] Checked existing Langfuse trace integration.
- [x] Implemented only the requested session grouping behavior.
- [x] Added/updated unit tests.
- [x] Ran focused tests successfully.
- [x] Ran Python compile validation.
- [x] Ran whitespace diff validation.
## Summary
Fixes#15695.
The Python GraphRAG path already accumulates similarity when several
N-hop paths produce the same edge, but PageRank was overwritten by the
last path. That makes ranking depend on path order for repeated edges.
This keeps the strongest PageRank seen for a repeated edge in the Python
implementation:
- `rag/graphrag/search.py`
The similarity score still accumulates exactly as before.
## To verify
- `python -m py_compile rag\graphrag\search.py`
- `git diff --check`
- `git diff --stat upstream/main` -> only `rag/graphrag/search.py`
I originally included the Go implementation too, but removed it after
maintainer feedback because the Go version is still under development
and not released yet.
### Summary
Closes#15423
`rag/llm/embedding_model.py` hosts about 40 embedding providers that
shared several defects affecting indexing reliability, cost accounting,
and error visibility. This PR fixes four concrete bugs.
**Masked, inconsistent errors (27 sites).** Nearly every provider ran
`log_exception(_e, res)` followed by `raise Exception(f"Error: {res}")`.
Because `log_exception` always raises, the second line was dead code,
and the surfaced exception varied with whether the SDK response exposed
a `.text` attribute. Every failure path now raises a single
`EmbeddingError` that includes the underlying response detail, so the
cause of a failed embedding is consistent and visible.
**Fabricated token counts.** `LocalAIEmbed` returned a hardcoded `1024`
and `OllamaEmbed` added `128` per text. These values feed `used_tokens`
and therefore billing and usage tracking. Both now report the real count
from the API (Ollama `prompt_eval_count`, LocalAI `usage`) and fall back
to a local token count only when the server omits it.
**Truncation overshoot.** The `8196` limit used by Mistral and Bedrock
exceeded the standard `8192` ceiling and could push boundary sized
inputs past the model limit. Limits are corrected to `8192` and made
intentional per provider, and providers that rely on server side
truncation now request it explicitly (Ollama `truncate=True`, Cohere
`truncate="END"`).
**Missing batching on Zhipu and Ollama.** Both issued one request per
text. They now batch like the other OpenAI compatible providers, turning
N round trips into `ceil(N / batch_size)`. Batched results are realigned
by response `index` so a chunk always keeps its own vector.
A shared `Base._batched_encode` helper owns the batch loop, optional
truncation, result accumulation, and the single error path. It is the
mechanism that lets these fixes live in one place instead of across 27
duplicated sites. The public `encode()` and `encode_queries()` contract
stays the same, so existing callers are unaffected.
Tests covering all four fixes are added under
`test/unit_test/rag/llm/test_embedding_model.py`.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
## Summary
Improves the responsiveness of the User Settings layout by converting
the left navigation sidebar into a compact icon-only rail on mobile
devices.
Previously, the sidebar retained its full desktop width on narrow
viewports, reducing the available space for settings content and making
pages such as **Data Sources** difficult to use on phones and smaller
tablets.
With this change:
- Desktop layouts retain the existing full sidebar experience
- Mobile layouts (<768px) display a compact 64px icon-only navigation
rail
- Main content receives significantly more horizontal space
- Navigation and logout actions remain fully accessible on mobile
## Type of Change
- [x] Bug fix
## Screenshots
| Before | After |
|---------|---------|
| <img width="557" height="760" alt="image"
src="https://github.com/user-attachments/assets/fb0d6a90-2d57-464c-90c6-9097418c7c13"
/> | <img width="557" height="760" alt="image"
src="https://github.com/user-attachments/assets/8db36d0f-7070-41e1-b7b2-0fe9d0cceefb"
/> |
## What Changed
### Mobile Sidebar Optimization
- Added responsive mobile behavior using `useIsMobile()`
- Displays avatar and navigation icons only on mobile
- Hides user email, navigation labels, version information, theme
switcher, and logout text on smaller screens
- Preserves navigation and logout functionality through icon actions
### Layout Improvements
- Updated settings page grid layout to use fixed sidebar widths:
- Mobile: `4rem` (64px)
- Desktop: `303px`
- Uses `minmax(0, 1fr)` for the content panel to prevent overflow and
allow proper shrinking
- Prevents sidebar width from expanding based on content
## Impact
- Improves usability of User Settings pages on phones and small tablets
- Increases available space for settings content
- Reduces horizontal crowding and overflow issues
- Maintains the existing desktop experience
## Test Plan
### Desktop (≥768px)
- Verify the full sidebar is displayed
- Confirm email, navigation labels, version information, theme switch,
and logout text are visible
- Ensure all navigation items function correctly
### Mobile (<768px)
- Verify the sidebar collapses to a 64px icon-only rail
- Confirm main content remains readable without horizontal crowding
- Verify navigation icons route correctly:
- Data Sources
- Model Providers
- MCP
- Team
- Profile
- API
- Confirm logout works from the icon button
### Verification
- Run `npm run build`
- Hard refresh when testing production or Docker deployments
- Verify responsive behavior using browser device emulation
### What problem does this PR solve?
The Go model-driver layer () has ~38,700 lines across 109 files. Roughly
74% of that is boilerplate duplicated into every driver: identical HTTP
client setup, the same 65-line SSE scanner loop, and 10-11 one-line "not
supported" stub methods per driver. Any fix must be manually propagated
to every file. Closes#15820.
This PR establishes the three shared utility files that form the
foundation for incremental driver migration:
---
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
- [x] Refactoring
---------
Co-authored-by: Haruko386 <tryeverypossible@163.com>
### What problem does this PR solve?
As title
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
- [x] New Feature (non-breaking change which adds functionality)
- [x] Refactoring
### What problem does this PR solve?
- Update version tags in README files (including translations) from
v0.25.6 to v0.26.0
- Modify Docker image references and documentation to reflect new
version
- Update version badges and image descriptions
- Maintain consistency across all language variants of README files
### Type of change
- [x] Documentation Update
### What problem does this PR solve?
- Replace embedding model `dimension` metadata with `max_dimension`.
- Add optional `dimensions` metadata for models with fixed selectable
output dimensions.
- Include `max_dimension` and `dimensions` in model list responses.
- Validate requested embedding dimensions before calling provider
embedding APIs.
- Forward SiliconFlow embedding dimensions with the correct `dimensions`
request field.
- Add unit coverage for embedding dimension validation rules.
## Summary
`get_model_config_from_provider_instance()` was not including
`max_tokens` in its returned dict, causing all downstream consumers
(dialog truncation, message fitting, knowledge base trimming, embedding,
graphrag, RAPTOR) to fall back to the hardcoded default of **8192
tokens** regardless of the actual model context window size (e.g.,
GPT-4o 128K, Claude 200K).
Closes#15944
## Root Cause
The function builds `model_config` with only: `llm_factory`, `api_key`,
`llm_name`, `api_base`, `model_type`, `is_tools`. `max_tokens` is never
included.
Yet the data exists in four independent sources:
1. `TenantModel.extra` JSON field — written by
`provider_api_service.py:659`
2. `conf/llm_factories.json` — every model entry has `max_tokens`
3. `rag/llm/model_meta.py` — 9 provider classes fetch real context
windows from APIs
4. `TenantLLM.max_tokens` database column
None of them are read by this function.
## Fix
Two lines added, one per return path:
- **Path B** (model_obj exists → provider-instance model): reads
`max_tokens` from `model_obj.extra` JSON
- **Path C** (fallback → factory config): reads `max_tokens` from
`llm_info` (sourced from `llm_factories.json`)
Both fall back to 8192 when the value is absent, preserving backward
compatibility.
## Impact
This single 5-line change fixes the context window budget for all **78+
call sites** across **20 files** that construct `LLMBundle` or read
`max_tokens` from the config dict, including:
| Consumer | File | Effect |
|---|---|---|
| Dialog chat truncation | `dialog_service.py:562` |
`message_fit_in(msg, max_tokens * 0.95)` now uses real context window |
| Knowledge base trimming | `dialog_service.py:752` |
`kb_prompt(kbinfos, max_tokens)` now fits more retrieved content |
| Agent message fitting | `agent/component/llm.py:322` | Agent prompts
no longer truncated at 7946 tokens |
| Embedding truncation | `task_executor.py:704` | Embedding input uses
actual model limit |
| GraphRAG extraction | `graphrag/*/extractor.py` | Entity extraction
gets full context budget |
| LLM4Tenant.max_length | `tenant_llm_service.py:513` | Chat model
wrapper exposes real context window |
### What problem does this PR solve?
- Add `web/src/locales/ko.ts` with full Korean translation (~3100 keys)
- Register `Ko = 'ko'` in `LanguageAbbreviation` enum (`common.ts`)
- Add `[LanguageAbbreviation.Ko]: '한국어'` to `LanguageAbbreviationMap`
- Add lazy-load entry in `web/src/locales/config.ts`
- Add `korean` key to all existing locale files (`ja`, `id`, `es`,
`pt-br`, `vi`, `zh-traditional`)
- Fix duplicate enum value `FileMimeType.Mdx` (`'text/markdown'` →
`'text/mdx'`)
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
- [x] Other (please describe): Korean (한국어) i18n translation + fix
duplicate FileMimeType.Mdx enum value
### What problem does this PR solve?
Fix: Remove the pagination from the search and retrieval pages.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
## What problem does this PR solve?
Closes#15874
Both the `POST /api/v1/datasets/<dataset_id>/chunks` (re-parse) and
`DELETE /api/v1/datasets/<dataset_id>/chunks` (stop-parsing) handlers
called `settings.docStoreConn.delete` unconditionally. When the
tenant/dataset index has not been created yet — fresh dataset, first
parse interrupted before any chunks were indexed, or index manually
removed — the delete call throws and the handler returns HTTP 500
**after** the document state was already mutated (RUNNING with zeroed
counters for the parse path; CANCEL with zeroed counters for the stop
path), leaving the document in an inconsistent state.
The newer `parse_documents` path in `document_api.py` already uses
`index_exist` before deleting:
## How to fix?
Apply the same `index_exist` guard to both call sites in `chunk_api.py`:
- **`parse`** (POST path, line ~192): guard the delete before
`TaskService.filter_delete`.
- **`stop_parsing`** (DELETE path, line ~242): guard the delete after
`DocumentService.update_by_id`.
Both sites already have the correct `search.index_name(tenant_id)` and
`dataset_id` parameters; the guard is a one-line addition at each site.
## Type of change
- [x] Bug fix (non-breaking change which fixes an issue)
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Wang Qi <wangq8@outlook.com>
Fixes#15587
## Problem
`AzureEmbed.__init__` in `rag/llm/embedding_model.py` and
`AzureGptV4.__init__` in `rag/llm/cv_model.py` both call
`json.loads(key)` unconditionally:
```python
api_key = json.loads(key).get("api_key", "")
api_version = json.loads(key).get("api_version", "2024-02-01")
```
When a user stores a plain API key string (not a JSON object) in the
model configuration — which is a valid and common way to configure Azure
OpenAI — `json.loads` raises `JSONDecodeError`. This makes the model
fail to initialize and causes document parsing/embedding to return a 500
error.
## Fix
Wrap `json.loads` in `try/except (json.JSONDecodeError, TypeError)` and
fall back to using the raw string as the `api_key` with the default
`api_version`. This is the same pattern already applied to the Azure
chat model in PR #15604.
## Files changed
- `rag/llm/embedding_model.py` — `AzureEmbed.__init__`
- `rag/llm/cv_model.py` — `AzureGptV4.__init__`
Fixes#15857
---------
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Wang Qi <wangq8@outlook.com>
Fixes#15529 .
### Problem
`async_ask()` accessed `kbs[0]` without verifying that
`KnowledgebaseService.get_by_ids()` returned any knowledge bases. Empty
or stale `kb_ids` raised `IndexError`, which surfaced as HTTP 500 on
search/bot SSE endpoints.
### Fix
- Add an early guard when `kbs` is empty, yielding a final SSE error
event (consistent with `gen_mindmap()` in the same module).
- Add regression tests for empty `kb_ids` and deleted/invalid KB IDs.
### Test plan
- [ ] `pytest
test/unit_test/api/db/services/test_dialog_service_final_answer.py -k
"async_ask_empty or async_ask_stale"`
- [ ] Manual: `POST /api/v1/searchbots/ask` with invalid `kb_ids`
returns SSE error, not HTTP 500
---------
Co-authored-by: Wang Qi <wangq8@outlook.com>
### What problem does this PR solve?
`POST /api/v1/datasets/{dataset_id}/documents/stop`
(`stop_parse_documents`) cancels parsing tasks and sets `run` to
`CANCEL`, but it does **not** remove chunks already indexed in the doc
store or reset `progress` / `chunk_num`. REST callers can end up with a
“cancelled” document that still returns partial chunks in `GET
.../chunks` and in retrieval.
Legacy `DELETE /api/v1/datasets/{dataset_id}/chunks` (`stop_parsing`)
already performs full cleanup: it resets counters and calls
`docStoreConn.delete`. This PR aligns the newer stop endpoint with that
behavior so both paths leave the dataset consistent.
Fixes [#15788](https://github.com/infiniflow/ragflow/issues/15788).
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
- [ ] New Feature (non-breaking change which adds functionality)
- [ ] Documentation Update
- [ ] Refactoring
- [ ] Performance Improvement
- [ ] Other (please describe):
### Changes
- Update `stop_parse_documents` in `document_api.py` to reset `progress`
and `chunk_num` to `0` and delete partial chunks via
`docStoreConn.delete` after `cancel_all_task_of`.
- Add unit test `test_stop_parse_documents_cleans_partial_chunks` to
assert counters reset and doc store delete is invoked.
### Test plan
- [x] Unit test: `pytest
test/testcases/test_http_api/test_file_management_within_dataset/test_doc_sdk_routes_unit.py::TestDocRoutesUnit::test_stop_parse_documents_cleans_partial_chunks
-v`
- [ ] Manual: upload a slow document, start parse, call `POST
.../documents/stop` while `RUNNING`, verify `GET .../chunks` returns
zero chunks and UI `chunk_count` is 0
- [ ] Control: legacy `DELETE .../chunks` behavior unchanged
---------
Co-authored-by: Wang Qi <wangq8@outlook.com>
## Summary
- Normalize WebDAV file-size metadata before applying the sync size
threshold.
- Enforce the same threshold for numeric string sizes in both document
sync and slim snapshot paths.
- Add focused WebDAV unit coverage for size parsing and over-threshold
skips.
## Why
Some WebDAV servers return file sizes from PROPFIND metadata as strings.
The previous threshold check only handled integer values, so oversized
files could still be downloaded and sent into the chunking pipeline.
Closes#15724.
## Validation
- `uv run --no-project --with pytest --with pytest-asyncio pytest
test/unit_test/data_source/test_webdav_connector_unit.py -q`
- `uvx ruff check common/data_source/webdav_connector.py
test/unit_test/data_source/test_webdav_connector_unit.py`
- `python -m compileall -q common/data_source/webdav_connector.py
test/unit_test/data_source/test_webdav_connector_unit.py`
- `git diff --check`
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
## Summary
Fixes#15699.
User upgrades to v0.25.6 against an existing MySQL database, tries to
add an Ollama provider instance, and gets:
```
MySQL IntegrityError: Duplicate entry 'dbaafbfe608a11f1a5516d6066988224'
for key 'tenant_model_instance.tenantmodelinstance_api_key_provider_id'
```
The route at
[api/apps/restful_apis/provider_api.py:354](api/apps/restful_apis/provider_api.py#L354)
catches it and returns `get_error_data_result(message="Internal server
error")` — which by RAGFlow's convention is HTTP 200 with an error
`code` on the body — hence the reporter's "200 status code but the
database errored" complaint.
### Root cause
The provider-instance refactor in [PR
#15460](https://github.com/infiniflow/ragflow/pull/15460) dropped the
unique-compound-index tuple from `TenantModelInstance`:
```python
# Removed in #15460
class Meta:
db_table = "tenant_model_instance"
indexes = (
(("api_key", "provider_id"), True), # unique
)
```
and added a one-shot drop in `migrate_db()` for existing databases. But
the drop targets the wrong index name:
```python
# Before this PR — wrong name
for table_name, index_name in [
("tenant_model_instance", "idx_api_key_provider_id"), # ← doesn't exist
("tenant_model", "idx_provider_model_instance"),
]:
```
Peewee's auto-derived index name is `<lowercase
classname>_<col1>_<col2>` →
**`tenantmodelinstance_api_key_provider_id`**, which matches the user's
error verbatim. The drop raises `OperationalError: 1091 (HY000): Can't
DROP …`, the surrounding `except` clause at
[db_models.py:1736](api/db/db_models.py#L1736) swallows it as
expected-on-fresh-installs, and the legacy unique index lives on
indefinitely.
### Why Ollama hits it specifically
Ollama doesn't require an API key. The form posts `api_key: ""`. The
app-layer dedupe at
[provider_api_service.py:288-292](api/apps/services/provider_api_service.py#L288-L292):
```python
api_key_str = ""
if api_key: # ← skipped for ""
...
same_key_instance = TenantModelInstanceService.get_by_provider_id_and_api_key(...)
if same_key_instance:
return False, f"Already exist instance: ... with api_key {api_key}"
```
falls through for empty keys. Control reaches
`TenantModelInstanceService.create_instance(..., api_key="")` which
inserts a row whose `(api_key, provider_id) = ("", <provider_uuid>)`
collides with any prior Ollama row that already shipped that same pair →
the still-present unique index throws.
(`dbaafbfe608a11f1a5516d6066988224` in the user's error is the
duplicated `provider_id` UUID, paired with the empty `api_key`.)
### Fix
Add the Peewee auto-name alongside the existing `idx_*` entry so the
migration finally drops the obsolete index on next restart:
```python
legacy_indexes = [
("tenant_model_instance", "idx_api_key_provider_id"),
("tenant_model_instance", "tenantmodelinstance_api_key_provider_id"), # ← added
("tenant_model", "idx_provider_model_instance"),
]
```
The surrounding `try/except (OperationalError, ProgrammingError)`
matches `1091` / `can't DROP` / `does not exist` and treats them as
success, so every state is idempotent (see Test plan).
### Idempotency matrix
| Database state | First entry (`idx_api_key_provider_id`) | New entry
(`tenantmodelinstance_api_key_provider_id`) |
| --- | --- | --- |
| Fresh install (≥ #15460) — neither index exists | `1091` → swallowed |
`1091` → swallowed |
| Upgraded from before dc4b82523 (the user's case) — auto-name present |
`1091` → swallowed | **drops the index** |
| Upgraded after a manual rename to `idx_*` | drops the index | `1091` →
swallowed |
| Re-run of `migrate_db()` after either of the above | `1091` →
swallowed | `1091` → swallowed |
No rollback hazard: nothing depends on this unique constraint anymore
(`create_instance` dedupes by `instance_name` via `duplicate_name`, see
[tenant_model_instance_service.py:27](api/db/services/tenant_model_instance_service.py#L27)).
### What this PR does NOT change
- **`provider_api_service.create_provider_instance`** — its `if
api_key:` gate is correct *for the post-migration world*: multiple
Ollama instances with empty keys under one provider are legitimate, so
we shouldn't tighten the app-layer check.
- **`TenantModelInstance` Peewee model** — the `indexes` tuple was
already removed in #15460. New databases never get the constraint in the
first place.
- **The `except → get_error_data_result` → HTTP 200 pattern at
`provider_api.py:354`** — that's a project-wide convention; changing one
route to HTTP 500 would be inconsistent and out of scope.
## Test plan
- [ ] **Reproducer (pre-fix):** on a database originally created before
#15460, configure an Ollama provider with an empty `api_key`, then try
to create a *second* instance under the same provider — confirm the
`Duplicate entry … 'tenantmodelinstance_api_key_provider_id'` error in
the server log.
- [ ] **Verify the index is present pre-restart:** `SHOW INDEX FROM
tenant_model_instance WHERE Key_name =
'tenantmodelinstance_api_key_provider_id';` — non-empty result.
- [ ] **Restart with the fix applied:** server starts cleanly,
`migrate_db()` runs, no `Failed to drop index` in critical logs.
- [ ] **Verify the index is gone post-restart:** same `SHOW INDEX` query
— empty result.
- [ ] **Re-run the reproducer:** two Ollama instances under the same
provider, both `api_key=""`, both succeed.
- [ ] **Restart a second time** — no new errors; the matching `1091`
swallow keeps the migration idempotent.
- [ ] **Fresh install smoke test:** drop the DB volume, start clean — no
`1091` noise (the new index never existed), no functional regression.
## Files changed
- [api/db/db_models.py](api/db/db_models.py) — extend the legacy-index
drop list with `tenantmodelinstance_api_key_provider_id`; refactor the
inline list to a named `legacy_indexes` local with a comment pointing at
#15460 and #15699.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
- [ ] New Feature (non-breaking change which adds functionality)
- [ ] Documentation Update
- [ ] Refactoring
- [ ] Performance Improvement
- [ ] Other (please describe):
Co-authored-by: Wang Qi <wangq8@outlook.com>
## Summary
Fixes [#15585](https://github.com/infiniflow/ragflow/issues/15585).
- Route markdown preview through the shared `request` client (same as
txt/image previewers) so `Authorization` headers and interceptors are
applied consistently.
- Add a unit test covering `AUTH_BETA` token loading for embedded search
auth.
## Root cause
Search result preview for `.md`/`.mdx` used raw `fetch`, which did not
apply the same auth path as other preview types. That led to `401` on
`GET /api/v1/documents/{id}/preview` even when the user was logged in or
using an embedded search `auth` query param.
## Test plan
- [ ] Log in, run a search, open a markdown citation link — preview
loads (no 401).
- [ ] Open an embedded shared search URL with `auth` query param,
preview a markdown file — preview loads.
- [ ] Confirm PDF/txt preview still works in the same search UI.
---------
Co-authored-by: MkDev11 <89318445+bitloi@users.noreply.github.com>
Co-authored-by: Wang Qi <wangq8@outlook.com>
## Summary
Fixes#15532 — `delete_datasets()` crashes with `IndexError` when a
document has no `File2Document` row.
`delete_datasets()` in `dataset_api_service.py` called
`File2DocumentService.get_by_document_id()` and immediately accessed
`f2d[0].file_id` without checking whether the lookup returned any rows.
Documents created via API ingestion or connector sync may exist without
a linked file record, causing dataset deletion to abort with HTTP 500.
This PR mirrors the existing guard already used in `file_service.py` and
`document_api_service.py`.
### What problem does this PR solve?
FIx replicate model provider failing with valid api key
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
---------
Co-authored-by: Wang Qi <wangq8@outlook.com>
## Summary
Fixes#15409.
Reporter sees scary ERROR-level stack traces in `ragflow_server.log` on
every chat turn against a knowledge base whose spreadsheet has many
columns with embedded IDs (e.g. `id-wstc-bios fvt-322-wstc-bios
fvt-323`). Simple queries work; complex ones return "No answer" with
logs that look like a hard crash.
### What's actually happening
1. The user uploads a wide Excel/CSV.
[rag/app/table.py:477-493](rag/app/table.py#L477-L493) turns each header
into an ES field with a type suffix, e.g. `id-wstc-bios
fvt-322-wstc-bios fvt-323_tks`. This is correct — the parser faithfully
encodes the user's column names.
2. The user asks about test case `fvt-085`. The SQL chat path in
[api/db/services/dialog_service.py:914
use_sql](api/db/services/dialog_service.py#L914) asks the LLM to write
SQL using the field list. The LLM sees the `id-wstc-bios
fvt-NNN-wstc-bios fvt-MMM_tks` pattern and pattern-completes a
plausible-but-nonexistent column.
3. Elasticsearch rejects with `BadRequestError(400,
'verification_exception')`: `Unknown column [id-wstc-bios
fvt-085-wstc-bios fvt-086_tks]` and suggests the closest valid column.
4. **The recovery path already exists**: `use_sql` catches the
exception, re-prompts the LLM with the error text (which contains ES's
"did you mean" hint), and on second failure the caller at
[api/db/services/dialog_service.py:626](api/db/services/dialog_service.py#L626)
falls back to vector search. The chat does produce an answer — it's just
generated from the vector hits instead of SQL.
The only real bug is logging:
-
[common/doc_store/es_conn_base.py:399](common/doc_store/es_conn_base.py#L399)
catches every exception with `self.logger.exception(...)`, which writes
a full traceback at **ERROR** level.
- For LLM-generated SQL this is the hot path, not an exceptional
condition — it can fire twice per turn before the fallback runs.
### Fix
Catch `elasticsearch.BadRequestError` (the parent class of
`verification_exception` / `parsing_exception` / similar SQL-validity
errors) separately and log it at **WARNING** with the SQL plus ES error
message. The message still carries the unknown column name and ES's
suggested alternative, so it's actionable for anyone investigating "why
is my LLM producing bad SQL?" — just without the misleading stack trace.
Other exception types (`ConnectionTimeout`, generic `Exception`) keep
their original `ERROR`-level traceback treatment; those represent real
connectivity / library bugs.
This is a one-file, two-line-net change. The retry loop in `use_sql`,
the `add_kb_filter` injection, and the vector-search fallback are all
unchanged.
### What this PR does NOT change
- **The LLM prompts in `use_sql`** — they already specify `Use EXACT
field names from the schema` and pass the field list explicitly.
Strengthening them risks regressing well-behaved cases and is out of
scope for #15409.
- **The single-retry policy** — extending it to multi-retry with
extracted ES suggestions is a separate enhancement.
- **The parser at `rag/app/table.py`** — the field names match the
user's actual column headers; the parser is doing its job.
## Files changed
- [common/doc_store/es_conn_base.py](common/doc_store/es_conn_base.py)
- Add `BadRequestError` to the `elasticsearch` import.
- In `ESConnectionBase.sql()`, add an `except BadRequestError` arm above
the generic `except Exception` that logs at WARNING and re-raises (so
`use_sql` retry/fallback still triggers).
### What problem does this PR solve?
When embedding a chatbot, the API returned `"Model Name is required"`.
The embed widget now includes the assistant's `llm_id` as `model_name`
in the completion request.
### Type of change
- [x] Bug Fix
### How has this been tested?
- Created a chatbot with a default model.
- Embedded it and sent a message – the error is gone and the assistant
replies correctly.
### Related Issue
Closes#15883
Co-authored-by: RAGFlow Dev <dev@ragflow.local>
Co-authored-by: Wang Qi <wangq8@outlook.com>
### What problem does this PR solve?
Fix: The regular expression configuration for pipeline header-based
chunking will be reset.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
### What problem does this PR solve?
Two commands are used for ingestion file testing
```
RAGFlow(api/default)> chunk 'file' with 'dsl';
Chunk file: file, DSL: dsl
SUCCESS
RAGFlow(api/default)> parse file 'filename' chat 'xxx';
Success to parse local file "filename", vision: , chat: xxx, asr: , ocr: , embedding: , doc_parse:
SUCCESS
```
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
Signed-off-by: Jin Hai <haijin.chn@gmail.com>
### What problem does this PR solve?
`ChatStreamlyWithSender` in two Go model drivers could panic on nil
pointer dereferences when a caller passes a nil model config or omits
the reasoning `Effort`:
- **deepseek.go** - `switch *chatModelConfig.Effort` dereferenced
`Effort` without a nil check. It now defaults to `"high"` when nil.
- **volcengine.go** - the `modelConfig` pointer itself was dereferenced
(`Stream`, `MaxTokens`, `Temperature`, .) with no guard, and `Effort`
was dereferenced unchecked. `modelConfig` now defaults to an empty
`&ChatConfig{}` when nil so the optional-field accesses are safe, and
`Effort` defaults to `"medium"` when nil.
Addresses the CodeRabbit review on `volcengine.go`
`ChatStreamlyWithSender`. Per maintainer feedback ("one PR do one
thing"), the unrelated `handler/auth.go` and
`service/heartbeat_sender.go` changes were removed so this PR is scoped
to the model-provider fixes.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
## Summary
Fixes#15790.
Every Discord sync launched from the current Web UI crashes immediately
with:
```
'list' object has no attribute 'split'
```
The error is raised in
[rag/svr/sync_data_source.py:650-651](rag/svr/sync_data_source.py#L650-L651):
```python
server_ids=server_ids.split(",") if server_ids else [],
channel_names=channel_names.split(",") if channel_names else [],
```
### Root cause
Three independent bugs stack here, all in the Discord branch of
`sync_data_source.py`:
1. **Type mismatch (the user's exact error).** The current form at
[web/src/pages/user-setting/data-source/constant/index.tsx:833-843](web/src/pages/user-setting/data-source/constant/index.tsx#L833-L843)
uses `FormFieldType.Tag` for both **Server IDs** and **Channels**:
```tsx
{ label: 'Server IDs', name: 'config.server_ids', type:
FormFieldType.Tag, required: false },
{ label: 'Channels', name: 'config.channels', type: FormFieldType.Tag,
required: false },
```
Tag inputs serialise to **lists**, not comma-separated strings. The
backend `.split(",")` then explodes on the very first sync.
2. **Field-name mismatch.** The form writes `config.channels`. The
backend reads `self.conf.get("channel_names", None)`. Even if
`.split(",")` were fixed, channels would silently be empty for every
UI-created source.
3. **Int conversion missing.**
[common/data_source/discord_connector.py:82](common/data_source/discord_connector.py#L82)
types `server_ids` as `list[int]` (Discord guild IDs are integers); the
previous `.split(",")` produced strings, so the `channel.guild.id not in
server_ids` filter at
[discord_connector.py:92](common/data_source/discord_connector.py#L92)
silently never matched.
So even the configurations that didn't crash were also broken — there is
no path through the current code that actually filtered by server id
from a UI-created source.
### Fix
A 39-line patch in one function:
- New `Discord._coerce_str_list` static method: accepts `None` / `""` /
`list` / `tuple` / `set` / scalar / comma-separated str, returns a clean
`list[str]` with whitespace trimmed and empty entries dropped.
Smoke-tested against the 10 input shapes that can hit it (see Test
plan).
- `_generate` reads `config.channels` first (the form's actual key) and
falls back to `config.channel_names`, so SDK callers and legacy configs
that already shipped with the old key keep working.
- `server_ids` is coerced to `list[int]`. Non-integer entries are logged
and dropped instead of crashing the sync, so a single malformed tag from
the form doesn't tank the rest of the run.
### What this PR does NOT change
- **Web form key (`config.channels`)** — kept as-is. Renaming it to
`channel_names` would force a UI migration and break in-flight configs;
the backend fallback solves the same problem more safely.
- **`common/data_source/discord_connector.py`** — its signature was
already correct.
- **Other connectors (Slack, Gmail, Confluence, etc.)** — they don't
crash today and were not in the issue's scope.
## Test plan
`Discord._coerce_str_list` has been exercised against all ten realistic
input shapes — list, tuple, set, comma-separated string, str with extra
whitespace, empty entries, integers from a Tag input, None, empty list,
single trailing comma. All pass.
### What problem does this PR solve?
Two bugs in the Agent Categorize component:
1. The backend rejected `message_history_window_size = 0` while frontend
allowed it, causing API errors.
2. When calling the agent API without a `session_id`, a new session was
created but retained history from previous conversations.
### Type of change
- [x] Bug Fix
### How has this been tested?
- Issue 1: `CategorizeParam().check()` now accepts `0` and rejects
negative values.
- Issue 2: `canvas.clear_history()` is called for new sessions (no
`session_id`), ensuring fresh conversation state. Verified via UI and
API that a second call without `session_id` does not remember the first
conversation.
### Related Issue
Closes#15897
Co-authored-by: RAGFlow Dev <dev@ragflow.local>
Co-authored-by: Wang Qi <wangq8@outlook.com>
## Summary
Implements **chunk 1** of #15282 — the four `/api/v1/auth/password/...`
endpoints from the login-page Go port. **Chunk 2 (OAuth/OIDC) is
deferred** to its own subtask, matching the issue author's own
confidence-low recommendation ("multi-provider, stateful redirect flow
with external dependencies; recommend its own subtask").
New endpoints, all registered under `apiNoAuth` (forgot-password users
are unauthenticated by definition):
| Method | Path | Status |
|--------|------|--------|
| `POST` | `/api/v1/auth/password/forgot/captcha` | new |
| `POST` | `/api/v1/auth/password/forgot/otp` | new |
| `POST` | `/api/v1/auth/password/forgot/otp/verify` | new |
| `POST` | `/api/v1/auth/password/reset` | new |
## Wire compatibility with the Python backend
The two backends share state through Redis, so the Go port had to use
identical keys, encodings, and constants. Either backend can now
validate a code the other minted.
- **Redis keys**: `captcha:<email>`, `otp:<email>`,
`otp_attempts:<email>`, `otp_last_sent:<email>`, `otp_lock:<email>`,
`otp:verified:<email>` — same as `api/utils/web_utils.py`.
- **Stored OTP value**: `"<hex_hash>:<hex_salt>"` — same as Python.
- **Hash**: HMAC-SHA256 with a `crypto/rand` 16-byte salt — same as
`hash_code()`.
- **Constants**: `OTP_LENGTH=4`, `OTP_TTL=5min`, `ATTEMPT_LIMIT=5`,
`ATTEMPT_LOCK_SECONDS=30min`, `RESEND_COOLDOWN_SECONDS=60s` — all match
`api/utils/web_utils.py`.
- **Email body**: matches `RESET_CODE_EMAIL_TMPL` byte-for-byte.
## Files
### New
| File | Purpose |
|---|---|
| `internal/utility/otp.go` | OTP/captcha constants, Redis key builders
(`CaptchaRedisKey`, `OTPRedisKeys`, `OTPVerifiedRedisKey`),
`HashOTPCode`, `GenerateOTPCode` / `GenerateCaptchaCode` /
`GenerateOTPSalt` via `crypto/rand`, and `EncodeOTPStorageValue` /
`DecodeOTPStorageValue` matching Python's storage shape. |
| `internal/utility/smtp.go` | Minimal stdlib `net/smtp` sender.
`SendResetCodeEmail(to, otp, ttlMin)` builds an RFC 5322 plain-text
message and dispatches via implicit TLS / STARTTLS / plain — same
selectors as Python `aiosmtplib`. Returns `SMTPNotConfiguredError` if
the config block is empty. |
### Modified
| File | Change |
|---|---|
| `internal/server/config.go` | New `SMTPConfig` struct + `Config.SMTP`
field. Field names mirror the `smtp:` keys in `common/settings.py`
(`mail_server`, `mail_port`, `mail_use_ssl`, `mail_use_tls`,
`mail_username`, `mail_password`, `mail_from_name`, `mail_from_address`,
`mail_frontend_url`) so a single `conf/service_conf.yaml` powers both
backends. |
| `internal/service/user.go` | Four methods — `ForgotIssueCaptcha`,
`ForgotSendOTP`, `ForgotVerifyOTP`, `ForgotResetPassword`. Reuses the
existing `decryptPassword`, `HashPassword`, `userDAO.Update`, and
`utility.GenerateToken` so the reset+auto-login path is identical to
`LoginByEmail`. |
| `internal/handler/user.go` | Four handlers in the same `c.JSON` shape
as `LoginByEmail`. The reset handler rotates the access token and emits
an `Authorization` header for auto-login (matches Python
`construct_response(auth=user.get_id())`). |
| `internal/router/router.go` | Routes registered under `apiNoAuth`,
with an explanatory comment on why they sit outside the auth middleware.
|
## Known divergence — captcha rendering
The Python endpoint returns a rendered `image/JPEG` from the
`python-captcha` library. The Go side has **no image-captcha dependency
vendored** in `go.mod`, and hand-rolling a raster generator was out of
scope for this PR.
This commit returns JSON `{captcha: "<text>"}` instead. Implications:
- **Backend gate is identical** — the OTP step still verifies the
user-submitted captcha string against the Redis value, so the security
model is unchanged.
- **Frontend impact**: the password-reset page rendering needs a small
tweak (text display instead of `<img>`) until a Go captcha library is
wired in.
- The handler comments call this out explicitly so the next PR knows
what to swap.
Possible follow-ups (any one closes the gap):
1. Add `github.com/mojocn/base64Captcha` or `github.com/dchest/captcha`
to `go.mod` and replace the JSON response with an `image/JPEG`.
2. Hand-roll a 5x7 bitmap font + `image/png` writer using only the
stdlib.
3. Render a server-side SVG (cheap, but trivially OCR-able — only useful
as a UI shim).
## Test plan
- [ ] **Captcha**: `POST
/api/v1/auth/password/forgot/captcha?email=<existing>` returns `{code:
0, data: {captcha: "ABCD"}}`. Redis shows `captcha:<email>` with that
value and ~60s TTL. Unknown email returns `code: CodeDataError`.
- [ ] **OTP send**: `POST /api/v1/auth/password/forgot/otp` with the
right captcha mints an OTP, stores `<hash>:<salt>` under `otp:<email>`
for 5 min, sends an email, returns success. With a wrong captcha returns
`CodeAuthenticationError`. Hitting it again within 60s returns "you
still have to wait …" with `CodeNotEffective`.
- [ ] **OTP verify**: correct OTP → `code: 0`, OTP keys cleared,
`otp:verified:<email>` = `"1"`. Wrong OTP → `code:
CodeAuthenticationError`, attempt counter bumped; after 5 wrong tries
`otp_lock:<email>` is set and further attempts hit `CodeNotEffective`.
- [ ] **Reset**: with the verified flag set, supply a new password
(RSA-encrypted+base64, same as `LoginByEmail`). Returns `code: 0`,
`Authorization` header set, verified flag deleted. Without the verified
flag returns `CodeAuthenticationError`.
- [ ] **Wire-compat smoke**: mint an OTP from the Python backend, verify
it via the Go endpoint, and vice versa. Should both succeed.
- [ ] **SMTP misconfigured**: drop `smtp.mail_server` from
`conf/service_conf.yaml`. The OTP-send endpoint should now return
"failed to send email" without panicking; check the log for the
`SMTPNotConfiguredError` warning.
- [ ] **End-to-end FE**: hit the password-reset flow from
`web/src/pages/login-next/`. Confirm the text-captcha shim works after
the FE tweak.
- [ ] `go build ./...` and `go vet ./...` — I could not run these in the
sandbox; please confirm a clean build before merging.
- [ ] `uv run pytest` to confirm no Python regressions (shared Redis
schema).
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
## Summary
- Implemented the Go API endpoint for Memory message forgetting:
- `DELETE /api/v1/messages/{memory_id}:{message_id}`
- Added route registration for the Memory message DELETE endpoint only.
- Added request path validation for `memory_id:message_id`.
- Added service logic to mark a message as forgotten by setting
`forget_at`.
- Preserved Python-compatible response behavior:
- Success returns `code: 0`, `message: true`, `data: null`.
- Added focused unit tests for message path parsing and invalid message
ID handling.
- Fixed Linux cgo linker config to use the installed shared PCRE2
library so Go tests/builds can run in this environment.
## Related Issue
Closes: #15240
## Change Type
- [x] Feature
- [x] Test
- [x] Build / CI compatibility
## Implemented API
- `DELETE /api/v1/messages/{memory_id}:{message_id}`
## Real Behavior Proof
Validated with targeted Go tests:
```bash
/tmp/go1.25.0/bin/go test ./internal/handler ./internal/router
```
Result:
```text
ok ragflow/internal/handler
? ragflow/internal/router [no test files]
```
Validated server entrypoint build:
```bash
/tmp/go1.25.0/bin/go build -o /tmp/ragflow-server-main ./cmd/server_main.go
```
Result:
```text
build succeeded
```
Validated patch formatting:
```bash
git diff --check
```
Result:
```text
no whitespace errors
```
## Checklist
- [x] Implemented only `DELETE
/api/v1/messages/{memory_id}:{message_id}`.
- [x] Did not implement unrelated Memory message APIs.
- [x] Added route registration.
- [x] Added handler validation.
- [x] Added service-level memory access check.
- [x] Added tests.
- [x] Ran targeted Go tests.
- [x] Ran server build validation.
- [x] Ran `git diff --check`.
### What problem does this PR solve?
1. Fix go test, some cases still failed.
2. Remove unused code.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
---------
Signed-off-by: Jin Hai <haijin.chn@gmail.com>
## Summary
- Add official Qwen models to `conf/all_models.json` with `qwen/`
canonical names
- Include verified aliases from official Qwen/Hugging Face model IDs and
common provider naming
- Add metadata for context length, model types, thinking support, and
embedding dimensions
## Details
- Added Qwen model families from the official Hugging Face Qwen
organization
- Normalized canonical model names to the `qwen/...` format
- Preserved official HF IDs and lowercase/common aliases for lookup
compatibility
- Added `dimension` for Qwen embedding models
- Added or corrected `max_tokens` for Qwen model families, including:
- Qwen2.5 Instruct variants
- Qwen3 original, 2507, VL, Coder, Coder-Next, Next, Embedding, and
Reranker models
- Qwen3.5 and Qwen3.6 models
- QwQ models
- Added verified `thinking` metadata where officially supported
- Corrected `model_types` for Qwen Image, Omni, Audio, VL, embedding,
reranker, benchmark, and tokenizer entries
## Summary
- Normalize model alias index keys to lowercase
- Detect lowercase alias collisions during provider manager
initialization
- Fix ListModels metadata mapping for mixed-case provider aliases
### What problem does this PR solve?
Fix: add image2text/speech2text/ocr support
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
### What problem does this PR solve?
Fix: add thin scrollbar styling for x-spreadsheet component
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
### What problem does this PR solve?
Fix: The dataset retrieval test returned an incorrect total number.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
---------
Co-authored-by: balibabu <assassin_cike@163.com>
### What problem does this PR solve?
Ensure agent components with image inputs route to `image2text` models
instead of staying on the chat path, so visual requests use the CV
wrapper when supported.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
## Summary
LocalAI exposes two API surfaces with conflicting naming conventions:
- `GET /api/tags` returns model names with `:latest` suffix (Ollama
format)
- `POST /v1/chat/completions` expects names without `:latest` (OpenAI
format)
RAGFlow discovered models via `/api/tags` and stored the tagged name,
then used it with `/v1/chat/completions`, causing a 404 error because
LocalAI didn't recognize `model:latest`.
## Fix
In `LocalAI.get_model_list()`, strip the tag suffix from model names
using `model["name"].rsplit(":", 1)[0]`, so stored names match what the
OpenAI-compatible endpoints expect.
### What problem does this PR solve?
Set OpenDataLoader and call in parser and naive
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
## What does this PR do?
This PR migrates the Agent Temporary File Download endpoint (`GET
/api/v1/agents/download`) from the Python backend to the Go backend,
optimizing the data retrieval flow and maintaining strict functional
parity. It also fixes a persistent parsing error in the Sandbox code
execution node.
## Checklist
- [x] Code logic matches Python implementation
- [x] All local unit tests passed
- [x] No breaking changes to existing router interfaces
Co-authored-by: Yingfeng <yingfeng.zhang@gmail.com>
### What problem does this PR solve?
1. remove unused code
2. fix login issue
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
- [x] Refactoring
Signed-off-by: Jin Hai <haijin.chn@gmail.com>
### What problem does this PR solve?
cohere api call failing because of missing prefix
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
### What problem does this PR solve?
Support factory models with multiple model types, so visual chat models
can be exposed as both image2text and chat while preserving the database
model-type-per-record design.
This also updates the SILICONFLOW model list and adds a helper script to
refresh SiliconFlow models from the provider API.
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
### What problem does this PR solve?
fix: rename ark_api_key to api_key for volcengine provider config
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
### What problem does this PR solve?
Fix: Fix some model provider-related UI issues
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
This PR expands conf/all_models.json with DeepSeek model entries and
provider aliases.
Changes:
- Added DeepSeek model entries across `V4`, `V3.2`, `V3.1`, `V3`, `R1`,
`Coder`, `Math`, `VL`, `OCR`, `Prover`, `MoE`, and `LLM` series.
- Normalized model name values to lowercase canonical IDs.
- Added alias values for official DeepSeek/Hugging Face names and
provider-specific names from OpenRouter, VolcEngine, SiliconFlow,
HuaweiCloud, and QiniuCloud.
- Preserved model metadata such as max_tokens, model_types, and thinking
where applicable.
- Added Gitee ListModels tests to verify DeepSeek aliases map back to
model metadata from all_models.json.
- Added an optional Gitee integration test gated by
GITEE_LIST_MODELS_INTEGRATION=1.
Test:
/usr/local/go/bin/go clean -cache
/usr/local/go/bin/go test ./internal/entity/models -run
'TestGiteeListModels(MapsAllDeepSeekAliasesToModelMetadata|KeepsOwnedBySuffixAfterAliasMetadataLookup|
Integration)'
### What problem does this PR solve?
Fix QWen rerank error handling so DashScope error responses without a
text attribute do not raise a secondary KeyError and hide the real
provider error.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
### What problem does this PR solve?
The RAGFlow Docker image was 9.06 GB with build-only compiler packages
leaking into the runtime, duplicate frontend source shipped alongside
compiled assets, and no .dockerignore causing ~6 GB of unnecessary
context transfer per build.
### Type of change
- [x] Performance Improvement
## Summary
- Add knowledge-base retrieval support to Go chat completions.
## What changed
- Routes KB-backed chat sessions through the Go retrieval service
instead of falling back to solo chat.
- Resolves embedding and rerank models, validates accessible knowledge
bases, and preserves tenant-aware retrieval.
- Rejects mixed embedding models across selected knowledge bases before
retrieval to avoid incompatible vector dimensions.
- Threads the HTTP request context into streaming retrieval so cancelled
requests can stop downstream retrieval work.
- Applies metadata filters and message-level `doc_ids` before retrieval.
- Expands parent/child chunks before building references and prompt
context.
- Injects retrieved knowledge through a copied dialog prompt config so
the caller's original dialog is not mutated.
- Honors configured empty responses when no chunks are found.
- Names the metadata no-match sentinel and reuses it across
retrieval/handler paths.
- Adds a defensive content cast while appending streamed answers.
- Adds focused unit coverage for retrieval, metadata filtering,
authorization, multimodal messages, references, empty-response behavior,
prompt immutability, and mixed embedding models.
---------
Co-authored-by: Yingfeng <yingfeng.zhang@gmail.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
### What problem does this PR solve?
```
RAGFlow(api/default)> use admin
SUCCESS
RAGFlow(api/default)> use api 'abc';
SUCCESS
```
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
---------
Signed-off-by: Jin Hai <haijin.chn@gmail.com>
### What problem does this PR solve?
Updated supported model providers and the corresponding URLs.
~~Synced supported model providers and base URLs with
**llm_factories.json**, while keeping the AI Badgr configuration example
via the OpenAI-API-Compatible provider.~~
### Type of change
- [x] Documentation Update
### What problem does this PR solve?
As title
```
/api/v1/datasets/<dataset_id>/documents/<document_id>/metadata/config PUT
```
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
Co-authored-by: Yingfeng <yingfeng.zhang@gmail.com>
### What problem does this PR solve?
Add two API in go
```
/api/v1/agents/test_db_connection POST
/api/v1/agents/<agent_id>/sessions DELETE
```
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
---------
Co-authored-by: Yingfeng <yingfeng.zhang@gmail.com>
Implements POST /api/v1/searchbots/ask in Go with streaming SSE,
citations, and think-tag processing. 23 files, 90+ unit tests.
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
## Summary
This PR re-enables the Go test steps in CI that were previously
commented out, and fixes all compilation errors that have accumulated in
`internal/entity/models/` since the `ListModels` return type was changed
from `[]string` to `[]ListModelResponse`.
## Changes
### CI (`.github/workflows/tests.yml`)
- Re-enable **Prepare test resources** step (clones resource repo with
WordNet data)
- Re-enable **Test Go packages** step (runs `go test ./internal/...`)
- Fix resource path race condition by using
`/tmp/resource-${GITHUB_RUN_ID}` instead of `/tmp/resource`
- Exclude `/cli` package from Go tests (contains `main` redeclarations)
### Test fixes (16 model provider test files)
All errors were caused by the upstream change from `[]string` to
`[]ListModelResponse` in the `ListModels` interface:
- Add `joinModelNames` test helper to extract `.Name` from
`[]ListModelResponse` slices
- `strings.Join(models, ",")` → `joinModelNames(models, ",")` (11 files)
- `ids[i] != "..."` → `ids[i].Name != "..."` (cometapi, mistral)
- `got[i] != want[i]` → `got[i].Name != want[i]` (bedrock)
- `[]string` return types → `[]ListModelResponse` (google)
### Pre-existing bugs in model_test.go
Bugs introduced by the upstream `entity/` → `entity/models/` directory
rename:
- Add missing `pm := GetProviderManager()` calls in 3 test functions
- Fix `InitProviderManager` signature (`_, err :=` → `err :=`)
- Fix `MaxTokens` `*int` dereference (6 comparisons)
- Fix `readProviderConfig` relative path (3 levels up instead of 2)
### model.go
- Add `findRepoRoot()` to make `conf/all_models.json` resolution work
from any CWD, fixing `TestSiliconFlowProviderConfigLoadsLatestProModels`
### Test validation
```bash
go build ./internal/... # ✅
go test ./internal/entity/models/... -count=1 # ✅ all pass
```
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
### Summary
Closes#15795
Knowledge-graph queries rank entities by `pagerank * sim` in `KGSearch`,
but the entity chunks written at index time stopped carrying the values
that ranking depends on. `graph_node_to_chunk` only stored
`entity_type`, `description`, and `source_id`, dropping the node
`pagerank` and the n-hop neighbour paths, while `search.py` still read
them back as `rank_flt` and `n_hop_with_weight`.
The producer of these fields, `update_nodes_pagerank_nhop_neighbour`,
was removed in #6513, but the read side in `KGSearch` was never updated.
The result is that on every knowledge-graph query:
- `pagerank` resolves to `0`, so the `pagerank * sim` sort key is `0`
for every entity and selection falls back to arbitrary order.
- Every displayed entity score is `0.00`.
- The n-hop relation-enrichment block is dead code because `n_hop_ents`
is always empty, leaving `merge_tuples` and `is_continuous_subsequence`
orphaned.
This PR restores the missing index-time fields so the documented `P(E|Q)
= pagerank * sim` ranking and the n-hop enrichment work again.
What changed:
- `graph_node_to_chunk` now writes `rank_flt` from the node pagerank and
`n_hop_with_weight` from the recomputed n-hop neighbour paths.
- Reintroduced the n-hop path computation (`n_neighbor`) in
`rag/graphrag/utils.py`, reusing the previously orphaned `merge_tuples`
/ `is_continuous_subsequence` helpers, with a direction-agnostic
edge-weight lookup for undirected graphs. `set_graph` computes the paths
per added or updated node and passes them through.
- `KGSearch` now selects `n_hop_with_weight` in the entity keyword
search so Infinity and OceanBase return it (Elasticsearch and OpenSearch
already read it from `_source`), and the read is hardened against
missing keys or empty strings before `json.loads`.
- Added the `n_hop_with_weight` column to OceanBase, including the
`EXTRA_COLUMNS` migration entry so existing tables get it. The other
engines already map both fields via dynamic templates or the Infinity
mapping.
Scope note: pagerank and n-hop are re-indexed for the added or updated
nodes in each pass, consistent with the existing incremental indexing
design.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
### Testing
Added unit tests in
`test/unit_test/rag/graphrag/test_graphrag_utils.py`:
- `n_neighbor`: path and weight shape, one-hop vs two-hop, isolated
nodes, missing weights, and direction-agnostic lookup.
- `graph_node_to_chunk`: `rank_flt` populated from pagerank and
defaulting to `0`, `n_hop_with_weight` serialized and defaulting to an
empty list.
```
uv run pytest test/unit_test/rag/graphrag/ # 106 passed
uv run ruff check rag/graphrag/ rag/utils/ob_conn.py
```
## Summary
- Complete the Go Elasticsearch result functions that remained stubbed
after #15160.
- Add focused unit coverage for field mapping, aggregation, IDs, and
highlighting behavior.
- Update a stale query-builder test type import discovered during
validation.
## What changed
- Keep the Elasticsearch Go implementation merged in #15160 and fill in
`GetFields`, `GetAggregation`, `GetHighlight`, and `GetDocIDs` in
`internal/engine/elasticsearch/chunk.go`.
- Add regression and invariant coverage in
`internal/engine/elasticsearch/chunk_helpers_test.go`.
- Update `internal/service/nlp/query_builder_test.go` to use the current
`types.MatchTextExpr` type.
## Why
- #15160 implemented the main Go Elasticsearch surface, but
retrieval/tag flows still call result functions that returned stubs.
- Completing these functions keeps Elasticsearch result processing
aligned with the expected document-engine behavior for field extraction,
tag aggregation, doc ID extraction, and snippet highlighting.
## Validation
- `go test ./internal/engine/elasticsearch`
- `GOARCH=arm64 CGO_ENABLED=1 go test ./internal/service/nlp -run
TestQueryBuilder`
- `git diff --check`
- CodeRabbit review reported 0 issues after follow-up fixes.
- Codex Security diff scan found no reportable issues.
## Notes
- This PR is now a follow-up to #15160 rather than a competing
implementation.
- A full local `go test ./internal/service/nlp` run is blocked by local
WordNet resource prerequisites; the query-builder tests touched by this
PR pass with the arm64 CGO path.
### What problem does this PR solve?
Fix: The variables in the Visual Input File of the agent operator are
not displayed.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
### What problem does this PR solve?
Fixes four Go paths that dereference a pointer with no prior nil check,
each
causing a **runtime panic**. Closes#15814.
| # | File | Bug | Fix |
|---|------|-----|-----|
| 1 | `internal/entity/models/deepseek.go` | streaming path runs `switch
*chatModelConfig.Effort` inside `if *Thinking`; panics when
`Thinking=true` and `Effort==nil` | nil-check with default `"high"`,
matching the non-streaming path in the same file |
| 2 | `internal/entity/models/volcengine.go` | identical oversight:
`switch *modelConfig.Effort` with no guard | nil-check with default
`"medium"`, matching its non-streaming path |
| 3 | `internal/handler/auth.go` | `AuthMiddleware` does `if
*user.IsSuperuser`; panics on every authenticated request when the DB
column is `NULL` | guard with `user.IsSuperuser != nil &&`, matching
every other call site (`admin/handler.go`, `admin/service.go`,
`user.go`) |
| 4 | `internal/service/heartbeat_sender.go` |
`responseBody["code"].(float64)` panics on any non-200 response lacking
a numeric `code`; the upstream `recover()` calls `Fatal()` →
`os.Exit(1)`, taking down the whole server | comma-ok assertion (`code,
ok := ...`); return an error instead of panicking |
- [x] Bug Fix (non-breaking change which fixes an issue)
### What problem does this PR solve?
Fixes four panic / spurious-error paths in the Go model layer. Closes
#15818.
| # | File | Bug | Fix |
|---|------|-----|-----|
| 1 | | Thinking-mode streaming path: accessed unconditionally; Gemini
emits usage-only chunks with an empty slice, causing a runtime panic |
Guard each step: , , before indexing |
| 2 | | is a plain for ordinary requests; the cast to silently returns ,
then panics immediately | Switch on concrete type; handle both and |
| 3 | | Identical panic on the streaming path | Same switch-on-type fix
|
| 4 | | The field is optional (absent for non-thinking models) but the
code returned an error when it was missing, breaking every ordinary
Ollama completion | Change to a silent comma-ok assertion; is empty
string when the field is absent |
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
### What problem does this PR solve?
feat: support custom editing for model list
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
## Problem
When users configure auto-metadata for a dataset, parsing crashes with:
```
KeyError: 'properties' in gen_metadata → schema["properties"]
```
## Root Cause
Pydantic `AutoMetadataField` defaults `enum` and `description` to `None`
when the frontend omits these fields:
```python
class AutoMetadataField(Base):
enum: Annotated[list[str] | None, Field(default=None)]
description: Annotated[str | None, Field(default=None)]
```
These `None` values propagate through the call chain and cause two
crashes:
### What problem does this PR solve?
Dedup api_key and migrate `is_tools `in migration.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
### What problem does this PR solve?
When setting the API key for the BaiduYiyan provider, all model
validations fail with the error "Fail to access model using this api
key. No valid response received".
**Root cause:**
1. `BaiduYiyanChat` in `rag/llm/chat_model.py` does not override
`async_chat_streamly()`. The `verify_api_key()` function uses
`mdl.async_chat_streamly()` to validate, but `BaiduYiyanChat` inherits
`Base.async_chat_streamly()` which uses the OpenAI client, not the Baidu
Qianfan SDK (qianfan). Since BaiduYiyan has no OpenAI-compatible
base_url, validation always fails.
2. `verify_api_key()` in `provider_api_service.py` does not format the
raw API key string into the JSON format (`{"yiyan_ak": "...",
"yiyan_sk": "..."}`) that `BaiduYiyanChat.__init__()` expects via
`json.loads(key)`.
**Fix:**
1. Add `async_chat_streamly()` method to `BaiduYiyanChat` using the
qianfan SDK, consistent with the existing `chat_streamly()` method.
2. Add BaiduYiyan API key formatting in `provider_api_service.py`
`verify_api_key()` to match the format expected by
`BaiduYiyanChat.__init__()`.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
- [ ] New Feature (non-breaking change which adds functionality)
- [ ] Documentation Update
- [ ] Refactoring
- [ ] Performance Improvement
- [ ] Other (please describe):
### What problem does this PR solve?
Force image parser runtime output format to JSON so downstream chunking
reads OCR results from the JSON output and image parser chunks can be
displayed.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
Co-authored-by: Wang Qi <wangq8@outlook.com>
### What problem does this PR solve?
The last column definition `INDEX idx_instance_id (instance_id),` in the
`CREATE TABLE tenant_model` statement has a trailing comma, which causes
a MySQL syntax error during deployment.
Closes#15832
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
### How was this tested?
- [x] Visual inspection: the trailing comma on line 837 has been removed
### What problem does this PR solve?
Propagate `tenant_id` from memory task messages into task collection so
refactored task execution can build a valid context.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
### Description
This PR ports the `GET /api/v1/agents/prompts` endpoint from the Python
backend to the Go backend.
### Changes Made
- **Handler**: Added `GetPrompts` method to `internal/handler/agent.go`.
- **Router**: Registered the `agents.GET("/prompts")` route in
`internal/router/router.go`.
- **Logic**: Leveraged the existing `service.LoadPrompt` utility to read
`analyze_task_system`, `analyze_task_user`, `next_step`, `reflect`, and
`citation_prompt` templates directly from the `rag/prompts` directory.
- **Unit Test**: Added `TestGetPrompts_Success` to
`internal/handler/agent_test.go` to mock the HTTP context and validate
the JSON response structure.
### Motivation
This is part of the ongoing effort to port the Agent API surface to Go.
Since this specific endpoint only serves static prompt templates and
does not require the complex DAG/Canvas execution engine, it can be
seamlessly and safely handled by the Go backend.
### Testing
- [x] Added automated unit test `TestGetPrompts_Success` (verified
passing).
- [x] Tested locally via `curl` against the Go server (port 9380) and
Python server (port 9384).
- [x] Verified that the Go JSON response structure and loaded prompt
text are logically 100% identical to the Python implementation.
### What problem does this PR solve?
Fix: An error message appears when accessing the agent's launch page:
"pagesize exceeds maximum value".
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
---------
Co-authored-by: balibabu <assassin_cike@163.com>
## Summary
This PR adds checkpoint/resume support for the GraphRAG
`extract_community` and `resolve_entities` stages.
The implementation stores successful intermediate results in the
document store so interrupted ingestion can resume without repeating
already-completed LLM work. Checkpoints are loaded before each stage,
reused when available, saved after successful batch/community
processing, and cleaned up after the stage completes successfully.
## Related Issue
Closes: #15518
## Change Type
- [x] Feature
- [x] Bug fix
- [x] Test
- [ ] Refactor
- [ ] Documentation
- [ ] Breaking change
## Real Behavior Proof
Validation commands run locally:
```bash
uv run python -m py_compile \
rag/graphrag/checkpoints.py \
rag/graphrag/general/community_reports_extractor.py \
rag/graphrag/entity_resolution.py \
rag/graphrag/general/index.py \
test/unit_test/rag/graphrag/test_checkpoints.py
```
Result:
```text
Passed
```
```bash
uv run pytest test/unit_test/rag/graphrag/test_checkpoints.py
```
Result:
```text
4 passed
```
```bash
uv run pytest \
test/unit_test/rag/graphrag/test_phase_markers.py \
test/unit_test/rag/graphrag/test_graphrag_utils.py \
test/unit_test/rag/graphrag/test_checkpoints.py
```
Result:
```text
95 passed
```
```bash
git diff --check
```
Result:
```text
Passed
```
## Checklist
- [x] Implemented checkpoint/resume support for `extract_community`.
- [x] Implemented checkpoint/resume support for `resolve_entities`.
- [x] Avoided touching unrelated API behavior.
- [x] Added unit tests for the new checkpoint helper logic.
- [x] Verified Python syntax compilation.
- [x] Ran related GraphRAG unit tests successfully.
- [x] Ran `git diff --check`.
- [ ] Ran full project test suite.
---------
Co-authored-by: Wang Qi <wangq8@outlook.com>
### What problem does this PR solve?
Display intl `base_url` for Tongyi-Qianwen
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
### What problem does this PR solve?
fix: resolve issue where some models do not use modelInfo parameter
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
### What problem does this PR solve?
OceanBase could start without the `ragflow` tenant, so RAGFlow failed to
connect with `root@ragflow`. This PR adds a safe startup reconcile step
and documents the required host limits before using OceanBase.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
- [x] Documentation Update
## Summary
The embedding branch of `verify_api_key` was missing `await` on
`asyncio.wait_for(...)`, so valid embedding-only providers always failed
API-key verification.
## Root cause
`arr, tc = asyncio.wait_for(...)` (no `await`) returns a coroutine;
unpacking it raises `TypeError: cannot unpack non-iterable coroutine`,
which the `except` swallows as a failure. The chat branch (`await
asyncio.wait_for(check_streamly())`) and rerank branch (`arr, tc = await
asyncio.wait_for(...)`) already `await` correctly.
## Fix
```diff
- arr, tc = asyncio.wait_for(
+ arr, tc = await asyncio.wait_for(
asyncio.to_thread(mdl.encode, ["Test if the api key is available"]),
timeout=timeout_seconds,
)
```
## Files changed
- `api/apps/services/provider_api_service.py`
## Verification
- `ruff check` — clean
- Fix mirrors the already-correct chat/rerank branches in the same
function. Local full pytest not run (heavy RAG deps); CI validates.
## Note
Implemented with LLM assistance (model: claude-opus-4-8).
Closes#15619
Co-authored-by: dearsishs <MCarter112116@outlook.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
## Summary
`SearchService.get_detail` crashed with `AttributeError` (HTTP 500) when
no matching row existed, because it called `.first().to_dict()` before
the `if not search` guard — making that guard dead code.
## Root cause
`.first()` returns `None` when the query matches nothing (deleted search
app, or joined `User` not `VALID`). `None.to_dict()` raises before the
guard runs.
## Fix
```diff
.first()
- .to_dict()
)
if not search:
return {}
- return search
+ return search.to_dict()
```
Guard the `None` first, then serialize — restoring the intended `{}`
"not found" return that every caller (`search_api`, `bot_api`,
`chat_api`, `dataset_api_service`) already handles.
## Files changed
- `api/db/services/search_service.py`
## Verification
- `ruff check` — clean
- Logic: `.first()` → `None` now hits `return {}` instead of
`None.to_dict()`. Local full pytest not run (heavy RAG deps); CI
validates.
## Note
Implemented with LLM assistance (model: claude-opus-4-8).
Closes#15621
Co-authored-by: dearsishs <MCarter112116@outlook.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
## Problem
`TestXiaomiNewModelWithCustomDefaultTransport` panics on Go 1.25:
```
panic: interface conversion: http.RoundTripper is models.roundTripperFunc, not *http.Transport
```
In Go 1.25, `http.DefaultTransport` is no longer `*http.Transport`, so
the unchecked type assertion in `NewXiaomiModel` panics when the test
replaces it with a `roundTripperFunc`.
## Fix
Use a safe type assertion with fallback to a new `http.Transport`,
matching the pattern already used in `modelscope.go`.
## Verification
```bash
go test -run TestXiaomiNewModelWithCustomDefaultTransport ./internal/entity/models/...
# PASS
```
Internal contributors only.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
### What problem does this PR solve?
`is_english()` in `rag/nlp/__init__.py` compiles a **single-character**
regex class and `fullmatch`es it against each item:
```python
pattern = re.compile(r"[`a-zA-Z0-9\s.,':;/\"?<>!\(\)\-]") # no quantifier
...
eng = sum(1 for t in texts if pattern.fullmatch(t.strip()))
```
For a **string** argument the text is first split into single characters
(`texts = list(texts)`), so each `fullmatch` sees one character and
works. But for a **list** argument each item is a whole multi-character
string, and `fullmatch` of a one-character pattern against a
multi-character string always fails — so `is_english()` returns `False`
for **any** list, regardless of content.
```python
is_english("This is English") # True (ok)
is_english(["The quick brown fox jumps.", "Hello world."]) # False (bug — should be True)
is_english(["这是中文。"]) # False (right answer, wrong reason)
```
Many call sites pass lists and were therefore silently always-`False`,
e.g.:
- `rag/llm/chat_model.py:1088`, `rag/llm/cv_model.py:168,1155` —
`is_english([ans])` when an answer is truncated at `max_tokens`, so an
English reply gets the Chinese "······由于长度的原因，回答被截断了，要继续吗？" continuation
suffix instead of the English one.
- `rag/app/book.py` — `remove_contents_table(...,
eng=is_english([...sections...]))`, so English books have their contents
table stripped in Chinese mode.
- `common/doc_store/es_conn_base.py:339`,
`rag/utils/opensearch_conn.py:733` — `is_english(txt.split())` in
highlight handling.
- plus `rag/app/qa.py`, `rag/flow/parser/utils.py`,
`common/doc_store/infinity_conn_base.py`.
### Fix
Add a `+` quantifier so an all-English multi-character item matches:
```python
pattern = re.compile(r"[`a-zA-Z0-9\s.,':;/\"?<>!\(\)\-]+")
```
The string path is unchanged (single characters still match) and
non-English lists still return `False`. Adds
`test/unit_test/rag/test_is_english.py`; the two list cases fail before
this change and pass after.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
Used the Claude CLI while working on this.
### What problem does this PR solve?
Support model list for VolcEngine.
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
## What problem does this PR solve?
Go test files are never compiled in CI — only production binaries via
`go build`. This allowed a missing `"sort"` import in
`metadata_filter_test.go` to be merged without detection.
## Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
- [x] New Feature (non-breaking change which adds functionality)
## Changes
- Add `go test -count=1 ./internal/...` step after Go build in CI
workflow
- Fix missing `"sort"` import in `metadata_filter_test.go` (pre-existing
compile error)
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
## Summary
Normalizes Qwen model-family names before reasoning extraction so
provider-prefixed Qwen models use the existing `<think>...</think>`
fallback.
## Summary
- keep MiniMax chat calls in non-streaming mode and streaming calls in
SSE mode
- make MiniMax model listing and connection checks use a bodyless GET
/v1/models
- add focused MiniMax request/response regression tests
### What problem does this PR solve?
**Verified from CLI**
```
RAGFlow(user)> chat with 'mimo-v2.5@test@xiaomi' message 'who r u'
Answer: Hello! I'm MiMo-v2.5, a large language model developed by Xiaomi's LLM Core Team. You can think of me as a friendly AI assistant ready to help you answer questions, have conversations, or work on creative tasks. My context window can handle up to 1 million tokens, so we can dive into pretty long discussions or documents if you'd like. What can I help you with today?
Time: 3.831830
RAGFlow(user)> stream chat with 'mimo-v2.5@test@xiaomi' message 'who r u'
Answer: there! I'm MiMo-v2.5, an AI assistant created by the Xiaomi LLM Core Team. I'm here to chat, help out, answer questions, or just have a friendly conversation. Think of me as a helpful buddy with a pretty big memory (1 million tokens worth!). What can I do for you today?😊
Time: 2.421630
RAGFlow(user)> think chat with 'mimo-v2.5@test@xiaomi' message 'who r u'
Thinking: The user is asking a simple question about who I am. According to my system prompt, I should:
- Identify myself as **MiMo-v2.5**
- State that I was developed by the **Xiaomi LLM Core Team**
- Answer in first person and be warm and conversational
Answer: Hey there! 👋
I'm **MiMo**, an AI assistant created by the **Xiaomi LLM Core Team**. Think of me as a friendly chat buddy who's here to help you with all sorts of questions and tasks!
I love having conversations, answering questions, brainstorming ideas, and helping people figure things out. Whether you want to chat, need help with something specific, or just want to explore ideas together — I'm here for it! 😊
What can I help you with today?
Time: 6.651589
RAGFlow(user)> tts with 'mimo-v2.5-tts@test@xiaomi' text 'hello? show yourself' play format 'wav' param '{"voice": "Chloe"}'
SUCCESS
RAGFlow(user)> asr with 'mimo-v2.5-asr@test@xiaomi' audio './internal/test.wav' param '{"language": "zh"}'
+------------------------------------------------------------------------------------------------------------------------+
| text |
+------------------------------------------------------------------------------------------------------------------------+
| 1 The examination and testimony of the experts enabled the commission to conclude that five shots may have been fired. |
+------------------------------------------------------------------------------------------------------------------------+
```
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
- [x] Refactoring
## Summary
- keep Moonshot chat calls in non-streaming mode and streaming calls in
SSE mode
- make Moonshot model listing and balance checks use bodyless GET
requests
- add focused Moonshot request/response regression tests
### What problem does this PR solve?
Fix LM-Studio provider connection verification so embedding checks await
the async wrapper correctly and log the full traceback on failures.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
### What problem does this PR solve?
Fix: Add a waiting status to the messages on the chat page.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
### What problem does this PR solve?
fix: Resolve error when checking pipeline parsing result
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
### What problem does this PR solve?
Table parser metadata aggregation previously only ran when
`table_column_mode` was set to `manual`. In auto mode (default), all
columns default to `"both"` role, meaning they should also be aggregated
into document-level metadata for UI/chat filters. Additionally, the task
snapshot could be stale — `table_column_names` are written to KB
`parser_config` during `chunk()` but the task may have been created
before that.
Changes:
- Renames `aggregate_table_manual_doc_metadata` →
`aggregate_table_doc_metadata`
- Supports both `"manual"` and `"auto"` `table_column_mode` (defaults to
`"auto"`)
- Reloads `table_column_names` from KB DB when missing from task
snapshot
- Removes the manual-only guard in `task_executor` and refactored
`post_processor`
- Updates all tests with new function name and adds auto mode test cases
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
### What problem does this PR solve?
This PR adds `ModelMeta` implementations for four additional LLM/RAG
ecosystem platforms, building on the ModelMeta infrastructure introduced
in #15711.
Currently, only `Ollama` and `VolcEngine` have `ModelMeta` classes that
enable remote model list fetching. This PR extends that support to four
more platforms.
### Changes
Added four new `ModelMeta` subclasses in `rag/llm/model_meta.py`:
| Platform | `_FACTORY_NAME` | Has model list | Has full model info |
Approach |
|----------|-----------------|----------------|---------------------|----------|
| **Xinference** | `"Xinference"` | ✅ | ✅ | Parses `model_type` and
`context_length` from `/v1/models` response. Maps 6 model types
(LLM/embedding/rerank/image/TTS/speech2text). |
| **LocalAI** | `"LocalAI"` | ✅ | ✅ | Uses Ollama-compatible `GET
/api/tags` + `POST /api/show` endpoints. Returns capabilities
(completion/embedding/vision/tools/thinking) and
`general.context_length`. |
| **BaiduYiyan** | `"BaiduYiyan"` | ✅ | ✅ | Uses Qianfan SDK static
model catalog + `get_model_info()` for `max_input_tokens`. Returns 60
models (56 chat + 4 embedding) with real context lengths. |
| **Tencent Cloud** | `"Tencent Cloud"` | ❌ | ❌ | `NotImplementedError`
— uses SDK-based SID/SK HMAC signing, no model list REST API available.
|
All classes are automatically discovered and registered via the existing
`__init__.py` mechanism — no additional configuration needed.
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
### What problem does this PR solve?
Browser parsed sys.query from prompts but never called set_input_value,
so node_finished inputs displayed null in the agent orchestration run
log.
Additionally, Browser’s tenant-model path could trigger unsupported
structured-output modes (response_format/tool_choice) for some
OpenAI-compatible providers (notably DeepSeek thinking models), causing
step failures.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
Fixes#15413
### What problem does this PR solve?
In **Settings → Model providers**, the *Available models* panel lets you
filter
providers by model type (All, LLM, Embedding, Rerank, TTS, ASR, VLM, …),
but the
filter tags gave no hint of how many providers fall under each type.
Users had to
click a tag to find out, and an empty category looked identical to a
populated one.
This PR adds a count to each filter tag in `AvailableModels`:
- The **All** tag shows the total number of providers currently listed.
- Each model-type tag shows how many providers offer that model type.
- Counts respect the active search term, so the badge always matches the
number of
cards shown once that tag is selected.
- Each provider is counted once per model type (deduplicated via a
`Set`), so a
provider that lists the same type more than once isn't double-counted.
Counts are rendered with `tabular-nums` for stable width and dimmed via
`opacity-60`
so they read as secondary to the label. No API changes; the existing
filter logic is
untouched — this is purely an additive UI affordance.
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
### What problem does this PR solve?
Follow-up to #15393. After #15393 fixed the OpenSearch `search()`
signature and
the doc-meta mapping, document metadata still renders as **"0 fields"**
for every
document on the OpenSearch backend (`DOC_ENGINE=opensearch`).
**Root cause.** `OSConnection.insert()` pops `id` out of the document
before
indexing:
meta_id = d_copy.pop("id", "") # id used as _id, then DROPPED from
_source
so the stored `_source` never contains an `id` field. But the doc-meta
read path
filters and sorts on that field:
- `DocMetadataService.get_metadata_for_documents()` builds
`condition = {"kb_id": kb_id, "id": doc_ids}` -> `OSConnection.search()`
emits
`Q("terms", id=doc_ids)` (a term query on the `id` field), and
- `_search_metadata()` sorts with `order_by.asc("id")`.
With `id` absent from `_source`, the terms filter matches nothing, so
`get_metadata_for_documents()` returns an empty map and the UI shows "0
fields"
-- even though the metadata was written correctly (it is visible via a
kb_id-only query).
`ESConnection.insert()` already keeps `id` (`d_copy.get("id", "")`) with
the
comment *"also keep 'id' as a regular field for sorting"*. This is a
plain
OpenSearch-only divergence (`pop()` vs `get()`).
### Fix
Mirror Elasticsearch: use `get("id")` instead of `pop("id")` so `id`
survives in
`_source`. The doc-meta mapping already declares `id` as `keyword`, so
the field
is searchable/sortable once populated.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
### Affected backends
OpenSearch only. Elasticsearch already keeps `id`; Infinity / OceanBase
unaffected.
### How to reproduce
1. `DOC_ENGINE=opensearch`, create a KB, upload/parse a document, set
metadata.
2. Open the document list -> every document shows "0 fields" (the
metadata exists
in the `ragflow_doc_meta_*` index but its `_source` has no `id` field).
### Risk & backward compatibility
`insert()` is shared with the main chunk index; keeping `id` in
`_source` brings
OpenSearch in line with Elasticsearch (which already does this), so it
is parity,
not new behavior. No default / ES / Infinity / OceanBase behavior
change.
Note: affects new inserts only. Existing `ragflow_doc_meta_*` indices
created
before this change have no `id` in `_source`; re-sync metadata, or
backfill once
with `_update_by_query` (`ctx._source.id = ctx._id`).
### Test plan
- [ ] OpenSearch: after the fix the document list shows correct metadata
field
counts (not "0 fields"); metadata filter/sort by id works.
- [ ] Elasticsearch regression: unchanged.
### What problem does this PR solve?
`RAGFlowExcelParser.html()` iterates `(len(rows) - 1) // chunk_rows + 1`
times. `rows[0]` is the header, so `len(rows) - 1` is the data-row
count. When that count is an exact multiple of `chunk_rows`, the `+ 1`
over-counts by one: the final iteration's data slice is empty, but the
header row is still appended — producing a chunk that contains only the
table header and no data.
This is reachable via `rag/app/naive.py` (`html4excel`, `chunk_rows=12`)
and `rag/app/one.py`. A sheet with 12/24/36… data rows (or 256/512… with
the default `chunk_rows=256`) produces an extra
`<table><caption>…</caption><tr><th>…</th></tr></table>` chunk. It is
non-empty, so it passes the `if _` filter and gets indexed as a real
(empty) chunk.
| data rows (chunk_rows=12) | before | after |
|---|---|---|
| 12 | 2 chunks (1 header-only) | 1 |
| 24 | 3 chunks (1 header-only) | 2 |
| 13 | 2 (unchanged) | 2 |
### Fix
Iterate `ceil(n_data / chunk_rows)` times instead of `n_data //
chunk_rows + 1`. Adds
`test/unit_test/deepdoc/parser/test_excel_parser.py`; the
header-only-chunk cases fail before this change and pass after.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
Used the Claude CLI while working on this.
## Summary
Closes#15720
`FulltextQueryer.paragraph` normalized its `content_tks` token string
with `[c.strip() for c in content_tks.strip() ...]`, which iterates the
string **character by character** — `"machine learning model"` becomes
20 single characters instead of 3 tokens. Those single chars are fed to
`tw.weights(..., preprocess=False)`, producing meaningless term weights
and a garbage `MatchTextExpr`.
`paragraph()` backs `Dealer.tag_content` (the KB auto-tagging feature),
so tag retrieval/scoring is silently broken for tag-enabled knowledge
bases. Every other method in this file tokenizes with `.split()` — this
is a `.strip()`-vs-`.split()` typo.
## Change
- `rag/nlp/query.py` — change `content_tks.strip()` to
`content_tks.split()` in the `paragraph` token-normalization line.
## Why it's safe
- The caller passes a space-separated token string; `.split()` recovers
the real tokens, matching the contract of `tw.weights` and the
`.split()` tokenization used by the sibling methods (`similarity`,
`question`).
- No behavior depends on the per-character expansion.
## Verification
- `python -m py_compile rag/nlp/query.py` — OK.
- Demonstrated: `"machine learning model"` → 20 single-character entries
before, 3 real tokens after. No test references `paragraph`.
Co-authored-by: seekmistar01 <seekmistar01@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
## What problem does this PR solve?
Implements `FetchChunkVectors` — the infrastructure needed to hydrate
chunk embedding vectors on demand. This is a prerequisite for
`insert_citations` (citation insertion in the `searchbots/ask`
endpoint), matching the Python `Dealer.fetch_chunk_vectors` pattern.
Without this, citation insertion cannot compute answer-vs-chunk vector
similarity.
## Type of change
- [x] New Feature (non-breaking change which adds functionality)
## Changes
### New Function
- `FetchChunkVectors(engine, chunkIDs, tenantIDs, kbIDs, dim)` — fetches
embedding vectors for a set of chunk IDs
- Consumer-side `vectorFetcher` interface with only `Search` + `GetType`
methods
- Both `*elasticsearchEngine` and `*infinityEngine` implicitly satisfy
the interface
### Engine Behavior
- **ES**: queries by chunk ID list via `Search` with filter `{"id":
chunkIDs}`, parses tab-separated `q_N_vec` string format
- **Infinity / OceanBase**: skips the round-trip (vectors already
shipped with chunks)
- **Degrades gracefully**: engine errors return zero vectors — citation
insertion will use placeholders instead of failing
### Vector Parsing
- Handles ES tab-separated string format (`"0.1\t0.2\t0.3"`)
- Handles `[]float64` and `[]interface{}` formats
- Returns zero vector for wrong-dimension or unparseable input
### Bug Fix
- `metadata_filter_test.go`: add missing `"sort"` import (pre-existing
build break)
### Tests
- 12 unit tests: empty input, Infinity/OceanBase skip, ES string vector,
ES float slice, ES interface slice, search error degradation, missing
chunk → zero, wrong dimension → zero, parse edge cases
## Files Changed
| File | Change |
|------|--------|
| `internal/service/chunk_vector.go` | New — FetchChunkVectors + parse
helpers |
| `internal/service/chunk_vector_test.go` | New — 12 tests |
| `internal/service/metadata_filter_test.go` | Fix missing `"sort"`
import |
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
### What problem does this PR solve?
Refine the stream parsing for `<think>` / `</think>` so MiniMax and
DeepSeek-style chunking both flush in the right order without mixing
think and answer buffers.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
### What problem does this PR solve?
feat：Unify the 'Add Model Provider' modal
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
- [x] Refactoring
### What problem does this PR solve?
As Title
Codes were tested by Postman
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
### What problem does this PR solve?
Fixes the OpenSearch side of #10747: hybrid search drops the keyword
(BM25) leg and
ends up doing plain vector search.
When a search has both a text and a vector leg, `OSConnection.search()`
throws the text
query away:
del q["query"]
q["query"] = {"knn": knn_query}
The text clause only stays on as a filter inside the knn query, so it
narrows the
candidate set but doesn't count towards scoring. So hybrid search on
OpenSearch behaves
like plain vector search, unlike the Elasticsearch backend.
What I changed:
- when both legs are present, send a real hybrid query
`{"hybrid": {"queries": [bm25, {"knn": ...}]}}` and let a
normalization-processor
search pipeline score and combine the two legs
- only the actual filters (kb_id, available_int, ...) go in the knn
filter, not the
text must clause
- create the pipeline on startup if it's missing, so there's no separate
provisioning
step. name and weights can be set under `os:` in service_conf.yaml, or
via
`OS_HYBRID_PIPELINE`; defaults are `ragflow_hybrid_pipeline` and `[0.5,
0.5]`
- normalization-processor needs OpenSearch 2.10+. on older clusters, or
when the
pipeline can't be created, log a warning and fall back to vector-only
instead of
pointing at a pipeline that doesn't exist
This is only the hybrid-search fix; `create_doc_meta_idx` is already on
main.
Testing (there's no OpenSearch path in CI): added a unit test
(`test/unit_test/rag/utils/test_opensearch_hybrid_search.py`, no
services needed) that
checks the query built in each case — hybrid + pipeline param for
text+vector, plain knn
for vector-only, plain bool for text-only, the knn filter never carrying
the text
query_string, and the vector-only fallback when the pipeline isn't
available. Also ran
it against a real OpenSearch 2.19.1 container with a doc that matches
the keyword but
sits outside the knn top-k: pure knn returns `['D1','D2','D5']` (keyword
doc missing),
the hybrid query returns `['A','D1','D2','D5']` (keyword doc present).
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
Signed-off-by: Danut Matei <matei.danut.dm@gmail.com>
## What problem does this PR solve?
Implements `POST /api/v1/searchbots/retrieval_test` in the Go API
server, aligning with the Python `bot_api.py` counterpart. Also applies
security hardening and consistency fixes discovered during CTO-level
code review:
- **Missing endpoint**: `retrieval_test` was not available in Go,
requiring Python fallback
- **Security**: Both `chunkHandler` and `searchBotHandler` leaked
`err.Error()` to API consumers
- **Python alignment**: Default values, empty question handling, and
`top_k <= 0` validation differed from Python behavior
- **Test gaps**: `chunkHandler.RetrievalTest` had zero unit tests;
several edge cases uncovered
## Type of change
- [x] New Feature (non-breaking change which adds functionality)
- [x] Bug Fix (non-breaking change which fixes an issue)
- [x] Refactoring
## Summary
### New Endpoint
- `POST /api/v1/searchbots/retrieval_test` — retrieval test with full
field support (page, size, top_k, use_kg, cross_languages, keyword,
similarity_threshold, vector_similarity_weight)
### New Type
- `common.StringSlice` — JSON type that accepts both `"kb1"` and
`["kb1", "kb2"]`, matching Python API flexibility
### Security
- Both `searchBotHandler` and `chunkHandler` now use `common.Warn()` +
generic error messages instead of leaking `err.Error()` to API consumers
- All error responses include consistent `"data": nil` shape
- `chunkHandler.RetrievalTest` uses interface-based DI (`chunkService`)
to enable testability
### Python Alignment
- Handler-level defaults align with Python `bot_api.py` (page=1,
size=30, top_k=1024, similarity_threshold=0.0,
vector_similarity_weight=0.3)
- `top_k <= 0` validation matching Python behavior
- Empty/whitespace question returns 200 + empty result (matches
`chunk_api.py`)
- `chunkHandler` `Datasets` field uses `common.StringSlice` for
string-or-array flexibility
### Refactoring
- `ChunkServiceIface` → `ChunkRetriever`, `chunkSvcIface` →
`chunkService` (Go-conventional naming)
- Extracted `applyRetrievalDefaults`, `toRetrievalServiceRequest` from
handler body
- Regex moved to package-level var in `parseRelatedQuestions`
- `service.RetrievalTestRequest.Datasets` type changed to
`common.StringSlice`
- `chunkHandler` now uses consumer-side interface for DI
### Tests
- 37 unit tests across both handlers: auth, validation, defaults,
StringSlice edge cases, empty/whitespace KbID, service errors, JSON
format, `top_k <= 0`, field mapping verification
## Files Changed
| File | Change |
|------|--------|
| `cmd/server_main.go` | Wire new handler + chunkService +
difyRetrievalHandler |
| `internal/common/json_types.go` | New StringSlice type |
| `internal/common/json_types_test.go` | StringSlice tests |
| `internal/handler/chunk.go` | Interface-based DI, security, Python
alignment, defaults |
| `internal/handler/chunk_test.go` | New — 9 comprehensive tests |
| `internal/handler/searchbot.go` | New endpoint + refactoring + `top_k
<= 0` validation |
| `internal/handler/searchbot_test.go` | 18 tests covering all edge
cases |
| `internal/router/router.go` | Register new route +
difyRetrievalHandler |
| `internal/service/chunk.go` | Datasets type → StringSlice, Question
binding relaxed |
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
### What problem does this PR solve?
Fix: The embedded website floating component on the agent page does not
display citations.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
### What problem does this PR solve?
Normalize agent session chunk references so they are mapped through a
dedicated helper instead of duplicating the field extraction inline.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
### What problem does this PR solve?
Fix: The time zone is not displayed on the personal profile page.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
### What problem does this PR solve?
Closes#15611.
RAGFlow's fallback reasoning parser only recognized the exact model
family `qwen3`. For provider-prefixed Qwen model names such as
SiliconFlow's `qwen/qwen3-8b`, the derived model class can be
`qwen/qwen3`, so inline `<think>...</think>` content was not split from
the visible answer when `reasoning_content` was absent.
This PR normalizes model-family detection before fallback reasoning
extraction, keeps the parser nil-safe, and adds focused tests for Qwen3
variants plus Gitee and SiliconFlow chat responses.
It also makes SiliconFlow propagate `ChatConfig.Thinking` into the chat
request body, matching the existing Gitee behavior, so Qwen thinking
mode is actually enabled when requested.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
- [x] Refactoring
### Validation
- `/root/go/bin/gofmt -l internal/entity/models/common.go
internal/entity/models/common_test.go
internal/entity/models/reasoning_family_provider_test.go
internal/entity/models/siliconflow.go`
- `git diff --check`
- `/root/go/bin/go test ./internal/entity/models -run
'Test(NormalizeModelFamily|GetThinkingAndAnswer|GiteeChatExtractsQwenThinkingFromInlineContent|SiliconflowChatExtractsProviderPrefixedQwenThinkingFromInlineContent)'
-vet=off -count=1`
Note: the full package command `/root/go/bin/go test
./internal/entity/models -vet=off -count=1` now runs locally, but it
currently fails on an unrelated existing
`TestAstraflowEmbedReturnsNoSuchMethod` panic in
`internal/entity/models/astraflow.go:482`.
### What problem does this PR solve?
Closes#15428
The hybrid score in `rag/nlp/search.py` (`rerank_by_model`) blends
reranker similarity with token similarity on a fixed `[0, 1]` scale:
```python
return tkweight * np.array(tksim) + vtweight * vtsim + rank_fea # tkweight=0.3, vtweight=0.7
```
The reranker implementations did not agree on that scale. Only three of
roughly 17 providers normalized their output, and `NvidiaRerank`
returned raw, unbounded logits. Weighted at `0.7`, a negative logit
could push a genuinely relevant chunk below pure keyword matches, and
its magnitude swamped `tksim`, which lives in `[0, 1]`. The practical
effect was that the same query produced differently scaled scores
depending on the configured reranker, and logit based providers degraded
retrieval quality instead of improving it.
This PR enforces a single scoring contract in one place:
- `Base.similarity` is now the only public entry point. It
short-circuits empty input and guarantees a normalized result. Each
provider implements its raw scoring in `_compute_rank`, which removes
sixteen duplicated empty input guards and the three scattered
normalization calls.
- Normalization is range aware. Providers that already return calibrated
`[0, 1]` relevance scores (Cohere, Jina, Voyage, and others) keep their
absolute magnitudes, so `similarity_threshold` filtering and the
reported `vector_similarity` stay meaningful. Only out-of-range output
such as NVIDIA logits is min-max rescaled into `[0, 1]`.
- The twelve leftover `[DEBUG ...]` prints in `rerank_by_model`,
introduced in #14231, are removed. They ran on every retrieval, added
per chunk overhead, and leaked queries, keywords, and document content
to stdout and logs.
A new regression suite in
`test/unit_test/rag/llm/test_rerank_normalization.py` covers logit
rescaling (positive, negative, and flat batches), preservation of
already calibrated scores, ordering, empty input handling, and the per
provider HTTP path. It also asserts that no provider overrides
`similarity()`, so the contract cannot silently drift.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
## Summary
Port the Python `GET /v1/plugin/tools` endpoint to the Go API server.
Listed in the Go-API port checklist of #15240.
Returns the metadata of every embedded LLM tool plugin in the same JSON
shape the Python endpoint emits (camelCase keys preserved), so existing
frontends bind to the Go server without changes.
### What problem does this PR solve?
Closes#15433
Reranked retrieval drops results and returns short pages once pagination
crosses the first candidate block, for the common page sizes 10 and 30.
In `rag/nlp/search.py`, the candidate window (`RERANK_LIMIT`) is rounded
up to a multiple of `page_size` to keep block based pagination aligned,
and then clamped back to 64:
```python
RERANK_LIMIT = math.ceil(64 / page_size) * page_size if page_size > 1 else 1 # e.g. 70 for page_size=10
RERANK_LIMIT = max(30, RERANK_LIMIT)
if rerank_mdl and top > 0:
RERANK_LIMIT = min(RERANK_LIMIT, top, 64) # clamps back to 64, breaking the multiple
```
`RERANK_LIMIT` is used both as the backend block size (`page =
global_offset // RERANK_LIMIT`) and as the modulus that slices a page
out of a reranked block (`begin = global_offset % RERANK_LIMIT`). When
it stops being a multiple of `page_size`, the block that gets fetched
and the slice taken from it no longer agree. With `page_size=10` and
`top=1024`, page 7 returns only 4 of 10 results and the head of the next
block is never shown on any page. This happens whenever the result set
spans more than one block, which is the default.
**Fix**
The window math is moved into a small reusable helper,
`Dealer._rerank_window`, which:
- targets a pool of about 64 candidates,
- bounds it by `top` when a reranker is active, and
- always rounds to a whole number of pages, so the window stays an exact
multiple of `page_size`.
The call site becomes a single line, and the alignment invariant now
lives in one documented place. Behavior is unchanged on every path that
was already aligned (the non reranked path and any `top` that already
produced a page multiple).
**Verification**
A simulation of the full retrieval path (per block rerank, similarity
threshold filter, and the exact `page // window` and `offset % window`
math) confirms the fix loses nothing where the old code lost real
results:
```
ps=10 top=1024: new window=70 dropped_valid=0 | old window=64 dropped_valid=16
ps=30 top=1024: new window=90 dropped_valid=0 | old window=64 dropped_valid=66
```
New unit tests in `test/unit_test/rag/test_search_pagination.py` cover
the alignment invariant, cross block pagination (every candidate
surfaced once, in order, no gaps, no short interior pages), the reported
regression, and parity with the old window on the previously correct
paths. All 114 cases pass and `ruff check` is clean.
Fixes the reranked deep pagination data loss described above.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
### Description
This PR syncs the `documentServiceIface` interface and introduces
handler methods for document preview, artifact fetching, and downloading
in the Go API. It also ensures that strict dataset alignment and access
checks are enforced when retrieving or downloading documents.
Furthermore, this PR introduces comprehensive unit tests for both the
newly added Handler and Service methods to ensure robustness and prevent
future regressions.
### Key Changes
* **Router & Handler Integration**:
* Added and wired new API endpoints in `internal/router/router.go`.
* Synchronized the `documentServiceIface` with `GetDocumentArtifact`,
`GetDocumentPreview`, and `DownloadDocument`.
* Implemented handlers for these endpoints in
`internal/handler/document.go`.
* **Access & Validation Enforcement**:
* Refactored `internal/service/document.go` to strictly check if a
document belongs to the requested dataset before allowing downloads or
previews.
* Added robust artifact file sanitization (`sanitizeArtifactFilename`)
and attachment handling (`shouldForceArtifactAttachment`).
* **Comprehensive Unit Testing**:
* **Handler Layer (`internal/handler/document_test.go`)**: Added mock
service implementations and Gin router tests covering success,
not-found, and internal error states for all 3 new endpoints.
* **Service Layer (`internal/service/document_test.go`)**: Added
extensive business logic tests including dataset mismatch checks,
non-existent document checks, and artifact file validation.
### What problem does this PR solve?
Feat：
- Get model list from remote provider.
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
Prepend a leading slash and reject `..` segments so scoped OneDrive
delta queries use `root:/path:/delta` instead of `root:path:/delta`.
Fixes#15500
### What problem does this PR solve?
The OneDrive connector builds Microsoft Graph delta URLs from optional
`config.folder_path`. When users enter a path without a leading slash
(e.g. `Documents/Reports` instead of `/Documents/Reports`), the
connector produces a malformed URL such as
`root:Documents/Reports:/delta`. Per [Microsoft Graph path-based
addressing](https://learn.microsoft.com/en-us/graph/onedrive-addressing-driveitems),
the segment after `root:` must start with `/` (e.g.
`root:/Documents/Reports:/delta`). Sync and validation then fail or
return no documents, which is hard to diagnose from the UI because the
optional folder field does not enforce the format.
This PR normalizes `folder_path` at connector construction time (prepend
`/`, trim whitespace and trailing slashes) and rejects `..` segments
before any Graph request is made.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
## Summary
This PR improves the Go CLI in two areas:
1. It adds batch model management support, allowing multiple models to
be added or removed in a single command.
2. It makes the `dimension` argument optional for the `embed text`
command.
These changes keep the existing single-model and explicit-dimension
behaviors compatible while making the CLI more convenient for common
workflows.
## What Changed
### 1. Batch model add/remove support
The CLI now supports operating on multiple model names provided in a
single quoted string.
Supported commands include:
```
add model 'x1 x2 x3' to provider 'vllm' instance 'test' with tokens 1024 chat think vision, token 2048 chat, token 1024 think vision;
drop model 'x1 x2 x3' from 'vllm' 'test';
remove model 'x1 x2 x3' from 'vllm' 'test';
```
For add model, each config segment after with is matched to the
corresponding model name by position.
Example mapping:
- x1 -> tokens 1024, chat + vision, thinking=true
- x2 -> tokens 2048, chat
- x3 -> tokens 1024, vision, thinking=true
The existing single-model syntax remains supported.
### 2. Optional embedding dimension
Previously, the Go CLI required dimension to be explicitly provided for
embed text.
Before:
embed text 'what is rag' 'who are you' with 'model@test@provider'
dimension 8192;
Now both forms are supported:
embed text 'what is rag' 'who are you' with 'model@test@provider'
dimension 8192;
embed text 'what is rag' 'who are you' with 'model@test@provider';
When omitted, the CLI leaves dimension unset and relies on
provider/backend behavior.
## Tests
Added parser tests covering:
- Multiple models with multiple config segments
- Model type deduplication
- Model/config count mismatch
- Drop/remove multiple models
- Optional embedding dimension parsing
### What problem does this PR solve?
Fix: When adding a chat in the main interface, a warning will
automatically pop up (even if embedding and LLM model have already been
configured).
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
## What
- make `Switch` ignore conditions that have no evaluable items
- add a regression for blank `cpn_id` items falling through to the else
branch
- keep the existing non-empty `and` condition behavior covered
Fixes#15643.
## Verified
- `python -m py_compile agent\component\switch.py
test\unit_test\agent\component\test_switch.py`
- `python -m pytest test\unit_test\agent\component\test_switch.py -q` ->
`2 passed`
- `python -m ruff check agent\component\switch.py
test\unit_test\agent\component\test_switch.py`
- `git diff --check`
I also checked `python -m ruff format --check` on the touched files. It
would reformat pre-existing style in `agent/component/switch.py` beyond
this bug fix, so I kept the patch scoped instead of reformatting the
whole file.
### What problem does this PR solve?
Update Dockerfile and release workflow to use GitHub mirror instead of
Gitee
### Type of change
- [x] Other (please describe): CI
### What problem does this PR solve?
Fixes#15542.
AWS Bedrock support for the Go model provider layer was added in #15166,
but embedding support was intentionally left out of scope and
`BedrockModel.Embed(...)` still returned the `no such method` sentinel.
This PR implements Bedrock text embeddings under the umbrella provider
tracker #14736.
### What this PR includes
- `internal/entity/models/bedrock.go`: implement
`BedrockModel.Embed(...)` through Bedrock Runtime `InvokeModel` with
existing SigV4 auth, region resolution, and runtime URL helpers.
- Titan embeddings: supports `amazon.titan-embed-text-v1` and
`amazon.titan-embed-text-v2:0`; v2 forwards `EmbeddingConfig.Dimension`
as `dimensions` when provided, while v1 keeps the payload minimal.
- Cohere embeddings: supports `cohere.embed-english-v3`,
`cohere.embed-multilingual-v3`, and `cohere.embed-v4:0`; batches input
texts and maps returned vectors to RAGFlow `EmbeddingData` in input
order.
- `conf/models/bedrock.json`: adds the `embedding` URL suffix (`invoke`)
and Bedrock embedding model entries.
- `internal/entity/models/bedrock_test.go`: adds unit tests for Titan,
Cohere, typed Cohere responses, validation, empty input, unsupported
models, and HTTP error propagation.
Reference docs:
- Bedrock InvokeModel API:
https://docs.aws.amazon.com/bedrock/latest/APIReference/API_runtime_InvokeModel.html
- Titan Text Embeddings:
https://docs.aws.amazon.com/bedrock/latest/userguide/titan-embedding-models.html
- Cohere Embed models on Bedrock:
https://docs.aws.amazon.com/bedrock/latest/userguide/model-parameters-embed.html
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
### How was this tested?
- [x] `jq empty conf/models/bedrock.json`
- [x] `git diff --check`
- [x] `go test ./internal/entity/models/... -run Bedrock -count=1`
- [x] `go test ./internal/entity/models/... -run '^$' -count=1`
- [x] `go test ./internal/entity/models/... -run Bedrock -race -count=1`
Note: `go test ./internal/entity/models/... -count=1` currently fails in
unrelated existing Astraflow coverage
(`TestAstraflowEmbedReturnsNoSuchMethod` panics in
`internal/entity/models/astraflow.go`). The Bedrock-specific tests and
compile-only package check pass.
## Summary
Ports the MCP (Model Context Protocol) server management endpoints that
power `web/src/pages/user-setting/mcp/` from Python
(`api/apps/restful_apis/mcp_api.py`) to Go. There were no MCP routes in
the Go server before this change.
Closes#15275 (subtask of #15240).
## Endpoints implemented (base path `/api/v1`)
| Method | Path | Description |
|--------|------|-------------|
| GET | `/mcp/servers` | List tenant servers (keyword / order /
pagination) |
| POST | `/mcp/servers` | Create a server |
| GET | `/mcp/servers/{mcp_id}` | Get one (`?mode=download` exports
config) |
| PUT | `/mcp/servers/{mcp_id}` | Update a server |
| DELETE | `/mcp/servers/{mcp_id}` | Delete a server |
| POST | `/mcp/import` | Bulk import from JSON config |
| POST | `/mcp/servers/{mcp_id}/test` | Connect + list tools (see notes)
|
## Implementation
Follows the existing `handler → service → dao` layering (per PR #14790):
- **entity** (`internal/entity/mcp.go`): added `MCPServerType` constants
and `IsValidMCPServerType` over the existing `MCPServer` model.
- **dao** (`internal/dao/mcp.go`): new `MCPServerDAO` with tenant-scoped
CRUD, a keyword filter, and a **whitelisted order-column map** (guards
against SQL injection via the caller-supplied `orderby`).
- **service** (`internal/service/mcp.go`): new `MCPService` —
list/get/export/create/update/delete/import/test — mirroring
`MCPServerService` and the `mcp_api` request validation, with sentinel
errors for clean code mapping.
- **handler** (`internal/handler/mcp.go`): new `MCPHandler` with the
seven handlers and Python-compatible response codes.
- **router / server_main**: registered the `/mcp` group and wired the
handler.
## Deviations from Python (documented in code)
1. **Bulk import is at `POST /mcp/import`, not `/mcp/servers/import`.**
gin (v1.9.1) cannot register a static segment and a path param at the
same tree node, so `/mcp/servers/import` would collide with
`/mcp/servers/:mcp_id` and panic at startup. The frontend should call
`/mcp/import`.
2. **No live tool discovery on create/update/import.** The Python path
runs `get_mcp_tools` over SSE / streamable-HTTP and stores
`variables.tools`. The Go server has no MCP client yet, so these persist
`variables`/`headers` but leave `variables.tools` unpopulated.
3. **`/test` returns a data error (`ErrMCPTestUnsupported`)** until a Go
MCP client lands. Per the issue, the live-connection path is scoped as a
follow-up; the handler still validates `url` + `server_type`.
## Testing
- Added `internal/service/mcp_test.go` covering `IsValidMCPServerType`
and the `TestServer` validation/short-circuit paths (no DB required).
- No Go toolchain was available in the dev environment, so `go build
./...` / `go vet ./...` verification is left to CI.
## Follow-ups
- Go MCP client (SSE / streamable-HTTP) to enable live tool discovery
and the real `/test` behavior.
- Reconcile the `/mcp/import` vs `/mcp/servers/import` path with the
frontend.
---------
### What problem does this PR solve?
Closes#15461.
RAGFlow had no way to ingest Salesforce CRM data, so support / sales
teams couldn't ground responses on live Accounts, Contacts,
Opportunities, Cases, or Knowledge articles. This adds a first-class
Salesforce data source connector that authenticates against a Connected
App via OAuth 2.0 client-credentials, queries selected SObjects via
SOQL, and turns each record into an indexable document with incremental
sync.
**Highlights**
- `common/data_source/salesforce_connector.py`: new
`SalesforceConnector` (`CheckpointedConnectorWithPermSync` +
`SlimConnectorWithPermSync`).
- OAuth 2.0 client-credentials flow; canonical `instance_url` from the
token response so multi-pod orgs route correctly.
- Per-object `SystemModstamp` cursor stored in
`SalesforceCheckpoint.cursors` — a failure mid-object doesn't rewind
sibling objects, and re-syncs only fetch changed rows.
- Deterministic record-to-text formatter (sorted keys) so SOQL field
reordering on the server doesn't mark every row "changed" on each poll.
- `_get_json` raises on non-2xx so 429 / 5xx never silently advance the
checkpoint past missing data.
- `Knowledge__kav` is in the default object set but is skipped silently
when the org doesn't have Salesforce Knowledge enabled (404 on
describe).
- Slim-doc IDs are scoped as `<Object>/<Id>` so prune deletes can't
collide across object types.
- `common/constants.py`, `common/data_source/config.py`,
`common/data_source/__init__.py`: register `salesforce` in `FileSource`
/ `DocumentSource` and export `SalesforceConnector`.
- `rag/svr/sync_data_source.py`: new `Salesforce(SyncBase)` class routed
through `load_from_checkpoint` (poll_source would re-walk every object
each run) and added to `func_factory`.
- Frontend:
- `web/src/pages/user-setting/data-source/constant/index.tsx`: new
`DataSourceKey.SALESFORCE`, form fields (instance URL, client ID/secret,
objects, api_version, batch size), `syncDeletedFiles` capability,
default form values, and tile entry with the new icon.
- `web/src/locales/{en,zh}.ts`: description + per-field tooltips.
- `web/src/assets/svg/data-source/salesforce.svg`: 48x48 brand-style
icon to match the other Microsoft / cloud tiles.
**Verification**
- `npm run build` (vite + esbuild) passes (1m 26s).
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
### What problem does this PR solve?
This fixes duplicated post-think text in streamed chat responses. When
the model emits text immediately after `</think>`, the stream state now
advances its cursor correctly so the same visible prefix is not emitted
twice.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
## Summary
This PR fixes case-sensitivity regressions introduced in #15656 and
consolidates the metadata filtering pipeline by removing the duplicate
`applySingleCondition` adapter layer.
### Bug fixes
1. **contains / not contains**: restored case-insensitive matching (was
lost when `applySingleCondition` was replaced by
`common.MetaFilter.matchValue` which lacked `strings.ToLower`)
2. **not in**: restored case-insensitive matching (was lost for same
reason; uses `strings.EqualFold`)
3. **!= with date filter values**: non-date metadata values now
correctly match the `≠` operator (a non-date value IS not equal to any
date, but was returning false)
### Architecture
4. **Removed `applySingleCondition`** (65 lines) — the inline switch was
a duplicate of `common.MetaFilter` logic. `ApplyMetaFilter` now converts
conditions and delegates to `common.MetaFilter` once per filter set,
eliminating ~25 lines of duplicate AND/OR merge logic.
5. **Added `filterSet`** — O(n+m) hash-map fast path for `in`/`not in`
operators, replacing the O(n*m) linear scan in `matchValue`.
6. **Exported `NormalizeOperator`** from `common` for consistent
operator alias handling.
### Cleanup
7. Removed 18 lines of dead code (`matchValue`'s `in`/`not in` branches
already bypassed by `filterOut` delegation)
8. Fixed orphaned godoc comment for `convertOperator`
9. Fixed incorrect `filterSet` doc comment (claimed "matching EqualFold"
but used `strings.ToLower`)
10. Completed `convertToMetaCondition` operator normalization
documentation
### Testing
- 60 tests (24 service + 36 common), all passing
- New tests: `==`, `≠`, `>`, `<`, `≥`, `≤`, `empty`, `not empty` through
`ApplyMetaFilter`
- New tests: `<`, `≤`, `≠` through `MetaFilter`; `not-in-empty-list`
through `filterSet`
- All 18 `MetaFilter` tests pass; all 10 `filterSet` unit tests pass
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
## Summary
Add `EnrichChunksWithDocMetadata` as a method on `MetadataService` that
attaches document metadata to retrieval chunks in-place. Equivalent to
Python's `enrich_chunks_with_document_metadata()` from
`api/utils/reference_metadata_utils.py`.
### Usage
```go
metadataSvc.EnrichChunksWithDocMetadata(chunks, tenantID, metadataFields)
```
### Changes
- **`service/metadata.go`**: Added `EnrichChunksWithDocMetadata` method
- **`service/enrich_metadata_test.go`** (new): 7 test cases
### Algorithm
1. Collect unique `(kb_id, doc_id)` pairs from chunks
2. Fetch metadata from ES via `SearchMetadata(kbID, tenantID, docIDs)`
3. Attach `document_metadata` field to each matching chunk
4. Optionally filter to specified `metadataFields`
### Testing
All 7 tests pass:
```
=== RUN TestEnrichChunksWithDocMetadata_NoChunks --- PASS
=== RUN TestEnrichChunksWithDocMetadata_EmptyChunks --- PASS
=== RUN TestEnrichChunksWithDocMetadata_EmptyDocID --- PASS
=== RUN TestEnrichChunksWithDocMetadata_DuplicateDocIDs --- PASS
=== RUN TestEnrichChunksWithDocMetadata_MultipleKBs --- PASS
=== RUN TestEnrichChunksWithDocMetadata_WithMetadataFields --- PASS
=== RUN TestEnrichChunksWithDocMetadata_MixedFields --- PASS
```
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
### What problem does this PR solve?
Markdown extraction can split tables row by row when delimiter-based
extraction uses a newline delimiter. That loses table structure during
chunking even though delimiters should still split normally outside
tables.
This PR keeps the follow-up to #15482 intentionally narrow:
- preserve Markdown pipe tables during delimiter-based extraction
- preserve borderless pipe tables during delimiter-based extraction
- preserve multiline HTML tables during delimiter-based extraction
- keep delimiter splitting unchanged outside protected table ranges
Refs #15482
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
### Testing
- `ruff check deepdoc/parser/markdown_parser.py
test/unit_test/deepdoc/parser/test_markdown_parser.py`
- `python3 run_tests.py -t
test/unit_test/deepdoc/parser/test_markdown_parser.py`
- `git diff --check`
## Summary
Port the canvas-template catalogue endpoint to the Go API server. Listed
in the Go-API port checklist of #15240.
Mirrors `list_agent_template` in `api/apps/restful_apis/agent_api.py`:
returns every row from the `canvas_template` table so that the UI can
render the template gallery on the New-Agent screen.
## What
- `internal/dao/canvas_template.go` — new `CanvasTemplateDAO.GetAll()`
ordered by `create_time desc` (newest templates first).
- `internal/service/agent.go` — wire the new DAO into `AgentService` and
expose `ListTemplates() ([]*entity.CanvasTemplate, error)`.
- `internal/handler/agent.go` — new `AgentHandler.ListTemplates` HTTP
handler (auth-gated, mirrors Python `@login_required`).
- `internal/router/router.go` — `agents.GET("/templates",
r.agentHandler.ListTemplates)` registered alongside the existing `GET
/agents`.
- `internal/handler/agent_test.go` — three new tests covering: success
path, empty-list → JSON array (not `null`), and the auth gate.
## Notes
- `CanvasTemplate` entity, GORM tags, and DB migration already exist in
`internal/entity/canvas.go` and `internal/dao/database.go` — no schema
change required.
- The handler coerces a `nil` slice to `[]*entity.CanvasTemplate{}` so
the JSON payload is always an array (the frontend does `data.map(...)`
on it).
## Test plan
- [x] `go vet ./internal/handler ./internal/service ./internal/dao
./internal/router` clean
- [x] Three unit tests added; existing `TestListAgents_Success`
untouched
- [ ] CI runs `go test ./internal/handler` with cgo binding linked
## Related
- Tracker: #15240
QueryRewrite prompt builder and response parser. Zero external
dependencies.
### Functions
- `BuildQueryRewritePrompt`: Renders `minirag_query2kwd` prompt with
query and type pool
- `ParseQueryRewriteResponse`: Parses LLM JSON response with fallback
for markdown and extra text
### Testing
```
=== RUN TestBuildQueryRewritePrompt --- PASS
=== RUN TestParseQueryRewriteResponse_ValidJSON --- PASS
=== RUN TestParseQueryRewriteResponse_MarkdownBlock --- PASS
=== RUN TestParseQueryRewriteResponse_ExtraText --- PASS
=== RUN TestParseQueryRewriteResponse_Invalid --- PASS
=== RUN TestParseQueryRewriteResponse_EmptyEntities --- PASS
```
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
## Summary
- Infer `Content-Type` from the stored document filename on SDK download
routes.
- Covers `GET /api/v1/datasets/<dataset_id>/documents/<document_id>` and
`GET /api/v1/documents/<document_id>`.
- Aligns with REST preview/download via `CONTENT_TYPE_MAP`.
## Test plan
- [x] `pytest
test/testcases/test_http_api/test_file_management_within_dataset/test_doc_sdk_routes_unit.py::TestDocRoutesUnit::test_download_mimetype_from_filename`
- [x] Manual: `curl -sSI` on SDK dataset document download for a PDF;
expect `Content-Type: application/pdf`
Fixes#15112.
### What problem does this PR solve?
Fix:
- VolcEngine adapt to new api_key format
- Save dict api_key as json
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
### What problem does this PR solve?
On some Linux hosts (e.g. x86_64 with enforced POSIX ACL on overlay
storage), the official `elasticsearch` Docker image cannot start because
`docker-entrypoint.sh` needs to create temporary files under `/tmp` for
bash here-documents, while the image ACL grants `user:elasticsearch`
only `r-x` on `/tmp`:
```
/usr/local/bin/docker-entrypoint.sh: line 73/84: cannot create temp file for here-document: Permission denied
```
RAGFlow users hit this when running `docker compose` with the default
`es01` service. See also Refs #284.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
## Summary
Mount a writable `tmpfs` at `/tmp` for the `es01` service so
Elasticsearch entrypoint scripts can run on ACL-enforced environments.
Closes the startup failure described in #284 for non-ARM deployments.
## Changes
- Add `tmpfs: /tmp:mode=1777,size=512m` to `es01` in
`docker/docker-compose-base.yml`
- Document why the mount is required (ES image `/tmp` ACL vs entrypoint
here-documents)
## Test plan
- [x] Verified on Linux (x86_64): `docker run --rm elasticsearch:8.11.3
bash -c 'mktemp'` fails without tmpfs and succeeds with `--tmpfs
/tmp:mode=1777,size=512m`
- [x] Verified `es01` becomes healthy after `docker compose up -d es01`
with this change
- [ ] Upstream maintainers: `docker compose -f
docker/docker-compose-base.yml --profile elasticsearch up -d es01` on a
host where ACL is enforced
Made with [Cursor](https://cursor.com)
Co-authored-by: Cursor <cursoragent@cursor.com>
Add `ResolveReferenceMetadata` to parse `include_metadata` /
`metadata_fields` from request and config payloads.
### Changes
- **New**: `internal/common/reference_metadata.go` — pure function, zero
dependencies
- **New**: `internal/common/reference_metadata_test.go` — 8 test cases
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
## Summary
Change `GetFlattedMetaByKBs` return type from `map[string]interface{}`
to strongly-typed `common.MetaData`.
**Depends on**: #15648 (provides `MetaData`, `MetaValueDocs` types)
### Changes
- `service/metadata.go`: Changed return type, removed type assertions
- `service/metadata_filter.go`: Updated all metadata function signatures
- `service/metadata_filter_test.go` (new): 12 test cases
### Bug fix
`applySingleCondition` used `.([]interface{})` assertions on `[]string`
data, silently breaking operators like `!=`, `contains`, `start with`,
etc.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
### What problem does this PR solve?
Closes#15465.
RAGFlow supports S3, Google Cloud Storage, R2, and OCI as data sources
but not Azure Blob Storage, leaving Azure users without a way to index
container objects into a knowledge base. This adds a first-class Azure
Blob Storage data-source connector — distinct from RAGFlow's existing
Azure storage *backends* (`rag/utils/azure_sas_conn.py`,
`rag/utils/azure_spn_conn.py`) which store RAGFlow's own files.
**Highlights**
- `common/data_source/azure_blob_connector.py`: new `AzureBlobConnector`
(`CheckpointedConnectorWithPermSync` + `SlimConnectorWithPermSync`).
- Uses the existing `azure-storage-blob` dependency (already in
`pyproject.toml`).
- Three auth modes, tried in order of precedence:
1. **Account key** — `account_name` + `account_key` + `container_name`.
2. **Connection string** — `connection_string` + `container_name`.
3. **SAS token** — `container_url` + `sas_token` (same shape as
`RAGFlowAzureSasBlob`).
- ETag fingerprint stored per blob in `AzureBlobCheckpoint.etags` —
unchanged blobs (same ETag as last run) are skipped without a download.
Only new/modified blobs are fetched.
- Optional `prefix` scopes indexing to a virtual folder.
- `validate_connector_settings()` probes `get_container_properties()`
and maps `AuthenticationFailed / 403 / ContainerNotFound` to typed
connector exceptions.
- Slim-doc IDs are blob names so prune reconciles correctly.
- `common/constants.py`, `common/data_source/config.py`,
`common/data_source/__init__.py`: register `azure_blob` in `FileSource`
/ `DocumentSource` and export `AzureBlobConnector`.
- `rag/svr/sync_data_source.py`: new `AzureBlob(SyncBase)` class routed
through `load_from_checkpoint` (ETag fingerprint owns change-detection)
and added to `func_factory`.
- Frontend:
- `web/src/pages/user-setting/data-source/constant/index.tsx`: new
`DataSourceKey.AZURE_BLOB`, auth-mode selector (account key / connection
string / SAS token), all credential fields, prefix + batch-size,
`syncDeletedFiles` capability, default form values, tile entry with
icon.
- `web/src/locales/{en,zh}.ts`: description + per-field tooltips for all
9 new keys.
- `web/src/assets/svg/data-source/azure-blob.svg`: Azure-branded
stacked-cylinders icon.
**Verification**
- `npm run build` (vite + esbuild) passes (37 s).
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
## Summary
`DocumentDAO.GetByIDs()` generated `WHERE id IN ()` for empty/nil ID
slices, which is invalid SQL and would fail on most databases. This PR
adds a nil guard and comprehensive tests.
### Changes
- **Modified**: `internal/dao/document.go` — Added `len(ids) == 0` guard
to `GetByIDs`
- **New**: `internal/dao/document_test.go` — 4 test cases covering
success, empty IDs, nil IDs, and no-match
### Testing
```
=== RUN TestDocumentGetByIDs_Success --- PASS
=== RUN TestDocumentGetByIDs_EmptyIDs --- PASS
=== RUN TestDocumentGetByIDs_NilIDs --- PASS
=== RUN TestDocumentGetByIDs_NoMatch --- PASS
```
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
## Summary
Migrate the metadata filtering utilities `meta_filter` and
`convert_conditions` from `common/metadata_utils.py` to Go as pure
functions with zero external dependencies.
These functions are used by `dify/retrieval`, `openai/chat/completions`,
`document_api`, and `chunk_api` for filtering documents by metadata
conditions.
### Changes
- **New**: `internal/common/metadata_utils.go` — `ConvertConditions()`
and `MetaFilter()` with full operator support
- **New**: `internal/common/metadata_utils_test.go` — 18 test cases
covering all operators and edge cases
### Supported Operators
`=`, `≠`, `>`, `<`, `≥`, `≤`, `contains`, `not contains`, `in`, `not
in`, `start with`, `end with`, `empty`, `not empty`
### Design
- Numeric comparison via `strconv.ParseFloat`
- Date comparison via YYYY-MM-DD format detection
- Case-insensitive string comparison fallback
- `and` / `or` logic support for multiple conditions
- Zero external dependencies — pure functions only
## Summary
Implement the `GET /api/v1/agents/<agent_id>/versions/<version_id>`
endpoint in Go, returning full version details including DSL.
Depends on #15629 which introduced the version list endpoint and
`UserCanvasVersionDAO` infrastructure.
### Changes
- **Modified**: `internal/handler/agent.go` — Added `GetAgentVersion`
handler with auth check and ownership verification
- **Modified**: `internal/router/router.go` — Registered `GET
/:agent_id/versions/:version_id` route
- **New/Modified tests**: Service and handler tests for the version
detail endpoint
### Testing
```
=== RUN TestGetVersion_Success --- PASS
=== RUN TestGetVersion_WrongCanvas --- PASS
=== RUN TestGetVersion_NotFound --- PASS
=== RUN TestGetAgentVersionHandler_Success --- PASS
=== RUN TestGetAgentVersionHandler_VersionNotFound --- PASS
```
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
### What problem does this PR solve?
When a document is rerun or updated concurrently, the previous
unconditional update could overwrite a newer task state.
This change adds an `update_time`-based optimistic lock so the update
only succeeds if the record has not been modified by another flow in the
meantime.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
### What problem does this PR solve?
This change ensures `/searchbots/ask` receives `search_id` from the
frontend, so the backend can load the matching search configuration when
the shared search flow invokes the endpoint.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
### What problem does this PR solve?
Not display `success` when check not passed.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
## Summary
- keep the native Docling chunking path when it returns usable chunks
- fall back to the standard Docling response parser when a chunked
request gets HTTP 200 but returns no usable chunks
- add a regression test for older Docling servers that accept the
chunking request but return a standard conversion payload
## Why
Older external Docling servers can accept a request containing
`do_chunking: true` and still return the standard conversion response
shape. The current code treats any HTTP 200 from the chunked request as
a native chunk response, finds no chunk entries, and returns zero
sections without trying the standard response parser.
Fixes#15569.
## Validation
- `python -m pytest
test\\unit_test\\deepdoc\\parser\\test_docling_parser_remote.py -q`
- `python -m py_compile deepdoc\\parser\\docling_parser.py
test\\unit_test\\deepdoc\\parser\\test_docling_parser_remote.py`
- `python -m ruff check deepdoc\\parser\\docling_parser.py
test\\unit_test\\deepdoc\\parser\\test_docling_parser_remote.py`
- `git diff --check`
### What problem does this PR solve?
Markdown extraction currently applies custom delimiters before
respecting fenced code blocks. When a delimiter such as a newline is
configured, fenced code can be split into separate chunks, and longer
outer fences can be closed incorrectly by shorter nested fences.
This PR keeps the fix intentionally narrow for the Markdown chunking
discussion in #15482:
- preserve fenced code blocks when delimiter-based extraction is used
- support both backtick and tilde fences
- respect fence length so longer outer fences can contain shorter inner
fences
- keep delimiter splitting unchanged outside fenced blocks
Refs #15482
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
### Testing
- `ruff check deepdoc/parser/markdown_parser.py
test/unit_test/deepdoc/parser/test_markdown_parser.py`
- `python3 run_tests.py -t
test/unit_test/deepdoc/parser/test_markdown_parser.py`
### What problem does this PR solve?
remove duplicate document preview access check
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
### What problem does this PR solve?
Fix:
- Verify provider with empty llm list in llm_factories.json
- Set search bot's chat_llm_name, use tenant default chat model as
default
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
Restore `DocumentService.accessible` on `GET
/api/v1/documents/{doc_id}/preview` so cross-tenant users cannot stream
documents by UUID.
Fixes#15501
### What problem does this PR solve?
PR #15146 (`71a52d579`) moved the agent attachment download route and
accidentally removed the `DocumentService.accessible(doc_id,
current_user.id)` guard from the REST preview handler. The endpoint
still requires login, but any authenticated user who knows another
tenant's `doc_id` can download the raw file bytes.
This restores the same authorization check that existed before #15146,
returning a generic `"Document not found!"` when access is denied (no
cross-tenant ID enumeration). SDK download routes tracked in #15125 are
unchanged.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
## Summary
Restore the `DocumentService.accessible(doc_id, current_user.id)` check
that PR #15146 dropped from the REST document preview handler. Any
authenticated caller could download any tenant's document bytes by
guessing/knowing the `doc_id`.
## Root cause
`api/apps/restful_apis/document_api.py` — the `GET
/documents/<doc_id>/preview` handler called `DocumentService.get_by_id`
and went straight to `File2DocumentService.get_storage_address` +
`STORAGE_IMPL.get`, with no tenant check between the lookup and the
read. The handler's docstring even promises "user must belong to the
tenant that owns the document's knowledge base" — the code didn't
enforce it.
## Fix
- Add `current_user` to the existing `api.apps` import.
- Immediately after `get_by_id`, call
`DocumentService.accessible(doc_id, current_user.id)`; on denial, return
the **same** `get_data_error_result(message="Document not found!")`
shape used for the missing-doc branch. That makes a cross-tenant probe
indistinguishable from a missing-doc probe, preventing ID enumeration
(the issue body calls this out explicitly).
- Emit `logging.warning` with caller user + doc_id for audit.
- Restores symmetry with peer routes that already call
`accessible(doc_id, user_id)` (e.g. `_run_sync` at
`document_api.py:1380`).
## Test plan
Adds
`test/unit_test/api/apps/restful_apis/test_document_preview_accessible.py`:
- **`test_cross_tenant_preview_is_denied`** — owner tenant ≠ caller
tenant; asserts the response shape is `Document not found!` and the
storage backend (`thread_pool_exec(STORAGE_IMPL.get, ...)`) is **never**
invoked.
- **`test_missing_doc_returns_not_found`** — missing-doc behaviour
unchanged.
Stub-loader pattern mirrors
`test/unit_test/api/apps/sdk/test_dify_retrieval.py` (added in #15028,
passing in CI).
## Provenance — how this fix was produced
This PR was authored against a small cited knowledge base committed in
the working tree as a `.vouch/` (see
[vouchdev/vouch](https://github.com/vouchdev/vouch)). The loop used
here:
1. **Grounding first.** Before reading the handler, queried the KB for
prior context: `vouch context "tenant scoped accessible authorization"`
→ retrieved a cited claim distilled from PR #15028 (which restored the
same `accessible()` check on `/dify/retrieval`). The retrieved rule:
> *ragflow REST endpoints that load by tenant-scoped id must call
`<Service>.accessible(id, tenant_id)` after `get_by_id` and before
storage/DB read; deny with code 109 'No authorization.' and log a
warning. Established by PR #15028.*
2. **Applied the pattern with a domain refinement.** For an API/JSON
endpoint, `No authorization.` is the right denial shape. For a
**byte-streaming, browser-facing** endpoint like `/preview`, leaking
*existence* itself enables enumeration — so per the issue's expected
behaviour, this PR denies with `Document not found!` (indistinguishable
from missing) instead. Same auth check, narrower response.
3. **Recorded the refinement back into the KB** as a new cited claim, so
the next IDOR-class issue starts already grounded in both the general
pattern and the byte-route nuance.
Net effect of the workflow: the fix replicates a known-good pattern
instead of reinventing it, *and* the place where the pattern was nuanced
is now retrievable for the next pass. Mechanism is fully independent of
this PR — it's not a runtime dependency, just process discipline.
Closes#15501
### What problem does this PR solve?
### Problem
On the Model Providers page, the Embedding Model dropdown in System
Model Settings shows empty (no default selected), even though a default
embedding model is configured in `service_conf.yaml`.
### Root Cause
Two issues were identified:
1. **Backend: `_get_model_info` fails for unregistered providers**
The tenant's `embd_id` is set to `bge-m3@xxxx` during initialization
(from the placeholder config `factory: 'xxxx'`). The `_get_model_info`
function requires the provider to exist in `tenant_model_provider`
table, but `xxxx` is never a real provider. Even after the user adds a
real provider (e.g., ZHIPU-AI), the stale `embd_id` still references the
non-existent one, causing the function to return `None`.
2. **Frontend: default models cache not invalidated after adding
provider**
`useAddProviderInstance` only invalidates `addedProviders` and
`allModels` caches after adding a provider instance, but does **not**
invalidate the `defaultModels` cache. This means the default model list
is not re-fetched until the user manually refreshes the page.
### Fix
**`api/apps/services/models_api_service.py`**
- Added `_resolve_model_from_tenant_providers()` helper: when the
default model's provider doesn't exist (e.g., placeholder `xxxx`), it
searches through the tenant's actually registered providers for a model
of the same type and returns the first match.
- When an instance name doesn't match (e.g., `"default"` vs actual name
`"1"`), the function now auto-resolves to the first real instance under
that provider.
- Falls back to `FACTORY_LLM_INFOS` validation when neither provider nor
instance exists.
**`web/src/hooks/use-llm-request.tsx`**
- Added `queryClient.invalidateQueries({ queryKey:
LlmKeys.defaultModels() })` to `useAddProviderInstance` so that the
default model list is re-fetched immediately after a provider instance
is added, eliminating the need for a manual page refresh.
### Testing
- Verified with a tenant whose `embd_id=bge-m3@xxxx` and only provider
is ZHIPU-AI (instance `1`): `_resolve_model_from_tenant_providers`
correctly resolves to `embedding-2@1@ZHIPU-AI`.
- After adding a provider via the UI, the embedding model dropdown now
immediately shows the resolved default without requiring a page refresh.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
---------
Signed-off-by: noob <yixiao121314@outlook.com>
### What problem does this PR solve?
The following API is available in go
> /api/v1/connectors/google/oauth/web/start POST
> /api/v1/connectors/gmail/oauth/web/callback GET
> /api/v1/connectors/google-drive/oauth/web/callback GET
> /api/v1/connectors/google/oauth/web/result POST
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
### What problem does this PR solve?
Adds a shared safe default implementation for unsupported Go
model-driver capability methods and migrates the confirmed panic-stub
providers to use it.
The Go `ModelDriver` interface requires providers to implement many
capability methods even when the provider does not support them. XunFei
had unsupported capability methods implemented as `panic("implement
me")`, Mistral still had a panic in `ParseFile`, and HuaweiCloud carried
an unreachable `panic("implement me")` after a normal chat return.
### Type of change
- [x] Refactoring
Co-authored-by: Haruko386 <tryeverypossible@163.com>
## Summary
Fixes#15534 — `update_document_name_only()` crashes with
`AttributeError` when `File2Document` exists but the linked `File` row
was deleted.
`update_document_name_only()` in `document_api_service.py` called
`FileService.get_by_id()` when a `File2Document` row existed, then
accessed `file.id` without checking the lookup result. An orphan
`File2Document` link (file deleted, mapping left behind) caused document
rename via `PATCH /api/v1/datasets/{dataset_id}/documents/{document_id}`
to return HTTP 500.
This PR mirrors guards used in `file2document_api.py` and
`file_api_service.py`: skip the optional file rename when the file is
missing, and still update the document record and search index.
## Changes
- `api/apps/services/document_api_service.py` — check `e and file`
before `FileService.update_by_id`
- `test/unit_test/api/apps/services/test_update_document_name_only.py` —
regression tests (orphan link + happy path)
## Test plan
- [x] `pytest
test/unit_test/api/apps/services/test_update_document_name_only.py -v`
- [ ] Manual: PATCH document `name` when `File2Document` points to a
non-existent `file_id` → 200, document/index renamed, no 500
### What problem does this PR solve?
1. Add license announcement
2. Add sanity check on API config
3. Add base class: BaseModel
4. Add GetBaseURL
### Type of change
- [x] Refactoring
---------
Signed-off-by: Jin Hai <haijin.chn@gmail.com>
### What problem does this PR solve?
Fix: Switching pagesize on a chunk page did not reset the current page.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
### What problem does this PR solve?
remove old and add latest cohere models
### Type of change
- [x] Refactoring
- [x] Other (please describe): update models
### What problem does this PR solve?
Fix: Model provider add verify and fixed form in modal not resetting
issue
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
### What problem does this PR solve?
Closes#15388.
Chat completion routes did not reliably honor per-request generation
settings:
- `/api/v1/chat/completions` copied generation settings with a
truthiness check, so valid zero values such as `temperature: 0`, `top_p:
0`, `frequency_penalty: 0`, `presence_penalty: 0`, and `max_tokens: 0`
were dropped.
- `/api/v1/openai/{chat_id}/chat/completions` did not forward standard
generation settings into the request-specific dialog LLM settings before
calling `async_chat`.
This PR preserves explicitly supplied generation parameters, including
zero values, and merges request-level overrides into existing dialog
settings where appropriate.
The supported generation parameter keys and merge behavior live in a
shared REST API helper to keep both completion routes aligned.
Validation:
- `git diff --check`
- `python3 -m py_compile api/apps/restful_apis/_generation_params.py
api/apps/restful_apis/chat_api.py api/apps/restful_apis/openai_api.py
test/testcases/test_http_api/test_session_management/test_session_sdk_routes_unit.py`
- `uv run ruff check api/apps/restful_apis/_generation_params.py
api/apps/restful_apis/chat_api.py api/apps/restful_apis/openai_api.py
test/testcases/test_http_api/test_session_management/test_session_sdk_routes_unit.py`
- `ZHIPU_AI_API_KEY=dummy uv run pytest
test/testcases/test_http_api/test_session_management/test_session_sdk_routes_unit.py
-q -k generation_params`
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
## Problem
When uploading `.md` files with `parser=naive` and `delimiter="\n"`,
markdown headers (e.g., `## Quick Travel`) become separate chunks with
very short content (16-18 characters). This causes retrieval issues:
when the header is matched, the corresponding body text is not included
in the chunk.
## Related Issues
Closes#15487
## Checklist
- [x] Code changes are minimal and focused
- [x] Unit tests added (12/12 passed)
- [x] No breaking changes
### What problem does this PR solve?
Fix:
- Handle siliconflow and siliconflow_intl api_key
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
add the newanthropic and voyage models. Strip opus 4.7 and 4.8 of
certain usnspported keys
Co-authored-by: Idriss Sbaaoui <112825897+6ba3i@users.noreply.github.com>
## What
#15240
implementation for PUT /api/v1/mcp/servers/:mcp_id
## Changes
- Adds the Go implementation for `PUT /api/v1/mcp/servers/:mcp_id`.
- Wires MCP service and handler into the Go server/router for the update
route.
- Preserves Python-style behavior for ownership checks, partial update
fields, MCP type/name/URL validation, `headers`/`variables`
normalization, and tool metadata scrubbing.
### What problem does this PR solve?
Closes#15379
Around 29 Go model providers in `internal/entity/models/` share an
`http.Client` configured with `Timeout: 120 * time.Second`, and reuse
that same client for `ChatStreamlyWithSender`. Go's
`http.Client.Timeout` is a hard ceiling on the whole request that also
covers reading the response body, so it behaves as a wall clock on
streaming. Any streamed chat response that lasts longer than 120 seconds
gets cut off in the middle with a timeout error. Long generations,
reasoning model outputs, and slow or overloaded upstreams are the common
victims.
The providers that already behave correctly (`groq`, `mistral`,
`voyage`, `anthropic`) set no client `Timeout` and instead wrap each
request in a `context.WithTimeout`. This change converges the affected
providers onto that same pattern.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
---------
Co-authored-by: Jin Hai <haijin.chn@gmail.com>
### What problem does this PR solve?
mark mysql migrations as applied
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
## Summary
- Harden `NewN1NModel` to avoid panics when `http.DefaultTransport` is a
custom non-`*http.Transport` RoundTripper.
- Fallback to a safe transport (`ProxyFromEnvironment`) while preserving
existing pooling/timeout settings.
- Add `n1n_test.go` with coverage for name/factory plus
`TestN1NNewModelWithCustomDefaultTransport`.
Co-authored-by: Cursor <cursoragent@cursor.com>
### What problem does this PR solve?
This PR aligns `POST /api/v1/system/tokens` in Go with the Python
implementation.
### Type of change
- Keep the token creation flow under the system API route.
- Preserve the owner-tenant authorization check.
- Generate and persist API tokens consistently with the current Go
service flow.
- Return the created token payload in the standard API response format.
Co-authored-by: Jin Hai <haijin.chn@gmail.com>
### What problem does this PR solve?
Fix:
- Use @ to avoid split by `_` in model_name.
- Verify api_key when add instance.
- Pop api_key in list intances response.
- Remove useless index.
- Sort providers, instances and models by name.
- Get `is_tools` from llm_factories.json
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
### What problem does this PR solve?
implement /api/v1/datasets/<dataset_id>/metadata/config
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
### What problem does this PR solve?
This PR fixes the issue where Qwen3.5/Qwen3.6 series models may spend
excessive time on simple document-parsing tasks, such as Auto Metadata
extraction, keyword extraction, question generation, and image
description when using the MinerU parser.
For these tasks, Qwen3.5/Qwen3.6 models may perform unnecessary
reasoning by default, which can lead to very long response times, high
token consumption, and, in some cases, potential infinite output loops.
Since Qwen3.5/Qwen3.6 multimodal models are instantiated as `CvModel`
when configured as `image2text`, the existing `enable_thinking=False`
logic in `chat_model.py` does not apply to them. This PR adds the
corresponding handling for the CV/image-to-text model path as well.
This helps reduce unnecessary thinking time, avoid potential infinite
loops, and improve parsing efficiency without noticeably affecting
output quality for these simple extraction and image-description tasks.
Fixes#15083.
### What problem does this PR solve?
Fixes#15286.
When calling `/api/v1/openai/<chat_id>/chat/completions` with `"stream":
true`, the response contains the answer **twice** — the final message
repeats everything that was already streamed.
#### Root cause
RAGFlow's `async_chat` streams the body as incremental `delta.content`
chunks, then emits a terminating `final` event whose `answer` is the
**complete** (decorated) message. The handler re-emitted that full
answer as one more `delta.content` chunk:
```python
if ans.get("final"):
if ans.get("answer"):
full_content = ans["answer"]
response["choices"][0]["delta"]["content"] = full_content # <-- whole answer again
yield ...
```
So a client accumulating `delta.content` ends up with the message
duplicated.
#### Fix
Drop the re-emission. The complete answer from the `final` event is now
surfaced **only** through the trailing chunk's `final_content` and
`reference` fields, which matches OpenAI streaming semantics: deltas are
incremental, and the final chunk carries only `finish_reason` / `usage`
(plus RAGFlow's `reference` / `final_content` extensions).
This matches the expected behavior described in the issue: "The stream
should only yield content chunks once, and the final message should only
contain reference, usage, and finish_reason."
#### Testability refactor
The streaming SSE assembly was a closure inside the request handler, so
it could only be exercised against a live server + real LLM. I extracted
it into a module-level `_stream_chat_completion_sse` async generator
(behavior-preserving) so it can be unit-tested with a fake event stream.
#### Tests
Adds
`test/unit_test/api/apps/restful_apis/test_openai_stream_no_duplicate.py`
(same import-stub pattern as the existing `test_get_agent_session.py`):
- body is streamed exactly once (the regression);
- the complete answer is never re-emitted as a content chunk;
- the terminating chunk has `finish_reason="stop"`, `content=None`, and
correct `usage`;
- `final_content` / `reference` are present on the trailing chunk;
- reasoning (`think`) deltas stream separately and are not duplicated.
> Note: this is unrelated to #15442, which only changes the `stream`
default — it does not touch the duplication logic.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
- [x] Added test cases
---------
Co-authored-by: Wang Qi <wangq8@outlook.com>
### What problem does this PR solve?
Feature: #14961
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
- [x] Refactoring
### What problem does this PR solve?
Fixes#15115.
`GET /api/v1/documents/images/<image_id>` returned **Image not found**
when the thumbnail storage object key contained hyphens (e.g.
`page-1.png`). Document APIs build URLs as `{dataset_id}-{thumbnail}`,
but `get_document_image()` used `image_id.split("-")` and required
exactly two segments, so keys like `<kb_id>-page-1.png` were rejected
even though the blob existed.
This PR splits only on the first hyphen (`split("-", 1)`) and sets
`Content-Type` from the object key extension via `CONTENT_TYPE_MAP`
instead of hardcoding `image/JPEG`.
### What problem does this PR solve?
Fixes#15117.
Chunk images are stored with `img_id = f"{bucket}-{objname}"` in
`image2id()` (`rag/utils/base64_image.py`). When loading via
`id2image()`, the code used `image_id.split("-")` and required exactly
two segments. Object keys that contain hyphens (e.g. `page-1.jpg`)
produce more than two segments, so `id2image` returns `None` and chunk
image previews fail even though the blob exists.
This is the same parsing issue as #15115 (HTTP thumbnail route); this PR
fixes the indexing/retrieval path.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
- [ ] New Feature (non-breaking change which adds functionality)
- [ ] Documentation Update
- [ ] Refactoring
- [ ] Performance Improvement
- [ ] Other (please describe):
### Test plan
- [x] `pytest test/unit_test/rag/utils/test_base64_image.py`
- [ ] Manual: index a chunk with an `objname` containing hyphens and
confirm `img_id` resolves to an image in retrieval
Fixes#15117.
### Related issues
Closes#15312
### What problem does this PR solve?
`tools/scripts/mysql_migration.py` built batch INSERT SQL for the
`tenant_model_provider` stage using f-strings with raw `llm_factory` and
`tenant_id` values. If either value contained a single quote, migration
SQL could fail; this also created unnecessary SQL-injection risk in the
migration path.
This PR replaces string interpolation with parameterized SQL
placeholders in `TenantModelProviderStage.execute()`. The migration now
safely handles quoted values and executes deterministically across
existing tenant data.
### Related issues
Closes#15358
<!-- After filing upstream, replace XXXX with your issue number. -->
---
### What problem does this PR solve?
`POST /api/v1/openai/<chat_id>/chat/completions` forwards `messages` to
`async_chat` without normalizing `content`. Downstream, `dialog_service`
assumes string content:
```python
re.sub(r"##\d+\$\$", "", m["content"])
```
OpenAI-compatible clients may send `content` as an **array** of parts
(text, `image_url`, etc.), including text-only arrays. That causes
`TypeError` and HTTP **500** instead of a valid response or a clear
**400**.
`openai_api.py` also reads `messages[-1]["content"]` directly for
`prompt` without handling list-shaped content.
This PR normalizes array `content` to a string (concatenating `type:
text` parts) before calling `async_chat`, matching a minimal
OpenAI-compat path. Image parts can be documented as unsupported or
handled in a follow-up if vision integration is required.
### What problem does this PR solve?
Fix auto metadata type issue
https://github.com/infiniflow/ragflow/issues/15323
Type information is missing at frontend - backend correctly store the
type information for the auto metadata type.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
## Summary
Fixes#15245 — `POST /api/v1/chat/completions` with `stream=true`
intermittently returns 500:
```
data:{"code": 500, "message": "failed to encode response: json:
unsupported value: NaN (status code: 500)", "data": {...}}
```
…even though "the same question" works on retry.
## Root cause
The streaming path serialized the answer with bare `json.dumps(...)`
(`api/apps/restful_apis/chat_api.py:1221`). `json.dumps` defaults to
`allow_nan=True` and emits the literal token `NaN` for NaN /
Infinity float values. That is valid Python-flavored JSON but
**invalid per RFC 8259**, so downstream consumers reject it. The
reporter's gateway is Go-based and the error wording
(`failed to encode response: json: unsupported value: NaN`) is
straight from Go's `encoding/json`.
How NaN gets into the payload: retrieval scoring in
`rag/nlp/search.py` runs `np.mean(...)` over aggregations that can
be empty, and similarity denominators can be zero. Reference chunk
fields like `similarity`, `vector_similarity`, `term_similarity`
can therefore be NaN depending on which chunks a given query
retrieves — which is exactly why the failure is intermittent for
the same question.
The non-streaming branch (`get_json_result(data=answer)`,
`chat_api.py:1243`) has the same vulnerability — Quart's `jsonify`
also defaults to `allow_nan=True` and the same retrieval pipeline
feeds both branches.
`agent/tools/exesql.py:88-102` already has the same NaN/Inf guard
for SQL results. This PR brings the chat completions path up to
parity.
## Fix
Add a small `_sanitize_json_floats(obj)` helper near the top of
`api/apps/restful_apis/chat_api.py`. It walks `dict` / `list` /
`tuple` and replaces any `float` that is `NaN` or `±Infinity` with
`None`. Apply it at the two serialization boundaries:
- **Streaming branch** (`stream()`): sanitize the SSE payload before
`json.dumps`.
- **Non-streaming branch**: sanitize the `answer` dict before
`get_json_result(data=...)`.
The terminal `data:True` frame and the `code:500` error frame carry
no scores and are left untouched.
Added `import math` to the existing alphabetical import block.
No change to retrieval logic — replacing NaN with `null` at the
serialization boundary is conservative: clients still parse the
JSON, a missing-score chunk is a strictly better failure mode than
a 500 that kills the whole reply.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
### What problem does this PR solve?
Fixes#15427.
All LiteLLM-routed chats fail with:
- Anthropic: `litellm.BadRequestError: AnthropicException -
{"type":"invalid_request_error","message":"model_type: Extra inputs are
not permitted"}`
- OpenAI: `litellm.BadRequestError: OpenAIException - Unknown parameter:
'model_type'`
This is a regression from v0.25.4.
#### Root cause
A chat assistant's `llm_setting` is forwarded to the model as
`gen_conf`. `llm_setting` can legitimately carry RAGFlow-internal
metadata such as `model_type` (the chat REST APIs in
`api/apps/restful_apis/` read it back out of `llm_setting`), so that key
ends up inside `gen_conf`.
`Base._clean_conf` (OpenAI-compatible providers) already **whitelists**
the keys it forwards, so direct-OpenAI providers were unaffected.
`LiteLLMBase._clean_conf` only dropped `max_tokens` and passed
everything else straight through to `litellm.acompletion`, which
forwarded `model_type` to the upstream provider — and Anthropic / OpenAI
reject it. Because both Claude and GPT route through LiteLLM, every chat
broke.
#### Fix
- Extract the allowed-key set into a shared `ALLOWED_GEN_CONF_KEYS`
constant and reuse it in `Base._clean_conf`.
- Apply the same whitelist in `LiteLLMBase._clean_conf`, plus the
LiteLLM-specific reasoning params (`thinking`, `reasoning_effort`,
`extra_body`) that the model-family policies inject for reasoning
models.
This covers all four LiteLLM completion paths (`async_chat`,
`async_chat_streamly`, `async_chat_with_tools`,
`async_chat_streamly_with_tools`), since they all route through
`_clean_conf`.
#### Tests
Adds `test/unit_test/rag/llm/test_clean_conf_whitelist.py` covering both
backends: `model_type` (and other stray keys) are dropped, genuine
generation params and `thinking` survive, `max_tokens` is removed, and
the whitelist invariants hold.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
- [x] Added test cases
## What
#15240
Implements `GET /api/v1/mcp/servers` in the Go API server.
## Changes
- Added MCP server DAO list query with tenant scoping.
- Added MCP service response wrapper.
- Added MCP handler for list request parsing and response formatting.
- Wired `GET /api/v1/mcp/servers` under authenticated `/api/v1` routes.
- Initialized MCP service and handler in the Go server startup.
- update_time and update_date now both map to update_date
- create_time and create_date now both map to create_date
- default ordering now returns create_date
## API Behavior
Matches the Python endpoint behavior:
- Requires authenticated user.
- Lists MCP servers for the current user tenant.
- Supports `keywords`.
- Supports `mcp_id` and repeated/comma-separated `mcp_ids`.
- Supports `page`, `page_size`, `orderby`, and `desc`.
- Returns:
```json
{
"code": 0,
"message": "success",
"data": {
"mcp_servers": [],
"total": 0
}
}
```
## Summary
Add Tigris configuration to the Configuration and Backup & migration
pages, using the existing AWS_S3 backend — no code changes required.
Fix `region` → `region_name` in the existing S3 config example in
`backup_and_migration.md`. The code in `s3_conn.py` reads `region_name`,
so the previous field name was silently ignored.
##Context
With MinIO's open-source repository archived (#13840 on
infiniflow/ragflow), users need documented alternatives for object
storage. Tigris is S3-compatible and works with RAGFlow's existing
AWS_S3 backend out of the box.
## Changes
`configurations.md`: Added `### s3 (Tigris)` section after `### minio`,
matching the existing reference style. Includes config block, field
descriptions, and a pointer to `service_conf.yaml.template` for other
S3-compatible backends.
`backup_and_migration.md`: Added Tigris config block under single-bucket
mode. Fixed region → region_name in the existing S3 example. Added
Tigris to the supported backends list.
##Notes
No new files — edits to existing docs only.
Config field names (`access_key`, `secret_key`, `region_name`,
`endpoint_url`, `bucket`, `prefix_path`, `signature_version`,
`addressing_style`) verified against `rag/utils/s3_conn.py`.
## Summary
- Skip MinerU `header`, `footer`, and `page_number` blocks when
converting `content_list.json` into sections.
- Ignore unsupported block types explicitly so future MinerU output
types cannot re-emit the previous text block.
Fixes duplicate text in General/naive chunks when parsing PDFs via
MinerU (reported with repeated page headers and body text in slices).
Closes#15335
## Test plan
- [x] `pytest test/unit_test/deepdoc/parser/test_mineru_parser.py -v`
(4/4 passed)
## Summary
- Add custom `base_url` support to the Google Go model driver.
- Preserve Google URL suffix configuration when creating custom base URL
driver instances.
- Validate Google chat/stream request inputs before constructing the SDK
client.
- Cover Google model listing, connection checks, base URL resolution,
and request validation with focused tests.
## What changed
- `GoogleModel.NewInstance` now returns a Google driver configured with
the supplied base URL map.
- Google SDK client creation now resolves configured base URLs through
`genai.HTTPOptions.BaseURL`.
- Base URL lookup supports configured regions, empty-region keys, and
`default` fallback.
- Google chat, streaming chat, embeddings, and model listing now reject
blank API keys before creating SDK clients.
- Google chat and streaming chat now reject blank model names locally,
and streaming chat rejects a nil sender.
- Existing message handling, embeddings, pagination, and provider errors
are preserved.
## Why
Google custom model instances could not use configured base URLs because
`NewInstance` returned `nil` and the SDK client path ignored the driver
base URL map. The request validation keeps invalid Google calls from
reaching SDK client construction with blank credentials or incomplete
chat inputs.
### What problem does this PR solve?
## Problem
When parsing PDFs containing English figure/table captions (e.g. "Fig.
20", "Figure 20", "Table 20"), the `is_caption` method in
`TableStructureRecognizer` failed to recognize them as captions. This
caused figure numbering gaps in the parsed output (e.g. Fig. 19 → Fig.
21, skipping Fig. 20).
## Root Cause
The `is_caption` regex only matched Chinese caption formats:
```python
patt = [r"[图表]+[ 0-9:：]{2,}"]
```
When the layout recognizer also failed to assign a `caption` layout type
to a given text block, English captions were entirely missed.
## Fix
Added three case-insensitive English caption patterns to `is_caption` in
`deepdoc/vision/table_structure_recognizer.py`:
- `(?i)Fig\.?\s*\d+` — matches `Fig. 20`, `Fig 20`, `FIG. 20`, etc.
- `(?i)Figure\s+\d+` — matches `Figure 20`, `FIGURE 20`, etc.
- `(?i)Table\s+\d+` — matches `Table 20`, `TABLE 20`, etc.
## Files Changed
- `deepdoc/vision/table_structure_recognizer.py` — extended `is_caption`
regex patterns
- [x] Bug Fix (non-breaking change which fixes an issue)
Signed-off-by: noob <yixiao121314@outlook.com>
### What problem does this PR solve?
Fix: The newly added model did not appear in the drop-down menu.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
fix: restore TitleChunker output for json/chunks upstream formats
## Summary
The refactor commit e194027b (#14247) introduced two regressions that
caused `TitleChunker` to produce zero chunks when the upstream Parser
node outputs `json` or `chunks` format (e.g. PDF parsing).
## Root Cause
### 1. Dead code in `extract_line_records` (critical)
After refactor, when `payload` is `None` (which is the case for `json`
and `chunks` output formats), the method returns an empty list
immediately via `return []`, so no records are ever extracted from
structured upstream output. The original `json`/`chunks` handling code
became unreachable dead code.
### 2. Unconditional overwrite in `build_chunks_from_record_groups`
The `chunks` variable assigned in the `if` branch for markdown/text/html
formats was unconditionally overwritten by the statement below it, due
to a missing `else` keyword.
## Fix
- Remove the premature `return []` so the `json`/`chunks` branch is
reachable again.
- Add `else` branch in `build_chunks_from_record_groups` so the two
format families are handled independently.
## Test Plan
- [x] Verified no lint errors on the changed file
- [ ] Tested with a PDF document parsed via DeepDOC → TitleChunker
pipeline
- [ ] Tested with markdown input through TitleChunker
- [ ] Tested hierarchy and group chunking modes
## Impact
- Fixes the regression where documents parsed with `json`/`chunks`
output format produced no chunks from `TitleChunker`.
- No API or configuration changes. Fully backward compatible.
Signed-off-by: noob <yixiao121314@outlook.com>
### What problem does this PR solve?
Part of the Python → Go API server rewrite tracked in #15240 (Dataset
ingestion section). This PR implements the three dataset ingestion
endpoints in the Go API server, mirroring the existing Python
`dataset_api_service` behaviour:
- `GET /api/v1/datasets/<dataset_id>/ingestions/summary`
- `GET /api/v1/datasets/<dataset_id>/ingestions`
- `GET /api/v1/datasets/<dataset_id>/ingestions/<log_id>`
### Type of change
- [x] Refactoring
- [x] New Feature (non-breaking change which adds functionality)
Co-authored-by: sxxtony <sxxtony@users.noreply.github.com>
### What problem does this PR solve?
The Go GPUStack driver returned a stub error for `Embed()` even though
GPUStack exposes OpenAI-compatible embeddings on the **v1-openai** route
(not `v1/embeddings`).
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
### What problem does this PR solve?
As title
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
- [x] New Feature (non-breaking change which adds functionality
### What problem does this PR solve?
extend restful api suite
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
- [x] Other (please describe): test
### What problem does this PR solve?
Fix: If the filename is too long, it overflows the confirmation box for
deleting the file.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
### What problem does this PR solve?
Closes#15332.
RAGFlow can index Gmail and generic IMAP mailboxes but had no native
connector for Outlook / Microsoft 365 mail. Organisations on Microsoft
365 had no way to bring mailbox content into a knowledge base through
Microsoft Graph.
This PR adds a net-new Outlook data source that:
- Authenticates against Microsoft Graph with the same MSAL
client-credentials flow already used by the SharePoint and Teams
connectors (no new auth primitives).
- Pages over `/users/{id}/mailFolders/{folder}/messages/delta` per
mailbox and persists `@odata.deltaLink` values in
`OutlookCheckpoint.delta_links`, so incremental syncs only fetch changed
messages.
- Supports two scoping modes:
- **Tenant-wide** (default): enumerates every user in the tenant via
`/users` and syncs each mailbox. Requires `User.Read.All`.
- **Targeted**: when `user_ids` is provided (comma-separated UPNs or
object IDs), only those mailboxes are synced. `User.Read.All` is not
needed in this mode.
- Lets the caller pick the mail folder (`inbox`, `sentitems`, `archive`,
...). Defaults to `inbox`.
- Maps each message to a `Document` shaped after the Gmail connector:
one `TextSection` carrying `From/To/Cc/Subject` headers + body, with
HTML bodies stripped to text inline (no extra dependency).
- Surfaces typed errors on the validation probe:
401 → `ConnectorMissingCredentialError`, 403 →
`InsufficientPermissionsError` (with `Mail.Read` / `User.Read.All`
hint), 404 on a configured mailbox → `ConnectorValidationError`, 5xx →
`UnexpectedValidationError`.
- Skips messages flagged `@removed` by the delta semantics and messages
whose `receivedDateTime` is older than `poll_range_start`.
#### Files
| File | Change |
|------|--------|
| `common/data_source/outlook_connector.py` | **New** —
`OutlookConnector` (`CheckpointedConnectorWithPermSync` +
`SlimConnectorWithPermSync`) + `OutlookCheckpoint` + tiny `_strip_html`
helper. |
| `common/data_source/config.py` | `DocumentSource.OUTLOOK = "outlook"`.
|
| `common/constants.py` | `FileSource.OUTLOOK = "outlook"`. |
| `common/data_source/__init__.py` | Export `OutlookConnector`. |
| `rag/svr/sync_data_source.py` | `Outlook(SyncBase)` with `batch_size`
normalisation, CSV/list parsing of `user_ids`; registered in
`func_factory`. |
| `web/src/pages/user-setting/data-source/constant/index.tsx` |
`DataSourceKey.OUTLOOK`, visibility map (`syncDeletedFiles: true`), info
entry, form fields (tenant_id, client_id, client_secret, folder,
user_ids, batch_size), default values. |
| `web/src/locales/en.ts`, `web/src/locales/zh.ts` |
`outlookDescription` + 5 tooltip keys (EN + ZH). |
| `test/unit_test/data_source/test_outlook_connector_unit.py` | **New**
— 19 unit tests (`p1`/`p2`/`p3`) covering auth, validation (tenant-wide
vs specific user vs error paths), checkpoint helpers, user enumeration
pagination, message filtering, HTML body stripping. |
#### Required Azure AD permissions
- `Mail.Read` (Application, admin-granted) — always.
- `User.Read.All` (Application, admin-granted) — only when `user_ids` is
left blank so the connector can enumerate mailboxes.
#### Out of scope
- **Attachment indexing.** The current connector emits message body +
headers; binary attachments are flagged via `metadata.has_attachments`
but not pulled. Adding attachment hydration is straightforward but
scoped out per the issue's "decide whether attachments are indexed in
the first version" note.
- **Delegated (per-user) OAuth.** The connector uses app-only
credentials, consistent with the SharePoint / Teams precedent in this
codebase.
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
### What problem does this PR solve?
Document metadata is completely broken on the OpenSearch backend
(`DOC_ENGINE=opensearch`). Both failures were introduced by #14577,
which added
a doc-metadata dispatch surface but only validated it against
Elasticsearch.
**1. Index creation rejected (`mapper_parsing_exception`).**
`OSConnection.create_doc_meta_idx` feeds `conf/doc_meta_es_mapping.json`
verbatim to OpenSearch. That file declares a top-level `"dynamic":
"runtime"`.
Runtime fields are Elasticsearch-only; OpenSearch cannot parse the
value:
mapper_parsing_exception: Could not convert [dynamic.dynamic] to boolean
(400)
**2. `search()` signature mismatch (`TypeError`).**
`DocMetadataService` (added by #14577) calls `docStoreConn.search(...)`
with
snake_case kwargs (`select_fields=`, `index_names=`,
`knowledgebase_ids=`, …),
matching `ESConnection.search`. But `OSConnection.search` still uses
camelCase
parameters (`selectFields`, `indexNames`, `knowledgebaseIds`, …):
TypeError: OSConnection.search() got an unexpected keyword argument
'select_fields'
The UI then shows "0 fields" for every document on OpenSearch.
### Fix
1. In `OSConnection.create_doc_meta_idx`, normalize a top-level
`"dynamic": "runtime"` to `True` **for the OpenSearch request only**.
The
shared mapping file is left untouched, so the Elasticsearch backend
keeps its
runtime-field behavior. Dynamic field discovery is preserved on
OpenSearch.
2. Rename the `OSConnection.search()` parameters (and their in-method
local
uses) from camelCase to snake_case so they match `ESConnection.search()`
and
the `DocMetadataService` call sites. The change is confined to
`search()`;
`get/insert/update/delete` keep their existing positional signatures
(they
are called positionally from `rag/nlp/search.py`).
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
### Affected backends
OpenSearch only. Elasticsearch, Infinity and OceanBase are untouched.
### How to reproduce
1. `DOC_ENGINE=opensearch`, restart the stack.
2. Upload/parse a document, then open the dataset's document list / set
metadata.
- Before: index creation 400s (`Could not convert [dynamic.dynamic]`),
and/or
`TypeError ... 'select_fields'`; document metadata shows 0 fields.
### Risk & backward compatibility
- ES default deployment: no change. `doc_meta_es_mapping.json` is not
modified,
so ES still receives `"dynamic": "runtime"`.
- `search()` rename is internal; the only kwarg caller
(`DocMetadataService`)
already uses the snake_case names this PR aligns to.
### Test plan
- [ ] `DOC_ENGINE=opensearch`: per-tenant `ragflow_doc_meta_*` index is
created
(no `mapper_parsing_exception`); document metadata reads/writes work.
- [ ] `DOC_ENGINE=elasticsearch` regression: doc-meta index still
created with
runtime mapping; metadata unchanged.
### What problem does this PR solve?
On the OpenSearch backend (`DOC_ENGINE=opensearch`), every retrieval
that
performs the KNN second-pass scoring crashes with:
AttributeError: 'OSConnection' object has no attribute 'get_scores'
**Root cause.** #14970 ("Refactor: Drop the vector fetch for ES") added
a
`get_scores()` helper to `ESConnectionBase`
(`common/doc_store/es_conn_base.py`)
and introduced `Dealer._knn_scores()` in `rag/nlp/search.py`, which
calls
`self.dataStore.get_scores(res)`. `search.py` routes Infinity and
OceanBase to
their own similarity paths via `DOC_ENGINE_INFINITY` /
`DOC_ENGINE_OCEANBASE`,
but OpenSearch sets neither flag, so it falls into the Elasticsearch
branch and
calls `get_scores`. `OSConnection` (which subclasses
`DocStoreConnection`
directly, not `ESConnectionBase`) never received that method, so any
vector-search hit triggers the crash. It reproduces with any normal
embedding
(e.g. 1024-dim mistral-embed) as soon as a KNN query returns hits.
### Fix
Add `OSConnection.get_scores()`, mirroring
`ESConnectionBase.get_scores()`.
OpenSearch hit headers expose `_score` exactly like Elasticsearch (the
existing
`OSConnection.__getSource` already reads `d["_score"]`), so the
implementation
is identical.
Scope note: Infinity and OceanBase deliberately do not use `get_scores`
(#14970 routes them elsewhere), so this fix is intentionally limited to
the
OpenSearch backend, which is the only one reaching the ES KNN-score
path.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
### Affected backends
OpenSearch only. Elasticsearch already implements `get_scores`; Infinity
/
OceanBase are routed away from it.
### How to reproduce
1. `DOC_ENGINE=opensearch` (docker `.env`), restart the stack.
2. Create a knowledge base with any dense embedding model and parse a
document.
3. Run a retrieval / chat over that KB -> 500 with the AttributeError
above.
### Risk & backward compatibility
None for the default Elasticsearch deployment -- the change only adds a
method
to `OSConnection`. No default values or ES/Infinity/OceanBase behavior
change.
### Test plan
- [ ] With `DOC_ENGINE=opensearch`, retrieval over a KB returns scored
chunks
(no AttributeError).
- [ ] `DOC_ENGINE=elasticsearch` regression: retrieval unchanged.
- [ ] Empty-result path: `_knn_scores` early-returns `{}` (guarded),
get_scores
handles an empty `hits` list gracefully.
### Related issues
Closes#15310
### What problem does this PR solve?
`/api/v1/dify/retrieval` had duplicate `GET` route registrations in
`dify_retrieval_api.py`: one for authenticated retrieval and another for
unauthenticated health checks. Sharing the same path and method created
ambiguous routing behavior and an unstable API contract for Dify
external knowledge base integration.
This PR separates concerns by moving the health-check endpoint to `GET
/api/v1/dify/retrieval/health`, while keeping retrieval on
`/api/v1/dify/retrieval`. This makes auth behavior deterministic and
prevents route shadowing/conflicts.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
## Summary
Upgrade crawl4ai from 0.7.6 to 0.8.0 to fix CVE-2026-26217.
## Vulnerability
| Field | Value |
|-------|-------|
| **ID** | CVE-2026-26217 |
| **Severity** | CRITICAL |
| **Scanner** | trivy |
| **Rule** | `CVE-2026-26217` |
| **File** | `uv.lock` |
| **Assessment** | Likely exploitable |
**Description**: Crawl4AI Has Local File Inclusion in Docker API via
file:// URLs
## Evidence
**Scanner confirmation**: trivy rule `CVE-2026-26217` flagged this
pattern.
**Production code**: This file is in the production codebase, not
test-only code.
## Threat Model Context
This is a web service - vulnerabilities in request handlers are directly
exploitable by remote attackers.
## Changes
- `pyproject.toml`
- `uv.lock`
## Verification
- [x] Build passes
- [x] Scanner re-scan confirms fix
- [x] LLM code review passed
---
*This change addresses a pattern flagged by static analysis. The code
path handles user-influenced input and the fix reduces the attack
surface against both manual and automated exploitation.*
---
*Automated security fix by [OrbisAI Security](https://orbisappsec.com)*
This PR introduces a `.rooignore` file to the root of the repository to
optimize how AI coding assistants (like Roo) interact with the RAGFlow
codebase.
Currently, when AI agents index the workspace, they can waste tokens and
processing time reading through generated files, caches, large
dependency artifacts, and runtime logs. This `.rooignore` file provides
a standard configuration to exclude these irrelevant directories and
files (such as `.venv/`, `node_modules/`, `__pycache__/`, logs, and
large binaries). This significantly reduces indexing noise, prevents
accidental reads of sensitive or bulky local data, and ensures AI coding
agents remain focused strictly on relevant source code.
### Type of change
- [x] Other (please describe): Developer Experience (DX) / AI Tooling
configuration
### What problem does this PR solve?
This PR updates `SystemService.ListAPITokens` to lazily backfill missing
`beta` values for API tokens, matching the Python behavior of
`/api/v1/system/tokens`.
### Type of change
- When an API token has an empty `beta`, generate a new one.
- Persist the generated `beta` back to the `api_token` table.
- Keep the handler/routing unchanged.
- `GET /api/v1/system/tokens` now returns tokens with `beta` filled in
for older records that were missing it.
- This aligns Go behavior with the Python implementation.
## Summary
- Validate Hunyuan embedding model name and API key before building
requests.
- Reuse region-aware base URL validation for embedding requests.
- Replace the stale unsupported Embed test with happy-path and
validation coverage.
## What changed
- Added early Hunyuan Embed validation for missing model names and API
keys.
- Routed Embed through the same base URL region guard used by the other
Hunyuan methods.
- Updated Hunyuan tests to configure the embedding suffix and cover
Embed success plus invalid inputs.
## Why
Hunyuan Embed is implemented, but the existing test still expected it to
be unsupported and could panic before returning a normal validation
error. This keeps the implemented embedding path aligned with the
current driver behavior and prevents nil input panics.
Closes#15087
Refs #14736
### What problem does this PR solve?
Closes#15180.
`OIDCClient.parse_id_token` in `api/apps/auth/oidc.py` read the JWT
signing
algorithm from the **unverified** JWT header and passed it through to
`jwt.decode(..., algorithms=[alg], ...)` as the trust anchor. This is
the
textbook JWT algorithm-confusion vulnerability (CWE-345 / CWE-347). Any
unauthenticated client capable of reaching the OIDC callback could take
over
an arbitrary account on any RAGFlow deployment with OIDC login enabled:
1. **`alg: "none"`** — present a JWT with `{"alg": "none"}` and no
signature segment → `jwt.decode(..., algorithms=["none"])` → PyJWT's
`NoneAlgorithm` accepts the token without verification → login as any
user.
2. **RSA / HMAC confusion** — fetch the public RSA key from the
provider's
JWKS (it's public), forge a JWT with `{"alg": "HS256"}` HMAC-signed
using the public-key bytes as the secret → `jwt.decode(...,
algorithms=["HS256"], key=public_key)` → verifier accepts → login as
any user. (Modern PyJWT independently refuses to use a PEM-formatted
key as an HMAC secret, which mitigates this leg for PEM key formats;
the fix here is the only mitigation for raw / DER / JWK octet keys and
for older PyJWT versions.)
### What changed
**`api/apps/auth/oidc.py`:**
- New module constants `_ALLOWED_OIDC_SIGNING_ALGS` (asymmetric-only:
`RS*`, `ES*`, `PS*`, `EdDSA` — explicitly excludes `none` and `HS*`)
and `_DEFAULT_OIDC_SIGNING_ALGS = ("RS256",)` (the OIDC Core 1.0 §2
spec default).
- New helper `_resolve_id_token_signing_algs(metadata)` — intersects the
provider's advertised `id_token_signing_alg_values_supported` from
`/.well-known/openid-configuration` with the safe allowlist; falls back
to RS256 when the field is missing or contains only unsafe values.
- `OIDCClient.__init__` now stores the resolved allowlist on
`self.id_token_signing_algs` — pinned once, from a trusted source, at
construction time.
- `parse_id_token` no longer calls `jwt.get_unverified_header` and no
longer reads `alg` from the JWT header. It passes
`self.id_token_signing_algs` to `jwt.decode(..., algorithms=...)`.
`PyJWKClient.get_signing_key_from_jwt` still reads the `kid` from the
header internally for JWKS lookup — that's fine, `kid` is not a
security decision; the signature still proves which key was actually
used.
**`test/testcases/test_web_api/test_auth_app/test_oidc_client_unit.py`:**
- Existing `test_parse_id_token_success_and_error` drops its
`jwt.get_unverified_header` mock (no longer called by `parse_id_token`).
- `_metadata` and `_make_client` helpers grew an optional `signing_algs`
parameter so tests can configure what the discovery document advertises.
- New `TestSSRFValidation` / algorithm-confusion regression block (7
tests):
- `test_id_token_signing_algs_default_to_rs256_when_metadata_missing`
- `test_id_token_signing_algs_intersect_metadata_with_safe_allowlist`
- `test_id_token_signing_algs_fall_back_when_only_unsafe_advertised`
- `test_id_token_signing_algs_ignores_non_string_entries`
- `test_id_token_signing_algs_handles_non_list_metadata_field`
- `test_parse_id_token_passes_pinned_algorithms_to_jwt_decode` —
sabotages `jwt.get_unverified_header` to raise on call, proving the
verification path never consults the unverified header.
- `test_parse_id_token_rejects_alg_none` — uses real PyJWT to encode an
`alg: "none"` token; `parse_id_token` raises `ValueError("Error
parsing ID Token: …")` instead of accepting it.
- `test_parse_id_token_rejects_hs256_when_allowlist_is_asymmetric` —
uses real PyJWT to forge an `alg: "HS256"` token with a non-PEM
shared secret (so PyJWT's incidental PEM-as-HMAC refusal isn't what
blocks it); `parse_id_token` raises because `HS256` is not in the
pinned allowlist.
Sanity-checked end-to-end with real PyJWT outside the project test
runner:
- `alg=none` forged token + `algorithms=["RS256"]` →
`InvalidAlgorithmError` ✓
- `alg=HS256` forged token + `algorithms=["RS256"]` →
`InvalidAlgorithmError` ✓
- Same `alg=HS256` token + `algorithms=["HS256"]` → **accepted**
({'sub': 'admin'})
— confirming the attack path was real before the fix.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
Co-authored-by: galuis116 <contact@duerrimports.com>
### Summary
Closes#15381
Every provider in `internal/entity/models/` reads its streaming response
with `bufio.NewScanner(resp.Body)` and iterates over `scanner.Scan()`.
The default `bufio.Scanner` maximum token size is 64KB, so when an
upstream sends a single SSE `data:` line larger than 64KB (long content
deltas, large tool or function call argument blobs, bundled
`reasoning_content`, or providers that emit a whole message in one
event) `scanner.Scan()` returns `false` and `scanner.Err()` returns
`bufio.ErrTooLong`. Streaming chat then ends with an error partway
through the response.
This change adds `scanner.Buffer(make([]byte, 64*1024), 1024*1024)`
immediately after every SSE scanner that was still bare, raising the cap
to 1MB. 1MB is the value already used for streaming chat in `openai.go`,
`modelscope.go`, `groq.go`, `mistral.go`, `xai.go` and the other already
patched providers (the 8MB cap in the repo is reserved for TTS and
embedding paths), so this simply converges the remaining providers onto
the established pattern. Nothing else changes: line parsing, `data:`
prefix handling, `[DONE]` detection, JSON unmarshalling, error handling,
and the existing `scanner.Err()` checks all stay the same.
Providers covered (23 scanners across 22 files): 302ai, aliyun,
baichuan, baidu, cohere, deepinfra, deepseek, gitee, huggingface,
lmstudio, minimax (the chat scanner, whose TTS scanner was already
bumped), moonshot, nvidia, ollama, openrouter, orcarouter, paddleocr,
siliconflow, tokenhub, vllm, volcengine, xunfei, zhipu-ai. `jiekouai.go`
is excluded because it is covered by the in flight #15337.
A table driven regression test (`sse_scanner_buffer_test.go`) streams a
single 128KB `data:` content delta followed by `data: [DONE]` through an
`httptest` server and asserts that `ChatStreamlyWithSender` delivers the
full content with no error across a representative subset of providers.
Without the buffer fix the test fails with `bufio.Scanner: token too
long`.
This PR also removes three duplicate declarations of the package level
`roundTripperFunc` test helper that several recently merged provider PRs
each added independently, which had left the `internal/entity/models`
test package unable to compile. The helper now lives in a single place
and is shared.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
## Summary
- Fix `meta_filter()` AND logic so an empty result from an early
condition is not overwritten when a later condition matches.
- Add regression tests for empty-first AND, successful AND intersection,
and OR behavior after an empty first condition.
Fixes incorrect `/retrieval` metadata filtering when multiple AND
conditions are used and the first condition matches no documents.
Closes#15360
## Test plan
- [x] `pytest test/unit_test/common/test_metadata_filter_operators.py
-v` (19/19 passed)
### What problem does this PR solve?
Fixes custom `base_url` resolution when a model instance has no
configured region.
Some drivers read custom base URLs from `BaseURL[""]` when
`apiConfig.Region` is empty, while others normalize empty region to
`"default"` and read `BaseURL["default"]`. This PR adds the `"default"`
alias only for empty-region custom base URLs while preserving the
existing empty-region key.
Closes#15042
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
### What this PR fixes
This PR fixes an issue in the Python backend where user logout did not
reliably persist the invalidated access_token to the database.
Although the logout endpoint returned success and logged that the token
had been invalidated, the user.access_token value could remain
unchanged in the database, which meant the previous login token could
stay valid longer than expected.
### What changed
- Resolve the real user object before updating the token
- Persist the invalidated access_token before calling logout_user()
- Return a server error if the token update is not written successfully
### Impact
- Logging out now correctly replaces the stored access_token with an
INVALID_... value
- The previous login session is properly invalidated
- The change is limited to the logout flow and is intentionally small in
scope
### What problem does this PR solve?
This PR fixes several behavior gaps in the Go implementation of the user
registration API.
### Type of change
- Make `nickname` required for user registration.
- Align registration error messages and response data with expected API
behavior.
- Handle password decryption errors for registration more consistently.
- Generate UUID v1-style IDs for new users, access tokens, tenants,
user-tenant records, and root files.
- Initialize default user fields during registration, including:
- language
- color schema
- timezone
- last login time
- Create user, tenant, user-tenant relation, tenant LLM records, and
root folder in a single DB transaction.
- Initialize default tenant LLM records from configured default models.
- Avoid partial registration data when one creation step fails.
- Use locale-based default language fallback for user profile responses.
### What problem does this PR solve?
Added 4 new models:
deepseek-ai/DeepSeek-V4-Pro
deepseek-ai/DeepSeek-V4-Flash
Pro/moonshotai/Kimi-K2.6
Pro/zai-org/GLM-5.1
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
### What problem does this PR solve?
Closes#15330.
RAGFlow had no connector for OneDrive / OneDrive for Business. Users who
store working documents in OneDrive could not index them into a
knowledge base without manually downloading and re-uploading files.
This PR adds a net-new OneDrive data source that:
- Authenticates against Microsoft Graph with the same MSAL
client-credentials flow already used by the SharePoint and Teams
connectors (no new auth primitives).
- Enumerates every drive visible to the service principal and pages
through `/drives/{id}/root/delta`, persisting `@odata.deltaLink` values
per drive so subsequent syncs only fetch changed items.
- Optionally narrows ingestion to a sub-folder (`folder_path`) without
needing a separate code path.
- Surfaces typed errors on the validation probe (`GET /drives?$top=1`):
401 → `ConnectorMissingCredentialError`, 403 →
`InsufficientPermissionsError` (with a `Files.Read.All` hint), 5xx →
`UnexpectedValidationError`.
- Filters folders, soft-deleted items, and unsupported extensions (`.pdf
.docx .doc .xlsx .xls .pptx .ppt .txt .md .csv`).
#### Files
| File | Change |
|------|--------|
| `common/data_source/onedrive_connector.py` | **New** —
`OneDriveConnector` + `OneDriveCheckpoint`. |
| `common/data_source/config.py` | `DocumentSource.ONEDRIVE =
"onedrive"`. |
| `common/constants.py` | `FileSource.ONEDRIVE = "onedrive"`. |
| `common/data_source/__init__.py` | Export `OneDriveConnector`. |
| `rag/svr/sync_data_source.py` | `OneDrive(SyncBase)` with `batch_size`
normalisation; registered in `func_factory`. |
| `web/src/pages/user-setting/data-source/constant/index.tsx` |
`DataSourceKey.ONEDRIVE`, visibility map (`syncDeletedFiles: true`),
info entry, form fields (tenant_id, client_id, client_secret,
folder_path, batch_size), default values. |
| `web/src/locales/en.ts`, `web/src/locales/zh.ts` |
`onedriveDescription` + 4 tooltip keys (EN + ZH). |
| `test/unit_test/data_source/test_onedrive_connector_unit.py` | **New**
— 13 unit tests (`p1`/`p2`) covering auth, validation, checkpoint
helpers, and document filtering. |
#### Required Azure AD permission
`Files.Read.All` (Application, admin-granted).
#### Out of scope
- Interactive end-user OAuth (delegated permissions) — the connector
uses app-only credentials, consistent with the SharePoint / Teams
precedent.
- Binary download of file contents — the sync layer emits `Document`s
carrying `webUrl` + metadata; bytes are hydrated downstream by the parse
pipeline.
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
### What problem does this PR solve?
default OpenAI chat completions to non-stream when `stream` is omitted
https://github.com/infiniflow/ragflow/issues/15356
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
### What problem does this PR solve?
Python implementation of the Go-based model_provider API suite.
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
---------
Co-authored-by: bill <yibie_jingnian@163.com>
## Summary
- Harden `NewNovitaModel` to avoid panics when `http.DefaultTransport`
is a custom non-`*http.Transport` RoundTripper.
- Fallback to a safe transport (`ProxyFromEnvironment`) while preserving
existing pooling/timeout settings.
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Jin Hai <haijin.chn@gmail.com>
### What problem does this PR solve?
Closes#15199.
The add-custom-model endpoint is routed through
`/api/v1/providers/:provider_name/instances/:instance_name/models`, but
the handler previously trusted `provider_name` and `instance_name` from
the JSON body instead of the path target. A request could therefore hit
one provider/instance URL while operating on a different body
provider/instance.
The same handler only rejected `model_types` when the slice was nil. An
empty array passed validation and reached
`ModelProviderService.AddCustomModel`, where `request.ModelTypes[0]`
could panic.
This PR makes the path provider/instance authoritative, rejects
mismatched body values, rejects missing or empty `model_types`, and adds
a service-level guard so direct service callers cannot hit the same
panic path.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
## Summary
Ports the Python `tenant_api` team/member management endpoints to Go,
adding 4 endpoints under `/api/v1/tenants/:tenant_id/`:
- `GET /tenants/:tenant_id/users` — list non-owner members with user
details (owner only)
- `POST /tenants/:tenant_id/users` — invite a user by email; creates
invite-role join record (owner only)
- `DELETE /tenants/:tenant_id/users` — remove a member by `user_id`;
owner can remove anyone, members can remove themselves
- `PATCH /tenants/:tenant_id` — accept a pending invitation,
transitioning role `invite → normal`
Closes#15294
### What problem does this PR solve?
As title
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
- [x] New Feature (non-breaking change which adds functionality)
- [x] Refactoring
## Summary
- Harden JieKouAI request validation before outbound provider calls
- Force non-streaming and streaming chat methods to use their expected
stream modes
- Make model listing use a bodyless GET and parse model responses
without panics
Closes#14736
---------
Co-authored-by: Jin Hai <haijin.chn@gmail.com>
## Summary
Closes#15268.
The `UpdateMetadataSetting` handler at `internal/handler/kb.go:126`
retrieved the authenticated user via `GetUser(c)` but discarded the user
object (`_, errorCode, errorMessage := GetUser(c)`), then forwarded the
caller-supplied `kb_id` straight to the service layer with no ownership
check. Any authenticated user could mutate the `parser_config` /
metadata of any knowledge base in the system by guessing or harvesting a
`kb_id` — a classic IDOR (CWE-284, OWASP A01).
This is the only handler in `internal/handler/kb.go` missing the check;
every sibling (`ListTags`, `ListTagsFromKbs`, `RenameTag`,
`KnowledgeGraph`, `DeleteKnowledgeGraph`, `GetMeta`, `GetBasicInfo`)
already calls `h.kbService.Accessible(kbID, user.ID)`. The same
defensive check on the document preview endpoint was added in PR #14625
— this PR closes the matching gap on the KB metadata endpoint.
---------
Co-authored-by: Jin Hai <haijin.chn@gmail.com>
## Summary
- Harden `NewModelScopeModel` to avoid panics when
`http.DefaultTransport` is a custom non-`*http.Transport` RoundTripper.
- Fallback to a safe transport (`ProxyFromEnvironment`) while preserving
existing pooling/timeout settings.
- Add `TestModelScopeNewModelWithCustomDefaultTransport` regression
coverage.
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Jin Hai <haijin.chn@gmail.com>
## Summary
Closes: #15328
- Implements `GET /api/v1/agents` — the agent/canvas listing endpoint
needed to complete the Home dashboard tile in `web/src/pages/home/`.
- Mirrors Python `api/apps/restful_apis/agent_api.py::list_agents`
exactly: tenant-join auth, optional `owner_ids` guard, keyword filter,
pagination, ordering, and `canvas_category` filter (default:
`agent_canvas`).
- **Scope:** read-only list only. Full agent CRUD and canvas runtime are
explicitly out of scope (separate slice of #15240).
## Summary
Ports the connector (data source) management endpoints that power
`web/src/pages/user-setting/data-source/` from Python
(`api/apps/restful_apis/connector_api.py`) to Go. Previously only `GET
/connectors` (list) was implemented in Go; this adds the rest of the
lifecycle.
Closes#15273 (subtask of #15240).
## Endpoints implemented
All under base path `/api/v1` (mirrors the Python routes):
| Method | Path | Description |
|--------|------|-------------|
| POST | `/connectors/{connector_id}/test` | Validate stored credentials
|
`GET /connectors` (list) was already present and is unchanged.
---------
Co-authored-by: Jin Hai <haijin.chn@gmail.com>
### What problem does this PR solve?
Closes#15191.
RAGFlow shipped a Microsoft Teams connector stub
(`common/data_source/teams_connector.py`) whose document-loading methods
all returned `[]`, `Teams._generate()` was a `pass`, and Teams was
commented out of the data-source settings UI. As a result there was no
way to index Teams channel conversations into a knowledge base.
This PR implements the connector end to end on top of Microsoft Graph
(Office365-REST-Python-Client). It shares the MSAL client-credentials
auth shape with the SharePoint connector.
**Backend**
- `common/data_source/teams_connector.py`
- `load_credentials()` now builds the Graph client using an MSAL
client-credentials **token callback** — the form `GraphClient` actually
expects. (The previous stub passed a raw access-token string to
`GraphClient(...)`, which is not how that client is driven.) Token
acquisition is lazy, so credential loading performs no network call.
- `validate_connector_settings()` lists teams via Graph.
- `load_from_checkpoint()` is now a generator that pages teams →
channels → messages, flattens each top-level post together with its
replies into one blob-based `Document` (`extension` `.txt`/`.html`,
`blob`, `size_bytes`, `doc_updated_at`). Incremental syncs are bounded
by message `lastModifiedDateTime` (falling back to `createdDateTime`).
Per-message errors surface as `ConnectorFailure` instead of aborting the
run.
- `retrieve_all_slim_docs_perm_sync()` yields id-only `SlimDocument`
batches and the checkpoint helpers return proper `TeamsCheckpoint`s.
- ACL → `ExternalAccess` mapping is intentionally left best-effort
(`load_from_checkpoint_with_perm_sync` delegates to the standard load)
because the sync pipeline does not currently persist `ExternalAccess`.
- `rag/svr/sync_data_source.py`
- Implemented `Teams._generate()` using the existing
`CheckpointOutputWrapper` pattern (same shape as Confluence/Jira/Google
Drive), supporting full reindex and incremental polling from
`poll_range_start`.
- `TeamsConnector` is already exported from
`common/data_source/__init__.py`.
**Frontend (`web/`)**
- Enabled the `TEAMS` data-source enum and added its form fields
(`tenant_id`, `client_id`, `client_secret`), default values, display
metadata, and a Teams icon.
- Added `teamsDescription` / `teamsTenantIdTip` to `en.ts` and `zh.ts`.
**Tests**
- `test/unit_test/data_source/test_teams_connector_unit.py`: mock-based
unit tests covering credential loading (incomplete creds raise, happy
path sets the Graph client, fetch-without-creds raises), post/reply
flattening (incl. the HTML vs text extension), incremental
`lastModifiedDateTime` filtering, and slim-doc listing. All 6 pass;
`ruff check` is clean.
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
## Summary
- Harden `NewVoyageModel` to avoid panics when `http.DefaultTransport`
is a custom non-`*http.Transport` RoundTripper.
- Fallback to a safe transport (`ProxyFromEnvironment`) while preserving
existing pooling/timeout settings.
- Add `TestVoyageNewModelWithCustomDefaultTransport` regression
coverage.
Co-authored-by: Cursor <cursoragent@cursor.com>
## Summary
- Harden `NewLongCatModel` to avoid panics when `http.DefaultTransport`
is a custom non-`*http.Transport` RoundTripper.
- Fallback to a safe transport (`ProxyFromEnvironment`) while preserving
existing pooling/timeout settings.
- Add `TestLongCatNewModelWithCustomDefaultTransport` regression
coverage.
Co-authored-by: Cursor <cursoragent@cursor.com>
### What problem does this PR solve?
implement delete, rebuild api for connector
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
### What problem does this PR solve?
Closes#15187.
RAGFlow shipped a Slack connector
(`common/data_source/slack_connector.py`) but it was never usable:
`Slack._generate()` in the sync worker was a `pass` stub, the
connector's document-generating code was incompatible with the current
data model,
and Slack was commented out of the data-source settings UI. As a result,
teams had no way to index Slack channels/threads into a knowledge base.
This PR completes the connector end to end.
**Backend**
- `common/data_source/slack_connector.py`
- Rewrote `thread_to_doc` to produce a blob-based `Document`
(`extension`/`blob`/`size_bytes`). The previous implementation built the
doc with a `sections=[...]` argument and omitted the now-required
`blob`/`extension`/ `size_bytes` fields, so it raised a validation error
against the current `Document` model. Thread messages are now cleaned
and flattened into a single UTF-8 text blob.
- Added `load_from_state()` / `poll_source(start, end)` generators. The
connector's checkpoint interface is a no-op stub, so both full and
incremental syncs run through a single channel-iterating generator built
on the existing module helpers (`get_channels`, `filter_channels`,
`get_channel_messages`, `_process_message`), with per-channel thread
de-duplication.
- `rag/svr/sync_data_source.py`
- Implemented `Slack._generate()`. Credentials are loaded via
`StaticCredentialsProvider` (the connector requires `slack_bot_token`
and does not support `load_credentials`). Supports full reindex and
incremental polling from `poll_range_start`, plus the optional channel
filter. Modeled on the Confluence/Dropbox wrappers.
- `SlackConnector` was already exported from
`common/data_source/__init__.py`.
**Frontend (`web/`)**
- Enabled the `SLACK` data-source enum and added its form fields (Slack
bot token + optional channel filter), default values, display metadata,
and a Slack icon.
- Added `slackDescription` / `slackBotTokenTip` / `slackChannelsTip`
strings to `en.ts` and `zh.ts`.
**Tests**
- `test/unit_test/data_source/test_slack_connector_unit.py`: unit tests
covering credential loading (`load_credentials` raises,
`set_credentials_provider` initializes clients, missing credentials
raises) and document generation (standalone message + flattened thread,
blob/extension/size_bytes/metadata, and the incremental poll time
window). All 5 pass; `ruff check` is clean.
Required Slack scopes: `channels:read`, `channels:history`,
`users:read`.
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
## Summary
- Harden the 302.AI model driver request validation and response parsing
paths.
- Add focused tests for chat request mode, model listing, malformed
provider responses, and input validation.
## What changed
- Validate API keys, model names, rerank queries, ASR file paths, OCR
inputs, parse URLs, task IDs, and model-list IDs before use.
- Keep chat and streaming methods from accepting conflicting `stream`
values in request payloads.
- Send `ListModels` as a bodyless GET and parse the response with typed
JSON structs instead of unchecked assertions.
- Remove raw SSE event logging from stream handling.
## Why
The driver could panic or send inconsistent requests when optional
config fields were nil, empty, malformed, or contradicted the method
path. This keeps provider-driver behavior explicit while preserving the
existing supported 302.AI flows.
Closes#14736
### What problem does this PR solve?
Fixes the `/user/me` response so it returns the current user's model
settings correctly.
### Type of change
- Added model settings data to the `/user/me` response.
- Kept the response structure compatible with existing user profile
fields.
- Avoided changing unrelated user/session behavior.
## Summary
- Add Go REST support for `GET /api/v1/system/healthz`.
- Return Python-compatible `ok`/`nok` dependency fields for DB, Redis,
document engine, and storage.
- Return HTTP 200 only when all checks pass; otherwise return HTTP 500
with `_meta` failure details.
- Add focused service coverage for the unhealthy dependency response
when Go dependencies are not initialized.
## Scope
This is a small, isolated slice of #15240. It avoids current open
connector PRs (#15274, #15300, #15265, #15264), tenant/member PRs
(#15295, #15301, #15276), MCP PRs (#15281, #15253, #15254, #15260,
#15261, #15262), and the memory-message PR (#15256).
Refs #15240
### What problem does this PR solve?
Closes#15189.
RAGFlow shipped a SharePoint connector stub
(`common/data_source/sharepoint_connector.py`) whose document-loading
methods all returned `[]`, `SharePoint._generate()` was a `pass`, and
SharePoint was commented out of the data-source settings UI. As a result
there was no way to index files stored in SharePoint document libraries.
This PR implements the connector end to end on top of Microsoft Graph
(Office365-REST-Python-Client).
**Backend**
- `common/data_source/sharepoint_connector.py`
- `load_credentials()` now builds the Graph client using an MSAL
client-credentials **token callback** — the form `GraphClient` actually
expects. (The previous stub passed a raw access-token string to
`GraphClient(...)`, which is not how that client is driven.) Token
acquisition is lazy, so credential loading does no network call.
- `validate_connector_settings()` resolves the configured site via
Graph.
- `load_from_checkpoint()` is now a generator that enumerates every
document library under the site, walks folders depth-first, downloads
each file, and yields blob-based `Document` objects (`extension` /
`blob` / `size_bytes` / `doc_updated_at`). Incremental syncs are bounded
by file `lastModifiedDateTime`. Per-file errors are surfaced as
`ConnectorFailure` rather than aborting the run.
- `retrieve_all_slim_docs_perm_sync()` yields id-only `SlimDocument`
batches (no downloads) and the checkpoint helpers return proper
checkpoints.
- ACL → `ExternalAccess` mapping is intentionally left best-effort
(`load_from_checkpoint_with_perm_sync` delegates to the standard load)
because the sync pipeline does not currently persist `ExternalAccess`;
this can be extended once that plumbing exists.
- `rag/svr/sync_data_source.py`
- Implemented `SharePoint._generate()` using the existing
`CheckpointOutputWrapper` pattern (same shape as Confluence/Jira/Google
Drive), supporting full reindex and incremental polling from
`poll_range_start`.
- `SharePointConnector` is already exported from
`common/data_source/__init__.py`.
**Frontend (`web/`)**
- Enabled the `SHAREPOINT` data-source enum and added its form fields
`site_url`, `tenant_id`, `client_id`, `client_secret`), default values,
display metadata, and a SharePoint icon.
- Added `sharepointDescription` / `sharepointSiteUrlTip` to `en.ts` and
`zh.ts`.
**Tests**
- `test/unit_test/data_source/test_sharepoint_connector_unit.py`:
mock-based unit tests covering credential loading (incomplete creds
raise, happy path sets the Graph client, fetch-without-creds raises),
drive traversal + file download, incremental `lastModifiedDateTime`
filtering, and slim-doc listing. All 6 pass; `ruff check` is clean.
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
### What problem does this PR solve?
extend restful api suite
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
- [x] Other (please describe): test
### What problem does this PR solve?
- Add Go implementation parity for `PATCH /api/v1/users/me`.
- This updates the Go user settings endpoint to match the Python
behavior for updating the current user's profile settings.
### Changes
- Route `PATCH /api/v1/users/me` through the authenticated current user
from middleware.
- Add `password` and `new_password` support to `UpdateSettingsRequest`.
- Prevent `email` from being updated through this endpoint, matching the
Python blacklist behavior.
- Support updating:
- `nickname`
- `avatar`
- `language`
- `color_schema`
- `timezone`
- `password`
- Align password handling with Python:
- invalid plaintext password payload returns `CodeExceptionError`
- wrong old password returns `Password error!`
- successful update returns `{ code: 0, data: true, message: "success"
}`
### Test
Tested manually with Python and Go backends using the same request
bodies:
- `PATCH /api/v1/users/me` with nickname/timezone update
- plaintext password payload returns Python-compatible `Incorrect
padding`
- wrong old password returns `Password error!`
### What problem does this PR solve?
1. Break huge function into smaller pieces
2. Add unit test for the smaller pieces function
3. Layer-ed design
a. infra layer - task_context.py, recording_context.py,
write_operation_interceptor.py, ...
b. service layer - *_service.py
c. business layer - task_handler.py
4. Default behavior: use "refactor-ed version" - can switch to original
version by change env variable
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
- [x] Refactoring
- [x] Performance Improvement
---------
Co-authored-by: Liu An <asiro@qq.com>
Co-authored-by: Zhichang Yu <yuzhichang@gmail.com>
Wrap per-statement execution in both the generic and IBM DB2 loops so a
failing statement reports a friendly "SQL Execution Failed" message and
continues, instead of letting a raw driver exception abort the node and
discard results from statements that already succeeded.
Rolls back after a failure so PostgreSQL's aborted-transaction state
does not cascade into every subsequent statement in the batch.
### What problem does this PR solve?
Closes#14737
The **ExeSQL** agent node splits its input on `;` and runs each
statement in a loop. Both execution loops — the generic one
(`cursor.execute`) and the IBM DB2 one (`ibm_db.exec_immediate`) — were
wrapped only in a `try/finally` for resource cleanup, with **no
`except`** around statement execution.
As a result, when any single statement failed (e.g. the reporter's MSSQL
`('42S02', "[42S02] ... 对象名 'ASSET_AUDIT' 无效")`):
- The raw, unformatted driver exception bubbled up and the node failed
with an ugly `_ERROR` instead of friendly information.
- **The whole node aborted** — results from statements that had already
succeeded were discarded, and the remaining statements in the batch
never ran. The reporter confirmed this was the real pain point: *"after
reporting an exception, the previous normal query cannot be executed
properly … Do not interrupt the workflow for any issues."*
Connection-level failures were already wrapped with a friendly
`"Database Connection Failed!"` prefix — only per-statement execution
errors were missed.
**This PR** wraps per-statement execution in `try/except` in both loops.
A failing statement now:
- records a friendly `SQL Execution Failed: <sql>\n<error>` entry into
the `json` and `formalized_content` outputs (the actual DB error is kept
so the user can see *what* failed), and
- `continue`s to the next statement — so earlier results survive and
later statements still run.
After a failure in the generic loop, the connection is rolled back so
PostgreSQL's aborted-transaction state does not cascade into every
subsequent statement in the batch. The node returns normally (no
`_ERROR` raised), so the agent workflow proceeds instead of halting.
Connection failures remain fatal (correct — nothing can run without a
connection). The pre-existing `break` on `cursor.rowcount == 0` is
intentionally left unchanged; it is out of scope for this fix.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
### What problem does this PR solve?
implement create_connector API
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
### What problem does this PR solve?
Fix: Uploading TSV format documents to the knowledge base did not
generate any error messages.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
## Summary
- centralize TokenHub chat request validation for chat and streaming
calls
- reject blank TokenHub model names before sending provider requests
- send TokenHub model listing requests as bodyless GET requests
## What changed
- Added shared TokenHub chat request validation for API key, model name,
and messages.
- Updated `ListModels` to call `GET /models` without a request body.
- Added focused tests for blank model names and accidental GET request
bodies.
- Replaced an httptest handler callback `t.Fatalf` with `t.Errorf` plus
an HTTP error and return.
## Why
TokenHub chat requests should fail locally for invalid model names
instead of sending avoidable malformed requests upstream. Model listing
should also match normal GET semantics and avoid sending an empty JSON
body.
Closes#14736
Co-authored-by: Jin Hai <haijin.chn@gmail.com>
### What problem does this PR solve?
As title
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
- [x] New Feature (non-breaking change which adds functionality)
- [x] Refactoring
## Summary
- Add Go REST support for `GET /api/v1/connectors/:connector_id`.
- Reuse the Python API behavior by returning the connector only when the
current user can access its tenant.
- Add focused handler coverage for success and unauthorized responses.
Co-authored-by: Jin Hai <haijin.chn@gmail.com>
### What problem does this PR solve?
`ReplicateModel.Rerank` in `internal/entity/models/replicate.go` was a
`"replicate, no such method"` stub. The chat path landed in #14958 and
the embed path in #15073; rerank is the last major retrieval surface
still missing on this provider.
Until this PR, a tenant who selected a Replicate reranker model got the
sentinel error on every rerank call.
Co-authored-by: sxxtony <sxxtony@users.noreply.github.com>
Co-authored-by: Jin Hai <haijin.chn@gmail.com>
### What problem does this PR solve?
Closes#15090.
Adds GiteeAI support to the Go model-provider layer so GiteeAI chat
models can be routed through the Go API server using the same
OpenAI-compatible chat, streaming, model listing, and connection-check
flow used by other SaaS providers.
GiteeAI is implemented as a separate provider from the existing `gitee`
provider.
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
## Summary
- Added a GiteeAI Go model driver.
- Added the GiteeAI provider catalog with default base URL
`https://ai.gitee.com/v1`.
- Registered `giteeai` in the model factory separately from `gitee`.
- Added focused provider tests for sync chat, streaming chat, model
listing, connection checks, base URL override, SSE parsing, `[DONE]`
handling, and unsupported methods.
## What changed
- Implemented `ChatWithMessages` for `POST /chat/completions`.
- Implemented `ChatStreamlyWithSender` with SSE parsing, `delta`
extraction, `finish_reason`, `[DONE]`, and `<think>` tag handling.
- Implemented `ListModels` for `GET /models`.
- Implemented `CheckConnection` by delegating to `ListModels`.
- Returned standard `no such method` errors for unsupported embedding,
rerank, image-to-text, ASR, and TTS paths.
## Tests
```bash
go test -vet=off ./internal/entity/models -run 'TestGiteeAI' -count=1
go test -vet=off ./internal/entity -run 'Test.*Provider|Test.*Model' -count=1
```
---------
Co-authored-by: Jin Hai <haijin.chn@gmail.com>
### What problem does this PR solve?
This PR adds Qiniu provider integration for the Go model driver layer in
RAGFlow.
Supported capabilities:
- [X] Chat
- [X] Think Chat
- [X] Stream Chat
- [X] Stream Think Chat
- [X] Model listing
- [X] Provider configuration and factory registration
Verified examples from the CLI:
```
login user '***' password '***';
ADD PROVIDER 'qiniu';
CREATE PROVIDER 'qiniu' INSTANCE 'test' KEY '***';
chat with 'deepseek/deepseek-v3.1-terminus-thinking@test@qiniu' message
'hello';
think chat with 'deepseek/deepseek-v3.1-terminus-thinking@test@qiniu'
message 'hello';
stream chat with 'deepseek/deepseek-v3.1-terminus-thinking@test@qiniu'
message 'hello, what are you';
stream think chat with
'deepseek/deepseek-v3.1-terminus-thinking@test@qiniu' message 'hello,
what are you';
stream think chat with 'qwen3-max-2026-01-23@test@qiniu' message 'hello,
what are you';
LIST MODELS FROM 'qiniu' 'test';
```
### Type of change
- [X] New Feature
- [X] Provider integration
## Summary
- add the VolcEngine `models` URL suffix used by the existing Go
`ListModels` implementation
- return a clear error when the VolcEngine models suffix is missing
- add focused VolcEngine model-listing regression tests
## What changed
- Added `url_suffix.models` to `conf/models/volcengine.json`.
- Normalized the configured models suffix before building the request
URL.
- Covered config loading, successful model listing, upstream errors, and
missing suffix handling.
## Why
`VolcEngine.ListModels` already builds requests from `URLSuffix.Models`,
but the bundled VolcEngine config did not define that suffix. That left
the model-listing path unable to call the documented `/models` endpoint
from the existing provider config.
Fixes#14701
Co-authored-by: Jin Hai <haijin.chn@gmail.com>
### What problem does this PR solve?
extend restful api suite
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
- [x] Other (please describe): test
### What problem does this PR solve?
Fix: The Creativity parameter of chat was not saved.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
## Summary
- route hosted MinerU.Net and PaddleOCR.Net provider names to their
existing Go drivers
- add regression coverage for loading the hosted OCR provider configs
through ProviderManager
## What changed
- Added canonical provider-name aliases for the hosted OCR provider
display names.
- Covered both bundled configs with a focused provider-manager test.
## Why
The hosted provider configs use display names with `.Net`, while model
factory dispatch lowercases the provider name. Without aliases, those
configs fall through to `DummyModel` instead of using the existing
MinerU and PaddleOCR drivers.
---------
Co-authored-by: Jin Hai <haijin.chn@gmail.com>
## Summary
- Add LongCat model-list support through the documented
OpenAI-compatible models endpoint.
## What changed
- Add the LongCat `models` URL suffix for `/openai/v1/models`.
- Implement `ListModels` for the LongCat Go driver.
- Delegate `CheckConnection` to the lightweight model-list request.
- Add focused regression coverage for successful, malformed, oversized,
and missing-key responses.
## Why
LongCat documents a models endpoint under the OpenAI-compatible API
surface, but the Go driver still returned `no such method` for model
listing and connection checks.
## Validation
- `go test ./internal/entity/models -run TestLongCat -count=1`
- `go test -race ./internal/entity/models -run TestLongCat -count=1`
- `go test ./internal/entity -count=1`
- `git diff --check`
## Notes
- Related to the broader Go model provider tracking in #14736, but this
PR only handles LongCat model listing.
- `go test ./internal/entity/models -count=1` is currently blocked by an
unrelated Astraflow test panic outside this LongCat change.
---------
Co-authored-by: Jin Hai <haijin.chn@gmail.com>
## Summary
- add the xAI `models` URL suffix used by the existing Go `ListModels`
implementation
- return a clear error when the xAI models suffix is missing
- add focused xAI model-listing and connection-check regression tests
## What changed
- Added `url_suffix.models` to `conf/models/xai.json`.
- Normalized the configured models suffix before building the request
URL.
- Covered config loading, successful model listing, upstream errors,
API-key validation, missing suffix handling, and `CheckConnection`
delegation.
## Why
`XAIModel.ListModels` already builds requests from `URLSuffix.Models`,
and `CheckConnection` delegates to that method. The bundled xAI config
did not define that suffix, which left the model-listing path unable to
call the provider `/models` endpoint from the existing provider config.
## Validation
- `go test ./internal/entity/models -run TestXAI -count=1`
- `go test ./internal/entity -count=1`
- `git diff HEAD~1..HEAD --check`
## Notes
- `go test ./internal/entity/models -count=1` currently fails in
unchanged Astraflow coverage: `TestAstraflowEmbedReturnsNoSuchMethod`
panics before reaching any xAI assertions.
---------
Co-authored-by: Jin Hai <haijin.chn@gmail.com>
### What problem does this PR solve?
- Update version tags in README files (including translations) from
v0.25.5 to v0.25.6
- Modify Docker image references and documentation to reflect new
version
- Update version badges and image descriptions
- Maintain consistency across all language variants of README files
### Type of change
- [x] Documentation Update
### What problem does this PR solve?
implement provider `OrcaRouter`
**The following functionalities are now supported:**
**Cohere:**
- [x] Chat / Think Chat / Stream Chat / Stream Think Chat
- [x] Model listing
- [x] TTS
- [ ] Balance
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
---------
Co-authored-by: Jin Hai <haijin.chn@gmail.com>
Closes#15040.
ModelScope was listed unchecked in the Go-rewrite tracker #14736 and
already had an llm_factories.json entry (tags: LLM) but no Go driver, so
the new Go API server could not route ModelScope instances. The Python
side has supported it through the OpenAI-compatible base at
rag/llm/chat_model.py:618 (ModelScopeChat), which requires a
user-supplied base URL and appends /v1.
This adds:
- internal/entity/models/modelscope.go: self-hosted OpenAI-compatible
driver with chat (sync + SSE stream with idle-timeout cancellation),
list_models, and check_connection. Auth header is optional, matching the
xinference pattern, so deployments without auth and auth-enabled
deployments both work. Base URL is normalized so users can configure
either the root endpoint or the /v1 endpoint.
- internal/entity/models/modelscope_test.go: 12 tests covering name, URL
normalization, factory routing, chat happy path / auth header /
reasoning_content extraction, stream happy path / stream=false rejection
/ idle cancellation, list_models + check_connection, missing-base-URL
clear error, and the no-such-method sentinels.
- conf/models/modelscope.json: shipped config (class: "local",
url_suffix v1/chat/completions and v1/models).
- internal/entity/models/factory.go: case "modelscope" →
ModelScopeModel.
- internal/service/llm.go: ModelScope added to the selfDeployed map
alongside Ollama, Xinference, LocalAI, LM-Studio, GPUStack — the Python
side requires user-supplied URL with no default, so the Go side
classifies it the same way.
Follow-on issues will add Embed and Rerank, in line with how Novita,
NVIDIA, TogetherAI, and other providers landed method-by-method.
---------
Co-authored-by: Jin Hai <haijin.chn@gmail.com>
### What problem does this PR solve?
This PR adds HuaweiCloud provider integration in RAGFlow.
Supported capabilities:
- [x] Chat / Think Chat / Stream Chat / Stream Think Chat
- [x] Embedding
- [x] Rerank
- [x] Model listing
- [x] Provider connection checking
Verified examples from the CLI:
```
check instance 'test' from 'HuaweiCloud';
chat with 'deepseek-v4-flash@test@HuaweiCloud' message 'hello';
think chat with 'deepseek-v4-flash@test@HuaweiCloud' message 'hello';
stream chat with 'deepseek-v4-flash@test@HuaweiCloud' message 'hello';
stream think chat with 'deepseek-v4-flash@test@HuaweiCloud' message
'hello';
embed text 'what is rag' 'who are you' with 'bge-m3@test@HuaweiCloud'
dimension 1024;
rerank query 'what is rag' document 'rag is retrieval augmented
generation' 'rag need llm' 'famous rag
project includes ragflow' with 'bge-reranker-v2-m3@test@HuaweiCloud' top
3;
list supported models from 'HuaweiCloud' 'test';
LIST MODELS FROM 'HuaweiCloud' 'test';
```
### Type of change
- [x] New Feature
- [x] Provider integration
## Summary
- Wire the Go TokenHub provider through the model factory.
- Harden TokenHub request handling for chat, streaming, embeddings, and
model listing.
- Add focused TokenHub unit coverage for factory wiring and provider
behavior.
## Notes
- Refs #14736.
- Follows up #15159.
Co-authored-by: Jin Hai <haijin.chn@gmail.com>
## Summary
Closes#15165.
Implements the AWS Bedrock model provider for the Go API server, tracked
under #14736. Adds Converse + Converse-Stream chat and foundation-model
listing, with SigV4 signing over a hand-rolled `net/http` path that
matches the established pattern in `internal/entity/models/` (no new
direct `go.mod` deps).
## Linked tracker
Tracked under #14736 (Implement model providers of RAGFlow API server in
Go). Closes#15165.
### What problem does this PR solve?
extend restful api suite
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
- [x] Other (please describe): test
### What problem does this PR solve?
Fix: Fixed metadata issue
- The dataset's built-in metadata is now active, but it appears to be
disabled in the individual file configuration.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
### What problem does this PR solve?
Creating or updating an agent via `POST /api/v1/agents` and `PUT
/api/v1/agents/{agent_id}` did not persist `canvas_type` because the
handler `req` dict never assigned the field before
`UserCanvasService.save` / `update_by_id`.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
- [ ] New Feature (non-breaking change which adds functionality)
- [ ] Documentation Update
- [ ] Refactoring
- [ ] Performance Improvement
- [ ] Other (please describe):
Co-authored-by: Cursor <cursoragent@cursor.com>
### What problem does this PR solve?
The Go DeepInfra driver returned a stub error for `Rerank()` even though
DeepInfra serves reranker models at `POST /v1/inference/{model}` with
`query`, `documents`, and a `scores[]` response.
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
Co-authored-by: Cursor <cursoragent@cursor.com>
### What problem does this PR solve?
Add a Go driver for **FuturMix** (https://futurmix.ai/docs), one of the
unchecked providers on the umbrella tracking issue #14736. FuturMix is
documented as an "OpenAI-compatible API" aggregator over Claude / GPT /
Gemini / DeepSeek (~22 models per their `/models` page).
Until this PR, a tenant who configured `futurmix` as a model provider in
the Go layer fell through to the default branch of
`internal/entity/models/factory.go` and got the dummy driver.
---------
Co-authored-by: sxxtony <sxxtony@users.noreply.github.com>
Co-authored-by: Jin Hai <haijin.chn@gmail.com>
### What problem does this PR solve?
Closes#15167.
The Baidu Go provider advertises OCR support through
`paddleocr-vl-0.9b`, but `BaiduModel.OCRFile` dereferenced required
inputs before validating them. Calling OCR with a missing API config,
API key, or model name could panic instead of returning a normal error.
This PR adds explicit input validation for those required values.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
---------
Co-authored-by: Jin Hai <haijin.chn@gmail.com>
### What problem does this PR solve?
Closes#15142.
ZhipuAI lists `glm-ocr` as an OCR model, but the Go driver still
returned `no such method` from `OCRFile`. This wires the advertised
model to Z.AI's documented `layout_parsing` endpoint and returns the
`md_results` Markdown output through the existing `OCRFileResponse.Text`
field.
This PR also adds focused tests for URL input, raw file-content base64
input, and validation errors.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
- [ ] New Feature (non-breaking change which adds functionality)
- [ ] Documentation Update
- [ ] Refactoring
- [ ] Performance Improvement
- [ ] Other (please describe):
### Test
- [x] `go test -vet=off ./internal/entity/models -run
'TestZhipuAIOCRFile'`
### What problem does this PR solve?
extend restful api suite
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
- [x] Other (please describe): test
### What problem does this PR solve?
Fix [Bug]: Save parser configs in dataset configuration page is not
working #15175
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
## Summary
Fixes the confirmed asyncio anti-patterns from #14755. Only the three
verified bugs are addressed; patterns already correctly using
`asyncio.new_event_loop()` in a fresh thread are left untouched.
### Changes
**`api/apps/restful_apis/tenant_api.py` — fire-and-forget
`send_invite_email`**
`asyncio.create_task()` was called without storing the `Task` reference.
CPython's GC can collect an unfinished task, silently cancelling it and
swallowing exceptions. Fixed by storing the task in a module-level
`_background_tasks: set[Task]` with a `done_callback` to discard it on
completion — the standard Python idiom for safe background tasks.
**`api/apps/restful_apis/agent_api.py` — fire-and-forget
`background_run`**
Same root cause in the webhook "Immediately" execution path. Same fix
applied.
**`rag/llm/chat_model.py` (`LocalLLM._stream_response`) —
`asyncio.get_event_loop()` on running loop**
`asyncio.get_event_loop()` returns Quart's running event loop when
called from an async context.
Calling `loop.run_until_complete()` on it raises `RuntimeError`.
Replaced with `asyncio.new_event_loop()` so the generator
uses a dedicated fresh loop, closed in a `finally` block.
## What was NOT changed
- `llm_service._sync_from_async_stream` and
`evaluation_service._sync_from_async_gen`: both already correctly use
`asyncio.new_event_loop()` inside a fresh thread.
- `llm_service._run_coroutine_sync`: only caller is `rag/app/resume.py`
(sync context), so `thread.join()` is correct there.
- `requests` in agent tools: sync methods dispatched through thread
pools; httpx migration is a separate, larger refactor.
## Test plan
- [ ] Invite a team member and confirm the email is sent with no task
warnings in logs.
- [ ] Trigger a webhook agent in "Immediately" mode; confirm canvas
state is persisted after background run.
- [ ] Verify `LocalLLM` (Jina backend) chat and streaming work
end-to-end.
Closes#14755
---------
Co-authored-by: Zhichang Yu <yuzhichang@gmail.com>
### What problem does this PR solve?
Fix: The prompt variable for the agent operator disappears after input.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
### What problem does this PR solve?
extend restful api suite
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
- [x] Other (please describe): test
### What problem does this PR solve?
Feat: Enable agent messages to display base64 images
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
### What problem does this PR solve?
IDK how to implement **`Ollama`** on #14580 but it's totally wrong.
This is the rewrite version for **`Ollama`**
**Verified from CLI**
```
# Embed
RAGFlow(user)> embed text 'what is rag' 'who are you' with 'nomic-embed-text:latest@test12@ollama' dimension 1024;
+-----------+-------+
| dimension | index |
+-----------+-------+
| 768 | 0 |
| 768 | 1 |
+-----------+-------+
# Chat
RAGFlow(user)> think chat with 'qwen3:0.6b@test12@ollama' message 'who r u'
Thinking: Okay, the user asked, "Who r u?" I need to respond appropriately. First, I should acknowledge their question. Since I'm an AI, I don't have a physical form, but I can confirm that I'm a large language model. I should keep the response friendly and offer help. Let me make sure I'm not making up any information and that the response is natural. Also, I should check for any typos and ensure clarity. Alright, that should cover it.
Answer: I'm an AI language model, and I don't have a physical form. However, I can tell you that I'm designed to assist with questions and tasks. How can I help you today?
Time: 2.914285
RAGFlow(user)> stream think chat with 'qwen3:0.6b@test12@ollama' message 'who r u'
Thinking: , the user asked, "Who are you?" I need to respond appropriately. Since I'm an AI assistant, I should mention that I don't have a physical form or a mind. I should also clarify that I can help with various tasks like answering questions or providing information. It's important to keep the response friendly and informative while maintaining the correct tone.
Answer: don't have a physical form or a mind, but I'm here to help with your questions or tasks! What can I do for you today?
Time: 1.740047
# LisyModels
RAGFlow(user)> list supported models from 'ollama' 'test12'
+-------------------------+
| model_name |
+-------------------------+
| nomic-embed-text:latest |
| qwen3:0.6b |
+-------------------------+
```
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
- [x] Refactoring
### What problem does this PR solve?
Fix: Replace the red highlight at the top of the PDF document with
yellow.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
Follow on PR #15146 to reslove the backwad compatability issue.
1. /agents/<attachment_id>/download ->
/agents/attachments/<attachment_id>/download
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
## Summary
This change fixes ingestion quality issues where MinerU parser output
may contain HTML fragments (for example, table-related tags like `<tr>`,
`<td>`, `<br>`), which were previously passed directly into
chunking/tokenization and degraded chunk quality.
The fix adds a sanitization step in the MinerU parser path so parsed
sections are normalized to clean text before chunking.
## Change Type (select all)
- [x] Bug fix
- [x] Ingestion pipeline improvement
- [x] Parser/chunking quality fix
## Related Issue
- https://github.com/infiniflow/ragflow/issues/14831
### What problem does this PR solve?
This PR improves the table upload flow for CSV/Excel files by allowing
table column role configuration at upload time.
Previously, users had to:
1. Upload and parse a table file.
2. Open parser settings and manually set table column roles.
3. Re-parse the file for the roles to take effect.
This was inefficient and required an unnecessary second parse.
With this change:
1. When the knowledge base uses table parsing, the upload dialog
extracts CSV/Excel headers client-side.
2. Users can choose Auto mode or Manual mode.
3. In Manual mode, users can assign per-column roles before upload.
4. The selected parser config is sent with the upload request and
applied server-side during document creation.
Result: configured table column roles are applied from the first parse.
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
Co-authored-by: Ahmad Intisar <ahmadintisar@Ahmads-MacBook-M4-Pro.local>
Declare doc_id, filename, mime_type, and size as separate outputs on the
Document Generation component so downstream nodes (e.g., the Code
component) can consume them via the variable picker. The existing
download JSON blob is preserved unchanged for the Message component's
download-chip rendering.
### What problem does this PR solve?
The Document Generation component previously exposed only a single
`download` output —
a JSON-encoded blob containing the file's `doc_id`, `filename`,
`mime_type`, `size`,
and base64 payload. On top of that, the variable picker actively hides
this `download`
entry from every consumer except the Message component (because the
embedded base64 is
too heavy to splat into arbitrary downstream nodes).
The combined effect: users wiring the Doc Generator's output into a Code
component had
no way to retrieve basic file info such as `file_name` or `doc_id` from
the picker,
blocking workflows that need to post-process the generated file (e.g.,
registering it
elsewhere, custom delivery, follow-up API calls).
This PR declares `doc_id`, `filename`, `mime_type`, and `size` as
**discrete outputs**
on the Document Generation component, alongside the existing `download`
blob. The new
fields:
- Appear in the variable picker for **all** downstream nodes, including
the Code
component, so users can bind them directly to script arguments.
- Are cheap scalars only — no base64 payload leaks into other
components.
- Leave the existing `download` JSON blob completely untouched, so the
Message
component's download-chip rendering (which parses that blob via
`_is_download_info`)
keeps working with no behavior change.
Changes:
- `agent/component/docs_generator.py` — declare the four new outputs in
`DocGeneratorParam` and emit them via `set_output(...)` in `_invoke`.
- `web/src/pages/agent/constant/index.tsx` — extend
`initialDocGeneratorValues.outputs`
with the new keys.
- `web/src/pages/agent/form/doc-generator-form/index.tsx` — mirror the
new outputs in
the zod schema so the form is valid.
No changes needed to the picker's existing `download`-hiding filter — it
matches only
on the literal output name `download`, so the new metadata entries fall
through
naturally.
Reported in: https://github.com/infiniflow/ragflow/issues/14461.
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
### What problem does this PR solve?
Implement embed for Tencent Hunyuan
**Verified from CLI**
```
RAGFlow(user)> embed text 'what is rag' 'who are you' with 'hunyuan-embedding@test1@hunyuan' dimension 16;
+-----------+-------+
| dimension | index |
+-----------+-------+
| 1024 | 0 |
| 1024 | 1 |
+-----------+-------+
```
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
- [x] Refactoring
### What problem does this PR solve?
1. Fix /chat/completions to send only the latest message
2. Allo chat stream=False
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
### What problem does this PR solve?
Go: implement provider: PaddleOCR_Local
**Verified from CLI**
```
RAGFlow(user)> ocr with 'PaddleOCR-VL@test@paddleocr_local' file './internal/test1.jpg'
+----------------------+
| text |
+----------------------+
| ## Parallel to these |
+----------------------+
```
### Type of change
- [X] Bug Fix (non-breaking change which fixes an issue)
- [X] New Feature (non-breaking change which adds functionality)
- [X] Refactoring
## Summary
- Adds a `Hunyuan` Go driver so the new API server can route Tencent
Hunyuan chat instances (registered in `conf/llm_factories.json:3830` as
`Tencent Hunyuan`). Follows the same SaaS-driver shape used for
Astraflow, Avian, Novita, TogetherAI, Replicate, DeepInfra, Upstage, and
LongCat.
Closes#15087
---------
Co-authored-by: Jin Hai <haijin.chn@gmail.com>
### What problem does this PR solve?
Fix /chat/completions not aware of conversation_id
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
### What problem does this PR solve?
Closes#15102.
OpenAI's Go provider config advertises `whisper-1` as ASR and `tts-1` as
TTS, but the Go driver returned `openai, no such method` for both audio
paths and did not define `url_suffix.asr` / `url_suffix.tts`.
This PR:
- adds OpenAI audio URL suffixes for `audio/transcriptions` and
`audio/speech`
- implements non-streaming `TranscribeAudio` using multipart form
uploads
- implements non-streaming `AudioSpeech` using the OpenAI speech JSON
request shape
- keeps streaming TTS explicitly unsupported instead of sending binary
audio through the text SSE sender
- adds focused tests for config coverage, ASR/TTS request shape,
required TTS voice validation, and unsupported streaming TTS
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
---------
Co-authored-by: Jin Hai <haijin.chn@gmail.com>
### What problem does this PR solve?
Fix: /openai/<chat_id>/chat/completions not aware of session_id
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
### What problem does this PR solve?
Go: implement reasoning_chat, TTS, ASR for Groq
**Verify from CLI**
```
RAGFlow(user)> think chat with 'qwen/qwen3-32b@test@groq' message 'who r u'
Thinking: Okay, the user asked, who r u. I need to determine what the user is asking. They may be asking about my identity. I should introduce my name and basic functions. The user might want to know what I can do, so I should list some common use cases, such as answering questions, creating writing, coding, and expressing opinions. The user may be curious about how they can interact with me, so they can be advised to ask any questions or provide instructions. Keep your answers conversational, avoid overly technical terms, keep answers concise, and encourage further interaction. Check if there's any ambiguity in the answer and make sure it's accurate and meets the user's needs. Also consider if there are other aspects the user may be interested in, such as my training data or performance. But since the question is basic, I'll focus on the essentials first and invite the user to ask more. In summary, respond to the user's questions by introducing yourself, your functions, and encouraging further interaction.
Answer: Hello! I'm Qwen. I am a large-scale language model developed by Tongyi Lab, designed to assist you in various ways, such as answering questions, creating text, logical reasoning, programming, and more. I aim to provide clear, accurate, and helpful information and support. How can I assist you today? Feel free to ask any questions or give me tasks! 😊
Time: 2.199908
RAGFlow(user)> stream think chat with 'openai/gpt-oss-20b@test@groq' message 'who r u'
Thinking: to respond politely.
Answer: ’m ChatGPT—an AI language model created by OpenAI. I’m here to answer questions, offer explanations, and help with a wide range of topics. How can I assist you today?
RAGFlow(user)> tts with 'canopylabs/orpheus-arabic-saudi@test@groq' text 'hello? show yourself' play format 'wav' param '{"voice": "fahad"}'
SUCCESS
RAGFlow(user)> asr with 'whisper-large-v3-turbo@test@groq' audio './internal/test.wav' param '{"language": "en"}'
+----------------------------------------------------------------------------------------------------------------------+
| text |
+----------------------------------------------------------------------------------------------------------------------+
| The examination and testimony of the experts enabled the Commission to conclude that five shots may have been fired |
+----------------------------------------------------------------------------------------------------------------------+
```
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
### What problem does this PR solve?
Closes#15088.
Adds Groq support to the Go model-provider layer so Groq instances can
be routed through the Go API server with the same OpenAI-compatible
chat, streaming, model listing, and connection-check flow used by other
SaaS providers.
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
## Summary
- Added a Groq Go model driver.
- Added the Groq provider catalog and default OpenAI-compatible API URL.
- Registered Groq in the model factory.
- Added focused provider tests.
## What changed
- Implemented chat completions, SSE streaming, ListModels, and
CheckConnection for Groq.
- Covered request shape, stream termination, reasoning fallback, model
listing, custom base URLs, safe transport setup, and unsupported
methods.
- Kept the provider catalog scoped to current Groq chat-capable model
IDs.
- Cleaned up pre-existing Go model package validation blockers so the
package can be tested normally with vet enabled.
## Why
The existing Python/provider catalog path includes Groq, but the Go
model-provider layer did not have a Groq driver, so the Go API server
could not instantiate or use Groq as requested in #15088.
## Notes
The model package now validates without disabling vet.
---------
Co-authored-by: Jin Hai <haijin.chn@gmail.com>
### What problem does this PR solve?
move agent attachment download api to the correct route and update
frontend callers
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
### Notes
- Move the attachment download endpoint from document routes to agent
routes.
- Update frontend download callers to use the agent attachment endpoint.
- Reuse the shared file response header helper instead of duplicating it
in `agent_api.py`.
## Summary
- Adds a `TokenPony` Go driver so the new API server can route TokenPony
chat instances, matching the existing Python `TokenPonyChat`
(`rag/llm/chat_model.py:1210`). Follows the same SaaS-driver shape used
for Astraflow, Avian, Novita, TogetherAI, Replicate, DeepInfra, Upstage,
and LongCat.
Closes#15086
---------
Co-authored-by: Jin Hai <haijin.chn@gmail.com>
## Summary
Implements the TODO in `evaluation_service.py`: **Track token usage** in
evaluation results.
## Changes
- **Import** `num_tokens_from_string` from `common.token_utils`
- **Prompt tokens**: Use the full prompt returned by `async_chat` when
available (includes system prompt + knowledge base + query), otherwise
fall back to the question token count
- **Completion tokens**: Count tokens in the generated answer
- **Storage**: Store `token_usage` as `{prompt_tokens,
completion_tokens, total_tokens}` in each `EvaluationResult` instead of
`None`
## Why
The evaluation pipeline previously saved `token_usage: None` for every
result. This change allows downstream consumers (e.g. evaluation
dashboards, cost tracking) to see approximate token usage per test case
using the same tokenizer (tiktoken cl100k_base) used elsewhere in
RAGFlow.
## Testing
- No new tests added; existing evaluation flow unchanged
- Token counting uses existing `num_tokens_from_string` utility
---------
Co-authored-by: kiannidev <kiannidev@users.noreply.github.com>
### What problem does this PR solve?
Fixes#15066
OpenRouter now exposes an official speech-to-text endpoint at `POST
/api/v1/audio/transcriptions`, but the Go model driver still returned
`openrouter, no such method` from `TranscribeAudio`. This left
OpenRouter ASR models unavailable through the Go API server even though
the provider already has OpenRouter audio support for TTS.
Related provider-tracking context: #14736
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
Co-authored-by: Jin Hai <haijin.chn@gmail.com>
### What problem does this PR solve?
The agent API currently does not pass chat_template_kwargs to the
underlying LLM call path, so clients cannot control template-level model
behavior (such as thinking-mode toggles) when invoking
/agents/chat/completion. This PR adds passthrough support for
chat_template_kwargs across agent execution flows (session and
non-session, streaming and non-streaming) by propagating it through
canvas runtime state and into LLM invocation kwargs. This addresses the
feature gap raised in [Issue
#14182](https://github.com/infiniflow/ragflow/issues/14182).
Closes#14182
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
Closes#14789
### What problem does this PR solve?
User API endpoints (`login`, `user_profile`, `user_add`,
`forget_reset_password`) were returning full user objects via
`to_json()` / `to_dict()`, which included sensitive fields like
`password` and `access_token` in the response body. This leaks
credentials to the client.
This PR adds a `to_safe_dict()` method on the `User` model that strips
sensitive fields (`password`, `access_token`) and replaces all affected
call sites to use it.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
### What problem does this PR solve?
1. Enhance retry and timeout, and adjust the default timeout
2. NER: spacy do not batch chunks
3. extract _has_cancel_and_exit
4. enhance log messages
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
- [x] Refactoring
Closes#4310
### What problem does this PR solve?
Issue #4310 requests practical examples for the RAGFlow SDK and HTTP API
to help developers get started faster. The existing `example/sdk/`
folder only contains `dataset_example.py`. This PR fills the remaining
gaps by adding examples for three key API areas not yet covered in
`main` or by other open PRs (#13904, #13284):
- **Chunk management** — add, list, update, delete, and retrieve chunks
within a dataset
- **Chat assistant** — create a chat assistant, open a session, send
messages (streaming and non-streaming), and clean up
- **Retrieval** — perform semantic retrieval across one or multiple
datasets
### Type of change
- [x] Documentation Update
- [x] New Feature (non-breaking change which adds functionality)
### What problem does this PR solve?
Closes#14865
`download_img` in `common/misc_utils.py` is used for OAuth avatar URLs.
The previous implementation called `async_request` from
`common.http_client`, which followed redirects without re-validating
each hop and did not apply the same SSRF protections as this path needs.
That made it possible to reach non-public or disallowed targets (for
example via redirects or unsafe URLs) when fetching avatars.
This change replaces that flow with an explicit, bounded fetch: each URL
(including every redirect target) is checked with
`common.ssrf_guard.assert_url_is_safe`, DNS is pinned with
`pin_dns_global`, `httpx` streams the body with `follow_redirects=False`
and a manual redirect loop (capped by
`RAGFLOW_OAUTH_AVATAR_MAX_REDIRECTS`), and total response size is capped
(`RAGFLOW_OAUTH_AVATAR_MAX_BYTES`). Timeouts, proxy, and user agent
align with `HTTP_CLIENT_*` env vars without importing `http_client`, so
lightweight tests stay simple.
Unit tests cover empty/None URLs, loopback, cloud metadata-style
addresses, and disallowed schemes so SSRF regressions are caught early.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
---------
Co-authored-by: Kevin Hu <kevinhu.sh@gmail.com>
### What problem does this PR solve?
This PR implements ASR and TTS support for the ZhipuAI Go driver.
The ZhipuAI model config already advertises `glm-asr-2512` as an ASR
model, but the Go driver returned `zhipu, no such method` from
`TranscribeAudio`. This adds the documented audio transcription endpoint
suffix and sends multipart transcription requests with `model`,
`stream=false`, and `file` fields.
Per maintainer review, this also adds the ZhipuAI TTS endpoint suffix
and implements `AudioSpeech` / `AudioSpeechWithSender` for `glm-tts`.
Closes#15133
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
- [x] New Feature (non-breaking change which adds functionality)
### What problem does this PR solve?
Closes#15089.
Adds PPIO support to the Go model-provider layer so PPIO instances can
be routed through the Go API server with the same OpenAI-compatible
chat, streaming, model listing, and connection-check flow used by other
SaaS providers.
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
## Summary
- Added a PPIO Go model driver.
- Added the PPIO provider catalog and default OpenAI-compatible API URL.
- Registered PPIO in the model factory.
- Added focused provider and provider-manager tests.
## What changed
- Implemented chat completions, SSE streaming, ListModels, and
CheckConnection for PPIO.
- Covered request shape, stream termination, reasoning fallback, model
listing, custom base URLs, safe transport setup, unsupported methods,
and provider config loading.
- Kept the provider catalog aligned with the existing RAGFlow PPIO
factory model set.
- Cleaned up pre-existing Go model package validation blockers so the
scoped provider tests can run normally with vet enabled.
## Why
The existing Python/provider catalog path includes PPIO, but the Go
model-provider layer did not have a PPIO driver, so the Go API server
could not instantiate or use PPIO as requested in #15089.
### What problem does this PR solve?
implement rerank, asr, tts for TogetherAI
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
### What problem does this PR solve?
1. update python version to 3.13
2. upgrade ormsgpack to 1.6.0
### Type of change
- [x] Refactoring
---------
Signed-off-by: Jin Hai <haijin.chn@gmail.com>
### What problem does this PR solve?
implement ASR and TTS for Xinference
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
- [x] Refactoring
## Summary
Fixes 10 unguarded `response.choices[0]` accesses that cause
`IndexError` or `AttributeError` when the LLM returns an empty `choices`
list — the scenario described in #14711.
- `rag/llm/cv_model.py`
- `rag/llm/chat_model.py`
Each access site is now guarded with:
```python
if not response.choices:
raise ValueError("LLM returned empty response")
```
## Verification
Detected and verified by [pact](https://github.com/qizwiz/pact) — a
sheaf-cohomological LLM contract checker using Z3 as a local theory
solver.
**pact sheaf-cohomological proof status after fix:**
| File | Ȟ¹ (after) | Z3 |
|------|-----------|-----|
| `rag/llm/cv_model.py` | 0 | UNSAT ✓ |
| `rag/llm/chat_model.py` | 0 | UNSAT ✓ |
All access sites proven safe (Z3 UNSAT certificate).
The checker was also used to verify the autogen streaming-None fix in
[microsoft/autogen#7711](https://github.com/microsoft/autogen/pull/7711).
## Test plan
- [ ] Existing test suite passes
- [ ] Manually test with a provider that returns empty `choices` under
load (e.g. Vertex AI)
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Signed-off-by: Jonathan Hill <jonathan.f.hill@gmail.com>
`GET /agents/<agent_id>/sessions/<session_id>` crashed with
`AttributeError: 'NoneType' object has no attribute 'to_dict'` when the
session lookup failed: `_, conv =
API4ConversationService.get_by_id(...)` returned `(False, None)`, then
`conv.to_dict()` was called unconditionally.
This is reachable in multi-instance deployments: the session row may not
yet be visible on the node servicing the immediate follow-up GET after a
session is created on a different node.
Add the same `if not exists` guard already used by every other call site
of `API4ConversationService.get_by_id` (see agent_api.py:1147,
sdk/session.py:179, conversation_service.py:248, canvas_service.py:323).
Closes#14989
### What problem does this PR solve?
_Briefly describe what this PR aims to solve. Include background context
that will help reviewers understand the purpose of the PR._
### Type of change
- [ ] Bug Fix (non-breaking change which fixes an issue)
- [ ] New Feature (non-breaking change which adds functionality)
- [ ] Documentation Update
- [ ] Refactoring
- [ ] Performance Improvement
- [ ] Other (please describe):
Replace the RuntimeError with a warning + first-address fallback so a
single email whose From header contains multiple addresses no longer
crashes the entire IMAP sync task. Also add regression tests covering:
- #14963: RFC 5322 quoted display names with commas (e.g. "Schlüter,
Sabine" <s@x>) parsed as one address, not two.
- #14964: multi-address headers warn instead of raising.
Closes#14964
Refs #14963
## Summary
- Bump pinned nginx in `Dockerfile` from `1.29.5-1~noble` (vulnerable)
to `1.31.0-1~noble` to remediate **CVE-2026-42945**.
## Root Cause
`Dockerfile:58` pinned `ARG NGINX_VERSION=1.29.5-1~noble`. Per the
official nginx security advisory, **CVE-2026-42945** is a buffer
overflow in `ngx_http_rewrite_module` triggered via the `rewrite` and
`set` directives, affecting nginx **0.6.27 through 1.30.0**. `1.29.5`
falls inside that range, so the shipped image is vulnerable.
References:
- nginx security advisories:
https://nginx.org/en/security_advisories.html
- Vendor advisory: https://my.f5.com/manage/s/article/K000161019
- Fixed versions: `1.31.0` (mainline) and `1.30.1` (stable)
## Fix
Single-line change in `Dockerfile:58`:
```diff
-ARG NGINX_VERSION=1.29.5-1~noble
+ARG NGINX_VERSION=1.31.0-1~noble
### What problem does this PR solve?
Fixes#14997.
RAPTOR builds on the Infinity backend have been broken since v0.25.2
introduced the `extra` field in code (`rag/svr/task_executor.py:1011`)
without declaring it in `conf/infinity_mapping.json`. Every RAPTOR job
fails with:
```
infinity.common.InfinityException: (3013, 'Fail to bind the expression: extra@src/planner/expression_binder_impl.cpp:99')
```
The auto-migration in
`common/doc_store/infinity_conn_base.py:_migrate_db()` adds any columns
it finds in the mapping JSON to existing tables — so the only thing
standing between users and a working RAPTOR build is that one missing
declaration. OceanBase, ES, and OpenSearch were unaffected because they
store `extra` as a native JSON type; only Infinity (which has a strict
`varchar`/`integer`/`float` schema) needed the addition.
### The fix
Two-part change:
1. **`conf/infinity_mapping.json`**: declare `"extra": {"type":
"varchar", "default": ""}`. On next startup, `_migrate_db()` adds the
column to all existing chunk tables — no manual DDL needed for upgrading
installations.
2. **`rag/utils/infinity_conn.py` `insert()`**: serialize the `extra`
dict to a JSON string at write time, since Infinity's `varchar` can't
store a Python dict directly. Modelled on the existing `chunk_data`
handling a few lines above.
The read path (`rag/utils/raptor_utils.py:_as_extra_dict`) already
normalises both dict and JSON-string inputs, so no read-side change is
needed. Other backends are untouched — `task_executor.py` still writes
the dict, and the OceanBase/ES/OpenSearch insert paths handle dicts
natively.
### Verification
Tested on a v0.25.4 deployment with the Infinity backend by applying the
same two changes via mounted-volume override:
- Confirmed `_migrate_db()` adds the `extra` column to all pre-existing
chunk tables on startup (column visible via Infinity's
`show_columns()`).
- Triggered RAPTOR builds on four datasets (~21k chunks total) via `POST
/api/v1/datasets/<id>/index?type=raptor`.
- All four progressed past the previously-failing
`get_raptor_chunk_methods()` call into actual entity-extraction and
clustering work without the (3013) error.
- GraphRAG builds (which can trigger the same path indirectly via
`task_executor.py:857`) also progressed cleanly.
### Type of change
- [X] Bug Fix (non-breaking change which fixes an issue)
### What problem does this PR solve?
`UpstageModel.ChatStreamlyWithSender` (in the driver merged via #14819)
only extracted `delta.content` from each SSE event. For the `solar-pro3`
reasoning family (and any future Upstage model that follows the same
wire shape), the chain-of-thought is streamed in a **separate
`delta.reasoning` field**, and the driver was silently dropping all of
it.
The non-streaming path already extracts `message.reasoning` into
`ChatResponse.ReasonContent` (added earlier in this PR's history), so
the same model produced **inconsistent behavior** between streaming and
non-streaming: a tenant calling `solar-pro3` with `reasoning_effort:
high` would see the reasoning trace if they used `ChatWithMessages` but
not if they used `ChatStreamlyWithSender`.
### Live evidence
Probed against `api.upstage.ai/v1/chat/completions` with `solar-pro3` +
`reasoning_effort: high` + `stream: true` (8000-token budget so the
reasoning has room to finish):
```
$ curl -sN -H "Authorization: Bearer <key>" -H "Content-Type: application/json" \
-X POST https://api.upstage.ai/v1/chat/completions \
-d '{"model":"solar-pro3","messages":[{"role":"user","content":"Compute 15% of 80."}],
"max_tokens":8000,"stream":true,"reasoning_effort":"high"}'
# across 168 SSE events:
# delta keys seen: [content reasoning role]
# delta.content total len: 121 chars (the visible answer)
# delta.reasoning total len: 159 chars (the chain-of-thought) <- driver dropped this
```
A representative event showing both fields side by side:
```json
data: {"choices":[{"index":0,"delta":{"reasoning":"15% = 0.15."}}]}
data: {"choices":[{"index":0,"delta":{"content":"15% of 80 is "}}]}
```
The 159 chars of reasoning were arriving on the wire and being thrown
away. `solar-pro2` was also probed (625 events); it does **not** emit
`delta.reasoning` — its reasoning is inlined into `delta.content` — so
this change is a no-op for it and for `solar-mini`.
### What this PR includes
- `internal/entity/models/upstage.go`: in the SSE scanner loop, extract
`delta.reasoning` before `delta.content` and forward each non-empty
chunk via the sender's second arg (the existing `reasonContent` channel
the non-stream path already populates).
The ordering contract is documented inline: reasoning chunks within a
single SSE event are emitted before content chunks, so a UI that pipes
both sees the chain-of-thought start before the answer for that token,
matching the wire order Upstage emits.
- `internal/entity/models/upstage_test.go`: three new tests pinning the
new behavior:
- `TestUpstageStreamExtractsReasoningDelta` — reasoning + content
forwarded to the right sender args; one-of invariant per call
- `TestUpstageStreamReasoningChunksArriveBeforeContent` — ordering
pinned within a single SSE event that carries both fields
- `TestUpstageStreamWithoutReasoningStillWorks` — regression net:
non-reasoning models (`solar-mini`, `solar-pro2`) continue to work; the
reason callback never fires
No interface change. No factory change. No config change.
### How was this tested?
```
$ go test -vet=off -run TestUpstage -count=1 -v ./internal/entity/models/...
... (existing tests 1..9 still pass) ...
=== RUN TestUpstageStreamExtractsReasoningDelta
--- PASS: TestUpstageStreamExtractsReasoningDelta (0.01s)
=== RUN TestUpstageStreamReasoningChunksArriveBeforeContent
--- PASS: TestUpstageStreamReasoningChunksArriveBeforeContent (0.01s)
=== RUN TestUpstageStreamWithoutReasoningStillWorks
--- PASS: TestUpstageStreamWithoutReasoningStillWorks (0.00s)
PASS
ok ragflow/internal/entity/models 0.034s
```
12/12 Upstage tests pass on go 1.25. `go build
./internal/entity/models/...` exits 0.
**Live integration test** (smoke test not committed) — the patched
driver was run directly against `api.upstage.ai/v1` with the same prompt
that produced the curl evidence above:
```
=== RUN TestUpstageStreamReasoningLiveSmoke
[OK] visible content: 50 chunks, 84 chars
[OK] reasoning: 39 chunks, 90 chars
content head 200: "\\(15\\% = \\frac{15}{100}=0.15\\).\n\n\\[\n0.15 \\times 80 = 12.\n\\]\n\n**15 % of 80 is 12.**"
reasoning head 200: "We need to compute 15% of 80. That's 0.15 * 80 = 12. So answer is 12. Provide explanation."
UPSTAGE STREAM REASONING SMOKE PASSED
--- PASS: TestUpstageStreamReasoningLiveSmoke (1.97s)
```
Before this fix, the same call would have produced **0 reasoning
chunks**. The 90 chars of reasoning that the patched driver now surfaces
are the chain-of-thought solar-pro3 emits when reasoning_effort is high.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
### What problem does this PR solve?
`MistralModel.ChatWithMessages` (in the driver merged via #14807)
assumes that `choices[0].message.content` from `/v1/chat/completions` is
always a string and falls through to `return nil, fmt.Errorf("invalid
content format")` on anything else.
That assumption breaks for the **magistral reasoning family**
(`magistral-small-*`, `magistral-medium-*`). When the model needs a
chain-of-thought to answer, Mistral returns `content` as a **structured
array of typed parts**:
```json
"content": [
{"type": "thinking",
"thinking": [{"type": "text", "text": "Combined speed is 150 mph. 300 / 150 = 2 hours."}],
"closed": true},
{"type": "text", "text": "They will meet after **2 hours**."}
]
```
Concretely, this is what the live API returns today (probed against
`api.mistral.ai/v1`):
```
$ curl -H "Authorization: Bearer <key>" -H "Content-Type: application/json" \
-X POST https://api.mistral.ai/v1/chat/completions \
-d '{"model":"magistral-medium-latest",
"messages":[{"role":"user","content":"two trains 60mph and 90mph, 300mi apart, when do they meet? step by step."}],
"max_tokens":1024}'
HTTP 200
{ "choices":[{"message":{
"role":"assistant",
"content":[
{"type":"thinking","thinking":[{"type":"text","text":"Okay, let's see..."}],"closed":true},
{"type":"text","text":"To determine when the two trains meet..."}
]}}] }
```
With the current driver, every call like that returns the generic
`"invalid content format"` error. Trivial prompts that happen to fit in
a string answer still succeed, so the breakage is **non-deterministic
from the tenant's POV**: same model, same provider, sometimes works,
sometimes 500s with no useful error.
A secondary issue: `conf/models/mistral.json` does not include any
magistral model. The picker hid the broken path, which is why this
wasn't caught during #14807's review.
### What this PR includes
- New helper `extractMistralContent(raw interface{}) (answer,
reasonContent string, err error)` in
`internal/entity/models/mistral.go`, which normalizes both shapes
Mistral can return:
- `string` → historical path. `Answer = content`, `ReasonContent = ""`.
Preserves behavior for every non-reasoning model (`mistral-large-*`,
`mistral-small-*`, `ministral-*`, `codestral-*`, `pixtral-*`,
`open-mistral-nemo`).
- `[]interface{}` → walk the parts. Concatenate every `{"type":"text",
"text":...}` part into `Answer`; concatenate the inner text inside every
`{"type":"thinking", "thinking":[...]}` part into `ReasonContent`.
- `ChatWithMessages` now calls the helper instead of doing the raw
`.(string)` cast.
- Unknown part types are **skipped, not failed**. Mistral has been
adding new content variants quickly (audio chunks, citations, etc.);
this driver should not 500 every call when a new part type appears.
- `conf/models/mistral.json`: add `magistral-medium-latest` and
`magistral-small-latest`. Both are visible in `/v1/models` today.
No interface change. No factory change. No new dependencies.
### How was this tested?
**Unit tests** — 5 new tests in `internal/entity/models/mistral_test.go`
on top of the 27 already shipped via #14807:
- `TestMistralChatHandlesStringContent` — regression net for the
historical path
- `TestMistralChatExtractsReasoningFromStructuredContent` — the fixture
body is a trimmed copy of the actual `magistral-medium-latest` response
captured above; asserts both `Answer` and `ReasonContent` are populated
correctly
- `TestMistralChatHandlesStructuredContentWithoutThinking` —
`magistral-*` with a trivial answer returns a structured shape that has
only a `text` part; `ReasonContent` must stay empty
- `TestMistralChatIgnoresUnknownContentPartTypes` — `audio_url` and
`future_part_type` parts are skipped, `text` parts still flow through
- `TestExtractMistralContent` — table-driven unit coverage of the helper
for string, empty string, nil, empty array, text-only, thinking+text,
unsupported root type
```
$ go test -vet=off -run "TestMistral|TestExtractMistralContent" -count=1 -v ./internal/entity/models/...
=== RUN TestMistralChatHandlesStringContent
--- PASS: TestMistralChatHandlesStringContent (0.00s)
=== RUN TestMistralChatExtractsReasoningFromStructuredContent
--- PASS: TestMistralChatExtractsReasoningFromStructuredContent (0.00s)
=== RUN TestMistralChatHandlesStructuredContentWithoutThinking
--- PASS: TestMistralChatHandlesStructuredContentWithoutThinking (0.00s)
=== RUN TestMistralChatIgnoresUnknownContentPartTypes
--- PASS: TestMistralChatIgnoresUnknownContentPartTypes (0.00s)
=== RUN TestExtractMistralContent
=== RUN TestExtractMistralContent/plain_string
=== RUN TestExtractMistralContent/empty_string
=== RUN TestExtractMistralContent/nil
=== RUN TestExtractMistralContent/empty_array
=== RUN TestExtractMistralContent/text_only
=== RUN TestExtractMistralContent/thinking_then_text
=== RUN TestExtractMistralContent/unknown_root_type
--- PASS: TestExtractMistralContent (0.00s)
PASS
ok ragflow/internal/entity/models 0.046s
```
All 32 Mistral tests pass on go 1.25. `go build
./internal/entity/models/...` exits 0.
**Live integration test** — driver exercised against `api.mistral.ai/v1`
with the patched code:
```
=== RUN TestMistralMagistralSmoke
[OK] "magistral-small-latest" present upstream
[OK] "magistral-medium-latest" present upstream
[OK trivial] Answer="7" ReasonContent=""
[OK reasoning] Answer len=797 head="To determine when the two trains meet, we can follow these steps:\n\n1. **Identify..."
ReasonContent len=1069 head="Okay, let's see. There are two trains, one going 60 mph and the other going 90 mph. They're moving towards each other, s..."
MAGISTRAL SMOKE PASSED
--- PASS: TestMistralMagistralSmoke (18.09s)
PASS
ok ragflow/internal/entity/models 18.112s
```
What the live run proves on the wire:
- `magistral-small-latest` with a trivial prompt still uses the
string-content shape; the regression-net path is exercised against the
real server, not just the mock.
- `magistral-medium-latest` with a reasoning prompt uses the
structured-array shape; the new code path extracts a 1069-character
reasoning trace into `ChatResponse.ReasonContent` and a 797-character
visible answer into `ChatResponse.Answer`. Before this fix, the same
call returned `"invalid content format"` and the caller saw nothing.
The smoke-test file itself is not committed (live tests live outside the
PR diff, same convention used for prior provider PRs).
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
## Problem
The Go server build pipeline (`build.sh` + CMake + CGO bindings) was
tested on Ubuntu only. On macOS arm64 with Homebrew it fails in five
orthogonal places. None of these require platform-specific code paths —
the same source builds on both Linux and Darwin after these fixes.
## Reproduction (before)
```
$ uname -a
Darwin … 25.4.0 arm64
$ brew install cmake pcre2 simde
$ bash build.sh
…
error: 'simde/x86/sse4.1.h' file not found
error: implicit instantiation of undefined template 'std::basic_istringstream<char>'
error: no matching function for call to 'Join'
…
clang: error: no such file or directory: '/usr/local/lib/libpcre2-8.a'
```
## Fix (5 small, orthogonal changes)
### 1. `internal/cpp/CMakeLists.txt` — find Homebrew + libpcre2-8
portably
- Detect Apple platforms via `if(APPLE)`, call `brew --prefix` once, add
`${HOMEBREW_PREFIX}/include` and `${HOMEBREW_PREFIX}/lib`. No effect on
Linux.
- Replace the literal `libpcre2-8.a` link token (which only the Linux
linker finds in `/usr/local/lib` by default) with
`find_library(PCRE2_LIB NAMES pcre2-8 REQUIRED)`. Works on
`/usr/lib/x86_64-linux-gnu` (Debian/Ubuntu), `/usr/local/lib` (Intel Mac
& legacy Linux), `/opt/homebrew/lib` (Apple Silicon).
### 2. `internal/cpp/wordnet_lemmatizer.cpp` +
`internal/cpp/rag_analyzer.cpp` — explicit `#include <sstream>`
libstdc++ (Linux) pulls `<sstream>` in transitively via `<fstream>`;
libc++ (Apple Clang) doesn't, so the existing `std::istringstream` /
`std::ostringstream` uses fail to compile on macOS. One-line include in
each file.
### 3. `internal/cpp/rag_analyzer.cpp` — `Join` template overload fix
`Join(tokens, start, tokens.size(), delim)` at line 146 passes `size_t`
to an `int` parameter. C++23 strict mode in Apple Clang refuses the
implicit narrowing and reports the 4-arg overload as a substitution
failure, leaving the call ambiguous between the 3-arg and 4-arg
templates. Fix: explicit `static_cast<int>(tokens.size())`. Behaviour
identical on libstdc++ — the narrowing was always intentional.
### 4. `internal/binding/rag_analyzer.go` — split darwin CGO LDFLAGS
The existing `#cgo darwin LDFLAGS: ... /usr/local/lib/libpcre2-8.a` only
matches Intel Macs. Apple Silicon Homebrew installs to `/opt/homebrew`.
Split into `darwin,arm64` and `darwin,amd64` build constraints with the
right absolute path on each.
### 5. `build.sh` — accept Homebrew path in the pcre2 sanity check
The sanity check looked at two Linux paths only and then fell through to
`sudo apt -y install libpcre2-dev` on failure. Added
`/opt/homebrew/lib/libpcre2-8.a`, and on Darwin failure now exits
cleanly with the right `brew install pcre2` hint instead of trying
`apt`.
## Verified
- `bash build.sh` now completes on macOS arm64 (Apple Silicon, brew 4.x,
cmake 4.x, Apple Clang 17, Go 1.25, pcre2 10.x, simde 0.8.x).
- Produced binaries: `bin/server_main`, `bin/admin_server`,
`bin/ragflow_cli`.
- `bin/server_main` boots, connects MySQL, runs migrations, loads the 64
model provider configs cleanly.
- Still builds on Linux — the CMake additions are inside an `if(APPLE)`
guard, the `find_library` call matches Linux paths too, the build.sh
check still tries `apt` when not on Darwin.
## Out of scope
The Go server itself currently fails at runtime when not pointing at
Elasticsearch (`Failed to initialize doc engine: failed to ping
Elasticsearch`), but that's the placeholder Infinity engine documented
in `internal/engine/README.md` — unrelated to this build patchset.
---
Happy to split this into smaller PRs if you'd prefer (one per file). The
five changes are independent.
## What
- Add Perplexity as a chat and embedding provider backed by its
OpenAI-compatible `/chat/completions` and `/v1/embeddings` APIs
- Register Perplexity in the Go model factory and provider config
- Support non-streaming chat, SSE streaming chat, embeddings, model
listing, and connection checks
Refs #14736
---------
Co-authored-by: Jin Hai <haijin.chn@gmail.com>
- Adds an `Astraflow` Go driver so the new API server can route
Astraflow (UCloud ModelVerse) chat instances, matching the existing
Python `AstraflowChat` (`rag/llm/chat_model.py:1237`). Follows the same
SaaS-driver shape used for Avian, Novita, TogetherAI, Replicate,
DeepInfra, Upstage, and LongCat.
Closes#15062
---------
Co-authored-by: Jin Hai <haijin.chn@gmail.com>
Closes#15044.
Avian was listed unchecked in the Go-rewrite tracker #14736 and already
had an llm_factories.json entry with 4 preconfigured chat models
(deepseek-v3.2, kimi-k2.5, glm-5, minimax-m2.5), but the Go API server
had no driver to route them. The Python side has supported Avian at
rag/llm/chat_model.py:1220 (AvianChat) via the LiteLLM openai/ provider
with default base https://api.avian.io/v1.
Co-authored-by: Jin Hai <haijin.chn@gmail.com>
### What problem does this PR solve?
`ReplicateModel.Embed` in `internal/entity/models/replicate.go` was a
`"replicate, no such method"` stub. Tracking issue #14736 lists
Replicate's embedding surface as not implemented. This PR wires it up
against Replicate's documented embedding schema.
Until this PR, a tenant who selected a Replicate embedding model got the
sentinel error on every embed call.
Co-authored-by: sxxtony <sxxtony@users.noreply.github.com>
Co-authored-by: Jin Hai <haijin.chn@gmail.com>
### What problem does this PR solve?
This PR adds a new `Browser` operator to Agent workflows, enabling
prompt-driven browser automation in RAGFlow.Technically based
‘Browser-Use’
It includes:
- Backend browser component execution with tenant LLM integration
- Upload source support (file IDs, URLs, variables, CSV/JSON array)
- Downloaded file persistence to RAGFlow storage
- Frontend node/operator integration, form config, icon, and i18n
updates
- Unit tests for upload/download and ID parsing logic
- Dependency and Docker updates for browser-use runtime support
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
## Summary
- Adds a lightweight `@tool` decorator and `FunctionToolSession` adapter
in `rag/llm/tool_decorator.py` that let callers register plain Python
functions as LLM tools without hand-writing OpenAI function schemas or
building an MCP-style session.
- Refactors `Base.bind_tools` and `LiteLLMBase.bind_tools` in
`rag/llm/chat_model.py` to accept either the new decorator form
`bind_tools(tools=[fn1, fn2])` or the existing `(toolcall_session,
tools_schemas)` form, so existing agent/dialog call-sites in
`agent/component/agent_with_tools.py`, `api/db/services/llm_service.py`,
and `api/db/services/dialog_service.py` are unaffected.
- Adds 8 unit tests in `test/unit_test/rag/llm/test_tool_decorator.py`
covering schema shape, required/optional inference, sync + async
dispatch, and bad-input rejection.
## Usage
```python
from rag.llm.tool_decorator import tool
@tool
def get_weather(city: str) -> str:
"""Get current weather for a city.
:param city: City name to look up.
"""
return f"{city}: 21 C, partly cloudy"
chat_mdl.bind_tools(tools=[get_weather])
ans, tk = await chat_mdl.async_chat_with_tools(system, history)
```
The decorator introspects `inspect.signature` + type hints + the
docstring (`:param name:` style) and attaches an OpenAI-format
`openai_schema` to the callable. `FunctionToolSession` duck-types the
existing `ToolCallSession` protocol, dispatching async callables
directly and sync ones through `thread_pool_exec` so the event loop is
never blocked.
## Design notes
- `tool_decorator.py` deliberately does **not** live inside
`rag/llm/__init__.py` to avoid forcing every consumer through the heavy
provider auto-discovery loop and to sidestep a circular import
(`__init__.py` imports `chat_model`, which would otherwise need symbols
from `__init__.py`).
- `FunctionToolSession` is duck-typed against
`common.mcp_tool_call_conn.ToolCallSession` rather than explicitly
inheriting from it, so importing the decorator doesn't pull the MCP
client SDK into the import graph.
- Docstring parsing is intentionally minimal (`:param name:` only) to
keep this dependency-free; Google/NumPy styles can be added later via
`docstring_parser` if needed.
## Test plan
- [x] `python -m pytest test/unit_test/rag/llm/test_tool_decorator.py
-v` — 8 passed
- [x] `python -m pytest test/unit_test/rag/llm/
--ignore=test/unit_test/rag/llm/test_perplexity_embed.py` — 11 passed
(the ignored test has a pre-existing `numpy` import that's unrelated)
- [ ] Reviewer: smoke-test the new path end-to-end with a live model via
`chat_mdl.bind_tools(tools=[my_fn])` to confirm the OpenAI-format
schemas pass through unchanged
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
### What problem does this PR solve?
Closes#15048.
Several SDK session routes in `api/apps/sdk/session.py` called
`.split()` directly on `request.headers.get("Authorization")`. When
clients omitted the header, the handlers raised `AttributeError` before
returning the existing `Authorization is not valid!` response.
This PR centralizes SDK Authorization parsing in a small helper and
keeps the existing error response for missing, empty, or malformed
headers.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
### Tests
- `ZHIPU_AI_API_KEY=dummy uv run --python 3.13 --group test pytest
test/testcases/test_http_api/test_session_management/test_session_sdk_routes_unit.py::test_sdk_session_routes_missing_authorization_unit
-q`
- `uv run --python 3.13 --group test ruff check api/apps/sdk/session.py
test/testcases/test_http_api/test_session_management/test_session_sdk_routes_unit.py`
- `python3 -m py_compile api/apps/sdk/session.py
test/testcases/test_http_api/test_session_management/test_session_sdk_routes_unit.py`
- `git diff --check`
### What problem does this PR solve?
Remove duplicate function definitions in
`api/db/services/dialog_service.py`.
**Problem:** Two helper functions were defined twice in the same file,
but with different parameter orders:
- First definition (line 57):
`_resolve_reference_metadata(request_payload=None, config=None)`
- Second definition (line 136): `_resolve_reference_metadata(config,
request_payload=None)`
**Solution:** Keep the second definition (which is actually used by
other modules) and remove the first one to avoid confusion.
Additionally, remove duplicate `_enrich_chunks_with_document_metadata`
definition (keep line 140 version).
<img width="1584" height="313" alt="image"
src="https://github.com/user-attachments/assets/7daee832-244f-4bb2-8488-e3b65012a3f9"
/>
<img width="1672" height="359" alt="image"
src="https://github.com/user-attachments/assets/4fd2f523-273c-4b20-a7c9-ab35740b7834"
/>
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
- [ ] New Feature (non-breaking change which adds functionality)
- [ ] Documentation Update
- [x] Refactoring
- [ ] Performance Improvement
- [ ] Other (please describe):
## Summary
- Align **GET `/api/v1/documents/<doc_id>/download`** with
**`/preview`**: resolve extension and MIME type from the stored document
name when the **`ext` query parameter is omitted**, instead of
defaulting to `markdown`.
- When **`?ext=`** is present, behavior stays the same as before
(explicit extension / `Content-Type` mapping).
- Enforce the same access + document lookup pattern as preview
(**`accessible`** + **`get_by_id`**).
- Extend unit tests for the no-`ext` PDF filename case.
## Test plan
- [x] `uv run pytest
test/testcases/test_web_api/test_document_app/test_document_metadata.py::TestDocumentMetadataUnit::test_download_attachment_success_and_exception_unit`
- [x] Optional: `curl -sSI` against
`/api/v1/documents/<pdf_doc_id>/download` without `ext` and confirm
`Content-Type: application/pdf`
Fixes#15052.
POST /api/v1/dify/retrieval resolved the caller via @apikey_required
(injecting tenant_id) but then fetched the requested knowledge_id with
no tenant filter and ran the full retrieval pipeline against
kb.tenant_id (the owner). Any valid Dify-compatible API key could
retrieve chunks from any tenant whose KB UUID was known. Adds the
missing ownership check.
## Root Cause
api/apps/sdk/dify_retrieval.py line 253:
KnowledgebaseService.get_by_id(kb_id) fetched the KB by id alone, then
the handler used kb.tenant_id (the OWNER) to build the embedding model
and call the retriever. The caller tenant_id was only used downstream at
line 278 for retrieval_by_children, well after cross-tenant data was
already retrieved.
grep confirmed there was no KnowledgebaseService.accessible call
anywhere in the handler.
## Fix
Two-line guard immediately after the existing get_by_id lookup,
mirroring the pattern PR #14749 lands for the sibling sdk/doc.py routes
(download, parse, stop_parsing, retrieval_test):
e, kb = KnowledgebaseService.get_by_id(kb_id)
if not e:
return build_error_result(message="Knowledgebase not found!",
code=RetCode.NOT_FOUND)
+ if not KnowledgebaseService.accessible(kb_id, tenant_id):
+ return build_error_result(message="No authorization.",
code=RetCode.AUTHENTICATION_ERROR)
if kb.tenant_embd_id:
...
KnowledgebaseService.accessible already handles solo-tenant ownership,
team membership via TenantService.get_joined_tenants_by_user_id, and the
permission=ME distinction. No behavior change for legitimate callers;
cross-tenant callers now receive RetCode.AUTHENTICATION_ERROR (109).
## Test Plan
- [x] Regression test added:
test/unit_test/api/apps/sdk/test_dify_retrieval.py
- test_cross_tenant_request_is_rejected -- attacker tenant calling owner
tenant KB gets 109; retriever is not invoked
- test_same_tenant_request_succeeds -- owner tenant gets the records
back
- test_missing_knowledge_base_returns_not_found -- missing KB returns
404 BEFORE the access check fires (legit callers see the clearer
message)
- [x] All 3 tests pass after the fix
- [x] Cross-tenant test FAILS on pre-fix main (KeyError on result[code]
because handler leaks records dict instead of returning auth error)
- [x] ruff check clean on both changed files
- [x] No drive-by reformatting in dify_retrieval.py -- only the 2 added
lines
### Post-fix output
test_cross_tenant_request_is_rejected PASSED [ 33%]
test_same_tenant_request_succeeds PASSED [ 66%]
test_missing_knowledge_base_returns_not_found PASSED [100%]
============================== 3 passed in 0.04s
===============================
Closes#15027
### What problem does this PR solve?
Closes#15076
Two endpoints in `api/apps/restful_apis/chat_api.py` accepted a
`user_id` field from the request body and used it directly when creating
a session:
```python
# before (vulnerable)
"user_id": req.get("user_id", current_user.id) # create_session
conv = await _create_session_for_completion(chat_id, dia, req.get("user_id", current_user.id)) # session_completion
```
Any authenticated caller could supply an arbitrary `user_id` and have
the new session attributed to a different user — effectively spoofing
session ownership. Both call sites are now fixed to always use
`current_user.id`, which is set by the authentication middleware and
cannot be tampered with via the request payload.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
### Changes
| File | Change |
|------|--------|
| `api/apps/restful_apis/chat_api.py` | Remove `req.get("user_id", ...)`
fallback in `create_session` and `session_completion`; always use
`current_user.id` |
|
`test/testcases/test_http_api/test_session_management/test_session_sdk_routes_unit.py`
| Add `test_create_session_user_id_not_spoofable` and
`test_session_completion_user_id_not_spoofable` (both `@pytest.mark.p2`)
|
### Testing
Two new unit tests assert that a `user_id` value supplied in the request
body is silently ignored and the session is always owned by the
authenticated user:
```
test_create_session_user_id_not_spoofable
test_session_completion_user_id_not_spoofable
```
Run with:
```bash
uv run pytest test/testcases/test_http_api/test_session_management/test_session_sdk_routes_unit.py -k "spoofable" -v
```
## What problem does this PR solve?
Closes#15021.
The Go model-provider layer had no support for **Azure OpenAI**. Azure
OpenAI is *not* a drop-in base-URL swap of the OpenAI driver — it
differs in authentication, endpoint structure, and how models are listed
— so it needs its own `ModelDriver` implementation.
## Type of change
- [x] New feature (non-breaking change which adds functionality)
Co-authored-by: Jin Hai <haijin.chn@gmail.com>
### What problem does this PR solve?
Fixes#15023
GPUStack is listed as unchecked in the Go-rewrite tracker #14736, and
`internal/service/llm.go:171` already classifies it as a self-deployed
provider alongside Ollama, Xinference, LocalAI, and LM Studio — but
`internal/entity/models/` had no `gpustack.go` driver, so the new Go API
server could not route GPUStack instances. This PR adds the chat surface
for GPUStack so it lines up with the existing self-hosted Go drivers.
Co-authored-by: Jin Hai <haijin.chn@gmail.com>
## Summary
- Replaces the `"no such method"` stub on `XinferenceModel.Embed`
(`internal/entity/models/xinference.go`) with a real implementation
against Xinference's OpenAI-compatible `/v1/embeddings` endpoint.
- Adds the `"embedding": "v1/embeddings"` URL suffix to
`conf/models/xinference.json`.
- Mirrors the Python `XinferenceEmbed` class in
`rag/llm/embedding_model.py:407` for payload shape (OpenAI-compatible
`model + input` → `data[*].index + data[*].embedding`) and tolerates the
same no-auth default Xinference deployments use. Authorization is only
sent when a non-empty API key is configured, via the existing
`setXinferenceAuth` helper.
- Reuses the existing `normalizeXinferenceBaseURL` + `baseURLForRegion`
helpers so both `http://127.0.0.1:9997` and `http://127.0.0.1:9997/v1`
resolve to the same `/v1/embeddings` target without doubled `/v1`.
- Validates response indices — duplicate, missing, or out-of-range
`data[*].index` values fail with a clear error rather than silently
producing misaligned vectors.
- Returns `[]EmbeddingData` in original input order (placed by `Index`)
so downstream callers can index positionally without re-sorting.
- Forwards `EmbeddingConfig.Dimension` as `dimensions` when `> 0`,
matching the OpenAI cluster pattern.
Closes#14810
Co-authored-by: Jin Hai <haijin.chn@gmail.com>
### What problem does this PR solve?
Fixes#15012
The Novita Go driver landed in #14850 and shipped a stub `Rerank` method
that returned `"novita, no such method"`, so Novita could not be used as
a rerank provider in RAGFlow. This PR fills that gap, in the same way
#14895 filled the Embed gap on the same driver.
Novita exposes a public rerank endpoint at `POST
https://api.novita.ai/openai/v1/rerank` that accepts the
Cohere-compatible request shape (`{model, query, documents, top_n}`)
with `Authorization: Bearer <api_key>`. `baai/bge-reranker-v2-m3` is
documented in Novita's model library with a 1024-token limit.
### What problem does this PR solve?
Fixes#14816
The Xinference Go driver landed chat in #14938 and Embed is in review in
#14932, but `Rerank` shipped as a stub that returns `"xinference, no
such method"`. Tenants who launch a rerank model with `--model-type
rerank` on their Xinference instance cannot route it through the Go API
server. This PR fills the gap.
Xinference exposes an OpenAI-compatible REST API. The rerank endpoint is
at `POST <base>/v1/rerank` and accepts the Cohere-shaped body `{model,
query, documents, top_n}`, returning `{results: [{index,
relevance_score}]}` — the same wire shape used by the merged NVIDIA
(#14778), Aliyun (#14676), Gitee (#14656), ZhipuAI (#14608), Novita
(#15014), and LocalAI (#14813) Rerank implementations. Documented in
[Xinference rerank
docs](https://inference.readthedocs.io/en/v1.6.1/models/model_abilities/rerank.html);
the [builtin rerank model
catalog](https://inference.readthedocs.io/en/stable/models/builtin/rerank/)
lists `bge-reranker-base`, `bge-reranker-large`, `bge-reranker-v2-m3`,
and others.
### What problem does this PR solve?
Add a Go driver for **n1n.ai** (https://docs.n1n.ai), one of the
unchecked providers on the umbrella tracking issue #14736. n1n.ai is an
OpenAI-compatible aggregator hosting a 450+ model catalog (GPT, Claude,
Gemini, DeepSeek, Kimi, Qwen, embedding + reranker families) under
`https://api.n1n.ai/v1`.
Until this PR, a tenant who configured `n1n` as a model provider in the
Go layer fell through to the default branch of
`internal/entity/models/factory.go` and got the dummy driver.
---------
Co-authored-by: sxxtony <sxxtony@users.noreply.github.com>
### What problem does this PR solve?
Fixes#15015
The TogetherAI Go driver in `internal/entity/models/togetherai.go`
shipped a stub `Embed` method that returned `"TogetherAI, no such
method"`, so TogetherAI could not be used as an embedding provider in
RAGFlow. This PR fills that gap.
TogetherAI exposes a public OpenAI-compatible embeddings endpoint at
`POST https://api.together.ai/v1/embeddings` that accepts the standard
`{model, input}` shape with `Authorization: Bearer <api_key>` (confirmed
in TogetherAI's official docs:
https://docs.together.ai/docs/embeddings-overview). Documented embedding
models include `intfloat/multilingual-e5-large-instruct`,
`BAAI/bge-large-en-v1.5`, and `BAAI/bge-base-en-v1.5`.
### Changes
- `internal/entity/models/togetherai.go`: implement
`TogetherAIModel.Embed`.
- Validate inputs (api key, model name) and short-circuit on empty
texts.
- Resolve region with the existing `baseURLForRegion` helper.
- Build URL from `URLSuffix.Embedding`.
- Send `{model, input}` POST body, add `dimensions` when
`embeddingConfig.Dimension > 0` (matches the pattern in #14735).
- Bearer auth + JSON content type, mirroring the chat path.
- Parse `{data: [{embedding, index}]}` and reorder by `index`, rejecting
out-of-range indices, duplicates, and missing entries so the output
always lines up with the input. Same shape as the merged Mistral,
Upstage, and Novita Embed implementations.
- `conf/models/togetherai.json`:
- Add `"embedding": "embeddings"` to `url_suffix`.
- Add default embedding model entries for
`intfloat/multilingual-e5-large-instruct`, `BAAI/bge-large-en-v1.5`, and
`BAAI/bge-base-en-v1.5`.
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
### What problem does this PR solve?
Fix: The logs on the data source details page are not fully displayed.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
### What problem does this PR solve?
1. Add model types when add model
---
```
RAGFlow(user)> add model 'pipeline' to provider 'mineru_local' instance 'test' with tokens 131072 doc_parse;
SUCCESS
```
2. implement provider: MinerU_Local
---
**Verified from CLI**
```
RAGFlow(user)> parse with 'pipeline@test@mineru_local' file './internal/test.pdf'
+--------------------------------------+
| task_id |
+--------------------------------------+
| c7260e31-b6e2-4b36-955d-e9c60510c669 |
+--------------------------------------+
RAGFlow(user)> show 'test@mineru_local' task 'c7260e31-b6e2-4b36-955d-e9c60510c669'
+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-------+
| content | index |
+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-------+
| # Repurposing Diffusion-Based Image Generators for Monocular Depth Estimation
Bingxin Ke Anton Obukhov Shengyu Huang Nando Metzger Rodrigo Caye Daudt Konrad Schindler Photogrammetry and Remote Sensing, ETH Zurich ¨
### What problem does this PR solve?
RuntimeError: Cannot run the event loop while another loop is running
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
### What problem does this PR solve?
Feat: add local & ssh provider in admin panel
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
### What problem does this PR solve?
Closes#15025
Langfuse-enabled `dialog_service.async_chat()` regressed to
`langfuse_tracer.start_generation(...)` after the earlier Langfuse v4
migration. Langfuse v4 uses `start_observation(as_type="generation")`,
so the remaining `start_generation` call can fail when chat tracing is
enabled.
This restores the migrated `start_observation(as_type="generation")`
call for chat observations while preserving the existing trace context,
model, input payload, and update/end flow. It also adds a regression
test with a fake Langfuse v4-style client that exposes
`start_observation()` but not `start_generation()`.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
### Tests
- `.venv/bin/pytest
test/unit_test/api/db/services/test_dialog_service_final_answer.py -q`
- `.venv/bin/ruff check api/db/services/dialog_service.py
test/unit_test/api/db/services/test_dialog_service_final_answer.py`
### What problem does this PR solve?
Fix: The folder tree menu for moving folders cannot be scrolled.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
### What problem does this PR solve?
Closes#15029.
Some custom `base_url` paths in `ModelProviderService` call
`NewInstance(newURL)` and then immediately invoke methods on the
returned driver. Several real Go model drivers still return `nil` from
`NewInstance`, so those paths can panic instead of returning a normal
error.
This PR:
- centralizes custom base URL driver creation in `model_service.go`
- skips request-local driver creation when `base_url` is blank or
whitespace
- preserves the existing region key behavior when building the
request-local base URL map
- returns a clear error when the provider driver is missing or
`NewInstance` returns `nil`
- routes list/check/task and active model paths through the guarded
helper
- adds focused unit coverage for empty-region preservation, regional
base URLs, blank base URLs, nil drivers, and nil `NewInstance` results
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
### Test plan
- [x] `git diff --check upstream/main...HEAD`
- [x] `/root/go/bin/gofmt -w internal/service/model_service.go
internal/service/model_service_test.go`
- [x] `GOPATH=/root/gopath GOTOOLCHAIN=local /root/go/bin/go test
./internal/service -run TestNewModelDriverForBaseURL -count=1 -vet=off`
- [x] `GOPATH=/root/gopath GOTOOLCHAIN=local /root/go/bin/go build
./internal/service/... ./internal/entity/models/...`
Note: the same targeted `go test` command without `-vet=off` is
currently blocked by an existing unrelated vet finding in
`internal/service/llm.go:355` (`non-constant format string in call to
fmt.Errorf`).
### What problem does this PR solve?
extend restful api suite
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
- [x] Other (please describe): test
### What problem does this PR solve?
This PR implement implement provider 302.AI and JieKouAI
**The following functionalities are now supported:**
**302.ai**
- [x] chat / think chat / stream chat / stream think chat
- [x] Embedding
- [x] ASR
- [x] ListModels
- [x] Provider connection checking
- [x] Balance
- [x] Rerank
- [x] OCR
- [x] Doc Parse
- [x] Show task
- [ ] ~~List Tasks!~~
- [ ] TTS
**JieKouAI**
- [x] chat / think chat / stream chat / stream think chat
- [x] Embedding
- [x] Rerank
- [x] ListModels
**Verified examples from the CLI:**
```palintext
# jiekouAI
RAGFlow(user)> stream think chat with 'zai-org/glm-4.5@test@jiekouai' message 'Hi'
Thinking: Let me think about how to respond to this simple greeting. The user just said "Hi", which is a basic and friendly way to start a conversation. I should respond in a similarly warm and welcoming manner.First, I need to acknowledge their greeting and reciprocate with enthusiasm. Something like "Hello!" or "Hi there!" would work well to create a positive atmosphere right from the start.Next, I should make it clear that I'm ready to help. Since they haven't asked anything specific yet, I'll keep it open-ended and inviting. Perhaps offering assistance with a question or task would encourage them to engage further.I should also maintain a professional yet approachable tone. Being an AI assistant, I want to convey that I'm knowledgeable and capable, but also friendly and easy to talk to.Let me put this all together into a concise response. I'll start with a cheerful greeting, express my readiness to help, and finish with an open invitation for them to share what's on their mind. This should create a welcoming environment for whatever they want to discuss next.
Answer: ! I'm Claude, an AI assistant created by Anthropic. I'm here to help you with information, answer questions, or assist you with tasks. What can I help you with today?
RAGFlow(user)> think chat with 'zai-org/glm-4.5@test@jiekouai' message 'Hi'
Thinking: Let me consider how to respond to this greeting. The user initiated with a simple "Hi," so a friendly and open response would be most appropriate to encourage further conversation. I should maintain a welcoming tone while offering assistance.
The response should accomplish a few key things: return the greeting warmly, show openness to conversation, and offer specific ways I can help. This approach demonstrates both approachability and usefulness.
I'll start with a greeting in return, then express my availability to help, and finish by suggesting some areas where I can provide assistance. This creates a natural flow from acknowledgment to support.
It's important to keep the response concise but inviting. Since the user hasn't specified their needs yet, I'll present a few broad categories of assistance to spark their thinking about what they might want to discuss or ask about.
The response should end with an encouraging note that prompts them to share what's on their mind, keeping the conversational ball in their court while making it clear I'm ready to engage with whatever they need.
Answer: Hello! How can I help you today? Whether you have questions, need information, or just want to chat, I'm here to assist.
RAGFlow(user)> embed text 'walkerwhat' 'jumperwho' with 'text-embedding-3-large@test@jiekouai' dimension 16
+-----------+-------+
| dimension | index |
+-----------+-------+
| 3072 | 0 |
| 3072 | 1 |
+-----------+-------+
RAGFlow(user)> rerank query 'what is rag' document 'rag is retrieval augment generation' 'rag need llm' 'famous rag project includes ragflow' with 'baai/bge-reranker-v2-m3@test@jiekouai' top 3
+-------+-----------------+
| index | relevance_score |
+-------+-----------------+
| 0 | 0.9830034 |
| 2 | 0.06399203 |
| 1 | 0.04665664 |
+-------+-----------------+
# 302.ai
RAGFlow(user)> think chat with 'kimi-k2.6@test@302.ai' message 'who r u'
Thinking: The user is asking "who r u" which is a casual way of asking "who are you." I need to identify myself as an AI assistant created by Moonshot AI. I should be friendly, concise, and helpful.
Key points to include:
- I am Kimi, an AI assistant made by Moonshot AI
- I can help with various tasks like answering questions, writing, analysis, coding, etc.
- Keep it casual but informative since the user used "r u" (text speak)
I should not:
- Pretend to be human
- Claim to have personal experiences or emotions
- Be overly formal or robotic
Simple, friendly response is best.
Answer: I'm Kimi, an AI assistant made by Moonshot AI. I can help you with answering questions, writing, coding, analysis, or just chatting. What can I do for you?
Time: 17.687750
RAGFlow(user)> stream think chat with 'kimi-k2.6@test@302.ai' message 'who r u'
Thinking: user asked "who r u" which is a casual way of asking "who are you." I should introduce myself as Kimi, an AI assistant developed by Moonshot AI. I need to be friendly, concise, and accurate. I should mention my capabilities briefly and keep the tone helpful. Since the user used casual text speak ("r u"), I can match that energy with a friendly but still informative tone.Key points:- I'm Kimi, an AI assistant made by Moonshot AI- I can help with various tasks like answering questions, writing, coding, analysis, etc.- Keep it brief but warm- Don't claim to be human- Don't over-explainDraft:"I'm Kimi, an AI assistant created by Moonshot AI. I can help with answering questions, writing, coding, analysis, brainstorming, and lots of other tasks. What can I do for you?"This is good - direct, accurate, and inviting.
Answer: Kimi, an AI assistant made by Moonshot AI. I can help with answering questions, writing, coding, analysis, brainstorming, and lots of other stuff. What can I do for you?
Time: 14.912576
RAGFlow(user)> asr with 'whisper-v3-turbo@test@302.ai' audio './internal/test.wav' param ''
+---------------------------------------------------------------------------------------------------------------------+
| text |
+---------------------------------------------------------------------------------------------------------------------+
| The examination and testimony of the experts enabled the Commission to conclude that five shots may have been fired |
+---------------------------------------------------------------------------------------------------------------------+
RAGFlow(user)> ocr with 'mistral-ocr-latest@test@302.ai' file './internal/test.pdf'
+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| text |
+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| # Repurposing Diffusion-Based Image Generators for Monocular Depth Estimation
Bingxin Ke
Nando Metzger
Anton Obukhov
Rodrigo Caye Daudt
Shengyu Huang
Konrad Schindler
Photogrammetry and Remote Sensing, ETH Zürich
Figur... |
+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
RAGFlow(user)> parse with 'vlm@test@302.ai' file 'https://arxiv.org/pdf/2505.09358'
+--------------------------------------+
| task_id |
+--------------------------------------+
| 6de6eae6-c122-4b67-91e8-b061a0b8c087 |
+--------------------------------------+
RAGFlow(user)> show 'test@302.ai' task '6de6eae6-c122-4b67-91e8-b061a0b8c087'
+----------------------------------------------------------------------------+-------+
| content | index |
+----------------------------------------------------------------------------+-------+
| https://file.302.ai/gpt/imgs/20260519/b340fdff4774699c287fe4ee4658b317.zip | 0 |
+----------------------------------------------------------------------------+-------+
RAGFlow(user)> embed text 'walkerwhat' 'jumperwho' with 'jina-embeddings-v3@test@302.ai' dimension 16
+-----------+-------+
| dimension | index |
+-----------+-------+
| 1024 | 0 |
| 1024 | 1 |
+-----------+-------+
RAGFlow(user)> rerank query 'what is rag' document 'rag is retrieval augment generation' 'rag need llm' 'famous rag project includes ragflow' with 'jina-reranker-v2-base-multilingual@test@302.ai' top 3;
+-------+-----------------+
| index | relevance_score |
+-------+-----------------+
| 0 | 0.74167407 |
| 2 | 0.18832397 |
| 1 | 0.15713684 |
+-------+-----------------+
```
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
- [x] Refactoring
### What problem does this PR solve?
extend restful api suite
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
- [x] Other (please describe): test
### What problem does this PR solve?
Closes#14751.
The user reported that after adding a variable (e.g. `key1`) to an
agent's **Begin** component, the Python SDK gave them no way to pass it:
their call `session.ask(question=user_question, stream=False)` had no
parameter for `key1`, and the `ask()` signature was just `(question,
stream, **kwargs)` with a docstring that only described streaming
behavior.
The functionality already works — `_ask_agent` does
`json_data.update(kwargs)` and the server reads `inputs` from the
request body at `agent_api.py:902`. The canonical shape is also in the
public API docs (`docs/references/python_api_reference.md:1817-1840`):
```python
session.ask(
"",
stream=False,
inputs={"line_var": {"type": "line", "value": "I am line_var"}},
return_trace=True,
)
```
But because `inputs`, `release`, and `return_trace` were hidden behind
`**kwargs`, they did not appear in IDE signature help, and the docstring
did not mention them. Users had no path from "I added a key in the UI"
to "I need to pass `inputs=...` with this exact shape."
This PR promotes the three most relevant Begin-related arguments to
named parameters and rewrites the docstring with a worked example.
### What this PR changes
- `sdk/python/ragflow_sdk/modules/session.py`:
- `Session.ask()` signature becomes `ask(question="", stream=False,
inputs=None, release=None, return_trace=None, **kwargs)`.
- These three new named params are forwarded into the existing `kwargs`
dict before dispatch, so the wire format and downstream behavior are
unchanged.
- Docstring rewritten in numpy style, including the structured `{"type":
..., "value": ...}` shape that the Begin component requires (see
`agent/component/begin.py:45-60`).
No backend changes. `**kwargs` is preserved for forward compatibility
with other body fields (`session_id`, `files`, `user_id`,
`custom_header`, …).
### Test plan
- [ ] `session.ask(question="hi", stream=False)` — existing call still
works
- [ ] `session.ask("", stream=False, inputs={"key1": {"type": "line",
"value": "v"}})` — Begin component receives `key1 = "v"`
- [ ] `session.ask("", stream=True, return_trace=True)` — streaming
response includes trace events
- [ ] IDE / `help(Session.ask)` now shows `inputs`, `release`,
`return_trace` with descriptions
### Type of change
- [x] Refactoring
- [x] Documentation Update
## Summary
Closes#14869.
Adds VLM-based semantic descriptions to **image chunks produced by the
MinerU parser**, closing a long-standing parity gap with the deepdoc
parser's `VisionFigureParser`. A maintainer flagged this in #13342
("We may add the VLM enhancement to MinerU parser as well") and an
earlier proposal exists in #13824; this PR lands the change end-to-end
inside the existing parser plumbing.
## Why
Today the MinerU parser returns image chunks containing only the
native `image_caption` and `image_footnote` strings from MinerU's
JSON. When neither is present (or when both are sparse), the chunk
carries effectively no searchable content for the figure and
retrieval misses it entirely. Users who configured a local VLM
(reporter's case: Gemma-4-31B) had to post-process MinerU's
`tmp/*.json` themselves.
The deepdoc parser already solves this via
[`VisionFigureParser`](deepdoc/parser/figure_parser.py): when the
tenant has an `IMAGE2TEXT` model configured, each figure gets a
semantic description merged into its chunk. This PR brings the same
behavior to MinerU.
## What changed
### `deepdoc/parser/mineru_parser.py`
- **New method `_enhance_images_with_vlm(outputs, vision_model,
callback=None)`** —
collects every `IMAGE` block with a readable `img_path`, runs
`rag.app.picture.vision_llm_chunk` in a 10-worker
`ThreadPoolExecutor` using the existing
`vision_llm_figure_describe_prompt`, and writes the result back as
`vlm_description`. Per-image failures are logged and skipped — they
never abort the run.
- **`_transfer_to_sections` (IMAGE branch)** — folds
`vlm_description` into the section text alongside caption +
footnote, so the description becomes part of the chunk and is
searchable / retrievable.
- **`parse_pdf`** — after `_read_output`, calls
`_enhance_images_with_vlm(outputs, vision_model, callback=callback)`
when a `vision_model` kwarg is supplied. Wrapped in `try / except`
so a VLM outage cannot break parsing.
### `rag/app/naive.py` (`by_mineru`)
After successfully resolving the MinerU OCR parser, also resolves the
tenant's default `LLMType.IMAGE2TEXT` model via
`get_tenant_default_model_by_type`, wraps it in an `LLMBundle`, and
injects it as `kwargs["vision_model"]` before delegating to
`parse_pdf`.
## Behavior
| Tenant config | Behavior |
|---|---|
| `IMAGE2TEXT` model configured | MinerU image chunks contain `caption +
footnote + VLM description`. Retrieval against figures now actually
works. |
| No `IMAGE2TEXT` model configured | Exact same output as today (caption
+ footnote only). Lookup fails silently with an info log; no error, no
regression. |
| VLM call fails for a single image | That image silently falls back to
caption + footnote; other images proceed. |
| Caller already passes `vision_model` in kwargs | We don't override it
— `if "vision_model" not in kwargs` guards the lookup. |
## Files
- `deepdoc/parser/mineru_parser.py` (+56)
- `rag/app/naive.py` (+13)
### What problem does this PR solve?
extend restful api suite
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
- [x] Other (please describe): test
## Summary
Closes#14921.
Reconfiguring an existing LLM provider to enable **tool call** or
**vision** fails with `Your API key is invalid. Fail to access model.`
even when the saved API key is correct. The most visible report is
VLLM ("Cannot add vllm model" once `--enable-auto-tool-choice` /
vision is toggled on), but the bug applies to every provider whose
api_key field stays blank in edit mode.
## Root cause
PR #14885 ("Fix: llm add api key overridden") removed the existing-key
lookup in `api/apps/llm_app.py::add_llm`. The intent was correct —
stop the saved key from clobbering a user-provided new one — but the
removal was unconditional, so the edit path now has no fallback at all:
1. `web/src/pages/user-setting/setting-model/hooks.tsx:230` sets the
initial `api_key` form value to `''` in edit mode (the real key is
never returned to the browser).
2. The user toggles `is_tools` / `vision` without retyping the key.
3. `hooks.tsx:183-185` strips the empty `api_key` from the payload.
4. `add_llm` defaults to the placeholder `"x"`
(`api/apps/llm_app.py:182`).
5. The upstream provider rejects `"x"` with `Your API key is invalid`.
## Fix
Restore the fallback **narrowly**, before any factory-specific handler
runs:
- If `req.get("api_key") is None`, look up the tenant's existing record
(using the correctly suffixed `llm_name` for VLLM /
OpenAI-API-Compatible / LocalAI / HuggingFace).
- Decode the saved blob with `_decode_api_key_config` and write **only
the decoded `api_key` string** back into `req["api_key"]`. Never use
the raw JSON payload — that was the exact thing PR #14885 was trying
to avoid.
- When the user **does** type a new key, `req.get("api_key")` is not
`None` and the fallback is skipped, so PR #14885's fix is preserved.
| Scenario | Before this PR | After this PR |
|---|---|---|
| Plain factory (VLLM, Ollama, …), retype key | OK | OK |
| Plain factory, blank key in edit (the bug) | Fails with "API key is
invalid" | Recovers saved key, validates against the real one |
| OpenRouter / Bedrock, change `provider_order` only | Fails |
`apikey_json([...])` rebuilds the JSON with saved `api_key` + new field
|
| User clears the form and types a brand-new key | OK (key replaced) |
OK (key replaced — fallback skipped) |
## Files changed
- `api/apps/llm_app.py` — restored fallback in `add_llm` (no other call
sites touched).
## Test plan
- [ ] Add a VLLM chat model with a valid api_key, no toggles → save
succeeds.
- [ ] Edit the same model, toggle **tool call** on, leave api_key blank
→ save succeeds, validation runs against the saved key.
- [ ] Edit again, toggle **vision** on (model_type → `image2text`),
leave api_key blank → save succeeds.
- [ ] Edit again and **type a new api_key** → the new key replaces the
saved one (`is None` check skips the fallback). Verify via the DB
row or by deliberately typing a wrong key and observing the
validation failure.
- [ ] Repeat the blank-key edit with **OpenRouter**, changing only
`provider_order` → resulting api_key JSON contains the saved
`api_key` and the new `provider_order`.
- [ ] First-time add of a new model name → no existing record, fallback
no-ops, behaves as before.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
- [ ] New Feature (non-breaking change which adds functionality)
- [ ] Documentation Update
- [ ] Refactoring
- [ ] Performance Improvement
- [ ] Other (please describe):
## What
- Add TogetherAI as a chat provider backed by its OpenAI-compatible
`/v1/chat/completions` API
- Register TogetherAI in the Go model factory and provider config
- Support non-streaming chat, SSE streaming chat, model listing, and
connection checks
## Notes
- Uses the current TogetherAI OpenAI-compatible base URL
`https://api.together.ai/v1`
- Forwards documented chat parameters from `ChatConfig`: `max_tokens`,
`temperature`, `top_p`, `stop`, and GPT-OSS `reasoning_effort`
- Routes Together reasoning traces from `reasoning` /
`reasoning_content` into `ReasonContent`
## Tests
- `go test -vet=off -run TestTogetherAI -count=1
./internal/entity/models`
- `go test -vet=off -count=1 ./internal/entity/models`
Refs #14736
### What problem does this PR solve?
Closes#14808.
Adds a Go model driver for Xinference so self-hosted Xinference chat
models can be used through the Go provider layer instead of falling
through to the dummy driver. Xinference exposes an OpenAI-compatible API
under `/v1`; the driver accepts either a root endpoint such as
`http://127.0.0.1:9997` or an OpenAI-compatible endpoint such as
`http://127.0.0.1:9997/v1` and normalizes it before calling chat or
model-listing routes.
### What is changed?
- Add `internal/entity/models/xinference.go` implementing `ModelDriver`
for Xinference chat.
- Route provider name `xinference` in
`internal/entity/models/factory.go`.
- Add `conf/models/xinference.json` as a local provider config.
- Add focused unit tests in `internal/entity/models/xinference_test.go`.
Initial method coverage:
- `ChatWithMessages`: POST `/v1/chat/completions`.
- `ChatStreamlyWithSender`: SSE streaming from `/v1/chat/completions`.
- `ListModels`: GET `/v1/models`.
- `CheckConnection`: lightweight `ListModels` probe.
- Optional auth: send `Authorization: Bearer <api_key>` only when a
non-empty key is configured, matching Xinference no-auth and
auth-enabled deployments.
- `Balance`, `Embed`, `Rerank`, ASR, TTS, and OCR return `no such
method` for this initial chat-provider PR.
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
- [x] Bug Fix (non-breaking change which fixes an issue)
### Tests
- `go test -vet=off -run TestXinference -count=1
./internal/entity/models/...`
- `go test -vet=off -count=1 ./internal/entity/models/...`
### References
- Xinference docs:
https://inference.readthedocs.io/zh-cn/latest/index.html
- OpenAI-compatible chat usage:
https://inference.readthedocs.io/zh-cn/latest/getting_started/using_xinference.html
- API key auth:
https://inference.readthedocs.io/zh-cn/latest/user_guide/auth_system.html
---------
Co-authored-by: Jin Hai <haijin.chn@gmail.com>
## What problem does this PR solve?
Closes#12582.
When a Retrieval component sits inside an Iteration with a **manual**
metadata filter that references the iteration variable (e.g.
`{IterationItem:abc@item}`), every iteration reuses the value resolved
on the **first** pass.
Root cause: [`_resolve_manual_filter` in
`agent/tools/retrieval.py`](https://github.com/infiniflow/ragflow/blob/main/agent/tools/retrieval.py#L144-L171)
mutated `flt["value"]` in place. The `filters` list passed in is the
live `self._param.meta_data_filter["manual"]` (see
[`apply_meta_data_filter` in
`common/metadata_utils.py:257-261`](https://github.com/infiniflow/ragflow/blob/main/common/metadata_utils.py#L257-L261)),
so after the first iteration the param dict permanently held the
resolved string instead of the original variable reference.
```text
iter #1: flt["value"] = "{IterationItem:abc@item}" → resolved to "AI"
after mutation: flt["value"] = "AI" ← written back into _param
iter #2: flt["value"] = "AI" ← no {…} matches
retrieval keeps filtering by "AI" forever
```
This PR returns a shallow copy with the resolved value instead, leaving
the original filter (and its variable reference) intact for the next
iteration.
## Type of change
- [x] Bug fix (non-breaking change which fixes an issue)
## Test plan
- [ ] Build an agent: `Agent (structured output → list of areas) →
Iteration → Retrieval (manual filter: Area = {IterationItem/Item}) →
Message`. Run with a multi-area query and confirm each iteration's
Retrieval result matches its own item, not the first item.
- [ ] Regression: Retrieval with a manual metadata filter outside an
Iteration still resolves the variable correctly on each request.
- [ ] Regression: Retrieval with no metadata filter and with `auto` /
`semi_auto` filters behave unchanged.
## What problem does this PR solve?
Closes#12017.
TTS output is deterministic for a given `(model, text)` pair, so
re-running the same text through the same TTS model produces the same
bytes — yet `Canvas.tts` and `dialog_service.tts` re-synthesized on
every request. That's slow and wastes provider quota whenever the same
assistant response is replayed, shared across users, or repeated within
a session.
### Change
New helper `rag/utils/tts_cache.py` with `synthesize_with_cache(tts_mdl,
cleaned_text)`:
- **Key:** `tts:cache:{model_id}:{sha256(text)}` — separate namespace
per model, identical cleaned text reuses a single entry across both call
sites.
- **Value:** the hex-encoded audio blob both call sites already
returned. No format change for downstream consumers.
- **TTL:** 7 days by default, configurable via
`RAGFLOW_TTS_CACHE_TTL_SECONDS`.
- **Failure modes:** a Redis hiccup falls back to direct synthesis; a
failed synthesis still returns `None` (existing contract preserved).
[`Canvas.tts`](https://github.com/infiniflow/ragflow/blob/main/agent/canvas.py#L683-L724)
and
[`dialog_service.tts`](https://github.com/infiniflow/ragflow/blob/main/api/db/services/dialog_service.py#L1367-L1380)
now route through the helper; the per-file bytes-accumulation/hex-encode
loop has been removed in favor of one shared implementation.
## Type of change
- [x] New Feature (non-breaking change which adds functionality)
## Test plan
- [ ] **Cache hit, chat path:** Configure a dialog with TTS enabled, ask
the same question twice with `stream=false`. Verify the second response
returns the same `audio_binary` and that the second invocation doesn't
hit the TTS provider (e.g., observe provider-side logs / usage counters;
check no `LLMBundle.tts can't update token usage` log line on the second
run).
- [ ] **Cache hit, agent path:** Same exercise via a Conversational
Agent that includes a Message component playing back the answer.
- [ ] **Cache isolation per model:** Switch tenant's `tts_id` between
two models, run the same text against each — confirm the second model's
first synthesis still happens (no cross-model hits).
- [ ] **TTL override:** Set `RAGFLOW_TTS_CACHE_TTL_SECONDS=120`, confirm
the entry expires after 2 minutes.
- [ ] **Redis unavailable:** Stop Redis (or break the connection).
Verify the TTS endpoint still works — synthesis falls back to direct
calls, with a `TTS cache lookup failed` / `TTS cache store failed`
warning logged.
- [ ] **Failure path:** Configure a TTS model with an invalid API key,
ensure the response still returns successfully with `audio_binary=None`
(no regression vs. current behavior).
## Summary
Fix critical severity security issue in
`internal/cpp/opencc/dictionary/text.c`.
## Vulnerability
| Field | Value |
|-------|-------|
| **ID** | V-001 |
| **Severity** | CRITICAL |
| **Scanner** | multi_agent_ai |
| **Rule** | `V-001` |
| **File** | `internal/cpp/opencc/dictionary/text.c:107` |
**Description**: The OpenCC C library uses fgets() to read dictionary
and configuration files without proper bounds validation on subsequent
buffer operations. While fgets() itself is bounds-checked, the sprintf()
call at config_reader.c:174 constructs file paths by concatenating
home_path and filename without verifying the result fits in pkg_filename
buffer. An attacker providing malformed OpenCC configuration files with
excessively long path components can overflow the fixed-size buffer,
overwriting adjacent memory including return addresses and function
pointers.
## Changes
- `internal/cpp/opencc/config_reader.c`
- `internal/cpp/opencc/dictionary/text.c`
- `internal/cpp/opencc/utils.c`
## Verification
- [x] Build passes
- [x] Scanner re-scan confirms fix
- [x] LLM code review passed
---
*Automated security fix by [OrbisAI Security](https://orbisappsec.com)*
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Bug Fixes**
* Improved error detection and handling for malformed configuration and
dictionary entries during file parsing.
* Enhanced memory cleanup in error recovery paths to prevent potential
issues.
* Strengthened robustness of string operations and buffer handling
throughout the library.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Co-authored-by: Ubuntu <ubuntu@ip-172-31-32-15.us-west-2.compute.internal>
## Problem
When using MinerU with `vlm-http-client` backend, the parser fails to
find the output files because they are located in a `vlm/` subdirectory,
but the `_read_output`
method doesn't check this location.
## Error Message
[ERROR]MinerU not found.
[MinerU] Missing output file, tried: ...
## Root Cause
The MinerU API with `vlm-http-client` backend returns output files in
the following structure:
output_dir/
vlm/
filename_content_list.json
filename.md
images/
However, the `_read_output` method in `mineru_parser.py` only checks:
1. `output_dir/filename_content_list.json`
2. `output_dir/sanitized_filename_content_list.json`
3. `output_dir/sanitized_filename/sanitized_filename_content_list.json`
It doesn't check the `vlm/` subdirectory.
## Solution
Added two additional fallback paths to check the `vlm/` subdirectory:
- `output_dir/vlm/filename_content_list.json`
- `output_dir/vlm/sanitized_filename_content_list.json`
## Testing
Tested with MinerU API using `vlm-http-client` backend. The parser now
successfully finds and processes the output files.
## Related
This issue occurs specifically when using:
- MinerU backend: `vlm-http-client`
- MinerU server URL configured for remote vLLM inference
## What
- Add Replicate as a chat provider backed by the documented predictions
API
- Register Replicate in the Go model factory and provider config
- Support non-streaming chat through sync predictions, polling fallback,
streaming through `urls.stream`, model listing, and connection checks
## Notes
- Uses `POST /v1/predictions` with Replicate model identifiers in
`version`, which supports official and community model identifiers
- Maps RAGFlow messages into Replicate prompt-shaped inputs (`prompt`,
optional `system_prompt`) and forwards common documented LLM inputs:
`max_new_tokens`, `temperature`, `top_p`
- Preserves whitespace in SSE output chunks and emits RAGFlow `[DONE]`
at stream completion
## Tests
- `go test -vet=off -run TestReplicate -count=1
./internal/entity/models`
- `go test -vet=off -count=1 ./internal/entity/models`
Refs #14736
### What problem does this PR solve?
Fix minor code quality issues:
1. Fix typo in assertion error message: "Can't fine" → "Can't find"
2. Remove duplicate line in common/connection_utils.py
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
- [x] Refactoring
### Related issues
Closes#14922
### What problem does this PR solve?
`POST /memories` already resolves `tenant_llm_id` and `tenant_embd_id`
through `ensure_tenant_model_id_for_params`, but `PUT
/memories/<memory_id>` accepted client-supplied `tenant_llm_id` /
`tenant_embd_id` without checking that those `tenant_llm` rows belong to
the memory owner’s tenant. A caller could persist another tenant’s row
IDs and later trigger extraction or embedding that loaded foreign model
credentials via `get_model_config_by_id(tenant_model_id)` with no tenant
allow-list.
This change aligns the update path with create: updates that change
models must go through `llm_id` / `embd_id` and
`ensure_tenant_model_id_for_params` scoped to the **memory’s**
`tenant_id` (not only the current user, so team-access cases stay
correct). Direct `tenant_*` fields in the body without `llm_id` /
`embd_id` are rejected. As defense in depth, `memory_message_service`
passes `allowed_tenant_ids` / `requester_tenant_id` into
`get_model_config_by_id` for LLM and embedding resolution so mismatched
IDs cannot be used even if bad data existed. A regression test rejects
payloads that set only `tenant_llm_id` / `tenant_embd_id`.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
---------
Co-authored-by: jony376 <jony376@gmail.com>
### What problem does this PR solve?
This PR improves the connector dashboard task management experience and
adds better visibility into connector execution logs.
### Overview:
#### Before
<img width="700" alt="image"
src="https://github.com/user-attachments/assets/e4a8ed6f-2e18-4f0f-8528-41a514550052"
/>
#### Now:
<img width="700" alt="Screenshot from 2026-05-18 16-31-30"
src="https://github.com/user-attachments/assets/d4ca193b-847a-49ae-9e4f-5fbca60ea627"
/>
### 1. Add a new logging page to the connector dashboard
A new logging page has been added so users can view connector task
execution logs directly from the connector dashboard.
### 2. Merge the Resume button into Confirm
The separate **Resume** button has been removed. The **Confirm** button
now represents different actions depending on the current task state:
- **Save**: Save form changes and reschedule tasks.
- **Stop**: Cancel currently scheduled or running tasks.
- **Resume**: Create new scheduled tasks after the previous tasks have
been stopped.
- **Start**: Start tasks when no task has been started yet.
### 3. Separate syncing and pruning tasks
Connector tasks are now separated into **syncing** and **pruning**.
Pruning is controlled by the **Sync deleted files** option:
- When **Sync deleted files** is disabled, only syncing tasks are shown.
- When **Sync deleted files** is enabled, both syncing and pruning tasks
are shown.
**Now: Sync deleted files disabled**
<img width="700" alt="Sync deleted files disabled"
src="https://github.com/user-attachments/assets/dbd9232e-614a-407f-a0b1-c109e5fa567d"
/>
**Now: Sync deleted files enabled**
<img width="700" alt="Sync deleted files enabled"
src="https://github.com/user-attachments/assets/1f527f48-ccb3-4ee8-97ca-086891489296"
/>
### 4. Update logs in backend
<img width="700" alt="image"
src="https://github.com/user-attachments/assets/10a95a3f-98c1-4e67-8afa-ddf6cda5b0b2"
/>
### 5. Remove connector resume API
- Removed: `POST /v1/connectors/<connector_id>/resume`
- Replaced by: `PATCH /v1/connectors/<connector_id>`
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
### What problem does this PR solve?
fix forgetting policy validation and fix memory update diff checks
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
### What problem does this PR solve?
When _parse_doc_id_filter_with_metadata returns [], the empty list is
falsy so the WHERE id IN (...) clause was silently skipped, causing the
full dataset to be returned instead of an empty result.
Change `if doc_ids:` to `if doc_ids is not None:` in both get_list() and
get_by_kb_id() to distinguish between no filter (None) and a filter that
matched zero documents ([]).
Fixes#14962
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
### What problem does this PR solve?
1. expose batch_chunk_token_size for configuration
2. retrieve chunks when build subgraph for the doc, not retreive all
docs chunks at the begining
3. get all chunks for a document, used to be hard coded 10000
4. delete not used method run_graphrag
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
- [x] Refactoring
Follow on: #14617
### What problem does this PR solve?
Fixes#14746.
Adds tenant access checks for connector-by-id REST routes before reading
connector details, mutating connector config/status, deleting
connectors, rebuilding, or listing sync logs. Unauthorized callers now
receive `RetCode.AUTHENTICATION_ERROR` with `No authorization.` without
reaching the connector/log mutation paths.
Validation:
- `python3 -m pytest
--confcutdir=test/testcases/test_web_api/test_connector_app
test/testcases/test_web_api/test_connector_app/test_connector_routes_unit.py`
- `uvx ruff check api/apps/restful_apis/connector_api.py
api/db/services/connector_service.py
test/testcases/test_web_api/test_connector_app/test_connector_routes_unit.py`
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
Co-authored-by: dev111-actor <dev111-actor@users.noreply.github.com>
### What problem does this PR solve?
Refactor: speed up ragflow server, save startup memory, saved 200MiB,
and 5-9 seconds start time.
##### Before
1241292 | | \_ python3 api/ragflow_server.py
RAGFlow server is ready after 25.61845850944519s initialization.
##### After
1019968 | | \_ python3 api/ragflow_server.py
RAGFlow server is ready after 16.205134391784668s initialization.
### Type of change
- [x] Refactoring
### What problem does this PR solve?
Closes#14858
The `test_db_connection` endpoint in the agent API accepts a
user-supplied `host` and connects to it directly via database drivers
(MySQL/PostgreSQL) without any validation. This allows an attacker to
probe internal network addresses (e.g. `127.0.0.1`, `10.x.x.x`,
link-local, etc.) through the server — a classic Server-Side Request
Forgery (SSRF) vulnerability.
This PR adds an SSRF guard that resolves the host and rejects any
address that is not globally routable before the database connection is
attempted.
**Changes:**
- **`common/ssrf_guard.py`** — Added `assert_host_is_safe()`, a
host-level counterpart of the existing `assert_url_is_safe()`, designed
for non-HTTP protocols (database drivers) where there is no URL to
parse.
- **`api/apps/restful_apis/agent_api.py`** — Call
`assert_host_is_safe(req["host"])` at the top of `test_db_connection` so
that non-public hosts are rejected early with a clear error message.
Fixes#14858
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
---------
Co-authored-by: Jin Hai <haijin.chn@gmail.com>
### What problem does this PR solve?
Adds the Go model provider driver for CometAPI, which is listed as
unchecked in the Go provider tracking issue #14736 and requested in
#14804. Without this, the Go layer falls back to the dummy driver for
the `cometapi` provider.
Fixes#14804
### What this PR includes
- New `internal/entity/models/cometapi.go` implementing `ModelDriver`
for CometAPI.
- New `conf/models/cometapi.json` with CometAPI base URLs and
representative chat / embedding models from the public catalog.
- `factory.go`: route `"cometapi"` to `NewCometAPIModel`.
- Unit tests in `internal/entity/models/cometapi_test.go`.
### Method coverage
- `ChatWithMessages`: `POST /v1/chat/completions`.
- `ChatStreamlyWithSender`: SSE streaming on the same endpoint.
- `Embed`: `POST /v1/embeddings`, including optional `dimensions`.
- `ListModels`: `GET /api/models` public catalog.
- `Balance`: `GET https://query.cometapi.com/user/quota?key=...`.
- `CheckConnection`: delegates to the quota query to verify the key.
- `Rerank`, ASR, TTS, OCR: return `no such method` for now.
No ModelDriver interface change. No new dependencies.
### How was this tested?
```bash
go test -vet=off -run TestCometAPI -count=1 ./internal/entity/models/...
go test -vet=off -count=1 ./internal/entity/models/...
```
---------
Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: Jin Hai <haijin.chn@gmail.com>
Signed-off-by: majiayu000 <1835304752@qq.com>
Co-authored-by: 加帆 <Jiafan@users.noreply.github.com>
Co-authored-by: Kevin Hu <kevinhu.sh@gmail.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: bulexu <baiheng527@gmail.com>
Co-authored-by: xubh <xubh@wikiflyer.cn>
Co-authored-by: Jin Hai <haijin.chn@gmail.com>
Co-authored-by: Carve_ <75568342+Rynzie02@users.noreply.github.com>
Co-authored-by: Paul Y Hui <paulhui@seismic.com>
Co-authored-by: LIRUI YU <128563231+LiruiYu33@users.noreply.github.com>
Co-authored-by: yun.kou <koopking@gmail.com>
Co-authored-by: Yun.kou <yunkou@deepglint.com>
Co-authored-by: Ahmad Intisar <168020872+ahmadintisar@users.noreply.github.com>
Co-authored-by: Ahmad Intisar <ahmadintisar@Ahmads-MacBook-M4-Pro.local>
Co-authored-by: chanx <1243304602@qq.com>
Co-authored-by: Syed Shahmeer Ali <syedshahmeerali196@gmail.com>
Co-authored-by: Octopus <liyuan851277048@icloud.com>
Co-authored-by: lif <1835304752@qq.com>
## Summary
- Stop pulling chunk vectors (`q_*_vec`) back from Elasticsearch in the
main retrieval path. ES already knows them; shipping them was pure
bandwidth/memory overhead.
- Recover the per-chunk cosine similarity via a second KNN-only ES call
filtered by the candidate chunk ids. The new `_score` is merged with
locally computed term similarity using the user-configured
`vector_similarity_weight`.
- Lazily fetch the chunk embedding only for the chunks
`insert_citations` actually needs.
## Details
**`rag/nlp/search.py`**
- `Dealer.search`: no longer appends `q_*_vec` to the ES select list.
OceanBase still gets it (its rerank path is unchanged).
- New `Dealer._knn_scores(sres, idx_names, kb_ids)`: a `MatchDenseExpr`
over the cached query vector filtered by `id IN sres.ids`, returning
`{chunk_id: cosine_score}` via ES `_score`.
- New `Dealer.rerank_with_knn(...)`: term similarity from
`qryr.token_similarity` plus the ES-supplied KNN score, combined with
`tkweight`/`vtweight` and the existing rank-feature bonus.
- New `Dealer.fetch_chunk_vectors(chunk_ids, tenant_ids, kb_ids, dim)`:
on-demand vector fetch for citation use.
- `Dealer.retrieval` routes Infinity → unchanged, OceanBase → existing
local `rerank`, ES → new KNN-score path.
**`common/doc_store/es_conn_base.py`**
- New `get_scores(res)` helper returning `{_id: _score}` directly from
hit headers (ES doesn't surface `_score` through `get_fields`).
**`api/db/services/dialog_service.py`**
- New top-level `_hydrate_chunk_vectors(...)` helper. On ES it
back-fills `ck["vector"]` from `fetch_chunk_vectors` right before
`insert_citations`. No-op on Infinity / OB (their chunks already carry
vectors).
- Both `decorate_answer` closures became `async` and are `await`-ed at
all call sites in `async_chat` and `async_ask`.
## Backend behavior
| Backend | Returns chunk vec in main search | Sim source | Vectors for
citations |
|---|---|---|---|
| ES | No | second KNN call (`_score`) merged with term sim | fetched on
demand |
| Infinity | No (unchanged) | normalized `_score` | already on chunks |
| OceanBase | Yes (kept) | local hybrid rerank | already on chunks |
## Test plan
### What problem does this PR solve?
Adds the missing Anthropic provider implementation for the Go model
provider layer.
Closes#14939
### What changed
- Add `conf/models/anthropic.json` with Anthropic Claude chat/vision
models and API endpoints.
- Add `internal/entity/models/anthropic.go` implementing non-streaming
Messages API chat, model listing, and connection checking.
- Register `anthropic` in the Go model factory.
- Add httptest coverage for headers, payload mapping, response parsing,
validation errors, provider errors, model listing, connection checking,
factory registration, and unsupported methods.
### Notes
Streaming chat is left as an explicit `no such method` follow-up because
this initial provider focuses on non-streaming chat and connection
checking.
### Tests
- `docker run --rm -v
/home/ubuntu/Documents/gitTensor_repos/carlos/ragflow:/work -v
/tmp/ragflow-go-cache:/go/pkg/mod -v
/tmp/ragflow-go-build:/root/.cache/go-build -w /work golang:1.25 go test
-vet=off ./internal/entity/models -run Anthropic -count=1 -v`
- `docker run --rm -v
/home/ubuntu/Documents/gitTensor_repos/carlos/ragflow:/work -v
/tmp/ragflow-go-cache:/go/pkg/mod -v
/tmp/ragflow-go-build:/root/.cache/go-build -w /work golang:1.25 go test
-vet=off ./internal/entity -count=1`
- `git diff --check`
- `jq . conf/models/anthropic.json >/dev/null`
Plain `go test ./internal/entity/models` currently hits pre-existing
unrelated vet findings in other provider files (`baidu.go`, `cohere.go`,
`fishaudio.go`, `openrouter.go`).
---------
Co-authored-by: Jin Hai <haijin.chn@gmail.com>
### What problem does this PR solve?
This PR adds non-streaming chat support for the Jina Go model provider.
The Jina provider was added with embedding, rerank, model listing, and
connection checking, but `ChatWithMessages` still returned a
not-implemented error even though Jina exposes an OpenAI-compatible
`/v1/chat/completions` endpoint.
Closes#14933
**The following functionalities are now supported:**
### **Jina:**
- [x] Chat
- [ ] Stream Chat
- [x] Embedding
- [x] Rerank
- [x] Model listing
- [x] Provider connection checking
- [ ] Balance
### **Implementation details:**
- Implements `JinaModel.ChatWithMessages`
- Sends `Authorization: Bearer <api-key>` and JSON chat completion
requests
- Validates API key, model name, messages, and configured region before
making requests
- Forwards supported chat config fields: `max_tokens`, `temperature`,
`top_p`, and `stop`
- Parses the first chat completion choice into `ChatResponse.Answer`
- Adds `jina-ai/jina-vlm` as a chat-capable model in
`conf/models/jina.json`
- Adds focused unit tests for request construction, auth, response
parsing, validation errors, provider errors, and region handling
**Verification:**
```plaintext
docker run --rm -v $PWD:/repo -w /repo golang:1.25 sh -c '/usr/local/go/bin/gofmt -w internal/entity/models/jina.go internal/entity/models/jina_test.go && /usr/local/go/bin/go test -vet=off ./internal/entity/models -run TestJina -count=1'
ok ragflow/internal/entity/models 0.037s
```
Note: `go test ./internal/entity/models -run TestJina -count=1`
currently hits unrelated existing vet findings in other provider files,
so the focused Jina tests were run with `-vet=off`.
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
---------
Co-authored-by: Jin Hai <haijin.chn@gmail.com>
### What problem does this PR solve?
Fixes#14893
The Novita Go driver landed in #14850 and shipped a stub `Embed` method
that returned `"novita, no such method"`, so Novita could not be used as
an embedding provider in RAGFlow. This PR fills that gap.
Novita exposes a public embeddings endpoint at `POST
https://api.novita.ai/v3/embeddings` that accepts the standard
OpenAI-compatible request shape (`{model, input}`) with `Authorization:
Bearer <api_key>`. Two embedding models are documented in Novita's model
library: `baai/bge-m3` (multilingual, 8192 tokens) and
`baai/bge-large-en-v1.5`.
### Changes
- `internal/entity/models/novita.go`: implement `NovitaModel.Embed`.
- Validate inputs (api key, model name) and short-circuit on empty
texts.
- Resolve region with the existing `baseURLForRegion` helper.
- Build URL from `URLSuffix.Embedding` (the embeddings path lives under
`/v3/`, separate from the chat path under `/openai/v1/`).
- Send `{model, input}` POST body, add `dimensions` when
`embeddingConfig.Dimension > 0` (matches the pattern in #14735).
- Bearer auth + JSON content type, mirroring the chat path.
- Parse `{data: [{embedding, index}]}` and reorder by `index`, rejecting
out-of-range indices, duplicates, and missing entries so the output
always lines up with the input. Same shape as the merged Mistral and
Upstage Embed implementations.
- `conf/models/novita.json`:
- Add `"embedding": "v3/embeddings"` to `url_suffix`.
- Add default embedding model entries for `baai/bge-m3` and
`baai/bge-large-en-v1.5` so they appear in the model picker.
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
---------
Co-authored-by: Jin Hai <haijin.chn@gmail.com>
### What problem does this PR solve?
This fixes a bug where files uploaded in chat were left in storage after
the session was deleted. It now removes those chat-uploaded blobs during
session deletion. fixes#14965
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
## RAG Optimization Description
Optimize the core `BaseTitleChunker` in
`rag/flow/chunker/title_chunker/common.py` to improve RAG document
chunking quality and retrieval accuracy.
## Key Changes
1. **Format-branched text processing**: Preserve original whitespace &
indentation for Markdown/HTML payloads to maintain document semantics
and chunk fidelity; only perform full whitespace cleaning on plain text
content.
2. **Empty chunk filtering**: Thoroughly filter invalid pure-blank lines
to reduce noisy data in vector database.
3. **Code deduplication**: Unified markdown/text/html payload extraction
logic, removed redundant repeated code blocks.
4. **None serialization fix**: Avoid converting `None` value into
literal `"None"` string in chunk text fields.
5. **Production logging**: Added input/output line count logging for
filter logic, observable in online environment.
6. **100% backward compatible**: No changes to chunking hierarchy rules,
output format and all existing workflows.
## RAG Business Value
- Preserves document format fidelity for structured Markdown/HTML files
- Reduces invalid noisy chunks → improves RAG retrieval precision
- Cleans plain text data → optimizes vector embedding quality
- Improves code maintainability with no breaking changes
- Provides observable logging for chunk filtering behavior
## Compatibility
- ✅ No API changes
- ✅ No chunk logic modifications
- ✅ All document parsing/chunking workflows unaffected
- ✅ All pre-checks passed, no code conflicts
### Type of change
- [x] Refactoring
- [x] Performance Improvement
## Summary
This PR fixes 3 bugs in agent components:
### Bug 1: `DataOperations._invoke()` dispatches `"literal_eval"` to
wrong handler
**File:** `agent/component/data_operations.py`, line 76
The `_invoke()` method compares `self._param.operations` against
`"recursive_eval"` (line 76), but the valid value defined in
`DataOperationsParam.__init__()` (line 29) and validated in `check()`
(line 43) is `"literal_eval"`. This means selecting the `literal_eval`
operation from the frontend would never match, and the method
`_literal_eval()` would never be called.
**Fix:** Change `"recursive_eval"` to `"literal_eval"` in the dispatch
condition.
### Bug 2: `VariableAssigner._clear()` — `bool` branch unreachable
**File:** `agent/component/variable_assigner.py`, lines 95–100
In Python, `bool` is a subclass of `int` (`True` is `isinstance(True,
int) == True`). The `isinstance(variable, int)` check on line 95 catches
boolean values before the `isinstance(variable, bool)` check on line 99,
making the bool branch unreachable. A boolean variable would be cleared
to `0` instead of `False`.
**Fix:** Move the `isinstance(variable, bool)` check before
`isinstance(variable, int)`.
### Bug 3: `LoopItem.evaluate_condition()` — `bool` branch unreachable
**File:** `agent/component/loopitem.py`, lines 67–93
Same issue as Bug 2: `isinstance(var, (int, float))` on line 67 catches
boolean values before `isinstance(var, bool)` on line 85. Boolean
variables would be evaluated with numeric operators (`=`, `≠`, `>`,
etc.) instead of boolean operators (`is`, `is not`).
**Fix:** Move the `isinstance(var, bool)` check before `isinstance(var,
(int, float))`.
## Test plan
- [ ] Verify `DataOperations` with `literal_eval` operation correctly
invokes `_literal_eval()`
- [ ] Verify `VariableAssigner._clear()` returns `False` for boolean
variables (not `0`)
- [ ] Verify `LoopItem.evaluate_condition()` uses boolean operators for
`True`/`False` values
🤖 Generated with [Claude Code](https://claude.com/claude-code)
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Bug Fixes**
* Fixed operation routing logic to correctly dispatch the "literal_eval"
operation to its handler.
* **Refactor**
* Reorganized conditional branch ordering in agent components to improve
code structure and maintainability without affecting functional
behavior.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
### What problem does this PR solve?
LLM/chat and search UIs render Markdown in several places (document
preview, floating chat widget, next-search, etc.). Plugin lists and
behavior were duplicated or inconsistent, and single newlines in model
output were not always rendered as visible line breaks, which hurts
readability for chat-style content.
This PR centralizes shared **remark/rehype** configuration (including
**`remark-breaks`** for newline handling) and wires the main Markdown
surfaces to use it, so behavior is consistent and easier to maintain.
### Type of change
- [x] Refactoring
---------
Co-authored-by: Yingfeng Zhang <yingfeng.zhang@gmail.com>
### What problem does this PR solve?
This PR implement implement OCR for Baidu and Mistral, implement
PaddleOCR provider and implement ASR for CoHere
**Verified examples from the CLI:**
```
RAGFlow(user)> ocr with 'mistral-ocr-2512@test@mistral' file './internal/text.jpg'
+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| text |
+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Parallel to these organizational innovations there were significant complementary technical innovations (e.g., improved methods of manufacturing cast-iron pipe and of coating interiors for pressure maintenance, and newer paving and construction material... |
+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
RAGFlow(user)> ocr with 'paddleocr-vl-0.9b@test@baidu' file './internal/text.jpg'
+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| text |
+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Parallel to these organizational innovations there were significant complementary technical innovations (e.g., improved methods of manufacturing cast-iron pipe and of coating interiors for pressure maintenance, and newer paving and construction material... |
+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
# PaddleOCR
RAGFlow(user)> ocr with 'PaddleOCR-VL-1.5@test@paddleocr' file './internal/test.pdf'
+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| text |
+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| # Repurposing Diffusion-Based Image Generators for Monocular Depth Estimation
Bingxin Ke
Nando Metzger
Photogra
Anton Obukhov
Rodrigo Caye Daudt
netry and Remote Sensing,
Shengyu Huang
Konrad Schindler
ETH Zürich
<div style="text-align: c... |
+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
# Cohere
RAGFlow(user)> asr with 'cohere-transcribe-03-2026@test@cohere' audio './internal/test.wav' param '{"language": "en"}'
+-----------------------------------------------------------------------------------------------------------------------+
| text |
+-----------------------------------------------------------------------------------------------------------------------+
| The examination and testimony of the experts enabled the Commission to conclude that five shots may have been fired. |
+-----------------------------------------------------------------------------------------------------------------------+
```
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
- [x] Refactoring
Closes#14753
## What changed
| File | Change |
|---|---|
| `pyproject.toml` | `requires-python` → `>=3.13,<3.15`; remove
`strenum==0.4.15` |
| `Dockerfile` | `uv python install 3.13`, `uv sync --python 3.13` |
| `.github/workflows/tests.yml` | `uv sync --python 3.13` on both matrix
legs |
| `CLAUDE.md` | dev setup command + requirements note updated |
| `deepdoc/parser/mineru_parser.py` | `from strenum import StrEnum` →
`from enum import StrEnum` |
| `agent/tools/code_exec.py` | same |
`StrEnum` has been in the stdlib since Python 3.11 — the `strenum`
backport package is no longer needed once the floor is 3.13.
## Why uv.lock is not regenerated
`uv lock --python 3.13` fails because:
1. The infiniflow/graspologic fork pins `numpy>=1.26.4,<2.0.0`
2. `tensorflow-cpu>=2.20.0` (the first release with cp313 wheels)
depends on `ml-dtypes>=0.5.1`, which requires `numpy>=2.1.0`
3. These two constraints are irreconcilable on Python 3.13
The lockfile regeneration requires loosening the `numpy` upper bound in
the `infiniflow/graspologic` fork. Once that fork commit is updated and
the SHA in `pyproject.toml:49` is bumped, `uv lock --python 3.13` will
succeed.
## RFC corrections
Two claims in the original RFC (#14753) did not hold up under code
review:
- **"graspologic hard-blocks 3.13"** — the infiniflow fork at the pinned
commit has no `<3.13` Python constraint. The blocker is the transitive
`numpy<2.0.0` conflict with tensorflow-cpu's test dependency, not a
direct Python version cap.
- **"free-threading throughput gains for I/O-bound workload"** — Python
3.13 free-threading requires a special `--disable-gil` build and
provides no benefit for async I/O code (the GIL is already released
during I/O). The real motivation is forward compatibility and improved
error messages.
## Summary
- Rename misspelled attribute `model_speciess` to `model_species` across
4 files
- The extra `s` is a typo — `species` is already plural
## Test plan
- [ ] Verify PDF parsing with laws/manual/paper parser types still works
correctly
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: yuj <yuj@ztjzsoft.com>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
### What problem does this PR solve?
This PRimplement TTS, ASR for Siliconflow and TTs for StepFun
**The following functionalities are now supported:**
**SiliConFlow:**
- [x] Text To Speech
- [x] Audio To Text
- [x] Stream Audio To Text
**StrepFun:**
- [x] Audio To Text
- [x] Stream Audio To Text
**Verified examples from the CLI:**
```plaintext
# SiliconFlow
RAGFlow(user)> tts with 'FunAudioLLM/CosyVoice2-0.5B@test@Siliconflow' text 'hello? show yourself' play format 'wav' param '{"voice": "fnlp/MOSS-TTSD-v0.5:alex"}'
SUCCESS
RAGFlow(user)> asr with 'FunAudioLLM/SenseVoiceSmall@test@siliconflow' audio './internal/test.wav' param ''
+----------------------------------------------------------------------------------------------------------------------+
| text |
+----------------------------------------------------------------------------------------------------------------------+
| The examination and testimony of the experts enabled the commission to conclude that five shots may have been fired. |
+----------------------------------------------------------------------------------------------------------------------+
RAGFlow(user)> stream asr with 'FunAudioLLM/SenseVoiceSmall@test@siliconflow' audio './internal/test.wav' param ''
+----------------------------------------------------------------------------------------------------------------------+
| text |
+----------------------------------------------------------------------------------------------------------------------+
| The examination and testimony of the experts enabled the commission to conclude that five shots may have been fired. |
+----------------------------------------------------------------------------------------------------------------------+
```
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
- [x] New Feature (non-breaking change which adds functionality)
### What problem does this PR solve?
```
+---------------------+----------------------------------+-------------+-----------------+---------+--------+------+
| created_at | id | meta_fields | name | size | status | type |
+---------------------+----------------------------------+-------------+-----------------+---------+--------+------+
| 2026-05-08 19:35:08 | f6aa38bb4ad111f1ba6338a74640adcc | map[] | abc.pdf | 3387987 | 1 | pdf |
+---------------------+----------------------------------+-------------+-----------------+---------+--------+------+
```
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
---------
Signed-off-by: Jin Hai <haijin.chn@gmail.com>
### What problem does this PR solve?
Users frequently ask how to use Ollama for local LLM inference with
RAGFlow. This FAQ entry provides step-by-step instructions for setting
up Ollama as a local model provider.
### Type of change
- [x] Documentation update
### Description
Adds a new FAQ entry: "How do I use Ollama with RAGFlow for local LLM
inference?"
Covers:
1. Starting Ollama and pulling a model
2. Configuring Ollama as a model provider in RAGFlow Settings
3. Using the Ollama model in an assistant
### What problem does this PR solve?
Closes#14878.
`VllmModel.Rerank()` in
[internal/entity/models/vllm.go:551](internal/entity/models/vllm.go#L551)
is currently a stub returning `nil, fmt.Errorf("%s, Rerank not
implemented", z.Name())`, and
[conf/models/vllm.json](conf/models/vllm.json) is missing a `rerank`
entry in `url_suffix`. Chat (long-standing) and embeddings (#14688)
already work, so rerank is the last missing leg of the retrieval
pipeline for operators running everything on a single self-hosted vLLM
server — today they have to point rerank at a different provider, which
defeats the point of a fully local deployment.
Upstream vLLM has supported a Jina/Cohere-compatible `POST /v1/rerank`
endpoint since v0.7
([vllm-project/vllm#12376](https://github.com/vllm-project/vllm/pull/12376)).
The request/response shape is essentially identical to the NVIDIA driver
landed in #14778, so this PR mirrors that structure with two
vLLM-specific adjustments.
This PR replaces the stub with a real implementation against vLLM's
`/v1/rerank`:
- `POST {baseURL}/rerank`
- Request body: `{"model": "<modelName>", "query": "<query>",
"documents": [...], "top_n": <int>}` — documents are a flat `[]string`,
**not** wrapped as `{text: "..."}` like NVIDIA's `/ranking`.
- Response body: `{"results": [{"index": int, "relevance_score": float},
...]}` (Jina-compatible; the optional `document` field is ignored since
callers reconstruct text via `Index`).
- `Authorization: Bearer <ApiKey>` is set **only when `APIConfig.ApiKey`
is non-empty**, matching the existing `Embed`/`ListModels` behaviour in
this file. vLLM is a local driver and can be deployed without an API
key.
The return shape matches the existing `*RerankResponse` contract used by
the NVIDIA ([nvidia.go:461](internal/entity/models/nvidia.go#L461)),
Aliyun ([aliyun.go:507](internal/entity/models/aliyun.go#L507)), and
ZhipuAI ([zhipu-ai.go:554](internal/entity/models/zhipu-ai.go#L554))
drivers, i.e. `Data []RerankResult` carrying `{Index, RelevanceScore}`
in the API's ranking order. Callers that need original-input order sort
by `Index`.
Behaviour requirements from the issue, all covered:
1. Empty `documents` → returns `&RerankResponse{}` without an HTTP call.
2. Missing `modelName` → `"model name is required"` validation error.
3. `rerankConfig.TopN` honored when `0 < TopN < len(documents)`;
otherwise `top_n` defaults to `len(documents)` so callers get a score
per input.
4. Non-200 responses return an error including upstream status and body
(`"vLLM rerank API error: <status>, body: <body>"`).
5. Response `index` values are bounds-checked against `len(documents)`.
**Scope:**
- [internal/entity/models/vllm.go](internal/entity/models/vllm.go) —
replaces the `Rerank` stub at line 551 with a real implementation; adds
`vllmRerankRequest`/`vllmRerankResponse` types for the slim subset of
the payload we need. Region/baseURL resolution, 30s context timeout,
conditional bearer header, and error wrapping all follow the existing
patterns in this file.
- [conf/models/vllm.json](conf/models/vllm.json) — adds `"rerank":
"rerank"` to `url_suffix`, joined to the operator-configured vLLM base
URL the same way the NVIDIA driver joins at
[nvidia.go:485](internal/entity/models/nvidia.go#L485).
-
[internal/entity/models/vllm_rerank_test.go](internal/entity/models/vllm_rerank_test.go)
— adds 7 `httptest`-backed tests mirroring `nvidia_rerank_test.go`:
happy path (out-of-order ranking → Index preservation), `top_n` clamp to
`RerankConfig.TopN`, empty-documents short-circuit, missing-model-name
validation, HTTP error propagation, out-of-range index rejection, and a
vLLM-specific `TestVllmRerankWithoutAPIKey` locking in the optional-auth
behaviour that distinguishes this driver from NVIDIA.
**Out of scope:** no interface change, no DDL, no frontend change. Chat,
embeddings, and balance paths are untouched. No new user-facing docs
required beyond the existing rerank model setup page — vLLM joins the
list of providers whose rerank model can be selected once `/v1/rerank`
is exposed by the server.
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
Fixes#13975
## Problem
The GitHub data source connector had both `include_pull_requests` and
`include_issues` defaulting to `false` in both the frontend form and the
backend sync code. This meant that with the default configuration, **no
content was synced at all** from a GitHub repository — silently
producing zero results.
Additionally, the form field labels contained a typo: "Inlcude" instead
of "Include".
## Solution
- Changed `include_pull_requests` default from `false` to `true` in the
frontend form fields and default values
- Changed `include_issues` default from `false` to `true` in the
frontend form fields and default values
- Changed both backend defaults in `sync_data_source.py` from `False` to
`True`
- Fixed label typos: "Inlcude Pull Requests" → "Include Pull Requests"
and "Inlcude Issues" → "Include Issues"
This makes the GitHub connector consistent with the GitLab connector,
which already defaults `include_mrs`, `include_issues`, and
`include_code_files` all to `true`.
## Testing
- The connector now syncs both pull requests and issues by default when
a new GitHub data source is created
- Users who want to exclude PRs or issues can uncheck the corresponding
checkboxes in the form
Co-authored-by: octo-patch <octo-patch@github.com>
## What problem does this PR solve?
Closes#13384.
The `/api/v1/agentbots/<agent_id>/completions` non-streaming path
returned the first yielded SSE chunk and exited:
```python
async for answer in agent_completion(objs[0].tenant_id, agent_id, **req):
return get_result(data=answer)
```
That meant structured output, the full assistant message, and reference
data were all dropped when an agent was called with `stream=false`.
Streaming worked because each event was forwarded individually;
non-streaming was returning a raw SSE-formatted string from a single
early event.
The v1 endpoint at
[`agent_api.py:1006-1050`](https://github.com/infiniflow/ragflow/blob/main/api/apps/restful_apis/agent_api.py#L1006-L1050)
already handles this correctly. This PR mirrors that aggregation in the
SDK beta endpoint: parse each SSE line, accumulate `content` from
`message` events, merge `reference`, collect `outputs.structured` from
each `node_finished` event keyed by `component_id`, and attach all of
them to the final response.
## Type of change
- [x] Bug fix (non-breaking change which fixes an issue)
## Test plan
- [ ] Build an agent with a node that emits structured output, call
`POST /api/v1/agentbots/<agent_id>/completions` with `stream=false` and
a beta API token, verify `data.structured.<component_id>` is present in
the response.
- [ ] Same agent with `stream=true` — verify behavior is unchanged.
- [ ] Agent without structured output — verify `data.structured` is
omitted, `content` and `reference` still aggregated correctly.
### What problem does this PR solve?
```
RAGFlow(user)> ocr with 'hunyuanocr@test@gitee' file './picture.png'
+----------------------------------------------------------+
| text |
+----------------------------------------------------------+
| 生活不是等待风暴过去，而是学会在雨中翩翩起舞。
——佚名 |
+----------------------------------------------------------+
RAGFlow(user)> list 'test@gitee' tasks;
+---------+----------------------------------+
| status | task_id |
+---------+----------------------------------+
| success | C3FX4MQNKY5MGC6ZFMIXIAMJKHCEBQB5 |
+---------+----------------------------------+
RAGFlow(user)> show 'test@gitee' task 'C3FX4MQNKY5MGC6ZFMIXIAMJKHCEBQB5';
+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-------+
| content | index |
+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-------+
| # PDF 1: Purpose of RAGFlow
RAGFlow is an open source Retrieval-Augmented Generation (RAG) engine designed to turn raw documents into reliable context for large language models.Its purpose is to make it practical to build an Al assistant that can ans... | 1 |
+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-------+
```
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
---------
Signed-off-by: Jin Hai <haijin.chn@gmail.com>
## Summary
Harden `api/utils/configs.deserialize_b64` so that it always routes
pickle data through the existing `RestrictedUnpickler`
(`restricted_loads`) rather than falling back to bare `pickle.loads()`.
- **CWE-502** — Deserialization of Untrusted Data
- **File / function**: `api/utils/configs.py` → `deserialize_b64`
- **Caller**: `SerializedField.python_value` in `api/db/db_models.py`
(invoked by Peewee whenever a pickled DB column is read)
## The issue
Before this change, `deserialize_b64` consulted a
`use_deserialize_safe_module` config flag that **defaults to `False`**
and is not set anywhere in the repository:
```python
use_deserialize_safe_module = get_base_config('use_deserialize_safe_module', False)
if use_deserialize_safe_module:
return restricted_loads(src)
return pickle.loads(src) # <-- default path
```
So the default code path was unrestricted `pickle.loads()` on bytes read
from a MySQL `SerializedField(serialized_type=PICKLE)` column. Any
attacker who can influence those bytes (SQL injection elsewhere,
compromised DB credentials, a backup restored from an untrusted source,
or a compromised replication peer) can craft a pickle payload that
achieves arbitrary code execution on the ragflow application server when
the field is next read.
Today no model in-tree instantiates a `SerializedField` with the default
PICKLE type — only `JsonSerializedField` is used in practice — so the
attack surface is currently **latent** rather than actively reachable
through an HTTP endpoint. But the insecure-by-default behaviour is a
sharp edge: any future field that uses the default PICKLE serialization
would silently inherit RCE-on-read semantics.
## The fix
```diff
- use_deserialize_safe_module = get_base_config(
- 'use_deserialize_safe_module', False)
- if use_deserialize_safe_module:
- return restricted_loads(src)
- return pickle.loads(src)
+ return restricted_loads(src)
```
`restricted_loads` is the existing `RestrictedUnpickler` already defined
in the same file, which limits permitted modules to `numpy` and
`rag_flow`. The config flag (and the now-dead `get_base_config` import)
are removed.
Diff is 1 insertion / 6 deletions, scoped to a single function.
## Testing
- Built a malicious pickle whose `__reduce__` resolves to
`posix.system('id')`. Pre-fix: executes. Post-fix: `restricted_loads`
raises `UnpicklingError: global 'posix.system' is forbidden`.
- Round-tripped a benign `numpy.ndarray` through `serialize_b64` →
`deserialize_b64`. Values preserved bit-for-bit.
- Confirmed `use_deserialize_safe_module` is not set in any config file
in the tree, so removing the flag does not change any operator-facing
knob that was actually in use.
## A note on `restricted_loads` itself
The existing `SECURITY.md` notes that `restricted_loads`'s `numpy`
allow-list can still be reached via `numpy.f2py.diagnose.run_command`.
This PR does **not** attempt to fix that — it is a separate hardening
question about tightening the allow-list to specific symbols rather than
whole modules. The change here strictly improves on the status quo (bare
`pickle.loads`) and brings the default path in line with what the
`restricted_loads` helper was clearly designed for. Happy to follow up
with a separate PR narrowing the allow-list if that direction is
welcome.
## Adversarial review
Before submitting, we tried to argue this finding away. The two
strongest objections are (1) "no field uses PICKLE today, so this is
unreachable" — true, but the default behaviour of a security-sensitive
helper still matters because new fields silently inherit it; and (2)
"the attacker already needs DB write access, which is game over" —
partially true, but pickle-RCE meaningfully escalates *data tampering*
into *code execution on the application host* (filesystem, internal
network, in-process secrets), which is not equivalent. The fix is one
line of real code, has no behavioural cost for legitimate callers, and
removes an insecure default. We decided it was worth filing.
---
<sub>_Submitted by Sebastion — autonomous open-source security research
from [Foundation Machines](https://foundationmachines.ai). Free for
public repos via the [Sebastion AI GitHub
App](https://github.com/marketplace/sebastion-ai)._</sub>
### What problem does this PR solve?
This PR fixes two issues in Agent Retrieval behavior and configuration
UX:
1. `top_k` configured in Agent Retrieval was not passed down to the
backend retriever call, so retrieval could ignore the configured vector
recall limit.
2. Similarity weight slider semantics were confusing in Agent forms
because the Agent field stores `keywords_similarity_weight` while UI
interactions were interpreted as vector weight. This could cause
displayed values and actual behavior to diverge.
This PR ensures Agent retrieval uses configured `top_k`, and makes the
slider behavior consistent and explicit for both vector and keyword
weight modes.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
## Summary
- Fixes **Tongyi-Qianwen** (`QWenEmbed`) text embeddings when the
configured `base_url` points at DashScope **international**
(`dashscope-intl.aliyuncs.com`) or **China** (`dashscope.aliyuncs.com`)
hosts, including values copied from Model Studio that use the
**OpenAI-compatible** path (`.../compatible-mode/v1`).
- The `dashscope` Python SDK (`TextEmbedding.call`) expects the
**native** HTTP root (`https://<host>/api/v1`), not the
OpenAI-compatible base URL. Without mapping, international accounts
could hit the wrong host or path.
## Implementation
- Added `_dashscope_native_http_api_url()` to normalize known DashScope
hosts to `.../api/v1`, and wired `QWenEmbed` to set
`dashscope.base_http_api_url` before each embedding call (document and
query).
## Notes
- In-code comments document the Tongyi-Qianwen / DashScope intl vs CN
behavior for future maintainers.
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
## Problem
When a user uploads a file attachment in their first message (Q1) and
then sends a follow-up message (Q2) that triggers a backend response,
the uploaded file attachment disappears from Q1 in the chat UI.
Fixes#13959
## Root Cause
In `single-chat-box.tsx`, a `useEffect` hook syncs `derivedMessages`
from `conversation?.messages` whenever the conversation data changes
(e.g., after a new assistant reply arrives):
```typescript
useEffect(() => {
const messages = conversation?.messages;
if (Array.isArray(messages)) {
setDerivedMessages(messages); // ← overwrites local state
}
}, [conversation?.messages, setDerivedMessages]);
```
The problem is that `conversation.messages` comes from the server, which
stores messages as plain JSON. Browser `File` objects (uploaded by the
user) cannot be serialized to JSON, so they are never stored on the
server. Each time the server data is applied to local state, the `files`
array on the user's first message is lost.
## Fix
Instead of replacing the local messages wholesale, preserve any `files`
entries from the previous local state by ID before applying the server
data:
```typescript
useEffect(() => {
const messages = conversation?.messages;
if (Array.isArray(messages)) {
setDerivedMessages((prevMessages) => {
const filesMap = new Map(
prevMessages
.filter((m) => m.files?.length)
.map((m) => [m.id, m.files]),
);
if (filesMap.size === 0) {
return messages;
}
return messages.map((m) => ({
...m,
files: filesMap.get(m.id) ?? m.files,
}));
});
}
}, [conversation?.messages, setDerivedMessages]);
```
This is a minimal, targeted fix: when there are no local files to
preserve the behavior is identical to before (early return with plain
assignment). When local file objects exist they are re-attached to the
corresponding server messages by ID.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Bug Fixes**
* Improved search query processing to properly handle special characters
and apostrophes in search terms and synonyms.
* Fixed chat message file attachments to persist when syncing with
server.
* **Refactor**
* Simplified OCR detection return values by removing timing metadata.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Co-authored-by: ximi <octo-patch@github.com>
### What problem does this PR solve?
add document download endpoint and refactor existing download function
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
### What problem does this PR solve?
Feat: This enables SelectWithSearch to search by label.
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
Co-authored-by: balibabu <assassin_cike@163.com>
### What problem does this PR solve?
This PR implement TTS for FishAudio and MiniMax provider and ASR for
FishAudio
**The following functionalities are now supported:**
**FishAudio:**
- [x] Text To Speech
- [x] Stream Text To Speech
- [x] Audio To Text
**OpenRouter:**
- [x] Text To Speech
**Verified examples from the CLI:**
```plaintext
**FishAudio**
RAGFlow(user)> tts with 's1@test@fishaudio' text 'He who desires but acts not, breeds pestilence.' play format 'wav' save './internal' param '{"reference_id": "90e65eaaf50e4470b8e6d43ee6afd7d5", "temperature": 0.7, "top_p": 0.7, "prosody": {"speed": 1, "volume": 0, "normalize_loudness": true}, "chunk_length": 300, "normalize": true, "sample_rate": 44100, "mp3_bitrate": 128, "latency": "normal", "max_new_tokens": 1024, "repetition_penalty": 1.2, "min_chunk_length": 50, "condition_on_previous_chunks": true, "early_stop_threshold": 1}'
Saved to directory: /home/infiniflow/Documents/development/ragflow/internal/s1_output.wav
SUCCESS
RAGFlow(user)> stream tts with 's1@test@fishaudio' text 'He who desires but acts not, breeds pestilence.' play format 'wav' save './internal' param '{"reference_id": "90e65eaaf50e4470b8e6d43ee6afd7d5", "temperature": 0.7, "top_p": 0.7, "prosody": {"speed": 1, "volume": 0, "normalize_loudness": true}, "chunk_length": 300, "normalize": true, "sample_rate": 44100, "mp3_bitrate": 128, "latency": "normal", "max_new_tokens": 1024, "repetition_penalty": 1.2, "min_chunk_length": 50, "condition_on_previous_chunks": true, "early_stop_threshold": 1}'
Saved to directory: /home/infiniflow/Documents/development/ragflow/internal/s1_output.wav
SUCCESS
RAGFlow(user)> asr with 'transcribe-1@test@fishaudio' audio './internal/test.wav' param '{"language": "en", "ignore_timestamps": true}'
+----------------------------------------------------------------------------------------------------------------------+
| text |
+----------------------------------------------------------------------------------------------------------------------+
| The examination and testimony of the experts enabled the commission to conclude that five shots may have been fired. |
+----------------------------------------------------------------------------------------------------------------------+
```
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
- [x] New Feature (non-breaking change which adds functionality)
- [x] Refactoring
## Summary
- Fixes resource-management bugs in the `POST
/agents/test_db_connection` endpoint where database connections could be
left open on error (part of #14750)
## Changes
- `api/apps/restful_apis/agent_api.py` — `test_db_connection`:
- mysql / mariadb / oceanbase / postgres: replaced bare `db.connect()` /
`db.close()` fallthrough with `with db.connection_context()` and a probe
`SELECT 1` — guaranteed close on both success and exception
- mssql: nested `try/finally` blocks so `cursor.close()` and
`db.close()` are always called even when `cursor.execute()` raises
- trino: wrapped cursor ops in `try/finally` for the same reason
- Removed the `if req["db_type"] != "mssql": db.connect(); db.close()`
shared fallthrough block — each branch now owns its teardown
- Consolidated to a single `return get_json_result(...)` after the
if/elif chain
### What problem does this PR solve?
When multiple MCP servers expose tools with the same name, the agent
currently registers those tools using their original MCP names. This can
lead to two issues:
- later MCP tools may overwrite earlier ones in the agent tool map
- duplicate function names may be exposed to the LLM
This PR fixes duplicate MCP tool-name handling by applying the same
indexed naming strategy already used for native agent tools. Native
tools are exposed with generated names such as `<tool_name>_<index>` to
avoid collisions, and MCP tools now follow the same convention for
consistency.
Specifically, this PR:
- assigns unique indexed function names to MCP tools exposed to the LLM
- preserves each MCP tool's original server-side name in an
`MCPToolBinding`
- dispatches MCP calls using the original MCP tool name while keeping
the indexed name in the agent tool map
- allows MCP metadata conversion to override only the OpenAI function
name without modifying the original MCP tool metadata
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
### Validation
The validation was performed using two MCP servers. Both servers exposed
a tool with the same name: `mcp0`. Both tools take no input parameters.
**MCP Server One:**
<img width="1780" height="625" alt="ONE"
src="https://github.com/user-attachments/assets/801a2654-fc10-4b71-b31c-81841fd40c55"
/>
**MCP Server Two:**
<img width="1777" height="624" alt="Second"
src="https://github.com/user-attachments/assets/c095151d-7bdf-47c8-9bfe-6aaf4a01b944"
/>
**Before the fix:**
When invoking `mcp0`, only the `mcp0` tool from the MCP server injected
later could be called successfully. As shown below, both `mcp0` tools
were present, but only the later-registered one was actually invokable.
<img width="694" height="935" alt="Three"
src="https://github.com/user-attachments/assets/3b9d7ab2-1765-492c-b8e0-bf05a69933ca"
/>
**After the fix:**
Both `mcp0` tools can now be invoked correctly.
<img width="737" height="1095" alt="F"
src="https://github.com/user-attachments/assets/6e896627-2b7f-41bb-becc-daa0c73ff58f"
/>
<img width="730" height="1090" alt="six"
src="https://github.com/user-attachments/assets/aba75593-26ae-4e3b-951d-b45ff177fd32"
/>
### What problem does this PR solve?
Fixes#14866.
Previously, `DocumentService.increment_chunk_num` and
`decrement_chunk_num` updated the `Document` row and its parent
`Knowledgebase` row in two separate, non-transactional statements. If
the second update failed (DB error, connection drop, etc.) after the
first one succeeded, the document and knowledge base chunk/token
counters would drift apart and stay inconsistent.
There was also a behavioral asymmetry between the two methods:
- `increment_chunk_num` only logged a warning when the document row was
missing and returned a value that callers usually treated as success.
- `decrement_chunk_num` raised `LookupError` in the same situation.
This PR makes the counter updates atomic and aligns the missing-document
behavior between the two methods:
- Wrap the `Document` and `Knowledgebase` updates in
`increment_chunk_num` / `decrement_chunk_num` inside a `DB.atomic()`
block so both succeed or both roll back together.
- Raise `LookupError` from `increment_chunk_num` when the target
document no longer exists, matching `decrement_chunk_num`.
- Update `reset_document_for_reparse` in `document_api_service.py` to
catch the new `LookupError` and return a proper "Document not found!"
API error instead of propagating the exception.
No schema changes, no API contract changes for the success path; only
the failure mode for a missing document during reparse is now a clean
error response instead of an uncaught exception.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
## Summary
- Replace `gen_conf={}` with `gen_conf=None` + guard in
`rag/llm/chat_model.py` (12 instances across Base, BaiChuanChat,
LocalLLM, MistralChat, ReplicateChat, BaiduYiyanChat, GoogleChat
classes)
- Replace `doc_ids=[]` with `doc_ids=None` + guard in
`api/db/services/document_service.py` (1 instance)
- Mutable default arguments are shared across all calls, causing
potential cross-request state contamination
- See Python docs:
https://docs.python.org/3/faq/programming.html#why-are-default-values-shared-between-objects
## Test plan
- [x] Verify LLM calls work with and without explicit gen_conf
- [x] No behavior change for existing callers — `None` is replaced with
`{}` at function entry
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Co-authored-by: Jin Hai <haijin.chn@gmail.com>
Co-authored-by: Yingfeng <yingfeng.zhang@gmail.com>
Co-authored-by: Kevin Hu <kevinhu.sh@gmail.com>
### What problem does this PR solve?
Closes#14853
The `/agents/download` and `/agents/<agent_id>/upload` endpoints in the
agent API are missing `@login_required` and `@add_tenant_id_to_kwargs`
decorators, allowing unauthenticated access. This is a security issue —
any user can upload files to or download files from an agent without
being logged in. Additionally, the upload endpoint bypasses canvas
access control (`@_require_canvas_access_async`).
This PR adds the missing authentication and authorization decorators to
both endpoints and replaces the manual `user_id` / `created_by` lookups
with the `tenant_id` provided by the auth middleware, making these
endpoints consistent with the rest of the agent API.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
### What problem does this PR solve?
In `api/utils/web_utils.py`, `__get_pdf_from_html()` creates a Chrome
WebDriver but only calls `driver.quit()` inside the `TimeoutException`
handler. If the page element becomes stale before the timeout (no
exception raised), the WebDriver is never quit, leaking the Chrome
browser process and returning `None`.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
### Changes
- Move the PDF printing logic and `driver.quit()` outside the `except`
block so they execute on all code paths
- Use `try/finally` to ensure `driver.quit()` is always called, even if
the `Page.printToPDF` DevTools call fails
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
## Summary
`Graph.set_variable_param_value()` in `agent/canvas.py` has a bug in its
nested path traversal logic. The `for` loop iterates through **all**
keys in the path (including the last one), descending into every level.
After the loop, it then tries to set `cur[keys[-1]] = value`, but `cur`
has already descended one level too deep.
**Example:** For `path = "a.b"`, `value = "hello"`:
- **Before (bug):** `obj["a"]["b"]` becomes `{"b": "hello"}` instead of
`"hello"`
- **After (fix):** `obj["a"]["b"]` becomes `"hello"` as expected
The fix changes `for key in keys:` to `for key in keys[:-1]:`, so the
loop only navigates to the parent dict, and the final key is set
directly. This is consistent with how the read-side counterpart
`get_variable_param_value()` works.
This method is called by `set_variable_value()` when assigning to nested
variable paths (e.g., `component@root.nested.key`), which is used by the
`VariableAssigner` component.
## Test plan
- [ ] Create a canvas with a VariableAssigner that writes to a nested
path (e.g., `component@obj.nested.key`)
- [ ] Verify the value is set correctly at the expected path, not
wrapped in an extra dict layer
- [ ] Verify single-key paths (e.g., `component@key`) still work
correctly
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Bug Fixes**
* Fixed a bug in variable parameter assignment where nested structures
were being incorrectly modified, ensuring values are now properly set at
their intended locations without unintended overwrites.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
### What problem does this PR solve?
This PR implement TTS for MiniMax provider and CLI testing for TTS
**The following functionalities are now supported:**
**MiniMax:**
- [x] Chat / Stream Chat
- [x] Embedding
- [x] Rerank
- [x] Model listing
- [x] Provider connection checking
- [x] Text To Speech
- [ ] OCRFile
- [ ] ~~Audio To Text~~
- [ ] ~~Balance~~
**Verified examples from the CLI:**
```plaintext
RAGFlow(user)> tts with 'speech-2.8-hd@test@minimax' text 'He who desires but acts not, breeds pestilence.' play format 'wav' save './internal' param '{"voice_setting": {"voice_id": "English_radiant_girl", "speed": 1, "vol": 1, "pitch": 0}, "audio_setting": {"sample_rate": 32000, "bitrate": 128000, "format": "wav", "channel": 1}, "output_format": "hex"}'
Saved to directory: /home/infiniflow/Documents/development/ragflow/internal/speech-2.8-hd_output.wav
SUCCESS
RAGFlow(user)> stream tts with 'speech-2.8-hd@test@minimax' text 'He who desires but acts not, breeds pestilence.' play format 'wav' save './internal' param '{"voice_setting": {"voice_id": "English_radiant_girl", "speed": 1, "vol": 1, "pitch": 0}, "audio_setting": {"sample_rate": 32000, "bitrate": 128000, "format": "wav", "channel": 1}, "output_format": "hex"}'
Saved to directory: /home/infiniflow/Documents/development/ragflow/internal/speech-2.8-hd_output.wav
SUCCESS
```
Set `Play` to play audio in CLI
Set `Save` `PATH_TO_SAVE` to save file
Set `format` to save file in wav or mp3
Set `Param` align with official request body
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
### What problem does this PR solve?
Closes#9837
The Langfuse integration currently only sends the output text to
`langfuse_generation.update()` without including token usage
information. This means Langfuse cannot track input/output token
consumption for cost analysis and monitoring.
### Solution
Add the `usage` parameter to `langfuse_generation.update()` with:
- `input`: approximate input token count from `message_fit_in()`
- `output`: approximate output token count from
`num_tokens_from_string(answer)`
- `total`: sum of input and output
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
---------
Co-authored-by: Kevin Hu <kevinhu.sh@gmail.com>
\`assert \"string\"\` always passes in Python because non-empty strings
are truthy. This silently skips input validation:
- **variable_assigner.py line 51**: \`assert \"Variable is not
complete.\"\` → \`raise ValueError(\"Variable is not complete.\")\`
- **loop.py line 59**: \`assert \"Loop Variable is not complete.\"\` →
\`raise ValueError(\"Loop Variable is not complete.\")\`
Without this fix, incomplete variables pass validation silently and
cause a confusing KeyError on the next line.
## Description
This PR fixes critical bugs and improves the robustness of the RAG
reranking module while maintaining **100% backward compatibility** with
all existing functionality and providers.
## Key Changes
1. **Network Stability**: Added 30s timeout to all API requests to
prevent service blocking
2. **Boundary Protection**: Added empty query/text validation for all
rerank models
3. **Response Fault Tolerance**: Replaced hardcoded key access with
`.get()` to avoid KeyError crashes
4. **Bug Fixes**:
- Fixed `Ai302Rerank` (completely non-functional before)
- Fixed `GPUStackRerank` incorrect exception catching
- Fixed `_normalize_rank` empty array crash
5. **Code Specification**: Added type annotations, standardized
unimplemented class prompts
## Compatibility
- ✅ No changes to any class/method names
- ✅ All rerank providers (Jina/Cohere/NVIDIA/HuggingFace etc.) work as
before
- ✅ No breaking changes, zero impact on existing workflows
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
- [x] Refactoring
---------
Co-authored-by: Kevin Hu <kevinhu.sh@gmail.com>
### What problem does this PR solve?
- Update version tags in README files (including translations) from
v0.25.3 to v0.25.4
- Modify Docker image references and documentation to reflect new
version
- Update version badges and image descriptions
- Maintain consistency across all language variants of README files
### Type of change
- [x] Documentation Update
### What problem does this PR solve?
Includes gpt-5.4-mini and gpt-5.4-nano to the OpenAI model list
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
### What problem does this PR solve?
Add a Go driver for Voyage AI (https://voyageai.com), one of the
unchecked providers on the umbrella tracking issue #14736. Voyage AI is
**embed + rerank only** — no chat, no streaming, no `/v1/models`
endpoint. It's the first provider in the Go layer of this shape.
Until this PR, a tenant who configured `voyage` as a model provider in
the Go layer fell through to the default branch of
`internal/entity/models/factory.go` and got the dummy driver.
### What this PR includes
- New `internal/entity/models/voyage.go` with a `VoyageModel`
implementing the `ModelDriver` interface.
- New `conf/models/voyage.json` with 6 embedding models (`voyage-3.5`,
`voyage-3.5-lite`, `voyage-3-large`, `voyage-code-3`, `voyage-law-2`,
`voyage-finance-2`) and 2 rerank models (`rerank-2`, `rerank-2-lite`).
- `factory.go`: route `"voyage"` to `NewVoyageModel`.
- `internal/entity/models/voyage_test.go`: 19 unit tests.
### How the driver works
- **Embed**: `POST /v1/embeddings`. Response is OpenAI-shaped (`{data:
[{embedding, index, object, text}], model, usage}`). Driver reorders by
`index`, rejects duplicate / out-of-range / missing slots, and
short-circuits empty input without an HTTP call.
- **Rerank**: `POST /v1/rerank`. Voyage uses **`top_k`** as the request
param name (not `top_n` like Aliyun/SiliconFlow); the driver translates
`RerankConfig.TopN` → `top_k`. Response is Cohere-shaped (`{data:
[{relevance_score, index}], model}`), so the existing
`RerankResponse{Data: []RerankResult{Index, RelevanceScore}}` shape fits
cleanly.
- **`ListModels`**: returns a hardcoded list of `voyageKnownModels`.
Voyage does **not** expose `/v1/models` (probed live, returns 404), so
the driver synthesizes the list from the same set the config ships. New
upstream models are added by extending one slice.
- **`CheckConnection`**: pings a 1-input embed call against
`voyage-3.5`. Without `/v1/models`, this is the cheapest way to verify
the API key + network path before a tenant tries a real workload.
- **`ChatWithMessages` / `ChatStreamlyWithSender` / `Balance` /
`TranscribeAudio` / `AudioSpeech` / `OCRFile`**: all return `"no such
method"`. Voyage does not host any of these surfaces.
No interface change. No new dependencies.
### How was this tested?
**19 unit tests** in `internal/entity/models/voyage_test.go` — all pass
on go 1.25:
```
$ go test -vet=off -run TestVoyage -count=1 ./internal/entity/models/...
ok ragflow/internal/entity/models 0.036s
```
Coverage: Name; Embed (happy path, reorder, empty-input, missing
key/model, duplicate index, out-of-range index, missing slot); Rerank
(happy path with `top_k` assertion, default-to-len-documents, empty
documents, out-of-range index); ListModels (static list, missing key);
CheckConnection (happy, 401); chat methods sentinels; Balance sentinel;
audio/OCR sentinels.
`go build ./internal/entity/models/...` exits 0.
**Live integration test** against `api.voyageai.com`:
```
=== RUN TestVoyageLiveSmoke
[OK] Name() = "voyage"
[OK] ListModels (static): 8 models -> [voyage-3.5 voyage-3.5-lite voyage-3-large voyage-code-3 voyage-law-2 voyage-finance-2 rerank-2 rerank-2-lite]
[OK] CheckConnection
[OK] Embed vectors=3 dim=1024 indices=[0 1 2]
[OK] Embed(empty) -> 0 vectors
[OK] Rerank results=3 scores=[0.8125 0.59765625 0.39453125]
[OK] ChatWithMessages returns voyage, no such method
[OK] Balance returns voyage, no such method
VOYAGE LIVE SMOKE PASSED
--- PASS: TestVoyageLiveSmoke (0.81s)
```
What the live run proves on the wire:
- Auth (`Bearer <key>`) accepted by `api.voyageai.com`.
- Embed `voyage-3.5` on 3 inputs returns 3 vectors at dim 1024 with
`index` field preserved as `[0, 1, 2]` — the reorder-by-index code is
exercised on real data.
- Empty input short-circuits without an HTTP call (mock server would
have been hit if it did).
- Rerank `rerank-2` on 3 docs returns 3 real `relevance_score` floats
`[0.8125, 0.598, 0.395]`. The `top_k` translation works on the live
wire.
- All sentinel methods return the documented `"no such method"` strings.
### Note on PR history
This branch was previously named for LocalAI Embed work which is now
consolidated into PR #14813. The branch was reset to `upstream/main` and
rebuilt for Voyage. Diff against `main` is a clean +838 lines across 4
files.
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
Tracking: #14736
---------
Co-authored-by: Jin Hai <haijin.chn@gmail.com>
## Summary
Closes#14774.
Adds free-form tags on agents (UserCanvas) with full UI + API:
- Stored as comma-separated `tags` column on `UserCanvas` with online
migration.
- New endpoints: `GET /v1/agents/tags` (aggregate counts) and `PUT
/v1/agent/<id>/tags` (write). `GET /v1/agents` accepts a `tags=` query.
- "Edit tags" item in agent dropdown opens a chip-style editor dialog;
tags render as badges on each agent card.
- New "Tags" facet in the agents filter bar, with counts.
## Implementation notes
- **Tag matching is exact-token**: the SQL filter wraps stored tags as
`,…,` and matches `,ml,` so `ml` doesn't match `ml-ops`.
- **Server-side normalization** in `UserCanvasService.update_tags`:
dedup (case-insensitive), per-tag cap of 64 chars, total length capped
at 512 chars to fit the column, commas inside tag values are replaced
with spaces.
- **Tenant authorization**: `PUT /v1/agent/<id>/tags` gates on
`UserCanvasService.accessible(canvas_id, tenant_id)`.
- **Tag listing scope**: `UserCanvasService.list_tags` follows the same
own + team-shared rule as `get_by_tenant_ids`.
- **i18n**: keys added to `en.ts` and `zh.ts` only (per project
convention; other locales fall back).
- **`HomeCard`** gets a non-breaking `extra?: ReactNode` slot for the
chip row; no `src/components/ui/` files modified.
## Test plan
- [ ] Backend boot runs `migrate_db` → confirm `user_canvas.tags` column
exists (`DESCRIBE user_canvas`).
- [ ] Agents page renders cards normally (no console error from missing
field).
- [ ] `⋯ → Edit tags` opens a dialog that stays open (regression: dialog
was unmounting with the dropdown).
- [ ] Typing a tag without pressing Enter and clicking Save persists it
(regression: last typed tag was being dropped).
- [ ] Chip input supports Enter/comma to commit, Backspace on empty to
remove, `×` to remove individual chip.
- [ ] Tag containing a comma sent via API is stored with the comma
replaced by a space.
- [ ] 20 long tags sent via API does not error (length cap silently
truncates).
- [ ] "Tags" filter in the filter bar shows counts and narrows the list.
- [ ] Filtering by `ml` does **not** return agents tagged `ml-ops`.
- [ ] UI in Chinese shows 编辑标签 / 添加标签以整理和筛选你的智能体 etc.
- [ ] `PUT /v1/agent/<other-tenant-id>/tags` returns `Agent not found or
no permission.`
Introduce comprehensive floating widget customization: add new widget
settings (title, subtitle, footer, colors, mute, streaming) with types
and defaults, and expose them via EmbedDialog UI (split into Embed Setup
and Widget Customization tabs). Persist and load settings through Agent
page by reading/writing globals and wiring an onSaveWidgetSettings
handler to setAgent; show a loading ButtonLoading for saving. Update
embed iframe query params and FloatingChatWidget to honor URL params
(colors, text, mute/streaming) with validation/normalization, color
darkening for gradients, footer link normalization, and improved
styling. Also add copy-to-clipboard in message toolbar, adjust syntax
highlighter layout and Copy button, and add i18n key for muteWidget.
### What problem does this PR solve?
Adds a few fields to the embed widget modal to customize the appearance
of the floating widget when embedded into a page.
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
---------
Co-authored-by: Noah <Noah.Thompson@ecn.forces.gc.ca>
## Summary
- Convert `pdfplumber.open()` to use `with` context manager in
`api/utils/file_utils.py` (`thumbnail_img` function)
- If any exception occurs between `open()` and `close()`, the PDF file
handle leaks
- The rest of the codebase (e.g. `read_potential_broken_pdf` in the same
file) already uses `with pdfplumber.open(...)` correctly
## Test plan
- [x] PDF thumbnail generation works correctly with context manager
- [x] Resources properly cleaned up on exceptions
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
# feat: Add Generic REST API Connector
## What problem does this PR solve?
RAGFlow supports many specific data source connectors (MySQL, Slack,
Google Drive, etc.), but there was no way to connect an arbitrary REST
API as a data source. Users with custom or third-party APIs had to write
a new connector class for each one.
This PR adds a **generic, configuration-driven REST API connector** that
lets users connect any REST API as a data source entirely through the UI
— no code changes needed per API.
---
## Features
### Core Connector (`common/data_source/rest_api_connector.py`)
- Implements `LoadConnector` and `PollConnector` interfaces for full and
incremental sync
- **Configurable authentication:** None, API Key (custom header), Bearer
Token, Basic Auth
- **Pluggable pagination:** Page-based, Offset-based, Cursor-based, or
None
- Smart page-size inference from user's query parameters to avoid
duplicate/conflicting params
- Configurable request delay between pages to prevent API rate limiting
- Auto-detection of the items array in JSON responses (`items`,
`results`, `data`, `records`, or first list found)
- **Advanced field mapping** with dot-notation (`country.name`), array
wildcards (`newsType[*].name`), type hints, and default values
- Optional content template rendering (`"Title: {title}\nBody: {body}"`)
- HTML stripping for content fields
- Stable document IDs via `hash128` from a configurable ID field or
auto-generated from item content
- Pydantic configuration schema with automatic coercion of UI string
inputs to dicts/lists
### Backend Registration (`rag/svr/sync_data_source.py`,
`common/constants.py`, `common/data_source/config.py`)
- `REST_API` sync class wired into RAGFlow's `func_factory`
- Full sync (`load_from_state`) and incremental polling (`poll_source`)
support
- Credentials and config passed from task to connector following
existing patterns (MySQL, SeaFile, etc.)
### Test Connection Endpoint (`api/apps/connector_app.py`)
- `POST /v1/connector/<id>/test` validates config schema,
authentication, and API connectivity without triggering a sync
- Clear error messages for auth failures vs. config issues
### Frontend UI (`web/src/pages/user-setting/data-source/constant/`)
- **Postman-style configuration:** Base URL, Query Parameters (key=value
per line), Auth, Content Fields, Metadata Fields, Pagination Type
- Auth-type-aware form: fields for API key header/value, Bearer token,
or Basic username/password appear only when relevant
- **Advanced Settings** toggle for: Custom Headers, Max Pages, Request
Delay, Poll Timestamp Field, Request Body (POST)
- Connector icon (SVG) and i18n strings (English)
- **"Test Connection"** button to validate before syncing
---
## Controls & Safety
- Configurable max pages safety cap (default: 1000, adjustable in UI)
- Configurable request delay between pages (default: 0.5s, adjustable in
UI)
- Auth errors (401/403) fail immediately without retries; transient
errors retry with exponential backoff
- Diagnostic logging: auth setup confirmation, request details on
failure, content field extraction status
---
## Type of change
- [x] New Feature (non-breaking change which adds functionality)
##Visual Screenshots of Features
<img width="482" height="510" alt="Screenshot 2026-03-11 at 5 19 52 PM"
src="https://github.com/user-attachments/assets/dcb7ab4a-1622-44f3-bb02-d6f0527314c4"
/>
(Connector can be configured within the external data sources tab)
Configuration Parameters:
<img width="661" height="682" alt="Screenshot 2026-03-11 at 5 20 46 PM"
src="https://github.com/user-attachments/assets/5e154e71-4ab5-4872-bfb2-04f02b73c18a"
/>
<img width="661" height="682" alt="Screenshot 2026-03-11 at 5 20 54 PM"
src="https://github.com/user-attachments/assets/00cb14b7-0bcf-4b94-9d71-34e93369ecb2"
/>
Connection can be tested before attaching to dataset:
<img width="981" height="681" alt="Screenshot 2026-03-11 at 5 21 40 PM"
src="https://github.com/user-attachments/assets/aaa6eeeb-89a7-4349-bc34-2423bf8be9ee"
/>
Ingestion tested with API connector (works perfectly fine):
<img width="1062" height="705" alt="Screenshot 2026-03-11 at 5 22 30 PM"
src="https://github.com/user-attachments/assets/afcd0d58-cadd-4152-badc-d2f14d96fbec"
/>
Search & Retrieval works as well with metadata flow:
<img width="1062" height="705" alt="Screenshot 2026-03-11 at 5 23 05 PM"
src="https://github.com/user-attachments/assets/d41ee935-dcf7-4456-b317-22a76ca032c0"
/>
---------
Co-authored-by: Ahmad Intisar <ahmadintisar@Ahmads-MacBook-M4-Pro.local>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
### What problem does this PR solve?
As title
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
Signed-off-by: Jin Hai <haijin.chn@gmail.com>
### Related issues
Closes#14781
### What problem does this PR solve?
Some retrieval endpoints accepted caller-supplied `tenant_rerank_id` and
resolved it through `get_model_config_by_id(...)`. That helper loaded
`TenantLLM` rows by global database id and returned decoded model
configuration without checking whether the model belonged to the
authenticated tenant or the dataset owner tenant.
This meant dataset access was validated, but rerank-model selection was
not. A caller who knew or could guess another tenant's
`tenant_rerank_id` could attempt retrieval with a foreign rerank model
config, creating a cross-tenant authorization gap for model usage.
This PR closes that gap by making `tenant_rerank_id` resolution
tenant-aware across the retrieval paths that accept it.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
- [ ] New Feature (non-breaking change which adds functionality)
- [ ] Documentation Update
- [ ] Refactoring
- [ ] Performance Improvement
- [ ] Other (please describe):
### Solution
- Extend `get_model_config_by_id(...)` to accept an optional
`allowed_tenant_ids` set and reject `TenantLLM` rows whose `tenant_id`
is outside that set.
- Pass the allowed tenant scope from retrieval endpoints that accept
`tenant_rerank_id`:
- `api/apps/sdk/doc.py`
- `api/apps/sdk/session.py`
- `api/apps/services/dataset_api_service.py`
- Use the authenticated tenant plus dataset-owner tenant ids already
derived by each retrieval flow as the authorization boundary for rerank
model selection.
- Add focused unit coverage to assert unauthorized `tenant_rerank_id`
values are rejected and that the allowed tenant set is propagated
correctly.
### Testing
- `python -m py_compile` on:
- `api/db/joint_services/tenant_model_service.py`
- `api/apps/services/dataset_api_service.py`
- `api/apps/sdk/doc.py`
- `api/apps/sdk/session.py`
- Added unit tests in:
-
`test/testcases/test_http_api/test_file_management_within_dataset/test_doc_sdk_routes_unit.py`
-
`test/testcases/test_http_api/test_session_management/test_session_sdk_routes_unit.py`
### Notes for reviewers
- This change is intentionally narrow: it affects only the
`tenant_rerank_id` path, not the normal `rerank_id` name-based
resolution path.
- Local lint/syntax checks passed.
- Full pytest execution could not be completed in this environment
because the local test runtime is missing `strenum`, so the route-test
files fail during collection before exercising the updated cases.
---------
Co-authored-by: jony376 <jony376@gmail.com>
### What problem does this PR solve?
A draft 0.25.3 release note.
### Type of change
- [x] Documentation Update
Co-authored-by: Jin Hai <haijin.chn@gmail.com>
### What problem does this PR solve?
The LocalAI Go driver landed in #14809 and Embed landed in #14811.
`Rerank` was left as a stub that returns `"not implemented"`. This PR
fills the gap.
LocalAI exposes a public rerank endpoint at `<tenant-url>/v1/rerank`
with a Cohere-shaped request and response (`{model, query, documents,
top_n}` → `{results: [{index, relevance_score}]}`). The Python side has
had `LocalAIRerank` in `rag/llm/rerank_model.py` for a long time. Until
this PR, a tenant who wanted to use LocalAI for reranking in the Go
layer got `"not implemented"`.
### What this PR includes
- `conf/models/localai.json`: add `"rerank": "rerank"` under
`url_suffix` so the driver can build the URL from config. This matches
the `URLSuffix.Rerank` field already used by aliyun and siliconflow.
- `internal/entity/models/localai.go`: replace the `Rerank` stub with a
real implementation that POSTs to `/v1/rerank`. Adds local
request/response types `localAIRerankRequest` and
`localAIRerankResponse`.
No factory change. No interface change.
### How the implementation works
- Validate the model name and resolve the tenant-supplied base URL with
the existing `resolveBaseURL` helper.
- Wrap the request with `context.WithTimeout(nonStreamCallTimeout)` so
the call has a clear deadline. Same pattern `ChatWithMessages`,
`ListModels`, and `Embed` already use in this file.
- Only set the `Authorization` header when a non-empty API key was
supplied. LocalAI accepts an empty key by default, so this preserves the
optional-auth contract.
- Default `top_n` to `len(documents)` when `rerankConfig.TopN == 0`,
matching the existing Aliyun and SiliconFlow rerank implementations.
- Validate every `results[].index` against `len(documents)`. If the
upstream returns an out-of-range index, fail clearly instead of silently
writing past the slice.
- An empty `documents` slice returns `&RerankResponse{}` with no HTTP
call.
- Non-200 responses propagate the upstream status line and body.
### Note on stacking
This PR builds on #14809 (LocalAI driver) and #14811 (LocalAI Embed).
Until both merge, this PR's diff on GitHub will include all three
commits. After #14809 and #14811 land on `main`, GitHub will auto-reduce
this PR to only the `Rerank` changes (one commit, ~99 line diff in
`localai.go` plus 1 line in `localai.json`).
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
### How was this tested?
- `go build ./internal/entity/models/...` returns exit 0 on go 1.25 (the
`go.mod` minimum).
- The full method set on `LocalAIModel` still matches the `ModelDriver`
interface.
- Pattern parity with the existing Aliyun Rerank
(`internal/entity/models/aliyun.go`) and SiliconFlow Rerank
(`internal/entity/models/siliconflow.go`) implementations.
Closes#14812
Depends on #14809, #14811
Tracking: #14736
Co-authored-by: Jin Hai <haijin.chn@gmail.com>
### What problem does this PR solve?
- Update version tags in README files (including translations) from
v0.25.2 to v0.25.3
- Modify Docker image references and documentation to reflect new
version
- Update version badges and image descriptions
- Maintain consistency across all language variants of README files
### Type of change
- [x] Documentation Update
### What problem does this PR solve?
RAGFlow(user)> ocr with 'hunyuanocr@test@gitee' file './picture.png'
+----------------------------------------------------------+
| text |
+----------------------------------------------------------+
| 生活不是等待风暴过去，而是学会在雨中翩翩起舞。
——佚名 |
+----------------------------------------------------------+
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
---------
Signed-off-by: Jin Hai <haijin.chn@gmail.com>
### What problem does this PR solve?
Fixes#14884
The ZhipuAI Go driver in `internal/entity/models/zhipu-ai.go` had a stub
`ListModels` method that always returned `"zhipu-ai, no such method"`.
The DeepSeek, Gitee, NVIDIA, OpenAI, SiliconFlow, and OpenRouter drivers
in the same package already implement `ListModels` against the
OpenAI-compatible `/models` endpoint, and the model picker UI relies on
it. This PR brings ZhipuAI in line with that pattern.
### Changes
- `internal/entity/models/zhipu-ai.go`: implement
`ZhipuAIModel.ListModels`.
- Resolve region with default fallback.
- GET `${BaseURL[region]}/${URLSuffix.Models}` (resolves to
`https://open.bigmodel.cn/api/paas/v4/models` with the default region).
- Send `Authorization: Bearer <api_key>` when an API key is configured.
Omit the header when the key is empty, so an unauthenticated caller gets
a clear `401` from upstream.
- Surface non-200 responses with the upstream status line and body,
matching the other Go drivers.
- Parse the response via the package-level `DSModelList` / `DSModel`
types already used by DeepSeek, Gitee, and SiliconFlow.
- When the response includes `owned_by`, render the entry as
`id@owned_by`, matching the convention of Gitee and SiliconFlow.
- `conf/models/zhipu-ai.json`: add `"models": "models"` to `url_suffix`.
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
### What problem does this PR solve?
Fix: Set embedded models during form initialization.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
### What problem does this PR solve?
Add a Go driver for LongCat (Meituan, https://longcat.chat), one of the
unchecked providers on the umbrella tracking issue #14736. LongCat
exposes an OpenAI-compatible REST API at
`https://api.longcat.chat/openai/v1` with three public chat models
including `LongCat-Flash-Thinking`, a reasoning model that returns
chain-of-thought in `reasoning_content` (OpenAI o-series shape).
Until this PR, a tenant who configured `longcat` as a model provider in
the Go layer fell through to the default branch of
`internal/entity/models/factory.go` and got the dummy driver.
### What this PR includes
- New `internal/entity/models/longcat.go` with a `LongCatModel`
implementing the `ModelDriver` interface.
- New `conf/models/longcat.json` with the 3 public chat models
(Flash-Chat, Flash-Lite, Flash-Thinking) and `url_suffix` for `chat` and
`models`.
- `factory.go`: route `"longcat"` to `NewLongCatModel`.
Method coverage:
- `ChatWithMessages`: `POST /openai/v1/chat/completions`, non-streaming
- `ChatStreamlyWithSender`: SSE stream against the same endpoint
- `ListModels` / `CheckConnection`: `GET /openai/v1/models`
- **Reasoning extraction**: `message.reasoning_content` (non-stream) and
`delta.reasoning_content` (stream) flow into
`ChatResponse.ReasonContent` / the sender's second arg. Matches the
OpenAI o-series convention also used by kimi-k2.6 and DeepSeek-R1.
- **`reasoning_effort` propagation**: `ChatConfig.Effort` → request body
`reasoning_effort` (LongCat-Flash-Thinking honors it; non-reasoning
models ignore it).
- `Embed` / `Rerank` / `Balance` / `TranscribeAudio` / `AudioSpeech` /
`OCRFile` return `"no such method"` (LongCat does not expose any of
these surfaces).
No interface change. No new dependencies.
### How was this tested?
**21 unit tests** in `internal/entity/models/longcat_test.go` — all
pass:
```
$ go test -vet=off -run TestLongCat -count=1 -v ./internal/entity/models/...
=== RUN TestLongCatName
--- PASS: TestLongCatName (0.00s)
=== RUN TestLongCatChatHappyPath
--- PASS: TestLongCatChatHappyPath (0.00s)
=== RUN TestLongCatChatExtractsReasoningContent
--- PASS: TestLongCatChatExtractsReasoningContent (0.00s)
=== RUN TestLongCatChatPropagatesReasoningEffort
--- PASS: TestLongCatChatPropagatesReasoningEffort (0.00s)
=== RUN TestLongCatChatOmitsReasoningEffortWhenUnset
--- PASS: TestLongCatChatOmitsReasoningEffortWhenUnset (0.00s)
=== RUN TestLongCatChatRequiresAPIKey
--- PASS: TestLongCatChatRequiresAPIKey (0.00s)
=== RUN TestLongCatChatRequiresMessages
--- PASS: TestLongCatChatRequiresMessages (0.00s)
=== RUN TestLongCatChatRejectsHTTPError
--- PASS: TestLongCatChatRejectsHTTPError (0.00s)
=== RUN TestLongCatStreamHappyPath
--- PASS: TestLongCatStreamHappyPath (0.00s)
=== RUN TestLongCatStreamExtractsReasoningContent
--- PASS: TestLongCatStreamExtractsReasoningContent (0.00s)
=== RUN TestLongCatStreamRejectsExplicitFalse
--- PASS: TestLongCatStreamRejectsExplicitFalse (0.00s)
=== RUN TestLongCatStreamRequiresSender
--- PASS: TestLongCatStreamRequiresSender (0.00s)
=== RUN TestLongCatStreamFailsWithoutTerminal
--- PASS: TestLongCatStreamFailsWithoutTerminal (0.00s)
=== RUN TestLongCatListModelsHappyPath
--- PASS: TestLongCatListModelsHappyPath (0.00s)
=== RUN TestLongCatListModelsRequiresAPIKey
--- PASS: TestLongCatListModelsRequiresAPIKey (0.00s)
=== RUN TestLongCatCheckConnectionDelegatesToListModels
--- PASS: TestLongCatCheckConnectionDelegatesToListModels (0.00s)
=== RUN TestLongCatEmbedReturnsNoSuchMethod
--- PASS: TestLongCatEmbedReturnsNoSuchMethod (0.00s)
=== RUN TestLongCatRerankReturnsNoSuchMethod
--- PASS: TestLongCatRerankReturnsNoSuchMethod (0.00s)
=== RUN TestLongCatBalanceReturnsNoSuchMethod
--- PASS: TestLongCatBalanceReturnsNoSuchMethod (0.00s)
=== RUN TestLongCatAudioOCRReturnNoSuchMethod
--- PASS: TestLongCatAudioOCRReturnNoSuchMethod (0.00s)
PASS
ok ragflow/internal/entity/models 0.020s
```
`go build ./internal/entity/models/...` exits 0 on go 1.25.
**Live integration test** against `api.longcat.chat`:
```
=== RUN TestLongCatLiveSmoke
[OK] Name() = "longcat"
[OK] CheckConnection
[OK] ListModels: 5 models -> [LongCat-Flash-Lite LongCat-Flash-Chat LongCat-Flash-Thinking-2601 LongCat-Flash-Omni-2603 LongCat-2.0-Preview]
[OK] Chat (Flash-Chat) answer="Got it! Let me know if you" reason=""
[OK] Chat (Flash-Thinking) answer len=443 head="To find 15 % of 80, follow these steps:\n\n1. **Convert the percentage to a frac..."
ReasonContent len=557 head="The user asks: \"15% of 80?\" They want step by step reasoning and final answer in \\boxed{}. So we need to compute 15% of ..."
[OK] Stream content: 78 chunks, 351 chars
[OK] Stream reasoning: 107 chunks, 537 chars
[OK] Balance returns longcat, no such method
[OK] Embed returns longcat, no such method
[OK] Rerank returns longcat, no such method
LONGCAT LIVE SMOKE PASSED
--- PASS: TestLongCatLiveSmoke (31.01s)
```
What the live run proves on the wire:
- Auth header (`Bearer <key>`) is accepted by `api.longcat.chat`.
- `/openai/v1/models` parser handles the real 5-model response (note:
live API returns versioned aliases `LongCat-Flash-Thinking-2601`,
`LongCat-Flash-Omni-2603`, `LongCat-2.0-Preview` plus the un-versioned
`LongCat-Flash-Chat` and `LongCat-Flash-Lite`).
- Non-stream chat against `LongCat-Flash-Chat`: visible answer parses
correctly, `ReasonContent` correctly empty.
- Non-stream chat against `LongCat-Flash-Thinking`: 443-char answer
flows into `Answer`, 557-char chain-of-thought flows into
`ReasonContent` via the new `message.reasoning_content` extraction.
- Streaming chat against `LongCat-Flash-Thinking`: 107 reasoning chunks
(537 chars) reach the sender's second arg via `delta.reasoning_content`;
78 content chunks (351 chars) reach the first arg. Before this code, the
reasoning chunks would have been silently dropped.
- All sentinel methods (Balance, Embed, Rerank, audio/OCR) return the
documented `"no such method"` strings.
### Note on PR history
This branch was previously named for LocalAI work which is now
consolidated into PR #14813. The branch was reset to `upstream/main` and
rebuilt for LongCat. The diff against `main` is a clean +969 lines
across 4 files.
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
Tracking: #14736
---------
Co-authored-by: Jin Hai <haijin.chn@gmail.com>
### What problem does this PR solve?
The delete /graph is duplicated of
`/datasets/<dataset_id>/<index_type>`, delete it.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
### What problem does this PR solve?
add new testing suite for the new restful api endpoints meant to replace
http and web api tests
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
- [x] Other (please describe): test
### What problem does this PR solve?
Add a Go driver for Novita.ai (https://novita.ai), one of the unchecked
providers on the umbrella tracking issue #14736. Novita exposes an
OpenAI-compatible REST API at `https://api.novita.ai/v3/openai` and
proxies a large catalog of third-party models (DeepSeek, Llama, Qwen3,
Kimi, Gemma, Mistral, MiniMax, GLM, etc.) behind a single OpenAI-shaped
surface — 102 models live at the time of writing.
Until this PR, a tenant who configured `novita` as a model provider in
the Go layer fell through to the default branch of
`internal/entity/models/factory.go` and got the dummy driver.
### What this PR includes
- New `internal/entity/models/novita.go` with a `NovitaModel`
implementing the `ModelDriver` interface (~520 lines).
- New `conf/models/novita.json` with 7 representative chat models
(DeepSeek-V4, Llama-3.3-70B, Qwen3-30B/235B reasoning, Kimi-K2,
Gemma-3-27B, Mistral-Nemo).
- `factory.go`: route `"novita"` to `NewNovitaModel`.
- `internal/entity/models/novita_test.go`: 23 unit tests.
### Notable design point: `<think>...</think>` reasoning extraction
Novita-routed reasoning models like `qwen3-*` and `deepseek-r1-*` embed
their chain-of-thought **inline inside content as `<think>...</think>`
tags**, rather than in a separate `reasoning_content` field. Verified
live by probing `api.novita.ai`:
```
content head 200: <think>
Okay, let's see. I need to find 15% of 80. Hmm, percentages can sometimes be tricky, but I think
content tail 100: h, that works.
Alternatively, 0.15 × 80. If I move the decimal two places to the left for </think>
```
Without handling, a tenant picking qwen3 via Novita would see raw
`<think>` tags in their UI answer — different from every other reasoning
provider in the Go layer.
The driver detects those tags and routes the inner text to
`ChatResponse.ReasonContent` (non-stream) or the sender's second arg
(stream), keeping the visible answer clean of tag clutter:
- **`splitNovitaThink`** — scans a complete content string. Used by the
non-streaming path. Handles multiple `<think>` blocks, unclosed tags
(the model got cut off mid-reasoning), pure-text content with no tags.
- **`novitaThinkExtractor`** — stateful streaming version. Buffers
trailing bytes that might be the start of a tag (e.g. `<thi` held back
when the next chunk completes `nk>`), then emits segments in routing
order so callers can pipe them to a UI. Tested with byte-level chunk
boundaries and tag-spanning scenarios.
### Method coverage
| Method | Behavior |
|---|---|
| `ChatWithMessages` | `POST /v3/openai/chat/completions`, `<think>`
extraction on response |
| `ChatStreamlyWithSender` | SSE stream, stateful `<think>` extraction
across deltas |
| `ListModels` / `CheckConnection` | `GET /v3/openai/models` (102 live)
|
| `Embed` / `Rerank` / `Balance` / `TranscribeAudio` / `AudioSpeech` /
`OCRFile` | `"no such method"` — Novita's OpenAI-compatible surface does
not expose any |
No interface change. No new dependencies.
### How was this tested?
**23 unit tests** in `internal/entity/models/novita_test.go` — all pass:
```
$ go test -vet=off -run "TestNovita|TestSplitNovita" -count=1 ./internal/entity/models/...
ok ragflow/internal/entity/models 0.020s
```
Coverage:
- `splitNovitaThink` (5 cases: pure text, single block, leading text,
multiple blocks, unclosed tag)
- `novitaThinkExtractor` (6 cases: single-chunk, opening tag span,
closing tag span, byte-level chunking, no tags, lone `<` not as tag
start)
- `ChatWithMessages`: pure text, with `<think>` tags, missing API key,
empty messages, HTTP error
- `ChatStreamlyWithSender`: tag-stripping with spanning deltas, pure
content, sender-required, stream-true-required
- `ListModels` / `CheckConnection` (happy paths)
- All sentinel methods
`go build ./internal/entity/models/...` exits 0 on go 1.25.
**Live integration test** against `api.novita.ai/v3/openai`:
```
=== RUN TestNovitaLiveSmoke
[OK] Name() = "novita"
[OK] CheckConnection
[OK] ListModels: 102 models (showing first 6) [deepseek/deepseek-v4-pro deepseek/deepseek-v4-flash deepseek/deepseek-v3.2 xiaomimimo/mimo-v2.5-pro moonshotai/kimi-k2.6 zai-org/glm-5.1]
[OK] Chat (llama-3.3) answer="ok" reason=""
[OK] Chat (qwen3) answer len=0 head=""
ReasonContent len=1657 head="Okay, so I need to figure out what 15% of 80 is. Hmm, percentages can sometimes trip me up, but let ..."
[OK] Stream content: 0 chunks, 0 chars; reasoning: 600 chunks, 1667 chars
[OK] Embed/Rerank/Balance/TranscribeAudio/AudioSpeech/OCRFile all return "novita, no such method"
NOVITA LIVE SMOKE PASSED
--- PASS: TestNovitaLiveSmoke (26.18s)
```
What the live run proves on the wire:
- Auth (`Bearer <key>`) accepted by `api.novita.ai`.
- `/v3/openai/models` parser handles the real 102-model response.
- Non-stream chat against `meta-llama/llama-3.3-70b-instruct`: clean
string answer, empty ReasonContent (non-reasoning model, pure-text
path).
- Non-stream chat against `qwen/qwen3-30b-a3b-fp8`: 1657-char reasoning
extracted from `<think>...</think>` and routed to
`ChatResponse.ReasonContent`. Visible answer is 0 chars in this run
because qwen3 spent its 600-token budget entirely on reasoning before
reaching the answer phase — that's the model's behavior, not a driver
bug. The important thing: **no `<think>` tags leaked into the visible
Answer field**.
- Streaming against qwen3: 600 reasoning chunks (1667 chars) emitted via
the sender's 2nd arg across SSE deltas; **no `<think>` tag fragments
leaked into the content channel** despite tag boundaries crossing chunk
boundaries on the wire.
- All 6 sentinel methods return the documented `"no such method"`
strings.
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
Tracking: #14736
… modal
- Add vertical resizing functionality for the text field
### What problem does this PR solve?
_Fix the issue where the text content of the knowledge base editing
parsing block is too long to scroll._
<img width="701" height="775" alt="image"
src="https://github.com/user-attachments/assets/b258422e-fbc1-466d-abab-062e642c21d5"
/>
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
Co-authored-by: chenyun <chenyun@chenyundemacbook-pro.local>
### What problem does this PR solve?
Fix delete graphrag not take effect in UI
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
### What problem does this PR solve?
The Baidu (Qianfan) `Encode` method silently swallowed malformed
responses. If a `data[]` item from the API was missing a field (`index`,
`embedding`, or unexpected shape), the loop did `continue` instead of
returning an error, leaving `nil` entries in the result slice. Callers
got back partial results with no indication anything went wrong, which
then crashes downstream consumers when they try to use a `nil` vector.
Concrete gaps fixed:
- No count-mismatch check between `data` length and input texts (only
checked for empty)
- No duplicate-index detection (a duplicate would silently overwrite)
- No missing-index final scan
- No empty-embedding rejection
- No per-call context timeout
- `EmbeddingConfig.Dimension` (added in #14735) was not propagated
This PR replaces `map[string]interface{}` parsing with a typed
`baiduEmbeddingResponse` struct, applies the standard four-layer
validation (count → out-of-range → duplicate → empty → final
missing-index scan), adds `context.WithTimeout(nonStreamCallTimeout)`,
and forwards `embeddingConfig.Dimension` as the `dimensions` parameter
(Baidu Qianfan v2 uses an OpenAI-compatible API).
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
## Problem
When parsing DOCX files with many tables, DeepDOC generates chunks
containing only empty HTML table tags, such as:
```html
<table><tr><td></td></tr><tr><td></td></tr><tr><td></td></tr><tr><td></td></tr></table>
```
After the regex cleanup at `task_executor.py:584`, this becomes `" "`
(whitespace only).
The guard at line 585 (`if not c`) only catches empty strings `""`, but
whitespace strings are truthy in Python and pass through. When sent to
Zhipu `embedding-3` API, it rejects them with error 1213:
`未正常接收到prompt参数`.
## Root Cause
```python
c = re.sub(r"</?(table|td|caption|tr|th)( [^<>]{0,12})?>", " ", c)
if not c: # ← only catches "", not " " / "\n" / "\t"
c = "None"
```
Verified with Zhipu `embedding-3`:
| Input | Result |
|---|---|
| `""` | error 1213 |
| `" "` | error 1213 |
| `"\n"` | error 1213 |
| `"None"` | OK |
## Fix
```diff
- if not c:
+ if not c.strip():
c = "None"
```
## Testing
Reproduced with a 678KB DOCX file (166 tables, 270 chunks). Chunk #89 is
the empty table above. After fix, `"None"` is sent instead and embedding
succeeds.
---------
Co-authored-by: Kevin Hu <kevinhu.sh@gmail.com>
Closes#14768
### What problem does this PR solve?
The `list_chats` and `list_searches` REST API endpoints did not enforce
authorization on the `owner_ids` query parameter. Any authenticated user
could pass arbitrary tenant IDs to `owner_ids` and retrieve chats or
search apps belonging to other tenants they are not a member of.
This PR resolves the issue by:
1. Looking up the current user's authorized tenants via
`TenantService.get_joined_tenants_by_user_id` and rejecting any
`owner_ids` that fall outside that set.
2. When no `owner_ids` are provided, scoping the query to only the
user's authorized tenants instead of returning an unfiltered result.
3. Adding unit tests that verify unauthorized `owner_ids` are rejected
with `OPERATING_ERROR`.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
What problem does this PR solve?
In rag/app/audio.py, the supported audio extensions list contains
duplicate entries: .wav appears twice (positions 3 and 5) and .aac
appears twice (positions 6 and 14). While this does not affect runtime
behavior, it is redundant and makes the code harder to maintain.
This PR removes the duplicate entries to keep the list clean and
consistent.
Type of change
- [X] Bug Fix (non-breaking change which fixes an issue)
### What problem does this PR solve?
```
RAGFlow(user)> chat with 'glm-ocr@test@zhipu-ai' message 'what is this'
CLI error: expect model glm-ocr@zhipu-ai is a chat or multimodal model
```
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
Signed-off-by: Jin Hai <haijin.chn@gmail.com>
### What problem does this PR solve?
Feat: When a Wait Node precedes a Message Node within a Loop Node, the
outgoing message is split into two separate messages.
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
### What problem does this PR solve?
OpenAI model catalogs used in provider selection flows were missing the
latest GPT models (`gpt-5.5` and `gpt-5.4`).
Because model availability is driven by seeded catalog data
(`conf/llm_factories.json` → DB seed → API response), these models were
not selectable in the UI or `/llm/list` responses.
This PR updates and synchronizes the OpenAI catalog definitions across
configuration sources and ensures the new models are correctly exposed
through the API layer and validated in tests.
---
### Type of change
* [x] New Feature (non-breaking change which adds functionality)
---
### Changes Made
* Added `gpt-5.5` and `gpt-5.4` to OpenAI catalog definitions in:
* `conf/llm_factories.json`
* `conf/models/openai.json` (chat + vision support)
* Ensured consistency between DB-seeded factory config and provider
model configuration
* Updated test coverage in:
* `test_llm_list_unit.py`
* seeded OpenAI catalog entries
* added response-level assertion validating `/llm/list` includes both
new model IDs under OpenAI grouping
---
### Root Cause
OpenAI model listings in selection flows are generated from catalog data
seeded via `conf/llm_factories.json`.
The catalog had not been updated to include the latest GPT models,
resulting in missing availability in UI and API responses.
---
### Testing
* Created isolated test environment:
* `python -m venv .venv-review`
* installed `pytest`
* Ran targeted and full test suite:
* `test_list_app_grouping_availability_and_merge`: ✅ passed
* Full `test_llm_list_unit.py`: ✅ 10 passed
---
### Risks / Limitations
* Adding models to the catalog does not guarantee upstream provider
availability or account entitlement.
* Environments with pre-seeded DB catalogs may require reseed or refresh
to reflect updated configuration.
---
### Notes
* Changes are minimal and scoped strictly to catalog configuration and
related test coverage.
* Ensures `/llm/list` API remains aligned with expected latest OpenAI
model availability.
* Closes#14827
### What problem does this PR solve?
This PR completes the Jina provider
**The following functionalities are now supported:**
**Jina:**
- [ ] Chat / Stream Chat (Not available for now: [(Jina chat API
docs)](https://api.jina.ai/docs#/Search%20Foundation%20Models/chat_completions_v1_chat_completions_post))
- [x] Embedding
- [x] Rerank
- [x] Model listing
- [x] Provider connection checking
- [ ] ~~Balance~~
**Verified examples from the CLI:**
```plaintext
RAGFlow(user)> embed text 'walkerwhat' 'jumperwho' with 'jina-embeddings-v2-base-en@test@jina' dimension 16
+-----------+-------+
| dimension | index |
+-----------+-------+
| 768 | 0 |
| 768 | 1 |
+-----------+-------+
RAGFlow(user)> rerank query 'what is rag' document 'rag is retrieval augment generation' 'rag need llm' 'famous rag project includes ragflow' with 'jina-reranker-v2-base-multilingual@test@jina' top 3;
+-------+-----------------+
| index | relevance_score |
+-------+-----------------+
| 0 | 0.74316794 |
| 2 | 0.18713269 |
| 1 | 0.15817434 |
+-------+-----------------+
RAGFlow(user)> list supported models from 'jina' 'test'
+---------------------------------------------+
| model_name |
+---------------------------------------------+
| Jina AI: Jina VLM |
| Jina AI: Jina Reranker v3 |
| Jina AI: Jina Code Embeddings 0.5b |
| Jina AI: Jina Code Embeddings 1.5b |
| Jina AI: Jina Embeddings v4 |
| Jina AI: Jina Reranker M0 |
| Jina AI: ReaderLM v2 |
| Jina AI: Jina Clip v2 |
| Jina AI: Jina Embeddings v3 |
| Jina AI: Jina Colbert v2 |
| Jina AI: Reader LM 0.5b |
| Jina AI: Reader LM 1.5b |
| Jina AI: Jina Reranker v2 Base Multilingual |
| Jina AI: Jina Clip v1 |
| Jina AI: Jina Reranker v1 Tiny EN |
| Jina AI: Jina Reranker v1 Turbo EN |
| Jina AI: Jina Reranker v1 Base EN |
| Jina AI: Jina Colbert v1 EN |
| Jina AI: Jina Embeddings v2 Base ES |
| Jina AI: Jina Embeddings v2 Base Code |
| Jina AI: Jina Embeddings v2 Base DE |
| Jina AI: Jina Embeddings v2 Base ZH |
| Jina AI: Jina Embeddings v2 Base EN |
| Jina AI: Jina Embedding B EN v1 |
| Jina AI: Jina Embeddings v5 Text Small |
| Jina AI: Jina Embeddings v5 Omni Small |
| Jina AI: Jina Embeddings v5 Omni Nano |
| Jina AI: Jina Embeddings v5 Text Nano |
+---------------------------------------------+
RAGFlow(user)> check instance 'test' from 'jina'
SUCCESS
```
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
### What problem does this PR solve?
The Mistral Go driver landed in #14805 with chat, list models, and check
connection. `Embed` was left as a stub that returns `"not implemented"`.
This PR fills the gap.
`conf/models/mistral.json` did not list any embedding model out of the
box, so a tenant who wanted to use Mistral end to end (chat +
embeddings) could not run an embedding call. This PR adds
`mistral-embed` to the config and a real `/v1/embeddings`
implementation.
### What this PR includes
- `conf/models/mistral.json`: add `"embedding": "embeddings"` under
`url_suffix` so the driver can build the URL from config (matches the
`URLSuffix.Embedding` field already used by openai, siliconflow,
zhipu-ai), and add a `mistral-embed` entry under `models`
(1024-dimensional vectors, 8192 max input tokens).
- `internal/entity/models/mistral.go`: replace the `Embed` stub with a
real implementation that POSTs to `/v1/embeddings`. Adds local response
types `mistralEmbeddingData` and `mistralEmbeddingResponse`.
No factory change. No interface change.
### How the implementation works
- Validate `apiConfig`, the API key, and the model name. Use the
existing `baseURLForRegion` helper so an unknown region fails fast with
a clear error.
- Wrap the request with `context.WithTimeout(nonStreamCallTimeout)` so
the call has a clear deadline. Same pattern as `ChatWithMessages` and
`ListModels` already use in this file.
- Send all input texts in one request. The Mistral API accepts the
`input` field as an array.
- Parse `data[*].embedding` and copy each slice into a `[]EmbeddingData`
indexed by `data[*].index` so the output order matches the input order
even if the API returns items in a different order.
- An empty input slice returns `[]EmbeddingData{}` with no HTTP call.
- Non-200 responses propagate the upstream status line and body.
- A final pass checks that every input slot got a vector. If any slot is
still empty, return a clear error so the caller does not silently use a
zero vector.
### Note on stacking
This PR builds on #14805 (the Mistral driver). Until #14805 merges, this
PR's diff on GitHub will include both that PR's commits and this one.
After #14805 lands on `main`, GitHub will auto-reduce this PR to only
the `Embed` changes (one commit, ~111 line diff in `mistral.go` plus 8
lines in `mistral.json`).
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
### How was this tested?
- `go build ./internal/entity/models/...` returns exit 0 on go 1.25 (the
`go.mod` minimum).
- The full method set on `MistralModel` still matches the `ModelDriver`
interface.
- Pattern parity with the existing OpenAI Embed implementation
(`internal/entity/models/openai.go`).
Closes#14806
Depends on #14805
Tracking: #14736
---------
Co-authored-by: Jin Hai <haijin.chn@gmail.com>
### What problem does this PR solve?
```
RAGFlow(user)> asr with 'glm-asr-2512@test@zhipu-ai' audio './speech.wav';
CLI error: zhipu, no such method
RAGFlow(user)> stream asr with 'glm-asr-2512@test@zhipu-ai' audio './speech.wav';
CLI error: zhipu, no such method
RAGFlow(user)> tts with 'glm-tts@test@zhipu-ai' text 'how are you';
CLI error: zhipu, no such method
RAGFlow(user)> stream tts with 'glm-tts@test@zhipu-ai' text 'how are you';
CLI error: zhipu, no such method
RAGFlow(user)> ocr with 'glm-ocr@test@zhipu-ai' file './test.log';
CLI error: zhipu, no such method
```
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
Signed-off-by: Jin Hai <haijin.chn@gmail.com>
### What problem does this PR solve?
The Upstage Go driver landed in #14817 with chat, list models, and check
connection. `Embed` was left as a stub that returns `"not implemented"`.
This PR fills the gap.
Upstage exposes an OpenAI-compatible embeddings endpoint at
`https://api.upstage.ai/v1/solar/embeddings` via the
`solar-embedding-1-large` family (`solar-embedding-1-large-query` for
queries, `solar-embedding-1-large-passage` for passages), and the Python
side has had `UpstageEmbed(OpenAIEmbed)` in `rag/llm/embedding_model.py`
for a long time targeting this same path. The existing
`conf/models/upstage.json` did not list any embedding model out of the
box, so a tenant who wanted to use Upstage end to end could not run an
embedding call. This PR fills the gap.
### What this PR includes
- `conf/models/upstage.json`: add `"embedding": "embeddings"` under
`url_suffix` so the driver can build the URL from config (matches the
`URLSuffix.Embedding` field already used by openai, mistral,
siliconflow, zhipu-ai), and add `solar-embedding-1-large-query` and
`solar-embedding-1-large-passage` entries under `models`.
- `internal/entity/models/upstage.go`: replace the `Embed` stub with a
real implementation that POSTs to `/v1/solar/embeddings`. Adds local
response types `upstageEmbeddingData` and `upstageEmbeddingResponse`.
No factory change. No interface change.
### How the implementation works
- Validate `apiConfig`, the API key, and the model name. Use the
existing `baseURLForRegion` helper so an unknown region fails fast with
a clear error.
- Wrap the request with `context.WithTimeout(nonStreamCallTimeout)` so
the call has a clear deadline. Same pattern as `ChatWithMessages` and
`ListModels` already use in this file.
- Send all input texts in one request. The Upstage API accepts the
`input` field as an array.
- Parse `data[*].embedding` and copy each slice into a `[]EmbeddingData`
indexed by `data[*].index` so the output order matches the input order
even if the API returns items in a different order.
- An empty input slice returns `[]EmbeddingData{}` with no HTTP call.
- Non-200 responses propagate the upstream status line and body.
- A final pass checks that every input slot got a vector. If any slot is
still empty, return a clear error so the caller does not silently use a
zero vector.
### Note on stacking
This PR builds on #14817 (the Upstage driver). Until #14817 merges, this
PR's diff on GitHub will include both that PR's commits and this one.
After #14817 lands on `main`, GitHub will auto-reduce this PR to only
the `Embed` changes (one commit, ~119 line diff in `upstage.go` plus ~15
lines in `upstage.json`).
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
### How was this tested?
- `go build ./internal/entity/models/...` returns exit 0 on go 1.25 (the
`go.mod` minimum).
- The full method set on `UpstageModel` still matches the `ModelDriver`
interface.
- Pattern parity with the existing Mistral Embed
(`internal/entity/models/mistral.go`) and OpenAI Embed
(`internal/entity/models/openai.go`) implementations.
Closes#14818
Depends on #14817
Tracking: #14736
---------
Co-authored-by: Jin Hai <haijin.chn@gmail.com>
### What problem does this PR solve?
This PR completes the Baichuan provider
**The following functionalities are now supported:**
**Baichuan:**
- [x] Chat / Stream Chat
- [x] Embedding
- [ ] ~~Rerank~~
- [ ] ~~Model listing~~
- [ ] ~~Provider connection checking~~
- [ ] ~~Balance~~
**Verified examples from the CLI:**
```plaintext
# Baichuan
RAGFlow(user)> embed text 'walkerwhat' 'jumperwho' with 'Baichuan-Text-Embedding@test@baichuan' dimension 16;
+-----------+-------+
| dimension | index |
+-----------+-------+
| 1024 | 0 |
| 1024 | 1 |
+-----------+-------+
AGFlow(user)> chat with 'Baichuan-M2@test@baichuan' message 'who r u'
Answer: I'm BaiChuan, a helpful AI assistant created by Baichuan-AI. I'm designed to be a knowledgeable, friendly, and reliable assistant for various tasks like answering questions, explaining concepts, writing content, and more. Feel free to ask me anything! 😊
Time: 1.637975
RAGFlow(user)> stream chat with 'Baichuan-M2@test@baichuan' message 'who r u'
Answer: I'm BaiChuan-m2, an AI assistant developed by Baichuan-AI. My purpose is to help you with a wide range of tasks by providing information, answering questions, solving problems, and assisting with creative projects. Think of me as a helpful digital companion! If you have any questions or need assistance, just let me know.😊
Time: 1.692321
```
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
- [x] Refactoring
This PR adds focused unit tests for aggregate_by_field in OceanBase
memory utilities to improve behavior coverage for real-world input
shapes.
- Adds test coverage for list-valued aggregation fields, including
whitespace trimming and skipping invalid list entries.
- Adds test coverage for scalar field values to ensure blank/non-string
values are ignored.
- Confirms aggregation output remains correct and stable for
mixed-quality message payloads.
### Why this helps
It strengthens regression protection for aggregation logic used by
memory retrieval flows, with no production code changes and minimal
review risk.
### What problem does this PR solve?
fix:
update null checks to use 'is None' for better clarity
replace RAGFlowSelect with SelectWithSearch in DebugContent
add max height and overflow to DialogContent in ParameterDialog
remove unused types from DataOperationsForm
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
Fixes#13817
### What problem does this PR solve?
The "knowledge graph construction" link on line 21 of
`docs/guides/dataset/run_retrieval_test.md` points to
`./construct_knowledge_graph.md`, which doesn't exist. The actual file
is at `./advanced/construct_knowledge_graph.md`.
### Type of change
- [x] Documentation Update
Signed-off-by: majiayu000 <1835304752@qq.com>
### What problem does this PR solve?
Add a Go driver for StepFun (阶跃星辰), one of the unchecked providers on
the umbrella tracking issue #14736.
Until this PR, a tenant who configured `stepfun` as a model provider in
the Go layer fell through to the default branch of
`internal/entity/models/factory.go` and got the dummy driver. Chat, list
models, and check connection all returned `"not implemented"` instead of
reaching the StepFun API.
The Python side has had StepFun registered in `rag/llm/__init__.py` as a
`SupportedLiteLLMProvider` with base URL `https://api.stepfun.com/v1`,
plus `StepFunCV` for vision and `StepFunSeq2txt` for ASR, but no Go
path. StepFun's chat API is OpenAI-compatible, so the implementation
pattern is the same as the merged Moonshot driver (#14433) and OpenAI
driver (#14605).
### What this PR includes
- New file `internal/entity/models/stepfun.go` with a `StepFunModel`
that implements the `ModelDriver` interface.
- `factory.go`: route the `"stepfun"` provider name to
`NewStepFunModel`.
- New `conf/models/stepfun.json` with the public StepFun chat models
(step-2-16k, step-1 family in 8k/32k/128k/256k context lengths,
step-1-flash, and the step-1v / step-1o vision models) and `url_suffix`
entries for `chat` and `models`.
### How the driver works
- StepFun exposes the OpenAI-compatible API at
`https://api.stepfun.com/v1`.
- `ChatWithMessages` and `ChatStreamlyWithSender` post to
`/chat/completions` in the same shape as the merged moonshot,
openrouter, and openai drivers.
- `ListModels` and `CheckConnection` call `/models` to list available
ids and confirm the API key works.
- `Embed` is left as `"not implemented"`. StepFun has not advertised a
public embeddings endpoint in the API reference linked from the umbrella
issue
(`https://platform.stepfun.com/docs/en/api-reference/chat/chat-completion-create`
is the chat endpoint), so any real implementation belongs in a separate
follow-up only after the endpoint is verified.
- `Rerank` and `Balance` return `"no such method"` because StepFun does
not expose either.
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
### How was this tested?
- `go build ./internal/entity/models/...` returns exit 0 with no errors
on go 1.25 (the `go.mod` minimum).
- Method set of `StepFunModel` matches the `ModelDriver` interface:
`NewInstance`, `Name`, `ChatWithMessages`, `ChatStreamlyWithSender`,
`Embed`, `Rerank`, `ListModels`, `Balance`, `CheckConnection`.
- Pattern parity with the merged moonshot (#14433), openai (#14605),
openrouter (#14652), and xai (#14550) drivers.
Closes#14814
Tracking: #14736
Bumps [urllib3](https://github.com/urllib3/urllib3) from 2.6.3 to 2.7.0.
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a
href="https://github.com/urllib3/urllib3/releases">urllib3's
releases</a>.</em></p>
<blockquote>
<h2>2.7.0</h2>
<h2>🚀 urllib3 is fundraising for HTTP/2 support</h2>
<p><a
href="https://sethmlarson.dev/urllib3-is-fundraising-for-http2-support">urllib3
is raising ~$40,000 USD</a> to release HTTP/2 support and ensure
long-term sustainable maintenance of the project after a sharp decline
in financial support. If your company or organization uses Python and
would benefit from HTTP/2 support in Requests, pip, cloud SDKs, and
thousands of other projects <a
href="https://opencollective.com/urllib3">please consider contributing
financially</a> to ensure HTTP/2 support is developed sustainably and
maintained for the long-haul.</p>
<p>Thank you for your support.</p>
<h2>Security</h2>
<p>Addressed high-severity security issues. Impact was limited to
specific use cases detailed in the accompanying advisories; overall user
exposure was estimated to be marginal.</p>
<ul>
<li>
<p>Decompression-bomb safeguards of the streaming API were bypassed:</p>
<ol>
<li>When <code>HTTPResponse.drain_conn()</code> was called after the
response had been read and decompressed partially. (Reported by <a
href="https://github.com/Cycloctane"><code>@Cycloctane</code></a>)</li>
<li>During the second <code>HTTPResponse.read(amt=N)</code> or
<code>HTTPResponse.stream(amt=N)</code> call when the response was
decompressed using the official <a
href="https://pypi.org/project/brotli/">Brotli</a> library. (Reported by
<a
href="https://github.com/kimkou2024"><code>@kimkou2024</code></a>)</li>
</ol>
<p>See GHSA-mf9v-mfxr-j63j for details.</p>
</li>
<li>
<p>HTTP pools created using
<code>ProxyManager.connection_from_url</code> did not strip sensitive
headers specified in <code>Retry.remove_headers_on_redirect</code> when
redirecting to a different host. (GHSA-qccp-gfcp-xxvc reported by <a
href="https://github.com/christos-spearbit"><code>@christos-spearbit</code></a>)</p>
</li>
</ul>
<h2>Deprecations and Removals</h2>
<ul>
<li>Used <code>FutureWarning</code> instead of
<code>DeprecationWarning</code> for better visibility of existing
deprecation notices. Rescheduled the removal of deprecated features to
version 3.0. (<a
href="https://redirect.github.com/urllib3/urllib3/issues/3763">urllib3/urllib3#3763</a>)</li>
<li>Removed support for end-of-life Python 3.9. (<a
href="https://redirect.github.com/urllib3/urllib3/issues/3720">urllib3/urllib3#3720</a>)</li>
<li>Removed support for end-of-life PyPy3.10. (<a
href="https://redirect.github.com/urllib3/urllib3/issues/4979">urllib3/urllib3#4979</a>)</li>
<li>Bumped the minimum supported pyOpenSSL version to 19.0.0. (<a
href="https://redirect.github.com/urllib3/urllib3/issues/3777">urllib3/urllib3#3777</a>)</li>
</ul>
<h2>Bugfixes</h2>
<ul>
<li>Fixed a bug where <code>HTTPResponse.read(amt=None)</code> was
ignoring decompressed data buffered from previous partial reads. (<a
href="https://redirect.github.com/urllib3/urllib3/issues/3636">urllib3/urllib3#3636</a>)</li>
<li>Fixed a bug where <code>HTTPResponse.read()</code> could cache only
part of the response after a partial read when
<code>cache_content=True</code>. (<a
href="https://redirect.github.com/urllib3/urllib3/issues/4967">urllib3/urllib3#4967</a>)</li>
<li>Fixed <code>HTTPResponse.stream()</code> and
<code>HTTPResponse.read_chunked()</code> to handle <code>amt=0</code>.
(<a
href="https://redirect.github.com/urllib3/urllib3/issues/3793">urllib3/urllib3#3793</a>)</li>
<li>Updated <code>_TYPE_BODY</code> type alias to include missing
<code>Iterable[str]</code>, matching the documented and runtime behavior
of chunked request bodies. (<a
href="https://redirect.github.com/urllib3/urllib3/issues/3798">urllib3/urllib3#3798</a>)</li>
<li>Fixed <code>LocationParseError</code> when paths resembling
schemeless URIs were passed to
<code>HTTPConnectionPool.urlopen()</code>. (<a
href="https://redirect.github.com/urllib3/urllib3/issues/3352">urllib3/urllib3#3352</a>)</li>
<li>Fixed <code>BaseHTTPResponse.readinto()</code> type annotation to
accept <code>memoryview</code> in addition to <code>bytearray</code>,
matching the <code>io.RawIOBase.readinto</code> contract and enabling
use with <code>io.BufferedReader</code> without type errors. (<a
href="https://redirect.github.com/urllib3/urllib3/issues/3764">urllib3/urllib3#3764</a>)</li>
</ul>
</blockquote>
</details>
<details>
<summary>Changelog</summary>
<p><em>Sourced from <a
href="https://github.com/urllib3/urllib3/blob/main/CHANGES.rst">urllib3's
changelog</a>.</em></p>
<blockquote>
<h1>2.7.0 (2026-05-07)</h1>
<h2>Security</h2>
<p>Addressed high-severity security issues.
Impact was limited to specific use cases detailed in the accompanying
advisories; overall user exposure was estimated to be marginal.</p>
<ul>
<li>
<p>Decompression-bomb safeguards of the streaming API were bypassed:</p>
<ol>
<li>When <code>HTTPResponse.drain_conn()</code> was called after the
response had been
read and decompressed partially.</li>
<li>During the second <code>HTTPResponse.read(amt=N)</code> or
<code>HTTPResponse.stream(amt=N)</code> call when the response was
decompressed
using the official <code>Brotli
<https://pypi.org/project/brotli/></code>__ library.</li>
</ol>
<p>See <code>GHSA-mf9v-mfxr-j63j
<https://github.com/urllib3/urllib3/security/advisories/GHSA-mf9v-mfxr-j63j></code>__
for details.</p>
</li>
<li>
<p>HTTP pools created using
<code>ProxyManager.connection_from_url</code> did not strip
sensitive headers specified in
<code>Retry.remove_headers_on_redirect</code> when
redirecting to a different host.
(<code>GHSA-qccp-gfcp-xxvc
<https://github.com/urllib3/urllib3/security/advisories/GHSA-qccp-gfcp-xxvc></code>__)</p>
</li>
</ul>
<h2>Deprecations and Removals</h2>
<ul>
<li>Used <code>FutureWarning</code> instead of
<code>DeprecationWarning</code> for better
visibility of existing deprecation notices. Rescheduled the removal of
deprecated features to version 3.0.
(<code>[#3763](https://github.com/urllib3/urllib3/issues/3763)
<https://github.com/urllib3/urllib3/issues/3763></code>__)</li>
<li>Removed support for end-of-life Python 3.9.
(<code>[#3720](https://github.com/urllib3/urllib3/issues/3720)
<https://github.com/urllib3/urllib3/issues/3720></code>__)</li>
<li>Removed support for end-of-life PyPy3.10.
(<code>[#4979](https://github.com/urllib3/urllib3/issues/4979)
<https://github.com/urllib3/urllib3/issues/4979></code>__)</li>
<li>Bumped the minimum supported pyOpenSSL version to 19.0.0.
(<code>[#3777](https://github.com/urllib3/urllib3/issues/3777)
<https://github.com/urllib3/urllib3/issues/3777></code>__)</li>
</ul>
<h2>Bugfixes</h2>
<ul>
<li>Fixed a bug where <code>HTTPResponse.read(amt=None)</code> was
ignoring decompressed
data buffered from previous partial reads.
(<code>[#3636](https://github.com/urllib3/urllib3/issues/3636)
<https://github.com/urllib3/urllib3/issues/3636></code>__)</li>
<li>Fixed a bug where <code>HTTPResponse.read()</code> could cache only
part of the
response after a partial read when <code>cache_content=True</code>.</li>
</ul>
<!-- raw HTML omitted -->
</blockquote>
<p>... (truncated)</p>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a
href="9a950b92d9"><code>9a950b9</code></a>
Release 2.7.0</li>
<li><a
href="5ec0de499b"><code>5ec0de4</code></a>
Merge commit from fork</li>
<li><a
href="2bdcc44d1e"><code>2bdcc44</code></a>
Merge commit from fork</li>
<li><a
href="f45b0df09d"><code>f45b0df</code></a>
Fix a misleading example for <code>ProxyManager</code> (<a
href="https://redirect.github.com/urllib3/urllib3/issues/4970">#4970</a>)</li>
<li><a
href="577193ca02"><code>577193c</code></a>
Switch to nightly PyPy3.11 in CI for now (<a
href="https://redirect.github.com/urllib3/urllib3/issues/4984">#4984</a>)</li>
<li><a
href="e90af45bb0"><code>e90af45</code></a>
Avoid infinite loop in <code>HTTPResponse.read_chunked</code> when
<code>amt=0</code> (<a
href="https://redirect.github.com/urllib3/urllib3/issues/4974">#4974</a>)</li>
<li><a
href="67ed74fdae"><code>67ed74f</code></a>
Bump dev dependencies (<a
href="https://redirect.github.com/urllib3/urllib3/issues/4972">#4972</a>)</li>
<li><a
href="3abd481097"><code>3abd481</code></a>
Upgrade mypy to version 1.20.2 (<a
href="https://redirect.github.com/urllib3/urllib3/issues/4978">#4978</a>)</li>
<li><a
href="2b8725dfca"><code>2b8725d</code></a>
Drop support for EOL PyPy3.10 (<a
href="https://redirect.github.com/urllib3/urllib3/issues/4979">#4979</a>)</li>
<li><a
href="2944b2a0a6"><code>2944b2a</code></a>
Upgrade <code>setup-chrome</code> and <code>setup-firefox</code> to fix
warnings (<a
href="https://redirect.github.com/urllib3/urllib3/issues/4973">#4973</a>)</li>
<li>Additional commits viewable in <a
href="https://github.com/urllib3/urllib3/compare/2.6.3...2.7.0">compare
view</a></li>
</ul>
</details>
<br />
[](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)
Dependabot will resolve any conflicts with this PR as long as you don't
alter it yourself. You can also trigger a rebase manually by commenting
`@dependabot rebase`.
[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)
---
<details>
<summary>Dependabot commands and options</summary>
<br />
You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits
that have been made to it
- `@dependabot show <dependency name> ignore conditions` will show all
of the ignore conditions of the specified dependency
- `@dependabot ignore this major version` will close this PR and stop
Dependabot creating any more for this major version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop
Dependabot creating any more for this minor version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop
Dependabot creating any more for this dependency (unless you reopen the
PR or upgrade to it yourself)
You can disable automated security fix PRs for this repo from the
[Security Alerts
page](https://github.com/infiniflow/ragflow/network/alerts).
</details>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
### What problem does this PR solve?
Closes#14674.
This PR improves RAPTOR configuration and tree construction while
preserving the existing RAPTOR behavior as the default.
RAPTOR currently builds summary layers with the original UMAP + GMM
clustering path. This PR keeps that default path, and adds:
- A hidden backend tree-builder option:
- `tree_builder="raptor"`: default, existing RAPTOR behavior.
- `tree_builder="psi"`: rank-aware Psi-style tree builder using original
embedding-space cosine ranking.
- A user-facing clustering method option for the default RAPTOR builder:
- `clustering_method="gmm"`: existing default.
- `clustering_method="ahc"`: agglomerative hierarchical clustering path.
- A RAPTOR UI setting for `Clustering method` and `Max cluster`.
### What changed
#### Backend
- Added `tree_builder` support for RAPTOR/Psi.
- Added `clustering_method` support for GMM/AHC.
- Kept existing RAPTOR + GMM as the default.
- Added Psi tree building from original-space cosine similarity.
- Added bucketed Psi building controls for large inputs:
- `raptor.ext.psi_exact_max_leaves`
- `raptor.ext.psi_bucket_size`
- Added method-aware RAPTOR summary metadata using existing
`extra.raptor_method`.
- Avoided adding a dedicated DB schema field for experimental method
tracking.
- Added cleanup/migration logic to avoid mixing stale RAPTOR summary
trees.
- Added defensive checks for Psi tree construction and summary failures.
#### Frontend/UI
- Added `Clustering method` in RAPTOR settings with `GMM` and `AHC`.
- Added/kept `Max cluster` in RAPTOR settings.
- Enlarged max cluster UI limit to `1024`, matching backend validation.
- Kept AHC editable even when a RAPTOR task has already finished.
- Fixed the UI save payload so `clustering_method` and `tree_builder`
are serialized through `parser_config.raptor.ext`, avoiding backend
validation errors for extra top-level RAPTOR fields.
Example saved RAPTOR config:
```json
{
"raptor": {
"max_cluster": 317,
"ext": {
"clustering_method": "ahc",
"tree_builder": "raptor"
}
}
}
Co-authored-by: CaptainTimon <CaptainTimon@users.noreply.github.com>
## Summary
- Add GET method handler to `/api/v1/dify/retrieval` endpoint for Dify
external knowledge base connectivity verification
- GET requests return a simple success response; POST requests retain
existing retrieval logic unchanged
## Problem
When Dify integrates with RAGFlow as an external knowledge base, it
sends periodic GET requests to the retrieval endpoint for
health/connectivity checks. The endpoint only accepted POST, causing
werkzeug to return `405 Method Not Allowed`. After several successful
POST retrievals, the failing GET health checks trigger Dify's circuit
breaker, causing all subsequent requests to fail.
Traceback from the issue:
```
werkzeug.exceptions.MethodNotAllowed: 405 Method Not Allowed: The method is not allowed for the requested URL.
```
## Changes
- `api/apps/sdk/dify_retrieval.py`: Added a separate GET route handler
(`retrieval_health_check`) that returns `get_json_result(data=True)`
## Test plan
- [ ] Verify `GET /api/v1/dify/retrieval` returns `{"code": 0,
"message": "success", "data": true}`
- [ ] Verify `POST /api/v1/dify/retrieval` with valid API key and body
still works as before
- [ ] Verify Dify external knowledge base integration no longer returns
405 errors
Closes#13788🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Asksksn <Asksksn@noreply.gitcode.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Co-authored-by: Kevin Hu <kevinhu.sh@gmail.com>
### What problem does this PR solve?
1. Add region check in zhipu-ai embed method
2. Fix retrieval test
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
Signed-off-by: Jin Hai <haijin.chn@gmail.com>
### What problem does this PR solve?
This PR completes the Cohere provider integration (upgrading to the new
Cohere V2 API) and enhances the Fish Audio provider in RAGFlow.
**The following functionalities are now supported:**
**Cohere:**
- [x] Chat / Think Chat / Stream Chat / Stream Think Chat
- [x] Embedding
- [x] Rerank
- [x] Model listing
- [x] Provider connection checking
- [ ] Balance
**Fish Audio:**
- [x] Model listing (`ListModels`)
- [x] Balance (`Balance`)
-----
**Verified examples from the CLI:**
```plaintext
# Cohere
RAGFlow(user)> think chat with 'command-a-reasoning-08-2025@test3@cohere' message 'jumperwho'
Thinking: Okay, the user wrote "jumperwho". Let me try to figure out what they might be asking. First, I'll check if it's a misspelling. "Jumper" ...... Hmm. Since the query is unclear, the best approach is to ask the user to provide more context or correct any possible typos.
Answer: It seems there might be a typo or missing context in your query "jumperwho." Could you clarify what you're referring to? For example:
- Are you asking about a **jumper** (a type of sweater, a person who jumps, or a component in electronics)?
- Is this related to a specific context, like a movie (e.g., the 2008 film *Jumper*) or a game?
- Did you mean to ask about a person ("who") associated with jumping (e.g., a parachutist)?
Let me know so I can provide a helpful response! 😊
Time: 6.710331
RAGFlow(user)> stream think chat with 'command-a-reasoning-08-2025@test3@cohere' message 'jumperwho'
Thinking: , the user mentioned "jumperwho". Let me try to figure out what they're referring to. First, I'll check if it's a misspelling. "Jumper" could be a typo for "jumper" or maybe a username. Alternatively, it might be a combination of words like "jumper who",....... the best approach is to inform the user that I don't recognize the term and ask if they can provide more context or clarify what they mean by "jumperwho". That way, I can assist them better once I have more information.
Answer: seems "jumperwho" isn't a widely recognized term, proper noun, or acronym in common usage. Could you provide more context or clarify what you mean by "jumperwho"? This will help me understand your question or request better!
Time: 4.513596
RAGFlow(user)> embed text 'walkerwhat' 'jumperwho' with 'embed-v4.0@test3@cohere' dimension 16;
+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-------+
| embedding | index |
+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-------+
| [-0.016643638 -0.001957038 0.0055713872 0.009027058 0.05275187 -0.024542313 -0.044006906 0.024119169 0.0014192933 0.006558722 0.0019129605 -0.021016119 -0.026516981 -0.017489925 0.021298215 0.017772019 0.04569948 0.008886009 0.012059584 -0.0014721862 0.... | 0 |
| [0.018778935 -0.0063459855 -0.0006839742 0.0046623563 0.0067668925 -0.018001877 -0.03963003 0.035744734 -0.014246088 -0.0020721585 -0.006313608 0.025124922 -0.010749322 0.01217393 -0.010231283 -0.025254432 0.021498645 -0.028880708 0.019167464 -0.0058279... | 1 |
+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-------+
RAGFlow(user)> rerank query 'what is rag' document 'rag is retrieval augment generation' 'rag need llm' 'famous rag project includes ragflow' with 'rerank-v4.0-pro@test@cohere' top 3;
+-------+-----------------+
| index | relevance_score |
+-------+-----------------+
| 0 | 0.91744334 |
| 1 | 0.7458429 |
| 2 | 0.68729424 |
+-------+-----------------+
RAGFlow(user)> list supported models from 'cohere' 'test'
+-------------------------------------+
| model_name |
+-------------------------------------+
| c4ai-aya-expanse-32b |
| c4ai-aya-vision-32b |
| cohere-transcribe-03-2026 |
| command-a-03-2025 |
| command-a-reasoning-08-2025 |
| command-a-translate-08-2025 |
| command-a-vision-07-2025 |
| command-r-08-2024 |
| command-r-plus-08-2024 |
| command-r7b-12-2024 |
| command-r7b-arabic-02-2025 |
| embed-english-light-v3.0 |
| embed-english-light-v3.0-image |
| embed-english-v3.0 |
| embed-english-v3.0-image |
| embed-multilingual-light-v3.0 |
| embed-multilingual-light-v3.0-image |
| embed-multilingual-v3.0 |
| embed-multilingual-v3.0-image |
| embed-v4.0 |
+-------------------------------------+
RAGFlow(user)> check instance 'test' from 'cohere'
SUCCESS
# FishAudio
RAGFlow(user)> list supported models from 'fishaudio' 'test'
+----------------------------------------+
| model_name |
+----------------------------------------+
| Valentino Narración Biblica Fer |
| Super Smash Bros. 4/Ultimate Announcer |
| Farid Dieck |
| عصام الشوالي |
| ALEX_CHIKNA |
| Energetic Male |
| voz de locutor k |
| يي |
| ELITE |
| Mortal Kombat |
+----------------------------------------+
RAGFlow(user)> show balance from 'fishaudio' 'test'
+----------------------------------+-----------------------------+--------+-----------------+------------------+-----------------------------+----------------------------------+
| _id | created_at | credit | has_free_credit | has_phone_sha256 | updated_at | user_id |
+----------------------------------+-----------------------------+--------+-----------------+------------------+-----------------------------+----------------------------------+
| 82ffec12cf984d88a30ec504d7909812 | 2026-05-09T07:52:16.119000Z | 0 | | false | 2026-05-09T07:52:16.119000Z | 2578ab1126804d6eaa630552400d7ff3 |
+----------------------------------+-----------------------------+--------+-----------------+------------------+-----------------------------+----------------------------------+
```
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
- [x] Refactoring
## Summary
- Replaces the `"no such method"` stub on `NvidiaModel.Rerank`
(`internal/entity/models/nvidia.go`) with a real implementation against
NVIDIA NIM's `/ranking` endpoint.
- Mirrors the existing Python `NvidiaRerank` class at
`rag/llm/rerank_model.py:149-190` for behavior parity: same
`passages`/`query.text`/`logit` payload shape; `top_n` set to
`len(documents)` so every input gets a score returned in original order
(the issue body's spec omitted `top_n`, which would cause silent data
loss).
- Adds the `"rerank": "ranking"` URL suffix and two NIM rerank model
entries (`nvidia/nv-rerankqa-mistral-4b-v3`,
`nvidia/llama-3.2-nv-rerankqa-1b-v2`) to `conf/models/nvidia.json` so
the picker exposes them.
- Follows the same shape as the recently merged Aliyun (#14676), Gitee
(#14656), and ZhipuAI (#14608) Rerank implementations: lowercase
per-driver request/response types, conversion to the project-wide
`RerankResponse{Data: []RerankResult}`, per-call `context.WithTimeout`
of 30s.
Closes#14720
## Test plan
- [x] `gofmt -l internal/entity/models/nvidia.go` — clean
- [x] `go vet ./internal/entity/models/...` — no new errors introduced
(the two pre-existing vet errors in `baidu.go:642` and
`openrouter.go:566` are unrelated to this PR)
- [x] `go build ./internal/entity/models/...` — succeeds
- [x] `python3 -c "import json;
json.load(open('conf/models/nvidia.json'))"` — JSON valid
- [ ] Live smoke test against NVIDIA NIM with a real API key (requires
reviewer with NIM credentials)
## Notes for reviewers
- The issue body suggested omitting `top_n`. The Python reference
includes it (`top_n: len(texts)`), and without it NVIDIA returns only
the default top-K rankings rather than scores for every input. This PR
follows the Python.
- The URL host is `integrate.api.nvidia.com` (kept consistent with the
existing chat/embeddings BaseURL in `nvidia.go`), not the legacy
`ai.api.nvidia.com` host the Python uses. NIM's unified endpoint accepts
the model names as-is, so no per-model URL transform is needed.
### What problem does this PR solve?
As the title suggests.
### Type of change
- [x] Documentation Update
Signed-off-by: Jin Hai <haijin.chn@gmail.com>
### What problem does this PR solve?
Fixes#14570. On OpenSearch backends (`DOC_ENGINE=opensearch`) every
document-metadata write failed with `'OSConnection' object has no
attribute 'create_doc_meta_idx'`, so both `PATCH
/api/v1/datasets/{ds}/documents/{doc}` with `meta_fields` and `POST
/api/v1/datasets/{ds}/metadata/update` were unusable while every other
document operation (retrieval, parsing, name update, chunk management)
worked correctly on the same OpenSearch cluster.
The bug runs deeper than the missing method name in the error message
suggests. `DocMetadataService` also reached into
`settings.docStoreConn.es.*` directly for the index refresh, the
scripted partial update, and the count call, which means that even after
adding `create_doc_meta_idx` to `OSConnection` the very next call in the
same metadata flow would still raise `AttributeError` because
`OSConnection` exposes `self.os` rather than `self.es`. Fixing only the
reported symptom would have moved the failure one line down without
restoring the feature.
This PR adds a uniform document-metadata dispatch surface to both
connection classes so they present the same abstract API, and routes the
service layer through that surface via `getattr` guards instead of
poking at backend-specific attributes. The four new methods on
`OSConnection` and `ESConnectionBase` are `create_doc_meta_idx`,
`refresh_idx`, `count_idx`, and `replace_meta_fields`.
`OSConnection.create_doc_meta_idx` reuses the existing
`conf/doc_meta_es_mapping.json` schema in the OpenSearch `body=` form
because OpenSearch and Elasticsearch share the same index-creation
payload, and `replace_meta_fields` emits a full scripted assignment
(`ctx._source.meta_fields = params.meta_fields`) on both backends so
removed keys actually disappear instead of being preserved by deep-merge
semantics.
The `getattr`-guarded dispatch in `DocMetadataService` keeps the
existing fall-through paths intact for Infinity and OceanBase, which
continue to rely on their search-based count fallback and on the
delete-then-insert metadata replacement they used before, so this change
is strictly additive for those two backends.
Verification: `pytest
test/unit_test/rag/utils/test_opensearch_doc_meta.py` runs 16 new unit
tests that pass locally and pin the `OSConnection` dispatch surface, the
`create_doc_meta_idx` short-circuit when the index already exists, the
mapping-file payload routing, the `IndicesClient.create` failure path,
the `refresh_idx` and `count_idx` success and error sentinels, and the
full-assignment script emitted by `replace_meta_fields`. The test module
stubs `common.settings` and `rag.nlp` at import time so the suite runs
without the heavy backend SDKs that the rest of the repository pulls in
transitively.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
---------
Co-authored-by: tmimmanuel <tmimmanuel@users.noreply.github.com>
### What problem does this PR solve?
fix some comments to improve readability
### Type of change
- [x] Documentation Update
---------
Signed-off-by: box4wangjing <box4wangjing@outlook.com>
Fixes#13851
## Problem
`OCR.detect()` in `deepdoc/vision/ocr.py` returns `None, None,
time_dict` (a truthy 3-tuple) when the text detector fails or receives a
`None` image. However, the caller in `pdf_parser.py:__ocr()` checks:
```python
bxs = self.ocr.detect(np.array(img), device_id)
if not bxs: # False! (None, None, time_dict) is a non-empty tuple → truthy
self.boxes.append([])
return
bxs = [(line[0], line[1][0]) for line in bxs] # iterates (None, None, time_dict)
# line = None → None[0] → TypeError: 'NoneType' object is not subscriptable
```
This causes the `NoneType object is not subscriptable` error that
appears after "OCR started" in the chunking pipeline when using PDF +
General parser.
## Solution
Simplified `OCR.detect()` to return `None` (falsy) instead of `None,
None, time_dict` on failure. The `time_dict` was unused by the only
caller of this method. The early-return guard `if not bxs:` in
`pdf_parser.py` then correctly catches it.
## Testing
- The method's only caller (`pdf_parser.py:__ocr`) already has a `if not
bxs:` guard that handles the `None` return correctly.
- No other callers of `OCR.detect()` exist in the codebase.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Refactor**
* Modified OCR detection function return behavior to streamline output.
The function now returns detection results only, without timing
metadata. Error cases now return `None` instead of empty tuple values.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
This PR fixes a UI issue where the .docx document preview was displayed
incompletely when clicking on a citation/reference link during a
knowledge base conversation.
### What problem does this PR solve?
The Issue:
In the chat interface, when a user clicks the source citation at the end
of an answer, the DocPreviewer opens. However, for .docx files, if the
content exceeded the window height, it was truncated and unscrollable,
preventing users from reading the full referenced text.
Changes:
web/src/components/document-preview/doc-preview.tsx: Added the
overflow-auto Tailwind class to the DocPreviewer root container to
ensure scrollbars appear automatically when content overflows.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
Co-authored-by: nie.weiyang <nie.weiyang@embedway.com>
### What problem does this PR solve?
The document parse status was set to DONE before the document chunks
were actually retrievable from Elasticsearch/Opensearch because it did
not wait for the index refresh. This meant that it was possible that the
document parse status returned by the API was DONE but when trying to
retrieve chunks there were none. Since the index refreshes every 1
second this was quite likely to happen when wait for document parsing by
polling with a short interval and then immediately trying to retrieve
chunks once the status was DONE.
I fixed this bug and added a test case that would have caught it.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
### What problem does this PR solve?
Added a private helper _visibility_and_status_filter(joined_tenant_ids,
user_id) that returns the Peewee condition: visible to user (team or
own) and status is VALID.
### Type of change
- [x] Refactoring
---------
Co-authored-by: Serobabov Aleksandr <40SerobabovAS@region.cbr.ru>
Co-authored-by: Yingfeng <yingfeng.zhang@gmail.com>
### What problem does this PR solve?
Addresses event-loop blocking under high concurrency reported in #13825.
When multiple requests hit the API simultaneously, synchronous DB/Redis
calls block the async event loop, preventing Quart from handling other
requests and causing cascading 502/504 timeouts.
This PR wraps all remaining blocking DB/Redis calls in `canvas_app.py`,
`chat_api.py`, `session.py`, and `canvas_service.py` with `await
thread_pool_exec()`
- Offload all synchronous `Service.*`, `REDIS_CONN.*`, and
`APIToken.query` calls to the thread pool
- Convert sync endpoint handlers (`list_chats`, `get_chat`, `templates`,
`sessions`, etc.) to `async def`
- Convert sync helper functions (`_ensure_owned_chat`,
`_validate_llm_id`, `_validate_dataset_ids`, etc.) to async - no
duplicate sync/async pairs
- Wrap `CanvasReplicaService` Redis IO calls (`bootstrap`,
`replace_for_set`, `commit_after_run`)
- Use `asyncio.gather()` for concurrent file uploads and chat response
building
**Note:** This fixes the code-level event-loop blocking, which is a
prerequisite for handling concurrent requests. For the full "30
concurrent requests without 502/504" goal described in the issue, users
should also tune deployment config:
- `WS=4` or higher (HTTP worker processes, default 1)
- `MAX_CONCURRENT_CHATS=50` (default 10)
- `SANDBOX_EXECUTOR_MANAGER_POOL_SIZE` for workflow-heavy workloads
### Performance verification
Reviewer asked for a before-vs-after comparison
([comment](https://github.com/infiniflow/ragflow/pull/13941#issuecomment-4393667231)).
I built a self-contained microbenchmark that reproduces the exact
failure mode this PR targets: an async handler that performs blocking
DB/Redis-style calls (50 ms each, 3 per request, 30 concurrent requests)
is run twice — once with the pre-PR pattern (sync call directly inside
the async handler) and once with the post-PR pattern (`await
thread_pool_exec(...)`). The benchmark imports nothing from RAGFlow
except `thread_pool_exec` itself, so it is hermetic and reproducible
(`THREAD_POOL_MAX_WORKERS=128`, Python 3.13.12).
**Throughput — wall-clock for 30 concurrent requests (lower is better)**
| flavour | wall(s) | p50(s) | p95(s) | max(s) |
|---|---:|---:|---:|---:|
| before | 4.986 | 0.158 | 0.207 | 0.269 |
| after | 0.248 | 0.181 | 0.230 | 0.231 |
The pre-PR handler serializes the entire load on the event-loop thread,
so 30 × 3 × 50 ms ≈ 4.5 s shows up as the wall time. The post-PR handler
parallelizes the blocking work across the thread pool and finishes the
same load in 248 ms — a **~20× speedup** on this workload.
**Event-loop responsiveness — latency of an unrelated probe coroutine
while the 30 slow requests are running (lower is better)**
| flavour | samples | probe p50 (ms) | probe p95 (ms) | probe max (ms) |
|---|---:|---:|---:|---:|
| before | 1 | 5442.26 | 5442.26 | 5442.26 |
| after | 28 | 0.88 | 11.53 | 98.02 |
This is the metric that maps directly to "the API still answers other
requests while one is busy". A 5 ms-interval probe was scheduled while
the 30 slow handlers ran. With the pre-PR code the event loop was frozen
for the entire duration of the blocking work, so only one probe sample
was ever picked up and it waited **5,442 ms**. After the PR, 28 probe
samples landed with **p50 0.88 ms / p95 11.53 ms**, meaning unrelated
requests are no longer starved by the slow ones. That is the regression
mode behind the cascading 502/504s reported in #13825.
<details>
<summary>Raw benchmark output</summary>
```
config: 30 concurrent requests, 3 blocking calls of 50ms each per request, THREAD_POOL_MAX_WORKERS=128
=== Throughput (lower wall is better) ===
flavour wall(s) p50(s) p95(s) max(s)
before 4.986 0.158 0.207 0.269
after 0.248 0.181 0.230 0.231
=== Event-loop responsiveness (lower probe latency is better) ===
flavour samples probe p50(ms) probe p95(ms) probe max(ms)
before 1 5442.26 5442.26 5442.26
after 28 0.88 11.53 98.02
```
</details>
The benchmark script is included as a comment on the PR for
reproducibility.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
- [x] Performance Improvement
Closes [#13825](https://github.com/infiniflow/ragflow/issues/13825)
---------
Co-authored-by: tmimmanuel <tmimmanuel@users.noreply.github.com>
Co-authored-by: Kevin Hu <kevinhu.sh@gmail.com>
### What problem does this PR solve?
- Moved if not all([email, new_pwd, new_pwd2]) guard to the top, before
any decryption that could crash on None value
- Removed the redundant REDIS_CONN.get() call — one call is sufficient
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
- [x] Refactoring
### What problem does this PR solve?
Provide embedding index according to the input text
### Type of change
- [x] Refactoring
---------
Signed-off-by: Jin Hai <haijin.chn@gmail.com>
## Summary
- Wrap 2 `ThreadPoolExecutor` instances in `file_service.py` with `with`
statement
- Ensures threads are properly shut down after all futures complete
## Problem
`parse_docs()` (line 532) and the file processing method (line 694)
create `ThreadPoolExecutor` instances that are never shut down. In a
long-running server process, this leaks thread resources on every
invocation — threads remain alive consuming memory even after all
submitted work is complete.
## Fix
Replace bare `ThreadPoolExecutor()` with `with ThreadPoolExecutor() as
exe:` context manager, which calls `executor.shutdown(wait=True)` on
exit.
## Test plan
- [x] Verified both call sites use `with` statement after fix
- [x] No remaining bare `ThreadPoolExecutor` in `file_service.py`
- [x] `document_service.py:1066` is a module-level executor (different
pattern, not changed in this PR)
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-authored-by: Kevin Hu <kevinhu.sh@gmail.com>
### What problem does this PR solve?
issue: https://github.com/infiniflow/ragflow/issues/14748
change: dataset search rerank id type
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
### Related issues
Closes#14744
### What problem does this PR solve?
The Memory REST endpoint `POST /api/v1/messages` previously persisted
whatever `user_id` the client sent in the JSON body. Memory rows were
therefore attributed to an arbitrary string, even when the caller
authenticated as a normal workspace user via JWT (browser/session-style
bearer token decoded into an access token). That broke attribution and
audit semantics for shared memories (team visibility): any authorized
writer could spoof another subject id.
The Python SDK already sends an optional `user_id` for integrations
using **API keys** (`APIToken`) to tag an external subject distinct from
the tenant owner user.
### Solution
- Record **`g.auth_via_api_token`** in `_load_user`
(`api/apps/__init__.py`): set `True` only when authentication resolves
via `APIToken`, otherwise `False` after JWT-based login succeeds.
- In **`POST /messages`** (`memory_api.add_message`): if the request was
authenticated with an API key, keep accepting optional `user_id` from
the body (default empty string). For JWT-authenticated users, **always**
set stored `user_id` to **`current_user.id`** and ignore the client
field.
- Guard reads of `g` with **`RuntimeError`** handling so isolated
imports or tests without a Quart application context do not fail when
resolving `user_id`.
- Document on **`RAGFlow.add_message`** that `user_id` is only
meaningful for API-key authentication.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
- [ ] New Feature (non-breaking change which adds functionality)
- [ ] Documentation Update
- [ ] Refactoring
- [ ] Performance Improvement
- [ ] Other (please describe):
### Testing
- `python -m py_compile` on modified modules (`api/apps/__init__.py`,
`api/apps/restful_apis/memory_api.py`).
- Recommended: run web/SDK memory message tests (`test_add_message`,
`test_message_routes_unit`) against a full environment with `quart` and
configured services.
### Notes for reviewers
- Behavior change **only** for callers using JWT-style authorization on
`POST /messages`; API-key callers keep prior optional `user_id`
semantics.
Co-authored-by: jony376 <jony376@gmail.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
## What problem does this PR solve?
The Dify-compatible `/dify/retrieval` endpoint recently gained stricter
parsing and validation for its request payload, including:
- Normalized `retrieval_setting.top_k` and
`retrieval_setting.score_threshold` types.
- Clear separation between malformed arguments vs missing required
fields.
Previously, there was no unit test explicitly guarding the exact error
code and message contract for these cases.
## What does this PR change?
- **Add guard-style unit test** in `test_dify_retrieval_routes_unit.py`:
- `test_retrieval_argument_error_messages`:
- Sends a request with malformed numeric options:
- `retrieval_setting = {"top_k": "not-int", "score_threshold":
"not-float"}`
- Asserts `code == RetCode.ARGUMENT_ERROR` and message contains
`"invalid or malformed arguments:"`.
- Sends a request with required fields missing:
- Empty payload (`{}`)
- Asserts `code == RetCode.ARGUMENT_ERROR` and message contains
`"required arguments are missing:"`.
This test encodes the intended behavior of the Dify retrieval API so
future refactors cannot silently regress error handling.
## Type of change
- [x] Tests (add coverage and guardrails for existing behavior)
Co-authored-by: Kevin Hu <kevinhu.sh@gmail.com>
pending_cell_images should be scoped by sheet
### What problem does this PR solve?
_Briefly describe what this PR aims to solve. Include background context
that will help reviewers understand the purpose of the PR._
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
### What problem does this PR solve?
GraphRAG feature - Part 1 - add spacy to extract entity and relation
<img width="1621" height="1288" alt="image"
src="https://github.com/user-attachments/assets/aadeddad-94da-46c6-adad-9c3784181f61"
/>
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
## Summary
- Wrap the `ThreadPoolExecutor` instances in `FileService.parse_docs`
and `FileService.get_files` with `with ... as exe:` blocks for
deterministic cleanup
- Replace the `concurrent.futures.ThreadPoolExecutor` in
`do_handle_task` with `asyncio.create_task(asyncio.to_thread(build_TOC,
...))`, preserving the existing parallelism with chunk insertion while
leveraging the surrounding async context
- Drop the now-unused `import concurrent` and the
`executor.shutdown(wait=False)` call in the `finally` block
Closes#14622.
No behavioral change, no public API change. Net diff: ~19 insertions /
25 deletions across two files.
## Test plan
- [ ] `uv run ruff check api/db/services/file_service.py
rag/svr/task_executor.py` passes
- [ ] Upload a multi-file batch through the chat/file endpoint and
confirm `FileService.parse_docs` still returns combined parsed text
- [ ] Trigger `FileService.get_files` via the chat reference flow with a
mix of image and non-image files; verify both `raw=True` and `raw=False`
paths return correctly
- [ ] Run a `naive`-parser document task with `toc_extraction: true` and
confirm the TOC chunk is generated and inserted exactly as before
- [ ] Run a `naive`-parser document task with `toc_extraction: false`
and confirm the path with `toc_thread = None` is unaffected
- [ ] Cancel a running task to exercise the `finally` block and confirm
cleanup still works without the executor shutdown call
---------
Co-authored-by: web-dev0521 <jasonpette1783@gmail.com>
Co-authored-by: Wang Qi <wangq8@outlook.com>
### What problem does this PR solve?
The OpenRouter `Encode` method silently swallowed malformed responses.
If a `data[]` item from the API was missing a field (`index`,
`embedding`, or unexpected shape), the loop did `continue` instead of
returning an error — leaving `nil` entries in the result slice. Callers
got back partial results with no indication anything went wrong, which
then crashes downstream consumers when they try to use a `nil` vector.
There were three concrete gaps:
- No count-mismatch check between `data` length and input texts (only
checked for empty)
- No duplicate-index detection (a duplicate would silently overwrite)
- Parse failures on individual items returned partial slices instead of
erroring
This PR replaces `map[string]interface{}` parsing with a typed
`openrouterEmbeddingResponse` struct and applies the same 3-layer
validation used in the other drivers (count mismatch → out-of-range
index → duplicate index), so any malformed response produces a clear
error instead of corrupted data.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
### What problem does this PR solve?
The LM Studio Go driver shipped with a stub \`Encode\` method that
returned \`no such method\`, even though LM Studio is one of the most
common local LLM runners on macOS and Windows and exposes an
OpenAI-compatible embeddings endpoint at \`/v1/embeddings\`.
LM Studio users routinely load local embedding models such as
\`nomic-ai/nomic-embed-text-v1.5\`,
\`mixedbread-ai/mxbai-embed-large-v1\`, or \`BAAI/bge-m3\`. They run on
the same \`/v1\` namespace as chat. The existing \`ListModels\` already
discovers them, but because \`Encode\` was a stub, a tenant who picked
one of these models in the Go layer could not actually run an embedding
call.
This finishes the local-LLM trio: Ollama Encode (#14664) and vLLM Encode
(#14688) are already in flight, both using the
same OpenAI-compatible \`/embeddings\` shape.
### What this PR includes
- \`conf/models/lmstudio.json\`: add \`\"embedding\": \"embeddings\"\`
under \`url_suffix\` so the driver can build the URL from config.
- \`internal/entity/models/lmstudio.go\`: replace the \`Encode\` stub
with a real implementation. Adds a small local response type that
matches the OpenAI-compatible shape.
No factory change. No interface change.
### How the driver works
- Validate the model name. The API key is optional for local LM Studio,
so the Authorization header is only set when both \`apiConfig\` and
\`ApiKey\` are non-nil and non-empty, the same pattern the recently
merged CheckConnection PR (#14614) uses.
- Resolve the region with a default fallback. Return a clear "missing
base URL" error when the user has not configured
the local access address yet.
- Use a per-call \`context.WithTimeout(30s)\` and
\`http.NewRequestWithContext\`, the same pattern the merged
Aliyun Encode (#14647) and the in-flight Ollama Encode (#14664) and vLLM
Encode (#14688) use.
- Send \`{model, input: [texts]}\` in one request.
- Parse \`data[*].embedding\` and copy each slice into a \`[][]float64\`
indexed by \`data[*].index\`, so the output
order matches the input order.
- Handle both \`float64\` and \`float32\` element types.
- Empty input returns \`[][]float64{}\` with no HTTP call.
- Length mismatch between input and result, out-of-range index, and any
missing slot all return clear errors instead
of silent zero vectors.
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
### How was this tested?
- \`go build ./internal/entity/models/...\` in a clean go 1.25 image
returns exit 0.
- The full method set on \`LmStudioModel\` still matches the
\`ModelDriver\` interface.
- Pattern parity with the merged Aliyun Encode (#14647), the in-flight
Ollama Encode (#14664) and vLLM Encode (#14688), and the existing
SiliconFlow Encode.
Closes#14693
### What problem does this PR solve?
The SiliconFlow `Encode` method sent one HTTP request per text, which is
wasteful and slow when indexing many documents (e.g., 100 docs = 100
round-trips).
SiliconFlow's `/v1/embeddings` is OpenAI-compatible and accepts an array
of strings in `input` (officially documented at
https://docs.siliconflow.cn/en/api-reference/embeddings/create-embeddings,
with a documented max array size of 32). This PR batches the requests up
to that limit, reducing 100 docs to ~4 round-trips, and replaces
`map[string]interface{}` parsing with a typed struct using the same
3-layer validation (count mismatch, out-of-range index, duplicate index)
used in the other drivers.
### Type of change
- [x] Performance Improvement
### What problem does this PR solve?
The NVIDIA Go driver in `internal/entity/models/nvidia.go` shipped with
a stub `Encode`
method that returned `no such method`. `conf/models/nvidia.json` already
lists
`nvidia/llama-3.2-nemoretriever-1b-vlm-embed-v1` as an embedding model,
but the conf had
no `embedding` URL suffix, so the picker had nothing wired even if
`Encode` worked.
A tenant who wanted to use NVIDIA NIM for chat (already working) and
embeddings from a
single provider could not, even though the upstream endpoint is public
at
`https://integrate.api.nvidia.com/v1/embeddings` and uses an
OpenAI-compatible request
body extended with the NVIDIA-specific `input_type` and `truncate`
fields. Several other
Go drivers already implement `Encode` (siliconflow, zhipu-ai, aliyun),
so the interface
and the pattern are well-established.
This PR fills the gap.
### What this PR includes
* `conf/models/nvidia.json`: declare the `embedding` URL suffix
alongside the existing
`chat` and `models` entries. The embedding model entry was already
present, so no
model addition is needed.
* `internal/entity/models/nvidia.go`: replace the `Encode` stub with a
real
implementation. Adds a small local response type that matches the
OpenAI-compatible
shape NVIDIA NIM returns.
No factory change. No interface change.
### How the driver works
* Validates `apiConfig` and the API key, validates the model name,
resolves the region
with a default fallback (matching the pattern the merged `ListModels`
and
`CheckConnection` paths in this driver already use), and builds the URL
from
`BaseURL[region] + URLSuffix.Embedding`.
* Sends all input texts in one request as the `input` array, with the
NVIDIA-specific `input_type: "query"`, `encoding_format: "float"`, and
`truncate: "END"`
fields, mirroring the Python `NvidiaEmbed` reference.
* Parses `data[*].embedding` and copies each slice into `[][]float64`
indexed by
`data[*].index` so the output order matches the input order even if the
API returns
items in a different order.
* Handles both `float64` and `float32` element types.
* Empty input returns `[][]float64{}` with no HTTP call.
* Non-200 responses propagate the upstream status line and body.
* A final pass checks every input slot got a vector and returns a clear
error if any
slot is still nil.
* Per-call 30s context deadline so a slow call cannot block forever.
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
### How was this tested?
* `go build ./internal/entity/models/...` returns exit 0.
* `go vet ./internal/entity/models/...` is clean.
* `gofmt -l internal/entity/models/nvidia.go` is clean.
* The full method set on `NvidiaModel` still matches the `ModelDriver`
interface.
* Pattern parity with the just-merged Aliyun `Encode` (#14647).
Closes#14699
### What problem does this PR solve?
The Ollama Go driver shipped with a stub \`Encode\` method that returned
\`no such method\`, even though Ollama is one of the most common local
LLM runners and exposes an OpenAI-compatible embeddings endpoint at
\`/v1/embeddings\`.
Ollama users routinely run local embedding models such as
\`nomic-embed-text\`, \`mxbai-embed-large\`, or \`bge-m3\`.
Pulled with \`ollama pull <model>\` and served on the same \`/v1\`
namespace as chat. The existing \`ListModels\` already
discovers them, but because \`Encode\` was a stub, a tenant who picked
one of these models in the Go layer could not
actually run an embedding call.
### What this PR includes
- \`conf/models/ollama.json\`: add \`\"embedding\": \"embeddings\"\`
under \`url_suffix\` so the
driver can build the URL from config.
- \`internal/entity/models/ollama.go\`: replace the \`Encode\` stub with
a real implementation. Adds a small local response
type that matches the OpenAI-compatible shape.
No factory change. No interface change.
### How the driver works
- Validate the model name. The API key is optional for local Ollama, so
the Authorization header is only set when both
\`apiConfig\` and \`ApiKey\` are non-nil and non-empty, the same pattern
the recently merged CheckConnection PR (#14614) uses.
- Resolve the region with a default fallback. Return a clear "missing
base URL" error when the user has not configured
the local access address yet.
- Use a per-call \`context.WithTimeout(30s)\` and
\`http.NewRequestWithContext\`, the same pattern the merged
Aliyun Encode (#14647) uses.
- Send \`{model, input: [texts]}\` in one request.
- Parse \`data[*].embedding\` and copy each slice into a \`[][]float64\`
indexed by \`data[*].index\`, so the output
order matches the input order.
- Handle both \`float64\` and \`float32\` element types.
- Empty input returns \`[][]float64{}\` with no HTTP call.
- Length mismatch between input and result, out-of-range index, and any
missing slot all return clear errors instead
of silent zero vectors.
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
### How was this tested?
- \`go build ./internal/entity/models/...\` in a clean go 1.25 image
returns exit 0.
- The full method set on \`OllamaModel\` still matches the
\`ModelDriver\` interface.
- Pattern parity with the merged Aliyun Encode (#14647) and the existing
SiliconFlow Encode.
Closes#14662
## Summary
This PR fixes the `message_fit_in()` truncation bug reported in #13607.
Changes:
- fix the user-message truncation branch to reserve room for the system
prompt token budget
- guard the zero-token edge case to avoid dividing by zero in the
truncation ratio check
- add focused regression tests covering both the user-dominant
truncation path and the zero-token boundary case
## Validation
```bash
pytest -q --noconftest test/unit_test/rag/prompts/test_generator_message_fit_in.py
```
Result: `2 passed`
Closes#13607
## Summary
This PR fully addresses all CodeRabbit review feedback and enhances the
robustness of the reranking module with 100% backward compatibility.
## Key Fixes
1. Fixed JinaRerank hardcoded base_url to support subclass endpoint
overrides
2. Corrected GPUStackRerank exception handling to use proper requests
exceptions and preserve stack traces
3. Added 30s timeout to all API calls to prevent service hanging
4. Added empty input validation for all rerank providers
5. Replaced direct dict key access with .get() to eliminate KeyError
crashes
6. Fixed _normalize_rank edge case for empty arrays
7. Implemented missing functionality for Ai302Rerank
8. Standardized type hints and fixed typo issues
## Compatibility
- No breaking changes to any existing functionality
- All rerank providers work as originally intended
- Fully compatible with existing configurations and workflows
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
- [x] Refactoring
---------
Co-authored-by: Kevin Hu <kevinhu.sh@gmail.com>
### What problem does this PR solve?
The vLLM Go driver shipped with a stub \`Encode\` method that returned
\`not implemented\`, even though vLLM is one of the most common
production-grade self-hosted inference servers and exposes an
OpenAI-compatible embeddings endpoint at \`/v1/embeddings\`.
Users who self-host \`BAAI/bge-m3\`, \`Qwen3-Embedding-*\`,
\`NV-Embed-v2\`, or similar models on vLLM could not run an embedding
call through the Go layer. The existing \`ListModels\` already discovers
the loaded models, but the embedding path failed because \`Encode\` was
a stub.
### What this PR includes
- \`conf/models/vllm.json\`: add \`\"embedding\": \"embeddings\"\` under
\`url_suffix\` so the driver can build the URL from config.
- \`internal/entity/models/vllm.go\`: replace the \`Encode\` stub with a
real implementation. Adds a small local response
type that matches the OpenAI-compatible shape.
No factory change. No interface change.
### How the driver works
- Validate the model name. The API key is optional for self-hosted vLLM,
so the Authorization header is only set when both \`apiConfig\` and
\`ApiKey\` are non-nil and non-empty, the same pattern the recently
merged CheckConnection PR (#14614) uses.
- Resolve the region with a default fallback. Return a clear "missing
base URL" error when the user has not configured
the local access address yet.
- Use a per-call \`context.WithTimeout(30s)\` and
\`http.NewRequestWithContext\`, the same pattern the merged
Aliyun Encode (#14647) and in-flight Ollama Encode (#14664) use.
- Send \`{model, input: [texts]}\` in one request.
- Parse \`data[*].embedding\` and copy each slice into a \`[][]float64\`
indexed by \`data[*].index\`, so the output
order matches the input order.
- Handle both \`float64\` and \`float32\` element types.
- Empty input returns \`[][]float64{}\` with no HTTP call.
- Length mismatch between input and result, out-of-range index, and any
missing slot all return clear errors instead
of silent zero vectors.
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
### How was this tested?
- \`go build ./internal/entity/models/...\` in a clean go 1.25 image
returns exit 0.
- The full method set on \`VllmModel\` still matches the \`ModelDriver\`
interface.
- Pattern parity with the merged Aliyun Encode (#14647), the in-flight
Ollama Encode (#14664), and the existing
SiliconFlow Encode.
Closes#14687
## What
Widen the keyword delimiter in `rag/svr/task_executor.py`:
both `build_chunks` (LLM `keyword_extraction` cache parsing) and
`run_dataflow` (chunk-level `keywords` ingestion) now split on
`, ， ; ； 、 \r \n` instead of only ASCII comma.
## Why
`rag/prompts/keyword_prompt.md` instructs the LLM:
> The keywords are delimited by ENGLISH COMMA.
In practice, Chinese-leaning models (Qwen / Tongyi-Qianwen, GLM,
etc.) frequently ignore this instruction when the source content is
Chinese and emit Chinese commas (`，`) instead. Result:
`cached.split(",")` sees the full LLM output as a *single* keyword.
Repro: `auto_keywords>=4` + Chinese docs + `qwen-plus@Tongyi-Qianwen`.
We observed entries in `important_kwd` like
`"功能介绍，配置说明，参数详解，问题排查"` — one bucket instead of four.
## Impact
- Silent data-quality bug; no exception thrown.
- BM25 `important_kwd^30` boost effectively stops firing — the
indexed term is the whole list, never matches user query tokens.
- Any downstream aggregating `important_kwd` (tagging, analytics,
candidate-keyword review UIs) sees garbage.
## Compatibility
- Pure widening of the splitter; ASCII-comma-only outputs continue
to work identically.
- No schema / API change.
## Test plan
Manually verified against `qwen-plus@Tongyi-Qianwen` with
`auto_keywords=10` on Chinese .txt files:
- Before: `important_kwd` contains one element per chunk that is the
full LLM string with `，`-separated phrases inside.
- After: `important_kwd` contains N elements, one per phrase, as the
LLM intended.
### What problem does this PR solve?
The Gitee AI Go driver in `internal/entity/models/gitee.go` shipped with
a stub `Encode` method that returned `gitee, no such method`, even
though `conf/models/gitee.json` already wires the `embedding` URL
suffix. The conf also listed no embedding models, so the picker had
nothing to select.
This blocked any tenant who wanted to use Gitee AI for chat, rerank
(already working, see #14656), and embeddings from a single provider.
This PR fills the gap, mirroring the just-merged Aliyun `Encode`
(#14647):
- `internal/entity/models/gitee.go`: replace the `Encode` stub with a
real implementation.
Validates inputs, resolves the region with a default fallback, POSTs the
standard OpenAI-compatible `{"model", "input": [...]}` body to
`BaseURL[region] + URLSuffix.Embedding`, parses `data[*].embedding`
indexed by `data[*].index` so output order matches input order, handles
both `float64` and `float32` element types, and uses a 30s per-call
context deadline matching the merged `Rerank`.
- `conf/models/gitee.json`: add `BAAI/bge-m3` so the embedding picker
has something to select.
No factory change. No interface change. No URL suffix change.
Verified with `go build`, `go vet`, and `gofmt -l` : all clean.
Closes#14697
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
### What problem does this PR solve?
`retrieval_by_children()` in `rag/nlp/search.py` crashes with a
`TypeError: 'NoneType' object is not subscriptable` when a parent
("mom") chunk referenced by child chunks is missing from the index.
This happens when the index is in an inconsistent state — for example
after a partial re-index, a document deletion that didn't clean up all
children, or a race condition during ingestion. `dataStore.get()`
returns `None` for the missing parent, and the subsequent access to
`chunk["content_with_weight"]` raises a `TypeError`.
**Stack trace:**
```
TypeError: 'NoneType' object is not subscriptable
File "rag/nlp/search.py", line 792, in retrieval_by_children
"content_with_weight": chunk["content_with_weight"],
```
### Type of change
- [x] Bug Fix
### Fix
When `dataStore.get()` returns `None` for a parent chunk, fall back to
using the child chunks directly and continue processing the remaining
parents. This preserves retrieval results for all other chunks rather
than aborting the entire query with an exception.
```python
chunk = self.dataStore.get(id, idx_nms[0], [ck["kb_id"] for ck in cks])
if chunk is None:
chunks.extend(cks)
continue
```
---------
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
## Summary
Two bypass vectors in the sandbox code security analyzer allowed
malicious code to pass the safety check undetected and reach the Docker
executor.
### 1. JavaScript: template-literal bypass of `require()` block
The `SecureJavaScriptAnalyzer` regex patterns used `['"]` to match
module names, covering only single and double quotes. An attacker could
use ES6 template literals to bypass all three `require` checks:
`javascript
const cp = require(`child_process`);
async function main() {
return cp.execSync('cat /etc/passwd').toString();
}
`
The same bypass applied to `fs` and `worker_threads`.
**Fix:** Updated all three `require` patterns from `['"]` to `['"\]` to
also match backtick template literals.
### 2. Python: `builtins` not blocked + attribute-call blind spot in
`visit_Call`
`visit_Call` only checked `ast.Name` nodes, so attribute-style calls
like `module.func()` were invisible to the analyzer. Additionally,
`builtins` was absent from `DANGEROUS_IMPORTS`. Combined, this allowed:
`python
import builtins
def main():
builtins.exec('import os; os.system("id")')
`
Neither the import nor the exec call triggered any flag.
**Fix:** Added `builtins` to `DANGEROUS_IMPORTS` and added an
`ast.Attribute` branch to `visit_Call` so that `module.dangerous_func()`
style calls are caught alongside bare `dangerous_func()` calls.
## Tests
Added four regression tests covering each new bypass vector:
- `test_javascript_child_process_template_literal_is_rejected`
- `test_javascript_fs_template_literal_is_rejected`
- `test_python_builtins_import_is_rejected`
- `test_python_attribute_eval_call_is_rejected`
---------
Co-authored-by: bounty-hunter <bounty@hunter.local>
### What problem does this PR solve?
Two bugs in the Aliyun Go driver:
1. **`Name()` returns `"siliconflow"`** — a copy-paste bug from when the
driver was created. `Name()` is used in error messages and log output,
so every Aliyun error incorrectly attributed itself to SiliconFlow.
2. **Silent empty URL for unknown regions in `ChatWithMessages`,
`ChatStreamlyWithSender`, and `ListModels`** — all three methods
construct the request URL as `z.BaseURL[region]` without checking
whether the key exists. For an unrecognised region this returns `""`,
producing a malformed URL like `"/chat/completions"` that the HTTP
transport rejects with a confusing error. `Encode` and `Rerank` (already
merged) correctly fall back to `"default"` and return a clear error.
This PR applies the same pattern to the remaining three methods.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
### What problem does this PR solve?
Closes#14703
`GoogleModel.CheckConnection` currently returns a hardcoded `no such
method` error even though the Google Go driver already supports
`ListModels`. This makes provider connection checks fail regardless of
whether the configured API key can list Google models.
This PR makes `CheckConnection` call `ListModels`, adds a small API-key
guard for nil, empty, and whitespace-only keys, and keeps `ListModels`
useful by following paginated Google model responses.
### What stays unchanged
* Google model listing still uses the Google GenAI SDK with
`genai.BackendGeminiAPI`.
* Model names still come from `models.Items[*].Name`.
* `Balance`, `Encode`, chat, streaming, provider config, and factory
wiring are unchanged.
### Tests and validation
Added focused unit coverage for:
* `CheckConnection` delegating to `ListModels` and returning its error
* nil, missing, empty, and whitespace-only API key validation
* model-name passthrough from the list-models adapter
* paginated model listing, empty-result preservation, and next-page
error propagation
Validated current PR head `17ceef43515ba8c46c254dd349b9085bf26dcbea`
locally with Go 1.25.0:
* `go test ./internal/entity/models -run
'TestGoogleModel|TestCollectGoogleModelNames' -count=1 -v` - PASS
* `go test ./internal/entity/models -count=1` - PASS
* `go test -race ./internal/entity/models -count=1` - PASS
* `gofmt -w internal/entity/models/google.go
internal/entity/models/google_test.go` - PASS, no diff
* `git diff --check` - PASS
### Type of change
* [x] Bug Fix (non-breaking change which fixes an issue)
Co-authored-by: Jin Hai <haijin.chn@gmail.com>
### What problem does this PR solve?
- Implements the `Encode` method in the Google Gemini driver, which was
previously a stub returning `not implemented`
- Uses the `google.golang.org/genai` SDK's `EmbedContent` API, which
routes to the `batchEmbedContents` endpoint internally — all texts are
sent in a single request
- Adds `text-embedding-004` (max 2048 tokens) to
`conf/models/google.json`
- Response values are `[]float32` from the SDK and are cast to
`[]float64` to satisfy the `ModelDriver` interface
## Files changed
- `internal/entity/models/google.go` — full `Encode` implementation
- `conf/models/google.json` — adds `text-embedding-004` embedding model
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
### What problem does this PR solve?
Multiple `requests.post()` calls across the LLM integration layer lack a
`timeout` parameter. Without a timeout, a single unresponsive upstream
service can block the calling thread **indefinitely**, eventually
exhausting the thread pool and degrading the entire system.
This is a well-known issue — Python's `requests` library defaults to
`timeout=None` (infinite wait), and [the library docs explicitly
recommend](https://requests.readthedocs.io/en/latest/user/advanced/#timeouts)
always setting a timeout.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
### Change
Added `timeout` to all `requests.post()` calls missing it:
| File | Calls fixed | Timeout |
|------|-------------|---------|
| `rag/llm/rerank_model.py` | 9 | 30s |
| `rag/llm/embedding_model.py` | 8 | 30s |
| `rag/llm/cv_model.py` | 3 | 60s |
| `rag/llm/tts_model.py` | 2 | 60s |
| `rag/llm/sequence2txt_model.py` | 2 | 60s |
Embedding/rerank calls use 30s (lightweight API calls). Vision, TTS, and
audio transcription use 60s (heavier workloads with file uploads).
Note: other files in the codebase (e.g. `check_minio_alive`,
`check_ragflow_server_alive`) already use `timeout=10`, so this PR
brings the LLM layer in line with existing practice.
Signed-off-by: Ricardo-M-L <Sibyl_Hartmanbnb@webname.com>
Co-authored-by: Kevin Hu <kevinhu.sh@gmail.com>
### What problem does this PR solve?
Resolves#14447. *(Note: This supersedes stalled PR #14448 and
implements the requested CodeRabbitAI fixes).*
Currently, the Dockerfiles inside `agent/sandbox/sandbox_base_image`
(both Python and Node.js) have hardcoded Chinese package mirrors. This
forces the mirrors on all users globally, which causes build network
timeouts for contributors outside of China.
This PR introduces an enhancement to fix the issue by:
1. Implementing the `NEED_MIRROR` build argument in the sandbox
Dockerfiles.
2. Replacing static `ENV` instructions with conditional shell logic
inside `RUN` blocks to dynamically set the package registries.
3. Allowing the build to cleanly fall back to default global registries
(`pypi.org` and `npmjs.org`) when `--build-arg NEED_MIRROR=0` is passed.
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
- [x] Refactoring
---------
Co-authored-by: Jin Hai <haijin.chn@gmail.com>
### What problem does this PR solve?
The VolcEngine Go driver in `internal/entity/models/volcengine.go`
shipped with a
`ListModels` stub that returned `volcengine, no such method`.
`conf/models/volcengine.json`
also did not declare a `models` URL suffix, so the model picker had
nothing to call even
if the method body were filled in.
A tenant who configured Volcengine (Doubao / Ark) as a provider could
not see the list of
available endpoints from the RAGFlow UI. Several other Go drivers
already implement
`ListModels` against the OpenAI-compatible `/models` endpoint (deepseek,
gitee, nvidia,
openai, siliconflow), so the interface and pattern are well-established.
This PR fills the gap.
### What this PR includes
* `conf/models/volcengine.json`: declare the `models` URL suffix
alongside the existing
`chat`, `files`, and `embedding` entries. The Ark v3 API exposes
`https://ark.cn-beijing.volces.com/api/v3/models`, so the suffix is just
`models`.
* `internal/entity/models/volcengine.go`: replace the `ListModels` stub
with a real
implementation. Reuses the package-level `DSModelList` / `DSModel` types
that
DeepSeek, Gitee, and SiliconFlow already use to parse the
OpenAI-compatible models
response shape.
No factory change. No interface change.
### How the driver works
* Resolves the region with a default fallback, the same way the other
VolcEngine methods
in this driver already do.
* Builds the URL from `BaseURL[region] + URLSuffix.Models`, with
`strings.TrimSuffix` on
the base to keep the join robust.
* Issues a `GET` with optional `Authorization: Bearer <api_key>` (the
header is omitted
when no key is configured, mirroring the existing NVIDIA `ListModels`).
* Reads the response body once, surfaces a non-200 with the upstream
status line plus
body, and parses the JSON via the shared `DSModelList` type.
* Returns the model id list in input order. When the response includes
an `owned_by`
field, the entry is rendered as `id@owned_by`, matching the convention
used by the
other Go drivers.
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
### How was this tested?
* `go build ./internal/entity/models/...` returns exit 0.
* `go vet ./internal/entity/models/...` is clean.
* `gofmt -l internal/entity/models/volcengine.go` is clean.
* The full method set on `VolcEngine` still matches the `ModelDriver`
interface.
* Endpoint reachability check: `GET
https://ark.cn-beijing.volces.com/api/v3/models`
returns `401 Unauthorized` without an API key, confirming the path
exists and accepts
Bearer authentication.
* Pattern parity with DeepSeek, Gitee, NVIDIA, and SiliconFlow
`ListModels`.
Fixes#14701
Co-authored-by: Jin Hai <haijin.chn@gmail.com>
## Summary
- `CvModel["Bedrock"]` was absent from `rag/llm/cv_model.py`, causing
`model_instance()` to return `None` when a Bedrock model was used as a
PDF parser — even after correct model resolution.
- This PR adds `BedrockCV`, enabling Bedrock vision models (e.g.
`amazon.nova-pro-v1:0`, `anthropic.claude-3-5-sonnet`) to be used as PDF
parsers.
## What problem does this PR solve?
When a Bedrock model is selected as the PDF parser in a knowledge base,
ingestion failed with:
```
'LiteLLMBase' object has no attribute 'describe_with_prompt'
```
The root cause: `LiteLLMBase` (the Bedrock chat implementation) was the
only registered handler for the Bedrock factory. It does not implement
`describe_with_prompt`. `CvModel` had no Bedrock entry, so
`model_instance()` returned `None` for `image2text` requests.
## Type of change
- [x] New Feature (non-breaking change which adds functionality)
## Changes
**`rag/llm/cv_model.py`**
Adds `BedrockCV(Base)` with `_FACTORY_NAME = "Bedrock"`:
- Uses `litellm.completion` with the `bedrock/` prefix (consistent with
`LiteLLMBase`)
- Parses AWS credentials from the JSON key assembled by `add_llm`
(`auth_mode`, `bedrock_ak`, `bedrock_sk`, `bedrock_region`,
`aws_role_arn`)
- Supports three auth modes: `access_key_secret`, `iam_role` (via STS
`assume_role`), and default credential chain (IRSA, instance profile)
- Implements `describe_with_prompt` and `describe`
## Test plan
- [ ] Configure a Bedrock vision model (e.g. `amazon.nova-pro-v1:0`)
with valid AWS credentials
- [ ] Select it as PDF parser in a knowledge base
- [ ] Verify ingestion of a PDF document completes without errors
- [ ] Verify `CvModel["Bedrock"]` resolves to `BedrockCV`
🤖 Generated with [Claude Code](https://claude.ai/claude-code)
---------
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
### What problem does this PR solve?
The table file parser (CSV/Excel) currently treats all columns
identically — every column is both vectorized (embedded in chunk text)
and stored as filterable metadata. There's no way for users to control
which columns should be searchable by semantic meaning versus which
should only be filterable attributes.
For example, when ingesting a news articles CSV with columns like title,
content, country, category, source, etc., the embedding includes
metadata fields like country: Brazil and source: Reuters in the chunk
text, which dilutes the semantic quality of the embedding without adding
retrieval value.
The RDBMS connector (MySQL/PostgreSQL) already supports content_columns
/ metadata_columns, but this capability was missing for file-based table
ingestion.
This PR adds column-level control (vectorize / metadata / both) for the
table file parser, following RAGFlow's existing patterns.
Backward compatible: Datasets without table_column_roles or with
table_column_mode: auto behave exactly as before (all columns = both).
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
### What problem does this PR solve?
HuggingfaceRerank.post() unconditionally prepends `http://` to base_url,
which already contains a protocol. This creates invalid URLs like
http://http://127.0.0.1:8080/rerank, breaking all requests. The fix
normalizes URL handling to match the rest of the codebase, removing
redunant `http://`.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
### Related Issues
- #7318
- #7796
---------
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
## Summary
- Tool-type components (Email, Invoke, etc.) fail to resolve template
strings that mix variable references with literal text in their
parameters.
- This adds template string resolution to `get_input()` in
`ComponentBase`, reusing existing `get_input_elements_from_text()` and
`string_format()` methods.
## Problem
`get_input()` in `ComponentBase` handles two cases:
1. **Pure reference** (`{Component:ID@field}`) — resolved via
`is_reff()` + `get_variable_value()`
2. **Literal value** — passed through as-is
But template strings like `{UserFillUp:X@name}@duke.edu` or `Question
from {Agent:Y@topic}` fall through to the literal branch because
`is_reff()` returns `False` (it expects the entire string to be a single
reference). The unresolved template is passed directly to the tool.
This affects **all** tool components (Email, Invoke, etc.) that need
mixed reference + text parameters — for example, constructing email
addresses or subjects dynamically.
## Fix
```python
# In get_input(), between is_reff check and literal fallback:
elif isinstance(v, str) and re.search(self.variable_ref_patt, v):
elements = self.get_input_elements_from_text(v)
kv = {k: e.get('value', '') for k, e in elements.items()}
self.set_input_value(var, self.string_format(v, kv))
```
This reuses `get_input_elements_from_text()` and `string_format()` which
are already used by `Message` components for the same purpose. The fix
only activates when the string contains at least one variable reference
pattern but is not a pure reference.
## Test plan
- [x] Pure references (`{Component:ID@field}`) still resolve correctly
via `is_reff()` path
- [x] Literal values without references pass through unchanged
- [x] Template strings like `{ref}@duke.edu` resolve the reference and
keep the literal suffix
- [x] Template strings like `Question from {ref}` resolve correctly
- [x] Multiple references in one string (`{ref1} and {ref2}`) both
resolve
- [x] Message components unaffected (they use their own template
resolution in `_run`)
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: wanghualoong <wanghualoong@gmail.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
## Summary
Closes#13663.
OAuth / OIDC callbacks call `login_user(user)` which writes `_user_id`
into the session cookie, but `_load_user()` in `api/apps/__init__.py`
only ever looked at the `Authorization` header. The SPA's response
interceptor wipes the Authorization value from `localStorage` on the
first 401 it sees — meaning that during the post-redirect window after
an OAuth login, a single transient 401 sends every subsequent request
back to the login page even though `login_user()` had already
established a perfectly good server-side session.
The reporter's analysis traces this all the way through the redirect →
`navigate('/')` → first request → empty header → 401 → `removeAll()` →
infinite-redirect-to-login chain.
## What changed
- New `_load_user_from_session()` helper that reads
`session["_user_id"]`, looks up the user in `UserService` (with the same
`StatusEnum.VALID` and `access_token` checks already used elsewhere),
and assigns `g.user`.
- Every `return None` path in `_load_user()` now routes through that
helper before giving up:
- missing `Authorization` header
- malformed `bearer ` prefix
- empty / too-short JWT payload
- JWT signature failure
- JWT-resolved user not found / has no `access_token`
- `APIToken.query()` fallback exhausted
The JWT and API-token paths still take precedence — the session is only
consulted when those can't authenticate the request. So existing
local-login and SDK callers see no behaviour change; only OAuth / OIDC
users that hit the original race now stay logged in.
The Bearer-prefix issue called out in #13663 (lines 103-110) is already
handled in the current code, so this PR only addresses the second half
of the report.
## Test plan
- [ ] Configure OIDC under `oauth` in `service_conf.yaml`
- [ ] Click the OIDC login button, complete auth at the IdP
- [ ] Confirm that navigating between pages no longer bounces back to
`/login`
- [ ] Confirm local email/password login still issues + accepts JWTs
- [ ] Confirm SDK/API key callers still authenticate via `Authorization:
Bearer <api-token>`
---------
Co-authored-by: Kevin Hu <kevinhu.sh@gmail.com>
### What problem does this PR solve?
`Radio.Group` in `web/src/components/ui/radio.tsx` injects the parent's
`disabled` prop into each child via `React.cloneElement` with
`as React.ReactElement` and no validation.
This throws at runtime when a consumer passes strings, numbers, `null`,
`false`, or other non-element nodes, while the cast hides the unsafe
access from TypeScript.
Use `React.isValidElement<RadioProps>(child)` as a type guard before
calling `cloneElement`. Non-element children pass through unchanged,
and `child.props` access becomes type-checked without an `as` cast.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
### What problem does this PR solve?
The OpenAI Go driver landed in #14605 with chat, list models, and check
connection. Encode was left as a stub that returns \`not implemented\`.
\`conf/models/openai.json\` already lists three embedding models out of
the box:
- text-embedding-ada-002
- text-embedding-3-small
- text-embedding-3-large
So a tenant who picked one of these in the Go layer could not actually
run an embedding call. This PR fills the gap.
### What this PR includes
- \`conf/models/openai.json\`: add \`\"embedding\": \"embeddings\"\`
under \`url_suffix\` so the driver can build the URL from config. This
matches the \`URLSuffix.Embedding\` field used by other drivers
(siliconflow, zhipu-ai).
- \`internal/entity/models/openai.go\`: replace the Encode stub with a
real implementation that POSTs to \`/v1/embeddings\`. Adds a small local
response type \`openaiEmbeddingResponse\`.
No factory change. No interface change.
### How the implementation works
- Validate \`apiConfig\` and the API key, validate the model name. Use
the existing \`baseURLForRegion\` helper so an unknown region fails fast
with a clear error.
- Wrap the request with \`context.WithTimeout(nonStreamCallTimeout)\` so
the call has a clear deadline. Same pattern as \`ChatWithMessages\` and
\`ListModels\` already use in this file.
- Send all input texts in one request. The OpenAI API accepts the
\`input\` field as an array.
- Parse \`data[*].embedding\` and copy each slice into a \`[][]float64\`
indexed by \`data[*].index\` so the output order matches the input order
even if the API returns items in a different order.
- Handle both \`float64\` and \`float32\` element types, the way the
SiliconFlow driver does.
- An empty input slice returns \`[][]float64{}\` with no HTTP call.
- Non-200 responses propagate the upstream status line and body.
- A final pass checks that every input slot got a vector. If any slot is
still nil, return a clear error so the caller does not silently use a
zero vector.
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
### How was this tested?
- \`go build ./internal/entity/models/...\` in a clean go 1.25 image
(the go.mod minimum) returns exit 0.
- The full method set on \`OpenAIModel\` still matches the
\`ModelDriver\` interface.
- Pattern parity with the existing SiliconFlow Encode implementation
(\`internal/entity/models/siliconflow.go\`).
Closes#14629
---------
Co-authored-by: Jin Hai <haijin.chn@gmail.com>
### What problem does this PR solve?
As title.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
Signed-off-by: Jin Hai <haijin.chn@gmail.com>
### What problem does this PR solve?
top_n is missing
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
Signed-off-by: Jin Hai <haijin.chn@gmail.com>
### What problem does this PR solve?
S3-family connector syncs currently re-download every in-window object
just so we can compute `xxhash128(blob)` and compare against
`Document.content_hash`. Anything that bumps `LastModified` without
changing bytes (`aws s3 cp` touches, bucket re-encryption, etc.) pays
full bandwidth and re-parses files that didn't actually change. #14628
covers the broader incremental-ingestion redesign; this PR is the first
slice.
The fix is a pre-listing short-circuit. `BlobStorageConnector` (S3 / R2
/ GCS / OCI / S3-compat) now implements a new `FingerprintConnector`
interface: `list_keys()` paginates `list_objects_v2` and yields
`KeyRecord(key, fingerprint)` where `fingerprint = xxhash128(ETag)`. The
orchestrator joins those against the connector's existing `{doc_id:
content_hash}` map and only calls `get_value(key)` when the fingerprint
differs. Unchanged keys are skipped entirely — no `GetObject`, no
re-parse.
No DDL. xxhash128(ETag) is 32 hex chars and reuses the existing
`Document.content_hash` column per @yingfeng's suggestion; the connector
decides at listing time whether to populate it. Local uploads and
connectors that don't opt in fall through to the existing post-download
`xxhash128(blob)` path with no behavior change.
This is PR-1 of a 4-PR series — full design lives on #14628. Subsequent
PRs extend tier 1 to local FS / WebDAV / Dropbox / Seafile / RDBMS
(PR-2), wire up tier 2 cursor connectors with `SyncLogs.next_checkpoint`
(PR-3), and unify deletion via `KeyRecord(deleted=True)` reconciliation
(PR-4). Holding those back keeps this PR additive and reviewable on its
own.
#### Files touched
- `common/data_source/models.py` — new `KeyRecord`; optional
`fingerprint` on `Document`
- `common/data_source/interfaces.py` — `IncrementalCapability` enum,
`FingerprintConnector` ABC
- `common/data_source/blob_connector.py` — `BlobStorageConnector`
implements `FingerprintConnector`; per-object download factored into
`_build_document_from_obj()` so `_yield_blob_objects`, `list_keys`,
`get_value` all share it
- `rag/svr/sync_data_source.py` —
`_BlobLikeBase._fingerprint_filtered_generator` does the bypass loop;
`_run_task_logic` plumbs `doc.fingerprint` into the upload dict
- `api/db/services/document_service.py` —
`list_id_content_hash_map_by_kb_and_source_type()` helper
- `api/db/services/connector_service.py` + `file_service.py` —
fingerprint flows through `duplicate_and_parse → upload_document` and
lands in `content_hash`
- `test/unit_test/common/test_blob_connector_fingerprint.py` — 14 tests
covering ETag normalization (single-part, multipart, quoted, empty),
`list_keys()` not calling `GetObject`, `get_value()` materializing with
fingerprint, deterministic/stable fingerprints, and the bypass loop
asserting `GetObject` is *not* called on a match
#### Worth flagging for review
Old `_BlobLikeBase._generate` called `poll_source(start, now)` with a
`LastModified` window when `poll_range_start` was set. New code uses
`_fingerprint_filtered_generator` (full bucket listing + fingerprint
compare) outside of explicit `reindex=1`. Strictly better for
unchanged-bucket cases since it skips `GetObject`, but it does mean
every sync now does a full `list_objects_v2` paginate. Should still be
cheap for most buckets — flagging in case anyone has a very large bucket
where the time-window filter was meaningful.
On migration: existing rows have `content_hash = xxhash128(blob)` from
the old code. The first sync after this lands sees ETag-derived
fingerprints that don't match, re-fetches every object once, and writes
the new fingerprint. From the second sync onward the bypass works as
expected. "Slow day one, fast every day after." A `fingerprint_backfill:
trust` opt-out is sketched in the design doc but not in this PR.
#### Test plan
- [x] `uv run ruff check` — clean on all 8 touched files
- [x] `uv run pytest
test/unit_test/common/test_blob_connector_fingerprint.py -v` — 14 passed
- [x] Broader unit-test suite — no regressions in anything I touched
- [ ] Manual smoke against a real S3 bucket — configure a connector, run
sync twice, expect the second sync to log `bypassed=N, fetched=0` and no
`GetObject` calls in CloudTrail / bucket access logs
- [ ] Manual smoke with `reindex=1` — confirm the full re-download path
still works
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
---------
Co-authored-by: Yingfeng <yingfeng.zhang@gmail.com>
### What problem does this PR solve?
This PR completes the Baidu Qianfan provider integration in RAGFlow.
**The following functionalities are now supported:**
- [x] Chat / Think Chat / Stream Chat / Stream Think Chat
- [x] Embedding
- [x] Rerank
- [x] Model listing
- [x] Provider connection checking
- [ ] Balance
-----
**Verified examples from the CLI:**
```plaintext
RAGFlow(user)> embed text 'what is rag' 'who are you' with 'embedding-3@test@zhipu-ai' dimension 16;
+-----------+-------+
| dimension | index |
+-----------+-------+
| 16 | 0 |
| 16 | 1 |
+-----------+-------+
RAGFlow(user)> rerank query 'what is rag' document 'rag is retrieval augment generation' 'rag need llm' 'famous rag project includes ragflow' with 'qwen3-reranker-4b@test@baidu' top 2;
+-------+---------------------+
| index | relevance_score |
+-------+---------------------+
| 0 | 0.974821150302887 |
| 1 | 0.14223189651966095 |
| 2 | 0.08632347732782364 |
+-------+---------------------+
RAGFlow(user)> think chat with 'deepseek-v3.2@test@baidu' message 'who r u'
Thinking: Hmm, the user is asking for a simple introduction. This is straightforward – no need for overcomplication.
I should give a clear, friendly response that covers my basic identity as an AI assistant, my purpose, and my capabilities. Keeping it concise but informative is key here.
Mentioning my creator Anthropic adds credibility, and ending with an offer to help invites further interaction. No need for technical details unless the user asks later.
Answer: Hello! I'm an AI assistant created by Anthropic, designed to help with a wide variety of tasks. You can think of me as a helpful digital companion—I can answer questions, assist with writing, help solve problems, provide explanations, and engage in conversation on many topics. I'm here to help with whatever you need! How can I assist you today?
Time: 8.103902
RAGFlow(user)> stream think chat with 'deepseek-v3.2@test@baidu' message 'who r u'
Thinking: mm, the user is asking "who r u" with casual spelling. This is a straightforward identity question. should give a clear, friendly introduction without overcomplicating it. Can start with my core function as an AI assistant, mention my creator, and briefly state my key capabilities. response should be welcoming and invite further interaction since this seems like an introductory question. Keeping it concise but covering the essentials: who I am, what I do, and how I can help.
Answer: ! I am DeepSeek, an AI assistant created by DeepSeek Company. I'm designed to help answer questions, provide information, assist with various tasks, and engage in conversations on a wide range of topics. I'm here to assist you with whatever you need - whether it's answering questions, helping with analysis, writing, coding, or just having a friendly chat!Is there anything specific I can help you with today? 😊
Time: 7.219703
RAGFlow(user)> list supported models from 'baidu' 'test'
+--------------------------------------+
| model_name |
+--------------------------------------+
| ernie-3.5-8k-preview |
| ernie-4.0-8k |
| ernie-4.0-turbo-8k-latest |
| ernie-4.0-turbo-8k-preview |
| ernie-4.0-8k-preview |
| ernie-speed-pro-128k |
| ernie-char-fiction-8k |
| ernie-3.5-8k |
| ernie-3.5-128k |
| ernie-lite-pro-128k |
| ernie-novel-8k |
| ernie-4.0-turbo-8k |
| ernie-4.0-turbo-128k |
| ernie-4.0-8k-latest |
| irag-1.0 |
| ........... |
| glm-5.1 |
| ernie-image-turbo |
| deepseek-v4-pro |
| deepseek-v4-flash |
| ernie-5.1 |
+--------------------------------------+
RAGFlow(user)> check instance 'test' from 'baidu'
SUCCESS
```
Additionally, this PR fixes an incorrect error message typo:
Before:
```go
fmt.Errorf("API requestssss failed with status %d: %s : %s", ...)
```
After:
```go
fmt.Errorf("API request failed with status %d: %s", ...)
```
This PR mainly improves provider compatibility, API completeness, and
runtime stability.
### Type of change
* [x] Bug Fix (non-breaking change which fixes an issue)
* [x] New Feature (non-breaking change which adds functionality)
* [x] Refactoring
### What problem does this PR solve?
- Update version tags in README files (including translations) from
v0.25.1 to v0.25.2
- Modify Docker image references and documentation to reflect new
version
- Update version badges and image descriptions
- Maintain consistency across all language variants of README files
### Type of change
- [x] Documentation Update
### What problem does this PR solve?
## Problem
During the REST API refactoring (#13690), the
`/api/v2/kb/check_embedding` endpoint was removed and never migrated to
the new RESTful structure. The frontend was pointed to the
`/api/v1/datasets/{id}/embedding` endpoint (which is `run_embedding` — a
completely different function). Additionally, a hard guard was
introduced that rejects any `embd_id` change when `chunk_num > 0`,
making it impossible to switch embedding models on datasets with
existing chunks.
## Root Cause
1. **Missing endpoint**: The old `check_embedding` logic (sample random
chunks, re-embed with the new model, compare cosine similarity) was not
carried over to the new REST API service layer.
2. **Wrong frontend URL**: `checkEmbedding` in `api.ts` pointed to
`/datasets/{id}/embedding` (`run_embedding`) instead of a dedicated
check endpoint.
3. **Overly restrictive guard**: `dataset_api_service.py` line 310
blocked all `embd_id` updates when `chunk_num > 0`. This check did not
exist in the pre-refactor code — it was incorrectly introduced during
the refactor.
## Changes
### Backend
- **`api/apps/services/dataset_api_service.py`**
- Remove the `chunk_num > 0` hard guard on `embd_id` updates
- Add `check_embedding()` service function: samples random chunks,
re-embeds them with the candidate model, computes cosine similarity,
returns compatibility result (avg ≥ 0.9 = compatible)
- Add `import re` for the `_clean()` helper
- **`api/apps/restful_apis/dataset_api.py`**
- Add `POST /datasets/<dataset_id>/embedding/check` endpoint following
the new REST API conventions
- Clean up unused top-level imports (`random`, `re`, `numpy`)
### Frontend
- **`web/src/utils/api.ts`**
- Fix `checkEmbedding` URL from `/datasets/${datasetId}/embedding` →
`/datasets/${datasetId}/embedding/check`
### Tests
-
**`test/testcases/test_http_api/test_dataset_management/test_update_dataset.py`**
- Update `test_embedding_model_with_existing_chunks` to assert success
(`code == 0`) instead of expecting the old `102` error
-
**`test/testcases/test_web_api/test_dataset_management/test_dataset_sdk_routes_unit.py`**
- Update `test_update_route_branch_matrix_unit` to assert
`RetCode.SUCCESS` when updating `embd_id` on a chunked dataset,
replacing the old `chunk_num` error assertion
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
---------
Signed-off-by: noob <yixiao121314@outlook.com>
### What problem does this PR solve?
```
RAGFlow(user)> embed text 'what is rag' 'who are you' with 'embedding-3@test@zhipu-ai' dimension 16;
+-----------+-------+
| dimension | index |
+-----------+-------+
| 16 | 0 |
| 16 | 1 |
+-----------+-------+
RAGFlow(user)> rerank query 'what is rag' document 'rag is retrieval augment generation' 'rag need llm' 'famous rag project includes ragflow' with 'rerank@test@zhipu-ai' top 2;
+-------+-----------------+
| index | relevance_score |
+-------+-----------------+
| 0 | 1 |
| 2 | 0.99999976 |
+-------+-----------------+
```
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
Signed-off-by: Jin Hai <haijin.chn@gmail.com>
### What problem does this PR solve?
Update mapping.json to treat id as a keyword.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
### What problem does this PR solve?
Fix: Some bugs
- Error during batch modification of metadata in the Knowledge Base
- Manually configured metadata is not displayed in search settings
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
Close#14292
## Issue
File ancestry endpoints return folder metadata without validating tenant
permissions, allowing any authenticated user to query arbitrary
`file_id` values across tenant boundaries.
## Affected Endpoints
- `GET /v1/file/parent_folder?file_id={file_id}`
- `GET /v1/file/all_parent_folder?file_id={file_id}`
- `GET /api/v1/files/{id}/ancestors`
## Root Cause
These endpoints **skip the permission check** that other file operations
(Delete, Download, Move) perform.
## Expected Permission Check
All file operations should follow this 3-step validation:
- Check file.tenant_id
- Check if user_id belongs to this tenant (via user_tenant join table)
- Check KB permission type (team permission)
**Code reference:** This is implemented in `checkFileTeamPermission()`
and used by Delete/Download/Move, but **missing** from
GetParentFolder/GetAllParentFolders.
## Reproduction
```bash
# User B (tenant: BBB) accessing User A's file (tenant: AAA)
curl -H "Authorization: Bearer USER_B_TOKEN" \
"http://localhost:9384/v1/file/parent_folder?file_id=AAA_FILE_123"
# Result: Returns User A's folder metadata ❌
# Expected: "No authorization." ✅
Fix
Pass userID from handler to service and call checkFileTeamPermission() — same as Download/Delete/Move handlers.
---------
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
### What problem does this PR solve?
Implement `HuggingFace` provider
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
### Related issues
Closes#14644
### What problem does this PR solve?
This PR fixes an authorization bug where datasets marked with
`permission = me` could still be accessed by other members of the same
tenant through APIs that relied on `KnowledgebaseService.accessible()`
or `DocumentService.accessible()`.
Before this change, those shared access helpers only checked tenant
membership and did not enforce the dataset's permission mode. As a
result, a non-owner who knew a private `dataset_id` could still reach
downstream document and chunk operations even though the dataset was
intended to be owner-only.
This change updates the central access checks so that:
- dataset owners always retain access
- joined tenant members only get access when the dataset permission is
`TEAM`
- private datasets with `permission = me` remain inaccessible to
non-owners
- document-level access follows the same dataset permission rules
The PR also adds regression coverage for private-vs-team dataset access
behavior.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
- [ ] New Feature (non-breaking change which adds functionality)
- [ ] Documentation Update
- [ ] Refactoring
- [ ] Performance Improvement
- [ ] Other (please describe):
### Testing
- Added
`test/unit_test/api/db/services/test_dataset_access_permissions.py`
- Attempted to run: `python -m pytest
test\\unit_test\\api\\db\\services\\test_dataset_access_permissions.py
-q`
- Local execution in this workspace is currently blocked during test
collection because the environment is missing the `strenum` dependency
---------
Signed-off-by: Jin Hai <haijin.chn@gmail.com>
Co-authored-by: jony376 <jony376@gmail.com>
Co-authored-by: Wang Qi <wangq8@outlook.com>
Co-authored-by: d 🔹 <liusway405@gmail.com>
Co-authored-by: Jin Hai <haijin.chn@gmail.com>
Co-authored-by: Magicbook1108 <newyorkupperbay@gmail.com>
Co-authored-by: chanx <1243304602@qq.com>
Co-authored-by: sxxtony <166789813+sxxtony@users.noreply.github.com>
Co-authored-by: sxxtony <sxxtony@users.noreply.github.com>
Co-authored-by: Baki Burak Öğün <63836730+bakiburakogun@users.noreply.github.com>
Co-authored-by: bakiburakogun <bakiburakogun@users.noreply.github.com>
Co-authored-by: Panda Dev <56657208+pandadev66@users.noreply.github.com>
Co-authored-by: Haruko386 <tryeverypossible@163.com>
Co-authored-by: D2758695161 <13510221939@163.com>
Co-authored-by: Hunter <hunter@yitong.ai>
Co-authored-by: Lynn <lynn_inf@hotmail.com>
Co-authored-by: buua436 <sz_buua@foxmail.com>
Co-authored-by: web-dev0521 <jasonpette1783@gmail.com>
Co-authored-by: Tim Wang <38489718+wanghualoong@users.noreply.github.com>
Co-authored-by: wanghualoong <wanghualoong@gmail.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Co-authored-by: qinling0210 <88864212+qinling0210@users.noreply.github.com>
Co-authored-by: dale053 <star05223@outlook.com>
### What
19 methods across `rag/llm/chat_model.py` and `rag/llm/cv_model.py`
declare `gen_conf={}` (or `gen_conf: dict = {}`) as a parameter default
and then mutate `gen_conf` in place — typically `del
gen_conf["max_tokens"]`, `gen_conf["penalty_score"] = ...`, or
`gen_conf.pop(...)` as part of provider-specific normalization.
### The two bugs in this pattern
**1. Mutable default argument (Python footgun).** Python evaluates
default values **once** at function-definition time, so the single `{}`
dict is *shared* across every caller that doesn't pass `gen_conf`. The
first such call's mutations leak into the default seen by every
subsequent call.
```python
# Before
def chat_streamly(self, system, history, gen_conf={}, **kwargs):
if "max_tokens" in gen_conf:
del gen_conf["max_tokens"] # mutates the SHARED default dict
...
```
After call N with `max_tokens` set, call N+1 that omits `gen_conf` no
longer sees `max_tokens` — even though the caller never touched it.
**2. Caller-dict pollution.** When the caller *does* pass a `gen_conf`
dict, the same in-place mutations modify the caller's dict. A reused
`gen_conf` (very common for chat-loop callers that build the config once
and pass it on every turn) silently loses `max_tokens`,
`presence_penalty`, etc. after the first round.
### The fix
In every affected method:
- Change `gen_conf={}` (or `gen_conf: dict = {}`) → `gen_conf=None`.
- Add `gen_conf = dict(gen_conf or {})` as the first statement of the
body so all subsequent mutations operate on a fresh local copy.
```python
# After
def chat_streamly(self, system, history, gen_conf=None, **kwargs):
gen_conf = dict(gen_conf or {})
if "max_tokens" in gen_conf:
del gen_conf["max_tokens"] # local copy — safe
...
```
This is byte-for-byte identical provider-side behavior for callers that
already pass a fresh `gen_conf` per call. The new `dict(...)` copy is
O(small constant) per call.
### Files changed
- `rag/llm/chat_model.py` — 17 methods
- `rag/llm/cv_model.py` — 2 methods
### Tests
Adds `test/unit_test/rag/llm/test_gen_conf_no_mutable_default.py` — an
`ast`-based regression guard that walks both modules and asserts no
parameter named `gen_conf` ever has a mutable literal (`{}` or `[]`) as
its default. The test caught **five additional `gen_conf: dict = {}`
sites** that an initial `gen_conf={}` text grep had missed (annotated
parameters with whitespace), and would fail again if the pattern is ever
reintroduced.
```
$ pytest test/unit_test/rag/llm/test_gen_conf_no_mutable_default.py -v
============================== 3 passed in 0.04s ===============================
```
`ruff check` passes on all touched files.
### Notes
- This PR is intentionally focused on **just** the `gen_conf` default +
copy fix. There's a related (but separate) `history.insert(0, ...)`
pattern in the same files that mutates the caller's history list in 12
places — left for a follow-up so this PR stays mechanical and easy to
review.
### Latest revision (`700bb54a7`) — addresses CodeRabbit review
- Type annotation: `gen_conf: dict = None` → `gen_conf: dict | None =
None` (5 occurrences in `chat_model.py`). The old annotation was a
static-checker mismatch since `None` isn't a `dict`.
- Regression test: the AST check accessed `default.keys` directly.
`ast.List` has no `.keys` attribute — a future `gen_conf=[]` would crash
with `AttributeError` instead of being caught. Use `getattr` for both
`.keys` (Dict) and `.elts` (List). Manually verified the updated check
correctly catches both `gen_conf={}` and `gen_conf=[]` while ignoring
`gen_conf=None` and non-empty literals.
---------
Co-authored-by: Ricardo <ricardo@example.com>
### What problem does this PR solve?
Bugfix: keep document api backward compatible
Fix 1: https://github.com/infiniflow/ragflow/issues/14634
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
## Summary
- When KB retrieval fails (e.g. ES `AssertionError` on empty
`index_names`), `kbinfos` falls back to a dict without a `total` key
- `_async_update_chunk_info` then iterates over `chunk_info.keys()`
(which includes `total`) and tries `kbinfos['total']`, raising a
`KeyError`
- This error surfaces when using Tavily web retrieval in a chat with no
knowledge base attached
## Changes
- Add `'total': 0` to all default `kbinfos` dicts in
`_retrieve_information`
- Add `setdefault('total', 0)` guard after successful KB retrieval to
handle cases where the retrieval result omits the key
- Accumulate `total` correctly in the merge branch of
`_async_update_chunk_info`
## Test plan
- [ ] Start a chat with Tavily configured and no knowledge base
- [ ] Verify no `KeyError: 'total'` is raised
- [ ] Verify Tavily results are returned correctly
---------
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
## Summary
- Adding a Bedrock model from the frontend fails with `Fail to access
model(Bedrock/<model>).Expecting value: line 1 column 1 (char 0)`.
- The assembled Bedrock JSON credentials are silently replaced by `"x"`
before the connection test, causing `json.loads("x")` to raise a
`JSONDecodeError`.
## What problem does this PR solve?
Commit `050113482` introduced a fallback in `add_llm()` that reuses the
existing DB key when `req.get("api_key") is None`:
```python
if req.get("api_key") is None:
api_key = existing_api_key if existing_api_key is not None else "x"
```
For Bedrock, credentials are sent as separate fields (`auth_mode`,
`bedrock_ak`, `bedrock_sk`, `bedrock_region`, `aws_role_arn`) — the
frontend does not send an `api_key` field. The function correctly
assembles the JSON key:
```python
api_key = apikey_json(["auth_mode", "bedrock_ak", "bedrock_sk", "bedrock_region", "aws_role_arn"])
```
But since `req.get("api_key")` is `None`, the override immediately
replaces `api_key` with `"x"` (or a stale DB value). `LiteLLMBase` then
calls `json.loads("x")` for Bedrock auth → `JSONDecodeError`.
## Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
## Changes
**`api/apps/llm_app.py`**
Write the assembled key into `req["api_key"]` so the `None` check
evaluates to `False` and the override is skipped — consistent with how
`Tencent Cloud` is already handled.
```python
# Before
api_key = apikey_json(["auth_mode", "bedrock_ak", "bedrock_sk", "bedrock_region", "aws_role_arn"])
# After
req["api_key"] = apikey_json(["auth_mode", "bedrock_ak", "bedrock_sk", "bedrock_region", "aws_role_arn"])
api_key = req["api_key"]
```
## Test plan
- [ ] Configure a Bedrock provider in Model Providers with valid AWS
credentials
- [ ] Add a Bedrock chat model — verify no `Expecting value` error
- [ ] Update the same model — verify the existing key is reused
correctly when credentials fields are left empty
🤖 Generated with [Claude Code](https://claude.ai/claude-code)
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
### What problem does this PR solve?
The use_sql() function in dialog_service.py constructed SQL WHERE
clauses and Infinity table names by directly interpolating kb_id values
using Python f-strings, with no validation of the input values. A
malformed or maliciously crafted kb_id (introduced via a compromised
admin account or a separate injection vector) could alter the structure
of the generated SQL query, potentially leading to unauthorized data
access or data manipulation.
This PR adds strict UUID format validation for all kb_id values before
they are interpolated into any SQL string, causing requests with invalid
IDs to fail fast with a ValueError rather than executing a tampered
query.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
---------
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
## Summary
- When a model is registered as `chat` in `tenant_llm` but has the
`IMAGE2TEXT` tag in `llm_factories.json`, requesting it as `image2text`
(e.g. PDF parser) fails with `Tenant Model with name <model> and type
image2text not found`.
- After resolution via the new fallback, the returned
`config_dict["model_type"]` was still `"chat"`, causing
`tenant_llm_service.model_instance()` to instantiate `ChatModel` instead
of `CvModel` — breaking `describe_with_prompt` at ingestion time.
## What problem does this PR solve?
RAGFlow already has a `CHAT→IMAGE2TEXT` fallback: when a chat model is
not found, it retries with `image2text`. The symmetric fallback
(`IMAGE2TEXT→CHAT`) was missing.
This matters for multimodal models declared as `model_type: "chat"` with
an `IMAGE2TEXT` tag in `llm_factories.json` (e.g. models added after
tenant creation, or providers where a single model serves both
purposes). The frontend PDF parser selector correctly surfaces these
models via the `IMAGE2TEXT` tag, but the backend fails to resolve them
at runtime.
## Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
## Changes
**`api/db/joint_services/tenant_model_service.py`**
1. Add `IMAGE2TEXT→CHAT` fallback in
`get_model_config_by_type_and_name`: when an `image2text` model is not
found in `tenant_llm`, retry with `chat` — but only if the `llm` table
confirms `IMAGE2TEXT` capability via the `tags` field. This mirrors the
philosophy of the existing `CHAT→IMAGE2TEXT` fallback: substitution is
only allowed when the model has declared the required capability.
2. Normalize `config_dict["model_type"]` to `image2text` after the
fallback, so the caller (`model_instance`) correctly routes to `CvModel`
instead of `ChatModel`.
3. Extend the type validation guard to allow `(requested=image2text,
found=chat)` alongside the existing `(requested=chat, found=image2text)`
exception.
## Test plan
- [ ] Add a model with `model_type=chat` and `tags` containing
`IMAGE2TEXT` to a tenant
- [ ] Select it as PDF parser in a knowledge base
- [ ] Verify ingestion succeeds without `image2text not found` or
`describe_with_prompt` errors
- [ ] Verify the same model still works correctly in chat context
🤖 Generated with [Claude Code](https://claude.ai/claude-code)
---------
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Fixes#14360
## Problem
When the same blob storage bucket is connected to multiple knowledge
bases (each through a different data source connector), the sync
pipeline hashes only the blob path
(`bucket_type:bucket_name:object_key`) to derive the document ID. Every
connector pointing at the same bucket therefore produces **identical
IDs** for the same object. The collision guard in
`FileService.upload_document` then fires for the second knowledge base:
```
Existing document id collision with another knowledge base; skipping update.
```
This makes it impossible to index the same bucket into more than one KB
simultaneously.
## Solution
Include `connector_id` in the hash input so that each connector produces
a distinct document ID even when the underlying blob path is identical:
```python
# Before
"id": hash128(doc.id),
# After
"id": hash128(f"{task['connector_id']}:{doc.id}"),
```
Because each KB connection uses its own connector (with a unique
`connector_id`), documents are now namespaced per connector and no
collision occurs.
**Note:** This is a breaking change for existing synced data sources.
After upgrading, a re-sync will create new documents with the updated ID
format. Old documents (indexed under the previous format) will remain in
the database but can be manually deleted or cleaned up via a re-sync
with reindex enabled.
## Testing
- Verified that the one-line change produces unique IDs for two
connectors pointing at the same S3 path.
- Existing unit test
`test_upload_document_skips_cross_kb_document_id_collision` continues to
pass — the collision guard in `FileService` is still valid for genuinely
colliding IDs from other sources.
---------
Co-authored-by: octo-patch <octo-patch@github.com>
### What problem does this PR solve?
As title
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
Signed-off-by: Jin Hai <haijin.chn@gmail.com>
### What problem does this PR solve?
The Aliyun Go driver has a stub `Rerank` method that always returns
`"Aliyun, Rerank not implemented"`. DashScope exposes an
OpenAI-compatible rerank endpoint (`compatible-mode/v1/rerank`) and
hosts dedicated bilingual rerankers (`gte-rerank-v2`, `gte-rerank`) that
are a natural pairing with the embedding models already in
`aliyun.json`. Without this, Aliyun users cannot use reranking within
RAGFlow.
Closes#14675
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
### What problem does this PR solve?
Fix: Route error in dataset files page
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
### What problem does this PR solve?
Refactor : Allow search multiple datasets
1. support /datasets/search
2. get rid of /graph/search, use /graph
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
- [x] Refactoring
Closes#14590
## Self Checks
- [x] I have searched for existing issues [search for existing
issues](https://github.com/infiniflow/ragflow/issues), including closed
ones.
- [x] I confirm that I am using English to submit this report ([Language
Policy](https://github.com/infiniflow/ragflow/issues/5910)).
- [x] Non-english title submitions will be closed directly (
非英文标题的提交将会被直接关闭 ) ([Language
Policy](https://github.com/infiniflow/ragflow/issues/5910)).
- [x] Please do not modify this template :) and fill in all the required
fields.
## RAGFlow workspace code commit ID
`a1b2c3d4e5f67890123456789abcdef12345678`
## RAGFlow image version
`0.13.1`
## Other environment information
- Hardware parameters: N/A
- OS type: Linux 6.17.0-22-generic
- Others: API key authentication via `Authorization: Bearer <token>`
## Actual behavior
The chatbot API endpoints:
- `POST /chatbots/<dialog_id>/completions`
- `GET /chatbots/<dialog_id>/info`
validate only that the bearer token exists in `APIToken`, but do not
verify that `dialog_id` belongs to the same tenant as that token.
Current flow (simplified):
1. Route extracts bearer token and checks `APIToken.query(beta=token)`.
2. If token exists, request is accepted.
3. Downstream service resolves dialog globally by ID
(`DialogService.get_by_id(dialog_id)` in `conversation_service.py`).
4. No tenant ownership check is enforced for `dialog_id`.
Impact: Any user with a valid API key can attempt arbitrary `dialog_id`
values and access/invoke chatbots outside their own tenant boundary if
IDs are known/guessed/leaked.
Security classification:
- Vulnerability class: Broken Access Control (IDOR, OWASP Top 10 A01)
- Severity recommendation: Critical
- Exploit prerequisite: any valid API key + discoverable target
`dialog_id`
## Expected behavior
Requests to `/chatbots/<dialog_id>/completions` and
`/chatbots/<dialog_id>/info` must be authorized only when:
1. bearer token is valid, and
2. `dialog_id` belongs to the same `tenant_id` as the token.
Otherwise, reject with authorization failure (e.g., 403 or
404-equivalent policy).
## Steps to reproduce
1. Prepare two tenants:
- Tenant A with API key `TOKEN_A`
- Tenant B with chatbot `dialog_id = DIALOG_B`
2. Send request from Tenant A to Tenant B chatbot completion endpoint:
```bash
curl -X POST "https://<host>/chatbots/DIALOG_B/completions" \
-H "Authorization: Bearer TOKEN_A" \
-H "Content-Type: application/json" \
-d '{"question":"hello","stream":false}'
```
3. Observe request is processed (or reaches dialog resolution) without
tenant ownership rejection.
4. Repeat against info endpoint:
```bash
curl -X GET "https://<host>/chatbots/DIALOG_B/info" \
-H "Authorization: Bearer TOKEN_A"
```
5. Observe the same missing ownership enforcement.
## Additional information
Affected code paths:
- `api/apps/sdk/session.py`
- `chatbot_completions(dialog_id)`
- `chatbots_inputs(dialog_id)`
- `api/db/services/conversation_service.py`
- `async_iframe_completion(...)` uses global dialog lookup
Suggested fix:
1. In both chatbot endpoints:
- Resolve `tenant_id = objs[0].tenant_id` from validated token.
- Fetch dialog with tenant-scoped query
(`DialogService.query(id=dialog_id, tenant_id=tenant_id)`).
- Reject if dialog is not found/owned by tenant.
2. Defense in depth:
- Require and enforce `tenant_id` in service-layer dialog resolution for
external flows.
- Avoid global `get_by_id(dialog_id)` where user-controlled dialog IDs
are reachable.
3. Add regression tests:
- Positive: same-tenant token + dialog succeeds.
- Negative: cross-tenant token + dialog fails for both endpoints.
### What problem does this PR solve?
Restrict file move operations: prevent moving a folder to itself or to
one of its own subfolders.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
### What problem does this PR solve?
Do not bypass threshold for rerank when metadata filter is enabled
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
### What problem does this PR solve?
Fixes#14651.
`kb_prompt()` in `rag/prompts/generator.py` crashes with
`AttributeError: 'NoneType' object has no attribute 'items'` during
agent citation generation when a retrieved chunk carries
`document_metadata: null`.
**Root cause.** The crash happens at `rag/prompts/generator.py:132-133`:
```python
meta = ck.get("document_metadata", {})
for k, v in meta.items():
```
`dict.get(key, default)` only returns the default when the key is
*missing*. When the key is present with an explicit `None` value,
`.get()` returns `None`, and `.items()` crashes.
**How the chunk gets `None`.** It's a round-trip inside RAGFlow itself,
not bad input from retrieval:
1. The agent stores retrieved chunks via `agent/canvas.py:814`, which
routes them through `chunks_format()`.
2. `rag/prompts/generator.py:61` canonicalizes the field with
`chunk.get("document_metadata")` (no default), so chunks without
metadata become `{"document_metadata": None, ...}`.
3. `agent/component/agent_with_tools.py:314` feeds those canonicalized
chunks back into `kb_prompt()` for citation generation, and
`.get("document_metadata", {})` no longer protects us.
**Fix.** One-line change at `rag/prompts/generator.py:132`: use
`ck.get("document_metadata") or {}` so an explicit `None` is also
coerced to `{}`.
The line-61 `None` is intentionally part of the API/UI contract — the
frontend handles it via optional chaining
(`web/src/components/markdown-content/index.tsx:184`,
`web/src/pages/next-search/search-view.tsx:217`) — so the fix belongs at
the consumer, not the producer.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
- [ ] New Feature (non-breaking change which adds functionality)
- [ ] Documentation Update
- [ ] Refactoring
- [ ] Performance Improvement
- [ ] Other (please describe):
### What problem does this PR solve?
```
RAGFlow(user)> logout;
SUCCESS
```
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
Signed-off-by: Jin Hai <haijin.chn@gmail.com>
### What problem does this PR solve?
1. **Fix Global State Pollution in Local Providers (Critical Bug):** -
Resolved a severe concurrency and architecture issue in
`model_service.go`. Previously, `ListSupportedModels` would permanently
overwrite the global provider singleton with a localized URL instance
(`driver.NewInstance`). This caused cross-request contamination in
multi-tenant environments.
- Fixed `CheckProviderConnection` for local models (LM Studio, vLLM,
Ollama). It now properly creates a localized driver copy and injects the
`base_url` before testing the connection, entirely eliminating the
false-positive `missing base URL` error without polluting the global
state.
2. **Implement `VolcEngine` Embeddings:** - Fully implemented the
`Encode` method for the `volcengine` provider, enabling text embedding
capabilities for VolcEngine models.
3. **Enhance Region Validation in `SiliconFlow`:** - Added a strict
empty string check (`*apiConfig.Region != ""`) alongside the existing
`nil` check when parsing regions. This ensures that if an empty string
is passed, the system safely falls back to the `"default"` region,
preventing malformed URL requests and `unsupported protocol scheme`
errors.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
- [x] New Feature (non-breaking change which adds functionality)
### What problem does this PR solve?
1. Update API URL
2. Add password encryption
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
Signed-off-by: Jin Hai <haijin.chn@gmail.com>
### What problem does this PR solve?
Update the type of tenant_rerank_id in validation.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
## Summary
- When an agent workflow has multiple `UserFillUp` pause points,
`canvas.run()` calls `reset(True)` on **all** components at the start of
each run. This clears outputs from components that completed in prior
runs, so downstream references like `{Agent:XXX@content}` resolve to
`None`.
- This fix only resets components on the **current execution path**
(`self.path`), preserving outputs from previously completed components.
## Problem
In a multi-step agent (e.g. draft email → user confirms → send email):
1. First `run()`: Agent drafts content, UserFillUp pauses for user input
→ Agent output is saved
2. Second `run()`: User submits input, but `reset(True)` clears **all**
components including the Agent that already completed
3. Email component references `{Agent:XXX@content}` → gets `None`
instead of the draft
This affects **all** agents that reference upstream component outputs
after a UserFillUp pause point.
## Fix
```python
# Before: reset ALL components
for k, cpn in self.components.items():
self.components[k]["obj"].reset(True)
# After: only reset components on current execution path
path_set = set(self.path)
for k, cpn in self.components.items():
if k in path_set:
self.components[k]["obj"].reset(True)
```
`self.path` already tracks the current execution path. For agents
without UserFillUp (single run), `path` contains all components, so
behavior is unchanged.
## Test plan
- [x] Agent with single UserFillUp: outputs from prior components are
preserved after resume
- [x] Agent with multiple UserFillUp: each resume preserves all
previously completed outputs
- [x] Agent without UserFillUp: behavior unchanged (all components in
path, all reset)
- [x] Webhook-triggered agents: unaffected (path includes all components
on first run)
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: wanghualoong <wanghualoong@gmail.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
### What problem does this PR solve?
add compatibility route for document download under /v1
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
## Summary
- **Collapsible thinking**: Replace `<section>` with `<details>` for
`<think>` content, so model thinking output is collapsed by default
(click to expand). Works for all models that output `<think>` tags
(Qwen3, DeepSeek, Gemini, Claude, etc.).
- **Fix double thinking tags**: When reasoning/deep research mode is
enabled in knowledge base chat, both the retrieval progress and model
thinking were wrapped in `<think>` tags, producing two "Thinking..."
blocks. Now retrieval progress uses a dedicated `<retrieving>` tag
rendered as a separate "Retrieving..." collapsible with a distinct green
accent.
### Before
- Thinking content displayed as flat gray-bordered `<section>`,
occupying significant screen space
- Deep research + model thinking both use `<think>` → two identical
"Thinking..." blocks
### After
- Thinking content collapsed by default in a `<details>` element, click
"Thinking..." to expand
- Deep research shows "Retrieving..." (green border), model thinking
shows "Thinking..." (gray border)
## Changes
**Backend (`api/db/services/dialog_service.py`)**
- Deep research callback: replace `start_to_think`/`end_to_think` marker
flags with direct `<retrieving>`/`</retrieving>` answer text
**Frontend**
- `web/src/utils/chat.ts`: `replaceThinkToSection()` now uses
`<details>` instead of `<section>`; add new
`replaceRetrievingToSection()`
- 4 tsx files: import and pipe `replaceRetrievingToSection`, whitelist
`details`, `summary`, `retrieving` in DOMPurify `ADD_TAGS`
- 4 less files: `section.think` → `details.think` with `<summary>`
styles; add `details.retrieving` with green accent; dark mode and RTL
variants
## Test plan
- [ ] Open a chat WITHOUT knowledge base, ask a question to a model with
thinking (e.g. Qwen3) → thinking content should be collapsed by default,
click "Thinking..." to expand
- [ ] Open a chat WITH knowledge base and reasoning enabled, ask a
question → "Retrieving..." (green) shows retrieval progress,
"Thinking..." (gray) shows model thinking, each independently
collapsible
- [ ] Verify dark mode renders correctly for both collapsible blocks
- [ ] Verify RTL layout renders correctly
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: wanghualoong <wanghualoong@gmail.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
### What problem does this PR solve?
Closes#14618.
The `GET /v1/document/get/<doc_id>` endpoint in
`api/apps/document_app.py` was protected only by `@login_required` and
called `DocumentService.get_by_id(doc_id)` without verifying that the
document's knowledge base belonged to the requesting user's tenant. Any
authenticated user who knew (or guessed) a document ID could download
files belonging to any other tenant — a cross-tenant IDOR.
This PR adds a `DocumentService.accessible(doc_id, current_user.id)`
check before serving the file. The helper already exists and joins
`Document` → `Knowledgebase` → `UserTenant` to verify the requesting
user belongs to the tenant that owns the document's KB. The same pattern
is already used by `api/apps/restful_apis/document_api.py` and mirrors
the tenant scoping in the SDK route at `api/apps/sdk/doc.py`.
The check returns the existing `"Document not found!"` error for both
non-existent and inaccessible documents, so attackers cannot use the
response to enumerate valid doc IDs across tenants.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
- [x] Other (please describe): Security fix (cross-tenant IDOR /
authorization bypass)
### What problem does this PR solve?
The Aliyun Go driver shipped with a stub \`Encode\` method that returned
\`no such method\`, even though \`conf/models/aliyun.json\` already
wires the OpenAI-compatible embeddings URL suffix at
\`compatible-mode/v1/embeddings\`. The same config also did not list any
embedding models, so the picker had nothing to select.
So an Aliyun tenant who wanted to use Tongyi text-embedding-v3 or v4 in
the Go layer could not, even though the upstream endpoint is public and
uses the standard \`POST /v1/embeddings\` shape that the SiliconFlow and
ZhipuAI
drivers already support.
This PR fills the gap.
### What this PR includes
- \`conf/models/aliyun.json\`: add \`text-embedding-v4\` and
\`text-embedding-v3\` to the \`models\` array.
- \`internal/entity/models/aliyun.go\`: replace the \`Encode\` stub with
a real implementation. Adds a small local response type that matches the
OpenAI-compatible shape.
No factory change. No interface change.
### How the driver works
- Validate \`apiConfig\` and the API key, validate the model name,
resolve the region with a default fallback, build the
URL from \`BaseURL[region] + URLSuffix.Embedding\`.
- Send all input texts in one request as the \`input\` array, the same
OpenAI-compatible shape the SiliconFlow \`Encode\`
uses.
- Parse \`data[*].embedding\` and copy each slice into a \`[][]float64\`
indexed by \`data[*].index\` so the output order matches the input order
even if the API returns items in a different order.
- Handle both \`float64\` and \`float32\` element types.
- Empty input returns \`[][]float64{}\` with no HTTP call.
- Non-200 responses propagate the upstream status line and body.
- A final pass checks every input slot got a vector and returns a clear
error if any slot is still nil.
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
### How was this tested?
- \`go build ./internal/entity/models/...\` in a clean go 1.25 image
returns exit 0.
- The full method set on \`AliyunModel\` still matches the
\`ModelDriver\` interface.
- Pattern parity with the existing SiliconFlow Encode implementation.
Closes#14646
---------
Co-authored-by: Jin Hai <haijin.chn@gmail.com>
### What problem does this PR solve?
1. implement `rerank`, `embedding`, `balance`, `checkConnet` method for
`OpenRouter`
2. delete `chat` method in `internal/entity/models/volcengine.go`
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
- [x] Refactoring
### What problem does this PR solve?
Since API is updated, CLI login failed.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
Signed-off-by: Jin Hai <haijin.chn@gmail.com>
### Related issues
Closes#14648
### What problem does this PR solve?
This PR fixes an authorization flaw in `POST /files/link-to-datasets`.
Before this change, the endpoint only checked whether the supplied
`file_ids` and `kb_ids` existed. It did not verify whether the
authenticated user was actually allowed to access those files or target
datasets. As a result, an authenticated user who knew valid IDs could
relink another user's files to arbitrary datasets.
This was especially risky because the relinking flow is state-changing:
the background worker removes existing file-document mappings and then
recreates documents under the attacker-supplied dataset IDs.
This change makes the route enforce the same permission model already
used by nearby file and document operations:
- each resolved file must pass `check_file_team_permission(...)`
- each target dataset must pass `check_kb_team_permission(...)`
- authorization is enforced before scheduling background relinking work
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
- [ ] New Feature (non-breaking change which adds functionality)
- [ ] Documentation Update
- [ ] Refactoring
- [ ] Performance Improvement
- [ ] Other (please describe):
### Testing
- Added regression coverage in
`test/testcases/test_web_api/test_file_app/test_file2document_routes_unit.py`
- Covered:
- unauthorized file access is rejected
- unauthorized dataset access is rejected
- existing success path still returns immediately after scheduling
background work
- Attempted to run:
- `python -m pytest
test\\testcases\\test_web_api\\test_file_app\\test_file2document_routes_unit.py
-q`
- Local execution in this workspace is currently blocked by missing test
dependencies during bootstrap, including `ragflow_sdk`
---------
Co-authored-by: jony376 <jony376@gmail.com>
### What problem does this PR solve?
The Gitee AI Go driver shipped with a stub \`Rerank\` method that
returned \`Rerank not implemented\`, even though
\`conf/models/gitee.json\` already wires the rerank URL suffix at
\`\"rerank\": \"rerank\"\`. The same config did not list any
rerank model, so the picker had nothing to select.
So a Gitee tenant could not use BAAI/bge-reranker-v2-m3 as a reranker
through the Go layer today, even though the
infrastructure was one config entry and one method body away.
### What this PR includes
- \`conf/models/gitee.json\`: add \`BAAI/bge-reranker-v2-m3\` to the
\`models\` array.
- \`internal/entity/models/gitee.go\`: replace the \`Rerank\` stub with
a real implementation. Adds two small local types
that match the OpenAI-compatible \`/rerank\` shape already used by the
SiliconFlow and ZhipuAI drivers.
No factory change. No interface change.
### How the driver works
- Validate \`apiConfig\` and the API key, validate the model name,
resolve the region with a default fallback, build the
URL from \`BaseURL[region] + URLSuffix.Rerank\`.
- Use a per-call \`context.WithTimeout(30s)\` and
\`http.NewRequestWithContext\`, matching the pattern the
recently merged Aliyun Encode and the OpenAI driver already use.
- Send \`{model, query, documents, top_n, return_documents:false}\` in
the body.
- Parse \`results[*].relevance_score\` and copy each score into the
output slice indexed by \`results[*].index\`, so the
output order matches the input order even if the API returns items in a
different order.
- Empty input returns \`[]float64{}\` with no HTTP call.
- An out-of-range result index returns a clear error rather than
silently skipping the entry.
- Non-200 responses propagate the upstream status line and body.
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
### How was this tested?
- \`go build ./internal/entity/models/...\` in a clean go 1.25 image
returns exit 0.
- The full method set on \`GiteeModel\` still matches the
\`ModelDriver\` interface.
- Pattern parity with the existing SiliconFlow Rerank and the recently
merged ZhipuAI Rerank (#14608).
Closes#14655
## Summary
Fixes file collision between different datasets when using Azure Blob
storage (SPN or SAS authentication).
## Bug
azure_spn_conn.py and zure_sas_conn.py ignored the ucket parameter
entirely, storing all files flat with just the filename. This caused
files with the same name from different datasets (knowledge bases) to
overwrite each other.
## Fix
Prepend bucket/ as a path prefix in all methods (put,
m, get, obj_exist, get_presigned_url, health) to match the behavior of
MinIO and S3 implementations.
## Changes
- **rag/utils/azure_spn_conn.py**: Added {bucket}/ prefix to file paths
in all operations
- **rag/utils/azure_sas_conn.py**: Same fix applied for consistency
(also noted in the original issue)
## Testing
Manual verification: files from different datasets now stored under
distinct bucket/ prefixes, preventing collisions.
Fixes#14159
Co-authored-by: Hunter <hunter@yitong.ai>
Co-authored-by: Jin Hai <haijin.chn@gmail.com>
### What problem does this PR solve?
The NVIDIA Go driver added in #14623 has a real chat path, but
\`ListModels\` and \`CheckConnection\` are stubs that always return \`no
such method\`. So:
- The model picker cannot auto-populate available NVIDIA NIM model ids.
Users have to type the full id by hand (e.g.
\`abacusai/dracarys-llama-3.1-70b-instruct\`).
- The "Check connection" button always fails for NVIDIA, even when the
base URL is reachable and the API key is accepted.
NVIDIA NIM is OpenAI-compatible. \`/v1/models\` works with the same
Bearer token used for chat. The
\`conf/models/nvidia.json\` file already wires the \`models\`
url_suffix, so no config change is needed.
### What this PR includes
- \`internal/entity/models/nvidia.go\`:
- \`ListModels\` now calls
\`GET ${BaseURL}/${URLSuffix.Models}\`, parses
\`response.data[*].id\`, and returns the list. Same shape
as the moonshot, xai, and openai drivers.
- \`CheckConnection\` now calls \`ListModels\` and returns its
error. Same pattern xai, moonshot, deepseek, aliyun, and
gitee already use.
\`Balance\`, \`Encode\`, and \`Rerank\` are still stubs in this PR and
can be added in follow-ups.
No JSON change. No factory change. No interface change.
### How the implementation works
- Region resolution falls back to \`default\` when the supplied region
is unknown, so a stray region value does not break a valid request.
- The Authorization header is only set when \`apiConfig\` and \`ApiKey\`
are non-nil and non-empty. This avoids a nil-pointer dereference and
lets self-hosted NIM deployments without a key still work.
- Non-200 responses propagate the upstream status line and body so the
user sees a real error message.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
### How was this tested?
- \`go build ./internal/entity/models/...\` in a clean go 1.25 image
(the go.mod minimum) returns exit 0.
- The full method set on \`NvidiaModel\` still matches the
\`ModelDriver\` interface.
- Pattern parity with the existing xai, moonshot, deepseek, aliyun,
gitee, and openai drivers.
Closes#14635
Closes#14631
### What problem does this PR solve?
The DeepSeek Go driver shipped with a stub \`Balance\` method that
returned \`no such method\`, even though DeepSeek exposes a public \`GET
/user/balance\` endpoint that works with the same Bearer token used for
chat.
So the "Balance" panel in the model provider UI always shows an error
for DeepSeek tenants, while it already works for Moonshot and Gitee.
This PR fills the gap.
### What this PR includes
- \`conf/models/deepseek.json\`: add \`\"balance\": \"user/balance\"\`
under \`url_suffix\` so the driver can build the URL from config the
same way the other endpoints do.
- \`internal/entity/models/deepseek.go\`: replace the \`Balance\` stub
with a real implementation. Adds a small local response type
\`deepseekBalanceResponse\` that matches the upstream shape.
No factory change. No interface change.
### How the driver works
- Validate \`apiConfig\` and the API key, resolve the region (with a
\`default\` fallback), and build the URL from \`BaseURL[region] +
URLSuffix.Balance\`.
- GET the URL with \`Authorization: Bearer <api_key>\`.
- Parse the upstream response:
\`\`\`json
{
\"is_available\": true,
\"balance_infos\": [
{\"currency\": \"USD\", \"total_balance\": \"10.00\", ...},
{\"currency\": \"CNY\", \"total_balance\": \"70.00\", ...}
]
}
\`\`\`
\`total_balance\` is a string in the upstream API, so the driver parses
it with \`strconv.ParseFloat\`.
- Return the first balance entry as \`{\"balance\": <float>,
\"currency\": <string>}\`, the same shape the Moonshot driver returns.
The UI can render it with no provider-specific code.
### Edge cases
- Missing or empty API key returns a clear local error before any HTTP
call.
- Empty \`balance_infos\` returns a clear \"no balance info in
response\" error rather than a zero-value silent success.
- Non-numeric \`total_balance\` returns a clear parse error.
- Non-200 responses propagate the upstream status line and body so the
user can see why the call failed.
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
### How was this tested?
- \`go build ./internal/entity/models/...\` in a clean go 1.25 image
(the go.mod minimum) returns exit 0.
- The full method set on \`DeepSeekModel\` still matches the
\`ModelDriver\` interface.
- Pattern parity with the existing Moonshot and Gitee Balance
implementations.
### What problem does this PR solve?
1. **Implement `OpenRouter` Provider:** Fully support OpenRouter AI
models (e.g., `gemma`, `minimax`). Includes robust handling of
Server-Sent Events (SSE) streams, error event interception, and proper
parsing of both `reasoning_content` and standard `content`.
2. **Fix BaseURL Resolution Bug:** Fixed a critical edge case in region
configuration parsing. Added a strict empty string check
(`*apiConfig.Region != ""`) alongside the `nil` check. This ensures that
if the UI passes an empty string, the system correctly falls back to the
`"default"` region, preventing `unsupported protocol scheme ""` errors
during HTTP requests.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
- [x] New Feature (non-breaking change which adds functionality)
### What problem does this PR solve?
The SiliconFlow Go driver shipped with a stub \`Balance\` method that
returned \`no such method\`, even though SiliconFlow exposes a public
\`GET /v1/user/info\` endpoint that returns the account balance per
currency.
So the "Balance" panel in the model provider UI always shows an error
for SiliconFlow tenants, while it already works for
Moonshot and Gitee. This PR fills the gap.
### What this PR includes
- \`conf/models/siliconflow.json\`: add \`\"balance\": \"user/info\"\`
under \`url_suffix\` so the driver builds the URL from config.
- \`internal/entity/models/siliconflow.go\`: replace the \`Balance\`
stub with a real implementation. Adds a small local response type that
matches the upstream shape.
No factory change. No interface change.
### How the driver works
- Validate \`apiConfig\` and the API key, resolve the region with a
default fallback, and build the URL from \`BaseURL[region] +
URLSuffix.Balance\`.
- GET the URL with \`Authorization: Bearer <api_key>\`.
- Parse the upstream response. SiliconFlow returns balance fields as
strings, so the driver parses them with \`strconv.ParseFloat\`. It
prefers \`totalBalance\` over \`balance\` when both are present.
- Return \`{\"balance\": <float>, \"currency\": \"CNY\"}\`, the same
shape the Moonshot driver returns. The UI can render it
with no provider-specific code.
### Edge cases
- Missing or empty API key returns a clear local error before any HTTP
call.
- An unknown region falls back to the default base URL.
- Empty \`balance\` and \`totalBalance\` returns a clear "no balance
info in response" error rather than a zero-value silent success.
- Non-numeric balance string returns a clear parse error.
- Non-200 responses propagate the upstream status line and body.
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
### How was this tested?
- \`go build ./internal/entity/models/...\` in a clean go 1.25 image
returns exit 0.
- The full method set on \`SiliconflowModel\` still matches the
\`ModelDriver\` interface.
- Pattern parity with the existing Moonshot and Gitee Balance
implementations.
Closes#14642
### What problem does this PR solve?
Three Go drivers had `CheckConnection` returning a hardcoded `no such
method` error, even though each one already has a working `ListModels`
that hits the configured base URL with the configured API key. So the
"Check connection" button in the model provider UI always failed for
these three providers, even when the underlying setup was fine.
Affected drivers:
- `internal/entity/models/ollama.go`
- `internal/entity/models/lmstudio.go`
- `internal/entity/models/vllm.go`
This is a real user-facing gap because Ollama and LM Studio are two of
the most popular local LLM runners, and vLLM is widely used for
self-hosted deployments.
### What this PR includes
For each of the three drivers, replace the stub with a small
implementation that calls `ListModels` and returns its error:
```go
func (o *OllamaModel) CheckConnection(apiConfig *APIConfig) error {
_, err := o.ListModels(apiConfig)
return err
}
```
This is the exact pattern that xai, moonshot, deepseek, aliyun, and
gitee already use for the same method.
No JSON change. No factory change. No interface change.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
### How was this tested?
- `go build ./internal/entity/models/...` in a clean go 1.25 image (the
go.mod minimum) returns exit 0.
- The full ModelDriver interface still resolves on each driver
(NewInstance, Name, ChatWithMessages, ChatStreamlyWithSender, Encode,
Rerank, ListModels, Balance, CheckConnection).
- Pattern parity with the existing xai, moonshot, deepseek, aliyun, and
gitee CheckConnection methods.
Closes#14609
## Summary
Update the Turkish locale file to match the latest English locale keys.
## Changes
- Add missing Turkish translations for the new Skills and Skill Search
sections
- Add newly introduced common, header, dataset, settings, and agent
workflow strings
- Align renamed flow keys such as file format options and list
operations with the English source
- Add empty-state strings for skill spaces
## Validation
- Compared web/src/locales/en.ts and web/src/locales/tr.ts: 0 missing
keys, 0 extra keys
- Checked jsonjoy-builder locale: Turkish is already complete
- Checked translated README variants: no new Turkish-specific
documentation gap found
- VS Code diagnostics: no errors in web/src/locales/tr.ts
Co-authored-by: bakiburakogun <bakiburakogun@users.noreply.github.com>
### What problem does this PR solve?
Fixes#14412.
`common.metadata_utils.meta_filter` evaluates user-defined metadata
conditions in Python after `DocMetadataService.get_flatted_meta_by_kbs`
loads the entire `meta_fields` table into memory. Past a few thousand
documents per knowledge base this becomes a memory bottleneck and a
wasted ES round-trip — every filter request currently fetches up to
10000 metadata rows even when the resulting `doc_ids` list is tiny.
This PR adds an ES push-down path that translates the same filter
language into a `bool` query and returns just the matching document IDs.
**Changes**
- `common/metadata_es_filter.py` *(new)*: pure-Python translator from
the RAGflow filter list to ES DSL. Covers every operator the in-memory
path supports (`=`, `≠`, `>`, `<`, `≥`, `≤`, `in`, `not in`, `contains`,
`not contains`, `start with`, `end with`, `empty`, `not empty`) with
`case_insensitive: true` on `prefix` and `wildcard` for parity with the
existing lower-cased Python comparisons. User wildcard metacharacters
are escaped before being injected into `wildcard` patterns. Negative
operators (`≠`, `not in`, `not contains`, ranges) are wrapped with an
`exists` guard so they do not accidentally match documents missing the
key, matching the legacy `if k not in metas` behaviour.
- `api/db/services/doc_metadata_service.py`: new
`DocMetadataService.filter_doc_ids_by_meta_pushdown(kb_ids, filters,
logic)` that returns the doc IDs ES matched, or `None` to signal the
caller should fall back to the in-memory path. Returns `None` when the
active doc store is Infinity (`meta_fields` is a JSON column, not a
dotted-object mapping), when any filter cannot be expressed in DSL
(`UnsupportedMetaFilter`), or when the ES request or metadata index
lookup errors.
- `common/metadata_utils.py`: `apply_meta_data_filter` accepts an
optional `kb_ids` argument. When supplied, conditions go through
push-down first via a new `_try_meta_pushdown` helper; on `None` the
function falls back to the original `meta_filter` call. Default
behaviour is unchanged for callers that don't pass `kb_ids`.
- Updated all four callers (`agent/tools/retrieval.py`,
`api/db/services/dialog_service.py` ×2,
`api/apps/services/dataset_api_service.py`, `api/apps/sdk/session.py`)
to forward `kb_ids` so the push-down path is exercised in production.
- `test/unit_test/common/test_metadata_es_filter.py` *(new)*: 35 unit
tests covering every operator's DSL shape, value coercion
(`ast.literal_eval`, lowercasing, ISO-date pass-through), wildcard
escaping, OR-logic wrapping that protects negative clauses, and the
doc-ID extractor.
**Behaviour preserved**
- The in-memory `meta_filter` is untouched and still services every
fallback case (Infinity backend, unknown operators, ES outages).
- The eligibility / credibility / issue-multiplier semantics described
in the LLM-driven `auto` and `semi_auto` modes still hand the LLM the
full in-memory `metas` dict to choose conditions from. Only the
*evaluation* of those generated conditions is pushed down.
- Existing tests in
`test/unit_test/common/test_metadata_filter_operators.py` continue to
pass (14/14).
**Test plan**
- `pytest test/unit_test/common/test_metadata_es_filter.py` — 35 passed.
- `pytest test/unit_test/common/test_metadata_filter_operators.py` — 14
passed.
- `ruff check` clean on every modified file.
- Reviewer please validate the ES query shapes against a live cluster —
particularly `case_insensitive` on `wildcard` and `prefix` (requires ES
7.10+) and the `exists` + `must_not` pairing for `≠`.
**Notes**
- The first cut caps each push-down request at 10000 results, matching
the existing `get_flatted_meta_by_kbs` limit, and logs a warning when
the cap is hit. A `search_after` follow-up would let us drop the cap
entirely once the push-down path is validated.
- Operator parity with the in-memory path is exact for the canonical
unicode operators (`≥`, `≤`, `≠`) used internally; the ASCII aliases
(`>=`, `<=`, `!=`) are normalised by `convert_conditions` before they
reach the translator.
### Type of change
- [x] Performance Improvement
---------
Co-authored-by: sxxtony <sxxtony@users.noreply.github.com>
### What problem does this PR solve?
Feat: support local provider for code exec component & remove some
outdated models
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
## Summary
Fixes#14159 — files from different datasets can overwrite each other in
Azure Blob storage.
## Problem
Both `azure_spn_conn.py` and `azure_sas_conn.py` ignore the `bucket`
parameter in all storage operations (`put`, `get`, `rm`, `obj_exist`,
`get_presigned_url`). Files are stored flat using only the filename, so
two datasets containing a file with the same name will overwrite each
other.
The MinIO and S3 implementations correctly use the bucket (typically the
knowledge base ID) as a path prefix to create logical folder isolation:
- MinIO: uses `use_prefix_path` decorator → `{orig_bucket}/{fnm}`
- S3: uses `use_prefix_path` decorator → `{prefix_path}/{bucket}/{fnm}`
## Fix
Prepend `{bucket}/` to the file path in all 5 operations across both
Azure connector files:
| File | Methods fixed |
|------|---------------|
| `azure_spn_conn.py` | `put`, `get`, `rm`, `obj_exist`,
`get_presigned_url` |
| `azure_sas_conn.py` | `put`, `get`, `rm`, `obj_exist`,
`get_presigned_url` |
This matches the existing convention where `bucket` is the knowledge
base ID used as a directory prefix.
## ⚠️ Migration Note
Existing Azure SPN/SAS deployments have files stored without the bucket
prefix. After this fix, new files will be stored under
`{bucket}/{filename}` while existing files remain at `{filename}`. A
one-time migration script or manual file move may be needed for existing
deployments. New deployments are unaffected.
## Testing
- Verified the fix is consistent across all 5 methods in both files
- The `health()` method is intentionally left unchanged as it uses a
hardcoded test filename without bucket semantics
Co-authored-by: Jin Hai <haijin.chn@gmail.com>
### What problem does this PR solve?
The ZhipuAI Go driver had a stub Rerank method that returned "not
implemented", even though conf/models/zhipu-ai.json already ships
glm-rerank as a rerank model and the rerank URL suffix is already wired
in url_suffix:
```json
"url_suffix": {
...
"rerank": "rerank"
},
"models": [
{"name": "glm-rerank", "model_types": ["rerank"]},
...
]
```
So the config was ready but the driver was not. A tenant who picked
glm-rerank in the Go layer could not actually run a rerank call. This PR
fills the gap so the listed model works end to end.
### What this PR includes
- `internal/entity/models/zhipu-ai.go`: real implementation of
`ZhipuAIModel.Rerank`, plus two small local types (`zhipuRerankRequest`,
`zhipuRerankResponse`) that mirror the standard OpenAI-compatible rerank
shape used by SiliconFlow.
No factory change. No JSON change. No interface change.
### How the driver works
- POST to `${BaseURL}/${URLSuffix.Rerank}` (resolves to
`https://open.bigmodel.cn/api/paas/v4/rerank` with the default config),
reusing the existing httpClient on the driver.
- Validate apiConfig and the API key, validate the model name, and
resolve the region. Return a clear local error before any HTTP call when
something is missing.
- Send `{model, query, documents, top_n, return_documents: false}` in
the body, the same shape the SiliconFlow driver already uses.
- Walk `results[*].relevance_score` and copy each score into the output
slice indexed by `results[*].index`, so the output order matches the
input order even if the API returns results in a different order.
- Empty `texts` input returns an empty `[]float64` with no HTTP call.
- Non-200 responses propagate the upstream status line and body.
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
### How was this tested?
- `go build ./internal/entity/models/...` in a clean go 1.25 image (the
go.mod minimum) returns exit 0.
- The full method set on `ZhipuAIModel` still matches the `ModelDriver`
interface (NewInstance, Name, ChatWithMessages, ChatStreamlyWithSender,
Encode, ListModels, Balance, CheckConnection, Rerank).
- Pattern parity with the existing SiliconFlow Rerank implementation
(`internal/entity/models/siliconflow.go`).
Closes#14607
Fixes#14551
### What problem does this PR solve?
The Moodle connector did not let the sync runner clean up indexed
documents that were deleted from the source. Other connectors such as
dropbox, seafile, webdav, and rss already do this through a slim
snapshot pass. This PR adds the same support for Moodle.
When `sync_deleted_files` is on, the runner now asks the Moodle
connector for a lightweight list of every module id that could be
indexed. The runner then compares this list with the index and removes
any indexed document whose id is not in the list.
The slim pass does not download files. It only goes through courses and
modules and yields ids. The id format matches the ids that the loader
produces, so the match is exact.
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
### Notes
- `MoodleConnector` now also implements `SlimConnectorWithPermSync`.
- New `retrieve_all_slim_docs_perm_sync` yields slim docs with the same
ids the loader uses (`moodle_resource_<id>`, `moodle_forum_<id>`,
`moodle_page_<id>`, `moodle_book_<id>`, `moodle_assign_<id>`,
`moodle_quiz_<id>`).
- The `Moodle` sync class now returns `(document_generator, file_list)`
so the runner can do the cleanup. If the slim snapshot fails,
`file_list` is set back to `None` and the run continues without cleanup.
- The web data source map exposes `syncDeletedFiles` for Moodle so the
option shows up in the UI.
### How was this tested?
- `ruff check` passes on the changed Python files.
- Manual review of the produced slim ids against the ids the loader
builds in `_process_resource`, `_process_forum`, `_process_page`,
`_process_book`, and `_process_activity`.
- Behavior parity with the merged dropbox (#14476), seafile (#14499),
webdav (#14491), and rss (#14493) PRs.
### What problem does this PR solve?
Since secret key get and set logic is updated, the go server also need
to update.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
---------
Signed-off-by: Jin Hai <haijin.chn@gmail.com>
Fixes#14159
## Problem
The `put()`, `get()`, `rm()`, and `obj_exist()` methods in both
`azure_spn_conn.py` and `azure_sas_conn.py` ignore the `bucket`
parameter entirely, storing all files flat using only the filename. This
causes files from different datasets to overwrite each other when they
share the same filename.
By contrast, the MinIO and S3 implementations correctly use the bucket
(typically the knowledge base ID) as a path prefix, creating logical
folder isolation like `{kb_id}/{filename}`.
## Solution
Prepend the `bucket` parameter as a path prefix to all file operations
in both Azure storage implementations:
- `azure_spn_conn.py`: `create_file`, `delete_file`, `get_file_client`
now use `f"{bucket}/{fnm}"`
- `azure_sas_conn.py`: `upload_blob`, `delete_blob`, `download_blob`,
`get_blob_client` now use `f"{bucket}/{fnm}"`
This matches the behavior of all other storage backends (MinIO, S3) and
prevents filename collisions across knowledge bases.
## Testing
- Verified the fix aligns with how MinIO/S3 connectors handle the bucket
parameter
- The `health()` method is left unchanged as it uses a fixed test path
for connectivity checks only
Co-authored-by: octo-patch <octo-patch@github.com>
Co-authored-by: Jin Hai <haijin.chn@gmail.com>
### What problem does this PR solve?
preserve doc generator download metadata
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
### What problem does this PR solve?
Follow on PR: https://github.com/infiniflow/ragflow/pull/14602
to fix: team member cannot edit agent.
new behavior: beside delete, everything is allowed for team member.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
### What problem does this PR solve?
1. **Implement `Nvidia` Provider:** Fully support NVIDIA NIM APIs with
robust parameter handling (including the `thinking` parameter) and safe
URL merging in `NewInstance`.
2. **Fix Misleading CLI Errors:** Corrected a bug in `common_command.go`
where failed chat requests inaccurately reported `failed to list
instance models`.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
- [x] New Feature (non-breaking change which adds functionality)
### What problem does this PR solve?
Feat: enable sync deleted files for RDBMS & fix remove last file issue
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
- [x] New Feature (non-breaking change which adds functionality)
### What problem does this PR solve?
Add a Go driver for OpenAI (GPT models).
The config file conf/models/openai.json has been in the repo for a while
with the full GPT-5 model list, but
internal/entity/models/factory.go had no case for "openai". So any
tenant that configured OpenAI as a model provider in the Go layer fell
through to the default branch and got the dummy driver. Chat, list
models, and check connection all returned dummy responses instead of
reaching the API.
OpenAI is the most commonly requested provider and the JSON config
already ships with the repo, so this gap is high impact even though the
JSON has been there for some time.
### What this PR includes
- New file internal/entity/models/openai.go with an OpenAIModel that
implements the ModelDriver interface.
- factory.go: route the "openai" provider name to NewOpenAIModel.
- conf/models/openai.json: add "models": "models" under url_suffix so
ListModels can hit /v1/models with no hardcoded fallback.
### How the driver works
- OpenAI exposes the canonical OpenAI-compatible API at
https://api.openai.com/v1.
- ChatWithMessages and ChatStreamlyWithSender post to /chat/completions
in the same shape the moonshot, vllm, and xai drivers use.
- ListModels and CheckConnection call /models to list available ids and
confirm the API key works.
- reasoning_content is passed through for the o-series and other
reasoning models, in both the non-stream and stream paths.
- Encode (embeddings) is left as "not implemented" for now, the same way
the other recent provider drivers do it. Rerank and Balance are not part
of OpenAI's public API surface in this layer and return a clear "not
implemented" or "no such method" error.
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
### How was this tested?
- go build ./internal/entity/models/... in a clean go 1.25 image (the
go.mod minimum) returns exit 0 with no errors.
- Method set of OpenAIModel matches the ModelDriver interface:
NewInstance, Name, ChatWithMessages, ChatStreamlyWithSender, Encode,
Rerank, ListModels, Balance, CheckConnection.
- Pattern parity with the merged moonshot (#14433), volcengine (#14460),
minimax (#14478), vllm (#14532), xai (#14550), and lm-studio (#14586)
PRs.
Closes#14604
## Summary
- Add MiniMax provider GroupId query parameter support in `LiteLLMBase`
- Extract `group_id` from key configuration in `__init__`
- Append `GroupId` as query parameter to `api_base` in
`_construct_complete_args`
## Why this change is needed
MiniMax provides an OpenAI-compatible API endpoint
(`/v1/chat/completions`), but `GroupId` is a MiniMax-specific account
identifier required for billing and rate limiting - it is not part of
the OpenAI standard.
Looking at LiteLLM's `MinimaxChatConfig`:
- `get_complete_url()` only constructs the base URL (e.g.,
`https://api.minimaxi.com/v1/chat/completions`)
- LiteLLM does **not** automatically inject `GroupId` into requests
- This must be handled by the caller (ragflow's chat_model.py)
The implementation appends `GroupId` as a query parameter to `api_base`:
```python
api_base = completion_args.get("api_base", self.base_url)
separator = "&" if "?" in api_base else "?"
completion_args["api_base"] = f"{api_base}{separator}GroupId={self.group_id}"
```
This matches MiniMax's official API format (as documented by
LlamaFactory):
```bash
curl --location 'https://api.minimaxi.chat/v1/text/chatcompletion?GroupId=你的GroupId' \
--header 'Authorization: Bearer 你的API_Key'
```
## Test plan
- [ ] Verify MiniMax API calls work with GroupId query parameter
- [ ] Verify backward compatibility for other providers
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
### What problem does this PR solve?
Bump to infinity v0.7.0-dev6
(uv lock --upgrade-package infinity-sdk)
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
### What problem does this PR solve?
A and B, two API servers and a REDIS server.
If A and REDIS restart, B will hold the obsolete secret key and will
lead to error.
TODO:
app.config['SECRET_KEY'] and app.secret_key still hold obsolete secret
key.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
---------
Signed-off-by: Jin Hai <haijin.chn@gmail.com>
### What problem does this PR solve?
support non-stream runtime agent completion
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
### What problem does this PR solve?
implement `lm-studio` provider
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
- [x] Refactoring
Fixes#14562
## Problem
LLMs like DeepSeek V4 Flash and Qwen3-MAX return \\( and \\[
(double backslash) in LaTeX output. The preprocessLaTeX() function
only handled single backslash delimiters, so equations showed as raw
text.
HTML entities like < and > were also not decoded.
## Solution
Added normalization step before existing delimiter conversion:
- \\( → \( and \\[ → \[
- < → < and > → > and & → &
---------
Co-authored-by: Vivek <viveksantoshkumardubey@email.com>
### What problem does this PR solve?
add file convert backward compatibility
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
This PR addresses three related GraphRAG reliability issues that
together allow long-running GraphRAG tasks (10+ hours of LLM extraction)
to be resumed after a crash or pause without re-doing completed work. It
builds on #14096 (per-doc subgraph cache) and extends the same idea to
the resolution and community-detection phases.
Fixes#14236.
## 1. Fix concurrent merge crash
Long GraphRAG runs would crash near the end of entity resolution with:
```
RuntimeError: dictionary keys changed during iteration
```
in `Extractor._merge_graph_nodes`. Two changes:
- `rag/graphrag/general/extractor.py`: snapshot `graph.neighbors(node1)`
via `list(...)` before iterating, so concurrent `add_edge` /
`remove_node` mutations on the shared `nx.Graph` cannot invalidate the
iterator. Also tracks each redirected neighbour in `node0_neighbors` so
a later merged node sharing the same external neighbour takes the
edge-merge branch instead of overwriting via `add_edge`.
- `rag/graphrag/entity_resolution.py`: serialize the merge step with a
dedicated `asyncio.Semaphore(1)`. `nx.Graph` is not thread-safe and
concurrent merges on overlapping neighbourhoods can produce incorrect
results even with the snapshot fix.
## 2. Don't wipe partial graph on pause
Previously the pause / cancel UI path called
`settings.docStoreConn.delete({"knowledge_graph_kwd": [...]}, ...)`,
destroying every subgraph, entity, relation, and graph row.
Re-triggering then started GraphRAG from scratch even though #14096 had
already added `load_subgraph_from_store`.
After main was merged in (which deleted `api/apps/kb_app.py` per
#14394), the pause path now lives on the new REST surface `DELETE
/v1/datasets/<id>/<index_type>`:
- `api/apps/services/dataset_api_service.py`: `delete_index` accepts a
`wipe: bool = True` parameter. When `False` the doc-store rows and
GraphRAG phase markers are left intact and only the running task is
cancelled. Default preserves historical behaviour.
- `api/apps/restful_apis/dataset_api.py`: parses `?wipe=false|0|no|off`
from the query string and forwards it.
- `web/src/utils/api.ts` + `web/src/services/knowledge-service.ts`:
`unbindPipelineTask` appends `?wipe=false` when explicitly false.
- The GraphRAG pause action in
`web/src/pages/dataset/dataset/generate-button/hook.ts` passes `wipe:
false` for `KnowledgeGraph`; raptor is unchanged.
**UX impact:** the pause icon next to a running GraphRAG task no longer
wipes graph data. The only path that still wipes is the explicit Delete
action in `GenerateLogButton` (trash icon behind a confirmation modal).
## 3. Phase-completion markers (`rag/graphrag/phase_markers.py`)
A small Redis-backed marker layer at
`graphrag:phase:{kb_id}:{resolution_done|community_done}` (7-day TTL).
`run_graphrag_for_kb` consults the markers on entry and skips phases
that already completed in a prior run. Markers are cleared automatically
when:
- new docs are merged into the graph (which invalidates prior resolution
and community results),
- `delete_index` wipes the graph, or
- `delete_knowledge_graph` is called.
Redis failures never block a run -- markers are an optimization, not a
gate.
## 4. Idempotent community detection
`extract_community` previously did `delete-then-insert` on
`community_report` rows; a crash mid-insert left the dataset with no
reports. Now report IDs are derived deterministically from `(kb_id,
community.title)`, the existing report IDs are snapshotted before
insert, new rows are written, then only stale rows are pruned. A failure
at any step leaves either the prior or the new report set intact --
never a partial mix.
## 5. Tunable doc-store insert pipeline
The GraphRAG insert loop in `rag/graphrag/utils.py` and the
`community_report` insert in `rag/graphrag/general/index.py` were both
hardcoded to `es_bulk_size = 4` and ran strictly sequentially. On a real
KB this meant 1077 chunks took ~21 minutes for a 100-chunk slice -- pure
round-trip overhead.
- New `insert_chunks_bounded()` helper in `rag/graphrag/utils.py`
batches inserts via a bounded `asyncio.Semaphore`. Same retry / timeout
semantics as the prior loop.
- Defaults: 64 docs per batch, 4 batches in flight (matches the regular
ingest pipeline in `document_service.py`). Tunable per-deployment via
`GRAPHRAG_INSERT_BULK_SIZE` and `GRAPHRAG_INSERT_CONCURRENCY`.
- Both `set_graph` and `extract_community` now use the helper.
This dropped the same 1077-chunk insert from minutes to seconds in local
testing without measurable extra pressure on Infinity (total in-flight
docs ≤ `BULK_SIZE × CONCURRENCY` = 256 by default).
## Tests
- `test/unit_test/rag/graphrag/test_merge_graph_nodes.py` (3 tests):
dense neighbourhood merge, neighbour-snapshot regression, concurrent
serialized merges.
- `test/unit_test/rag/graphrag/test_phase_markers.py` (4 tests): set/has
round-trip, kb-scoped clear, no-op on empty input, graceful Redis
failure.
-
`test/testcases/test_web_api/test_dataset_management/test_dataset_sdk_routes_unit.py`:
new `test_delete_index_wipe_flag_unit` covers `wipe=false` for both
GraphRAG and raptor on the new REST route, and confirms the default
still wipes and clears phase markers.
## Compatibility
- Backward compatible: tasks queued before this change behave
identically (default `wipe=true`, no markers expected).
- No schema/migration changes; all new state lives in Redis.
- New optional REST query param `wipe` on `DELETE
/v1/datasets/<id>/<index_type>`.
- New optional env vars `GRAPHRAG_INSERT_BULK_SIZE` and
`GRAPHRAG_INSERT_CONCURRENCY`; defaults preserve safe behaviour.
## Example of resume
Screenshot below shows a test resuming knowledge graph generation after
applying the concurrency fix and re-deploying.
<img width="521" height="677" alt="image"
src="https://github.com/user-attachments/assets/9ef0d405-cbb3-420d-a1a1-e51f3e7e9b7a"
/>
### Type of change
- [X] Bug Fix (non-breaking change which fixes an issue)
- [ ] New Feature (non-breaking change which adds functionality)
- [ ] Documentation Update
- [ ] Refactoring
- [ ] Performance Improvement
- [ ] Other (please describe):
### What problem does this PR solve?
This PR fixes a bug where `layout_recognize="<name>@OpenDataLoader"` was
misrouted and then failed during parsing in the naive parser path. It
now routes correctly to OpenDataLoader and avoids passing unsupported
arguments that caused runtime errors. fixes#14572
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
## Summary
This fixes a missing authorization check in the beta API document
download endpoint:
- **CWE:** CWE-862 (Missing Authorization)
- **Severity:** Medium
- **Affected route/file:** `GET /api/v1/documents/<document_id>` in
`api/apps/sdk/doc.py`
- **Data flow:** the route reads a bearer beta API token, resolves the
token with `APIToken.query(beta=token)`, accepts `document_id` directly
from the URL, loads the document with
`DocumentService.query(id=document_id)`, and then fetches the backing
object through `File2DocumentService.get_storage_address()` /
`settings.STORAGE_IMPL.get()`.
Before this change, that flow verified that the API token was valid, but
it did not verify that the token's tenant owned the document's knowledge
base. A caller with any valid beta API token and a known document ID
could therefore reach storage for a document belonging to another
tenant.
## Fix
The endpoint now takes the tenant ID from the resolved API token and
checks the document's knowledge base with:
```python
KnowledgebaseService.query(id=doc[0].kb_id, tenant_id=tenant_id)
```
If the knowledge base is not owned by the token tenant, the request
returns an access error before any storage lookup occurs. This mirrors
the tenant-scoped ownership checks used by the dataset-scoped document
download path and keeps the patch small.
## Tests
Added unit coverage for `download_doc()` to assert that:
- the beta token tenant ID is used in the knowledge-base ownership
lookup;
- cross-tenant access returns `You do not have access to this
document.`;
- storage resolution is not called before tenant authorization succeeds;
- the existing same-tenant empty-file and successful-download paths
still run after the authorization gate passes.
I also verified the final patch is limited to `api/apps/sdk/doc.py` and
the related document SDK route unit test. A local `pytest` invocation
could not complete in this checkout because the shared test fixture
attempts to log in to a RAGFlow server at `127.0.0.1:9380`, which was
not running in the local environment.
## Security analysis
This is exploitable when an attacker has a valid beta API token for
their own tenant and obtains or guesses a document ID from another
tenant. The token alone should not grant access to other tenants' files,
but the direct document route previously authorized only the token
itself and not the requested resource. The new tenant-scoped
knowledge-base check binds the requested document back to the token
tenant before storage is accessed, preventing cross-tenant document
downloads through this endpoint.
Before submitting, we attempted to disprove this by checking whether
existing dataset-scoped routes, token validation, or framework
protections already enforced ownership. They do not apply to this direct
document-ID route: it bypassed the dataset path parameter and used only
`DocumentService.query(id=document_id)` before reading storage.
cc @lewiswigmore
### What problem does this PR solve?
### Type of change
- [v] Bug Fix (non-breaking change which fixes an issue)
Co-authored-by: wiratama <dafa.wiratama@bankraya.co.id>
### What problem does this PR solve?
Fix#14340
## Problem Description
When using an **Agentic Agent** (not Workflow) with one or more
Retrieval tools (e.g., Dataset Retrieval + Memory Retrieval), the agent
silently returns an empty response (`agent_response: ""`) after hanging
for several minutes. The server logs show:
```
AttributeError: 'ChatCompletionMessageToolCall' object has no attribute 'index'
```
This error propagates as a `GENERIC_ERROR`, causing the canvas to return
an empty response. The subsequent Memory save task then receives the
empty `agent_response` and logs:
```
Document for referred_document_id XXXX not found
```
## Reproduction Steps
1. Set `DOC_ENGINE=infinity` (or `elasticsearch` — the engine itself is
not the root cause).
2. Create a blank **Agentic Agent** (not a Workflow).
3. Add **two Retrieval tools** to the Agent node:
- `Retrieval_DS` → Dataset (Knowledge Base)
- `Retrieval_Mem` → Memory component
4. Add a **Message** node with **Save to Memory** enabled.
5. Launch the agent and send any message (e.g., "hola").
6. The agent hangs and returns an empty response.
## Root Cause Analysis
The crash occurs in `_append_history` and `_append_history_batch` inside
`rag/llm/chat_model.py`. These methods directly access `.index` on tool
call objects:
```python
# _append_history_batch
{
"index": tc.index, # <-- crashes here
...
}
```
However, **non-streaming** LLM responses (`stream=False`) return
`ChatCompletionMessageToolCall` objects, which **do not have an `index`
field** according to the OpenAI API specification. The `index` field
only exists on `ChoiceDeltaToolCall` objects returned in **streaming**
responses (`stream=True`).
When the agentic agent triggers an internal `full_question` call (used
to compress multi-turn conversation history), the request is incorrectly
routed through `async_chat_with_tools` because `is_tools=True` is set at
the `LLMBundle` level. If the LLM decides to emit `tool_calls` during
this auxiliary request, the code enters the non-streaming tool loop and
crashes when trying to append history.
## Fix
Replaced all direct `.index` accesses with `getattr(..., "index", None)`
for safe, backward-compatible access:
| Method | File | Line | Change |
|--------|------|------|--------|
| `_append_history` | `rag/llm/chat_model.py` | ~L304 |
`tool_call.index` → `getattr(tool_call, "index", None)` |
| `_append_history_batch` | `rag/llm/chat_model.py` | ~L332 | `tc.index`
→ `getattr(tc, "index", None)` |
| `_append_history` | `rag/llm/chat_model.py` | ~L1467 |
`tool_call.index` → `getattr(tool_call, "index", None)` |
| `_append_history_batch` | `rag/llm/chat_model.py` | ~L1496 |
`tc.index` → `getattr(tc, "index", None)` |
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
Signed-off-by: noob <yixiao121314@outlook.com>
## Summary
- normalize string items for list-valued metadata filters in
`meta_filter`
- fix `in` / `not in` case asymmetry when document metadata is
lowercased but filter list values are not
- add regression tests that cover the original issue scenario using
uppercase list values
## Validation
- `PYTHONPATH=external/ragflow pytest
external/ragflow/test/unit_test/common/test_metadata_filter_operators.py
-q`
## Notes
- I commented on #14389 before opening this PR to claim the issue.
- The new tests use `value=["F2", "F11"]` so they fail on the old
implementation and pass with this fix.
- This also benefits other non-comparison operators that flow through
the same normalization path.
Co-authored-by: copizza <copizza@users.noreply.github.com>
Co-authored-by: Wang Qi <wangq8@outlook.com>
### What problem does this PR solve?
add legacy agent completion API compatibility
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
### What problem does this PR solve?
This PR fixes missing authorization checks in the Memory API.
Previously, several authenticated endpoints accepted caller-supplied
`tenant_id`, `owner_ids`, or `memory_id` values and used them directly
to list, read, update, delete, or search Memory data.
That could allow an authenticated user to access or mutate another
tenant's Memory records if they knew a tenant ID or memory ID. The fix
centralizes Memory access checks and applies them consistently across
Memory and Memory-message operations.
The change:
- Adds helper logic to parse list filters and compute tenant IDs
accessible to `current_user`.
- Requires direct `memory_id` operations to pass Memory access checks
before reading, updating, deleting, or changing message state.
- Filters list/search/recent-message requests to accessible memories
only.
- Applies Memory visibility filtering before count and pagination in
`MemoryService.get_by_filter`.
- Accepts `owner_ids` in the Memory list route, matching the frontend
owner filter while still intersecting values with the caller's
accessible tenants.
-
### Related issues
Closes#14534
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
Co-authored-by: jony376 <jony376@gmail.com>
### What problem does this PR solve?
add IMAP deleted document sync
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
### What problem does this PR solve?
Incremental DingTalk AI Table (Notable) sync did not reconcile rows
removed on the remote side with documents already in the knowledge base.
This follows the coordinated datasource work in #14362 (“sync deleted
files”).
This PR adds a **full slim snapshot**
(`retrieve_all_slim_docs_perm_sync`) that lists **current record IDs for
all sheets** without building document blobs, using the same logical
document IDs as full ingest
(`dingtalk_ai_table:{table_id}:{sheet_id}:{record_id}`). When
**`sync_deleted_files`** is enabled on incremental runs,
`DingTalkAITable._generate` returns **`(document_generator,
file_list)`** so **`SyncBase`** can run
**`cleanup_stale_documents_for_task`** and remove KB rows that no longer
exist remotely.
Design notes:
- **`_document_id`** centralizes the ID string so slim snapshots and
**`_convert_record_to_document`** stay aligned with
**`hash128(doc.id)`** semantics used during ingestion/cleanup.
- **`end_ts`** is captured before building **`file_list`**, then
**`poll_source`** uses the same upper bound (consistent with other
Dropbox-style connectors).
- **`batch_size`** from connector config is coerced to a positive
**`int`** before constructing the connector.
- Slim snapshot failures are caught in **`_generate`**; **`file_list`**
is set to **`None`** so cleanup is skipped rather than running on
partial/error state.
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
### Files changed (summary)
| Area | Change |
|------|--------|
| `common/data_source/dingtalk_ai_table_connector.py` |
`SlimConnectorWithPermSync`, `retrieve_all_slim_docs_perm_sync`,
`_document_id` shared with document conversion |
| `rag/svr/sync_data_source.py` | `DingTalkAITable._generate`: slim
snapshot + tuple return; `batch_size` validation; shared `end_ts` with
`poll_source` |
| `web/src/pages/user-setting/data-source/constant/index.tsx` |
`syncDeletedFiles` for DingTalk AI Table in
`DataSourceFeatureVisibilityMap` |
Closes / relates to: #14362
### What problem does this PR solve?
This fixes a MinerU parsing failure where output JSON was not found in
nested v0.24.0 layouts, and also fixes a `content_names` NameError in
`_read_output()`. As a result, successful MinerU API runs no longer end
with false “MinerU not found” parsing failures.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
Concurrent CREATE TABLE / CREATE INDEX / DROP TABLE on the same Infinity
instance can race on the catalog counter (e.g. db|1|next_table_id) and
fail with error 9003 "Resource busy" instead of waiting on a lock. Two
users creating a knowledge base at the same instant, or any deployment
with multiple backend workers behind one Infinity, can hit it.
Wrap the metadata paths in create_idx, create_doc_meta_idx, and
delete_idx with exponential backoff + jitter (5 attempts, 50ms base).
The wrapped operations already use ConflictType.Ignore, so retrying is
idempotent — worst case the second attempt is a no-op against an
already-created table. Tunable via INFINITY_META_RETRY_MAX /
INFINITY_META_RETRY_BASE_DELAY_MS.
Repro: stress 30 concurrent POST /api/v1/datasets against a 4-worker
backend → ~50% of requests fail without the patch (Resource busy from
the second worker that hits the counter), 100% succeed with it. At 100
concurrent requests, all 100 succeed in ~1.2s; the retry budget never
exhausted in our tests.
Scope is limited to metadata paths only — data-path operations (INSERT
chunks, SELECT for retrieval) go through per-table code paths and don't
share the contended counter.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
---------
Co-authored-by: yoan sapienza <Yoan Sapienza yoan.sapienza@orange.fr Yoan Sapienza zappy@macbookpro.home>
Closes#14552
### What problem does this PR solve?
Add a Go driver for xAI (Grok models).
The config file conf/models/xai.json has been in the repo since the
early Go provider work, but internal/entity/models/factory.go had no
case for "xai". So any xAI request fell through to the dummy driver
and never reached the API. This PR adds the missing driver and wires it
up.
### What this PR includes
- New file internal/entity/models/xai.go with an XAIModel that
implements the ModelDriver interface.
- factory.go: route the "xai" provider name to NewXAIModel.
### How the driver works
- xAI exposes an OpenAI-compatible API at https://api.x.ai/v1.
- ChatWithMessages and ChatStreamlyWithSender post to /chat/completions
in the same shape the moonshot and deepseek drivers use.
- ListModels and CheckConnection call /models to confirm the API key
works and to list available model ids.
- reasoning_content is passed through for grok-3-mini and other xAI
reasoning models, both in the non-stream and stream paths.
- Encode, Rerank, and Balance are not part of the public xAI API at the
moment, so they return a clear "not implemented" or "no such method"
error.
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
### How was this tested?
- go build ./internal/entity/models/... in a clean go 1.25 image (the
go.mod minimum) returns exit 0 with no errors.
- Method set of XAIModel matches the ModelDriver interface: NewInstance,
Name, ChatWithMessages, ChatStreamlyWithSender, Encode, Rerank,
ListModels, Balance, CheckConnection.
- Pattern parity with the merged moonshot (#14433), volcengine (#14460),
minimax (#14478), and vllm (#14532) PRs.
---------
Co-authored-by: Jin Hai <haijin.chn@gmail.com>
### What problem does this PR solve?
implement `Ollama` provider
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
- [x] Refactoring
---------
Co-authored-by: Jin Hai <haijin.chn@gmail.com>
### What problem does this PR solve?
Use GetChatModel, remove duplicate functions in model_service.go
### Type of change
- [x] Refactoring
Co-authored-by: Jin Hai <haijin.chn@gmail.com>
### What problem does this PR solve?
```
RAGFlow(user)> chat with 'glm-4.6v-flash@test@zhipu-ai' message 'What are the pics talk about?' image 'https://cdn.bigmodel.cn/static/logo/register.png' 'https://cdn.bigmodel.cn/static/logo/api-key.png'
Answer: The first picture shows a login/register modal with options for phone number login, account login, and WeChat QR code login, along with a prompt for new users to get a 20 million tokens experience package. The second picture displays the API keys management page of a platform, including a warning about API key security and a table listing existing API keys with details like creation time and usage history.
Time: 31.600545
RAGFlow(user)> chat with 'glm-4.6v-flash@test@zhipu-ai' message 'What are the video talk about?' video 'https://cdn.bigmodel.cn/agent-demos/lark/113123.mov'
Answer: Based on the sequence of frames provided, the video is a demonstration of a web search and navigation process.
1. The video starts with a blank Google search page.
2. The user types "智谱" (which is the Chinese name for the company Zhipu AI) into the search box.
3. The search is initiated and the page shows "About 0 results".
4. The search results load, showing information about Zhipu AI, including its website.
5. The user clicks on the main website link (www.zhipuai.cn).
6. The video ends by showing the homepage of Zhipu AI's website, titled "Z.ai GLM Large Model Open Platform".
In summary, the video is about searching for the company "智谱" (Zhipu AI) on Google and then navigating to its official website.
Time: 76.582520
```
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
Signed-off-by: Jin Hai <haijin.chn@gmail.com>
### What problem does this PR solve?
Currently, RAGFlow's Search and Chat interfaces display only raw
vectorized text chunks during retrieval, without contextual information
about their source documents. Users cannot see document titles, page
numbers, upload dates, or custom metadata fields that would help them
understand and trust the retrieved results.
This PR introduces an **optional metadata display feature** that
enriches retrieved chunks with document-level metadata in both the
Search tab and Chatbot interface.
**Key improvements:**
- **Search results**: Display document metadata as styled badges beneath
chunk snippets
- **Chat citations**: Show metadata in citation popovers and reference
lists for better source context
- **LLM context**: Metadata is injected into the LLM prompt to enable
more accurate, citation-aware responses
- **External API support**: Applications using RAGFlow's SDK retrieval
endpoints (`/v1/retrieval`, `/v1/searchbots/retrieval_test`) can opt-in
via request parameters
- **User control**: Multi-select dropdown UI allows users to choose
which metadata fields to display
**Implementation approach:**
- ✅ Reuses existing `DocMetadataService` infrastructure (no new database
tables or indices)
- ✅ Settings stored in existing JSON configuration fields
(`search_config.reference_metadata`, `prompt_config.reference_metadata`)
- ✅ No database migrations required
- ✅ Disabled by default (fully opt-in and backward-compatible)
- ✅ Dynamic metadata field selection populated from actual document
metadata keys
- ✅ Fixed critical bug where Python's builtin `set()` was shadowed by a
route handler function
**Modified endpoints (all backward-compatible):**
- `POST /v1/retrieval` (Public SDK)
- `POST /v1/searchbots/retrieval_test` (Searchbots)
- `POST /v1/chunk/retrieval_test` (UI/Internal)
- Chat completions endpoints (via `extra_body.reference_metadata` or
`prompt_config`)
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
###Images
-
<img width="879" height="1275" alt="image"
src="https://github.com/user-attachments/assets/95b2d731-31ae-45a1-b081-bf5893f52aeb"
/>
<br><br>
<br><br>
<img width="1532" height="362" alt="image"
src="https://github.com/user-attachments/assets/9cebc65b-b7a7-459f-b25e-3b13fa9b638e"
/>
<br><br>
<br><br>
<img width="2586" height="1320" alt="image"
src="https://github.com/user-attachments/assets/2153d493-d899-461f-a7a9-041391e07776"
/>
---------
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: Attili-sys <Attili-sys@users.noreply.github.com>
Co-authored-by: Ahmad Intisar <ahmadintisar@Ahmads-MacBook-M4-Pro.local>
### What problem does this PR solve?
remove delete_documents uuid validation
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
### What problem does this PR solve?
Partially addresses #14362.
This PR enables syncing deleted files for RSS data sources.
Previously, RSS incremental sync only returned feed entries whose
timestamps were inside the poll window. If an entry was removed from the
RSS feed, RAGFlow had no full current RSS snapshot to pass into the
shared stale-document cleanup path, so the deleted remote entry could
remain in the knowledge base.
This PR:
- adds `retrieve_all_slim_docs_perm_sync()` to `RSSConnector`
- reuses the same `rss:<md5(stable_key)>` document ID derivation used by
normal RSS ingest
- returns `(document_generator, file_list)` for incremental RSS sync
when `sync_deleted_files` is enabled
- captures the poll end timestamp before snapshot/poll so cleanup does
not race against the same sync window
- adds start/end logs around RSS slim snapshot collection
- exposes the deleted-file sync toggle for RSS in the data source UI
Per maintainer request on related datasource PRs, this PR contains no
test-case changes. Local verification was run with an external script.
Validation:
- `uv run ruff check common/data_source/rss_connector.py
rag/svr/sync_data_source.py`
- `uv run pytest test/unit_test/rag/test_sync_data_source.py -q`
- `./node_modules/.bin/eslint
src/pages/user-setting/data-source/constant/index.tsx`
- `git diff --check`
- `uv run python /tmp/verify_rss_deleted_sync.py --repo
/root/74/ragflow`
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
## What problem does this PR solve?
Incremental WebDAV sync only ingested files whose modification time fell
inside the poll window; documents removed on the WebDAV server were
never removed from the knowledge base. This aligns with
[#14362](https://github.com/infiniflow/ragflow/issues/14362)
(coordinated datasource “sync deleted files” work).
This PR adds a **full-tree slim snapshot**
(`retrieve_all_slim_docs_perm_sync`) that enumerates current remote
paths **without downloading file contents**, using the same logical
document IDs as full ingest (`webdav:{base_url}:{file_path}`). When
**`sync_deleted_files`** is enabled on incremental runs, sync returns
**`(document_generator, file_list)`** so **`SyncBase`** runs
**`cleanup_stale_documents_for_task`** and removes KB rows no longer
present remotely.
Design notes:
- **`_list_files_recursive`** gains **`filter_by_mtime`**: snapshot
passes **`filter_by_mtime=False`** (full tree under **`remote_path`**);
**`poll_source`** keeps mtime-window filtering as before.
- Slim snapshot applies the same **extension** and **`size_threshold`**
rules as **`_yield_webdav_documents`** so retain IDs match what would be
indexed.
- **`end_ts`** is captured before building **`file_list`**, then
**`poll_source`** uses the same upper bound (consistent with
Dropbox-style connectors).
## Type of change
- [x] New Feature (non-breaking change which adds functionality)
## Files changed
| Area | Change |
|------|--------|
| `common/data_source/webdav_connector.py` |
`SlimConnectorWithPermSync`, `retrieve_all_slim_docs_perm_sync`,
`filter_by_mtime` on `_list_files_recursive` |
| `rag/svr/sync_data_source.py` | WebDAV `_generate`: `file_list` +
tuple return; pass **`batch_size`** from connector config |
| `web/src/pages/user-setting/data-source/constant/index.tsx` |
`syncDeletedFiles` for WebDAV in `DataSourceFeatureVisibilityMap` |
### What problem does this PR solve?
Implement the vLLM model provider for RAGFlow to fully support local and
self-hosted open-source models (e.g., Qwen, GLM, Llama) via the vLLM
framework, and fix several critical bugs related to model instance
management and API requests.
**Key changes and fixes:**
1. **Added Standard vLLM Provider (`vllm.go`, `vllm.json`):**
- Implemented `VllmModel` driver strictly adhering to the OpenAI API
specification.
- Removed hardcoded and dangerous routing logic (e.g., forcing
`AsyncChat` for Qwen/GLM prefixes), ensuring standard
`/v1/chat/completions` compatibility.
- Refactored `ListModels` to use safe JSON parsing (resolving nil
pointer panics) and standard `GET` requests without bodies.
- Added `APIConfig.Region` fallback logic to prevent empty `base_url`
fetching when checking models.
2. **Fixed `ChatToModelStreamWithSender` Bug (`model_service.go`):**
- Resolved the `model is disabled` error when streaming chat with local
database-saved models.
- Added the missing `if modelInfo.Status == "active"` block to correctly
invoke `NewInstance` and inject the dynamic `base_url` into the provider
driver before starting the SSE stream.
3. **Fixed `ListSupportedModels` Bug (`model_service.go`):**
- Added dynamic `NewInstance` injection for `base_url`. Previously, the
list models function used the static JSON config without injecting
user-configured dynamic URLs from the database, resulting in an
`unsupported protocol scheme ""` error.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
- [x] New Feature (non-breaking change which adds functionality)
### What problem does this PR solve?
Fix: LaTeX formulas cannot be displayed on the chat page.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
### What problem does this PR solve?
id as "text", not a "keyword", order by it will cause error.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
### What problem does this PR solve?
Fix docker image version info in comment
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
Signed-off-by: Jin Hai <haijin.chn@gmail.com>
### What problem does this PR solve?
Refs #14362.
This PR enables syncing deleted files for Zendesk data sources.
Previously, Zendesk incremental sync never returned a slim remote
snapshot to the shared stale-document cleanup path, so deleted remote
Zendesk records could remain in RAGFlow. The existing Zendesk slim
snapshot also included records that ingestion intentionally skips, such
as draft articles, articles without bodies, skipped-label articles,
empty-body articles, and tickets with `status == "deleted"`.
This PR:
- exposes the deleted-file sync option for Zendesk in the data source UI
- returns Zendesk slim snapshots during incremental sync when
`sync_deleted_files` is enabled
- reuses Zendesk indexability rules so cleanup compares against the same
records ingestion can materialize
- adds start/end logs around Zendesk slim snapshot collection for
operational visibility
Per maintainer request, this PR contains no test-case changes. Manual
verification recording will be provided separately.
Validation:
- `uv run ruff check common/data_source/zendesk_connector.py
rag/svr/sync_data_source.py`
- `uv run pytest test/unit_test/rag/test_sync_data_source.py -q`
- `./node_modules/.bin/eslint
src/pages/user-setting/data-source/constant/index.tsx`
### Type of change
- [ ] Bug Fix (non-breaking change which fixes an issue)
- [x] New Feature (non-breaking change which adds functionality)
- [ ] Documentation Update
- [ ] Refactoring
- [ ] Performance Improvement
- [ ] Other (please describe):
### What problem does this PR solve?
Partially addresses #14362.
Adds deleted-file sync support for the Asana data source. Asana already
indexes task attachments as documents, but it did not provide the slim
document snapshot required by stale-document reconciliation, and the
sync wrapper never returned a `file_list` for cleanup.
This PR:
- adds `retrieve_all_slim_docs_perm_sync()` to `AsanaConnector`
- builds slim IDs with the same `asana:{task_id}:{attachment_gid}`
format used by indexed documents
- avoids downloading attachment blobs during the snapshot
- aborts the snapshot if Asana API errors occur, preventing partial
snapshots from deleting valid local docs
- captures the incremental poll end time before snapshotting and makes
`poll_source()` respect that boundary
- exposes the deleted-file sync toggle for Asana in the data source UI
Per maintainer request, this PR contains no test-case changes. Manual
verification recording will be provided separately.
Validation:
- `uv run ruff check common/data_source/asana_connector.py
rag/svr/sync_data_source.py`
- `uv run pytest test/unit_test/rag/test_sync_data_source.py -q`
- `./node_modules/.bin/eslint
src/pages/user-setting/data-source/constant/index.tsx`
- `git diff --check`
### Type of change
- [x] New Feature
## Summary
Fix critical severity security issue in `rag/utils/ob_conn.py`.
## Vulnerability
| Field | Value |
|-------|-------|
| **ID** | V-003 |
| **Severity** | CRITICAL |
| **Scanner** | multi_agent_ai |
| **Rule** | `V-003` |
| **File** | `rag/utils/ob_conn.py:691` |
**Description**: The OceanBase database connector constructs SQL WHERE
clauses by directly embedding user-controlled filter expressions using
Python f-strings at lines 726, 777, 781, 787, 793, 821, and 827. No
parameterization or allowlist validation is applied before the
expressions are incorporated into live SQL queries. This is the most
critical vulnerability in the codebase because it directly exposes the
RAG knowledge base — the platform's core business asset — to complete
compromise.
## Changes
- `rag/utils/ob_conn.py`
## Verification
- [x] Build passes
- [x] Scanner re-scan confirms fix
- [x] LLM code review passed
---
*Automated security fix by [OrbisAI Security](https://orbisappsec.com)*
### What problem does this PR solve?
Feat: add button for remove header & footer in pipeline
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
### What problem does this PR solve?
Incremental Seafile sync only ingests files whose modification time
falls in the poll window; documents removed in Seafile were never
removed from the knowledge base. This contributes to
[#14362](https://github.com/infiniflow/ragflow/issues/14362) (datasource
“sync deleted files” coordination).
This PR adds a **slim snapshot** (`retrieve_all_slim_docs_perm_sync`)
that enumerates current remote file IDs **without downloading content**,
using the same logical IDs as full ingest
(`seafile:{repo_id}:{file_id}`). When **`sync_deleted_files`** is
enabled on incremental runs, **`SeaFile._generate`** returns
**`(document_generator, file_list)`** so **`SyncBase`** can run
**`cleanup_stale_documents_for_task`** and remove stale KB documents.
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
### What changed
- **`common/data_source/seafile_connector.py`**: `SeaFileConnector`
implements **`SlimConnectorWithPermSync`**;
**`_list_files_recursive(..., filter_by_mtime=...)`** supports full-tree
listing for snapshots; **`retrieve_all_slim_docs_perm_sync()`** reuses
the same library/root scan as ingest and applies the same **size**
ceiling; logging for snapshot start/end and counts.
- **`rag/svr/sync_data_source.py`**: **`SeaFile._generate`** validates
**`batch_size`**, captures **`end_ts`** before snapshot +
**`poll_source`**, wraps slim retrieval in **`try`/`except`** (
**`file_list = None`** on failure so ingest continues), returns
**`(generator, file_list)`**.
- **`web/src/pages/user-setting/data-source/constant/index.tsx`**:
**`syncDeletedFiles`** for Seafile in
**`DataSourceFeatureVisibilityMap`**.
### What problem does this PR solve?
This fixes a crash in Manual and Naive parsing when PDF outlines include
page numbers as a third tuple value. It makes outline unpacking accept
extra values so parsing no longer fails. fixes#14411
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
### What problem does this PR solve?
Both tokenizer (`rag/flow/tokenizer/tokenizer.py`) and
`BuiltinEmbed.encode`
(`rag/llm/embedding_model.py`) currently accumulate embedding batches
via
`np.concatenate` inside the per-batch loop. `np.concatenate` allocates a
new
array and copies all existing data on every call, so accumulating N
batches
is O(N²) in both time and peak memory.
Replacing the incremental concatenate with a list-of-batches + a single
`np.vstack` at the end gives O(N) total work.
For tokenizer the title-vector broadcast `np.concatenate([vts[0]] * N)`
is
also replaced by `np.tile`, which does the same job with a single
contiguous
allocation instead of building a Python list of references.
This is purely a CPU/memory optimisation — output shape and dtype are
unchanged. Measured impact grows with document size:
- 1k chunks (batch 512, 2 iters): ~negligible
- 10k chunks (20 iters): ~10× speedup on this stage
- 100k chunks (195 iters): ~100× speedup, and peak RAM
drops from O(N) extra to near-zero
### Type of change
- [x] Performance Improvement
Co-authored-by: yoan sapienza <Yoan Sapienza yoan.sapienza@orange.fr Yoan Sapienza zappy@macbookpro.home>
### What problem does this PR solve?
- Update version tags in README files (including translations) from
v0.25.0 to v0.25.1
- Modify Docker image references and documentation to reflect new
version
- Update version badges and image descriptions
- Maintain consistency across all language variants of README files
### Type of change
- [x] Documentation Update
### What problem does this PR solve?
Fix: The GraphRAG icon is not displaying.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
### What problem does this PR solve?
## Summary
Fixed a bug where the **File Logs** tab in the dataset ingestion page
always showed "No logs" even after files were parsed successfully.
## Root Cause
Both the **File Logs** and **Dataset Logs** tabs on the frontend called
the same backend endpoint `/datasets/{dataset_id}/ingestions`. However,
the backend only queried `get_dataset_logs_by_kb_id`, which
hard-filtered records by `document_id == GRAPH_RAPTOR_FAKE_DOC_ID`
(dataset-level logs). As a result, real file-level logs were never
returned, causing the table to appear empty.
## Changes
### Backend
- **`api/apps/restful_apis/dataset_api.py`**
- Added two new query parameters to `list_ingestion_logs`:
- `log_type` — `"file"` or `"dataset"` (default: `"dataset"`)
- `keywords` — search keyword for filtering by document / task name
- **`api/apps/services/dataset_api_service.py`**
- Updated `list_ingestion_logs` signature to accept `log_type` and
`keywords`.
- Added conditional routing:
- When `log_type == "file"`, call
`PipelineOperationLogService.get_file_logs_by_kb_id`
- Otherwise, call
`PipelineOperationLogService.get_dataset_logs_by_kb_id`
- **`api/db/services/pipeline_operation_log_service.py`**
- Extended `get_dataset_logs_by_kb_id` with an optional `keywords`
parameter so dataset logs can also be searched.
### Frontend
- **`web/src/pages/dataset/dataset-overview/hook.ts`**
- Removed the separate API function switching (`listPipelineDatasetLogs`
vs `listDataPipelineLogDocument`).
- Unified both tabs to call `listDataPipelineLogDocument` with the new
`log_type` query parameter (`"file"` or `"dataset"`).
- Ensured `keywords` and filter values are passed through correctly.
## Behavior After Fix
| Tab | `log_type` | Returned Records | Searchable Field |
|---|---|---|---|
| File Logs | `file` | Real document-level logs | `document_name` (file
name) |
| Dataset Logs | `dataset` | GraphRAG / RAPTOR / MindMap logs |
`document_name` (task type) |
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
---------
Signed-off-by: noob <yixiao121314@outlook.com>
Co-authored-by: Wang Qi <wangq8@outlook.com>
Co-authored-by: Yingfeng Zhang <yingfeng.zhang@gmail.com>
### What problem does this PR solve?
Fix: The pipeline column header in the FileLogsTable is displaying
incorrectly.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
### What problem does this PR solve?
1. drop instance model
2. Fix issue of drop instance but not drop models.
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
Signed-off-by: Jin Hai <haijin.chn@gmail.com>
### What problem does this PR solve?
implement MiniMax provider
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
- [x] New Feature (non-breaking change which adds functionality)
### What problem does this PR solve?
Feat: enable sync deleted file for Discord
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
### What problem does this PR solve?
Partially addresses #14362 by adding deleted-file sync support for the
Dropbox data source.
Dropbox previously did not provide the slim current-file snapshot
required by stale document reconciliation, and its sync runner returned
only document batches. As a result, enabling deleted-file sync could not
remove local documents that had been deleted from Dropbox.
This PR:
- Adds `retrieve_all_slim_docs_perm_sync()` to `DropboxConnector`.
- Reuses Dropbox metadata traversal to collect current remote file IDs
without downloading file contents.
- Wires incremental Dropbox sync to return `(document_generator,
file_list)` when `sync_deleted_files` is enabled.
- Enables the deleted-file sync toggle for Dropbox in the data source
settings UI.
- Adds regression coverage for slim snapshots, nested folders, paginated
listings, duplicate filenames, and full reindex behavior.
Tests:
- `uv run pytest test/unit_test/common/test_dropbox_connector.py -q`
- `uv run pytest test/unit_test/rag/test_sync_data_source.py -q`
- `uv run pytest test/unit_test/common/test_dropbox_connector.py
test/unit_test/rag/test_sync_data_source.py -q`
- `uv run ruff check common/data_source/dropbox_connector.py
rag/svr/sync_data_source.py
test/unit_test/common/test_dropbox_connector.py
test/unit_test/rag/test_sync_data_source.py`
- `./node_modules/.bin/eslint
src/pages/user-setting/data-source/constant/index.tsx`
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
### What problem does this PR solve?
Feat: enable sync deleted files in gitlab
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
## Problem
In the Dataset Configuration page, changing the RAPTOR **Generation
scope** from "Single file" to "Dataset" and clicking **Save** did not
persist the change. After refreshing or re-entering the page, the scope
always reverted to "Single file".
## Root Cause
1. **Backend**: The `RaptorConfig` Pydantic model in
`api/utils/validation_utils.py` was configured with `extra="forbid"` but
did not declare a `scope` field. When the frontend sent `"scope":
"dataset"`, Pydantic rejected the request.
2. **Frontend**: The `extractRaptorConfigExt` utility in
`web/src/hooks/parser-config-utils.ts` treated `scope` as an unknown
field and moved it into the nested `ext` object. Consequently, the
backend could not read `raptor_config.get("scope", "file")` correctly,
so the default `"file"` was always used.
## Changes
- Added `scope: Literal["file", "dataset"]` to the backend
`RaptorConfig` model with a default of `"file"`.
- Added `scope` to the known-field whitelist in the frontend
`extractRaptorConfigExt` helper so it is transmitted as a top-level
raptor field instead of being buried in `ext`.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
---------
Signed-off-by: noob <yixiao121314@outlook.com>
### What problem does this PR solve?
1. support command:
```
RAGFlow(user)> create provider 'vllm' instance 'test' key 'test-key' url 'base-url' region 'abc';
SUCCESS
RAGFlow(user)> list instances from 'vllm';
+----------+----------------------------------------+----------------------------------+--------------+----------------------------------+--------+
| apiKey | extra | id | instanceName | providerID | status |
+----------+----------------------------------------+----------------------------------+--------------+----------------------------------+--------+
| test-key | {"base_url":"base-url","region":"abc"} | 40213c89430311f1a7cf38a74640adcc | test | b4d40e6142d311f1a4f938a74640adcc | enable |
+----------+----------------------------------------+----------------------------------+--------------+----------------------------------+--------+
```
2. support add vllm model
```
RAGFlow(user)> add model 'Qwen/Qwen2-0.5B' to provider 'vllm' instance 'test' with tokens 131072 chat;
SUCCESS
```
3. add vllm chat
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
- [x] Refactoring
---------
Signed-off-by: Jin Hai <haijin.chn@gmail.com>
### What problem does this PR solve?
Port PR14454 to GO (PruneDeletedChunks)
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
### What problem does this PR solve?
Feat: enable sync deleted files for Gmail && fix google drive issues
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
---------
Co-authored-by: bill <yibie_jingnian@163.com>
Co-authored-by: balibabu <assassin_cike@163.com>
### What problem does this PR solve?
Fix: Clicking the button in the bottom-right corner of the
`/chats/widget` page fails to display the dialog box.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
### What problem does this PR solve?
implement `volcengine` provider
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
### What problem does this PR solve?
Fix: Dataset: When configuring the "general chunk method," options such
as chunk size and parent-child slicing are unavailable.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
---------
Co-authored-by: balibabu <assassin_cike@163.com>
### What problem does this PR solve?
prune deleted doc chunks from retrieval
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
### What problem does this PR solve?
Update the URL to: /api/v1/chat/completions
### Type of change
- [x] Refactoring
Signed-off-by: Jin Hai <haijin.chn@gmail.com>
### What problem does this PR solve?
Feat: sync deleted files in Bitbucket
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
**Addresses the Google Drive integration for #14362**
This PR completely overhauls the Google Drive sync logic to accurately
detect remote deletions, while drastically reducing the memory footprint
during the snapshot phase.
### What changed under the hood:
* **Killed the memory bloat:** Swapped out the massive document
dictionary objects for a lightweight `collections.namedtuple` (`SlimDoc
= namedtuple('SlimDoc', ['id'])`). This prevents RAM spikes during
`retrieve_all_slim_docs_perm_sync` on massive enterprise drives.
* **Flawless downstream integration:** The `SlimDoc` object relies on
simple duck typing. It perfectly delivers the `.id` attribute required
by `ConnectorService.cleanup_stale_documents_for_task`, meaning your
core `hash128` vector cleanup logic runs natively without modification.
* **Fixed the Shared Drive blindspot:** The standard API query was
missing team folders. Injected the `corpora="allDrives"` and
`includeItemsFromAllDrives=True` override flags so the connector now
accurately maps state across both personal workspaces and organizational
Shared Drives.
### Testing:
Isolated the Google API retrieval logic locally to prove the `SlimDoc`
mapping works and correctly registers state drops when a file is trashed
remotely.
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
- [x] Performance Improvement
### What problem does this PR solve?
Fix: enable sync deleted file in airtable
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
### What problem does this PR solve?
## Summary
Migrate two web API endpoints to REST-style HTTP API endpoints,
following the pattern established in #14222:
| Old Endpoint | New Endpoint |
|---|---|
| `POST /v1/chunk/retrieval_test` | `POST
/api/v1/datasets/<dataset_id>/search` |
| `GET /v1/chunk/knowledge_graph` | `GET
/api/v1/datasets/<dataset_id>/graph` |
### What problem does this PR solve?
Fix: google authentication - gmail && google-drive
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
### What problem does this PR solve?
Steps to re-produce (existing bug before API migration):
create a new dataset
upload a file
click on "General" in "Parse" column and then click on "switch or
configure ingestion pipeline"
click on "Settings" (at right of "Auto metadata")
click "Add" to add new metadata
click on "Save"
re-open "Settings" and the newly added metadata is not there
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
### What problem does this PR solve?
agent toolcall null response & schema validation & DeepSeek think
history
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
### What problem does this PR solve?
Feat: enable sync delted files for connectors
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
## Summary
Fixes case-asymmetric matching for manual `meta_data_filter` when using
**`in`** / **`not in`** with a **list** `value`. Document metadata
strings were lowercased, but list elements were not, so values like
`"F2"` failed to match `["F2", "F11"]` even though **`=`** behaved
correctly.
Closes#14389
## Changes
- **`common/metadata_utils.py`**: For **`in`** / **`not in`**, normalize
string elements when `value` and/or `input` is a list, consistent with
scalar string lowercasing.
- **`test/unit_test/common/test_metadata_filter_operators.py`**:
Regression tests for list `value` case-insensitivity and **`not in`**.
## Type of change
- [x] Bug fix (non-breaking)
### What problem does this PR solve?
This PR fixes a regression where Manual pipeline + Naive (Plain Text)
PDF parsing crashed with `AttributeError: 'PlainParser' object has no
attribute 'extract_positions'` in `rag/app/manual.py`.
fixes#14411
### Type of change:
- [x] Bug Fix (non-breaking change which fixes an issue)
### What problem does this PR solve?
Add methods to volcengine
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
Signed-off-by: Jin Hai <haijin.chn@gmail.com>
### What problem does this PR solve?
Always return success if no such task id to follow existing code logic.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
### What problem does this PR solve?
align chat recommendation and thumbup APIs
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
### What problem does this PR solve?
preserve infinity available_int zero filter
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
### What problem does this PR solve?
1. Refactor server_main
2. Add volcengine
### Type of change
- [x] Refactoring
---------
Signed-off-by: Jin Hai <haijin.chn@gmail.com>
[Uploading part_4-13.pdf…]()
### What problem does this PR solve?
In chat, the thumbnails didn't display correctly
### Type of change
- [ ] Bug Fix (non-breaking change which fixes an issue)
Steps to reproduce:
1. create dataset and upload a file (see attached)
2. parse the document
3. once parsing completed, create a chat and associate it with the
dataset
4. ask a question (DAP VS DAPE comparison)
5. check result
### What problem does this PR solve?
Before migration
Web API: POST /v1/document/change_parser
HTTP API: PATCH /api/v1/datasets/<dataset_id>/documents
After consolidation, Restful API
PATCH /api/v1/datasets/<dataset_id>/documents
### Type of change
- [x] Refactoring
### What problem does this PR solve?
Add executor shutdown in finally clause to free resources.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
### What problem does this PR solve?
Before migration: GET /v1/document/thumbnails
After migration: GET /api/v1/thumbnails
### Type of change
- [x] Refactoring
### What problem does this PR solve?
Before migration: POST /v1/document/run
After migration: POST /api/v1/documents/ingest/
### Type of change
- [x] Refactoring
### What problem does this PR solve?
As title.
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
---------
Signed-off-by: Jin Hai <haijin.chn@gmail.com>
### What problem does this PR solve?
### Summary
PR #14222 consolidated KB (web) API endpoints into RESTful Dataset
(HTTP) API endpoints and deleted the web API test suite under
`test_web_api/test_kb_app/` and `test_web_api/test_document_app/`. While
most test coverage was migrated to the HTTP API test suite, some tests
were not ported over. This PR adds back the missing coverage.
### Route migration reference
| Old Web API | New HTTP API | Missing tests |
|---|---|---|
| `POST /v1/kb/update_metadata_setting` | `PUT
/api/v1/datasets/<id>/metadata/config` | auth & error paths |
| `GET /api/v1/datasets/<id>/auto_metadata` | `GET
/api/v1/datasets/<id>/metadata/config` | auth & CRUD |
| `PUT /api/v1/datasets/<id>/auto_metadata` | `PUT
/api/v1/datasets/<id>/metadata/config` | auth & CRUD |
| `GET /v1/kb/<kb_id>/basic_info` | `GET
/api/v1/datasets/<id>/ingestions/summary` | covered |
| `POST /v1/kb/list_pipeline_logs` | `GET
/api/v1/datasets/<id>/ingestions` | edge cases missing |
### Changes
#### `test_file_management_within_dataset/test_metadata_config.py` (new,
10 tests)
Covers `GET/PUT /datasets/<id>/metadata/config` (migrated from
`test_kb_tags_meta.py`'s `test_update_metadata_setting` and
`test_document_metadata.py`'s negative tests):
- Authorization for dataset metadata config GET/PUT
- Authorization for document metadata config PUT
- Success, invalid dataset, missing payload, not found scenarios
#### `test_dataset_management/test_ingestion_logs.py` (extended, +2
tests)
Covers `GET /datasets/<id>/ingestions` edge cases (migrated from
`test_kb_pipeline_tasks.py`):
- Missing dataset ID
- Abnormal date filter
### Type of change
- [x] Other: Test coverage improvement
---------
Signed-off-by: noob <yixiao121314@outlook.com>
### What problem does this PR solve?
Before migration
Web API: POST /v1/document/change_status
After consolidation, Restful API
POST /api/v1/datasets/<dataset_id>/documents/batch-update-status
### Type of change
- [x] Refactoring
### What problem does this PR solve?
prioritize explore session ID and reset default conversation variables
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
### What problem does this PR solve?
Dockerfile v0.25.0 expects nginx conf at path
/etc/nginx/ragflow.conf.python, see
[Dockerfile#L200](ca01c7a745/Dockerfile (L200))
However current helm template mount the conf at path
/etc/nginx/ragflow.conf causing runtime error at startup time.
### Type of change
- [X] Bug Fix (non-breaking change which fixes an issue)
---------
Co-authored-by: Mauro Gattari <mauro.gattari@infn.it>
### What problem does this PR solve?
Before migration: POST /v1/document/upload_info/
After migration: POST /api/v1/documentss/upload/
### Type of change
- [x] Refactoring
## Summary
- **Lazy img_np loading**: `np.array(img)` is now deferred until the
first OCR text extraction is actually needed, avoiding unnecessary
memory allocation for pages that already have text.
- **Chunked parse_into_bboxes**: Large PDFs (>50 pages, configurable via
`PDF_PARSER_PAGE_BATCH_SIZE`) are processed in batches. Each chunk's
boxes are normalized with `_to_global_boxes` to produce globally
consistent page numbers and position tags.
- **DLA early init**: Move remote-client initialization before model
loading in `LayoutRecognizer.__init__` so `DEEPDOC_URL` (or legacy
`TENSORRT_DLA_SVR`) short-circuits unnecessary model download for parser
containers relying on remote inference.
- **Fix outline regression**: Restore `self.outlines =
extract_pdf_outlines(fnm)` in `parse_into_bboxes`; this was dropped
during refactoring and is required by downstream `remove_toc` and
metadata handling in `rag/flow/parser/parser.py`.
## Test plan
- [ ] Small PDF (<=50 pages): verify parse succeeds and `self.outlines`
is populated
- [ ] Large PDF (>50 pages): verify chunked processing produces globally
consistent page numbers
- [ ] With `DEEPDOC_URL` set: verify remote DLA client is used and local
model is not downloaded
- [ ] With legacy `TENSORRT_DLA_SVR` set: verify backward compatibility
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
### What problem does this PR solve?
This PR fixes issue #14371 where file parsing failed after upgrading
from v0.24.0 to v0.25.0, because metadata config could be a JSON Schema
object but was handled like a list and later caused `KeyError:
'properties'`.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
### What problem does this PR solve?
Before migration: GET /v1/document/artifact/<filename>
After migration: GET /api/v1/documents/artifact/<filename>
### Type of change
- [x] Refactoring
### What problem does this PR solve?
Fixes#14196
## Problem
When using DeepDOC to parse large PDFs (over 1000 pages), the parser
silently truncated processing at 300 pages due to a hardcoded default
`page_to=299` in `RAGFlowPdfParser.__images__()`. This caused:
- **Errors** on pages beyond the limit
- **Poor image quality** as the parser attempted to compensate with
missing page data
- **Inconsistent chunk splitting** between full PDF imports and partial
imports
Additionally, the codebase scattered magic numbers (`299`, `600`,
`10000`, `100000`, `100000000`, `10000000000`, `10**9`) across 22 files
as sentinel values for "parse all pages", making future maintenance
error-prone.
## Root Cause
```python
# deepdoc/parser/pdf_parser.py (before)
def __images__(self, fnm, zoomin=3, page_from=0, page_to=299, callback=None):
# Only the first 300 pages were rendered; everything beyond was silently dropped
```
While most callers in `rag/app/*.py` correctly passed `to_page=100000`,
the base class `RAGFlowPdfParser.__call__()` and `parse_into_bboxes()`
invoked `__images__` **without** forwarding `page_from`/`page_to`,
falling back to the restrictive default of 299.
## Solution
### 1. Define constants in `common/constants.py`
```python
MAXIMUM_PAGE_NUMBER = 100000 # Used by the parsing layer
MAXIMUM_TASK_PAGE_NUMBER = MAXIMUM_PAGE_NUMBER * 1000 # Used by the task/DB layer
```
### 2. Replace all hardcoded sentinel values
| Layer | Files Changed | Old Values | New Value |
|---|---|---|---|
| **Deepdoc parsers** | `pdf_parser.py`, `mineru_parser.py`,
`docling_parser.py`, `opendataloader_parser.py`, `paddleocr_parser.py`,
`docx_parser.py` | `299`, `600`, `10**9`, `100000000` |
`MAXIMUM_PAGE_NUMBER` |
| **Chunk parsers** | `naive.py`, `book.py`, `qa.py`, `one.py`,
`manual.py`, `paper.py`, `presentation.py`, `laws.py`, `resume.py`,
`email.py`, `table.py` | `100000`, `10000`, `10000000000` |
`MAXIMUM_PAGE_NUMBER` |
| **Task/DB layer** | `db_models.py`, `task_service.py`,
`document_service.py`, `file_service.py` | `100000000` |
`MAXIMUM_TASK_PAGE_NUMBER` |
### 3. Fix `parse_into_bboxes()` missing parameters
Added `from_page`/`to_page` parameters to `parse_into_bboxes()` so that
the `rag/flow/parser/parser.py` DeepDOC path no longer falls back to the
restrictive default.
## Files Changed (22)
- `common/constants.py`
- `deepdoc/parser/pdf_parser.py`
- `deepdoc/parser/mineru_parser.py`
- `deepdoc/parser/docling_parser.py`
- `deepdoc/parser/opendataloader_parser.py`
- `deepdoc/parser/paddleocr_parser.py`
- `deepdoc/parser/docx_parser.py`
- `rag/app/naive.py`
- `rag/app/book.py`
- `rag/app/qa.py`
- `rag/app/one.py`
- `rag/app/manual.py`
- `rag/app/paper.py`
- `rag/app/presentation.py`
- `rag/app/laws.py`
- `rag/app/resume.py`
- `rag/app/email.py`
- `rag/app/table.py`
- `api/db/db_models.py`
- `api/db/services/task_service.py`
- `api/db/services/document_service.py`
- `api/db/services/file_service.py`
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
- [x] Refactoring
---------
Signed-off-by: noob <yixiao121314@outlook.com>
### What problem does this PR solve?
As title.
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
---------
Signed-off-by: Jin Hai <haijin.chn@gmail.com>
### What problem does this PR solve?
AI coding agents (Claude, Copilot, etc.) tend to directly edit files in
`src/components/ui/` when asked to tweak styles or add props, treating
them like ordinary feature code. This silently breaks the shared
component library that both shadcn primitives and project-authored
common components live in.
This PR adds a `Shared UI Component Lock` convention to `web/CLAUDE.md`
to instruct AI agents to treat the entire `src/components/ui/` directory
as read-only. Any customization must be done via wrappers or composition
outside the directory; exceptions require explicit user approval.
### Type of change
- [x] Other (please describe): Update `CLAUDE.md`
## Summary
PDF files often contain a bookmark/outline tree (table of contents built
into the file by the authoring tool). RAGFlow's `pdf_parser.outlines`
already extracts these `(title, depth)` tuples via pypdf, but they are
used ephemerally during chunking (`manual` parser uses them for
hierarchy detection) and then discarded.
This PR persists the outline as `doc.meta_fields["outline"]` — a JSON
array of `{"title": str, "depth": int}` objects — so downstream features
can use the structural information.
### Why this matters
- **Complementary to `toc_extraction`** — the existing `toc_extraction`
feature uses LLM calls to generate a TOC and only works for the `naive`
parser. The raw PDF outline is free (already extracted by pypdf), works
for all parsers, and captures the author's original document structure.
- **Document navigation** — frontends can render a clickable TOC from
the outline
- **Entity extraction** — the outline provides a structural map for
identifying document sections and key topics
- **Search result context** — knowing which section a chunk belongs to
helps users evaluate relevance
### Changes
| File | Change | LOC |
|------|--------|-----|
| `rag/app/naive.py` | Attach `pdf_parser.outlines` as `__outline__` on
first chunk dict | ~7 |
| `rag/app/manual.py` | Same for the manual parser | ~5 |
| `rag/svr/task_executor.py` | Extract `__outline__`, persist via
`DocMetadataService.update_document_metadata()` | ~12 |
### Design decisions
- **Transient key pattern**: The outline is passed from parser →
task_executor via `__outline__` on the first chunk dict, then removed
before indexing. This follows the same pattern as `metadata_obj` for
LLM-generated metadata.
- **No schema changes**: Uses the existing `meta_fields` JSON column on
the document table.
- **Graceful degradation**: If a PDF has no outline (common for scanned
docs), nothing is stored. If persistence fails, it logs a warning and
continues — parsing is not interrupted.
### Backward compatibility
- **Fully backward compatible** — no existing fields, behavior, or
schemas changed
- PDFs without outlines are unaffected
- Existing `meta_fields` data is preserved (merged, not overwritten)
## Test plan
- [ ] Parse a PDF with bookmarks (e.g. any multi-chapter document),
verify `meta_fields["outline"]` is populated
- [ ] Parse a PDF without bookmarks, verify no errors and no outline key
in meta_fields
- [ ] Verify existing `meta_fields` data is preserved (not overwritten)
when outline is added
- [ ] Verify `manual` parser also persists outlines
- [ ] Verify outline JSON structure: `[{"title": "Chapter 1", "depth":
0}, ...]`
Related: #9921 (Deterministic Document Access Layer)
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: yuch85 <yuch85.1@gmail.com>
Co-authored-by: Wang Qi <wangq8@outlook.com>
### What problem does this PR solve?
## Summary
Closes#6102
When using Infinity as the document store engine (GPU version), calling
`update()` on a non-existent table throws an unhandled
`InfinityException` with error code 3022 (`TABLE_NOT_EXIST`). This
causes users to see a raw "3022" error when clicking on a parsed
document.
## Root Cause
The `update()` methods in both `rag/utils/infinity_conn.py` and
`memory/utils/infinity_conn.py` call `db_instance.get_table(table_name)`
without catching `InfinityException`. In contrast, other CRUD methods
(`insert`, `delete`, `search`) all handle this exception gracefully:
| Method | Handles table-not-exist? | Behavior |
|----------|--------------------------|----------|
| `insert` | ✅ Yes | Auto-creates the table |
| `search` | ✅ Yes | Skips the table |
| `delete` | ✅ Yes | Returns 0 |
| `update` | ❌ **No** | Crashes with 3022 |
Additionally, `api/apps/document_app.py` worked around this with a
fragile string match (`"3022" in msg`) to detect the error.
## Changes
- **`rag/utils/infinity_conn.py`**: Catch `InfinityException` in
`update()`. When `TABLE_NOT_EXIST` is detected, log a warning and return
`False` — consistent with `delete()`.
- **`memory/utils/infinity_conn.py`**: Apply the same fix to its
`update()` method.
- **`api/apps/document_app.py`**: Remove the fragile `"3022"`
string-matching workaround. Table-not-exist is now handled by the `if
not ok` path with an improved error message.
### Type of change
- [x] Refactoring
---------
Signed-off-by: noob <yixiao121314@outlook.com>
## What does this PR do?
Fixes the `hint : 103 Only owner of canvas authorized for this
operation` error that appears when opening a **Chat** shared link
(`/chats/share?shared_id=...&from=chat`).
## Root Cause
The Chat shared page (`web/src/pages/next-chats/share/index.tsx`)
unconditionally calls `useFetchFlowSSE()`, which requests
`/api/canvas/getsse/{sharedId}`. This is an Agent Canvas endpoint that
validates canvas ownership. When sharing a **Chat** dialog (not an
Agent):
1. `sharedId` is a `dialog_id`, not a `canvas_id`
2. The API token's `tenant_id` doesn't match any canvas owner
3. The backend returns `code: 103, message: "Only owner of canvas
authorized for this operation."`
4. The global error interceptor in `request.ts` displays it as a
notification: `hint : 103 Only owner of canvas authorized for this
operation.`
## Changes
- **`web/src/hooks/use-agent-request.ts`**: Added an `enabled` parameter
to `useFetchFlowSSE` so callers can conditionally skip the query.
- **`web/src/pages/next-chats/share/index.tsx`**: Only enable
`useFetchFlowSSE` when `from === SharedFrom.Agent`. For Chat shares, the
hook is disabled, avoiding the unnecessary canvas API call entirely.
## Related Issue
Closes#14115
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
---------
Signed-off-by: noob <yixiao121314@outlook.com>
## Summary
RAPTOR's recursive clustering builds a `layers` list tracking
`(start_idx, end_idx)` boundaries per level, but currently discards this
information — only the flat `chunks` list is returned. This makes it
impossible to distinguish leaf-level summaries from top-level ones.
This PR:
- Returns `(chunks, layers)` tuple from `raptor.py`'s `__call__`
- Annotates each RAPTOR summary chunk with `raptor_layer_int` (1 = first
summary level, 2 = summary-of-summaries, etc.)
- Adds `raptor_layer_int` to `infinity_mapping.json` (Elasticsearch
handles it via existing `*_int` dynamic template)
### Why this matters
Downstream features need to know which RAPTOR layer a summary belongs
to:
- **Retrieving the top-level document summary** for entity extraction,
search snippets, or document comparison
- **Filtering by abstraction level** — users may want only high-level
summaries or only leaf-level cluster summaries
- **RAPTOR recall quality** — #10951 reports summaries not being
recalled for definition queries; layer metadata enables targeted
retrieval
### Changes
| File | Change | LOC |
|------|--------|-----|
| `rag/raptor.py` | Return `(chunks, layers)` tuple | ~3 |
| `rag/svr/task_executor.py` | Build `chunk_layer` mapping, set
`raptor_layer_int` | ~12 |
| `conf/infinity_mapping.json` | Add `raptor_layer_int` integer field |
~1 |
### Backward compatibility
- **Additive only** — no existing fields or behavior changed
- Existing RAPTOR chunks continue to work (they'll have
`raptor_layer_int = 0` by default)
- New RAPTOR chunks get layer metadata automatically
## Test plan
- [ ] Parse a document with RAPTOR enabled, verify `raptor_layer_int` is
set on indexed chunks
- [ ] Verify `raptor_layer_int` values increase with abstraction level
(layer 1 < layer 2 < ...)
- [ ] Verify existing RAPTOR deletion (`delete by raptor_kwd`) still
works
- [ ] Verify Infinity backend accepts the new field
Fixes#7488
Related: #4104, #11191, #10951🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: yuch85 <yuch85.1@gmail.com>
Co-authored-by: Wang Qi <wangq8@outlook.com>
### What problem does this PR solve?
The POST /upload_info?url=<url> endpoint accepted a user-supplied URL
and passed it directly to AsyncWebCrawler without any validation. There
were no restrictions on URL scheme, destination hostname, or resolved IP
address. This allowed any authenticated user to instruct the server to
make outbound HTTP requests to internal infrastructure — including RFC
1918 private networks, loopback addresses, and cloud metadata services
such as http://169.254.169.254 — effectively using the server as a proxy
for internal network reconnaissance or credential theft.
This PR adds an SSRF guard (_validate_url_for_crawl) that runs before
any crawl is initiated. It enforces an allowlist of safe schemes
(http/https), resolves the hostname at validation time, and rejects any
URL whose resolved IP falls within a private or reserved network range.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
### What problem does this PR solve?
Allow search id or _id when using es as doc_engine.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
### What problem does this PR solve?
Feat: introduce minimum type check for pipeline
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
### What problem does this PR solve?
As title
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
---------
Signed-off-by: Jin Hai <haijin.chn@gmail.com>
### What problem does this PR solve?
Fix: The button styles in the PaddleOCR dialog are not applying
correctly.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
Co-authored-by: Copilot <copilot@github.com>
### What problem does this PR solve?
Blob storage sync was downloading unsupported files first and rejecting
them later, which wasted bandwidth and made sync slower. This PR skips
unsupported extensions before download and applies `allow_images` in
blob sync. fixes#14338
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
Resolves#14211
**Background:** Currently, RAGFlow routes all Docling parsing through
the standard `/convert/source` endpoint. For large documents, this
returns massive, unchunked text that exceeds RAGFlow's internal
embedding model context limits, causing pipeline failures.
**Solution:**
This PR updates the `_parse_pdf_remote` ingestion logic in
`docling_parser.py` to prioritize `docling-serve`'s native chunking
endpoints (`/v1/chunk/source` and `/v1alpha/chunk/source`).
- By receiving pre-sliced chunk objects directly from Docling, RAGFlow
natively bypasses token limit overflows.
- Included a graceful fallback mechanism to the standard
`/convert/source` endpoints to maintain backwards compatibility for
users running older versions of the Docling server that return 404s on
the new routes.
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
### What problem does this PR solve?
Allow image2text models (multimodal) to be used as chat models.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
The Langfuse Python SDK v3+ removed `start_generation()` method.
RagFlow's code called this non-existent method, causing AttributeError
when Langfuse tracing is enabled.
Replace all `start_generation()` calls with
`start_observation(as_type="generation")` which is the correct v4 SDK
API.
Affected files:
- api/db/services/llm_service.py (12 occurrences)
- api/db/services/dialog_service.py (1 occurrence)
Fixes#14204
Related to #9243
### What problem does this PR solve?
_Briefly describe what this PR aims to solve. Include background context
that will help reviewers understand the purpose of the PR._
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
---------
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
### What problem does this PR solve?
when use azure blob as the file container, when click parse file, it
calls:
```python
partial(settings.STORAGE_IMPL.put, tenant_id=task["tenant_id"])
```
So any storage backend used there must accept tenant_id as a kwarg.
RAGFlowAzureSasBlob.put() did not, causing:
```
TypeError: ... got an unexpected keyword argument 'tenant_id'
```
Now it does, so parsing should proceed past this point.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
### What problem does this PR solve?
`check_ragflow_server_alive()` in `api/utils/health_utils.py` calls
`requests.get(url)` without a `timeout` parameter. Unlike
`check_minio_alive()` which correctly specifies `timeout=10`, this
health check can hang indefinitely if the server is unresponsive.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
### Changes
Added `timeout=10` to the `requests.get()` call, consistent with
`check_minio_alive()`.
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
Bumps [lxml](https://github.com/lxml/lxml) from 6.0.2 to 6.1.0.
<details>
<summary>Changelog</summary>
<p><em>Sourced from <a
href="https://github.com/lxml/lxml/blob/master/CHANGES.txt">lxml's
changelog</a>.</em></p>
<blockquote>
<h1>6.1.0 (2026-04-17)</h1>
<p>This release fixes a possible external entity injection (XXE)
vulnerability in
<code>iterparse()</code> and the <code>ETCompatXMLParser</code>.</p>
<h2>Features added</h2>
<ul>
<li>
<p>GH#486: The HTML ARIA accessibility attributes were added to the set
of safe attributes
in <code>lxml.html.defs</code>. This allows <code>lxml_html_clean</code>
to pass them through.
Patch by oomsveta.</p>
</li>
<li>
<p>The default chunk size for reading from file-likes in
<code>iterparse()</code> is now configurable
with a new <code>chunk_size</code> argument.</p>
</li>
</ul>
<h2>Bugs fixed</h2>
<ul>
<li>LP#2146291: The <code>resolve_entities</code> option was still set
to <code>True</code> for
<code>iterparse</code> and <code>ETCompatXMLParser</code>, allowing for
external entity injection (XXE)
when using these parsers without setting this option explicitly.
The default was now changed to <code>'internal'</code> only (as for the
normal XML and HTML parsers
since lxml 5.0).
Issue found by Sihao Qiu as CVE-2026-41066.</li>
</ul>
<h1>6.0.4 (2026-04-12)</h1>
<h2>Bugs fixed</h2>
<ul>
<li>LP#2148019: Spurious MemoryError during namespace cleanup.</li>
</ul>
<h1>6.0.3 (2026-04-09)</h1>
<h2>Bugs fixed</h2>
<ul>
<li>
<p>Several out of memory error cases now raise <code>MemoryError</code>
that were not handled before.</p>
</li>
<li>
<p>Slicing with large step values (outside of <code>+/-
sys.maxsize</code>) could trigger undefined C behaviour.</p>
</li>
<li>
<p>LP#2125399: Some failing tests were fixed or disabled in PyPy.</p>
</li>
<li>
<p>LP#2138421: Memory leak in error cases when setting the
<code>public_id</code> or <code>system_url</code> of a document.</p>
</li>
</ul>
<!-- raw HTML omitted -->
</blockquote>
<p>... (truncated)</p>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a
href="43722f4402"><code>43722f4</code></a>
Update changelog.</li>
<li><a
href="87470409b1"><code>8747040</code></a>
Name version of option change in docstring.</li>
<li><a
href="6c36e6cef7"><code>6c36e6c</code></a>
Fix pypistats URL in download statistics script.</li>
<li><a
href="c7d76d6cb8"><code>c7d76d6</code></a>
Change security policy to point to Github security advisories.</li>
<li><a
href="378ccf82db"><code>378ccf8</code></a>
Update project income report.</li>
<li><a
href="315270b810"><code>315270b</code></a>
Docs: Reduce TOC depth of package pages and move module contents
first.</li>
<li><a
href="6dbba7f3c7"><code>6dbba7f</code></a>
Docs: Show current year in copyright line.</li>
<li><a
href="e4385bfa5d"><code>e4385bf</code></a>
Update project income report.</li>
<li><a
href="5bed1e1a22"><code>5bed1e1</code></a>
Validate file hashes in release download script.</li>
<li><a
href="c13ee10a42"><code>c13ee10</code></a>
Prepare release of 6.1.0.</li>
<li>Additional commits viewable in <a
href="https://github.com/lxml/lxml/compare/lxml-6.0.2...lxml-6.1.0">compare
view</a></li>
</ul>
</details>
<br />
[](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)
Dependabot will resolve any conflicts with this PR as long as you don't
alter it yourself. You can also trigger a rebase manually by commenting
`@dependabot rebase`.
[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)
---
<details>
<summary>Dependabot commands and options</summary>
<br />
You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits
that have been made to it
- `@dependabot show <dependency name> ignore conditions` will show all
of the ignore conditions of the specified dependency
- `@dependabot ignore this major version` will close this PR and stop
Dependabot creating any more for this major version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop
Dependabot creating any more for this minor version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop
Dependabot creating any more for this dependency (unless you reopen the
PR or upgrade to it yourself)
You can disable automated security fix PRs for this repo from the
[Security Alerts
page](https://github.com/infiniflow/ragflow/network/alerts).
</details>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
### What problem does this PR solve?
Before migration
Web API: POST /v1/document/metadata/update
After migration, Restful API
PATCH /api/v2/datasets/<dataset_id>/documents/metadatas
### Type of change
- [x] Refactoring
### What problem does this PR solve?
Fix: Recall Test Page Metadata Not Displaying.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
### What problem does this PR solve?
1. Add new provider minimax
2. Add new command: CHECK INSTANCE 'instance_name' FROM 'provider_name';
```
RAGFlow(user)> check instance 'test' from 'minimax';
SUCCESS
```
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
Signed-off-by: Jin Hai <haijin.chn@gmail.com>
### What problem does this PR solve?
Fix: Some bugs
- Pipeline runtime log files could not be viewed
- Corrected TOC terminology errors in the English translation
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
---------
Co-authored-by: Yingfeng <yingfeng.zhang@gmail.com>
### What problem does this PR solve?
Fix: Remove duplicate text output from the thought model on the chat
page.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
### What problem does this PR solve?
Before migration
Web API: POST /v1/document/update_metadata_setting
After consolidation, Restful API
PUT
/api/v1/datasets/<dataset_id>/documents/<document_id>/metadata/config
### Type of change
- [x] Refactoring
### What problem does this PR solve?
This PR fixes the merge-phase crash reported in #14236 during GraphRAG
entity resolution.
The issue happens after candidate pair resolution completes, when
multiple merge coroutines mutate the same shared `networkx` graph
concurrently. In `_merge_graph_nodes`, the code iterates over
`graph.neighbors(node1)` and also awaits during edge/description
merging. That allows another coroutine to modify the graph adjacency
structure in between, which can trigger `RuntimeError: dictionary keys
changed during iteration` and can also lead to unsafe shared-graph
mutation.
This change keeps the PR scoped to that single issue by:
- serializing merge-time graph mutations with a dedicated merge lock
- snapshotting `graph.neighbors(node1)` with `list(...)` before
iteration
Together, these changes prevent concurrent mutation of the shared graph
during the merge phase and make the merge loop safe against live-view
invalidation.
Fixes#14236
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
## Summary
- Replace single `Read()` call in Go upload service with `io.ReadAll()`.
- Prevent potential truncated/corrupted file content during multipart
upload.
- Keep existing API behavior unchanged while fixing data integrity risk.
## Root Cause
`io.Reader.Read()` may return fewer bytes than requested without an
error. The previous implementation read once into a full buffer and
assumed all bytes were populated.
## Test plan
- Upload files of multiple sizes and verify uploaded content integrity.
- Confirm upload endpoint still returns successful responses.
- Verify downstream document parsing works on uploaded files.
## Issues
Closes#14266
### What problem does this PR solve?
As title
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
Signed-off-by: Jin Hai <haijin.chn@gmail.com>
## Add Astraflow Provider Support
This PR integrates [Astraflow](https://astraflow.ucloud.cn/) (by UCloud
/ 优刻得) as a new AI model provider in RAGFlow, with support for both
global and China endpoints.
### About Astraflow
Astraflow is an OpenAI-compatible AI model aggregation platform
supporting 200+ models from major providers including DeepSeek, Qwen,
GPT, Claude, Gemini, Llama, Mistral, and more.
| Variant | Factory Name | Endpoint | Env Var |
|---------|-------------|----------|---------|
| Global | `Astraflow` | `https://api-us-ca.umodelverse.ai/v1` |
`ASTRAFLOW_API_KEY` |
| China | `Astraflow-CN` | `https://api.modelverse.cn/v1` |
`ASTRAFLOW_CN_API_KEY` |
- **API key signup**: https://astraflow.ucloud.cn/
---
### Files Changed
| File | Change |
|------|--------|
| `rag/llm/__init__.py` | Register `Astraflow` and `Astraflow-CN` in
`SupportedLiteLLMProvider` enum, `FACTORY_DEFAULT_BASE_URL`, and
`LITELLM_PROVIDER_PREFIX` |
| `rag/llm/chat_model.py` | Add `AstraflowChat` and `AstraflowCNChat`
(OpenAI-compatible `Base` subclass) |
| `rag/llm/embedding_model.py` | Add `AstraflowEmbed` and
`AstraflowCNEmbed` (subclasses of `OpenAIEmbed`) |
| `rag/llm/rerank_model.py` | Add `AstraflowRerank` and
`AstraflowCNRerank` (subclasses of `OpenAI_APIRerank`) |
| `rag/llm/cv_model.py` | Add `AstraflowCV` and `AstraflowCNCV`
(subclasses of `GptV4`) |
| `rag/llm/tts_model.py` | Add `AstraflowTTS` and `AstraflowCNTTS`
(subclasses of `OpenAITTS`) |
| `rag/llm/sequence2txt_model.py` | Add `AstraflowSeq2txt` and
`AstraflowCNSeq2txt` (subclasses of `GPTSeq2txt`) |
| `conf/llm_factories.json` | Register `Astraflow` and `Astraflow-CN`
factories with a curated list of popular models |
---
### Supported Model Types
- ✅ **Chat / LLM** — DeepSeek-V3/R1, Qwen3, GPT-4o/4.1, Claude 3.5/3.7,
Gemini 2.0/2.5 Flash, Llama 3.3/4, Mistral, and 200+ more
- ✅ **Text Embedding** — text-embedding-3-small/large
- ✅ **Image / Vision (IMAGE2TEXT)** — GPT-4o, GPT-4.1, Claude, Gemini,
Llama-4, etc.
- ✅ **Text Re-Rank**
- ✅ **TTS** — tts-1
- ✅ **Speech-to-Text (SPEECH2TEXT)** — whisper-1
### Implementation Notes
- Uses the `openai/` LiteLLM prefix — consistent with other
OpenAI-compatible aggregation platforms (SILICONFLOW, DeerAPI, CometAPI,
OpenRouter, n1n, Avian, etc.)
- `Astraflow` (global, rank 250) and `Astraflow-CN` (China, rank 249)
are separate factory entries, allowing users to choose the optimal
endpoint based on their region.
- All model classes cleanly subclass existing base classes (`Base`,
`OpenAIEmbed`, `OpenAI_APIRerank`, `GptV4`, `OpenAITTS`, `GPTSeq2txt`)
with no custom logic needed — the provider is fully OpenAI-compatible.
---------
Co-authored-by: user <user@xzaaaMacBook-Air.local>
### What problem does this PR solve?
update MinerU parser to most recent minerU v3 logic
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
### What problem does this PR solve?
Add document of search message with user_id, add sdk support.
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
- [x] Documentation Update
### What problem does this PR solve?
As title.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
Signed-off-by: Jin Hai <haijin.chn@gmail.com>
### What problem does this PR solve?
update MinerU endpoint to /pdf_parse which has been exposed since v3.x.
fixes#14263
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
### What problem does this PR solve?
normalize think tags in final chat answer
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
### What problem does this PR solve?
Before consolidation
Web API: POST /v1/document/rm
Http API - DELETE /api/v1/datasets/<dataset_id>/documents
After consolidation, Restful API -- DELETE
/api/v1/datasets/<dataset_id>/documents
### Type of change
- [x] Refactoring
### What problem does this PR solve?
Refactor /api/v1/chats to be more RESTful.
### Type of change
- [x] Refactoring
---------
Co-authored-by: Jin Hai <haijin.chn@gmail.com>
### What problem does this PR solve?
Before consolidation
Web API: POST /v1/document/infos
Http API - GET /api/v1/datasets/<dataset_id>/documents
After consolidation, Restful API -- GET
/api/v1/datasets/<dataset_id>/documents?ids=id1&ids=id2
### Type of change
- [ ] Refactoring
Closes#14165
Add a short documentation page under Developer Guides introducing
DeepWiki as a resource for developers doing secondary development or
exploring RAGFlow's codebase internals.
---------
Co-authored-by: hyl64 <hyl64@users.noreply.github.com>
### What problem does this PR solve?
Before consolidation
Web API: POST /v1/document/filter
Http API - GET /api/v1/datasets/<dataset_id>/documents
After consolidation, Restful API -- GET
/api/v1/datasets/<dataset_id>/documents?type=filter
### Type of change
- [x] Refactoring
2026-04-21 18:55:30 +08:00
3093 changed files with 677202 additions and 89454 deletions
description: Go naming conventions and best practices. Use this skill when working with Go code and need to name packages, files, directories, structs, interfaces, functions, variables, or constants. Provides comprehensive naming guidelines following Go community standards.
---
Strictly follow the naming conventions in [rules/named.md](rules/named.md)
Strictly follow the naming conventions in [rules/named.md](../../rules/named.md)
# Persist computed ports into .env so docker-compose uses the correct host bindings.
# Remove previous CI overrides first; docker compose uses the last duplicate key.
sed -i '/^ES_PORT=/d;/^OS_PORT=/d;/^INFINITY_THRIFT_PORT=/d;/^INFINITY_HTTP_PORT=/d;/^INFINITY_PSQL_PORT=/d;/^EXPOSE_MYSQL_PORT=/d;/^MINIO_PORT=/d;/^MINIO_CONSOLE_PORT=/d;/^REDIS_PORT=/d;/^TEI_PORT=/d;/^KIBANA_PORT=/d;/^SVR_HTTP_PORT=/d;/^ADMIN_SVR_HTTP_PORT=/d;/^SVR_MCP_PORT=/d;/^GO_HTTP_PORT=/d;/^GO_ADMIN_PORT=/d;/^SANDBOX_EXECUTOR_MANAGER_PORT=/d;/^SVR_WEB_HTTP_PORT=/d;/^SVR_WEB_HTTPS_PORT=/d;/^NATS_PORT=/d;/^COMPOSE_PROFILES=/d;/^TEI_MODEL=/d;/^RAGFLOW_IMAGE=/d;/^DOC_ENGINE=/d' .env
if:always() # always run this step even if previous steps failed
run:|
# Sometimes `docker compose down` fail due to hang container, heavy load etc. Need to remove such containers to release resources(for example, listen ports).
# Persist computed ports into .env so docker-compose uses the correct host bindings.
# Remove previous CI overrides first; docker compose uses the last duplicate key.
sed -i '/^ES_PORT=/d;/^OS_PORT=/d;/^INFINITY_THRIFT_PORT=/d;/^INFINITY_HTTP_PORT=/d;/^INFINITY_PSQL_PORT=/d;/^EXPOSE_MYSQL_PORT=/d;/^MINIO_PORT=/d;/^MINIO_CONSOLE_PORT=/d;/^REDIS_PORT=/d;/^TEI_PORT=/d;/^KIBANA_PORT=/d;/^SVR_HTTP_PORT=/d;/^ADMIN_SVR_HTTP_PORT=/d;/^SVR_MCP_PORT=/d;/^GO_HTTP_PORT=/d;/^GO_ADMIN_PORT=/d;/^SANDBOX_EXECUTOR_MANAGER_PORT=/d;/^SVR_WEB_HTTP_PORT=/d;/^SVR_WEB_HTTPS_PORT=/d;/^NATS_PORT=/d;/^COMPOSE_PROFILES=/d;/^TEI_MODEL=/d;/^RAGFLOW_IMAGE=/d;/^DOC_ENGINE=/d' .env
- name:Generate server coverage report Elasticsearch
if:${{ !cancelled() }}
@@ -587,7 +997,7 @@ jobs:
else
echo ".coverage file not found!"
fi
- name:Collect ragflow log Elasticsearch
if:${{ !cancelled() }}
run:|
@@ -603,8 +1013,11 @@ jobs:
if:always() # always run this step even if previous steps failed
run:|
# Sometimes `docker compose down` fail due to hang container, heavy load etc. Need to remove such containers to release resources(for example, listen ports).
This file provides context, build instructions, and coding standards for the RAGFlow project.
It is structured to follow GitHub Copilot's [customization guidelines](https://docs.github.com/en/copilot/concepts/prompting/response-customization).
Use this file as the local operating guide for the current codebase. Prefer the code and the current CLAUDE.md over any older convention or remembered project shape.
## 1. Project Overview
RAGFlow is an open-source RAG (Retrieval-Augmented Generation) engine based on deep document understanding. It is a full-stack application with a Python backend and a React/TypeScript frontend.
## Core stance
- Treat legacy code as liability, not as a compatibility target.
- Prefer deletion over shims, deprecated branches, wrapper APIs, and dual-track migration notes.
- If old and new implementations coexist, converge to one path unless an external contract forces compatibility.
- Remove dead tests, commented-out code, stale docs, and "move later" notes instead of preserving them.
- Reduce public surface area when a helper can be made private or internal.
- Keep refactors centered on the owning abstraction, not on adjacent compatibility layers.
- **Backend**: Python 3.10+ (Flask/Quart)
-**Frontend**: TypeScript, React, UmiJS
-**Architecture**: Microservices based on Docker.
-`api/`: Backend API server.
-`rag/`: Core RAG logic (indexing, retrieval).
-`deepdoc/`: Document parsing and OCR.
-`web/`: Frontend application.
## Current stack
-Backend: Python 3.13+, Quart-based API server, Peewee ORM, async workers.
-Frontend: React + TypeScript + Vite in `web/`.
-Go: the repository also has a substantial Go module for servers, ingestion, parser/runtime, CLI, and supporting services.
-Runtime services commonly include MySQL/PostgreSQL, Redis, MinIO, and Elasticsearch/Infinity/OpenSearch depending on configuration.
## 2. Directory Structure
-`api/`: Backend API server (Flask/Quart).
-`apps/`: API Blueprints (Knowledge Base, Chat, etc.).
-`db/`: Database models and services.
-`rag/`: Core RAG logic.
-`llm/`: LLM, Embedding, and Rerank model abstractions.
-`deepdoc/`: Document parsing and OCR modules.
-`agent/`: Agentic reasoning components.
-`web/`: Frontend application (React + UmiJS).
-`docker/`: Docker deployment configurations.
-`sdk/`: Python SDK.
-`test/`: Backend tests.
## Code layout to expect
-`api/`: Python API server entrypoints, blueprints, services, and database code.
-`rag/`: ingestion, retrieval, LLM integration, and graph RAG logic.
-`deepdoc/`: parsing and OCR.
-`agent/`: workflow canvas, components, tools, and templates.
-`cmd/`: Go entrypoints. `ragflow_main` is the main server/admin/ingestor binary surface; `ragflow-cli` is the CLI entrypoint.
-`internal/`: main Go application code. Important subtrees:
docker compose -f docker/docker-compose-base.yml up -d
```
- **Launch**:
```bash
source .venv/bin/activate
export PYTHONPATH=$(pwd)
bash docker/launch_backend_service.sh
```
### Frontend (TypeScript/React)
Located in `web/`.
1. **Install Dependencies**:
```bash
cd web
npm install
```
2. **Run Dev Server**:
```bash
npm run dev
```
Runs on port 8000 by default.
### Docker Deployment
To run the full stack using Docker:
## Commands
### Backend
```bash
cd docker
docker compose -f docker-compose.yml up -d
uv sync --python 3.13 --all-extras
uv run python3 ragflow_deps/download_deps.py
docker compose -f docker/docker-compose-base.yml up -d
source .venv/bin/activate
exportPYTHONPATH=$(pwd)
bash docker/launch_backend_service.sh
uv run pytest
ruff check
ruff format
```
## 4. Testing Instructions
### Frontend
```bash
cd web
npm install
npm run dev
npm run build
npm run lint
npm run test
npm run type-check
```
### Backend Tests
- **Run All Tests**:
```bash
uv run pytest
```
- **Run Specific Test**:
```bash
uv run pytest test/test_api.py
```
### Go
```bash
uv run ragflow_deps/download_deps.py
bash build.sh --test ./path/to/package/...
bash build.sh --go
# or build specific binaries:
bash build.sh --all
```
### Frontend Tests
- **Run Tests**:
```bash
cd web
npm run test
```
## 5. Coding Standards & Guidelines
- **Python Formatting**: Use `ruff` for linting and formatting.
```bash
ruff check
ruff format
```
- **Frontend Linting**:
```bash
cd web
npm run lint
```
- **Pre-commit**: Ensure pre-commit hooks are installed.
```bash
pre-commit install
pre-commit run --all-files
```
## Validation preference
- Run the narrowest relevant test, lint, or build command after a change.
- For backend changes, prefer targeted pytest or ruff checks over full-suite runs.
- For frontend changes, prefer the touched-package lint, type-check, or test command.
- For Go changes, prefer package-scoped `bash build.sh --test ...` first.
- Do not default to raw `go test`, `go build`, or IDE Run/Debug for Go in this repo. They often miss the required CGO flags and native static libraries (`office_oxide`, `pdfium-static`, `pdf_oxide`) that `build.sh` wires correctly.
- If Go native builds fail, inspect `build.sh` and `internal/development.md` before changing code. Common environment issues are missing downloaded native deps and missing `lld` on Linux.
## Default review checklist
- Remove instead of retaining `deprecated`, `legacy`, or compatibility-only code.
- Collapse duplicate implementations to one path.
- Drop stale comments and documentation that describe a superseded design.
- Keep exported APIs only when the current code actually needs them.
- 🔧 [Build a Docker image](#-build-a-docker-image)
- 🔨 [Launch service from source for development](#-launch-service-from-source-for-development)
@@ -77,9 +76,9 @@
[RAGFlow](https://ragflow.io/) is a leading open-source Retrieval-Augmented Generation ([RAG](https://ragflow.io/basics/what-is-rag)) engine that fuses cutting-edge RAG with Agent capabilities to create a superior context layer for LLMs. It offers a streamlined RAG workflow adaptable to enterprises of any scale. Powered by a converged [context engine](https://ragflow.io/basics/what-is-agent-context-engine) and pre-built agent templates, RAGFlow enables developers to transform complex data into high-fidelity, production-ready AI systems with exceptional efficiency and precision.
## 🎮 Demo
## 🎮 Get Started
Try our demo at [https://cloud.ragflow.io](https://cloud.ragflow.io).
Try our cloud service at [https://cloud.ragflow.io](https://cloud.ragflow.io).
@@ -88,6 +87,8 @@ Try our demo at [https://cloud.ragflow.io](https://cloud.ragflow.io).
## 🔥 Latest Updates
- 2026-06-15 Support multiple chat channels such as Feishu, Discord, Telegram, Line, etc.
- 2026-04-24 Supports DeepSeek v4.
- 2026-03-24 [RAGFlow Skill on OpenClaw](https://clawhub.ai/yingfeng/ragflow-skill) — Provides an official skill for accessing RAGFlow datasets via OpenClaw.
- 2025-12-26 Supports 'Memory' for AI agent.
- 2025-11-19 Supports Gemini 3 Pro.
@@ -97,7 +98,6 @@ Try our demo at [https://cloud.ragflow.io](https://cloud.ragflow.io).
- 2025-08-08 Supports OpenAI's latest GPT-5 series models.
- 2025-08-01 Supports agentic workflow and MCP.
- 2025-05-23 Adds a Python/JavaScript code executor component to Agent.
- 2025-05-05 Supports cross-language query.
- 2025-03-19 Supports using a multi-modal model to make sense of images within PDF or DOCX files.
- [gVisor](https://gvisor.dev/docs/user_guide/install/): Required only if you intend to use the code executor (sandbox) feature of RAGFlow.
> [!TIP]
@@ -192,12 +193,12 @@ releases! 🌟
> All Docker images are built for x86 platforms. We don't currently offer Docker images for ARM64.
> If you are on an ARM64 platform, follow [this guide](https://ragflow.io/docs/dev/build_docker_image) to build a Docker image compatible with your system.
> The command below downloads the `v0.25.0` edition of the RAGFlow Docker image. See the following table for descriptions of different RAGFlow editions. To download a RAGFlow edition different from `v0.25.0`, update the `RAGFLOW_IMAGE` variable accordingly in **docker/.env** before using `docker compose` to start the server.
> The command below downloads the `v0.26.4` edition of the RAGFlow Docker image. See the following table for descriptions of different RAGFlow editions. To download a RAGFlow edition different from `v0.26.4`, update the `RAGFLOW_IMAGE` variable accordingly in **docker/.env** before using `docker compose` to start the server.
```bash
$ cd ragflow/docker
# git checkout v0.25.0
# git checkout v0.26.4
# Optional: use a stable tag (see releases: https://github.com/infiniflow/ragflow/releases)
# This step ensures the **entrypoint.sh** file in the code matches the Docker image version.
- 🔨 [إطلاق الخدمة من المصدر للتطوير](#-launch-service-from-source-for-development)
@@ -77,7 +76,7 @@
يُعد مشروع [RAGFlow](https://ragflow.io/) محركًا رائدًا ومفتوح المصدر للاسترجاع المعزز بالتوليد (<bdi dir="ltr">RAG</bdi>)، ويجمع أحدث تقنيات <bdi dir="ltr">RAG</bdi> مع قدرات الوكلاء لبناء طبقة سياق متقدمة لنماذج <bdi dir="ltr">LLMs</bdi>. يوفّر سير عمل <bdi dir="ltr">RAG</bdi> مبسّطًا وقابلًا للتكيّف مع المؤسسات بمختلف أحجامها. وبالاعتماد على [محرك سياق موحّد](https://ragflow.io/basics/what-is-agent-context-engine) وقوالب وكلاء جاهزة، يتيح <bdi dir="ltr">RAGFlow</bdi> للمطورين تحويل البيانات المعقّدة إلى أنظمة <bdi dir="ltr">AI</bdi> عالية الدقة وجاهزة للإنتاج بكفاءة وموثوقية.
## 🎮 Demo
## 🎮 ابدأ
جرّب النسخة التجريبية على [https://cloud.ragflow.io](https://cloud.ragflow.io).
@@ -88,8 +87,10 @@
## 🔥 آخر التحديثات
-2026-03-24 [RAGFlow Skill on OpenClaw](https://clawhub.ai/yingfeng/ragflow-skill) — توفر مهارة رسمية للوصول إلى مجموعات بيانات RAGFlow عبر OpenClaw.
- [gVisor](https://gvisor.dev/docs/user_guide/install/): مطلوب فقط إذا كنت تنوي استخدام ميزة منفذ التعليمات البرمجية (وضع الحماية) لـ RAGFlow.
> [!TIP]
@@ -192,12 +193,12 @@
> جميع الصور Docker مصممة لمنصات x86. لا نعرض حاليًا صور Docker لـ ARM64.
> إذا كنت تستخدم نظامًا أساسيًا ARM64، فاتبع [هذا الدليل](https://ragflow.io/docs/dev/build_docker_image) لإنشاء صورة Docker متوافقة مع نظامك.
> يقوم الأمر أدناه بتنزيل إصدار `v0.25.0` من الصورة RAGFlow Docker. راجع الجدول التالي للحصول على أوصاف لإصدارات RAGFlow المختلفة. لتنزيل إصدار RAGFlow مختلف عن `v0.25.0`، قم بتحديث المتغير `RAGFLOW_IMAGE` وفقًا لذلك في **docker/.env** قبل استخدام `docker compose` لبدء تشغيل الخادم.
> يقوم الأمر أدناه بتنزيل إصدار `v0.26.4` من الصورة RAGFlow Docker. راجع الجدول التالي للحصول على أوصاف لإصدارات RAGFlow المختلفة. لتنزيل إصدار RAGFlow مختلف عن `v0.26.4`، قم بتحديث المتغير `RAGFLOW_IMAGE` وفقًا لذلك في **docker/.env** قبل استخدام `docker compose` لبدء تشغيل الخادم.
```bash
$ cd ragflow/docker
# git checkout v0.25.0
# git checkout v0.26.4
# Optional: use a stable tag (see releases: https://github.com/infiniflow/ragflow/releases)
# This step ensures the **entrypoint.sh** file in the code matches the Docker image version.
- 🔎 [Architecture du système](#-architecture-du-système)
- 🎬 [Démarrage](#-démarrage)
- 🎬 [Auto-hébergement](#-auto-hébergement)
- 🔧 [Configurations](#-configurations)
- 🔧 [Construire une image Docker](#-construire-une-image-docker)
- 🔨 [Lancer le service depuis les sources pour le développement](#-lancer-le-service-depuis-les-sources-pour-le-développement)
@@ -77,9 +76,9 @@
[RAGFlow](https://ragflow.io/) est un moteur de [RAG](https://ragflow.io/basics/what-is-rag) (Retrieval-Augmented Generation) open-source de premier plan qui fusionne les technologies RAG de pointe avec des capacités Agent pour créer une couche de contexte supérieure pour les LLM. Il offre un flux de travail RAG rationalisé, adaptable aux entreprises de toute taille. Alimenté par un [moteur de contexte](https://ragflow.io/basics/what-is-agent-context-engine) convergent et des modèles d'agents préconstruits, RAGFlow permet aux développeurs de transformer des données complexes en systèmes d'IA haute-fidélité, prêts pour la production, avec une efficacité et une précision exceptionnelles.
## 🎮 Démo
## 🎮 Démarrage
Essayez notre démo sur [https://cloud.ragflow.io](https://cloud.ragflow.io).
Essayez notre service cloud sur [https://cloud.ragflow.io](https://cloud.ragflow.io).
@@ -88,6 +87,8 @@ Essayez notre démo sur [https://cloud.ragflow.io](https://cloud.ragflow.io).
## 🔥 Dernières mises à jour
- 15-06-2026 Prise en charge de plusieurs canaux de discussion tels que Feishu, Discord, Telegram, Line, etc.
- 24-04-2026 Prise en charge de DeepSeek v4.
- 24-03-2026 [RAGFlow Skill on OpenClaw](https://clawhub.ai/yingfeng/ragflow-skill) — Fournit un skill officiel pour accéder aux datasets RAGFlow via OpenClaw.
- 26-12-2025 Prise en charge de la « Mémoire » pour l'agent IA.
- 19-11-2025 Prise en charge de Gemini 3 Pro.
@@ -97,7 +98,6 @@ Essayez notre démo sur [https://cloud.ragflow.io](https://cloud.ragflow.io).
- 08-08-2025 Prise en charge des derniers modèles de la série GPT-5 d'OpenAI.
- 01-08-2025 Prise en charge du flux de travail agentique et de MCP.
- 23-05-2025 Ajout d'un composant exécuteur de code Python/JavaScript à l'Agent.
- 05-05-2025 Prise en charge des requêtes inter-langues.
- 19-03-2025 Prise en charge de l'utilisation d'un modèle multi-modal pour analyser les images dans les fichiers PDF ou DOCX.
## 🎉 Restez informé
@@ -142,7 +142,7 @@ Essayez notre démo sur [https://cloud.ragflow.io](https://cloud.ragflow.io).
@@ -150,6 +150,7 @@ Essayez notre démo sur [https://cloud.ragflow.io](https://cloud.ragflow.io).
- RAM >= 16 Go
- Disque >= 50 Go
- Docker >= 24.0.0 & Docker Compose >= v2.26.1
- Python >= 3.13
- [gVisor](https://gvisor.dev/docs/user_guide/install/) : Requis uniquement si vous souhaitez utiliser la fonctionnalité d'exécuteur de code (sandbox) de RAGFlow.
> [!TIP]
@@ -189,12 +190,12 @@ Essayez notre démo sur [https://cloud.ragflow.io](https://cloud.ragflow.io).
> Toutes les images Docker sont construites pour les plateformes x86. Nous ne proposons pas actuellement d'images Docker pour ARM64.
> Si vous êtes sur une plateforme ARM64, suivez [ce guide](https://ragflow.io/docs/dev/build_docker_image) pour construire une image Docker compatible avec votre système.
> La commande ci-dessous télécharge l'édition `v0.25.0` de l'image Docker RAGFlow. Consultez le tableau suivant pour les descriptions des différentes éditions de RAGFlow. Pour télécharger une édition de RAGFlow différente de `v0.25.0`, mettez à jour la variable `RAGFLOW_IMAGE` dans **docker/.env** avant d'utiliser `docker compose` pour démarrer le serveur.
> La commande ci-dessous télécharge l'édition `v0.26.4` de l'image Docker RAGFlow. Consultez le tableau suivant pour les descriptions des différentes éditions de RAGFlow. Pour télécharger une édition de RAGFlow différente de `v0.26.4`, mettez à jour la variable `RAGFLOW_IMAGE` dans **docker/.env** avant d'utiliser `docker compose` pour démarrer le serveur.
```bash
$ cd ragflow/docker
# git checkout v0.25.0
# git checkout v0.26.4
# Optionnel : utiliser un tag stable (voir les versions : https://github.com/infiniflow/ragflow/releases)
# Cette étape garantit que le fichier **entrypoint.sh** dans le code correspond à la version de l'image Docker.
- 🔨 [Meluncurkan aplikasi dari Sumber untuk Pengembangan](#-meluncurkan-aplikasi-dari-sumber-untuk-pengembangan)
@@ -77,9 +76,9 @@
[RAGFlow](https://ragflow.io/) adalah mesin [RAG](https://ragflow.io/basics/what-is-rag) (Retrieval-Augmented Generation) open-source terkemuka yang mengintegrasikan teknologi RAG mutakhir dengan kemampuan Agent untuk menciptakan lapisan kontekstual superior bagi LLM. Menyediakan alur kerja RAG yang efisien dan dapat diadaptasi untuk perusahaan segala skala. Didukung oleh mesin konteks terkonvergensi dan template Agent yang telah dipra-bangun, RAGFlow memungkinkan pengembang mengubah data kompleks menjadi sistem AI kesetiaan-tinggi dan siap-produksi dengan efisiensi dan presisi yang luar biasa.
## 🎮 Demo
## 🎮 Mulai
Coba demo kami di [https://cloud.ragflow.io](https://cloud.ragflow.io).
Coba layanan cloud kami di [https://cloud.ragflow.io](https://cloud.ragflow.io).
@@ -88,6 +87,8 @@ Coba demo kami di [https://cloud.ragflow.io](https://cloud.ragflow.io).
## 🔥 Pembaruan Terbaru
- 2026-06-15 Mendukung berbagai saluran obrolan seperti Feishu, Discord, Telegram, Line, dll.
- 2026-04-24 Mendukung DeepSeek v4.
- 2026-03-24 [RAGFlow Skill on OpenClaw](https://clawhub.ai/yingfeng/ragflow-skill) — Menyediakan skill resmi untuk mengakses dataset RAGFlow melalui OpenClaw.
- 2025-12-26 Mendukung 'Memori' untuk agen AI.
- 2025-11-19 Mendukung Gemini 3 Pro.
@@ -97,10 +98,7 @@ Coba demo kami di [https://cloud.ragflow.io](https://cloud.ragflow.io).
- 2025-08-08 Mendukung model seri GPT-5 terbaru dari OpenAI.
- 2025-08-01 Mendukung alur kerja agen dan MCP.
- 2025-05-23 Menambahkan komponen pelaksana kode Python/JS ke Agen.
- 2025-05-05 Mendukung kueri lintas bahasa.
- 2025-03-19 Mendukung penggunaan model multi-modal untuk memahami gambar di dalam file PDF atau DOCX.
- 2024-12-18 Meningkatkan model Analisis Tata Letak Dokumen di DeepDoc.
- 2024-08-22 Dukungan untuk teks ke pernyataan SQL melalui RAG.
## 🎉 Tetap Terkini
@@ -144,7 +142,7 @@ Coba demo kami di [https://cloud.ragflow.io](https://cloud.ragflow.io).
@@ -152,6 +150,7 @@ Coba demo kami di [https://cloud.ragflow.io](https://cloud.ragflow.io).
- RAM >= 16 GB
- Disk >= 50 GB
- Docker >= 24.0.0 & Docker Compose >= v2.26.1
- Python >= 3.13
- [gVisor](https://gvisor.dev/docs/user_guide/install/): Hanya diperlukan jika Anda ingin menggunakan fitur eksekutor kode (sandbox) dari RAGFlow.
> [!TIP]
@@ -192,12 +191,12 @@ Coba demo kami di [https://cloud.ragflow.io](https://cloud.ragflow.io).
> Semua gambar Docker dibangun untuk platform x86. Saat ini, kami tidak menawarkan gambar Docker untuk ARM64.
> Jika Anda menggunakan platform ARM64, [silakan gunakan panduan ini untuk membangun gambar Docker yang kompatibel dengan sistem Anda](https://ragflow.io/docs/dev/build_docker_image).
> Perintah di bawah ini mengunduh edisi v0.25.0 dari gambar Docker RAGFlow. Silakan merujuk ke tabel berikut untuk deskripsi berbagai edisi RAGFlow. Untuk mengunduh edisi RAGFlow yang berbeda dari v0.25.0, perbarui variabel RAGFLOW_IMAGE di docker/.env sebelum menggunakan docker compose untuk memulai server.
> Perintah di bawah ini mengunduh edisi v0.26.4 dari gambar Docker RAGFlow. Silakan merujuk ke tabel berikut untuk deskripsi berbagai edisi RAGFlow. Untuk mengunduh edisi RAGFlow yang berbeda dari v0.26.4, perbarui variabel RAGFLOW_IMAGE di docker/.env sebelum menggunakan docker compose untuk memulai server.
```bash
$ cd ragflow/docker
# git checkout v0.25.0
# git checkout v0.26.4
# Opsional: gunakan tag stabil (lihat releases: https://github.com/infiniflow/ragflow/releases)
# This steps ensures the **entrypoint.sh** file in the code matches the Docker image version.
[RAGFlow](https://ragflow.io/) 는 최첨단 [RAG](https://ragflow.io/basics/what-is-rag)(Retrieval-Augmented Generation)와 Agent 기능을 융합하여 대규모 언어 모델(LLM)을 위한 우수한 컨텍스트 계층을 생성하는 선도적인 오픈소스 RAG 엔진입니다. 모든 규모의 기업에 적용 가능한 효율적인 RAG 워크플로를 제공하며, 통합 [컨텍스트 엔진](https://ragflow.io/basics/what-is-agent-context-engine)과 사전 구축된 Agent 템플릿을 통해 개발자들이 복잡한 데이터를 예외적인 효율성과 정밀도로 고급 구현도의 프로덕션 준비 완료 AI 시스템으로 변환할 수 있도록 지원합니다.
- [gVisor](https://gvisor.dev/docs/user_guide/install/): RAGFlow의 코드 실행기(샌드박스) 기능을 사용하려는 경우에만 필요합니다.
> [!TIP]
@@ -174,12 +174,12 @@
> 모든 Docker 이미지는 x86 플랫폼을 위해 빌드되었습니다. 우리는 현재 ARM64 플랫폼을 위한 Docker 이미지를 제공하지 않습니다.
> ARM64 플랫폼을 사용 중이라면, [시스템과 호환되는 Docker 이미지를 빌드하려면 이 가이드를 사용해 주세요](https://ragflow.io/docs/dev/build_docker_image).
> 아래 명령어는 RAGFlow Docker 이미지의 v0.25.0 버전을 다운로드합니다. 다양한 RAGFlow 버전에 대한 설명은 다음 표를 참조하십시오. v0.25.0과 다른 RAGFlow 버전을 다운로드하려면, docker/.env 파일에서 RAGFLOW_IMAGE 변수를 적절히 업데이트한 후 docker compose를 사용하여 서버를 시작하십시오.
> 아래 명령어는 RAGFlow Docker 이미지의 v0.26.4 버전을 다운로드합니다. 다양한 RAGFlow 버전에 대한 설명은 다음 표를 참조하십시오. v0.26.4와 다른 RAGFlow 버전을 다운로드하려면, docker/.env 파일에서 RAGFLOW_IMAGE 변수를 적절히 업데이트한 후 docker compose를 사용하여 서버를 시작하십시오.
```bash
$ cd ragflow/docker
# git checkout v0.25.0
# git checkout v0.26.4
# Optional: use a stable tag (see releases: https://github.com/infiniflow/ragflow/releases)
# 이 단계는 코드의 entrypoint.sh 파일이 Docker 이미지 버전과 일치하도록 보장합니다.
- 🔎 [Arquitetura do Sistema](#-arquitetura-do-sistema)
- 🎬 [Primeiros Passos](#-primeiros-passos)
- 🎬 [Auto-hospedagem](#-auto-hospedagem)
- 🔧 [Configurações](#-configurações)
- 🔧 [Construir uma imagem docker sem incorporar modelos](#-construir-uma-imagem-docker-sem-incorporar-modelos)
- 🔧 [Construir uma imagem docker incluindo modelos](#-construir-uma-imagem-docker-incluindo-modelos)
@@ -78,9 +77,9 @@
[RAGFlow](https://ragflow.io/) é um mecanismo de [RAG](https://ragflow.io/basics/what-is-rag) (Retrieval-Augmented Generation) open-source líder que fusiona tecnologias RAG de ponta com funcionalidades Agent para criar uma camada contextual superior para LLMs. Oferece um fluxo de trabalho RAG otimizado adaptável a empresas de qualquer escala. Alimentado por [um motor de contexto](https://ragflow.io/basics/what-is-agent-context-engine) convergente e modelos Agent pré-construídos, o RAGFlow permite que desenvolvedores transformem dados complexos em sistemas de IA de alta fidelidade e pronto para produção com excepcional eficiência e precisão.
## 🎮 Demo
## 🎮 Primeiros Passos
Experimente nossa demo em [https://cloud.ragflow.io](https://cloud.ragflow.io).
Experimente o nosso serviço na nuvem em [https://cloud.ragflow.io](https://cloud.ragflow.io).
@@ -89,6 +88,8 @@ Experimente nossa demo em [https://cloud.ragflow.io](https://cloud.ragflow.io).
## 🔥 Últimas Atualizações
- 15-06-2026 Suporte a múltiplos canais de chat, como Feishu, Discord, Telegram, Line, etc..
- 24-04-2026 Suporta DeepSeek v4.
- 24-03-2026 [RAGFlow Skill on OpenClaw](https://clawhub.ai/yingfeng/ragflow-skill) — Fornece um skill oficial para acessar datasets do RAGFlow via OpenClaw.
- 26-12-2025 Suporte à função 'Memória' para agentes de IA.
- 19-11-2025 Suporta Gemini 3 Pro.
@@ -98,10 +99,7 @@ Experimente nossa demo em [https://cloud.ragflow.io](https://cloud.ragflow.io).
- 08-08-2025 Suporta a mais recente série GPT-5 da OpenAI.
- 01-08-2025 Suporta fluxo de trabalho agente e MCP.
- 23-05-2025 Adicione o componente executor de código Python/JS ao Agente.
- 05-05-2025 Suporte a consultas entre idiomas.
- 19-03-2025 Suporta o uso de um modelo multi-modal para entender imagens dentro de arquivos PDF ou DOCX.
- 18-12-2024 Atualiza o modelo de Análise de Layout de Documentos no DeepDoc.
- 22-08-2024 Suporta conversão de texto para comandos SQL via RAG.
## 🎉 Fique Ligado
@@ -145,7 +143,7 @@ Experimente nossa demo em [https://cloud.ragflow.io](https://cloud.ragflow.io).
@@ -153,6 +151,7 @@ Experimente nossa demo em [https://cloud.ragflow.io](https://cloud.ragflow.io).
- RAM >= 16 GB
- Disco >= 50 GB
- Docker >= 24.0.0 & Docker Compose >= v2.26.1
- Python >= 3.13
- [gVisor](https://gvisor.dev/docs/user_guide/install/): Necessário apenas se você pretende usar o recurso de executor de código (sandbox) do RAGFlow.
> [!TIP]
@@ -192,12 +191,12 @@ Experimente nossa demo em [https://cloud.ragflow.io](https://cloud.ragflow.io).
> Todas as imagens Docker são construídas para plataformas x86. Atualmente, não oferecemos imagens Docker para ARM64.
> Se você estiver usando uma plataforma ARM64, por favor, utilize [este guia](https://ragflow.io/docs/dev/build_docker_image) para construir uma imagem Docker compatível com o seu sistema.
> O comando abaixo baixa a edição`v0.25.0` da imagem Docker do RAGFlow. Consulte a tabela a seguir para descrições de diferentes edições do RAGFlow. Para baixar uma edição do RAGFlow diferente da `v0.25.0`, atualize a variável `RAGFLOW_IMAGE` conforme necessário no **docker/.env** antes de usar `docker compose` para iniciar o servidor.
> O comando abaixo baixa a edição`v0.26.4` da imagem Docker do RAGFlow. Consulte a tabela a seguir para descrições de diferentes edições do RAGFlow. Para baixar uma edição do RAGFlow diferente da `v0.26.4`, atualize a variável `RAGFLOW_IMAGE` conforme necessário no **docker/.env** antes de usar `docker compose` para iniciar o servidor.
```bash
$ cd ragflow/docker
# git checkout v0.25.0
# git checkout v0.26.4
# Opcional: use uma tag estável (veja releases: https://github.com/infiniflow/ragflow/releases)
# Esta etapa garante que o arquivo entrypoint.sh no código corresponda à versão da imagem do Docker.
- 🔨 [Geliştirme İçin Kaynaktan Hizmet Başlatma](#-geliştirme-i̇çin-kaynaktan-hizmet-başlatma)
@@ -77,9 +76,9 @@
[RAGFlow](https://ragflow.io/), derin doküman anlayışına dayalı, açık kaynaklı ve öncü bir Artırılmış Üretim ile Bilgi Erişimi ([RAG](https://ragflow.io/basics/what-is-rag)) motorudur. En son RAG teknolojisini Ajan yetenekleriyle birleştirerek LLM'ler için üstün bir bağlam katmanı oluşturur. Her ölçekteki kuruluşa uyarlanabilir, kolaylaştırılmış bir RAG iş akışı sunar. Yakınsanmış bir [bağlam motoru](https://ragflow.io/basics/what-is-agent-context-engine) ve hazır ajan şablonlarıyla donatılmış RAGFlow, geliştiricilerin karmaşık verileri yüksek doğrulukta, üretime hazır yapay zeka sistemlerine olağanüstü verimlilik ve hassasiyetle dönüştürmesini sağlar.
- 2026-06-15 Feishu, Discord, Telegram, Line vb. gibi birden fazla sohbet kanalını destekleyin.
- 2026-04-24 DeepSeek v4 desteği.
- 2026-03-24 [RAGFlow Skill on OpenClaw](https://clawhub.ai/yingfeng/ragflow-skill) — OpenClaw üzerinden RAGFlow veri setlerine erişmek için resmi bir skill sağlar.
- 2025-12-26 Yapay zeka ajanı için 'Bellek' desteği eklendi.
- [gVisor](https://gvisor.dev/docs/user_guide/install/): Yalnızca RAGFlow'un kod çalıştırıcı (sandbox) özelliğini kullanmayı planlıyorsanız gereklidir.
> Tüm Docker imajları x86 platformları için oluşturulmuştur. Şu anda ARM64 için Docker imajı sunmuyoruz.
> ARM64 platformundaysanız, sisteminizle uyumlu bir Docker imajı oluşturmak için [bu kılavuzu](https://ragflow.io/docs/dev/build_docker_image) takip edin.
> Aşağıdaki komut RAGFlow Docker imajının `v0.25.0` sürümünü indirir. Farklı RAGFlow sürümleri için aşağıdaki tabloya bakın. `v0.25.0` dışında bir sürüm indirmek için, `docker compose` ile sunucuyu başlatmadan önce **docker/.env** dosyasındaki `RAGFLOW_IMAGE` değişkenini güncelleyin.
> Aşağıdaki komut RAGFlow Docker imajının `v0.26.4` sürümünü indirir. Farklı RAGFlow sürümleri için aşağıdaki tabloya bakın. `v0.26.4` dışında bir sürüm indirmek için, `docker compose` ile sunucuyu başlatmadan önce **docker/.env** dosyasındaki `RAGFLOW_IMAGE` değişkenini güncelleyin.
```bash
$ cd ragflow/docker
# git checkout v0.25.0
# git checkout v0.26.4
# İsteğe bağlı: Kararlı bir etiket kullanın (sürümler: https://github.com/infiniflow/ragflow/releases)
# Bu adım, koddaki **entrypoint.sh** dosyasının Docker imaj sürümüyle eşleşmesini sağlar.
description="Admin Service's client of [RAGFlow](https://github.com/infiniflow/ragflow). The Admin Service provides user management and system monitoring. "
pub="-----BEGIN PUBLIC KEY-----\nMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEArq9XTUSeYr2+N1h3Afl/z8Dse/2yD0ZGrKwx+EEEcdsBLca9Ynmx3nIB5obmLlSfmskLpBo0UACBmB5rEjBp2Q2f3AG3Hjd4B+gNCG6BDaawuDlgANIhGnaTLrIqWrrcm4EMzJOnAOI1fgzJRsOOUEfaS318Eq9OVO3apEyCCt0lOQK6PuksduOjVxtltDav+guVAA068NrPYmRNabVKRNLJpL8w4D44sfth5RvZ3q9t+6RTArpEtc5sh5ChzvqPOzKGMXW83C95TxmXqpbK6olN4RevSfVjEAgCydH6HN6OhtOQEcnrU97r9H0iZOWwbw3pVrZiUkuRD1R56Wzs2wIDAQAB\n-----END PUBLIC KEY-----"
"system":"You are an intelligent assistant. Your primary function is to answer questions based strictly on the provided knowledge base.\n\n **Essential Rules:**\n - Your answer must be derived **solely** from this knowledge base: `{knowledge}`.\n - **When information is available**: Summarize the content to give a detailed answer.\n - **When information is unavailable**: Your response must contain this exact sentence: \"The answer you are looking for is not found in the knowledge base!\"\n - **Always consider** the entire conversation history.",
"system":'You are an intelligent assistant. Your primary function is to answer questions based strictly on the provided knowledge base.\n\n **Essential Rules:**\n - Your answer must be derived **solely** from this knowledge base: `{knowledge}`.\n - **When information is available**: Summarize the content to give a detailed answer.\n - **When information is unavailable**: Your response must contain this exact sentence: "The answer you are looking for is not found in the knowledge base!"\n - **Always consider** the entire conversation history.',
pub="-----BEGIN PUBLIC KEY-----\nMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEArq9XTUSeYr2+N1h3Afl/z8Dse/2yD0ZGrKwx+EEEcdsBLca9Ynmx3nIB5obmLlSfmskLpBo0UACBmB5rEjBp2Q2f3AG3Hjd4B+gNCG6BDaawuDlgANIhGnaTLrIqWrrcm4EMzJOnAOI1fgzJRsOOUEfaS318Eq9OVO3apEyCCt0lOQK6PuksduOjVxtltDav+guVAA068NrPYmRNabVKRNLJpL8w4D44sfth5RvZ3q9t+6RTArpEtc5sh5ChzvqPOzKGMXW83C95TxmXqpbK6olN4RevSfVjEAgCydH6HN6OhtOQEcnrU97r9H0iZOWwbw3pVrZiUkuRD1R56Wzs2wIDAQAB\n-----END PUBLIC KEY-----"
f"Added tenant for email: {user_info['email']}, A default tenant has been set; changing the default models after login is strongly recommended.")
# TenantLLMService.insert_many(tenant_llm)
logging.info(f"Added tenant for email: {user_info['email']}, A default tenant has been set; changing the default models after login is strongly recommended.")
_logger.debug("[Agent] Built user prompt with length=%d, reasoning=%s, context=%s",len(usr_pmt),bool(kwargs.get("reasoning")),bool(kwargs.get("context")))
ifnotself.tools:
ifself.check_if_canceled("Agent processing"):
return
_logger.debug("[Agent] No tools configured. Delegating to LLM._invoke_async. prompt_count=%d",len(self._param.prompts)ifself._param.promptselse0)
logging.warning("Browser no local browser executable found. Set BROWSER_USE_EXECUTABLE_PATH or preinstall chromium in image to avoid runtime playwright install.")
ifprofile_dir:
browser_kwargs["user_data_dir"]=profile_dir
# browser-use expects profile_directory to be a profile name
# such as "Default" / "Profile 1", not an absolute path.
You are an advanced classification system that categorizes user questions into specific types. Analyze the input question and classify it into ONE of the following categories:
@@ -83,10 +78,7 @@ Here's description of each category:
- Return only the category name without explanations
return"⌛Give me a moment—starting from: \n\n"+re.sub(r"(User's query:|[\\]+)",'',msg[-1]['content'],flags=re.DOTALL)+"\n\nI’ll figure out our best next move."
_,msg,_=self._prepare_prompt_variables()
return"⌛Give me a moment—starting from: \n\n"+re.sub(r"(User's query:|[\\]+)","",msg[-1]["content"],flags=re.DOTALL)+"\n\nI’ll figure out our best next move."
@@ -130,9 +124,7 @@ class E2BProvider(SandboxProvider):
# POST /sandbox/{sandboxID}/execute
raiseRuntimeError(
"E2B provider is not yet fully implemented. "
"Please use the self-managed provider or implement the E2B API integration. "
"See https://github.com/e2b-dev/e2b for API documentation."
"E2B provider is not yet fully implemented. Please use the self-managed provider or implement the E2B API integration. See https://github.com/e2b-dev/e2b for API documentation."
)
defdestroy_instance(self,instance_id:str)->bool:
@@ -208,7 +200,7 @@ class E2BProvider(SandboxProvider):
"min":5,
"max":300,
"description":"API request timeout for code execution",
"de":"CAJAL-Agent für wissenschaftliche Arbeiten",
"zh":"CAJAL 科学论文助手"
},
"description":{
"en":"A local-first scientific paper generation agent for RAGFlow. It is preconfigured for Agnuxo/CAJAL-4B-P2PCLAW through Ollama, retrieves knowledge-base evidence, and drafts citation-grounded LaTeX-ready academic sections.",
"de":"Ein lokal ausgerichteter Agent zur Erstellung wissenschaftlicher Arbeiten in RAGFlow. Er ist für Agnuxo/CAJAL-4B-P2PCLAW über Ollama vorkonfiguriert, ruft Evidenz aus der Wissensdatenbank ab und erstellt zitationsgestützte, LaTeX-fähige akademische Abschnitte.",
"content":"# Research request\n{sys.query}\n\nUse the configured knowledge base retrieval tool before drafting. If no evidence is retrieved, state that limitation clearly."
}
],
"sys_prompt":"## Role & Task\nYou are **CAJAL**, a local-first scientific writing agent specialized in citation-grounded paper generation.\n\nUse RAGFlow retrieval results as the source of truth. Write precise academic content that can be pasted into a paper draft, technical report, or LaTeX manuscript. You are optimized for the local GGUF model `Agnuxo/CAJAL-4B-P2PCLAW` served through Ollama as `agnuxo/cajal-4b-p2pclaw`.\n\n## Operating Rules\n1. Decompose the user request into research goals, target section type, expected citation style, and missing evidence.\n2. Retrieve relevant knowledge-base passages before drafting factual claims.\n3. Ground every substantive claim in retrieved evidence. If evidence is missing, mark it as a limitation instead of inventing a citation.\n4. Prefer structured scientific writing: abstract, introduction, related work, methodology, experiments, results, limitations, and conclusion.\n5. Use LaTeX formatting for equations, symbols, algorithms, tables, and section headings when it helps the manuscript.\n6. Preserve traceability: cite source titles, document names, page numbers, or chunk identifiers when available in retrieved context.\n7. Keep language technical, concise, and reproducible. Avoid marketing language and vague generalizations.\n\n## Output Contract\nReturn one of the following, depending on the user request:\n- A complete paper section with citation markers and a short evidence map.\n- A literature review organized by themes, methods, findings, and gaps.\n- A methodology or experiment section with reproducible steps, assumptions, and limitations.\n- A LaTeX-ready abstract, introduction, or conclusion.\n\nAlways include:\n- **Draft**: the requested scientific content.\n- **Evidence used**: concise bullets mapping claims to retrieved sources.\n- **Limitations**: missing evidence, weak support, or assumptions that require verification.\n",
"temperature":0.2,
"temperatureEnabled":true,
"tools":[
{
"component_name":"Retrieval",
"name":"Retrieval",
"params":{
"cross_languages":[],
"description":"Retrieve papers, reports, datasets, and notes that ground CAJAL scientific writing outputs.",
"empty_response":"",
"kb_ids":[],
"keywords_similarity_weight":0.7,
"outputs":{
"formalized_content":{
"type":"string",
"value":""
}
},
"rerank_id":"",
"similarity_threshold":0.2,
"top_k":1024,
"top_n":10,
"use_kg":false
}
}
],
"topPEnabled":false,
"top_p":0.75,
"user_prompt":"",
"visual_files_var":""
}
},
"upstream":[
"begin"
]
},
"Message:OrangeYearsShine":{
"downstream":[],
"obj":{
"component_name":"Message",
"params":{
"content":[
"{Agent:NewPumasLick@content}"
]
}
},
"upstream":[
"Agent:NewPumasLick"
]
},
"begin":{
"downstream":[
"Agent:NewPumasLick"
],
"obj":{
"component_name":"Begin",
"params":{
"enablePrologue":true,
"inputs":{},
"mode":"conversational",
"prologue":"Hi, I am CAJAL in RAGFlow. Add research papers or datasets to your knowledge base, then ask me to draft a citation-grounded paper section, literature review, methodology, or LaTeX-ready abstract."
"prologue":"Hi, I am CAJAL in RAGFlow. Add research papers or datasets to your knowledge base, then ask me to draft a citation-grounded paper section, literature review, methodology, or LaTeX-ready abstract."
},
"label":"Begin",
"name":"begin"
},
"dragging":false,
"id":"begin",
"measured":{
"height":48,
"width":200
},
"position":{
"x":-9.569875358221438,
"y":205.84018385864917
},
"selected":false,
"sourcePosition":"left",
"targetPosition":"right",
"type":"beginNode"
},
{
"data":{
"form":{
"content":[
"{Agent:NewPumasLick@content}"
]
},
"label":"Scientific Draft",
"name":"Response"
},
"dragging":false,
"id":"Message:OrangeYearsShine",
"measured":{
"height":56,
"width":200
},
"position":{
"x":734.4061285881053,
"y":199.9706031723009
},
"selected":false,
"sourcePosition":"right",
"targetPosition":"left",
"type":"messageNode"
},
{
"data":{
"form":{
"delay_after_error":1,
"description":"",
"exception_comment":"",
"exception_default_value":"",
"exception_goto":[],
"exception_method":null,
"frequencyPenaltyEnabled":false,
"frequency_penalty":0.5,
"llm_id":"agnuxo/cajal-4b-p2pclaw@Ollama",
"maxTokensEnabled":true,
"max_retries":3,
"max_rounds":3,
"max_tokens":32768,
"mcp":[],
"message_history_window_size":12,
"outputs":{
"content":{
"type":"string",
"value":""
}
},
"parameter":"Precise",
"presencePenaltyEnabled":false,
"presence_penalty":0.5,
"prompts":[
{
"role":"user",
"content":"# Research request\n{sys.query}\n\nUse the configured knowledge base retrieval tool before drafting. If no evidence is retrieved, state that limitation clearly."
}
],
"sys_prompt":"## Role & Task\nYou are **CAJAL**, a local-first scientific writing agent specialized in citation-grounded paper generation.\n\nUse RAGFlow retrieval results as the source of truth. Write precise academic content that can be pasted into a paper draft, technical report, or LaTeX manuscript. You are optimized for the local GGUF model `Agnuxo/CAJAL-4B-P2PCLAW` served through Ollama as `agnuxo/cajal-4b-p2pclaw`.\n\n## Operating Rules\n1. Decompose the user request into research goals, target section type, expected citation style, and missing evidence.\n2. Retrieve relevant knowledge-base passages before drafting factual claims.\n3. Ground every substantive claim in retrieved evidence. If evidence is missing, mark it as a limitation instead of inventing a citation.\n4. Prefer structured scientific writing: abstract, introduction, related work, methodology, experiments, results, limitations, and conclusion.\n5. Use LaTeX formatting for equations, symbols, algorithms, tables, and section headings when it helps the manuscript.\n6. Preserve traceability: cite source titles, document names, page numbers, or chunk identifiers when available in retrieved context.\n7. Keep language technical, concise, and reproducible. Avoid marketing language and vague generalizations.\n\n## Output Contract\nReturn one of the following, depending on the user request:\n- A complete paper section with citation markers and a short evidence map.\n- A literature review organized by themes, methods, findings, and gaps.\n- A methodology or experiment section with reproducible steps, assumptions, and limitations.\n- A LaTeX-ready abstract, introduction, or conclusion.\n\nAlways include:\n- **Draft**: the requested scientific content.\n- **Evidence used**: concise bullets mapping claims to retrieved sources.\n- **Limitations**: missing evidence, weak support, or assumptions that require verification.\n",
"temperature":0.2,
"temperatureEnabled":true,
"tools":[
{
"component_name":"Retrieval",
"name":"Retrieval",
"params":{
"cross_languages":[],
"description":"Retrieve papers, reports, datasets, and notes that ground CAJAL scientific writing outputs.",
Some files were not shown because too many files have changed in this diff
Show More
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
