Ports two Python fixes to Go: the variable_ref_patt underscore/colon fix
(#16792) and the TokenChunker upstream-chunks fix (#16825). Keeps Go
behavior aligned with upstream Python.
Fixes#16812
### Problem
In the `rag/flow` ingestion pipeline, when `TitleChunker` feeds
`TokenChunker`, the chapter-aware chunks are silently discarded and the
parser's raw flat json is re-chunked instead.
`TitleChunker` emits `output_format="chunks"` and writes its
chapter-aware output to the `chunks` field
(`rag/flow/chunker/title_chunker/common.py`,
`set_output("output_format", "chunks")`). But `TokenChunker._invoke`
only handles `output_format` in `["markdown", "text", "html"]`, then
falls through to the `# json` path which reads
`from_upstream.json_result`. There is no branch for `"chunks"`, so
`from_upstream.chunks` is never read.
Downstream effects reported in #16812: PageIndex/TOC extraction receives
flat line-level text instead of structured chapter blocks
(incorrect/duplicate/missing chapters), and retrieval quality degrades
because chunks are no longer aligned to document structure.
### Fix
Select the source list based on `output_format`, mirroring the exact
pattern already used in `title_chunker/common.py`:
```python
json_result = (from_upstream.chunks if from_upstream.output_format == "chunks" else from_upstream.json_result) or []
```
`chunks` items share the same dict shape as `json_result` items (both
consumed via `.get("text")`, `.get("doc_type_kwd")`, etc.), so they flow
through the existing token-sizing path unchanged. One-line change, no
behavior change for the `json`/`markdown`/`text`/`html` paths.
### Test
Adds `rag/flow/tests/test_token_chunker.py`, an isolated unit test that
runs the real `TokenChunker._invoke` (heavy deps stubbed; real pydantic
schema used when available) and asserts that with
`output_format="chunks"` the upstream `chunks` are consumed rather than
the raw parser `json`.
Verified RED -> GREEN: the test fails against the current code (reads
the raw json) and passes with the fix.
Signed-off-by: Yash Raj Pandey <yashpn62@gmail.com>
## Summary
`ComponentBase.variable_ref_patt` (and its duplicate in
`agent.canvas.Graph.get_value_with_variable`) is the regex the canvas
runtime uses to find `cpn_id@var_nm` template refs in component prompts.
The `cpn_id` half was constrained to `[a-zA-Z:0-9]+`, which silently
dropped underscores. Component ids emitted by the frontend all contain
underscores (`userfillup_abc`, `retrieval_xyz`, `llm_0`, `message_0`,
…), so any template ref like `{userfillup_abc@line}` failed to match.
The placeholder then leaked through to the LLM verbatim, and the Agent
answered only its system-prompt directive.
This is exactly the "unconsidered await response" symptom in #16758:
```
Begin(Task) -> Await response -> Agent -> Message
```
Widen `cpn_id` from `[a-zA-Z:0-9]+` to `[a-zA-Z0-9_]+`. Bare `{line}`
(no cpn_id) remains unrecognised so it stays literal until the user
wires it up — matching the existing `VARIABLE_REF_PATTERN` shape used by
`agent.dsl_migration` for the same purpose.
## Changes
- `agent/component/base.py` — fix `variable_ref_patt` class attribute.
- `agent/canvas.py` — same fix applied to the inline regex inside
`Graph.get_value_with_variable` (kept as the literal regex to avoid
coupling the two unrelated sites).
-
`test/testcases/test_web_api/test_canvas_app/test_variable_ref_pattern_unit.py`
— new regression test pinning both the regex shape and end-to-end
resolution.
## Regression coverage
```
test_variable_ref_patt_matches_underscored_component_ids PASSED
test_variable_ref_patt_still_matches_legacy_ids PASSED
test_get_input_elements_from_text_resolves_underscored_id PASSED
test_string_format_substitutes_underscored_ref PASSED
test_variable_ref_patt_does_not_match_bare_var_name PASSED
```
All five regression tests fail against the pre-fix regex (verified via
`git stash` round trip — drop fix, tests fail, restore fix, tests pass).
The two targeted existing tests in the same directory
(`test_fillup_unit.py`, `test_iterationitem_unit.py`) continue to pass.
## Repro before the fix
```python
import re
patt = r"\{* *\{([a-zA-Z:0-9]+@[A-Za-z0-9_.-]+|sys\.[A-Za-z0-9_.]+|env\.[A-Za-z0-9_.]+)\} *\}*"
list(re.finditer(patt, "{userfillup_abc@line}"))
# => [] # <-- bug
```
## Repro after the fix
```python
import re
patt = r"\{* *\{([a-zA-Z0-9_]+@[A-Za-z0-9_.-]+|sys\.[A-Za-z0-9_.]+|env\.[A-Za-z0-9_.]+)\} *\}*"
list(re.finditer(patt, "{userfillup_abc@line}"))
# => [<re.Match object; span=(0, 24), match='{userfillup_abc@line}'>]
```
Fixes#16758
## Test plan
- [x] New unit tests pass
- [x] Reverse-apply the fix and confirm the regression tests fail (they
do)
- [x] `test_fillup_unit.py` (existing sibling suite) still passes
- [x] `test_iterationitem_unit.py` (existing sibling suite) still passes
- [ ] Project CI green
---------
Co-authored-by: Taranum01 <taranum01@users.noreply.github.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
### Summary
In Go and python implementation, the dataset / KB id isn't validated if
it is accessible by this user.
---------
Signed-off-by: Jin Hai <haijin.chn@gmail.com>
### Summary
```
RAGFlow(admin)> show version;
+--------------+-----------------------+
| field | value |
+--------------+-----------------------+
| version | v0.26.4-84-g547bc8614 |
| version_type | open source |
+--------------+-----------------------+
```
---------
Signed-off-by: Jin Hai <haijin.chn@gmail.com>
## Summary
- Align Go WenCai and SearXNG behavior, schemas, and node parameters
with Python.
- Add the `WenCai` and `SearXNG` Canvas components and register their
tool factories.
- Match Python's current WenCai behavior by returning an empty report
while its upstream request is disabled.
- Add SearXNG request validation, SSRF-safe DNS pinning, raw result
preservation, and reference rendering.
- Support context cancellation, error envelopes, and lock-safe retrieval
references.
## Tests
Passed:
- `bash build.sh --test ./internal/agent/tool/...`
- `bash build.sh --test ./internal/agent/component/...`
- `bash build.sh --test ./internal/agent/runtime/...`
- `bash build.sh --test ./internal/agent/...`
- `cd web && npm run type-check`
<img width="1900" height="1102" alt="image"
src="https://github.com/user-attachments/assets/ec77d217-d9fd-455a-96ec-9aabf6841109"
/>
<img width="1900" height="1102" alt="image"
src="https://github.com/user-attachments/assets/52ac129f-cb65-453d-ae48-cc518803ac23"
/>
### Summary
Port the **QWeather** agent tool to the modern `ToolBase` / `_invoke`
interface. It was still written against the removed legacy
`ComponentBase` / `_run` / `be_output` API, so it was non-functional as
an Agent tool — adding it to an Agent raised `AttributeError` because it
had no `get_meta()`. This is the same defect that was fixed for the
AkShare tool in #16417.
**Changes**
- `QWeatherParam` now extends `ToolParamBase` with a `meta` exposing a
`query` (location) parameter, and adds `get_input_form()`. Existing
config (`web_apikey`, `lang`, `type`, `user_type`, `time_period`) is
preserved.
- `QWeather` now extends `ToolBase` and implements `_invoke(**kwargs)`
with the standard retry loop, cancellation checks,
`set_output("formalized_content", ...)`, and `thoughts()`. The weather /
indices / air-quality branches and the API error-code messages are kept.
- Added `test/unit_test/agent/component/test_qweather.py` covering the
restored `meta`, param validation, the weather-now and multi-day and
indices branches, the empty-query short-circuit, and the location-lookup
error message.
**Testing**
- `ruff check agent/tools/qweather.py
test/unit_test/agent/component/test_qweather.py` — clean
- `ruff format --check` — clean
- `pytest test/unit_test/agent/component/test_qweather.py`
## Summary
- Add the GitHub Canvas component with tool registration and reference
propagation.
- Align the Invoke component with the Python contract for node config,
input form, response output, and timing fields.
- GitHub search and HTTP Invoke now work correctly in the Go Canvas
runtime.
## Tests
- `bash build.sh --test ./internal/agent/tool/...`
- `bash build.sh --test ./internal/agent/component/...`
Note: the untracked go_ragflow_cli file is not part of the PR changes.
<img width="1813" height="1102" alt="image"
src="https://github.com/user-attachments/assets/f69cef32-59a0-4287-a06b-6843d85198cf"
/>
<img width="1813" height="1102" alt="image"
src="https://github.com/user-attachments/assets/b37dfc31-bc9b-4937-a38e-d2184bb157fe"
/>
## Summary
- register the Go `ArXiv` canvas component and add its input form
- align the Go ArXiv request/schema with Python by keeping only `query`
in runtime args and moving `top_n`/`sort_by` to node params
- keep ArXiv results consistent for canvas output and tool response
handling
## Test
- `bash build.sh --test ./internal/agent/tool
./internal/agent/component`
<img width="1817" height="972" alt="image"
src="https://github.com/user-attachments/assets/7f726dfa-a996-4561-b481-cb0b44bec81c"
/>
### Summary
1. update docker compose file to start NATS healthy
2. Add two commands
```
RAGFlow(admin)> live;
SUCCESS
RAGFlow(admin)> health;
+---------------+-------+
| field | value |
+---------------+-------+
| storage | ok |
| message_queue | ok |
| status | ok |
| db | ok |
| redis | ok |
| doc_engine | ok |
+---------------+-------+
```
---------
Signed-off-by: Jin Hai <haijin.chn@gmail.com>
### What problem does this PR solve?
Issue [#16758](https://github.com/infiniflow/ragflow/issues/16758) —
clicking a chunk whose data references a single-line variable from an
Await-Response (UserFillUp) component, the Agent's `user_prompt` is
being resolved against the **previous** canvas run's captured value
instead of the current run's value. The system-prompt path works only
because the system prompt is computed upstream and re-reads the value on
the new run.
### Root cause
`Canvas._run_impl` reset every path component with `only_output=True`,
so `_param.inputs` was never cleared between runs.
`ComponentBase.get_input()` calls `set_input_value(var, resolved)` at
line 482, which writes the resolved variable into
`self._param.inputs[var]["value"]`. On the next canvas run, that input
was never cleared, so the previous run's resolved value stuck around.
The Agent's `kwargs.get("user_prompt")` then read the stale string and
forwarded it to the LLM, which produced the "Understood. Please provide
the text..." fallback because the prompt looked empty.
### What changed?
- `agent/canvas.py` — differentiate `begin` (still `only_output=True`,
since it has no inputs and the webhook payload branch below populates
`request` explicitly) from non-begin path components (reset with
`only_output=False`, which clears both `inputs` and `outputs`).
- `test/unit_test/agent/test_canvas_input_reset.py` — new pytest module.
Pinned the contract: non-begin path components receive
`only_output=False`. The fix is small enough to verify with a stub
canvas rather than a full canvas-runtime test (the existing agent
conftest hits an unrelated `scholarly` import on Python 3.13, so a real
canvas import would require fixing that first).
### Backward compatibility
- `Begin` behaviour unchanged.
- All non-begin path components: previously persisted inputs across runs
(the bug); now reset between runs. Components that were relying on stale
inputs (none found in the existing test suite) would lose that as a side
effect, but that is the entire point of the fix.
- No API surface change. No backend change.
### Testing
```
$ uv run pytest test/unit_test/agent/test_canvas_input_reset.py -v
collected 4 items
test/unit_test/agent/test_canvas_input_reset.py::test_begin_is_reset_with_only_output_true PASSED
test/unit_test/agent/test_canvas_input_reset.py::test_non_begin_path_components_are_reset_with_only_output_false PASSED
test/unit_test/agent/test_canvas_input_reset.py::test_only_path_components_are_reset PASSED
test/unit_test/agent/test_canvas_input_reset.py::test_inputs_reset_flag_is_passed_to_non_begin_components PASSED
4 passed in 0.14s
```
`python3 -m py_compile agent/canvas.py` clean. Existing agent test files
(`test_switch.py`, `test_llm_prompt.py`) hit a pre-existing `scholarly`
import error on Python 3.13 (unrelated to this PR), so I couldn't run
the full agent suite. Recommend fixing the `scholarly` import
separately.
### Files changed
- `agent/canvas.py` (+9 / −1)
- `test/unit_test/agent/test_canvas_input_reset.py` (new, +104)
Fixes#16758
---------
Co-authored-by: Harsh Kashyap <harshkashyap@Harshs-MacBook-Pro.local>
Co-authored-by: Zhichang Yu <yuzhichang@gmail.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Fixes#7316.
## Problem
`deepdoc/vision/operators.py` defines the image-standardize
preprocessing op as `class StandardizeImag` (missing the final `e`), but
every caller — including
`deepdoc/vision/recognizer.py::Recognizer.preprocess` — looks the class
up by the canonical string `"StandardizeImage"` via:
```python
op_type = new_op_info.pop("type") # "StandardizeImage"
preprocess_ops.append(getattr(operators, op_type)(**new_op_info))
```
So `getattr(operators, "StandardizeImage")` raised `AttributeError`, and
the "StandardizeImage" preprocessing step silently never ran for any
image pipeline that used the dynamic dispatch (LayoutLMv3 and friends).
The user-visible symptom is that the standardize step is missing
entirely from the preprocessing chain, so the model gets un-normalized
images.
## Production fix
```diff
-class StandardizeImag:
+class StandardizeImage:
"""normalize image
Args:
mean (list): im - mean
std (list): im / std
is_scale (bool): whether need im / 255
norm_type (str): type in ['mean_std', 'none']
"""
```
That's the entire production change — a one-character class rename. The
misnamed `StandardizeImag` had no other references in the codebase
(verified via `git grep`), so removing it is safe; every caller uses the
canonical `"StandardizeImage"` string and will now resolve correctly.
## Tests
New `test/unit_test/deepdoc/vision/test_operators_standardize_image.py`
with six regression tests, all green locally:
```
test_standardize_image_class_resolves_by_canonical_name PASSED
test_standardize_image_callable_matches_legacy_alias_name PASSED
test_standardize_image_normalizes_input_with_mean_std_and_is_scale PASSED
test_standardize_image_skips_scaling_when_is_scale_false PASSED
test_standardize_image_norm_type_none_passes_image_through PASSED
test_standardize_image_via_module_getattr_dispatch_path PASSED
6 passed in 0.18s
```
The tests:
1. **Pin the dispatch contract** (`hasattr(operators,
"StandardizeImage")`) — this is the exact check the recognizer's
`getattr` would do, so any future regression fails the same way the
runtime would.
2. **Pin that the misspelled name is gone** — if a downstream caller
ever relied on it, this fails loudly.
3–5. **Behavioural coverage** of the three documented code paths:
`is_scale=True, norm_type="mean_std"`, `is_scale=False,
norm_type="mean_std"`, and `norm_type="none"`.
6. **End-to-end via the same `getattr(operators, "StandardizeImage")`
call** the recognizer uses, with a real numpy image, so any rename or
removal surfaces as `AttributeError` instead of silently skipping the
step.
Verified both ways:
- Without the fix → **all 6 tests fail** (Python even suggests
`'StandardizeImag' → 'StandardizeImage'`)
- With the fix → all 6 pass in 0.15s
The test file follows the project's existing pattern
(`test/unit_test/deepdoc/parser/test_html_parser.py`): load the target
module via `importlib.util.spec_from_file_location`, stub the only
project-internal import (`rag.utils.lazy_image`), and assert against the
loaded module — no full RAGFlow runtime required.
## Risk
Very low. The class is renamed; no public Python API was using the
misnamed class. The only reference path is the `"StandardizeImage"`
string in `recognizer.py:270`, which now resolves correctly.
## Out of scope
- No other ops in `operators.py` are affected; checked all the others
(DecodeImage, NormalizeImage, Permute, etc.) and they all use correct
names.
- The dynamic-dispatch lookups in `recognizer.py` for `LinearResize`,
`StandardizeImage`, `Permute`, `PadStride` all use the same dispatch
path; only the `StandardizeImage` key was broken. No other keys need
fixing.
Made with [Cursor](https://cursor.com)
---------
Co-authored-by: Taranum01 <Taranum01@users.noreply.github.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Zhichang Yu <yuzhichang@gmail.com>
## What
An **Await Response** (`UserFillUp`) node placed inside a **Loop** now
pauses and waits for a fresh user response on **every** iteration,
instead of only on the first one.
## Problem
When a `UserFillUp` node lives inside a `Loop`, it only paused for input
on the first iteration. On subsequent iterations the loop ran straight
through, silently reusing the answer the user gave the first time.
Root cause is in `UserFillUp._invoke` / the canvas wait-check
(`agent/canvas.py`). The wait-check decides whether to pause by calling
`Canvas._is_input_field_satisfied` on the node's form fields — a field
counts as satisfied as soon as its `value` is not `None`:
```python
@staticmethod
def _is_input_field_satisfied(field):
...
if value is None:
return False
return True
```
The same component object is reused across loop iterations, and
`UserFillUp._invoke` writes the answer into
`self._param.inputs[...]["value"]` via `set_input_value`. Nothing
cleared those values when the node was re-entered for the next
iteration, so:
| Iteration | Entry (no answer yet) | Field value | Satisfied? | Result
|
|---|---|---|---|---|
| 1 | fresh | `None` | no | pauses ✅ |
| 1 | resume w/ answer | `answer` | yes | continues ✅ |
| 2 | fresh | `answer` (**stale**) | yes | continues ❌ (should pause) |
## Fix
When a `UserFillUp` is entered without a fresh user answer
(`merged_inputs` is empty), clear the retained form values so the
wait-check treats the form as unsatisfied and pauses again:
```python
merged_inputs = self._merge_runtime_inputs(kwargs.get("inputs", {}))
if not merged_inputs:
self._clear_form_values()
```
- Fresh entry / new loop iteration → no answer supplied → values cleared
→ node pauses and waits.
- Resume with an answer → `merged_inputs` is non-empty → values applied
normally, nothing cleared.
- Non-loop behavior is unchanged: the first entry already had `None`
values, so clearing is a no-op there.
`Begin` overrides `_invoke` and is unaffected.
## Tests
Added to
`test/testcases/test_web_api/test_canvas_app/test_fillup_unit.py`:
- `test_user_fillup_clears_stale_values_on_reentry_without_answer` — a
retained value is cleared on a fresh entry with no answer (loop
re-entry).
- `test_user_fillup_keeps_values_when_answer_supplied` — a supplied
answer is applied and not cleared.
All unit tests pass and `ruff check` is clean.
## Scope
This targets the Python agent runtime (`agent/`). It is independent of
any other in-flight Await Response change.
### Summary
- Make `BLOB_STORAGE_SIZE_THRESHOLD` configurable through an environment
variable.
- Preserve the existing 20 MiB default.
- Add tests for the default and configured values.
### Why
Blob storage, Seafile, and WebDAV connectors currently use a hardcoded
20 MiB limit. Self-hosted users
cannot raise this limit without modifying the source code inside the
container.
### Testing
- `test/unit_test/data_source/test_config.py`: 2 passed
- `ruff check common/data_source/config.py
test/unit_test/data_source/test_config.py`
Fixes#16634
### What problem does this PR solve?
- Clear stale pipeline IDs and generated data when updating documents
without `pipeline_id`.
- Support tree compilation results in pipeline workflows.
- Update compilation templates in place while preserving existing
template IDs.
- Improve duplicate-template validation messages.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
- [x] New Feature (non-breaking change which adds functionality)
Co-authored-by: Jin Hai <haijin.chn@gmail.com>
### Summary
```
RAGFlow(admin)> show users plan quota 100;
+---------+------------------------------------------+
| field | value |
+---------+------------------------------------------+
| quota | 100 |
| command | show_users_plan_quota |
| error | 'Show users plan quota' is not supported |
+---------+------------------------------------------+
```
Signed-off-by: Jin Hai <haijin.chn@gmail.com>
## 🚨 Urgent: this breaks the Docker image build for **every** open PR
The `ragflow_tests_elasticsearch` and `ragflow_tests_infinity` jobs —
which build the image before running tests — currently **fail on all
pull requests**, not just this one. CI is effectively red repo-wide
until this lands. **Please review and merge ASAP.**
Example failing runs (from an unrelated PR):
-
https://github.com/infiniflow/ragflow/actions/runs/29086020168/job/86341032173
-
https://github.com/infiniflow/ragflow/actions/runs/29086020168/job/86341032196
## Symptom
```
Error: Cannot find module '/ragflow/web/scripts/prepare.js'
npm error command sh -c node scripts/prepare.js
ERROR: process "/bin/bash -c cd web && NODE_OPTIONS=..." did not complete successfully: exit code: 1
```
## Root cause
`web/package.json` defines:
```json
"prepare": "node scripts/prepare.js"
```
npm runs the `prepare` lifecycle script during `npm install`. But the
frontend-deps layer in the `Dockerfile` only copies the package
manifests before installing:
```dockerfile
COPY web/package.json web/package-lock.json web/.npmrc ./web/
RUN ... cd web && npm install # runs `node scripts/prepare.js` → file not present yet
```
Since `web/scripts/prepare.js` is not in the image at that point, `npm
install` aborts and the whole build fails.
## Fix
Copy `web/scripts` before `npm install` so the `prepare` script is
present:
```dockerfile
COPY web/package.json web/package-lock.json web/.npmrc ./web/
COPY web/scripts ./web/scripts
RUN ... cd web && npm install
```
- Minimal and safe: `scripts/prepare.js` only installs lefthook git
hooks and already no-ops (wrapped in try/catch) inside the image where
there is no Git repo.
- Preferred over `--ignore-scripts`, which would also disable
dependencies' legitimate install scripts.
- `web/scripts` changes rarely, so build-cache impact on this layer is
negligible.
## Verification
Build reaches the frontend `npm install` step without the
`MODULE_NOT_FOUND` error. No application code is touched — this is a
build-infrastructure fix only.
## Summary
- Register Wikipedia component + tool alias
`wikipedia`/`wikipedia_search`
- Use `generator=search` to get title/summary/url in one request (was
N+1)
- Node params `top_n`/`language` with validation
- Return `formalized_content` for downstream
- tests pass
<img width="1817" height="972" alt="image"
src="https://github.com/user-attachments/assets/f6d79599-6d1f-4ea6-84f7-ac06d0de13b0"
/>
Two refactors on the Go port (agent-go-port):
- Remove the dead per-tenant canvas runtime selector (write-only Redis
scaffolding with no runtime callers) and its dependent metrics/admin
code.
- Move the tokenizer embedding-model id from the shared ingestion
globals into a Tokenizer-scoped setup, and wire the production embedder
resolver in the ingestion task package.
32 files changed, 861 insertions, 1228 deletions.
## Summary
- align the Go Google Scholar component with the Python-side config
pattern
- merge node-level params with runtime inputs so canvas defaults are
preserved and per-run inputs can override them
- add tests covering node param fallback and runtime override behavior
## Verification
- `bash build.sh --test ./internal/agent/component/... -run
TestGoogleScholar`
<img width="1873" height="1165" alt="image"
src="https://github.com/user-attachments/assets/67198c6f-6a0e-43bf-a500-8e88d82b8751"
/>
### What problem does this PR solve?
Closes#8175.
The Agent OpenAI-compatible streaming path uses `get_data_openai(...,
stream=True)`, but that helper currently returns a minimal chunk shape.
The main OpenAI-compatible chat endpoint already includes chunk metadata
such as `created`, `system_fingerprint`, `usage`, `logprobs`, and
assistant role/tool placeholders.
This PR aligns the Agent stream helper with that existing
OpenAI-compatible chunk shape while keeping the current `delta.content`
behavior and existing reference injection path intact.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
- [ ] New Feature (non-breaking change which adds functionality)
- [ ] Documentation Update
- [ ] Refactoring
- [ ] Performance Improvement
- [ ] Other (please describe):
### Verification
- `./.venv/bin/python -m pytest
test/unit_test/api/utils/test_api_utils.py -q`
- `python3 -m py_compile api/utils/api_utils.py
test/unit_test/api/utils/test_api_utils.py`
- `uvx ruff check api/utils/api_utils.py
test/unit_test/api/utils/test_api_utils.py`
---------
Co-authored-by: Harsh Kashyap <harshkashyap@Harshs-MacBook-Pro.local>
feat(ingestion): mirror Go pipeline progress into the document table;
harden resume guards
- pipeline: bind the owning document via WithDocumentID; after each
TrackProgress event aggregate ingestion_task_log progress and mirror
progress/run/progress_msg back into the document table, so GET
/api/v1/datasets/{dataset_id}/documents reflects live Go pipeline
progress without a bespoke endpoint.
- canvas: extend the S3 resume guard to reject legacy no-op nodes (e.g.
ExitLoop) so component_total equals the count of progress-reporting
components and the aggregate percent can reach 100%.
- runtime/canvas: route progress through TrackProgress; add interrupt
test coverage (r3_interrupt_test.go).
- dao/entity: add IngestionTask.DocumentID column and AggregateProgress
support used by the mirror; IngestionTaskLog keeps a Checkpoint column
alongside the progress fields.
feat(deepdoc): cache DocAnalyzer inference results in Redis (1h TTL)
- Redis-backed DocAnalyzerCache decorator over inference.Client; cache
key = "ddoc:cache:<method>:" + sha256 of the JPEG-encoded image bytes
(deterministic).
- TTL = 1h; hits skip the inner HTTP call and return cached JSON; inner
errors are not cached.
refactor(deepdoc): align figure cropping with Python cropout + bounded
page caches
- CropSectionByDLA mirrors Python cropout: best-overlap DLA
figure/equation region, fallback to section bbox per page, vertical
concat on gray background.
- sliding-window page-image cache bounds peak memory to the recent
window instead of the whole PDF.
- rename DLADebug -> DLARegions across parser/chunker/tests.
refactor(parser): drop lib_type selector; align NewXxxParser with
NewPDFParser
- remove config["lib_type"] lookup and the libType param/field/switch
from all nine constructors; surface the CGO-required error at
ParseWithResult time instead of construction time; drop resolveLibType,
its test, and the four lib_type constants.
feat(utility): add a reusable workerpool for bounded concurrent
execution
- internal/utility/workerpool.go (+ tests).
refactor: translate Chinese prose comments to English in non-harness Go
files.
chore: upgrade github.com/cloudwego/eino from v0.9.9 to v0.9.12.
## Summary
- register the Go `DuckDuckGo` canvas component and restore its dynamic
input form metadata
- align the Go component input/output surface with the current canvas
usage for `query`, `channel`, and `top_n`
- fix DuckDuckGo news search in Go by fetching the required `vqd` token
before calling `news.js`, and add targeted regression tests
## Testing
Passed:
- `bash build.sh --test ./internal/agent/tool/... -run 'DuckDuckGo'`
- `bash build.sh --test ./internal/agent/component/... -run
'DuckDuckGo|TestVerifyRegistration_P1'`
- `bash build.sh --test ./internal/agent/component/... -run
'DuckDuckGo'`
Not run:
- frontend tests
- frontend build
- full Go test suite
<img width="1776" height="1092" alt="image"
src="https://github.com/user-attachments/assets/9f3f8e4b-f6b4-4915-b96c-3c5b8c7b8b30"
/>
### Summary
- Implemented googleComponent wrapper to bridge the canvas component
contract with Eino's SerpApi-backed GoogleTool.
- Added parameter alias mapping (query to q, max_results to num) and
content formatting logic to match Python search result representation.
- Registered the "Google" component and the "google" tool factory in the
Go agent runtime to support web search nodes.
<img width="1776" height="1092" alt="image"
src="https://github.com/user-attachments/assets/e295ab88-e48c-4fe2-bcb7-47ca5b977c9b"
/>
### Summary
- TavilySearch now stores api_key from component params and injects it
into tool calls when runtime inputs omit it.
- TavilyExtract and BGPT now follow the same stored api_key behavior.
- Canvas decoding now recovers api_key from graph.nodes[].data.form when
components[].obj.params.api_key is empty, matching frontend payload
behavior without changing frontend data.
- Added regression tests for graph form key recovery and stored key
injection / caller key precedence.
Tests: build.sh --test ./internal/agent/component ./internal/service —
all pass.
<img width="1476" height="850" alt="image"
src="https://github.com/user-attachments/assets/0be31587-c1ba-4f3e-b43a-4fe0fca5a44c"
/>
<img width="1476" height="850" alt="image"
src="https://github.com/user-attachments/assets/e3edd92c-c62e-4db4-abe2-772bdf4fe1b2"
/>
### Summary
Handle searching dataset without embedding model
In this PR, Searching datasets with different embedding models or
searching dataset with/without embedding models are not allowed. We will
improve the behavior later.
### Summary
Updates the NGINX package used in the RAGFlow Docker image from
`1.31.0-1~noble` to `1.31.2-1~noble`.
NGINX 1.31.0 is affected by CVE-2026-9256. NGINX 1.31.2 includes
the corresponding security fix and is available from the official
NGINX mainline repository for Ubuntu Noble.
### References
- nginx security advisories:
https://nginx.org/en/security_advisories.html
- Vendor advisory: https://my.f5.com/manage/s/article/K000161377
### Fix
Single-line change in `Dockerfile:62`:
```diff
-ARG NGINX_VERSION=1.31.0-1~noble
+ARG NGINX_VERSION=1.31.2-1~noble
Co-authored-by: duanyuan <duanyaun@uyuyue.com>
## Summary
This PR updates Go agent publish logic to persist the parent canvas
update and canvas-version save in the same transaction.
## Changes:
- Reuse SaveOrReplaceLatest semantics for published versions
- Add SaveOrReplaceLatestTx for transactional publish flow
- Keep canvas release update and version persistence atomic
- Add a focused publish test covering canvas and released version state
## Tested:
```
bash build.sh --test -run 'Test(PublishAgentUpdatesCanvasAndReleasedVersion|UpdateAgentDSLCreatesAndReplacesDraftVersion|
UpdateAgentDSLDoesNotOverwriteLatestReleasedVersion)$' ./internal/service ./internal/dao
```
<img width="1476" height="850" alt="image"
src="https://github.com/user-attachments/assets/2c576581-1143-420b-8750-a77aa3c4292d"
/>
### Summary
Add object as a begin-node parameter type with JSON editor UI, webhook
schema support, and backend parsing in UserFillUp.
Co-authored-by: Cursor <cursoragent@cursor.com>
### Summary
The Go backend never seeded the `canvas_template` table, so the "Create
agent from template" page was blank when the frontend proxies to the Go
API (`API_PROXY_SCHEME=go`). This PR adds `SeedCanvasTemplates()` in
`internal/dao`, invoked from `InitDB()` after migrations, which loads
`agent/templates/*.json` and mirrors Python's `add_graph_templates()`
behavior.
Changes:
- Add `internal/dao/canvas_template_seed.go` to parse and upsert
built-in templates.
- Call `SeedCanvasTemplates()` in `InitDB()`.
- Add `CanvasTypes` (`JSONSlice`) to `entity.CanvasTemplate` so the
frontend can filter/group by category.
- Skip seeding gracefully when the templates directory is absent.
This fixes the blank template catalogue in Go mode.
## Summary
Adds the missing input form metadata for the Go BGPT canvas component.
## Root Cause
The standalone BGPT component was registered in Go, but it did not
implement GetInputForm(). During component trial run, the backend asks
the component for its input_form. Since BGPT had none, the API returned:
component has no input_form: BGPT:<node_id>
Python BGPT already exposes the query input form, so the Go component
needed the same contract.
## Change
Added GetInputForm() to the Go BGPT component with a single query line
input.
Added test coverage to ensure BGPT exposes the input form.
## Validation
Backend:
bash build.sh --test -run TestBGPT ./internal/agent/component
<img width="1369" height="1184" alt="image"
src="https://github.com/user-attachments/assets/f99e4a81-2359-42e5-80bb-dcc4e6a63fea"
/>
<img width="1736" height="1152" alt="image"
src="https://github.com/user-attachments/assets/c11240a5-2c42-4d08-88e3-c6dfbf49eedb"
/>
## Summary
Fixes a page crash when opening the BGPT node configuration in the
canvas.
## Root Cause
BGPT was using the tool-form watcher call pattern in a normal canvas
component form.
Tool forms use:
useWatchFormChange(form)
Canvas component forms use:
useWatchFormChange(node?.id, form)
Tool is not equal to component. The BGPT canvas component imported the
component-level hook but called it like a tool-form hook, so the form
argument became undefined and React Hook Form tried to read control from
a null context.
## Change
Updated the BGPT canvas form to pass the node id and form instance
correctly.
## Validation
Ran ESLint for the changed file:
npx eslint src/pages/agent/form/bgpt-form/index.tsx
<img width="1369" height="1184" alt="image"
src="https://github.com/user-attachments/assets/a40c5202-7394-4f26-9da2-08329dcc7fbf"
/>
## Summary
- Add language-aware Snowball stemmer to `RagTokenizer` supporting 16
languages (Dutch, German, French, Spanish, etc.)
- Thread the KB `language` parameter through the full tokenization
pipeline (14 parser modules + task executor)
- Add Dutch to the frontend language lists and cross-language form
## Problem
RAGFlow uses the English Porter stemmer + WordNet lemmatizer for **all**
BM25 tokenization, regardless of the knowledge base language setting.
This produces incorrect stems for non-English text. For example:
| Dutch word | Dutch stemmer | English Porter |
|---|---|---|
| documenten | document | documenten (unchanged!) |
| gebruikers | gebruiker | gebruik (over-stemmed) |
| instellingen | instell | instellingen (unchanged!) |
This degrades BM25 recall for any non-English knowledge base.
## Solution
NLTK already ships Snowball stemmers for 16 languages. This PR:
1. **`rag/nlp/rag_tokenizer.py`**: Overrides `tokenize()` with
`set_language()` and `_normalize_token()` that selects the correct NLTK
Snowball stemmer. Falls back to Porter for unmapped languages (Chinese,
Japanese, Korean, etc. — these use character-based tokenization anyway).
2. **`rag/nlp/__init__.py`** + **14 `rag/app/*.py` parsers** +
**`rag/svr/task_executor.py`**: Threads the `language` parameter through
`tokenize()`, `tokenize_chunks()`, `tokenize_table()`, and all callers.
3. **Frontend**: Adds Dutch (`Nederlands`) to `LanguageList`,
`LanguageMap`, `LanguageAbbreviationMap`, `LanguageTranslationMap`,
cross-language form field, and `en.ts` locale.
## Backward Compatibility
- Default language is `"English"`, preserving existing behavior for all
current users
- Languages without a Snowball stemmer mapping fall back to Porter (no
change)
- No new dependencies — NLTK Snowball is already bundled
### Motivation
This PR evolves the harness from a pure execution runtime into an
**observable, replayable agent evaluation platform**. The current
`harness/graph` checkpoint mechanism is insufficient for true
event-sourced introspection—we need append-only event logs capturing
every tool call, state transition, memory write, and approval decision,
enabling deterministic replay, fork/diff, postmortem analysis, and
time-travel debugging.
### Key Design Goals
1. **Event-Sourced Execution Model**
Replace coarse checkpoints with granular, append-only event logs. Every
operation becomes a durable event: tool invocation, state mutation,
memory update, human approval. This unlocks deterministic replay,
branching execution histories, and regression datasets derived directly
from production failures.
2. **First-Class Replay & Evaluation Loop**
Replay is not an afterthought—it is a core primitive. A single live run
seeds an offline corpus that supports: repeated playback, model
substitution, tool result mocking, and strategy comparison. The harness
graduates from "executor" to "continuous evaluation platform" where
failed production traces convert directly into offline regression
suites.
3. **Operational Observability**
Beyond raw traces, expose metrics that prove stability over time:
- Tool success / failure rates
- Approval latency distributions
- Retry frequencies
- Checkpoint restore reliability
- Memory retrieval quality
- Cost per completed task
- Fork replay pass rates
The underlying thesis: the bottleneck for most agent systems is not
execution capability, but the inability to **demonstrate continuous,
measurable improvement**.
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
## Summary
Closes#15483.
Default workflow/session agent completions to non-streaming when
`stream` is omitted.
## Changes
- `api/apps/restful_apis/agent_api.py`: `req.get("stream", False)` on
workflow paths.
## Test plan
- [ ] POST workflow completion without `stream`; assert JSON response.
Ports remaining Go parser wiring and PDF backends, adds tenant-aware VLM
dispatch, aligns post-processing with Python, and adds end-to-end
pipeline coverage with a generated six-page PDF.
### Summary
Keep `data` as the uploaded document array when dataset document upload
partially succeeds.
This matches the Python API behavior and allows parse-on-creation to run
for successfully uploaded files when other files in the same folder are
unsupported.
## Summary
- Add Go dynamic input form support for ExeSQL and Browser components.
- Align their input form metadata with the Python implementation.
- Add regression tests for `/components/:component_id/input-form`.
## Summary
Debugging YahooFinance component in agent canvas returns "unknown
component" and "no input_form".
YahooFinance was only registered as an eino tool, not as a runtime
component. The component factory only searches the runtime registry.
- `universe_a_wrappers.go`: add `yahooFinanceComponent` wrapper
delegating to `agenttool.YahooFinanceTool` with `GetInputForm()`
- `fixture_stubs.go`: register `"YahooFinance"` component
## TEST
`go build` and `go test ./internal/agent/component/...` all pass.
## What this fixes
Closes#16400.
`get_data_openai()` currently returns `created: null` when callers do
not pass a timestamp, and it replaces explicit timestamp values with the
current time. This makes non-streaming OpenAI-compatible responses
inconsistent with the expected integer `created` timestamp field.
## Change
- Preserve explicit `created` values when provided.
- Default non-streaming responses to `int(time.time())` when `created`
is not provided.
- Add focused unit coverage for default timestamps, explicit timestamps,
and unchanged streaming chunk shape.
## Verification
- `./.venv/bin/python -m pytest
test/unit_test/api/utils/test_api_utils.py -q`
- `python3 -m py_compile api/utils/api_utils.py
test/unit_test/api/utils/test_api_utils.py`
- `uvx ruff check api/utils/api_utils.py
test/unit_test/api/utils/test_api_utils.py`
---------
Co-authored-by: Harsh Kashyap <harshkashyap@Harshs-MacBook-Pro.local>
### What problem does this PR solve?
The Go chunk pipeline's `PostprocessOperator` `filter` stage
(`internal/ingestion/chunk/postprocess.go`) only filtered by length
(`min_length`/`max_length`). It could not drop empty/whitespace-only
chunks or duplicate chunks — both standard RAG post-processing steps
(blank chunks shouldn't be indexed; identical chunks waste embedding
compute and add redundant retrieval results).
This adds two optional, default-off booleans to the `filter` config:
- `drop_empty` — drop chunks whose content is empty or whitespace-only.
- `drop_duplicates` — drop chunks whose exact content already appeared
(order-preserving; the first occurrence is kept).
They compose with the existing length bounds and are reflected in
`String()` for plan explainability. Also adds the first unit tests for
the postprocess filter (length bounds, drop_empty, drop_duplicates,
combined, exact-content matching, and config parsing).
Validation: `gofmt` clean, `go vet ./internal/ingestion/chunk/` clean,
`go build` ok, `go test ./internal/ingestion/chunk/` — all tests pass.
Closes#16048
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
Co-authored-by: Ling Qin <qinling0210@163.com>
### What problem does this PR solve?
The Go ingestion chunk pipeline's `SplitOperator`
(`internal/ingestion/chunk/split.go`) supported only `sentence`, `char`,
and `paragraph` strategies, but not **fixed-size (length) chunking with
overlap** — the canonical RAG strategy for bounding chunk length while
preserving cross-boundary context.
This adds a `length` strategy alongside the existing ones, configurable
via DSL `params`:
- `chunk_size` — target window size in **runes** (rune-aware:
multi-byte/CJK text is windowed by character, never split mid-rune).
- `overlap` — runes carried from the end of each window into the next.
The window advances by `chunk_size - overlap`. `chunk_size` falls back
to a default (256) when unset/non-positive, and `overlap` is clamped to
`[0, chunk_size-1]` so the window always advances and the operation
terminates. Implementation follows the existing
`splitByChar`/`splitByParagraph` pattern and reuses `DetectLanguage` for
chunk metadata.
It also adds `split_test.go` — the first unit tests for the `chunk`
package — covering basic windowing, overlap, overlap
clamping/termination, rune-awareness (CJK), default sizing, no-overlap
reconstruction, empty input, and DSL param parsing.
Validation: `gofmt` clean, `go vet ./internal/ingestion/chunk/` clean,
`go build` ok, `go test ./internal/ingestion/chunk/` — all tests pass.
Closes#16046
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
Co-authored-by: bittoby <218712309+bittoby@users.noreply.github.com>
## Summary
- derive Go Agent debug input forms from prompt variable references
instead of Agent meta fields
- seed `sys.*` debug params into `CanvasState.Sys` so single-component
debug resolves prompt variables like Python
- restore Agent test-run parity for form rendering and debug execution
## Tests
- `go test ./internal/agent/component -run
'TestAgent_(GetInputForm_UsesPromptReferences|GetInputForm_DeduplicatesPromptReferences|Meta_DefaultsToEmpty|Reset_NoTools)$'`
- `go test ./internal/handler -run
'Test(DebugComponent_SeedsSysInputsIntoCanvasState|DebugComponent_HappyPath_Begin|GetComponentInputForm_HappyPath)$'`
AFTER:
<img width="669" height="456" alt="image"
src="https://github.com/user-attachments/assets/4fd86559-aafc-4027-91ae-6e666137ee1b"
/>
## Related issues
Closes#15375
### What problem does this PR solve?
`GET /api/v1/messages` and `GET /api/v1/messages/search` accepted
unbounded `limit` / `top_n` query parameters while other REST list
endpoints enforce `REST_API_MAX_PAGE_SIZE` (100) via
`validate_rest_api_page_size()`. Oversized values can trigger expensive
memory index queries and large result sets (DoS risk).
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
### Changes
| File | Change |
|------|--------|
| `api/apps/restful_apis/memory_api.py` | Cap `limit` and `top_n` with
`validate_rest_api_page_size`; return argument error when exceeded |
|
`test/testcases/test_web_api/test_message_app/test_message_routes_unit.py`
| Regression tests for oversized `limit` / `top_n` |
### Test plan
- [x] Unit tests added
- [ ] `pytest
test/testcases/test_web_api/test_message_app/test_message_routes_unit.py`
Co-authored-by: Cursor <cursoragent@cursor.com>
Follow-up from #13896Fixes#13840
### What problem does this PR solve?
In #13896, only the docker-compose-base.yml was adjusted. However, in
the Helm chart, the unmaintained minio/minio image is still referenced.
This PR syncs the Helm chart with the docker compose setup again.
I also added a line to AGENTS.md, so agents should know to do this
automatically in the future.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
- [ ] New Feature (non-breaking change which adds functionality)
- [x] Documentation Update
- [ ] Refactoring
- [ ] Performance Improvement
- [ ] Other (please describe):
Co-authored-by: Zhichang Yu <yuzhichang@gmail.com>
## Summary
Add **per-node thinking mode control** for LLM components in RAGFlow
Agent canvas, supporting Qwen3/Qwen3.6/B200M thinking-capable models.
Users can now independently configure thinking mode
(thinking/non-thinking) for each LLM node via the existing UI dropdown.
## Motivation
When Qwen3.6-27B (and other thinking-capable models like Qwen3-32B,
B200M) are used in RAGFlow Agent nodes, different nodes need different
thinking behavior:
- **Reasoning nodes** (complex analysis, math, coding): thinking mode ON
- **Simple nodes** (direct Q&A, intent classification): thinking mode
OFF
The web UI already has a `thinking` dropdown (default/enabled/disabled)
in LLM settings, and the LLM backend `_apply_model_family_policies()`
already supports `enable_thinking`. **The missing link was `gen_conf()`
not forwarding the parameter.**
This 3-line fix completes the chain.
## Changes
**`agent/component/llm.py`** — `LLMParam.gen_conf()`:
```python
if hasattr(self, "thinking") and self.thinking and self.thinking != "default":
conf["thinking"] = self.thinking
```
## End-to-end flow
```
UI dropdown: default / enabled / disabled
→ DSL: {"thinking": "enabled"}
→ LLMParam.thinking = "enabled"
→ gen_conf() returns {"thinking": "enabled"}
→ _apply_model_family_policies()
→ extra_body {enable_thinking: true}
→ Model API call with thinking ON
```
## Backward compatibility
- **Fully backward compatible** — only 3 lines added, nothing changed
- When `thinking` is "default" or not set, existing behavior is
preserved
- Qwen3 models default to `enable_thinking: false` (non-thinking),
unchanged
## Related issues
- Closes#16321 (thinking content leaks in non-streaming agent API
responses)
- Closes#13957 (how to view model reasoning process in agent API)
## Testing
- Verified in
`test/unit_test/rag/llm/test_chat_model_thinking_policy.py` that
thinking policy already tested for Qwen3 models
- The 3-line change passes through existing tested code path
(`_apply_model_family_policies`)
Co-authored-by: Hermes Agent <hermes-agent@agent.local>
## Summary
Adds ownership/access checks before updating or deleting documents,
setting document metadata, and reading file contents from storage. Also
adds tests for authorized and unauthorized access paths.
## Summary
Fix Elasticsearch-backed skill search by mapping skill search fields to
their indexed token fields.
`name`, `tags`, `description`, and `content` are stored for display but
are not searchable in the skill ES mapping. Search queries now target
`name_tks`, `tags_tks`, `description_tks`, and `content_tks`.
## Testing
- Ran Go unit tests:
```bash
/usr/local/go/bin/go test -count=1 ./internal/engine/elasticsearch
```
- Frontend verification:
1. Open /files/skills.
2. Enter a skill space.
3. Reindex the skill space if existing skills were created before this
fix.
4. Search by skill name or description keyword.
5. Confirm matching skills are returned.
### Summary
This PR fixes two issues that prevented the Agent component's
single-component debug/test run from working under the Go backend:
1. **Dynamic input_form generation**: Some components (e.g. `Agent`) do
not store a static `input_form` in the DSL. The Go handler now falls
back to the runtime component's `GetInputForm()` method, matching
Python's `Canvas.get_component_input_form` behavior. This resolves the
frontend 102 error: `component has no input_form`.
2. **Tenant ID injection for debug**: Single-component debug runs use a
fresh `CanvasState` that previously lacked `tenant_id`.
`AgentComponent.Invoke` resolves LLM credentials via the tenant tables,
so the debug run failed with `api key is required`. The handler now
seeds `state.Sys["tenant_id"]` with the authenticated user's ID,
mirroring Python's `@add_tenant_id_to_kwargs` decorator.
### Changes
- `internal/handler/agent_component.go`:
- Added `componentInputForm` helper that first reads the static
`input_form` and, if missing, instantiates the component and calls
`GetInputForm()`.
- In `DebugComponent`, set `debugState.Sys["tenant_id"] = user.ID`
before invoking the component.
### Summary
The Go backend Agent component was not returning artifacts produced by
the CodeExec tool. While the Python agent collects the "`_ARTIFACTS`"
envelope from tool responses and appends artifact markdown to the final
content, the Go agent only returned the assistant text, so generated
images were missing from the chat output.
### Changes
- Wire `react.WithMessageFuture()` in `runEinoReActAgent` and store the
resulting `MessageFuture` in the invocation context.
- After the ReAct loop finishes, drain the future and extract
``_ARTIFACTS`` entries from every tool response message.
- Support reading the tool payload from both `msg.Content` and
`msg.UserInputMultiContent` to match eino's tool contract.
- De-duplicate artifacts by URL and render images as `!` and other files
as download links.
- Add `agent_artifact_test.go` with a regression test that simulates a
CodeExec-style tool response carrying an image artifact and verifies it
is collected and formatted.
### Verification
- `go test ./internal/agent/component/... -run
TestAgent_ReActAgent_CollectsArtifactsFromCodeExecTool` passes.
- `go test ./internal/agent/component/... -count=1` compiles; the only
failure is an unrelated DNS-pinning timeout test
(`TestInvoke_ProxyDNSPin`).
- `gofmt` clean for modified files.
### Related
Fixes the behavior shown in the screenshot where the Go agent ignored
the CodeExec-generated PNG artifact.
## Problem
`raptor.py` computes `n_neighbors = int((len(embeddings) - 1) ** 0.8)`
and
passes it to `umap.UMAP(...)`. In a dataset-scope RAPTOR build the first
layer's `embeddings` is the entire KB's chunk set, so this is
effectively
unbounded: ~93k chunks → n_neighbors ≈ 9,446.
UMAP's k-NN graph is `N × n_neighbors`; at these values the raw neighbor
arrays alone are ~14 GB (93k × 9446 × 16 B), and the symmetrized fuzzy
simplicial set + spectral init push peak well past 30 GB. The task
executor is OOM-killed inside `fit_transform` before any clustering runs
—
the log shows "Task has been received" with no "Cluster one layer" line
—
after which the unacked task re-queues and OOMs again in a loop.
The line above already flags this: `# Degrade too much ??`.
## Fix
Cap `n_neighbors` at 100. UMAP's neighborhood size has strongly
diminishing returns well below this (default 15; a few dozen already
captures global structure), so the ceiling preserves — likely improves —
cluster quality while bounding memory to O(N). Mirrors the existing
`n_components=min(12, len(embeddings) - 2)` clamp two lines down.
```diff
- n_neighbors = int((len(embeddings) - 1) ** 0.8)
+ n_neighbors = min(int((len(embeddings) - 1) ** 0.8), 100)
```
## Repro
Dataset-scope RAPTOR over a KB with ~90k+ chunks on a box with <~64 GB
available: executor OOM-killed in the first-layer UMAP `fit_transform`.
With the cap, first-layer UMAP peaks in low single-digit GB and the
build
proceeds to completion.
## Scope
Only affects large dataset-scope builds; file-scope RAPTOR already had
n_neighbors well under 100. No behavior change beyond the ceiling.
### Summary
1. Move common functions to format.go
2. modify show name spaces to _
3. move _order _columns column sort group;
4. add dao empty enterprise file
## Summary
While fixing #16467 (IterationItem crash on `@` in user-defined output
keys), an audit of `agent/**/*.py` revealed **three additional sites**
with the same vulnerability. This PR hardens all of them with
`maxsplit=1` and adds regression tests.
This is **defense-in-depth hardening**, not a behavior change. The
current `variable_ref_patt` regex constrains `var_nm` to
`[A-Za-z0-9_.-]+`, so single-`@` templates resolve exactly as before.
The `maxsplit=1` only kicks in if the trailing side itself contains `@`
— currently unreachable from the public DSL surface, but trivially
exploitable the moment a user-defined output key happens to contain `@`
(e.g. `user@email`) or the regex is ever relaxed.
> **Note on issue scope**: The primary fix for #16467 (the
`list_tenant_added_models` `ValueError` crash on `@` in model names) is
in PR #16468. This PR is a **follow-up hardening sweep** of the same
vulnerability class found in `agent/` during that audit; it does not
duplicate or replace #16468.
## Sites hardened
| File | Line | Method |
|------|------|--------|
| `agent/canvas.py` | 206 | `Graph.get_variable_value` |
| `agent/canvas.py` | 256 | `Graph.set_variable_value` |
| `agent/component/base.py` | 533 |
`ComponentBase.get_input_elements_from_text` |
| `agent/component/iterationitem.py` | 88 |
`IterationItem.output_collation` |
All now use `split("@", 1)` with an inline comment explaining the
rationale. The trailing side keeps any embedded `@`.
## Sites already safe (audited but left alone)
| File | Reason safe |
|------|------------|
| `agent/canvas.py:708` (`is_reff`) | Pre-checks `len(arr) != 2` |
| `agent/component/categorize.py` | Uses `rsplit` |
| `agent/component/iteration.py` | Pre-validates via regex |
| Other call sites | `rsplit` or regex pre-validation |
## Regression tests
9 new tests across 2 files, all `pytest.mark.p2`:
| File | Tests |
|------|-------|
| `test/unit_test/agent/test_canvas_at_split.py` | 6 —
`get_variable_value`, `set_variable_value`, round-trip, single-`@`,
missing-component |
| `test/unit_test/agent/component/test_iterationitem_at_split.py` | 3 —
`output_collation` with `@` in var, single-`@`, non-matching cid |
Each test was **verified to fail with `ValueError: too many values to
unpack (expected 2)`** when the corresponding fix is temporarily
reverted, confirming the tests actually catch the bug rather than just
exercising the happy path.
## Test results
```
9 passed in 0.04s
```
Full agent unit suite also clean (38 passed, 3 skipped; 6 unrelated
pre-existing collection errors from missing `peewee`/`requests` in local
venv — not caused by this PR).
## Related
- Issue: #16467
- Primary fix PR: #16468 (closes the issue)
- This PR: defense-in-depth follow-up, intentionally non-blocking on
#16467
---------
Co-authored-by: skbs-eng <skbs-eng@users.noreply.github.com>
### What problem does this PR solve?
Closes#16414.
The **Crawler** agent tool (`agent/tools/crawler.py`) was never ported
to the modern `ToolBase`/`_invoke` interface during the agent module
redesign, so it was broken in three independent ways:
1. **Crashed on construction.** `CrawlerParam` extends `ToolParamBase`,
whose `__init__` reads `self.meta["parameters"]`, but `CrawlerParam`
defined no `meta`. Constructing it raised `AttributeError:
'CrawlerParam' object has no attribute 'meta'`. Because
`agent/canvas.py` instantiates `component_class(component_name +
"Param")()` while loading a canvas, **any agent containing a Crawler
node failed to load.**
2. **`_invoke` missing.** It extends `ToolBase` (whose `invoke()`
dispatches to `self._invoke`) but only implemented the legacy `_run`, so
`_invoke` resolved to `ComponentBase._invoke` → `NotImplementedError`.
3. **`be_output` removed.** `_run` called `Crawler.be_output(...)`,
which no longer exists on the base classes.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
### Changes
- Add a `ToolMeta` to `CrawlerParam` (defined before
`super().__init__()`, matching every other ported tool such as
`ArXivParam`/`TavilyExtractParam`) advertising a required `query`
parameter — the URL to crawl, default `{sys.query}`, consistent with the
`{sys.query}` convention shared by the other tools.
- Replace the legacy `_run`/`be_output` with `_invoke`/`set_output`,
writing the extracted page content to `formalized_content` (errors
surfaced via `_ERROR`), consistent with the other tools.
- Preserve the existing SSRF guard (`assert_url_is_safe` +
`pin_dns_global`).
- Add regression tests
(`test/unit_test/agent/component/test_crawler.py`) covering param
construction, validation, and the tool descriptor.
Same class of defect as #16329 (DeepL). Backend-only; no frontend
changes.
---------
Co-authored-by: Zhichang Yu <yuzhichang@gmail.com>
## Summary
Use `DocumentService.RemoveDocumentKeepFile` when deleting files that
are linked to documents.
## Change
- inject `DocumentService` into `FileService`
- replace direct document deletion in `deleteSingleFile`
- remove the obsolete file-local engine deletion helper
## Result
Deleting a file now cleans up linked documents through the same service
path used elsewhere, keeping KB counters and document engine cleanup
consistent.
### What problem does this PR solve?
Closes#16416.
The **AkShare** agent tool (`agent/tools/akshare.py`) was never ported
to the modern `ToolBase`/`_invoke` interface during the agent module
redesign and was still written against the removed legacy
`_run`/`be_output` API, so it was non-functional:
1. **Adding it to an Agent raised `AttributeError`.** `AkShare` extended
`ComponentBase` (not `ToolBase`) and `AkShareParam` defined no `meta`,
so it had no `get_meta()`. `agent/component/agent_with_tools.py` builds
each tool's function descriptor via `cpn.get_meta()`, so constructing an
Agent that includes the AkShare tool raised `AttributeError: 'AkShare'
object has no attribute 'get_meta'`.
2. **It could never run.** `invoke()` dispatches to `self._invoke`, but
`AkShare` only implemented the legacy `_run`, so `_invoke` fell through
to `ComponentBase._invoke` → `NotImplementedError`. `_run` also called
`be_output(...)`, which no longer exists on the base classes.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
### Changes
- Port `AkShareParam` to `ToolParamBase` with a `ToolMeta` (defined
before `super().__init__()`, matching `ArXivParam`/`TavilyExtractParam`)
exposing a required `query` parameter — the stock symbol to look up,
default `{sys.query}`. `query` matches the `{sys.query}` convention
shared by the other tools.
- Rewrite the component with `_invoke`/`set_output("formalized_content",
...)` (errors surfaced via `_ERROR`), keeping `top_n` and importing
`akshare` lazily.
- Add regression tests
(`test/unit_test/agent/component/test_akshare.py`) covering param
construction, validation, and the tool descriptor.
Same class of defect as #16329 (DeepL) and #16414 (Crawler).
Backend-only; no frontend changes.
---------
Co-authored-by: Zhichang Yu <yuzhichang@gmail.com>
### Summary
Plan to start api_server, admin_server and ingestor in one binary:
- ./ragflow_main --admin
- ./ragflow_main --api
- ./ragflow_main --ingestor
---------
Signed-off-by: Jin Hai <haijin.chn@gmail.com>
## Summary
- Decrement document and knowledgebase chunk counts after chunks are
deleted
- Keep token counts unchanged because deleted chunk token totals are not
available
- Add tests for stats update, zero-delete behavior, error handling, and
transaction rollback
### Summary
1. env 'MINIO_PORT' is used for MINIO external access, which shouldn't
be used in Go config.
2. After RAGFlow 1.0 release, MINIO_PORT will be used for docker compose
internal usage. new ENV MINIO_EXTERNAL_PORT will be used for external
access.
Signed-off-by: Jin Hai <haijin.chn@gmail.com>
### Summary
Per discussion with @yuzhichang , disable agent test firstly.
https://github.com/infiniflow/ragflow/actions/runs/28562749273/job/84704079689?pr=16521
[0.094ms] [rows:0] SELECT * FROM `tenant_model_instance` WHERE
provider_id = "provider-1" AND instance_name = "default" ORDER BY
`tenant_model_instance`.`id` LIMIT 1
--- FAIL: TestInvoke_ProxyDNSPin (2.00s)
invoke_test.go:375: dial error = Invoke: do: Get "http://8.8.8.8/api":
context deadline exceeded; want pinned proxy IP 192.88.99.1:9999
(connection-refused is acceptable; an absent IP means the dialer fell
through to the default resolver and the pinning regression went
undetected)
2026/07/02 14:34:58
/home/infiniflow/runners_work/tower01-9dd627fd9c44/ragflow/ragflow/internal/dao/kb.go:79
record not found
[0.358ms] [rows:0] SELECT * FROM `knowledgebase` WHERE id = "da1" AND
status = "1" ORDER BY `knowledgebase`.`id` LIMIT 1
2026/07/02 14:34:58
/home/infiniflow/runners_work/tower01-9dd627fd9c44/ragflow/ragflow/internal/dao/kb.go:79
record not found
[0.283ms] [rows:0] SELECT * FROM `knowledgebase` WHERE id = "da1" AND
status = "1" ORDER BY `knowledgebase`.`id` LIMIT 1
2026/07/02 14:34:58
/home/infiniflow/runners_work/tower01-9dd627fd9c44/ragflow/ragflow/internal/dao/kb.go:79
record not found
[0.523ms] [rows:0] SELECT * FROM `knowledgebase` WHERE id = "da1" AND
status = "1" ORDER BY `knowledgebase`.`id` LIMIT 1
2026/07/02 14:34:58 ExpectPing will have no effect as monitoring pings
is disabled. Use MonitorPingsOption to enable.
FAIL
FAIL ragflow/internal/agent/component 2.759s
ok ragflow/internal/agent/component/io 0.026s
### What problem does this PR solve?
Part of #15240 (rewriting the RAGFlow API server in Go).
Implements the two public bot endpoints from
`api/apps/restful_apis/bot_api.py`:
- **`GET /api/v1/chatbots/<dialog_id>/info`** (`chatbots_inputs`) —
returns `{title, avatar, prologue, has_tavily_key}` for a dialog the
authenticated tenant owns (tenant match + `status == VALID`), otherwise
`"Authentication error: no access to this chatbot!"`.
- **`GET /api/v1/searchbots/detail`** (`detail_share_embedded`) —
returns search-app detail for a `search_id` the tenant can access.
Permission is checked across the tenant's joined tenants; denial returns
`"Has no permission for this operation."` (operating error, `data:
false`) and a missing app returns `"Can't find this Search App!"`.
Both endpoints authenticate with an SDK **beta token** (`Authorization:
Bearer <beta>`) rather than a session — the token is resolved to a
tenant via `APIToken.query(beta=token)`, backed by a new
`APITokenDAO.GetByBeta`. Because they perform their own token-based
auth, the routes are registered on the unauthenticated route group
(mirroring the Python blueprint, which has no `@login_required`).
Both live in a new `internal/handler/bot.go` + `internal/service/bot.go`
since they share the same source module. Handler unit tests cover the
auth, success, and error-mapping paths.
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
---------
Co-authored-by: Claude Code <claude@anthropic.com>
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-authored-by: Ling Qin <qinling0210@163.com>
## Summary
Fix error messages in `build.sh` and add documentation in
`internal/development.md` for downloading native static libraries
(pdfium, pdf_oxide, office_oxide).
## Changes
- `build.sh`: change error hint from `uv run download_deps.py` to `uv
run ragflow_deps/download_deps.py` (correct path from project root)
- `internal/development.md`: add section 2.1 documenting how to download
native libs and install lld
## Summary
- use the project-standard 32-character ID generator when creating
shared chatbot sessions
- fix MySQL insert failures caused by writing 36-character UUID strings
into `api_4_conversation.id`
### Summary
Adopt sync.WaitGroup.Go (Go 1.25) to simplify tracked goroutine
spawning. This replaces the error-prone trio of wg.Add(1), go func(),
and defer wg.Done() with a single, self-contained call.
More info: https://github.com/golang/go/issues/63796
Signed-off-by: grandpig <grandpig@outlook.com>
### Summary
This PR fixes a Go backend bug where updating agent settings, such as
description, could clear the agent DSL.
Root cause:
PUT /api/v1/agents/:canvas_id only bound the dsl field in Go. When the
frontend submitted settings without dsl, the service still updated the
canvas with an empty DSL value.
Changes:
- Treat agent updates as partial patches.
- Preserve existing DSL when dsl is not present in the request.
- Update only specified user_canvas fields instead of saving the full
row.
- Add a regression test for settings updates preserving DSL.
Test:
`go test ./internal/service ./internal/handler`
Co-authored-by: Zhichang Yu <yuzhichang@gmail.com>
## Summary
- align the Go `/api/v1/thumbnails` endpoint with the frontend request
format for repeated `doc_ids`
- return thumbnail mappings for multiple documents instead of failing on
a single missing document
- preserve Python-compatible thumbnail formatting, including base64
thumbnail passthrough
### Summary
Fixes a bug in the Go chunk list handler where the available` query
parser rejected `false` and `0` even though they were documented as
supported values.`
This caused requests from the "Disabled" chunk filter to return HTTP 400
and broke the chunk list page when filtering disabled chunks.
### Summary
As title
This PR fixes dataset index task creation failing with unsupported data
type: entity.JSONMap when loading document chunking config.
#### issues:
```
2026/06/30 15:19:40 /home/infiniflow/Documents/development/ragflow/internal/dao/document.go:162
[error] unsupported data type: ragflow/internal/entity.JSONMap
```
#### Changes:
+ Adds the missing GORM type:longtext tag to ParserConfig in
DocumentDAO.GetChunkingConfig.
+ Adds a DAO regression test covering GetChunkingConfig joins across
document, knowledgebase, and tenant while scanning parser_config.
### Summary
As title
Before, it return `update success` but never insert or update any
metadata
fixed:
```go
_, err = s.docEngine.InsertMetadata(nil, []map[string]interface{}{
{
"id": docID,
"kb_id": doc.KbID,
"meta_fields": meta,
},
}, tenantID)
```
### What problem does this PR solve?
Closes#12962
MCPToolCallSessions created during agent execution (in `Agent.__init__`)
are never explicitly closed. Each session starts its own event loop
thread and opens an SSE/HTTP connection to the MCP server. When the
canvas goes out of scope, these threads and connections remain alive
indefinitely, accumulating over time and causing resource exhaustion
after prolonged use.
### Solution
1. Add a `Graph.close()` method that iterates all components, finds
MCPToolCallSessions held by Agent tools, and calls `close_sync()` on
each to properly shut down the event loop, thread, and connection.
2. Call `canvas.close()` in `finally` blocks after `canvas.run()`
completes in `canvas_service.py` and `canvas_app.py`.
3. Move MCP session cleanup to `finally` blocks in `test_tool` endpoint
(`mcp_server_app.py`) and `get_mcp_tools` (`api_utils.py`) to ensure
sessions are closed even on exceptions.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
---------
Co-authored-by: conflict-resolver <conflict-resolver@local>
Co-authored-by: Zhichang Yu <yuzhichang@gmail.com>
Replace fragile wall-clock timeout assertions with semantic checks for
deadline errors, retry suppression, and event ordering. Keep only
lower-bound timing checks where they prove backoff behavior. This
reduces CPU-load flakes without weakening regression coverage.
### Summary
This PR fixes incorrect dataset document counters in the Go service.
Several document creation paths inserted document records directly
through documentDAO.Create, bypassing the shared InsertDocument logic
that increments knowledgebase.doc_num. As a result, datasets could
contain documents while doc_num remained 0.
### Summary
```
/api/v1/chats/<chat_id>/sessions/<session_id>/messages/<msg_id> DELETE
/api/v1/chats/<chat_id>/sessions/<session_id>/messages/<msg_id>/feedback PUT
```
Migrates the chat session message delete and feedback APIs to the Go
server, matching the Python behavior for authorization, session
ownership checks, message/reference updates, and feedback validation.
### What problem does this PR solve?
_Briefly describe what this PR aims to solve. Include background context
that will help reviewers understand the purpose of the PR._
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
- [x] New Feature (non-breaking change which adds functionality)
- [x] Other (please describe):
## Summary
Agent (Canvas) runs previously did not surface token usage in the SSE
stream, and RAGFlow's own Langfuse generations for agent runs were
missing the prompt/completion split and the session/user correlation.
This made it impossible for an external caller (or Langfuse) to
reconcile an agent turn's cost with the upstream provider (e.g.
OpenRouter), because a single turn can issue several distinct LLM calls
(query rewriting / cross-language translation, multi-round tool
reasoning, nested sub-agents, and the final answer).
This PR introduces a per-run token usage sink so that **every** LLM call
in a run is aggregated and reported once, and enriches Langfuse
generations with the prompt/completion split plus session/user
attributes.
## What changes
### 1. Per-run token usage sink (`common/token_utils.py`)
- Adds two `contextvars`: `token_usage_sink` (a mutable per-run
accumulator) and `langfuse_run_attrs` (session_id/user_id for the run).
- Adds `record_run_token_usage(...)` (thread-safe via a lock, because
`thread_pool_exec` copies the context into worker threads that share the
sink dict) and `usage_from_response(...)` which extracts a
`{prompt_tokens, completion_tokens, total_tokens}` split from
OpenAI/OpenRouter-style responses.
### 2. Provider layer captures the prompt/completion split
(`rag/llm/chat_model.py`)
- `LiteLLMBase` and `Base` now store `self.last_usage`
(prompt/completion/total) for the most recent chat call, in both the
plain and tool-calling paths.
- Streaming requests set `stream_options.include_usage = True` (LiteLLM
path) so the authoritative usage arrives on the final chunk; this is
read even on the usage-only chunk that carries no `choices`.
- Fixes a multi-round accounting bug in `*_with_tools`: token totals
were **overwritten** by each round (`total_tokens = tol`) instead of
accumulated, undercounting multi-round tool conversations. Each round is
now committed to a running aggregate.
### 3. LLMBundle reports usage once, per call
(`api/db/services/llm_service.py`)
- New `_report_usage(total_tokens)` records the call's usage into the
active run sink and returns the prompt/completion/total split for
Langfuse. The split is only used when it is consistent with the
authoritative total; otherwise only the total is reported.
- All three chat entry points (`async_chat`, `async_chat_streamly`,
`async_chat_streamly_delta`) now emit `usage_details` with
`input`/`output`/`total` instead of total-only.
- `_start_langfuse_observation` now applies `session_id`/`user_id` from
the per-run context (`langfuse_run_attrs`) so agent-run generations are
correctly grouped, even though agent LLMBundles are constructed without
those attributes.
### 4. Canvas installs the sink and emits the aggregate
(`agent/canvas.py`)
- `Canvas.run()` installs a fresh `token_usage_sink` and
`langfuse_run_attrs` (from `user_id`/`session_id`) at the start of every
turn.
- `message_end` now includes an aggregated `usage` object:
`{prompt_tokens, completion_tokens, total_tokens, calls}` covering all
LLM calls in the run.
### 5. Pass session id into the run
(`api/db/services/canvas_service.py`)
- `completion()` forwards `session_id` to `Canvas.run()` for Langfuse
session correlation.
## Why a context variable
LLM calls in an agent run originate from many places that each build
their own `LLMBundle` (e.g. `cross_languages`/`keyword_extraction`
helpers, the Agent component, and nested sub-agents invoked as tools). A
run-scoped context variable is the only non-invasive chokepoint that
captures all of them exactly once, including nested agents (which run in
the same async context) and thread-pool tools (the executor copies the
context).
## Behavior / compatibility
- No public API or wire-format removal: `message_end` gains an
additional optional `usage` field; existing consumers are unaffected.
- When a provider does not return authoritative usage, behavior falls
back to the previous token estimate (total only, no split).
- Non-agent flows (Dataflow `Pipeline`, sync `Graph.run`) are untouched.
## Testing
- [x] Simple agent answer: `message_end.usage.total_tokens` matches
provider usage.
- [x] Agent with cross-language retrieval: aggregate equals the sum of
both provider calls.
- [x] Tool-calling agent (multi-round): total accumulates across rounds.
- [x] Nested agent (agent-as-tool): sub-agent tokens included in the
parent run total.
- [x] Langfuse: agent generations show input/output split and are
grouped by session/user.
---------
Co-authored-by: yzc <yuzhichang@gmail.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
## Problem
When building or updating a knowledge graph with a large number of
entities and edges, `set_graph()` in `rag/graphrag/utils.py` creates one
`asyncio` task per entity and one per edge, each calling
`embd_mdl.encode([single_name])` — a single-item HTTP request to the
embedding server.
For a graph with 17,000+ nodes and edges (real case reported in #16205),
this generates **34,000+ individual embedding API round-trips** instead
of ~266 batched calls at the default `_INSERT_BULK_SIZE=128`. The
asyncio gather over thousands of tasks makes the embedding server the
bottleneck; under load, a single slow/failed call aborts all remaining
tasks, causing the pipeline to stall and never complete.
Closes#16205. Related: #15921.
## Root Cause
```python
# Before (in set_graph, node loop):
tasks = [asyncio.create_task(graph_node_to_chunk(n, ...)) for n in nodes]
# Each task calls embd_mdl.encode([single_name]) — 1 HTTP call per node
```
`graph_node_to_chunk` checks the embed cache first, but the cache is
cold on first build, so every task makes a live API call.
## Fix
Pre-warm the embedding cache with batched calls before spawning tasks.
Each batch pre-warm calls `embd_mdl.encode(batch_of_128)` once,
populating the cache. Then every individual task hits the cache and
makes zero embedding API calls.
- Only encodes names not already in cache (no-op on warm cache / small
incremental updates)
- Uses existing project idioms: `thread_pool_exec`, `chat_limiter`,
`_INSERT_BULK_SIZE`, `get_embed_cache`, `set_embed_cache`
- Mirrors the `ENABLE_TIMEOUT_ASSERTION` timeout pattern from
`graph_node_to_chunk`
- Zero behavior change: per-task encode logic remains as a correct
fallback
## Result
| Graph size | Before | After |
|---|---|---|
| 17,576 edges | ~17,576 embedding calls → stall | ~138 batched calls |
| 17,509 nodes | ~17,509 embedding calls → stall | ~137 batched calls |
| **Total** | **~35,000 calls** | **~275 calls** |
---------
Co-authored-by: Oti_B <oti@mac.speedport.ip>
### Summary
The addition of the Context method to Go's testing.T provides
significant improvements for writing concurrent tests. It allows better
management of goroutines, ensuring they properly exit and preventing
issues like deadlocks and unfinished processes.
By using Context, errors and cancellations can be handled more
effectively, making tests more robust and easier to reason about. This
change also enables tighter integration between tests and the
application code, especially for systems that span multiple concurrent
components. Overall, it simplifies test code and enhances test stability
and maintainability.
More info: [golang/go#18368](https://github.com/golang/go/issues/18368)
Signed-off-by: blackflytech <blackflytech@outlook.com>
### Summary
`truncateText` in the `reduction` and `summarization` middlewares
truncates with `s[:maxLen]`, which slices by byte. When `maxLen` lands
inside a multi-byte character (common with CJK or other non-ASCII
content flowing through the agent), the string is cut mid-rune and the
tail byte(s) become invalid UTF-8. That broken text then goes into the
reduced context / summary prompt.
`TruncateToolResult` in the same `reduction` package already avoids this
by slicing on a rune boundary and even notes it in a comment. This PR
makes the two `truncateText` helpers do the same, so they stay
consistent with the existing helper.
Both functions keep their existing output shape (summarization still
appends `...`). Added a small unit test in each package covering ASCII
truncation and a CJK string, asserting the result stays valid UTF-8.
## Summary
TuShare required non-empty upstream input but filtered fetched news with
the static `keyword` param (default empty string), so agent-provided
keywords were ignored.
Use `self._param.keyword or ans` when filtering, matching how AkShare
uses upstream input for its query.
Fixes#16360
## Test plan
- [x] `test_tushare_filters_with_upstream_keyword_when_param_empty`
mocks the API and asserts only rows matching the upstream keyword are
returned
---------
Co-authored-by: yzc <yuzhichang@gmail.com>
Co-authored-by: Harsh Kashyap <harshkashyap@Harshs-MacBook-Pro.local>
### What problem does this PR solve?
Closes#16418.
`scholarly.search_pubs(...)` returns a **lazy generator**, but
`agent/tools/googlescholar.py` treated it as a re-iterable, bounded
list:
```python
scholar_client = scholarly.search_pubs(kwargs["query"], ...) # lazy generator
self._retrieve_chunks(scholar_client, ...) # (1) iterates -> exhausts it
self.set_output("json", list(scholar_client)) # (2) already empty -> []
```
1. **`json` output was always empty.** `_retrieve_chunks` iterates
`scholar_client`, exhausting the generator; `list(scholar_client)` then
returns `[]`.
2. **`top_n` was never applied.** Unlike `ArXiv`
(`max_results=self._param.top_n`), the unbounded generator was passed
straight to `_retrieve_chunks`, which has no internal limit — so the
tool kept paginating well past Top N (until an error, rate-limit/block,
or `COMPONENT_EXEC_TIMEOUT`).
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
### Changes
- Materialize at most `top_n` results once with `itertools.islice`, and
reuse that list for both `_retrieve_chunks` and the `json` output.
- Add regression tests
(`test/unit_test/agent/component/test_googlescholar.py`, stubbing
`scholarly.search_pubs`) covering the `top_n` bound, the non-empty
`json` output, and the empty-query short-circuit.
Verified: against `main` the new tests fail with `assert 30 == 5` (top_n
ignored) and `assert 0 == 5` (empty json); with this fix all pass.
Backend-only.
---------
Co-authored-by: Zhichang Yu <yuzhichang@gmail.com>
## Summary
Add two new MCP tools to the RAGFlow MCP server:
1. **ragflow_list_datasets** - List all accessible datasets with IDs,
names, descriptions
2. **ragflow_list_chats** - List all accessible chat assistants with
IDs, names, descriptions
### Implementation
- Added `list_chats()` method to `RAGFlowConnector`
- Registered both tools in `list_tools()` and `call_tool()`
- Follows existing `ragflow_retrieval` pattern for error handling
### Usage via langchain-mcp-adapters
---------
Co-authored-by: saltsalt123 <saltsalt123@users.noreply.github.com>
Co-authored-by: yzc <yuzhichang@gmail.com>
### What problem does this PR solve?
Closes https://github.com/infiniflow/ragflow/issues/14571.
Adds CAJAL as a first-class local scientific-writing option in RAGFlow:
- registers `agnuxo/cajal-4b-p2pclaw` as a known Ollama chat model with
a 32K context setting
- adds a built-in “CAJAL scientific paper agent” template under the
existing agent template catalog
- preconfigures the agent for grounded scientific writing: retrieval
first, citation traceability, LaTeX-ready output, and explicit
limitations when evidence is missing
- adds unit coverage to ensure the template normalizes through RAGFlow’s
production template loader, keeps graph form data in sync, and exposes
the Ollama model option
Behavior/evidence gathered for the requested model:
- Hugging Face model metadata for `Agnuxo/CAJAL-4B-P2PCLAW` reports
`pipeline_tag=text-generation` and tags including `gguf`, `llama.cpp`,
`vllm`, `scientific-research`, `papers`, `academic-writing`, `latex`,
and `license:apache-2.0`.
- The model card documents CAJAL as a 4B scientific paper generation
model with 32K context, local inference, LaTeX/citation specialization,
and CPU-only support around 5 tok/s on Ryzen 7 5800X.
- Local CPU generation could not be completed on this machine because
the advertised Ollama model name is not currently resolvable from
Ollama’s registry: both
`https://registry.ollama.ai/v2/agnuxo/cajal-4b-p2pclaw/manifests/latest`
and
`https://registry.ollama.ai/v2/library/agnuxo/cajal-4b-p2pclaw/manifests/latest`
returned `404 Not Found`; the Hugging Face repo tree currently exposes
an 8.4 GB `model.safetensors` but no GGUF artifact in `main`. The
template therefore targets the documented Ollama model name for users
who have the local CAJAL deployment/model file available.
Verification run locally:
```bash
python3 -m pytest test/test_cajal_template_unit.py -q
# 3 passed in 0.34s
python3 - <<'PY'
import json, glob
for f in sorted(glob.glob('agent/templates/*.json') + ['conf/llm_factories.json']):
with open(f, encoding='utf-8') as fp: json.load(fp)
print('json_ok')
PY
# json_ok
python3 -m ruff check test/test_cajal_template_unit.py
# All checks passed!
git diff --check
```
`uv run pytest
test/testcases/test_web_api/test_agent_app/test_cajal_template_unit.py
-q` was also attempted first, but dependency setup failed before test
collection while building `ormsgpack==1.5.0` from uv with a package
metadata parse error. Clearing uv’s `ormsgpack` cache and retrying
reproduced the same build failure, so the focused unit test was run with
the system Python environment instead.
### Type of change
- [ ] Bug Fix (non-breaking change which fixes an issue)
- [x] New Feature (non-breaking change which adds functionality)
- [ ] Documentation Update
- [ ] Refactoring
- [ ] Performance Improvement
- [ ] Other (please describe):
---------
Co-authored-by: sxxtony <sxxtony@users.noreply.github.com>
Co-authored-by: yzc <yzc@users.noreply.github.com>
Co-authored-by: Zhichang Yu <yuzhichang@gmail.com>
…rialization
Agent components (llm.py, agent_with_tools.py, message.py) store
functools.partial objects as deferred streaming handles in their output
slots. When the canvas state gets serialized for SSE events, Redis
commits, or logging, these partials — plus non-copyable objects like
Langfuse clients — crash json.dumps and deepcopy.
Changes:
- canvas_app.py: add default=str to json.dumps for SSE event
serialization (lines 238, 296)
- canvas.py: wrap deepcopy calls in try/except to handle non-copyable
objects (Langfuse clients, etc.), add default=str to final json.dumps
- base.py: add default=str to ComponentParamBase.__str__ to handle
non-serializable objects in component parameters
Closes#14229
### What problem does this PR solve?
_Briefly describe what this PR aims to solve. Include background context
that will help reviewers understand the purpose of the PR._
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
- [ ] New Feature (non-breaking change which adds functionality)
- [ ] Documentation Update
- [ ] Refactoring
- [ ] Performance Improvement
- [ ] Other (please describe):
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-authored-by: yzc <yuzhichang@gmail.com>
## Summary
- Treat `max_tokens=0` as unset (`or 8192`) when building model context
budgets, fixing agents that silently zeroed prompts when a vLLM model
had `max_tokens: 0` in tenant config
- Replace trailing same-role canvas history in `LLM._sys_prompt_and_msg`
instead of skipping the current user prompt
- Add `LLM.fit_messages()` validation after `message_fit_in` on agent
paths so empty user content fails fast with a clear error instead of
reaching vLLM
Fixes#16411
## Root cause
Agent canvas workflow called `message_fit_in` with `int(max_length *
0.97)`. When `max_length` was `0`, both system and user content were
trimmed to empty strings. The `[HISTORY STREAMLY]` log showing only
`{"role":"user","content":""}` matches this. A secondary bug skipped
appending the formatted user prompt when history ended with a `user`
role message.
## Test plan
- [x] Added `test/unit_test/agent/component/test_llm_prompt.py` for
role-replace, validation, and zero-budget fitting
- [x] Added
`test_message_fit_in_zero_budget_preserves_non_empty_messages` in
`test_generator_message_fit_in.py`
- [ ] CI unit tests
- [ ] Manual: agent canvas `begin → Retrieval → Agent → Message` with
vLLM Qwen3; confirm user message reaches LLM
Made with [Cursor](https://cursor.com)
---------
Co-authored-by: Taranum Wasu <taranumwasu@Taranums-MacBook-Pro.local>
Co-authored-by: Cursor <cursoragent@cursor.com>
Recovery PR for #16173 after the fork branch was accidentally reset
during rewrite-cleanup.
Cherry-picked onto current `main`:
- fix(common/time_utils): correct fallback timestamp and ISO-8601
normalization
- fix(common/time_utils): preserve zero timestamps and mark regression
tests
- test(common/time_utils): make fallback assertions deterministic
Supersedes closed#16173 — same branch
`Harsh23Kashyap/fix/time-utils-edgecases`, rebuilt per @yuzhichang
recovery steps in
https://github.com/infiniflow/ragflow/pull/16173#issuecomment-4829663835
---------
Co-authored-by: Harsh Kashyap <harshkashyap@Harshs-MacBook-Pro.local>
Co-authored-by: Cursor <cursoragent@cursor.com>
## Summary
Two changes to make Go build \& run independent of native libraries
(office_oxide, pdfium, pdf_oxide).
## 1. Make native libraries optional (build.sh + Go source)
## 2. Roll back tests.yml CI changes from PR #16391
This PR migrates the Box web OAuth flow from Python to Go for:
- POST /api/v1/connectors/box/oauth/web/start
- GET /api/v1/connectors/box/oauth/web/callback
- POST /api/v1/connectors/box/oauth/web/result
### What problem does this PR solve?
GET /api/v1/agents (list_agents) already supports filtering by
canvas_category, keywords, tags, and owner_ids, but it does not support
canvas_type — even though canvas_type is a persisted field on UserCanvas
and is already accepted on agent create/update APIs.
This gap causes two issues:
Filtering — clients cannot list agents by business category (e.g.
Marketing, Agent, Ingestion Pipeline) without fetching all agents and
filtering client-side.
Response payload — list_agents did not return canvas_type in each canvas
item, so consumers had to call GET /api/v1/agents/{id} per agent to read
it.
This PR adds optional canvas_type query parameter support and includes
canvas_type in the list response.
### Type of change
- [√] Bug Fix (non-breaking change which fixes an issue)
- [ ] New Feature (non-breaking change which adds functionality)
- [ ] Documentation Update
- [ ] Refactoring
- [ ] Performance Improvement
- [ ] Other (please describe):
## Summary
Fixes#15169 — `POST /api/v1/agents/chat/completions` returned
`data: {}` with no `session_id` when the agent produced no events
(e.g. the reporter's payload sent `"query": ""`).
## Root cause
For `{"agent_id": "...", "query": "", "stream": false}`:
1. No `session_id` in the request → new-session branch at
`agent_api.py:1278`.
2. `session_id = get_uuid()` at `agent_api.py:1294`.
3. Falls into `_run_workflow_session`.
4. `canvas.run(query="")` produces no events, so `final_ans`
stays `{}`.
5. Non-streaming path then hit:
```python
if not final_ans:
await commit_runtime_replica()
return get_result(data={})
```
`session_id` was allocated but silently dropped on the way out.
The streaming path had the same shape (only a bare `[DONE]` was
yielded — no SSE event carrying `session_id`). The
session-continuation path at `agent_api.py:1463` had the same bug
for callers that passed `session_id` and got `{}` back.
The successful (non-empty) paths were fine because every canvas
event has `ans["session_id"] = session_id` attached before being
yielded / captured into `final_ans` (see
`agent_api.py:255` and `:303`).
## Fix
Three minimal changes, all in
`api/apps/restful_apis/agent_api.py`:
1. **`_run_workflow_session` (non-streaming)**:
`return get_result(data={"session_id": session_id})` instead of
`data={}`.
2. **`_run_workflow_session` (SSE)**: if the canvas loop emits no
events, yield one
`data:{"session_id": "...", "data": {}}` event before
`[DONE]`, so the client receives the id over the wire.
3. **`agent_chat_completion` session-continuation**: echo the
caller-supplied `session_id` back in the empty-events case
instead of `{}`.
No change needed on the happy paths — they already attach
`session_id` to every event.
## Test plan
- [ ] Repro from the issue: `POST /api/v1/agents/chat/completions`
with `{"agent_id": "<id>", "query": "", "stream": false}`.
Response `data` should now contain `session_id`.
- [ ] Same payload with `"stream": true`. SSE stream should
contain one event with `session_id` before `data:[DONE]`.
- [ ] Same shape but with a real, non-empty `"query"` (new
session). Response should be unchanged from before — every
event still carries `session_id`, final response still
includes it on `final_ans`.
- [ ] Pass an existing `session_id` plus `"query": ""`. Response
should echo that `session_id` back instead of `{}`.
- [ ] Pass an existing `session_id` plus a normal query. Response
should be unchanged from before.
- [ ] `openai-compatible: true` path is untouched — sanity-check
it still works.
- [ ] Run `uv run pytest` to make sure no existing tests regress.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
- [ ] New Feature (non-breaking change which adds functionality)
- [ ] Documentation Update
- [ ] Refactoring
- [ ] Performance Improvement
- [ ] Other (please describe):
## Summary
Fixes#15215 — attachments uploaded to an agent were not reaching the
LLM.
When a user uploads a file in an agent chat, `canvas.run` parses it into
the `sys.files` global (text content for documents, `data:image/...`
URIs
for images — see `agent/canvas.py:752-768`). But the LLM/Agent
component's
`_prepare_prompt_variables` only substitutes variables the user's prompt
template explicitly references via `{var}` placeholders. The default
prompt is `[{"role": "user", "content": "{sys.query}"}]` with no
`{sys.files}`, so the parsed attachment content never reaches the model.
In the reporter's logs, this is why the agent saw only the bare query
`附件 摘要 attachment summary` and went searching the dataset instead of
reading the uploaded PDF.
## Fix
`agent/component/llm.py` — added `_collect_sys_files()` and an
auto-injection step in `_prepare_prompt_variables`:
- If `sys.files` is non-empty **and** neither `sys_prompt` nor any entry
in `prompts` already contains `{sys.files}` (no double-injection),
split the entries into text vs. `data:image/...` URIs.
- Image URIs are merged into `self.imgs`, which the existing logic uses
to switch the chat model to `IMAGE2TEXT` and pass `images=...` to
`async_chat`.
- Text content is appended to the last `user` role message in `msg`,
mirroring how `dialog_service.async_chat_solo` handles attachments for
the non-agent chat path (`api/db/services/dialog_service.py:318-321`).
Both `LLM._invoke_async` and `Agent._invoke_async` (tool-using) go
through `_prepare_prompt_variables`, so plain LLM nodes and Agent nodes
are fixed in both streaming and non-streaming paths.
## Test plan
- [ ] Upload a PDF attachment to an agent with the default `{sys.query}`
prompt and ask "summarize the attachment" — the model should answer
from the file content rather than searching the knowledge base.
- [ ] Upload an image attachment to an agent and ask about its contents
—
the model should switch to the vision-capable LLM and answer from
the image.
- [ ] Verify that an agent whose prompt **does** include `{sys.files}`
still works and does **not** include the file content twice.
- [ ] Verify that an agent run with no attachments behaves unchanged.
- [ ] Run `uv run pytest` to make sure no existing tests regress.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
- [ ] New Feature (non-breaking change which adds functionality)
- [ ] Documentation Update
- [ ] Refactoring
- [ ] Performance Improvement
- [ ] Other (please describe):
---------
Co-authored-by: yzc <yuzhichang@gmail.com>
### Summary
```
RAGFlow(admin)> show users plan summary;
+---------+----------------------------------------------------------------+
| field | value |
+---------+----------------------------------------------------------------+
| command | show_users_plan_summary |
| error | 'Show users plan summary' is implemented in enterprise edition |
+---------+----------------------------------------------------------------+
```
Signed-off-by: Jin Hai <haijin.chn@gmail.com>
### What problem does this PR solve?
This PR adds Google BigQuery as a first-class data source connector in
RAGFlow.
It enables users to ingest and sync BigQuery data using the same
row-to-document model used by relational database connectors: selected
content columns become document text, metadata columns become document
metadata, an optional ID column provides stable document IDs, and an
optional timestamp column enables cursor-based incremental sync.
The connector supports service-account JSON credentials, table mode,
custom query mode, GoogleSQL queries, cursor-based incremental sync,
deleted-row pruning support, configurable query limits such as
`maximum_bytes_billed`, dry-run validation, batch loading, stable
document IDs, and BigQuery-aware value serialization.
### Summary
As title
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
- [x] Bug Fix (non-breaking change which fixes an issue)
### Summary
```
/api/v1/chats/<chat_id>/sessions/<session_id>/messages/<msg_id> DELETE
/api/v1/chats/<chat_id>/sessions/<session_id>/messages/<msg_id>/feedback PUT
```
Migrates the chat session message delete and feedback APIs to the Go
server, matching the Python behavior for authorization, session
ownership checks, message/reference updates, and feedback validation.
### Testing
- `/usr/local/go/bin/go test ./internal/service ./internal/handler`
- Verified through the frontend page for deleting chat messages and
updating message feedback
Fix the reference index used when deleting a chat message pair.
Each user/assistant message pair shares one reference entry, while the
first assistant prologue has no reference. Using `i // 2` correctly
removes the reference for the deleted pair and avoids deleting the
previous turn's reference.
### What problem does this PR solve?
As title
main fix:
```go
if _, ok := req["meta_data_filter"]; !ok || req["meta_data_filter"] == nil {
req["meta_data_filter"] = map[string]interface{}{}
}
```
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
- [x] Refactoring
### What problem does this PR solve?
Package refactor and PDF post process.
### Type of change
- [x] Refactoring
---------
Co-authored-by: Claude <noreply@anthropic.com>
### Summary
fix: user-setting modal fixes and DOMPurify cleanup
- HighlightMarkdown: drop post-process DOMPurify pass (ineffective after
preprocessLaTeX; Coderabbit CRITICAL
#3486038798)
- SettingTeam: add invite-only-registered-users hint to add-user modal
- SettingModel: reset provider loading state when add-provider modal
closes
- MCP edit dialog: set maskClosable=false to prevent accidental
dismissal
- Form: switch FormDescription color from text-muted-foreground to
text-text-disabled
## What problem does this PR solve?
DOCX parsing could crash when a paragraph used a `Heading`-prefixed
style without a trailing numeric level, such as `Heading`, `Heading1`,
or `Heading Title`.
`docx_question_level()` assumed every heading style looked like `Heading
N` and called `int(p.style.name.split(" ")[-1])`. For non-numbered
heading styles, that raises `ValueError` and breaks Manual, Q&A, and
Laws chunking.
This PR parses heading levels safely and falls back to level 1 for
Heading-prefixed styles without an explicit numeric suffix.
Closes#16163.
## Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
- [x] Test Case (non-breaking change which adds test coverage)
### What problem does this PR solve?
1. Add CREATE and DROP DATASET / MEMORY / AGENT / SEARCH / CHAT.
2. Add option to build.sh to strip RAGFlow binary.
### Type of change
- [x] Refactoring
---------
Signed-off-by: Jin Hai <haijin.chn@gmail.com>
Summary
Add module-level debug logging to track Agent canvas execution flow
(Closes#9306), enabling developers to diagnose component invocation,
input/output states, and variable resolution without modifying
production code.
Also fix related bugs in message.py: re.sub backreference issue and
unawaited _save_to_memory coroutine causing silent memory save failures.
Changes
agent/canvas.py: log workflow start, component invocation, and component
completion
agent/component/agent_with_tools.py: log Agent parameter resolution and
LLM invocation path; standardize json.dumps usage
agent/component/base.py: log get_input() variable resolution branches
agent/component/message.py: fix re.sub backreference issue; properly
await _save_to_memory coroutine
Design
Uses module-level loggers (logging.getLogger(__name__)) to support
selective debugging: LOG_LEVELS=agent=DEBUG
Zero performance impact in production (INFO level by default)
Works with existing PUT /system/config/log API for runtime level changes
Closes#9306
Note: While adding debug logging, I discovered and fixed two related
bugs in message.py:
- re.sub replacement value was interpreted as regex backreference
instead of literal string
- _save_to_memory coroutine was not properly awaited, causing silent
failures
---------
Co-authored-by: wills <willsgao@163.com>
closes#14769
### What problem does this PR solve?
_Briefly describe what this PR aims to solve. Include background context
that will help reviewers understand the purpose of the PR._
### Type of change
- [ ] Bug Fix (non-breaking change which fixes an issue)
- [x] New Feature (non-breaking change which adds functionality)
- [ ] Documentation Update
- [ ] Refactoring
- [ ] Performance Improvement
- [ ] Other (please describe):
---------
Co-authored-by: Zhichang Yu <yuzhichang@gmail.com>
### What problem does this PR solve?
Fixes a workflow editor bug where deleting an Iteration Box could leave
orphan child nodes and dangling edges in client state. Those stale
references could be exported with the workflow and later cause rendering
errors, broken connections, and unstable editing behavior.
### Root Cause
Iteration deletion logic only removed the container, its direct
children, and some internal edges. It did not consistently remove the
full descendant subtree or all edges connected to deleted child nodes,
and the keyboard delete path was not expanded to include Iteration
descendants.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
- [ ] New Feature (non-breaking change which adds functionality)
- [ ] Documentation Update
- [ ] Refactoring
- [ ] Performance Improvement
- [ ] Other (please describe):
---------
Co-authored-by: Zhichang Yu <yuzhichang@gmail.com>
## Related issues
Closes#15144
### What problem does this PR solve?
`POST /api/v1/agents/rerun` loaded a pipeline operation log by UUID via
`PipelineOperationLogService.get_documents_info` with no authorization,
then wiped chunks, reset document counters, deleted tasks, and re-queued
dataflow for the victim document.
Any authenticated user who knew a victim's pipeline log id could disrupt
parsing on documents they did not own.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
- [ ] New Feature (non-breaking change which adds functionality)
- [ ] Documentation Update
- [ ] Refactoring
- [ ] Performance Improvement
- [ ] Other (please describe):
### Changes
| File | Change |
|------|--------|
| `api/apps/restful_apis/agent_api.py` | Call
`DocumentService.accessible(doc["id"], tenant_id)` before destructive
rerun operations; deny with generic `"Document not found."` |
|
`test/unit_test/api/apps/restful_apis/test_rerun_agent_authorization.py`
| Unit tests: cross-tenant log rejected, missing/unauthorized same
message, authorized rerun proceeds |
### Security notes
- **CWE-639:** Closes cross-tenant pipeline rerun / chunk wipe via
leaked log UUID.
- `tenant_id` from `@add_tenant_id_to_kwargs` is `current_user.id`;
`DocumentService.accessible` covers team-shared KBs.
### Test plan
- [ ] `pytest
test/unit_test/api/apps/restful_apis/test_rerun_agent_authorization.py`
- [ ] Manual: attacker cannot rerun victim pipeline log id
```bash
cd ragflow
uv run pytest test/unit_test/api/apps/restful_apis/test_rerun_agent_authorization.py -q
```
---------
Co-authored-by: Zhichang Yu <yuzhichang@gmail.com>
## Summary
- **Backend**: `_iter_session_completion_events` in `agent_api.py` was
filtering out `user_inputs` and `workflow_finished` SSE events, causing
agents with UserFillUp components to silently fail in explore mode — the
interactive form never appeared, while the same agent worked correctly
in run (editor) mode.
- **Frontend**: `SessionChat` component in explore mode was missing
`DebugContent` children rendering inside `MessageItem`, so even if the
backend forwarded the events, the form UI would not render. Added
`DebugContent`, `MarkdownContent`, `useAwaitCompentData` hook, and
input-disabling logic to match the run mode's `chat/box.tsx` behavior.
## What was changed
### Backend (`api/apps/restful_apis/agent_api.py`)
- Line 266: Added `"user_inputs"` and `"workflow_finished"` to the
allowed event filter in `_iter_session_completion_events`
### Frontend (`web/src/pages/agent/explore/components/session-chat.tsx`)
- Added imports: `DebugContent`, `MarkdownContent`,
`useAwaitCompentData`, `useParams`
- Added `sendFormMessage` from `useSendSessionMessage()` hook
- Added `useAwaitCompentData` hook for form state management
- Added `DebugContent` as `MessageItem` children for the latest
assistant message (renders UserFillUp form)
- Added `MarkdownContent` + submitted values display for previous
assistant messages
- Updated `NextMessageInput` disabled states to respect `isWaitting`
(form submission in progress)
## Test plan
- [x] Agent with UserFillUp component (e.g., email draft with
send/edit/cancel options) shows interactive form in **explore mode**
- [x] Same agent continues to work correctly in **run (editor) mode**
- [x] Form submission sends data back to the agent and workflow
continues
- [x] Input field is disabled while waiting for form submission
- [ ] Agents without UserFillUp components are unaffected in explore
mode
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Co-authored-by: Zhichang Yu <yuzhichang@gmail.com>
Fixes#16168
## Summary
- Add session-scoped authorization for `GET
/api/v1/documents/artifact/<filename>`
- Allow download only when the artifact filename appears in the caller's
`api_4_conversation` message and
`UserCanvasService.accessible(dialog_id, user_id)` passes
- Deny with generic `"Artifact not found."` before storage access (no
cross-user enumeration)
- Return 4xx when the blob is missing (existing behavior preserved)
## Approach
Sandbox artifacts are runtime CodeExec outputs, not KB documents — this
uses the same session gate pattern as `agent_chat_completion`, not
`DocumentService.accessible`.
## Test plan
- [x] Unit: denied when filename not referenced in user sessions
- [x] Unit: denied when agent canvas is not accessible
- [x] Unit: authorized user receives bytes; missing blob returns
`"Artifact not found."`
- [ ] `pytest
test/testcases/test_web_api/test_document_app/test_document_metadata.py
-k get_artifact`
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Zhichang Yu <yuzhichang@gmail.com>
## Summary
Closes#15425. The agent **Invoke** (HTTP Request) component now calls
`assert_url_is_safe` and `pin_dns` before `requests.*`, matching Crawler
and SearXNG.
## Changes
- `agent/component/invoke.py`: SSRF guard + DNS pinning on outbound
requests.
- `test_invoke_component_unit.py`: unit test blocks loopback URL without
calling `requests.get`.
## Test plan
- [x] `pytest
test/testcases/test_web_api/test_canvas_app/test_invoke_component_unit.py::test_invoke_blocks_loopback_url_with_ssrf_guard`
(requires project test env / `ZHIPU_AI_API_KEY` in CI)
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Zhichang Yu <yuzhichang@gmail.com>
### What problem does this PR solve?
Fixes#14882
Agent webhook execution currently fails open when the saved webhook
`security` block is missing/empty, or when `auth_type` is set to `none`.
This allows unauthenticated webhook invocation without an explicit
operator opt-in.
This PR makes anonymous webhook access explicit:
- Rejects missing or empty webhook security config.
- Requires `allow_anonymous: true` when `auth_type` is `none`.
- Preserves explicit anonymous webhooks by having the frontend serialize
`allow_anonymous: true` when the user selects `None` auth.
- Updates webhook unit tests to cover both denied implicit-anonymous
configs and allowed explicit-anonymous configs.
### Type of change
- [x] Bug Fix
- [x] Security hardening
- [x] Test
### Tests
- [x] `ZHIPU_AI_API_KEY=dummy uv run python -m pytest
--confcutdir=test/testcases/test_web_api/test_agent_app
test/testcases/test_web_api/test_agent_app/test_agents_webhook_unit.py`
- [x] `uv run ruff check api/apps/restful_apis/agent_api.py
test/testcases/test_web_api/test_agent_app/test_agents_webhook_unit.py`
- [x] `npm exec eslint src/pages/agent/utils.ts
src/pages/agent/form/begin-form/schema.ts`
---------
Co-authored-by: Zhichang Yu <yuzhichang@gmail.com>
### What problem does this PR solve?
Fixes#15456.
The SDK agent-bot routes `POST /api/v1/agentbots/<agent_id>/completions`
and `GET /api/v1/agentbots/<agent_id>/inputs`
(`api/apps/restful_apis/bot_api.py`) authenticate the caller with a beta
API token — which only yields the caller's `tenant_id` — but then load
and run the agent named in the URL **without verifying the agent belongs
to the caller's tenant**. `UserCanvasService.get_agent_dsl_with_release`
even accepts a `tenant_id` it never uses, and `begin_inputs` calls
`get_by_id` directly. Any holder of a single valid beta token could
therefore run another tenant's agent (leaking its DSL/prompts/tool
config) or read another tenant's agent metadata and begin input form,
just by substituting a victim `agent_id`.
This PR adds the project's existing ownership gate,
`UserCanvasService.accessible(agent_id, tenant_id)`, to both endpoints
right after token authentication — mirroring the checks already enforced
on the equivalent first-party routes in
`api/apps/restful_apis/agent_api.py` (lines 75/578/775) and on the
sibling `chatbot_completions` / `create_agent_session` /
`delete_agent_session` handlers in the same file. On failure it returns
the same `Can't find agent by ID: <id>` message already used by
`begin_inputs`, so it does not reveal whether an `agent_id` exists in
another tenant.
Added a regression test
(`test/unit_test/api/apps/restful_apis/test_agentbots_access_control.py`,
following the existing stubbed-loader pattern from
`test_get_agent_session.py`) asserting that an inaccessible `agent_id`
is rejected before the agent is loaded (`begin_inputs`) or executed
(`completions`), and that an accessible agent still proceeds.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
- [ ] New Feature (non-breaking change which adds functionality)
- [ ] Documentation Update
- [ ] Refactoring
- [ ] Performance Improvement
- [ ] Other (please describe):
---------
Co-authored-by: Zhichang Yu <yuzhichang@gmail.com>
## Summary
Fixes#14985 — clicking the **Thinking** button in a shared/embedded
chat returns 401 and bounces the user to the login page, even though
the same share page can chat with the agent just fine.
## Root cause
In shared chat, `useGetSharedChatSearchParams` binds `conversationId`
to the URL's `shared_id` query param — which is the **beta APIToken**,
not the real agent id. That `conversationId` propagates through the
component tree:
```tsx
<WorkFlowTimeline canvasId={conversationId}>
→ useFetchMessageTrace(canvasId)
→ GET /api/v1/agents/<sharedId>/logs/<messageId>
```
But `/agents/<agent_id>/logs/<message_id>` is decorated with
`@login_required` (`api/apps/restful_apis/agent_api.py:842-846`).
The share page only holds the beta token — there is no session JWT
— so the request 401s and quart-auth redirects to the login page.
The reporter's server log matches exactly:
```
load_user from jwt got exception No b'.' found in value
load_user: No APIToken found for token=ULG10SWG3E...
Unauthorized request (quart_auth)
GET /api/v1/agents/394013f8d42211f0bad6123fa55e8ed9/logs/96fd72e2-... 1.1 401
```
The `394013f8...` segment in the URL is the `shared_id` (beta
token), not an actual agent id. `_load_user` already accepts the
regular `APIToken.token` field, but not `APIToken.beta`, by design
— beta is a much weaker share-link credential than a personal API
key.
The sibling endpoints `/agentbots/<id>/completions` and
`/agentbots/<id>/inputs` already use the right auth pattern for
this scope (beta-token via `_get_sdk_authorization_token` →
`APIToken.query(beta=token)`). Trace just didn't have a parallel.
## Fix
### Backend (`api/apps/restful_apis/bot_api.py`)
Added a beta-token sibling endpoint:
```
GET /api/v1/agentbots/<shared_id>/logs/<message_id>
```
- Same auth shape as the existing `agentbots` endpoints.
- The `<shared_id>` path segment is a client-supplied label only.
The real `agent_id` used to build the Redis key
(`<agent_id>-<message_id>-logs`) is taken from
`APIToken.dialog_id` on the looked-up token, so the endpoint
never trusts client-supplied identifiers for the data lookup.
- Returns the same `{data: ...}` shape as the existing
`/agents/<id>/logs/<message_id>` endpoint, so the frontend
doesn't need to reshape the response.
### Frontend
- `web/src/utils/api.ts`: added `sharedTrace(sharedId, messageId)`
URL builder.
- `web/src/services/agent-service.ts`: added
`fetchSharedTrace({ shared_id, message_id })`.
- `web/src/hooks/use-agent-request.ts`: `useFetchMessageTrace`
takes an optional `isShare` argument. When set, it calls
`fetchSharedTrace`; `isShare` is also folded into the
`queryKey` so the two modes never share cached results.
- `web/src/pages/agent/log-sheet/workflow-timeline.tsx`:
forwards the already-existing `isShare` prop into the hook.
All other existing call sites of `useFetchMessageTrace` (webhook
timeline, pipeline log, dataflow result) pass no `isShare`
argument → undefined → falsy → unchanged behavior.
## Test plan
- [ ] In the regular Agent UI (logged-in user): open the trace /
log sheet for any message and click into "Thinking" — the
timeline should still load via `/agents/<id>/logs/<msg>`,
same as before.
- [ ] From the Agent page, click **Chat in new tab** to open
`/chat/share?shared_id=<token>&from=agent`. Send a message,
wait for a response, then click **Thinking** on the
assistant turn. The trace panel should load instead of
redirecting to the login page.
- [ ] Same flow but with the agent embedded in an iframe ("Embed
into webpage") — confirm there is no login redirect.
- [ ] In DevTools → Network, confirm the share-chat trace request
goes to `/api/v1/agentbots/<sharedId>/logs/<msgId>` and
returns 200 with the same JSON shape as the logged-in path.
- [ ] Confirm the chat completions, inputs, and upload flows in
the share page still work — they were not touched.
- [ ] Send a bogus / expired beta token to the new endpoint and
confirm it returns the standard "Authentication error: API
key is invalid!" response (no traceback, no 500).
- [ ] Run `uv run pytest` to make sure no existing tests regress.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
- [ ] New Feature (non-breaking change which adds functionality)
- [ ] Documentation Update
- [ ] Refactoring
- [ ] Performance Improvement
- [ ] Other (please describe):
---------
Co-authored-by: Zhichang Yu <yuzhichang@gmail.com>
## Related issues
Closes#15128
### What problem does this PR solve?
`GET` and `DELETE` `/api/v1/agents/<agent_id>/sessions/<session_id>`
verified canvas access for `agent_id` in the URL but loaded/deleted
sessions only by `session_id`, without checking `conv.dialog_id ==
agent_id`.
Any user with access to **any** agent could read or delete another
agent's `API4Conversation` session (messages, references, DSL, etc.)
when they knew the session UUID.
Agent completions in the same file already enforce this binding; chat
sessions do too — these two routes were inconsistent.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
- [ ] New Feature (non-breaking change which adds functionality)
- [ ] Documentation Update
- [ ] Refactoring
- [ ] Performance Improvement
- [ ] Other (please describe):
### Changes
| File | Change |
|------|--------|
| `api/apps/restful_apis/agent_api.py` | Require `conv.dialog_id ==
agent_id` in `get_agent_session` and `delete_agent_session_item`; return
generic `"Session not found!"` on mismatch |
| `test/unit_test/api/apps/restful_apis/test_get_agent_session.py` | Add
IDOR regression tests for GET/DELETE; fix success fixture to include
`dialog_id`; track `delete_by_id` calls |
### Test plan
- [x] Unit tests added for GET/DELETE IDOR and success paths
- [ ] `pytest
test/unit_test/api/apps/restful_apis/test_get_agent_session.py`
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Zhichang Yu <yuzhichang@gmail.com>
## Summary
Fixes the agent `Switch` component matching an **empty/all-skipped
condition** unconditionally because `all([]) is True`.
## Root cause
`res` only accumulates for items with a non-empty `cpn_id` (blank ones
`continue`). For a condition with empty `items` (or all-blank `cpn_id`),
`res == []`, and `if all(res):` is `True`, so the Switch routes to that
condition's `to` target before reaching the else/`end_cpn_ids` branch.
## Fix
```diff
- if all(res):
+ if res and all(res):
```
An empty result set no longer counts as a match; genuinely-satisfied
"and" conditions still route (the real `all(res)` path is preserved).
## Files changed
- `agent/component/switch.py`
- `test/unit_test/agent/component/test_switch_empty_condition.py` (new)
## Verification
- `ruff check` / `ruff format --check` — clean
- Added unit tests (mirroring the existing `_FakeCanvas` component-test
pattern): an empty/all-skipped "and" condition now falls through to
`end_cpn_ids`; a genuinely-satisfied "and" condition still routes to its
target.
- Local full pytest not run (heavy RAG deps); CI validates.
## Note
Implemented with LLM assistance (model: claude-opus-4-8).
Closes#15643
---------
Co-authored-by: seekmistar01 <seekmistar01@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Co-authored-by: Zhichang Yu <yuzhichang@gmail.com>
### What problem does this PR solve?
Closes#15608.
The ExeSQL agent tool (`agent/tools/exesql.py`) opens database
connections to a node-author-controlled host/port with no SSRF
validation. The sibling `test_db_connection` endpoint already validates
the host via `common.ssrf_guard.assert_host_is_safe` (added by PR
#14860), but the tool that actually performs the connection at agent run
time was left unguarded — so the guard is bypassed simply by running the
agent. An agent author can point the host at `127.0.0.1`,
`169.254.169.254` (cloud metadata), or any internal RFC1918 host/port,
turning ExeSQL into an internal port-scanner / metadata-fetch primitive.
### Fix
Mirror the accepted endpoint guard: validate (and resolve) the host
once, before the `db_type` dispatch, and connect to the validated public
IP so a later DNS change cannot rebind the host to an internal address.
- Add `from common.ssrf_guard import assert_host_is_safe`.
- `safe_host = assert_host_is_safe(self._param.host)` before the
dispatch (rejects loopback, link-local/metadata, RFC1918, and
unresolvable hosts).
- Substitute the validated IP into all 6 driver branches: mysql/mariadb,
oceanbase, postgres, mssql, trino, IBM DB2.
Adds `test/unit_test/agent/tools/test_exesql_ssrf.py` covering loopback,
link-local/metadata, RFC1918, and empty-host rejection (before any
connection), plus an allowed host dialing the validated IP.
### Validation
- `python3 -m py_compile agent/tools/exesql.py`
- `ruff check agent/tools/exesql.py
test/unit_test/agent/tools/test_exesql_ssrf.py`
- `pytest test/unit_test/agent/tools/test_exesql_ssrf.py` — 5 passed
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
---------
Co-authored-by: Zhichang Yu <yuzhichang@gmail.com>
### What problem does this PR solve?
This PR adds an Agent LLM setting to control thinking mode for official
providers that expose a thinking switch.
Related to #12842.
Closes#15445.
Some providers expose thinking controls through provider-specific
request fields, but Agent LLM settings did not have a unified option for
users to enable or disable thinking mode.
This PR adds a `Thinking` selector with:
- System default
- Enabled
- Disabled
<img width="452" height="278" alt="8566b0b4-0546-4c8a-913d-f9bbd38319f6"
src="https://github.com/user-attachments/assets/25b497f7-1ba0-4bfe-940d-6fe79287d6ab"
/>
<img width="471" height="971" alt="8a0a6bee-f45f-48d5-bd83-17af260de3db"
src="https://github.com/user-attachments/assets/41ad43c1-5087-48f1-bf37-f2ca14c2be2f"
/>
Initial support is limited to the verified official providers:
- Qwen / DashScope: `enable_thinking`
- Kimi / Moonshot: `thinking.type`
- GLM / ZHIPU-AI: `thinking.type`
For LiteLLM-based providers, provider-specific fields are forwarded
through `extra_body` before `drop_params` filtering so the request
parameters are preserved.
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
---------
Co-authored-by: jiashi <jiashi19@outlook.com>
Co-authored-by: Zhichang Yu <yuzhichang@gmail.com>
## Summary
#16332 fixed the missing `return` in DeepL's except branch, but
`ComponentBase.be_output` was removed during the agent refactor (#9113)
while several components still call it. DeepL (and other tools) would
raise `AttributeError` before any error message could be returned.
- Restore `ComponentBase.be_output` as `pd.DataFrame([{"content": v}])`
(same as pre-refactor behavior)
- Add regression test that `_run` returns the `**Error**:` message when
translation fails
Related to #16329
## Test plan
- [x] `test_run_returns_error_on_translation_failure`
- [x] Existing `test_deepl.py` check() tests still pass
---------
Co-authored-by: Harsh Kashyap <harshkashyap@Harshs-MacBook-Pro.local>
Co-authored-by: Zhichang Yu <yuzhichang@gmail.com>
### What problem does this PR solve?
Closes#15435
Several agent tools call external HTTP APIs through `requests` with no
request timeout. When an upstream host accepts the connection but never
responds (a slow or overloaded API, a half open connection, a stuck load
balancer), the call blocks forever. These tools run inside agent canvas
execution, so a single stalled socket freezes the entire agent run with
no recovery.
Ten call sites were affected:
- `agent/tools/qweather.py` (4 calls)
- `agent/tools/jin10.py` (4 calls)
- `agent/tools/tushare.py` (1 call)
- `agent/tools/github.py` (1 call)
The `github.py` tool already carried the `@timeout` decorator from
`common/connection_utils.py`, but that does not protect against this
case. In the default configuration the decorator waits on its result
queue with no timeout, and a daemon thread blocked inside a socket read
cannot be killed, so the run still hangs. The per request timeout added
here is what actually bounds the call.
This is the same bug class as the merged Go stream timeout fix,
surfacing in the Python tool layer.
Changes:
- Pass `timeout=DEFAULT_TIMEOUT` on all 10 calls, reusing the existing
shared constant in `common/http_client.py` (configurable via
`HTTP_CLIENT_TIMEOUT`) so there is one source of truth rather than
scattered literals.
- Add an AST based unit test at
`test/unit_test/agent/tools/test_http_timeout.py` that scans every tool
module and fails if any `requests` or `httpx` request call omits a
`timeout`, guarding current and future call sites.
Verification:
- Reproduced the indefinite block against a stalling local server, and
confirmed that adding a timeout raises `ReadTimeout` promptly.
- Confirmed the `@timeout` decorator does not interrupt a blocked no
timeout request in its default configuration.
- The new test flags exactly the 10 original call sites on the pre fix
code and passes (22 modules) after the fix.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
- [ ] New Feature (non-breaking change which adds functionality)
- [ ] Documentation Update
- [ ] Refactoring
- [ ] Performance Improvement
- [ ] Other (please describe):
---------
Co-authored-by: Zhichang Yu <yuzhichang@gmail.com>
### What problem does this PR solve?
Closes#14773.
Today, Pipeline (`rag/flow/`) chunking strategies only run as part of a
dataset ingestion that always embeds and indexes the result. There is no
way to drive Pipeline-style chunking from an Agent workflow without
paying that vectorization/persistence cost.
This PR adds a single new Agent component, `PipelineChunker`, that:
- Takes one or more file references (from `Begin` / `UserFillUp`
uploads) as input.
- Runs the existing `rag.app.*` chunking strategies (`naive`, `paper`,
`qa`, `manual`, `book`, `presentation`, `laws`, `table`, `one`, `email`,
`picture`, `audio`, `resume`, `tag`) against each file.
- Emits the resulting chunks as `chunks: list[str]` and `chunks_full:
list[dict]` for downstream Agent nodes.
- Performs **no embedding and no persistence** — chunks live only in
canvas variables for the duration of the run, exactly as requested in
the issue.
The component is auto-discovered by `agent/component/__init__.py`; no
registry edits required. Chunker functions are imported lazily so the
component itself does not pull `deepdoc` / OCR / VLM at
component-discovery time. File resolution mirrors the existing
`ExcelProcessor` convention.
Out of scope for this PR (potential follow-ups):
- Vectorization / KB persistence (explicit ask in the issue).
- Frontend canvas UI for the new component.
- Bridging to the newer Pydantic-based `rag/flow/chunker/TokenChunker`
(consumes a parser node's structured output rather than a raw file — a
separate, larger feature).
### Type of change
- [ ] Bug Fix (non-breaking change which fixes an issue)
- [x] New Feature (non-breaking change which adds functionality)
- [ ] Documentation Update
- [ ] Refactoring
- [ ] Performance Improvement
- [ ] Other (please describe):
---
## Files changed
- `agent/component/pipeline_chunker.py` — new component (~180 lines)
- `test/unit_test/agent/test_pipeline_chunker.py` — unit tests (~120
lines)
## Test plan
- [x] `ruff check` on changed files — clean.
- [x] `ruff format` applied to the new component file.
- [x] `python -m py_compile` on both new files — both compile.
- [x] New unit test file carries `pytestmark = pytest.mark.p2` so it
runs under marker-filtered CI.
- [x] Every new function, method, and class has a docstring (CodeRabbit
80% docstring-coverage gate).
- [x] `python -m pytest test/unit_test/agent/test_pipeline_chunker.py -x
-q` — **7 passed in 1.95s** locally. Tests stub
`api.db.services.file_service` and `rag.app.*` so they exercise the
parameter validation and parser-id lookup table without requiring the
full backend / model stack.
## Manual integration plan (post-merge)
1. Drop the component into an Agent canvas after a `Begin` node with a
file input.
2. Set `parser_id = "naive"` (or any other strategy) and reference the
file input in `inputs`.
3. Wire the `chunks` output into a downstream `LLM` / `Message` /
`Iteration` node — chunks are available as plain text without any
embedding or KB write.
Co-authored-by: John Baillie <johnbaillie2007@gmail.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-authored-by: Zhichang Yu <yuzhichang@gmail.com>
## Summary
- harden reopened advisory fixes across REST connector, invoke, document
downloads, and markdown rendering
- add targeted regression coverage for redirect-safe SSRF handling,
invoke SSRF checks, document access control, and markdown sanitization
- verify each referenced GHSA against the original GitHub advisory text
and align the closed-advisory plan with the implemented remediation
## What changed
- add tenant access checks to document download endpoints to avoid
cross-tenant document disclosure
- add per-hop SSRF validation, DNS pinning, redirect handling, and
redirect limits to the REST API connector
- ensure invoke requests validate and pin the resolved host and never
follow redirects implicitly
- keep the generic rate-limited request path wrapped, not just GET and
POST helpers
- sanitize markdown HTML before rendering in the highlight markdown
component
## Validation
- `cd web && npm test -- --runInBand
src/components/highlight-markdown/__tests__/index.test.tsx`
- `.venv/bin/python -m pytest -q
test/unit_test/data_source/test_rest_api_connector.py`
- targeted `test/testcases/test_web_api/...` unit additions were
reviewed, but the suite cannot be executed end-to-end in this
environment because parent `test/testcases/conftest.py` requires a local
service on `127.0.0.1:9380`
## Notes
- all GHSA entries referenced by the plan were checked against the
original GitHub advisory text, not sampled
- the closed-advisory plan document was updated locally during review,
but is intentionally not included in this PR
## Problem
The CodeQL Go analysis was failing on the entire codebase with:
fatal error: office_oxide.h: No such file or directory
because six ingestion parser files (`doc`, `docx`, `ppt`, `pptx`, `xls`,
`xlsx`) import `github.com/yfedoseev/office_oxide/go`, a CGO binding to
a Rust library. The CodeQL runner image doesn't ship the
`office_oxide.h` native header, so the Go AST build aborts before CodeQL
can analyze anything.
This means **no Go-language alerts have been re-evaluated** since the
suppression comments were added in #16407 and #16408. The most recent
CodeQL run fixed 51 alerts (all Python/JS), but every Go alert stayed
open, including ones in files that have nothing to do with office_oxide.
## Fix
Add a `.github/codeql/codeql-config.yml` that uses `paths-ignore` to
skip the six parser files. The rest of the Go tree is pure Go (no CGO)
and analyzes cleanly.
The parser files are also excluded from local `go test` / `go build`
when the office_oxide C library isn't installed, so this brings CodeQL
in line with the existing toolchain.
## Expected outcome
After this PR merges, the next CodeQL run on main will:
1. Complete successfully (Go analysis no longer aborts)
2. Re-evaluate the alerts in the remaining files
3. Match the existing `// codeql[go/...] suppression comments` added in
#16407 and #16408
4. Close those alerts
This should drop the open-alert count from 44 to near zero (the 6 Python
clear-text-logging and 1 JS prototype-pollution alerts that were added
in #16408 will also be re-evaluated).
## Why not just install office_oxide in the CodeQL runner?
- The `office_oxide` Go binding is a 3rd-party module
(`github.com/yfedoseev/office_oxide/go`) with CGO that pulls in a Rust
crate
- The CodeQL runner uses a stock Go toolchain that doesn't include the C
library
- Installing it would require modifying the GitHub-managed CodeQL
workflow, which is owned by GitHub and not easily customizable
- The parsers are also unimplemented stubs (each `Parse` function logs
the filename and returns `nil` after my earlier clear-text-logging fix),
so they have no security-relevant code to scan anyway
🤖 Generated with [Claude Code](https://claude.com/claude-code)
## Summary
After #16407 merged, 44 of the original 93 CodeQL alerts were still open
on the default branch. This PR closes the remaining ones by:
1. **Moving 32 existing `// codeql[...]` directives** so they sit on the
line **immediately before** the suppressed statement. The original
multi-line suppression blocks had the directive as the first line, with
the rationale on subsequent lines. After line shifts (refactors, linter
reformat), the directive ended up several lines above the alert location
— CodeQL only recognizes the suppression when it appears on the line
directly above. (32 alerts across 27 files.)
2. **Adding 9 new `// codeql[...]` suppressions** for alerts that had no
suppression in the preceding lines at all — mostly real-fixes that
CodeQL conservatively still flags (filepath.Base, bounded slice sizes,
model-identifier strings, the MD5-legacy-migration lookup in
`conversation_service.py`).
## Files changed
- `api/db/services/conversation_service.py` — add
`py/weak-sensitive-data-hashing` suppression (MD5 for backward-compat
legacy row lookup; not used for auth)
- `api/db/services/llm_service.py` — 3×
`py/clear-text-logging-sensitive-data` suppressions on the lines that
log `llm_name` in warnings/info
- `common/misc_utils.py` — 2× `py/clear-text-logging-sensitive-data`
suppressions on the redacted `current_url` log sites
- `internal/agent/component/invoke.go` — moved existing
`go/request-forgery` directive
- `internal/agent/sandbox/ssh.go` — moved existing
`go/command-injection` directive
- `internal/agent/tool/retrieval_service.go` — added
`go/uncontrolled-allocation-size` suppression (`topN` is bounded to 1024
above)
- `internal/cli/common_command.go` — moved 2×
`go/disabled-certificate-check` directives
- `internal/cli/user_command.go` — added `go/clear-text-logging`
suppression (filepath.Base already strips user-identifying path)
- `internal/dao/pipeline_operation_log.go` — moved 2× `go/sql-injection`
directives
- `internal/dao/user_canvas.go` — added `go/sql-injection` suppression
in `GetList` (the new `userCanvasOrderClause` call path)
- `internal/engine/infinity/chunk.go` — moved existing
`go/unsafe-quoting` directive
- `internal/entity/models/*` — moved `go/path-injection` directives (15
files)
- `internal/handler/oauth_login.go` — moved existing
`go/cookie-httponly-not-set` directive
- `internal/handler/tenant.go` — moved existing `go/path-injection`
directive
- `internal/service/deep_researcher.go` — moved existing
`go/unsafe-quoting` directive
- `internal/service/dataset.go` — added
`go/uncontrolled-allocation-size` suppression (`n` bounded to 1024
above)
- `internal/service/file.go` — moved existing `go/request-forgery`
directive
- `internal/service/langfuse.go` — moved 2× `go/request-forgery`
directives
- `internal/utility/mcp_client.go` — moved 3× `go/request-forgery`
directives
- `internal/utility/smtp.go` — moved existing `go/email-injection`
directive
- `rag/prompts/generator.py` — added
`py/clear-text-logging-sensitive-data` suppression
- `web/.../use-provider-fields.tsx` — added
`js/prototype-pollution-utility` suppression (FORBIDDEN_KEYS guard is on
the line above)
## Why the previous PR left alerts open
`// codeql[query-id] explanation` must be on the line **immediately
before** the suppressed statement per the [GitHub CodeQL suppression
spec](https://docs.github.com/en/code-security/code-scanning/automatically-scanning-your-code-for-vulnerabilities-and-errors/customizing-code-scanning-with-codeql/suppressing-code-scanning-alerts).
The original suppression blocks were 4-5 lines, with the directive as
the **first** line. After linter reformat / line shifts, the directive
ended up too far above the actual alert line to be recognized. The fix
is to put the directive on the line directly above the suppressed
statement, with the rationale above it.
## Test plan
- All 9 modified Python files `ast.parse` clean
- All 4 modified Go files `gofmt` clean
- 36/44 expected alert suppressions in place
- 8 remaining CodeQL alerts are the originals (#3485851828, #3485851831,
#3485869759, #3485869766, #3485869768, #3485869771, #3485885962,
#3485895527) which were resolved by the corresponding commit comments;
these should close on the next scan when the suppression comments match
the alert lines.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
## Summary
Ports five Python agent APIs to Go under the v1 Gin router:
- `GET /api/v1/agents/attachments/<attachment_id>/download`
- `POST /api/v1/chatbots/<dialog_id>/completions` (SSE)
- `GET /api/v1/chatbots/<dialog_id>/info`
- `POST /api/v1/agentbots/<agent_id>/completions` (SSE)
- `GET /api/v1/agentbots/<agent_id>/inputs`
Mirrors the existing Python wire shape (`{code, message,
data:{answer,reference,...}}` per Python `canvas_service.completion`) so
the iframe SDK and existing JS widgets keep working.
## Behavioural parity with Python
| # | Concern | How it's met |
|---|---------|--------------|
| R0 | Bot routes must not require regular user session | Routes mount
on `apiNoAuth` (router.go:198-202), with `BetaAuthMiddleware` only |
| R3 | Two SSE formats in Go drift | F2: `AgentChatCompletions` and
`AgentbotCompletion` share `service.WriteChatbotRunEvent` |
| R7 | `GetBySessionID` returns `(nil, nil)` on miss | Defensive
nil-check before `session.UserID != tenantID` |
| R8 | Begin component name vs ID | `FindBeginComponentID` resolves name
→ ID first, then `ExtractComponentInputForm(dsl, beginID)` |
| R9 | Defensive PromptConfig parsing | `stringFromMap` helper used for
`prologue` and `tavily_api_key` |
| R10 | `BetaAuthMiddleware` Bearer-prefix pre-filter | Removed —
`GetUserByToken` is called unconditionally, falls back to
`GetUserByBetaAPIToken` |
| F8 | Multi-turn chatbot history | `ChatbotCompletion` reads prior
turns from `session.Message`, appends user turn, calls LLM, persists new
pair via new `API4ConversationDAO.Update` |
| F9 | UUID gate stricter than plan | Removed — only `filepath.Base` +
CR/LF/quote header sanitization remains |
| H2 | Defence-in-depth IDOR | `AgentbotCompletion` calls `loadCanvas`
before delegating to `RunAgent` |
| M2 | SSE error leakage | `WriteChatbotFrame` emits generic `"an
internal error occurred"`; real error logged via `common.Error` |
## Verification
```bash
$ go vet ./... # clean (only pre-existing issues)
$ go build ./... # success
$ go test ./internal/handler/ ./internal/service/ ./internal/agent/dsl/ ./internal/common/ ./internal/dao/
ok ragflow/internal/handler 0.617s
ok ragflow/internal/service 1.729s
ok ragflow/internal/agent/dsl 0.008s
ok ragflow/internal/common 0.087s
ok ragflow/internal/dao 0.083s
```
1199 tests pass across 5 packages.
## Known follow-ups (out of scope for this PR)
- **F1**: token-level streaming in `ChatbotCompletion` (currently emits
one frame per turn)
- **F3**: per-route `auth_types` attribute in Go (currently applied via
route group middleware)
---------
Co-authored-by: Claude <noreply@anthropic.com>
## Summary
- Added Go API route `PUT /api/v1/chats/:chat_id` to align with Python
`PUT /api/v1/chats/<chat_id>` chat update behavior.
- Added Go API route `PATCH /api/v1/chats/:chat_id` to align with Python
`PATCH /api/v1/chats/<chat_id>` partial chat update behavior.
- Added matching handler and service logic for owner checks, tenant
validation, persisted-field filtering, read-only field filtering,
`dataset_ids` to `kb_ids` conversion, and PATCH shallow merge semantics
for `prompt_config` and `llm_setting`.
### What problem does this PR solve?
RAGFlow(api/default)> show admin server;
RAGFlow(api/default)> show api server 'default';
RAGFlow(admin)> show admin server;
RAGFlow(admin)> show api server 'default';
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
---------
Signed-off-by: Jin Hai <haijin.chn@gmail.com>
## Summary
Add support for **"New API"** as a model provider, enabling connection
to [New API](https://github.com/QuantumNous/new-api) /
[one-api](https://github.com/songquanpeng/one-api) compatible gateways
that aggregate multiple LLM backends behind a unified OpenAI-compatible
`/v1` endpoint.
### Features
- **All model types**: Chat, Embedding, Rerank, Image2Text, TTS,
Speech2Text
- **List Models discovery**: `NewAPI(OpenAIAPICompatible)` class in
`model_meta.py` queries the gateway's `/v1/models` to auto-discover
available models via the native `GET /api/v1/providers/<name>/models`
endpoint
- **Model parameter editing**: Pencil icon on each discovered model row
to edit `model_type`, `max_tokens`, and `features` (e.g. tool call
support) before submitting
- **Custom model addition**: "Add Custom Model" button at the bottom of
the List Models dropdown for models not returned by the API
- **Gear icon settings**: Enabled the Settings gear button on provider
instances to manage models on existing instances (viewMode)
- **viewMode credential passthrough**: Fixed List Models in viewMode —
merges `initialValues` credentials when `api_key`/`base_url` fields are
hidden by `hideWhenInstanceExists`
### Changes
**Backend** (8 files):
- `rag/llm/chat_model.py` — `NewAPIChat(Base)` class
- `rag/llm/embedding_model.py` — `NewAPIEmbed(OpenAIEmbed)` class (no
auto `/v1` append)
- `rag/llm/rerank_model.py` — `NewAPIRerank(Base)` class (uses `/rerank`
endpoint)
- `rag/llm/cv_model.py` — `NewAPICv(GptV4)` class
- `rag/llm/tts_model.py` — `NewAPITTS(OpenAITTS)` class
- `rag/llm/sequence2txt_model.py` — `NewAPISeq2txt(GPTSeq2txt)` class
- `rag/llm/model_meta.py` — `NewAPI(OpenAIAPICompatible)` class for List
Models discovery
- `conf/llm_factories.json` — New API factory entry with all model type
tags
**Frontend** (8 files + 1 new SVG):
- `web/src/assets/svg/llm/new-api.svg` — New API logo icon
- `web/src/constants/llm.ts` — `LLMFactory.NewAPI` enum + `IconMap`
entry
- `web/src/components/svg-icon.tsx` — `NewAPI` added to `svgIcons`
-
`web/src/pages/user-setting/setting-model/modal/provider-modal/field-config/local-llm-configs.ts`
— New API `buildLocalConfig`
-
`web/src/pages/user-setting/setting-model/modal/provider-modal/constants.ts`
— `LIST_MODEL_PROVIDERS` includes NewAPI
- `web/src/pages/user-setting/setting-model/components/used-model.tsx` —
Enable Settings gear button
-
`web/src/pages/user-setting/setting-model/modal/provider-modal/hooks/use-list-models-picker.ts`
— viewMode credential merge + model editing state/handlers
-
`web/src/pages/user-setting/setting-model/modal/provider-modal/hooks/use-list-models-options.tsx`
— Pencil edit icon per model row
-
`web/src/pages/user-setting/setting-model/modal/provider-modal/index.tsx`
— `AddCustomModelDialog` import + edit dialog rendering
**Note on Go implementation**: A Go model driver (`NewAPIModel`
delegating to `OpenAIModel`) has been prepared but is deferred until the
Go runtime is enabled in a future release (current v0.26.0 images use
`API_PROXY_SCHEME=python` and do not compile Go binaries). Will submit
as a follow-up PR.
## Related
- Depends on: #15996 (provider instance API improvements — server-side
credential lookup, idempotent `add_model`, security fixes — required for
viewMode gear icon and batch model submission)
## Test plan
- [ ] Add New API provider with api_key and base_url pointing to an
OpenAI-compatible gateway
- [ ] Click "List Models" — should discover and display available models
from `/v1/models`
- [ ] Click pencil icon on a model — should open edit dialog to change
model_type, max_tokens, features
- [ ] Select multiple models and click OK — should add all selected
models
- [ ] Click gear icon on the added instance — should open viewMode with
List Models working
- [ ] In viewMode, select new models including pre-existing ones, click
OK — should succeed (requires #15996)
- [ ] Verify all model types work: create a Chat assistant, Embedding
KB, Rerank setting
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Tim Wang <wanghualoong@users.noreply.github.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
### What problem does this PR solve?
```
RAGFlow(api/default)> show model 'WiseDiag-Z1 Think';
RAGFlow(api/default)> list models;
RAGFlow(admin)> show model 'WiseDiag-Z1 Think';
RAGFlow(admin)> list models;
```
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
Signed-off-by: Jin Hai <haijin.chn@gmail.com>
### What problem does this PR solve?
```
RAGFlow(api/default)> show var 'mail.port';
+-----------+-----------+--------------+-------+
| data_type | name | setting_type | value |
+-----------+-----------+--------------+-------+
| integer | mail.port | config | 30 |
+-----------+-----------+--------------+-------+
```
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
---------
Signed-off-by: Jin Hai <haijin.chn@gmail.com>
### What problem does this PR solve?
```
RAGFlow(api/default)> show provider 'zhipu-ai'
RAGFlow(api/default)> show provider 'zhipu-ai' instance 'test';
RAGFlow(api/default)> show provider 'zhipu-ai' instance 'test' balance;
RAGFlow(api/default)> show provider 'zhipu-ai' model 'glm-4.5';
```
### Type of change
- [x] Refactoring
---------
Signed-off-by: Jin Hai <haijin.chn@gmail.com>
## Summary
- Align Langfuse API key set/get/delete behavior with the Python
implementation.
- Improve DAO handling for Langfuse credential save/delete flows.
- Add tests for Langfuse service error handling and API key lifecycle
behavior.
### What problem does this PR solve?
As title
/api/v1/connectors/<connector_id> PATCH was implemented in #15512
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
- [x] Refactoring
### What problem does this PR solve?
As title:
implement:
```
/api/v1/messages/search GET
/api/v1/messages GET
/api/v1/messages/<memory_id>:<message_id>/content GET
/api/v1/memories/<memory_id>/config GET
/api/v1/messages/<memory_id>:<message_id> PUT
```
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
### Bug
`RAGFlowHtmlParser.chunk_block()` splits an oversized block by slicing
the **tokenized** string and storing the joined tokens:
```python
tks_str = rag_tokenizer.tokenize(block)
...
tokens = tks_str.split(" ")
while start < len(tokens):
chunks.append(" ".join(tokens[start:start + chunk_token_num])) # tokenized form, not source
```
On the default (Elasticsearch) backend `rag_tokenizer.tokenize`
transforms text: it lowercases/stems Latin words and inserts spaces
between CJK characters. So any text block longer than `chunk_token_num`
is stored as garbled, lowercased, space-segmented text instead of the
source content. The small-block branch correctly stores the original
`block`, so only oversized blocks are corrupted. Affects HTML and EPUB
ingestion (both go through `chunk_block`), degrading retrieved chunks
and the answers generated from them.
### Real tokenizer behavior (infinity-sdk 0.7.0, ES backend)
```
tokenize("Hello World FOO Bar Baz Qux Jumps") -> "hello world foo bar baz qux jump" # lowercased + stemmed
tokenize("你好世界这是一个测试") -> "你好世界 这 是 一个 测试" # spaces inserted
```
### Fix
Split the **original** text: break it into atoms (whitespace-delimited
runs for space-separated scripts, per-character for spaceless scripts
such as Chinese) and pack them into pieces of at most `chunk_token_num`
tokens. This preserves the source characters and still splits scripts
that have no whitespace — a plain whitespace split would leave CJK as
one un-splittable chunk.
### Proof (real tokenizer, before/after)
Running the old vs new split against the real `infinity.rag_tokenizer`:
```
ENGLISH "Hello World FOO Bar Baz Qux Lazy Dogs" (chunk_token_num=4)
OLD: ['hello world foo bar', 'baz qux jump over', 'lazi dog'] # lowercased + stemmed
NEW: ['Hello World FOO Bar ', 'Baz Qux Jumps Over ', 'Lazy Dogs'] # preserved; each <= 4 tokens
NEW preserves text exactly: True
CHINESE "你好世界这是一个测试用例需要被切分成多个块" (chunk_token_num=3)
OLD: ['你好世界 这 是', '一个 测试用例 需要', ...] # spurious spaces
NEW: ['你好世', '界这是', '一个测', ...] # preserved; each <= 3 tokens
NEW preserves text exactly: True
```
### Tests
Added `test/unit_test/deepdoc/parser/test_html_parser.py` (English +
Chinese oversized blocks, plus small-block merge). Before the fix the
two oversized tests fail (English shows lowercasing, Chinese shows
inserted spaces); after the fix all pass. `ruff check` clean.
### What problem does this PR solve?
`DeepLParam.check()` validated `self.top_n`, but DeepL has no such
parameter (it is not defined on the param class or its base), so
`check()` always raised `AttributeError` and a DeepL component could
never pass validation. Removed the bogus `top_n` check.
Also fixed the `_run` except branch, which computed
`be_output("**Error**...")` but never returned it, silently dropping the
error message.
Closes#16329
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
- [x] Add test cases
### Testing
Added `test/unit_test/agent/component/test_deepl.py` covering
`DeepLParam.check()` with valid defaults and rejection of invalid
source/target languages.
### What problem does this PR solve?
Fixes the PubMed tool always emitting `Authors: Unknown Authors`. The
`safe_find` closure in `_format_pubmed_content` was hardcoded to search
from the article root, so the per-author `LastName`/`ForeName` lookups
never matched.
`safe_find` now accepts an optional `base` node (defaults to `child`,
preserving the existing field lookups), and the author loop passes the
current `<Author>` element.
Closes#16328
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
- [x] Add test cases
### Testing
Added `test/testcases/test_web_api/test_canvas_app/test_pubmed_unit.py`
covering per-author parsing, intact title/journal/DOI fields, and the
no-authors fallback.
Before: `Authors: Unknown Authors`
After: `Authors: Furqan Khan, Jane Smith`
## Summary
Add the Go implementation of `POST
/api/v1/datasets/{dataset_id}/documents/{document_id}/chunks`.
This wires the full create-chunk path in Go:
- router and handler registration
- request/response structs
- chunk creation service logic
- embedding generation
- chunk insert into doc engine
- chunk/token counter increment
- `tag_feas` validation
- `image_base64` decoding and chunk image storage/merge
- unit tests for handler and service
## Testing
Unit tests:
- `/usr/local/go/bin/go test ./internal/handler`
- `/usr/local/go/bin/go test ./internal/service/chunk`
- `/usr/local/go/bin/go test ./internal/service`
- `/usr/local/go/bin/go test ./...`
All passed locally.
Manual curl checks:
- basic text chunk: Go passed
- chunk with `important_keywords` / `questions` / `tag_kwd` /
`tag_feas`: Go passed
- blank content validation: Go matched expected `code=102`
- invalid `image_base64` validation: Go matched expected `code=102`
- image upload and repeated image upload / merge path: Go passed twice
### What problem does this PR solve?
```
RAGFlow(api/default)> list dataset 'ccc' files;
Total: 1
```
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
Signed-off-by: Jin Hai <haijin.chn@gmail.com>
## Summary
Migrated the dataset document upload API (`POST
/api/v1/datasets/:dataset_id/documents`) from Python to the Go backend.
It supports local file uploads (`type=local`), web page ingestion
(`type=web`), and empty document creation (`type=empty`).
## Changes
- **Router**: Registered `POST /api/v1/datasets/:dataset_id/documents`
route.
- **Handler**: Implemented `UploadDocuments` handler and its routing
functions (`uploadLocalDocuments`, `uploadWebDocument`,
`uploadEmptyDocument`).
- **Service**: Implemented `UploadLocalDocuments`, `UploadWebDocument`,
and `UploadEmptyDocument` in `DocumentService`.
- **Refactoring**: Moved permission checking logic to a shared helper
for reuse in file and document services.
- **Tests**: Added comprehensive unit tests for the new handler and
service upload paths.
## Verification
Ran and passed the test suite for service and handler packages:
- `go test ./internal/service`
- `go test ./internal/handler`
### What problem does this PR solve?
- added the new dataset search route and handler
- reused the existing shared SearchDatasets service by adapting
single-dataset requests into dataset_ids=[dataset_id]
- aligned handler error responses with Python behavior for argument/data
errors
- aligned key service error messages such as invalid search_id and mixed
embedding models
- added focused handler and service tests for request mapping and error
behavior
### Tests:
`/usr/local/go/bin/go test ./internal/service -run
'TestSearchDatasetRequestToSearchDatasetsRequest|TestDatasetServiceSearchDatasets'`
`/usr/local/go/bin/go test ./internal/handler -run
'TestDatasetsHandlerSearchDataset'`
## Problem
The Wikipedia tool silently swallows all exceptions with `except
Exception: pass`, making it impossible to debug failures when fetching
Wikipedia pages.
## Fix
Replace the bare `except Exception: pass` with specific exception
handling:
- `DisambiguationError`: log available options
- `PageError`: log page not found
- `Exception`: log unexpected errors with full traceback
Co-authored-by: wills <willsgao@163.com>
Co-authored-by: Zhichang Yu <yuzhichang@gmail.com>
Adds Keenable as a web search tool in the agent, alongside the existing
Tavily/DuckDuckGo/SearXNG/Google tools.
The main difference from the other search tools is that it doesn't
require an
API key. By default it uses Keenable's keyless public endpoint, so it
works out
of the box. Providing a key (in the tool config) switches to the
authenticated
endpoint and lifts the rate limits.
### Changes
- Backend: `agent/tools/keenable.py` — `KeenableSearch`, follows the
Tavily/DuckDuckGo tool shape (results go through `_retrieve_chunks`).
Auto-registered by `agent/tools/__init__.py`.
- Frontend: wired into the agent builder — operator + icon, config form
(optional API key, search mode, site filter, top N), the search tool
menu,
and the existing api_key export sanitizer.
### Config
- API key: optional. Blank = keyless free tier; set it to lift limits /
enable
`realtime` mode.
- `site`: restrict to a single domain.
- `mode`: `pro` (default) or `realtime`.
### Notes
`KEENABLE_API_URL` can override the API base (HTTPS enforced; defaults
to
`https://api.keenable.ai`). The tool only sends the query (no URL
fetch), so
there's no SSRF surface. Verified the frontend with `vite build` and the
backend search path against the public endpoint.
### What problem does this PR solve?
- OpenTelemetry integration
- Checkpoint conformance tests
- State inspector API
- Callbacks
- A series of fault injection tests
- Pregel integration tests
### Type of change
- [x] Refactoring
### What problem does this PR solve?
```
RAGFlow(api/default)> set key 'ragflow-JgnarFSCUiV99oOvvMDei7ZzZg1cVlqGd1AMHrHeKE4';
SUCCESS
RAGFlow(api/default)> unset key;
SUCCESS
RAGFlow(api/default)> list provider 'zhipu-ai' instances;
RAGFlow(api/default)> list providers;
RAGFlow(api/default)> list available providers;
RAGFlow(api/default)> list provider 'zhipu-ai' instance 'test' models;
```
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
---------
Signed-off-by: Jin Hai <haijin.chn@gmail.com>
## Summary
- add `GET /api/v1/datasets/:dataset_id/tags`
- add `PUT /api/v1/datasets/:dataset_id/tags`
- implement dataset tag listing and rename flow
- align rename tag validation and response shape with the Python API
- add handler and service tests for dataset tags
## Routes
- `GET /api/v1/datasets/:dataset_id/tags`
- `PUT /api/v1/datasets/:dataset_id/tags`
## Test
- Run specific tests for dataset tags:
```
go test -v ./internal/service ./internal/handler -run 'TestDatasetServiceListTags|TestDatasetServiceRenameTag|TestDatasetsHandlerListTags|TestDatasetsHandlerRenameTag'
```
- Run all tests for service and handler to verify no regressions:
```
go test ./internal/service ./internal/handler
```
- use curl cmd to test
## Summary
Align the Go implementations of these APIs with the Python behavior:
- `POST /api/v1/datasets/:dataset_id/metadata/update`
- `PATCH /api/v1/datasets/:dataset_id/documents/metadatas`
- `POST /api/v1/documents/upload`
## What changed
- Added the Go routes and handlers for the 3 APIs.
- Aligned batch document metadata updates with Python semantics:
- support `match` in update items
- support list append / replace behavior
- support deleting specific list values
- remove metadata entirely when it becomes empty
- create metadata for documents that previously had none when updates
apply
- count `updated` only when a document actually changes
- Aligned `documents/upload` file uploads with Python-style
`upload_info` behavior:
- store upload-info blobs in the per-user downloads bucket
- return lightweight upload descriptors instead of normal
file-management responses
- Improved URL upload behavior:
- SSRF-guarded fetch with redirect validation
- redirect limit aligned to Python behavior
- normalize filename and MIME type
- add `.pdf` when the fetched content is PDF
- normalize HTML content into readable text instead of storing raw HTML
shells
## Validation
### Unit tests
Passed:
- `go test ./internal/service`
- `go test ./internal/handler`
Also verified targeted cases for:
- batch metadata update semantics
- upload_info URL handling
- upload_info download bucket behavior
### curl checks
Verified the new Go endpoints with `curl` and compared the response
shape and behavior with Python for:
- `POST /api/v1/datasets/{dataset_id}/metadata/update`
- `PATCH /api/v1/datasets/{dataset_id}/documents/metadatas`
- `POST /api/v1/documents/upload`
The Go responses were checked against Python for:
- argument validation
- success response shape
- metadata update results
- upload_info result structure
- file vs URL input handling
### Description
Migrates the datasets tags aggregation API `GET
/api/v1/datasets/tags/aggregation` from Python to Go.
### Changes
- Registered the `GET /api/v1/datasets/tags/aggregation` route.
- Implemented `AggregateTags` in datasets `handler` and `service`.
- Added handler and service `unit tests`.
### Test Verification
- Verified by comparing results between Python (9380) and Go (9384)
services.
- Tested scenarios: single dataset, multiple datasets, empty parameters,
and unauthorized/invalid IDs.
- All tests and Go `unit tests` passed.
### What problem does this PR solve?
```
RAGFlow(api/default)> add admin host '127.0.0.1:9383';
SUCCESS
RAGFlow(api/default)> use admin;
SUCCESS
RAGFlow(admin)> delete api 'default';
SUCCESS
RAGFlow(admin)> delete api 'default';
CLI error: api server: default not found
RAGFlow(admin)> add api 'default' host '127.0.0.1:9384';
SUCCESS
RAGFlow(admin)> use api 'default';
SUCCESS
RAGFlow(api/default)> delete admin
SUCCESS
RAGFlow(api/default)> delete admin;
CLI error: admin server not exists
RAGFlow(api/default)> list api server;
+------------+---------------+-----------------+---------+
| api_server | api_server_ip | api_server_port | auth |
+------------+---------------+-----------------+---------+
| default | 127.0.0.1 | 9384 | no auth |
+------------+---------------+-----------------+---------+
RAGFlow(api/default)> add admin host '127.0.0.1:9383';
SUCCESS
RAGFlow(api/default)> show admin server;
+-------------------+-----------+
| field | value |
+-------------------+-----------+
| admin_server_ip | 127.0.0.1 |
| admin_server_port | 9383 |
| auth | no auth |
+-------------------+-----------+
```
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
Signed-off-by: Jin Hai <haijin.chn@gmail.com>
### What problem does this PR solve?
When I run RAGFlow_server.py:
```
2026-06-24 10:27:01,938 ERROR 3413485 fetch task exception
Traceback (most recent call last):
File "/home/infiniflow/Documents/development/ragflow/api/db/services/document_service.py", line 948, in _sync_progress
if t.progress_msg.strip():
^^^^^^^^^^^^^^^^^^^^
AttributeError: 'NoneType' object has no attribute 'strip'
```
fixed:
```python
if t.progress_msg.strip():
# fix:
if (t.progress_msg or "").strip():
```
Fix crash in `_sync_progress` when `progress_msg` is `None`.
#### Root Cause
`progress_msg` from task records can be `None`, causing:
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
### What problem does this PR solve?
This PR follows up on
[#15863](https://github.com/infiniflow/ragflow/pull/15863) (Korean i18n)
with translation refinements and i18n coverage for hardcoded strings
found in the UI.
- Refine awkward Korean phrasing (e.g. 'Chunk 만들기' → 'Chunk 생성', '유형' →
'타입', etc.)
- Apply i18n to hardcoded strings in `message-item`,
`next-message-item`, `multi-select`, `chat-prompt-engine`, and various
filter hooks
- Rename `use-selelct-filters.ts` → `use-select-filters.ts` (typo fix)
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
### What problem does this PR solve?
Fix release
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
---------
Signed-off-by: Jin Hai <haijin.chn@gmail.com>
## Summary
This PR fixes two issues discovered during testing of the PaddleOCR
async API refactoring:
### 1. PP-OCRv6 returns `ocrResults` instead of `layoutParsingResults`
Models like PP-OCRv6 are pure text recognition models that return
results in `ocrResults.prunedResult.rec_texts` format rather than the
`layoutParsingResults.prunedResult.parsing_res_list` format used by
layout-aware models (PaddleOCR-VL series).
**Changes:**
- `deepdoc/parser/paddleocr_parser.py`: Extract `ocrResults` alongside
`layoutParsingResults` in `_send_request()`, add fallback logic in
`_transfer_to_sections()` and `parse_image()`
- `internal/entity/models/paddleocr.go`: Add `ocrResults` struct and
fallback extraction in Go OCR handler
### 2. Image parsing not integrated into picture chunker
The `parse_image()` method existed in PaddleOCRParser but was never
called from `rag/app/picture.py` (the module that handles image file
uploads). Users configuring PaddleOCR as their layout recognizer would
still get local deepdoc OCR for images.
**Changes:**
- `rag/app/picture.py`: When `layout_recognize` is set to PaddleOCR, use
`PaddleOCROcrModel.parse_image()` instead of local OCR. Falls back
gracefully to local OCR on failure.
## Testing
Verified end-to-end in Docker:
- PaddleOCR-VL-1.6 PDF parsing: ✅ (10 text blocks with bbox)
- PaddleOCR-VL-1.6 image parsing: ✅ (219 chars)
- PP-OCRv6 PDF parsing with ocrResults fallback: ✅ (10 text blocks)
- PP-OCRv6 image parsing with ocrResults fallback: ✅ (136 chars)
## Related PRs
- #15967 (merged) - PaddleOCR async Job API refactoring + new models
- #16086 (merged) - PaddleOCR image parsing support
### What problem does this PR solve?
`_get_optimal_clusters` in `rag/raptor.py` had two edge-case issues in
GMM cluster-count selection:
1. It used `np.arange(1, max_clusters)`, which never evaluates the
upper-bound candidate (`max_clusters`).
2. When effective `max_clusters` becomes `1`, the candidate list was
empty and `argmin` crashed.
This PR makes candidate evaluation inclusive (`1..max_clusters`) and
guards the single-cluster case by returning `1` directly.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
### Validation
- `pytest test/unit_test/rag/test_raptor_psi_tree_builder.py
--config-file pyproject.toml -q`
- `ruff check rag/raptor.py
test/unit_test/rag/test_raptor_psi_tree_builder.py`
### Tests added
- Regression test for `max_cluster == 1` path (no crash, returns 1)
- Regression test verifying upper-bound candidate is evaluated and can
be selected
_AI-assistance disclosure: parts of this change (bug triage and test
scaffolding) were drafted with AI assistance and fully reviewed and
verified by me._
---------
Co-authored-by: Harsh Kashyap <harshkashyap@Harshs-MacBook-Pro.local>
Co-authored-by: Cursor <cursoragent@cursor.com>
### What problem does this PR solve?
- Tools management
- Pregel engine wrapper for better usage
- UT race
- Coding style
### Type of change
- [x] Refactoring
### What problem does this PR solve?
```
RAGFlow(admin)> show model 'abc';
+------------+----------------------------------------------------------------+
| field | value |
+------------+----------------------------------------------------------------+
| command | get_model_by_model_name |
| error | 'get model by model name' is implemented in enterprise edition |
| model_name | abc |
+------------+----------------------------------------------------------------+
RAGFlow(admin)> list models;
+-----------------+--------------------------------------------------------+
| command | error |
+-----------------+--------------------------------------------------------+
| list_all_models | 'list all models' is implemented in enterprise edition |
+-----------------+--------------------------------------------------------+
```
### Type of change
- [x] Refactoring
Signed-off-by: Jin Hai <haijin.chn@gmail.com>
### What problem does this PR solve?
Implement:
1. `/api/v1/datasets/<dataset_id>/documents/<document_id>/chunks GET`
2.
`/api/v1/datasets/<dataset_id>/documents/<document_id>/chunks/<chunk_id>
PATCH`
3. `/api/v1/datasets/<dataset_id>/documents/<document_id>/chunks PATCH`
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
### What problem does this PR solve?
Provider default public key for CLI
### Type of change
- [x] Refactoring
Signed-off-by: Jin Hai <haijin.chn@gmail.com>
Refactor the Go agent port's logging so every log line — gin access,
agent canvas events, harness warnings, fatal boot errors — flows through
a single common.Logger (zap) backed by a rotated file, with structured
fields, level filtering, and configurable rotation.
---------
Co-authored-by: Claude <noreply@anthropic.com>
## Problem
`thread_pool_exec()` dispatches work via `loop.run_in_executor()`, which
submits the callable with a plain `executor.submit(func, *args)` and
does **not** copy the caller's `contextvars.Context`. So a `ContextVar`
set in the async caller is not visible inside the function running in
the worker thread.
This differs from `asyncio.to_thread()`, which runs the callable inside
a copied context. `run_in_executor()` has never propagated context
(verified on Python 3.12 and 3.13) — so this is a pre-existing gap in
the helper, **not** a regression or a Python-version compatibility
issue.
Concretely, any code that sets a `ContextVar` in async code and reads it
inside a function dispatched via `thread_pool_exec` (request tracing,
per-task state, Langfuse trace propagation, etc.) silently loses that
context.
## Fix
Copy the current context before submitting and run the callable inside
it with `ctx.run()`, matching what `asyncio.to_thread()` does:
```python
async def thread_pool_exec(func, *args, **kwargs):
loop = asyncio.get_running_loop()
ctx = contextvars.copy_context()
if kwargs:
inner = functools.partial(func, *args, **kwargs)
return await loop.run_in_executor(_thread_pool_executor(), ctx.run, inner)
return await loop.run_in_executor(_thread_pool_executor(), ctx.run, func, *args)
```
This explicitly **adds** ContextVar propagation to the helper (it does
not restore any prior behavior). Backward-compatible.
## Tests
`TestThreadPoolExec` covers propagation, the kwargs path, per-call
isolation and the unset-default case.
> Note: the branch name still contains `python313` for historical
reasons; the change is unrelated to any Python version.
### What problem does this PR solve?
Updated the release workflow to install SIMDe headers into the MSYS2
toolchain include directory. Adjusted CMake flags to remove references
to the previous SIMDE_INCLUDE_DIR.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
## Summary
Stabilizes the Go unit-test surface so the test suite can run reliably
in CI and locally via \`bash build.sh --test\`.
## Verification
\`\`\`bash
bash build.sh --test -- -count=10 -run TestWithCancel_SequentialAgent
./internal/harness/core/
bash build.sh --test -- -count=5 -run TestSiliconflowChatExtracts
./internal/entity/models/
bash build.sh --test # full suite
\`\`\`
All previously failing packages (\`admin\`, \`cli\`, \`handler\`,
\`parser\`,
\`router\`, \`service\`, \`service/chunk\`) now build and test
successfully.
\`TestWithCancel_SequentialAgent\` passes 10/10 (was flaky). SiliconFlow
reasoning test passes after switching the assertion to the SiliconFlow
wire
format.
---------
Co-authored-by: Claude <noreply@anthropic.com>
## Summary
Several keys added in recent releases were missing from the French
(`fr.ts`) locale file.
- **`top`** — missing in both the common section and the dataset section
- **Chat channels** — all UI strings for the new chat channels feature
(`chatChannels`, `chatChannelDesc.*`, `connectDialog`, `notConnected`,
etc.)
- **Username validation** — `usernameMaxLength`,
`usernameInvalidCharacters`
- **Model editing** — `editCustomModelTitle`
## Changes
- `web/src/locales/fr.ts` — 47 lines added, no other files touched
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
### What problem does this PR solve?
```
RAGFlow(admin)> show role 'user' default models;
+--------------------------+-----------------------------------------------------------------+-----------+
| command | error | role_name |
+--------------------------+-----------------------------------------------------------------+-----------+
| show_role_default_models | 'show role default models' is implemented in enterprise edition | user |
+--------------------------+-----------------------------------------------------------------+-----------+
RAGFlow(admin)> set role 'user' default chat 'glm4.5@test@zhipu-ai';
+------------+---------------------------------------------------------------+
| field | value |
+------------+---------------------------------------------------------------+
| model_id | |
| model_type | chat |
| role_name | user |
| command | set_role_default_model |
| error | 'set role default model' is implemented in enterprise edition |
+------------+---------------------------------------------------------------+
RAGFlow(admin)> reset role 'user' default chat;
+------------+-----------------------------------------------------------------+
| field | value |
+------------+-----------------------------------------------------------------+
| command | reset_role_default_model |
| error | 'reset role default model' is implemented in enterprise edition |
| model_type | chat |
| role_name | user |
+------------+-----------------------------------------------------------------+
```
---------
Signed-off-by: Jin Hai <haijin.chn@gmail.com>
## Summary
- add public Go route for `/api/v1/searchbots/detail`
- implement beta-token auth flow for shared search access
- add tenant-based access check for shared search apps
- add joined search detail query for the share response
- align Go response shape with the current Python runtime behavior
- add DAO / service / handler tests for the new endpoint
close#16132
## Summary
This PR completes the Go-side merge and cleanup for chat channel APIs,
including handler/service wiring, route registration, and test coverage.
Implemented and aligned 5 chat channel APIs:
```
- POST `/api/v1/chat-channels`
- GET `/api/v1/chat-channels`
- GET `/api/v1/chat-channels/:channel_id`
- PATCH `/api/v1/chat-channels/:channel_id`
- DELETE `/api/v1/chat-channels/:channel_id`
```
Co-authored-by: Haruko386 <tryeverypossible@163.com>
### What problem does this PR solve?
```
fixed:
RAGFlow(api/default)> use model 'minimax-m2.5@test@minimax'
SUCCESS
RAGFlow(api/default)> chat message 'who r u'
Answer: Hey! I'm MiniMax-M2.5, an AI assistant here to help you with questions, tasks, or whatever you need. What can I do for you?
Time: 1.727263
```
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
### What problem does this PR solve?
Removed references to 'simde' from the package lists and updated paths
for compiler detection and CMake configuration to ensure proper handling
of Windows executables.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
### What problem does this PR solve?
As title.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
---------
Signed-off-by: Jin Hai <haijin.chn@gmail.com>
Closes#15139.
The "+ Add" flow in the Set/Edit Metadata modal posted updates with an
empty value, so backend saves were silent no-ops and the document's "X
fields" count stayed at 0 despite a "Success" toast.
The value `<Input>` updates `tempValues` synchronously per keystroke but
only writes through to `metaData.values` on blur (via
`handleValueBlur`). When the user clicks the nested modal's Confirm
button without first blurring, the click handler races the blur and
`handleSave` closes over the pre-blur `metaData.values` — still the
initial `['']`. `addUpdateValue` then queues an empty-string update; the
auto-fire save sends it, and after `resetOperations()` the outer Save
button posts `updates: []`.
Read from `tempValues` instead so the queued update carries the typed
value.
Regression test in `tests/use-manage-values-modal.test.ts` asserts that
`handleSave` passes the typed value (not the pre-blur empty string) to
`addUpdateValue` in the add-new code path.
### What problem does this PR solve?
Updated rust_target and added simde support for Windows builds. Modified
CMake commands to include simde and adjusted paths for compilers.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
### What problem does this PR solve?
Updated MSYS2 package list for Windows builds and added Rust target
specifications. Modified build steps for office_oxide and rag tokenizer
libraries to improve compatibility and streamline the build process.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
## What
Fixes#16008 — tables contained in a DOCX are silently dropped when the
document is parsed with the **laws** chunking method.
## Root cause
`Docx.__call__` in `rag/app/laws.py` iterated `self.doc.paragraphs`,
which only yields paragraph elements. Tables are separate `tbl` blocks
in the document body, so they were never visited and were lost from the
output. (The `naive` parser already handles tables by iterating the
document body.)
## Changes
- Iterate `self.doc._element.body` so tables are visited in document
order alongside paragraphs.
- Add a `__table_to_html` helper that renders each table to HTML,
including merged-cell `colspan` detection (mirrors the `naive` parser's
logic).
- Inject each table into the section tree with a sentinel level deeper
than any heading, so `Node.build_tree` merges it into its **enclosing
section** — keeping the chapter/article title path as retrieval context
rather than producing an orphaned chunk.
- Guard the `h2_level` computation against an empty heading set, so a
tables-only or empty DOCX no longer raises `IndexError`.
This keeps the laws parser's hierarchical chunking **and** adds table
extraction, so users no longer have to choose between losing structure
(naive) or losing tables (laws).
## Tests
Adds `test/unit_test/rag/test_laws_docx_tables.py` covering:
- table content is preserved and carries its section title path,
- merged adjacent cells collapse to `colspan`,
- tables-only document does not crash,
- empty document returns `[]`.
All four pass; `ruff check` / `ruff format` are clean.
### What problem does this PR solve?
- list configs;
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
---------
Signed-off-by: Jin Hai <haijin.chn@gmail.com>
### What problem does this PR solve?
- list resources
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
- [x] Refactoring
---------
Signed-off-by: Jin Hai <haijin.chn@gmail.com>
### What problem does this PR solve?
1. Remove unused file
2. Remove duplicate models
3. Resort the function order
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
- [x] Refactoring
Signed-off-by: Jin Hai <haijin.chn@gmail.com>
### What does this PR do?
This PR adds a new DingTalk chat channel integration and hardens the
inbound callback path.
### Summary
- Adds DingTalk as a selectable chat channel in the UI and backend
channel registry.
- Adds the DingTalk chat channel icon asset.
- Acknowledges DingTalk Stream callbacks and deduplicates repeated
inbound messages to avoid duplicate replies.
### What problem does this PR solve?
Implement OpenAI chat completions in GO
POST /api/v1/openai/<chat_id>/chat/completions
OpenAI chat cli: internal/development.md
### Type of change
- [x] Refactoring
### What problem does this PR solve?
1. add modelID for delete_model and update_status
2. fix the bug when update-status delete model
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
- [x] New Feature (non-breaking change which adds functionality)
### What problem does this PR solve?
RAGFlow(admin)> show tasks summary;
+---------+-----------------------------------------------------------------+
| field | value |
+---------+-----------------------------------------------------------------+
| command | show_users_quota_summary |
| error | 'Show users quota summary' is implemented in enterprise
edition |
+---------+-----------------------------------------------------------------+
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
Signed-off-by: Jin Hai <haijin.chn@gmail.com>
The pipeline file log detail hook (`useFetchPipelineFileLogDetail`) was
calling the legacy `kbService.getPipelineDetail({ log_id })` endpoint,
which does not match the current RESTful API contract. The backend now
expects both `datasetId` and `logId` to construct the correct URL (`GET
/api/v1/datasets/{datasetId}/ingestions/{logId}`).
### What problem does this PR solve?
Prepare for enterprise command
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
---------
Signed-off-by: Jin Hai <haijin.chn@gmail.com>
### What problem does this PR solve?
As title
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
- [x] New Feature (non-breaking change which adds functionality)
## Summary
Fixes#15487 — lone markdown headers are no longer isolated as empty
chunks when a custom `delimiter` is set.
- Merge consecutive lone headers before attaching to the following prose
body
- Skip code fences, tables, lists, and blockquotes via
`_is_attachable_body()`
- Unit tests include the `# Title / ## Intro / Body` regression from
CodeRabbit review
## Validation
- `pytest test/unit_test/deepdoc/parser/test_markdown_parser.py` (11
passed locally)
Closes#15487
### What problem does this PR solve?
- Migrated MCP server detail and export (download) API from Python to
Go.
- Registered route: `GET /api/v1/mcp/servers/:mcp_id` (supporting
`?mode=download` query parameter).
### What problem does this PR solve?
This PR implements the Go backend counterpart for the document partial
update API:
`PATCH /api/v1/datasets/:dataset_id/documents/:document_id`
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
- [x] Refactoring
### What problem does this PR solve?
This PR improves code readability in the CLI parser by renaming the loop
index `i` to `modelIndex`. It also renames the loop label `A` to
`optionsLoop` to align with standard Go naming conventions.
### Type of change
- [x] Refactoring
### What problem does this PR solve?
- Update version tags in README files (including translations) from
v0.26.0 to v0.26.1
- Modify Docker image references and documentation to reflect new
version
- Update version badges and image descriptions
- Maintain consistency across all language variants of README files
### Type of change
- [x] Documentation Update
### What problem does this PR solve?
Fixes two bugs in the OpenRouter streaming chat request builder
(`internal/entity/models/openrouter.go`, `ChatStreamlyWithSender`):
1. **qwen/glm models streamed to a broken URL.** The code routed any
`qwen`/`glm` model to
`URLSuffix.AsyncChat`, but `conf/models/openrouter.json` defines no
`async_chat` suffix
(empty), so the request was POSTed to `<base>/` instead of
`<base>/chat/completions` —
breaking streaming for every qwen/glm model. The non-stream path has no
such branch.
Fix: all models use the standard `Chat` suffix, consistent with the
non-stream path.
2. **Streaming reasoning was never enabled.** The request set reasoning
via a non-standard
`thinking` key, which OpenRouter ignores. OpenRouter's API — and this
provider's own
non-stream request (line ~110) and its streamed `delta.reasoning` parser
(line ~311) —
use the `reasoning` object. Fix: send `reasoning: {"enabled":
<thinking>}` (and
`{"effort": ...}` when set, taking precedence as in the non-stream
path).
Closes#16110
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
### What problem does this PR solve?
Add chat model factory for Xiaomi model.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
### What problem does this PR solve?
Adds qqbot as a built-in chat channel so it can be discovered and
started by the channel bootstrapper and shown in the chat channel
settings UI.
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
### What problem does this PR solve?
Part of #15853 (provider model-list refactor).
Refactors **Ollama** `ListModels` onto the shared `ParseListModel`
pattern and fixes two correctness issues:
- **Endpoint:** switch the models suffix from `api/ps` (only
currently-running models) to `api/tags` (all installed models) — the
latter is what a model picker should show.
- **Parsing:** Ollama returns `{"models":[{"name","model"}]}`, a
non-OpenAI shape. Decode it into a typed struct, map the names into
`ModelList`, then enrich through `ParseListModel`. This removes the
previous unchecked type assertions (`result["models"].([]interface{})` /
`.(map[string]interface{})` / `.(string)`) that **panicked** when the
body was missing the `models` array or any field, and adds a fallback to
the `model` field when `name` is blank.
- Drops the no-op GET request body and a dead base-URL reassignment.
#### Drive-by fix
Shared gitee_test.go `DSModelList` -> `ModelList` compile fix (renamed
in #15900) so the models test package builds; auto-resolves against the
sibling #15853 PRs.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
- [x] Refactoring
### What problem does this PR solve?
Fix: The pipeline created from the template fails immediately upon
execution.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
### What problem does this PR solve?
fix: misc frontend fixes for agent log, login, search settings
- agent-log: restore server-side pagination on export and search;
replace hardcoded labels with i18n keys; switch container to
text-text-primary
- login: validate register nickname against NICKNAME_PATTERN with
reusable setting i18n
- next-search: align llm_setting schema with chat (LlmSettingFieldSchema
+ LLMIdFormField nested, LlmSettingEnabledSchema at form
root) so the slider Switch reads the correct path; strip *Enabled flags
before submit to avoid backend "Unrecognized field name"
errors
- locales: add common.reset (zh/en)
- skills/go-naming: fix relative link to rules/named.md
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
### What problem does this PR solve?
Fix: The .docx file is not displaying fully; the hierarchy of the
pipeline created from the template is missing.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
### What problem does this PR solve?
Fix chat channel message routing to use the connected `chat_id`, and
make the Feishu websocket client bind to the thread-local event loop.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
### What problem does this PR solve?
Show Telegram in the chat channel picker alongside the existing Discord
and Feishu entries.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
### What problem does this PR solve?
Use a 95% max_length threshold before truncating embedding inputs, which
reduces the chance of provider-side invalid-parameter errors on
near-limit chunks.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
Replaces the Python agent canvas runtime with a Go implementation that
runs inside `cmd/server_main`.
The canvas compiles into an eino Workflow that pauses on wait-for-user
via native Interrupt/Resume (no sentinel flag) and resumes from a
Redis-backed CheckPointStore.
All 21 Python agent components and ~35 tools are ported with functional
parity.
Sandbox providers now read their JSON config from the admin-panel
system_settings table with env fallback.
234 files / +35,413 / -6,111. All Go files are gofmt-clean (CI gate
added); drops the v2 DSL E2E step and the gap-analysis plan (both
redundant after the port ships).
## Type of change
- [x] Refactoring
- [x] New feature
- [x] Bug fix
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude <noreply@anthropic.com>
### What problem does this PR solve?
The `get_ingestion_log` endpoint (both Python
`dataset_api_service.get_ingestion_log` and Go
`DatasetService.GetIngestionLog`) was returning only the
**dataset-level** field set, which omits critical fields such as `dsl`,
`document_id`, `parser_id`, `document_name`, `pipeline_id`, etc.
This caused the front-end **dataflow-result page** to be unable to
render the pipeline timeline and chunks when viewing a single ingestion
log, regardless of whether the log was a dataset-level operation
(graph/raptor/mindmap) or a per-file parse.
### Background
`PipelineOperationLogService` provides two field sets:
| Method | Fields |
|---|---|
| `get_dataset_logs_fields` | Minimal set (progress, status, timestamps,
etc.) |
| `get_file_logs_fields` | Superset — includes `document_id`, `dsl`,
`parser_id`, `document_name`, `pipeline_id`, … |
When listing logs, the API correctly distinguishes dataset-level vs
file-level logs and uses the appropriate converter. However, when
**fetching a single log by ID**, both the Python and Go implementations
were hardcoded to the dataset-level set, dropping the extra fields that
the front-end needs.
### What problem does this PR solve?
Part of #15853 (provider model-list refactor). Final two providers.
- **voyage:** Voyage AI exposes no live model-list endpoint — its public
API only has `/v1/embeddings` and `/v1/rerank` — so the previous
`ListModels` was a `no such method` stub. Replace it with a
static-catalog listing sourced from the loaded provider definition,
carrying each model's `max_tokens`, `model_types`, and embedding
`dimensions`. `list models from voyage` now returns the 13-model catalog
instead of erroring.
- **fishaudio:** route the existing `/model` voice listing through the
shared `ParseListModel` helper for consistency; keep the human-readable
`title` as the model name and fall back to `_id` when a title is blank.
#### Drive-by fix
Shared gitee_test.go `DSModelList` -> `ModelList` compile fix (renamed
in #15900); auto-resolves against the sibling #15853 PRs.
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
- [x] Refactoring
Co-authored-by: Haruko386 <tryeverypossible@163.com>
# Summary
- The culprit is commit b4c8711d5 / PR #15415 (fix: upgrade crawl4ai to
0.8.0).
- That upgrade brought in unclecode-litellm, which installs the same
top-level litellm namespace as upstream litellm.
- The crash happens when files from one LiteLLM distribution are mixed
with files from the other: custom_guardrail.py expects
GuardrailTracingDetail, but types/utils.py can come from the older
conflicting package.
## Changes
1. **Entity (`internal/entity/chat_channel.go`)**:
- Implemented `ChatChannel` struct mapping the `chat_channel` database
table.
- Declared `ChatChannelListResponse` as a DTO to filter out sensitive
credentials (`config` field) and fetch the associated `dialog_name` via
left join.
2. **GORM Migration (`internal/dao/database.go`)**:
- Registered `&entity.ChatChannel{}` in the `dataModels` array inside
`InitDB()` to enable safe GORM schema synchronization.
3. **DAO (`internal/dao/chat_channel.go`)**:
- Implemented `ChatChannelDAO` wrapping GORM CRUD methods (`Create`,
`GetByID`, `UpdateByID`, `DeleteByID`).
- Implemented `ListByTenantID` performing a `LEFT JOIN` on the `dialog`
table to retrieve `dialog_name` while excluding `config` values to avoid
credential leaks.
4. **Test (`internal/dao/chat_channel_test.go`)**:
- Added integration unit tests testing the full CRUD lifecycle and GORM
left-join mapping list querying.
### What problem does this PR solve?
Fix: Importing the MCP dialog causes duplicate submissions.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
The parser pods suffer from OOM kills when processing large PDF
documents. The root cause is in api/db/services/task_service.py: when
layout_recognize is not DeepDOC (e.g. Plain Text), page_size was set to
MAXIMUM_TASK_PAGE_NUMBER (100 million), causing the entire PDF to be
processed as a single task with all pages loaded into memory
simultaneously.
This PR fixes the issue by paginating non-DeepDOC PDF parsing tasks the
same way DeepDOC already does.
### What problem does this PR solve?
Syncs the /api/v1/chat/completions docs with the current behavior,
including the new legacy streaming mode.
### Type of change
- [x] Documentation Update
### What problem does this PR solve?
The parser pods suffer from OOM kills when processing large PDF
documents. The root cause is in api/db/services/task_service.py: when
layout_recognize is not DeepDOC (e.g. Plain Text), page_size was set to
MAXIMUM_TASK_PAGE_NUMBER (100 million), causing the entire PDF to be
processed as a single task with all pages loaded into memory
simultaneously.
This PR fixes the issue by paginating non-DeepDOC PDF parsing tasks the
same way DeepDOC already does.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
- [ ] New Feature (non-breaking change which adds functionality)
- [ ] Documentation Update
- [ ] Refactoring
- [x] Performance Improvement
- [ ] Other (please describe):
## Summary
Add image parsing capability to PaddleOCR integration, building on top
of #15967 (async Job API migration).
## Changes
### `deepdoc/parser/paddleocr_parser.py`
- Add `parse_image()` method that uses the same async Job API flow as
`parse_pdf()`
- Extracts text from `layoutParsingResults` → `prunedResult` →
`parsing_res_list`
- Returns concatenated block content as a single string
### `rag/llm/ocr_model.py`
- Add `parse_image()` wrapper to `PaddleOCROcrModel` with availability
check and logging
## Relationship to other PRs
- **Depends on**: #15967 (async Job API migration) — this PR is based on
that branch
- **Replaces**: #14826 (original image processing PR based on old sync
API)
## Notes
This PR uses `base_url` and the async Job API (submit → poll → fetch)
consistent with #15967, rather than the old `api_url` + sync POST
pattern from #14826.
## Summary
Migrate PaddleOCR integration from the deprecated synchronous HTTP API
to the new asynchronous Job API (`submit → poll → fetch`), aligning with
PaddleOCR 3.6.0+ architecture.
## Changes
### Python (`deepdoc/parser/paddleocr_parser.py`)
- Replace synchronous `requests.post()` with async Job API flow (submit
→ poll → fetch)
- Authentication: `token {token}` → `Bearer {token}`
- File transfer: base64 JSON body → multipart file upload
- Polling: exponential backoff (initial 3s, ×1.5, max 15s, timeout
controlled by `request_timeout`)
- Result: fetch full JSONL from result URL, preserving `prunedResult`
with bbox info for crop functionality
- Rename `api_url` → `base_url` (backward compatible: `api_url` still
accepted as fallback)
### Python (`rag/llm/ocr_model.py`)
- Prefer `paddleocr_base_url` / `PADDLEOCR_BASE_URL`, fallback to
`paddleocr_api_url` / `PADDLEOCR_API_URL`
### Go (`internal/entity/models/paddleocr.go`)
- Add `Client-Platform: ragflow` header to submit and poll requests
- Change polling from fixed 3s to exponential backoff (initial 3s, ×1.5,
max 15s)
### Python (`common/constants.py`)
- Add `PADDLEOCR_BASE_URL` to env keys and default config
## Backward Compatibility
- Old env var `PADDLEOCR_API_URL` still works (used as fallback)
- Frontend field `paddleocr_api_url` still works (backend reads it as
fallback)
- No user-facing configuration changes required for existing setups
## Why not use the `paddleocr` SDK package directly?
RAGFlow's `_transfer_to_sections()` relies on `prunedResult` (containing
`block_bbox`, `block_label`, `parsing_res_list`) from the raw API
response for PDF crop functionality. The SDK's public `parse_document()`
API only returns `DocParsingResult` with `markdown_text`, discarding the
bbox data. Therefore we implement the async Job API flow directly via
HTTP, following the same logic as the SDK internally.
### What problem does this PR solve?
Merge password related functions
### Type of change
- [x] Refactoring
Signed-off-by: Jin Hai <haijin.chn@gmail.com>
## Summary
- The `ChatChannel` DB column was renamed from `dialog_id` to `chat_id`
via a migration (added in a prior commit).
- Aligns the REST API layer (`chat_channel_api.py`,
`chat_channel_service.py`) to use `chat_id` consistently.
- Updates the frontend (`interface.ts`, `hooks.ts`,
`connect-dialog-modal.tsx`, `added-channel-card.tsx`) to read/write
`chat_id` instead of `dialog_id`.
- The joined `dialog_name` alias in the list query is unchanged (backend
still returns it under that name).
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
### What problem does this PR solve?
`rag/app/naive.py` `Markdown.load_images_from_urls` fetched image URLs
parsed
straight out of an untrusted uploaded markdown document via a raw
`requests.get`,
with no SSRF validation. Markdown chunking always reaches this path
(`return_section_images=True`), so any authenticated user who uploads a
`.md`/`.markdown`/`.mdx` file to a knowledge base could make the server
issue
requests to internal services or cloud-metadata endpoints, e.g.
``. The `image/`
Content-Type
check only gates decoding — the outbound request (the SSRF) always
fires.
This was the one user-controlled fetch site missed by the project's
existing
SSRF-hardening (`common/ssrf_guard.py`, already applied to the crawler,
SearXNG,
RSS connector, MCP/document APIs, and OAuth avatar download).
The fix validates and DNS-pins every hop with
`common.ssrf_guard.assert_url_is_safe`
before connecting, and follows redirects manually so each redirect
target is
re-validated (closing the DNS-rebinding / redirect-bypass window),
mirroring
`common/data_source/rss_connector.py`. Blocked URLs are skipped and
logged like
any other unreachable image, so legitimate public images are unaffected.
Adds a
regression test at `test/unit_test/rag/app/test_markdown_image_ssrf.py`.
Closes#15437
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
---------
Co-authored-by: Ubuntu <ubuntu@ubuntu-2204.linuxvmimages.local>
Co-authored-by: galuis116 <galuis116@users.noreply.github.com>
### What problem does this PR solve?
fix: remove unnecessary 'asChild' prop from FilterButton component
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
### What problem does this PR solve?
fix: remove unnecessary div in profile page layout
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
### What problem does this PR solve?
fix: update channelTemplates to filter for Discord and Lark only
- Fixed a display issue with chunks during pipeline parsing.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
### What problem does this PR solve?
fix: update localization keys for image2text and add ocr option
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
### What problem does this PR solve?
Fix DB migration issue.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
---------
Signed-off-by: Jin Hai <haijin.chn@gmail.com>
### What problem does this PR solve?
Fix:
- Pass session_id to langfuse.
- Get correct status for add model_type.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
### What problem does this PR solve?
This PR fixes Go admin server startup failure caused by duplicate model
aliases in conf/all_models.json.
The model provider loader builds a global lookup table from both model
name and alias values. Some aliases duplicated another model's name or
another
alias, for example amazon.titan-embed-text-v1, which caused startup to
fail with a duplicate alias error. This PR removes conflicting duplicate
aliases
while keeping all model definitions intact.
### What problem does this PR solve?
As title
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
- [x] Documentation Update
### What problem does this PR solve?
Fix register user
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
---------
Signed-off-by: Jin Hai <haijin.chn@gmail.com>
### What problem does this PR solve?
TOC chunks now include a toc field so the agent pipeline logs expose the
data the frontend expects.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
### What problem does this PR solve?
## What
- Add `group` class to wrapper div to enable hover state coordination
- Apply hover styles to checkbox via group-hover
- Make FormLabel clickable via onClick toggle + cursor-pointer
- Fix label color logic: disabled vs primary state
## Why
The "Remember me" label was not clickable and had no hover feedback,
making the UX inconsistent with standard checkbox behavior.
## How to test
0. Go to the demo video before/after attached below
1. Go to the login page
2. Click directly on the "Remember me" label → should toggle the
checkbox
3. Hover over the checkbox area → should show hover styles
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
## Before
https://github.com/user-attachments/assets/bd47d45c-09ea-437f-bd98-3397ce040c1e
## After
https://github.com/user-attachments/assets/45c65d1a-bec7-4ad6-8f1c-d149f7296f8f
### What problem does this PR solve?
This PR enhances the CLI parser to support dimension configurations for
custom embedding models. Users can now specify the maximum dimension and
other supported dimensions directly after the embedding keyword.
```
add model 'x1 x2 x3 x4 x5' to provider 'vllm' instance 'test' with
tokens 1024 chat think vision,
token 2048 chat,
token 1024 think vision,
token 0 embedding 2048 64 1024 2048,
token 0 embedding 2048;
```
- The first integer following embedding represents the max_dimension.
- Any subsequent integers represent specific alternative dimensions.
- If no subsequent integers are provided, dimensions defaults to empty,
indicating all sizes under max_dimension are supported.
### Description
Currently, when setting tenant default models (e.g., chat, embedding,
rerank), the API only accepts the composite name
(`model_name@model_instance@model_provider`). However, some integrations
and front-end features prefer using the database `model_id` (UUID)
directly.
This PR adds support for `model_id` in default model configuration:
1. **Request Binding**: Added `model_id` (optional field) to the request
body schema in the handler.
2. **Database Lookup**: If `model_id` is supplied, the service queries
the database to resolve the respective provider, instance, and model
names.
3. **Security Validation**: Verified that the provider associated with
the resolved `model_id` belongs to the requesting tenant.
4. **Unit Tests**: Added `TestSetTenantDefaultModels_WithModelID` to
verify DB ID resolution and tenant mapping.
### What problem does this PR solve?
core module for agent layer built on top of graph engine #16039
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
### What problem does this PR solve?
Fixes the sandbox config API method mismatch so the frontend and backend
use the same HTTP verb.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
### What problem does this PR solve?
Adds a legacy mode for /chat/completions that restores v0.23.0-style
output by converting start_to_think/end_to_think back into raw
<think></think> markers and streaming cumulative answer text.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
### What problem does this PR solve?
go-version of Pregel-based BSP engine
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
### What problem does this PR solve?
Feat:
- Allow upsert model_type for instance model
Fix:
- Allow create instance with duplicate api_key
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
- [x] New Feature (non-breaking change which adds functionality)
### What problem does this PR solve?
Feat: Move less important chat settings into a collapsible panel.
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
### What problem does this PR solve?
As title.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
Signed-off-by: Jin Hai <haijin.chn@gmail.com>
### What problem does this PR solve?
```
RAGFlow(api/default)> parse file 'test.html';
Parsing HTML file: test.html
<html>
......
```
Signed-off-by: Jin Hai <haijin.chn@gmail.com>
Guard the agent-attachment download against a missing or empty storage blob so the caller gets a structured 4xx (`Document not found!`) instead of an HTTP 500. Same bug class as #15365 on document preview.
Resolve#15502
### What problem does this PR solve?
```
RAGFlow(api/default)> parse file 'README.md';
Parsing Markdown file: README.md
--- AST tree:
HTMLBlock '<div align="center">\n<a href="https:…'
```
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
Signed-off-by: Jin Hai <haijin.chn@gmail.com>
### What problem does this PR solve?
Allow S3-compatible data source region fields to accept custom values
while preserving search-and-select behavior.
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
### What problem does this PR solve?
| # | Method | Endpoint | Description | Git Equivalent |
|---|--------|----------|-------------|----------------|
| 1 | `POST` | `/api/v1/{prefix}/{folder_id}/commits` | Create a
snapshot commit with file changes (add/modify/delete/rename) | `git add`
+ `git commit` |
| 2 | `GET` | `/api/v1/{prefix}/{folder_id}/commits` | List commit
history (paginated) | `git log` |
| 3 | `GET` | `/api/v1/{prefix}/{folder_id}/commits/{commit_id}` | Get
commit detail with file changes | `git show` |
| 4 | `GET` | `/api/v1/{prefix}/{folder_id}/commits/{commit_id}/files` |
List file changes in a commit | `git show --name-status` |
| 5 | `GET` |
`/api/v1/{prefix}/{folder_id}/commits/diff?from=...&to=...` | Compare
two commits and return differences | `git diff` |
| 6 | `GET` | `/api/v1/{prefix}/{folder_id}/changes` | Get uncommitted
changes (add/modify/delete) | `git status` |
| 7 | `GET` | `/api/v1/{prefix}/{folder_id}/commits/{commit_id}/tree` |
Get the folder tree snapshot at commit time | `git ls-tree` |
| 8 | `GET` |
`/api/v1/{prefix}/{folder_id}/commits/{commit_id}/files/{file_id}/content`
| Get a file's content as it existed in a specific commit | `git show
HEAD:file` |
| 9 | `GET` | `/api/v1/{prefix}/{file_id}/versions` | Get version
history for a specific file across all commits | `git log -- file` |
Where `{prefix}/{id}` can be:
- `folders/{folder_id}` — direct folder access
- `workspaces/{workspace_id}` — alias of `folders/{folder_id}`
- `datasets/{dataset_id}` — resolves to the dataset's folder
- `memories/{memory_id}` — resolves to the memory's folder
- `skills/{skill_id}` — resolves to the skill's folder
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
- [x] Documentation Update
After upgrading to v0.26.0, the Ollama provider returns an empty model
list because the Go rewrite uses `/api/ps` (only running models) instead
of `/api/tags` (all installed models). This PR changes the endpoint to
`/api/tags`, restoring the ability to list and add Ollama models.
Closes#16000
### What problem does this PR solve?
Not not only model_name@instance_name@provider_name is acceptable, but
also model_id is acceptable.
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
---------
Signed-off-by: Jin Hai <haijin.chn@gmail.com>
### What problem does this PR solve?
Add parser config
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
Signed-off-by: Jin Hai <haijin.chn@gmail.com>
### What problem does this PR solve?
Now we can parse 'pptx', 'ppt', 'doc', 'xls', 'xlsx'
```
RAGFlow(api/default)> parse file 'test.pptx';
Parsing PPTX file: test.pptx
Document format: pptx
```
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
Signed-off-by: Jin Hai <haijin.chn@gmail.com>
### What problem does this PR solve?
Fixes#15840.
The Go HTTP server sets `WriteTimeout: 120s`, which also applies to
long-lived SSE responses. Existing Go streaming handlers did not clear
the per-response write deadline, so streams that run longer than the
server timeout can be terminated mid-response.
This PR adds a small handler helper that clears the response write
deadline for SSE requests and calls it only in existing Go streaming
branches:
- conversation completion streaming
- provider chat streaming
- provider transcription streaming
- provider speech streaming
The global server `WriteTimeout` remains unchanged for non-streaming
requests.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
### Test plan
- `/root/go/bin/go test ./internal/handler -run
TestDisableWriteDeadlineForSSEAllowsLongLivedStream -count=1`
- `/root/go/bin/go test ./internal/handler -count=1`
### What problem does this PR solve?
As title.
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
---------
Signed-off-by: Jin Hai <haijin.chn@gmail.com>
### What problem does this PR solve?
#15844
Adds a **Chat channels** capability so a RAGFlow assistant (Dialog) can
be exposed as a bot on external messaging platforms (Feishu/Lark,
Discord, Telegram, Slack, WeCom, LINE, etc.). An admin configures a bot
in the UI, connects it to an assistant, and inbound messages are
answered from that assistant's knowledge base — replies are delivered
back on the channel.
**Feishu/Lark is implemented and tested end-to-end.** Discord, Telegram,
LINE, and WeCom are scaffolded against the same interface; the remaining
listed channels are tracked as follow-ups.
### Design
**Backend**
- New `chat_channel` table (`tenant_id`, `name`, `channel`, `config`
JSON holding `{credential: {...}}`, `dialog_id`, `status`) +
`ChatChannelService` and RESTful CRUD under `/api/v1/chat_channels`.
- Channel framework under `api/channels/`: a `core` registry +
per-channel packages that self-register a builder and implement a common
`Channel` interface (`start`/`stop`/`send` + inbound normalization) over
`IncomingMessage`/`OutgoingMessage`.
- Embedded **reconcile loop** in `ragflow_server`
(`api/channels/bootstrap.py`): loads enabled bots, and
starts/stops/restarts them as rows change (no server restart needed).
Inbound messages run the connected dialog via the non-streaming
completion path, keeping per-end-user conversation history.
- Missing optional channel SDKs degrade gracefully (channel skipped with
a warning; others unaffected). Channel-level errors are logged, not
crashed.
- Feishu's WebSocket client runs in a dedicated thread with its own
event loop to avoid cross-loop/contextvars conflicts with the channel
runtime.
**Frontend**
- **Settings → Chat channels** panel: available-channels grid +
configured-bots list with add/edit/delete and a **Connect assistant**
popup that binds a bot to a dialog.
- Brand icons via simple-icons / reused shared data-source assets, with
colored fallbacks for brands not available.
- Route, sidebar entry, i18n (en/zh), and a top-nav segment-boundary fix
so the settings page no longer highlights the Chat tab.
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
### Notes
- DB: new `chat_channel` table is auto-created; `chat_channel.dialog_id`
is also covered by a `migrate_db` `alter_db_add_column` for existing
installs.
- Channel SDKs (`lark-oapi`, `discord.py`, `python-telegram-bot`,
`line-bot-sdk`, `wechatpy`, `aiohttp`) added to dependencies.
- Screenshots / per-channel credential docs to follow.
<img width="1338" height="1290" alt="Image"
src="https://github.com/user-attachments/assets/042cb2f9-0dad-4e6a-bcf7-43ced4bbd704"
/>
<img width="1344" height="738" alt="Image"
src="https://github.com/user-attachments/assets/373cd08e-ec40-4c67-9c51-4d948b1ba617"
/>
<img width="672" height="887" alt="Image"
src="https://github.com/user-attachments/assets/5a34953f-a9a3-4c1e-869e-5eff0dc64c84"
/>
---------
### What problem does this PR solve?
Fix: chat/agent -- Default avatar is not displaying correctly.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
### What problem does this PR solve?
Refs #15743
Some Go API handlers return raw `err.Error()` strings in
`CodeServerError` responses. Those errors can include internal backend
details such as database, storage, search engine, or host information.
This PR adds a small shared `jsonInternalError` helper for handler-level
internal failures. The helper logs the raw error server-side with
request method/path context, then returns the existing generic
`common.CodeServerError.Message()` to API clients.
This first slice migrates the existing `jsonError(c,
common.CodeServerError, err.Error())` production call sites in agent,
dataset graph, file, and system handlers. It intentionally does not
close the full issue because direct `c.JSON` error responses in other
handlers remain for follow-up PRs.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
- [ ] New Feature (non-breaking change which adds functionality)
- [ ] Documentation Update
- [ ] Refactoring
- [ ] Performance Improvement
- [ ] Other (please describe):
### Tests
- `/root/go/bin/go test ./internal/handler -count=1`
---------
Co-authored-by: Yingfeng <yingfeng.zhang@gmail.com>
### What problem does this PR solve?
The Profile **Name** field currently lacks application-level validation
and allows users to save excessively long names and unsupported special
characters.
While the database enforces a maximum length of 100 characters, neither
the frontend nor backend validates nickname format before persistence.
This can result in inconsistent user data, poor user experience, and UI
layout issues when long names wrap across multiple lines.
This PR introduces consistent frontend and backend validation for
profile names, enforces length and character constraints, provides clear
validation feedback, and prevents invalid values from being saved.
Fixes#15693
### Type of change
* [x] Bug Fix (non-breaking change which fixes an issue)
## Summary
This PR passes `session_id` into Langfuse trace observations so
multi-turn chat messages can be grouped under the same session in
Langfuse.
Changes include:
- Propagate `session_id` from chat/session APIs into
`dialog_service.async_chat`.
- Pass `session_id` into Langfuse `start_observation(...)`.
- Share Langfuse `trace_context` with chat, embedding, rerank, and TTS
model bundles where applicable.
- Add unit coverage to verify Langfuse observations receive
`session_id`.
- Update affected test stubs for the new optional Langfuse context
arguments.
## Related Issue
Closes: #15636
## Change Type
- [x] Feature
- [x] Bug fix
- [x] Test
- [ ] Refactor
- [ ] Documentation
- [ ] Breaking change
## Real Behavior Proof
Before this change:
- Langfuse observations were created without `session_id`.
- Multi-turn chat traces could not be grouped by session in Langfuse.
After this change:
- Chat/session flows pass `session_id` into `async_chat`.
- Langfuse observations include `session_id`.
- Related model bundles receive shared trace context and session
metadata.
Validation result:
```bash
uv run python -m py_compile \
api/db/services/tenant_llm_service.py \
api/db/services/llm_service.py \
api/db/services/dialog_service.py \
api/db/services/conversation_service.py \
api/apps/restful_apis/chat_api.py \
test/unit_test/api/db/services/test_dialog_service_final_answer.py \
test/unit_test/api/db/services/test_dialog_service_use_sql_source_columns.py
```
Passed.
```bash
uv run pytest \
test/unit_test/api/db/services/test_dialog_service_final_answer.py \
test/unit_test/api/db/services/test_dialog_service_use_sql_source_columns.py -q
```
Result:
```text
11 passed in 16.89s
```
```bash
git diff --check
```
Passed.
## Checklist
- [x] Analyzed the issue requirement.
- [x] Checked existing Langfuse trace integration.
- [x] Implemented only the requested session grouping behavior.
- [x] Added/updated unit tests.
- [x] Ran focused tests successfully.
- [x] Ran Python compile validation.
- [x] Ran whitespace diff validation.
## Summary
Fixes#15695.
The Python GraphRAG path already accumulates similarity when several
N-hop paths produce the same edge, but PageRank was overwritten by the
last path. That makes ranking depend on path order for repeated edges.
This keeps the strongest PageRank seen for a repeated edge in the Python
implementation:
- `rag/graphrag/search.py`
The similarity score still accumulates exactly as before.
## To verify
- `python -m py_compile rag\graphrag\search.py`
- `git diff --check`
- `git diff --stat upstream/main` -> only `rag/graphrag/search.py`
I originally included the Go implementation too, but removed it after
maintainer feedback because the Go version is still under development
and not released yet.
### Summary
Closes#15423
`rag/llm/embedding_model.py` hosts about 40 embedding providers that
shared several defects affecting indexing reliability, cost accounting,
and error visibility. This PR fixes four concrete bugs.
**Masked, inconsistent errors (27 sites).** Nearly every provider ran
`log_exception(_e, res)` followed by `raise Exception(f"Error: {res}")`.
Because `log_exception` always raises, the second line was dead code,
and the surfaced exception varied with whether the SDK response exposed
a `.text` attribute. Every failure path now raises a single
`EmbeddingError` that includes the underlying response detail, so the
cause of a failed embedding is consistent and visible.
**Fabricated token counts.** `LocalAIEmbed` returned a hardcoded `1024`
and `OllamaEmbed` added `128` per text. These values feed `used_tokens`
and therefore billing and usage tracking. Both now report the real count
from the API (Ollama `prompt_eval_count`, LocalAI `usage`) and fall back
to a local token count only when the server omits it.
**Truncation overshoot.** The `8196` limit used by Mistral and Bedrock
exceeded the standard `8192` ceiling and could push boundary sized
inputs past the model limit. Limits are corrected to `8192` and made
intentional per provider, and providers that rely on server side
truncation now request it explicitly (Ollama `truncate=True`, Cohere
`truncate="END"`).
**Missing batching on Zhipu and Ollama.** Both issued one request per
text. They now batch like the other OpenAI compatible providers, turning
N round trips into `ceil(N / batch_size)`. Batched results are realigned
by response `index` so a chunk always keeps its own vector.
A shared `Base._batched_encode` helper owns the batch loop, optional
truncation, result accumulation, and the single error path. It is the
mechanism that lets these fixes live in one place instead of across 27
duplicated sites. The public `encode()` and `encode_queries()` contract
stays the same, so existing callers are unaffected.
Tests covering all four fixes are added under
`test/unit_test/rag/llm/test_embedding_model.py`.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
## Summary
Improves the responsiveness of the User Settings layout by converting
the left navigation sidebar into a compact icon-only rail on mobile
devices.
Previously, the sidebar retained its full desktop width on narrow
viewports, reducing the available space for settings content and making
pages such as **Data Sources** difficult to use on phones and smaller
tablets.
With this change:
- Desktop layouts retain the existing full sidebar experience
- Mobile layouts (<768px) display a compact 64px icon-only navigation
rail
- Main content receives significantly more horizontal space
- Navigation and logout actions remain fully accessible on mobile
## Type of Change
- [x] Bug fix
## Screenshots
| Before | After |
|---------|---------|
| <img width="557" height="760" alt="image"
src="https://github.com/user-attachments/assets/fb0d6a90-2d57-464c-90c6-9097418c7c13"
/> | <img width="557" height="760" alt="image"
src="https://github.com/user-attachments/assets/8db36d0f-7070-41e1-b7b2-0fe9d0cceefb"
/> |
## What Changed
### Mobile Sidebar Optimization
- Added responsive mobile behavior using `useIsMobile()`
- Displays avatar and navigation icons only on mobile
- Hides user email, navigation labels, version information, theme
switcher, and logout text on smaller screens
- Preserves navigation and logout functionality through icon actions
### Layout Improvements
- Updated settings page grid layout to use fixed sidebar widths:
- Mobile: `4rem` (64px)
- Desktop: `303px`
- Uses `minmax(0, 1fr)` for the content panel to prevent overflow and
allow proper shrinking
- Prevents sidebar width from expanding based on content
## Impact
- Improves usability of User Settings pages on phones and small tablets
- Increases available space for settings content
- Reduces horizontal crowding and overflow issues
- Maintains the existing desktop experience
## Test Plan
### Desktop (≥768px)
- Verify the full sidebar is displayed
- Confirm email, navigation labels, version information, theme switch,
and logout text are visible
- Ensure all navigation items function correctly
### Mobile (<768px)
- Verify the sidebar collapses to a 64px icon-only rail
- Confirm main content remains readable without horizontal crowding
- Verify navigation icons route correctly:
- Data Sources
- Model Providers
- MCP
- Team
- Profile
- API
- Confirm logout works from the icon button
### Verification
- Run `npm run build`
- Hard refresh when testing production or Docker deployments
- Verify responsive behavior using browser device emulation
### What problem does this PR solve?
The Go model-driver layer () has ~38,700 lines across 109 files. Roughly
74% of that is boilerplate duplicated into every driver: identical HTTP
client setup, the same 65-line SSE scanner loop, and 10-11 one-line "not
supported" stub methods per driver. Any fix must be manually propagated
to every file. Closes#15820.
This PR establishes the three shared utility files that form the
foundation for incremental driver migration:
---
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
- [x] Refactoring
---------
Co-authored-by: Haruko386 <tryeverypossible@163.com>
### What problem does this PR solve?
As title
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
- [x] New Feature (non-breaking change which adds functionality)
- [x] Refactoring
### What problem does this PR solve?
- Update version tags in README files (including translations) from
v0.25.6 to v0.26.0
- Modify Docker image references and documentation to reflect new
version
- Update version badges and image descriptions
- Maintain consistency across all language variants of README files
### Type of change
- [x] Documentation Update
### What problem does this PR solve?
- Replace embedding model `dimension` metadata with `max_dimension`.
- Add optional `dimensions` metadata for models with fixed selectable
output dimensions.
- Include `max_dimension` and `dimensions` in model list responses.
- Validate requested embedding dimensions before calling provider
embedding APIs.
- Forward SiliconFlow embedding dimensions with the correct `dimensions`
request field.
- Add unit coverage for embedding dimension validation rules.
## Summary
`get_model_config_from_provider_instance()` was not including
`max_tokens` in its returned dict, causing all downstream consumers
(dialog truncation, message fitting, knowledge base trimming, embedding,
graphrag, RAPTOR) to fall back to the hardcoded default of **8192
tokens** regardless of the actual model context window size (e.g.,
GPT-4o 128K, Claude 200K).
Closes#15944
## Root Cause
The function builds `model_config` with only: `llm_factory`, `api_key`,
`llm_name`, `api_base`, `model_type`, `is_tools`. `max_tokens` is never
included.
Yet the data exists in four independent sources:
1. `TenantModel.extra` JSON field — written by
`provider_api_service.py:659`
2. `conf/llm_factories.json` — every model entry has `max_tokens`
3. `rag/llm/model_meta.py` — 9 provider classes fetch real context
windows from APIs
4. `TenantLLM.max_tokens` database column
None of them are read by this function.
## Fix
Two lines added, one per return path:
- **Path B** (model_obj exists → provider-instance model): reads
`max_tokens` from `model_obj.extra` JSON
- **Path C** (fallback → factory config): reads `max_tokens` from
`llm_info` (sourced from `llm_factories.json`)
Both fall back to 8192 when the value is absent, preserving backward
compatibility.
## Impact
This single 5-line change fixes the context window budget for all **78+
call sites** across **20 files** that construct `LLMBundle` or read
`max_tokens` from the config dict, including:
| Consumer | File | Effect |
|---|---|---|
| Dialog chat truncation | `dialog_service.py:562` |
`message_fit_in(msg, max_tokens * 0.95)` now uses real context window |
| Knowledge base trimming | `dialog_service.py:752` |
`kb_prompt(kbinfos, max_tokens)` now fits more retrieved content |
| Agent message fitting | `agent/component/llm.py:322` | Agent prompts
no longer truncated at 7946 tokens |
| Embedding truncation | `task_executor.py:704` | Embedding input uses
actual model limit |
| GraphRAG extraction | `graphrag/*/extractor.py` | Entity extraction
gets full context budget |
| LLM4Tenant.max_length | `tenant_llm_service.py:513` | Chat model
wrapper exposes real context window |
### What problem does this PR solve?
- Add `web/src/locales/ko.ts` with full Korean translation (~3100 keys)
- Register `Ko = 'ko'` in `LanguageAbbreviation` enum (`common.ts`)
- Add `[LanguageAbbreviation.Ko]: '한국어'` to `LanguageAbbreviationMap`
- Add lazy-load entry in `web/src/locales/config.ts`
- Add `korean` key to all existing locale files (`ja`, `id`, `es`,
`pt-br`, `vi`, `zh-traditional`)
- Fix duplicate enum value `FileMimeType.Mdx` (`'text/markdown'` →
`'text/mdx'`)
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
- [x] Other (please describe): Korean (한국어) i18n translation + fix
duplicate FileMimeType.Mdx enum value
### What problem does this PR solve?
Fix: Remove the pagination from the search and retrieval pages.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
## What problem does this PR solve?
Closes#15874
Both the `POST /api/v1/datasets/<dataset_id>/chunks` (re-parse) and
`DELETE /api/v1/datasets/<dataset_id>/chunks` (stop-parsing) handlers
called `settings.docStoreConn.delete` unconditionally. When the
tenant/dataset index has not been created yet — fresh dataset, first
parse interrupted before any chunks were indexed, or index manually
removed — the delete call throws and the handler returns HTTP 500
**after** the document state was already mutated (RUNNING with zeroed
counters for the parse path; CANCEL with zeroed counters for the stop
path), leaving the document in an inconsistent state.
The newer `parse_documents` path in `document_api.py` already uses
`index_exist` before deleting:
## How to fix?
Apply the same `index_exist` guard to both call sites in `chunk_api.py`:
- **`parse`** (POST path, line ~192): guard the delete before
`TaskService.filter_delete`.
- **`stop_parsing`** (DELETE path, line ~242): guard the delete after
`DocumentService.update_by_id`.
Both sites already have the correct `search.index_name(tenant_id)` and
`dataset_id` parameters; the guard is a one-line addition at each site.
## Type of change
- [x] Bug fix (non-breaking change which fixes an issue)
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Wang Qi <wangq8@outlook.com>
Fixes#15587
## Problem
`AzureEmbed.__init__` in `rag/llm/embedding_model.py` and
`AzureGptV4.__init__` in `rag/llm/cv_model.py` both call
`json.loads(key)` unconditionally:
```python
api_key = json.loads(key).get("api_key", "")
api_version = json.loads(key).get("api_version", "2024-02-01")
```
When a user stores a plain API key string (not a JSON object) in the
model configuration — which is a valid and common way to configure Azure
OpenAI — `json.loads` raises `JSONDecodeError`. This makes the model
fail to initialize and causes document parsing/embedding to return a 500
error.
## Fix
Wrap `json.loads` in `try/except (json.JSONDecodeError, TypeError)` and
fall back to using the raw string as the `api_key` with the default
`api_version`. This is the same pattern already applied to the Azure
chat model in PR #15604.
## Files changed
- `rag/llm/embedding_model.py` — `AzureEmbed.__init__`
- `rag/llm/cv_model.py` — `AzureGptV4.__init__`
Fixes#15857
---------
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Wang Qi <wangq8@outlook.com>
Fixes#15529 .
### Problem
`async_ask()` accessed `kbs[0]` without verifying that
`KnowledgebaseService.get_by_ids()` returned any knowledge bases. Empty
or stale `kb_ids` raised `IndexError`, which surfaced as HTTP 500 on
search/bot SSE endpoints.
### Fix
- Add an early guard when `kbs` is empty, yielding a final SSE error
event (consistent with `gen_mindmap()` in the same module).
- Add regression tests for empty `kb_ids` and deleted/invalid KB IDs.
### Test plan
- [ ] `pytest
test/unit_test/api/db/services/test_dialog_service_final_answer.py -k
"async_ask_empty or async_ask_stale"`
- [ ] Manual: `POST /api/v1/searchbots/ask` with invalid `kb_ids`
returns SSE error, not HTTP 500
---------
Co-authored-by: Wang Qi <wangq8@outlook.com>
### What problem does this PR solve?
`POST /api/v1/datasets/{dataset_id}/documents/stop`
(`stop_parse_documents`) cancels parsing tasks and sets `run` to
`CANCEL`, but it does **not** remove chunks already indexed in the doc
store or reset `progress` / `chunk_num`. REST callers can end up with a
“cancelled” document that still returns partial chunks in `GET
.../chunks` and in retrieval.
Legacy `DELETE /api/v1/datasets/{dataset_id}/chunks` (`stop_parsing`)
already performs full cleanup: it resets counters and calls
`docStoreConn.delete`. This PR aligns the newer stop endpoint with that
behavior so both paths leave the dataset consistent.
Fixes [#15788](https://github.com/infiniflow/ragflow/issues/15788).
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
- [ ] New Feature (non-breaking change which adds functionality)
- [ ] Documentation Update
- [ ] Refactoring
- [ ] Performance Improvement
- [ ] Other (please describe):
### Changes
- Update `stop_parse_documents` in `document_api.py` to reset `progress`
and `chunk_num` to `0` and delete partial chunks via
`docStoreConn.delete` after `cancel_all_task_of`.
- Add unit test `test_stop_parse_documents_cleans_partial_chunks` to
assert counters reset and doc store delete is invoked.
### Test plan
- [x] Unit test: `pytest
test/testcases/test_http_api/test_file_management_within_dataset/test_doc_sdk_routes_unit.py::TestDocRoutesUnit::test_stop_parse_documents_cleans_partial_chunks
-v`
- [ ] Manual: upload a slow document, start parse, call `POST
.../documents/stop` while `RUNNING`, verify `GET .../chunks` returns
zero chunks and UI `chunk_count` is 0
- [ ] Control: legacy `DELETE .../chunks` behavior unchanged
---------
Co-authored-by: Wang Qi <wangq8@outlook.com>
## Summary
- Normalize WebDAV file-size metadata before applying the sync size
threshold.
- Enforce the same threshold for numeric string sizes in both document
sync and slim snapshot paths.
- Add focused WebDAV unit coverage for size parsing and over-threshold
skips.
## Why
Some WebDAV servers return file sizes from PROPFIND metadata as strings.
The previous threshold check only handled integer values, so oversized
files could still be downloaded and sent into the chunking pipeline.
Closes#15724.
## Validation
- `uv run --no-project --with pytest --with pytest-asyncio pytest
test/unit_test/data_source/test_webdav_connector_unit.py -q`
- `uvx ruff check common/data_source/webdav_connector.py
test/unit_test/data_source/test_webdav_connector_unit.py`
- `python -m compileall -q common/data_source/webdav_connector.py
test/unit_test/data_source/test_webdav_connector_unit.py`
- `git diff --check`
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
## Summary
Fixes#15699.
User upgrades to v0.25.6 against an existing MySQL database, tries to
add an Ollama provider instance, and gets:
```
MySQL IntegrityError: Duplicate entry 'dbaafbfe608a11f1a5516d6066988224'
for key 'tenant_model_instance.tenantmodelinstance_api_key_provider_id'
```
The route at
[api/apps/restful_apis/provider_api.py:354](api/apps/restful_apis/provider_api.py#L354)
catches it and returns `get_error_data_result(message="Internal server
error")` — which by RAGFlow's convention is HTTP 200 with an error
`code` on the body — hence the reporter's "200 status code but the
database errored" complaint.
### Root cause
The provider-instance refactor in [PR
#15460](https://github.com/infiniflow/ragflow/pull/15460) dropped the
unique-compound-index tuple from `TenantModelInstance`:
```python
# Removed in #15460
class Meta:
db_table = "tenant_model_instance"
indexes = (
(("api_key", "provider_id"), True), # unique
)
```
and added a one-shot drop in `migrate_db()` for existing databases. But
the drop targets the wrong index name:
```python
# Before this PR — wrong name
for table_name, index_name in [
("tenant_model_instance", "idx_api_key_provider_id"), # ← doesn't exist
("tenant_model", "idx_provider_model_instance"),
]:
```
Peewee's auto-derived index name is `<lowercase
classname>_<col1>_<col2>` →
**`tenantmodelinstance_api_key_provider_id`**, which matches the user's
error verbatim. The drop raises `OperationalError: 1091 (HY000): Can't
DROP …`, the surrounding `except` clause at
[db_models.py:1736](api/db/db_models.py#L1736) swallows it as
expected-on-fresh-installs, and the legacy unique index lives on
indefinitely.
### Why Ollama hits it specifically
Ollama doesn't require an API key. The form posts `api_key: ""`. The
app-layer dedupe at
[provider_api_service.py:288-292](api/apps/services/provider_api_service.py#L288-L292):
```python
api_key_str = ""
if api_key: # ← skipped for ""
...
same_key_instance = TenantModelInstanceService.get_by_provider_id_and_api_key(...)
if same_key_instance:
return False, f"Already exist instance: ... with api_key {api_key}"
```
falls through for empty keys. Control reaches
`TenantModelInstanceService.create_instance(..., api_key="")` which
inserts a row whose `(api_key, provider_id) = ("", <provider_uuid>)`
collides with any prior Ollama row that already shipped that same pair →
the still-present unique index throws.
(`dbaafbfe608a11f1a5516d6066988224` in the user's error is the
duplicated `provider_id` UUID, paired with the empty `api_key`.)
### Fix
Add the Peewee auto-name alongside the existing `idx_*` entry so the
migration finally drops the obsolete index on next restart:
```python
legacy_indexes = [
("tenant_model_instance", "idx_api_key_provider_id"),
("tenant_model_instance", "tenantmodelinstance_api_key_provider_id"), # ← added
("tenant_model", "idx_provider_model_instance"),
]
```
The surrounding `try/except (OperationalError, ProgrammingError)`
matches `1091` / `can't DROP` / `does not exist` and treats them as
success, so every state is idempotent (see Test plan).
### Idempotency matrix
| Database state | First entry (`idx_api_key_provider_id`) | New entry
(`tenantmodelinstance_api_key_provider_id`) |
| --- | --- | --- |
| Fresh install (≥ #15460) — neither index exists | `1091` → swallowed |
`1091` → swallowed |
| Upgraded from before dc4b82523 (the user's case) — auto-name present |
`1091` → swallowed | **drops the index** |
| Upgraded after a manual rename to `idx_*` | drops the index | `1091` →
swallowed |
| Re-run of `migrate_db()` after either of the above | `1091` →
swallowed | `1091` → swallowed |
No rollback hazard: nothing depends on this unique constraint anymore
(`create_instance` dedupes by `instance_name` via `duplicate_name`, see
[tenant_model_instance_service.py:27](api/db/services/tenant_model_instance_service.py#L27)).
### What this PR does NOT change
- **`provider_api_service.create_provider_instance`** — its `if
api_key:` gate is correct *for the post-migration world*: multiple
Ollama instances with empty keys under one provider are legitimate, so
we shouldn't tighten the app-layer check.
- **`TenantModelInstance` Peewee model** — the `indexes` tuple was
already removed in #15460. New databases never get the constraint in the
first place.
- **The `except → get_error_data_result` → HTTP 200 pattern at
`provider_api.py:354`** — that's a project-wide convention; changing one
route to HTTP 500 would be inconsistent and out of scope.
## Test plan
- [ ] **Reproducer (pre-fix):** on a database originally created before
#15460, configure an Ollama provider with an empty `api_key`, then try
to create a *second* instance under the same provider — confirm the
`Duplicate entry … 'tenantmodelinstance_api_key_provider_id'` error in
the server log.
- [ ] **Verify the index is present pre-restart:** `SHOW INDEX FROM
tenant_model_instance WHERE Key_name =
'tenantmodelinstance_api_key_provider_id';` — non-empty result.
- [ ] **Restart with the fix applied:** server starts cleanly,
`migrate_db()` runs, no `Failed to drop index` in critical logs.
- [ ] **Verify the index is gone post-restart:** same `SHOW INDEX` query
— empty result.
- [ ] **Re-run the reproducer:** two Ollama instances under the same
provider, both `api_key=""`, both succeed.
- [ ] **Restart a second time** — no new errors; the matching `1091`
swallow keeps the migration idempotent.
- [ ] **Fresh install smoke test:** drop the DB volume, start clean — no
`1091` noise (the new index never existed), no functional regression.
## Files changed
- [api/db/db_models.py](api/db/db_models.py) — extend the legacy-index
drop list with `tenantmodelinstance_api_key_provider_id`; refactor the
inline list to a named `legacy_indexes` local with a comment pointing at
#15460 and #15699.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
- [ ] New Feature (non-breaking change which adds functionality)
- [ ] Documentation Update
- [ ] Refactoring
- [ ] Performance Improvement
- [ ] Other (please describe):
Co-authored-by: Wang Qi <wangq8@outlook.com>
## Summary
Fixes [#15585](https://github.com/infiniflow/ragflow/issues/15585).
- Route markdown preview through the shared `request` client (same as
txt/image previewers) so `Authorization` headers and interceptors are
applied consistently.
- Add a unit test covering `AUTH_BETA` token loading for embedded search
auth.
## Root cause
Search result preview for `.md`/`.mdx` used raw `fetch`, which did not
apply the same auth path as other preview types. That led to `401` on
`GET /api/v1/documents/{id}/preview` even when the user was logged in or
using an embedded search `auth` query param.
## Test plan
- [ ] Log in, run a search, open a markdown citation link — preview
loads (no 401).
- [ ] Open an embedded shared search URL with `auth` query param,
preview a markdown file — preview loads.
- [ ] Confirm PDF/txt preview still works in the same search UI.
---------
Co-authored-by: MkDev11 <89318445+bitloi@users.noreply.github.com>
Co-authored-by: Wang Qi <wangq8@outlook.com>
## Summary
Fixes#15532 — `delete_datasets()` crashes with `IndexError` when a
document has no `File2Document` row.
`delete_datasets()` in `dataset_api_service.py` called
`File2DocumentService.get_by_document_id()` and immediately accessed
`f2d[0].file_id` without checking whether the lookup returned any rows.
Documents created via API ingestion or connector sync may exist without
a linked file record, causing dataset deletion to abort with HTTP 500.
This PR mirrors the existing guard already used in `file_service.py` and
`document_api_service.py`.
### What problem does this PR solve?
FIx replicate model provider failing with valid api key
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
---------
Co-authored-by: Wang Qi <wangq8@outlook.com>
## Summary
Fixes#15409.
Reporter sees scary ERROR-level stack traces in `ragflow_server.log` on
every chat turn against a knowledge base whose spreadsheet has many
columns with embedded IDs (e.g. `id-wstc-bios fvt-322-wstc-bios
fvt-323`). Simple queries work; complex ones return "No answer" with
logs that look like a hard crash.
### What's actually happening
1. The user uploads a wide Excel/CSV.
[rag/app/table.py:477-493](rag/app/table.py#L477-L493) turns each header
into an ES field with a type suffix, e.g. `id-wstc-bios
fvt-322-wstc-bios fvt-323_tks`. This is correct — the parser faithfully
encodes the user's column names.
2. The user asks about test case `fvt-085`. The SQL chat path in
[api/db/services/dialog_service.py:914
use_sql](api/db/services/dialog_service.py#L914) asks the LLM to write
SQL using the field list. The LLM sees the `id-wstc-bios
fvt-NNN-wstc-bios fvt-MMM_tks` pattern and pattern-completes a
plausible-but-nonexistent column.
3. Elasticsearch rejects with `BadRequestError(400,
'verification_exception')`: `Unknown column [id-wstc-bios
fvt-085-wstc-bios fvt-086_tks]` and suggests the closest valid column.
4. **The recovery path already exists**: `use_sql` catches the
exception, re-prompts the LLM with the error text (which contains ES's
"did you mean" hint), and on second failure the caller at
[api/db/services/dialog_service.py:626](api/db/services/dialog_service.py#L626)
falls back to vector search. The chat does produce an answer — it's just
generated from the vector hits instead of SQL.
The only real bug is logging:
-
[common/doc_store/es_conn_base.py:399](common/doc_store/es_conn_base.py#L399)
catches every exception with `self.logger.exception(...)`, which writes
a full traceback at **ERROR** level.
- For LLM-generated SQL this is the hot path, not an exceptional
condition — it can fire twice per turn before the fallback runs.
### Fix
Catch `elasticsearch.BadRequestError` (the parent class of
`verification_exception` / `parsing_exception` / similar SQL-validity
errors) separately and log it at **WARNING** with the SQL plus ES error
message. The message still carries the unknown column name and ES's
suggested alternative, so it's actionable for anyone investigating "why
is my LLM producing bad SQL?" — just without the misleading stack trace.
Other exception types (`ConnectionTimeout`, generic `Exception`) keep
their original `ERROR`-level traceback treatment; those represent real
connectivity / library bugs.
This is a one-file, two-line-net change. The retry loop in `use_sql`,
the `add_kb_filter` injection, and the vector-search fallback are all
unchanged.
### What this PR does NOT change
- **The LLM prompts in `use_sql`** — they already specify `Use EXACT
field names from the schema` and pass the field list explicitly.
Strengthening them risks regressing well-behaved cases and is out of
scope for #15409.
- **The single-retry policy** — extending it to multi-retry with
extracted ES suggestions is a separate enhancement.
- **The parser at `rag/app/table.py`** — the field names match the
user's actual column headers; the parser is doing its job.
## Files changed
- [common/doc_store/es_conn_base.py](common/doc_store/es_conn_base.py)
- Add `BadRequestError` to the `elasticsearch` import.
- In `ESConnectionBase.sql()`, add an `except BadRequestError` arm above
the generic `except Exception` that logs at WARNING and re-raises (so
`use_sql` retry/fallback still triggers).
### What problem does this PR solve?
When embedding a chatbot, the API returned `"Model Name is required"`.
The embed widget now includes the assistant's `llm_id` as `model_name`
in the completion request.
### Type of change
- [x] Bug Fix
### How has this been tested?
- Created a chatbot with a default model.
- Embedded it and sent a message – the error is gone and the assistant
replies correctly.
### Related Issue
Closes#15883
Co-authored-by: RAGFlow Dev <dev@ragflow.local>
Co-authored-by: Wang Qi <wangq8@outlook.com>
### What problem does this PR solve?
Fix: The regular expression configuration for pipeline header-based
chunking will be reset.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
### What problem does this PR solve?
Two commands are used for ingestion file testing
```
RAGFlow(api/default)> chunk 'file' with 'dsl';
Chunk file: file, DSL: dsl
SUCCESS
RAGFlow(api/default)> parse file 'filename' chat 'xxx';
Success to parse local file "filename", vision: , chat: xxx, asr: , ocr: , embedding: , doc_parse:
SUCCESS
```
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
Signed-off-by: Jin Hai <haijin.chn@gmail.com>
### What problem does this PR solve?
`ChatStreamlyWithSender` in two Go model drivers could panic on nil
pointer dereferences when a caller passes a nil model config or omits
the reasoning `Effort`:
- **deepseek.go** - `switch *chatModelConfig.Effort` dereferenced
`Effort` without a nil check. It now defaults to `"high"` when nil.
- **volcengine.go** - the `modelConfig` pointer itself was dereferenced
(`Stream`, `MaxTokens`, `Temperature`, .) with no guard, and `Effort`
was dereferenced unchecked. `modelConfig` now defaults to an empty
`&ChatConfig{}` when nil so the optional-field accesses are safe, and
`Effort` defaults to `"medium"` when nil.
Addresses the CodeRabbit review on `volcengine.go`
`ChatStreamlyWithSender`. Per maintainer feedback ("one PR do one
thing"), the unrelated `handler/auth.go` and
`service/heartbeat_sender.go` changes were removed so this PR is scoped
to the model-provider fixes.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
## Summary
Fixes#15790.
Every Discord sync launched from the current Web UI crashes immediately
with:
```
'list' object has no attribute 'split'
```
The error is raised in
[rag/svr/sync_data_source.py:650-651](rag/svr/sync_data_source.py#L650-L651):
```python
server_ids=server_ids.split(",") if server_ids else [],
channel_names=channel_names.split(",") if channel_names else [],
```
### Root cause
Three independent bugs stack here, all in the Discord branch of
`sync_data_source.py`:
1. **Type mismatch (the user's exact error).** The current form at
[web/src/pages/user-setting/data-source/constant/index.tsx:833-843](web/src/pages/user-setting/data-source/constant/index.tsx#L833-L843)
uses `FormFieldType.Tag` for both **Server IDs** and **Channels**:
```tsx
{ label: 'Server IDs', name: 'config.server_ids', type:
FormFieldType.Tag, required: false },
{ label: 'Channels', name: 'config.channels', type: FormFieldType.Tag,
required: false },
```
Tag inputs serialise to **lists**, not comma-separated strings. The
backend `.split(",")` then explodes on the very first sync.
2. **Field-name mismatch.** The form writes `config.channels`. The
backend reads `self.conf.get("channel_names", None)`. Even if
`.split(",")` were fixed, channels would silently be empty for every
UI-created source.
3. **Int conversion missing.**
[common/data_source/discord_connector.py:82](common/data_source/discord_connector.py#L82)
types `server_ids` as `list[int]` (Discord guild IDs are integers); the
previous `.split(",")` produced strings, so the `channel.guild.id not in
server_ids` filter at
[discord_connector.py:92](common/data_source/discord_connector.py#L92)
silently never matched.
So even the configurations that didn't crash were also broken — there is
no path through the current code that actually filtered by server id
from a UI-created source.
### Fix
A 39-line patch in one function:
- New `Discord._coerce_str_list` static method: accepts `None` / `""` /
`list` / `tuple` / `set` / scalar / comma-separated str, returns a clean
`list[str]` with whitespace trimmed and empty entries dropped.
Smoke-tested against the 10 input shapes that can hit it (see Test
plan).
- `_generate` reads `config.channels` first (the form's actual key) and
falls back to `config.channel_names`, so SDK callers and legacy configs
that already shipped with the old key keep working.
- `server_ids` is coerced to `list[int]`. Non-integer entries are logged
and dropped instead of crashing the sync, so a single malformed tag from
the form doesn't tank the rest of the run.
### What this PR does NOT change
- **Web form key (`config.channels`)** — kept as-is. Renaming it to
`channel_names` would force a UI migration and break in-flight configs;
the backend fallback solves the same problem more safely.
- **`common/data_source/discord_connector.py`** — its signature was
already correct.
- **Other connectors (Slack, Gmail, Confluence, etc.)** — they don't
crash today and were not in the issue's scope.
## Test plan
`Discord._coerce_str_list` has been exercised against all ten realistic
input shapes — list, tuple, set, comma-separated string, str with extra
whitespace, empty entries, integers from a Tag input, None, empty list,
single trailing comma. All pass.
### What problem does this PR solve?
Two bugs in the Agent Categorize component:
1. The backend rejected `message_history_window_size = 0` while frontend
allowed it, causing API errors.
2. When calling the agent API without a `session_id`, a new session was
created but retained history from previous conversations.
### Type of change
- [x] Bug Fix
### How has this been tested?
- Issue 1: `CategorizeParam().check()` now accepts `0` and rejects
negative values.
- Issue 2: `canvas.clear_history()` is called for new sessions (no
`session_id`), ensuring fresh conversation state. Verified via UI and
API that a second call without `session_id` does not remember the first
conversation.
### Related Issue
Closes#15897
Co-authored-by: RAGFlow Dev <dev@ragflow.local>
Co-authored-by: Wang Qi <wangq8@outlook.com>
## Summary
Implements **chunk 1** of #15282 — the four `/api/v1/auth/password/...`
endpoints from the login-page Go port. **Chunk 2 (OAuth/OIDC) is
deferred** to its own subtask, matching the issue author's own
confidence-low recommendation ("multi-provider, stateful redirect flow
with external dependencies; recommend its own subtask").
New endpoints, all registered under `apiNoAuth` (forgot-password users
are unauthenticated by definition):
| Method | Path | Status |
|--------|------|--------|
| `POST` | `/api/v1/auth/password/forgot/captcha` | new |
| `POST` | `/api/v1/auth/password/forgot/otp` | new |
| `POST` | `/api/v1/auth/password/forgot/otp/verify` | new |
| `POST` | `/api/v1/auth/password/reset` | new |
## Wire compatibility with the Python backend
The two backends share state through Redis, so the Go port had to use
identical keys, encodings, and constants. Either backend can now
validate a code the other minted.
- **Redis keys**: `captcha:<email>`, `otp:<email>`,
`otp_attempts:<email>`, `otp_last_sent:<email>`, `otp_lock:<email>`,
`otp:verified:<email>` — same as `api/utils/web_utils.py`.
- **Stored OTP value**: `"<hex_hash>:<hex_salt>"` — same as Python.
- **Hash**: HMAC-SHA256 with a `crypto/rand` 16-byte salt — same as
`hash_code()`.
- **Constants**: `OTP_LENGTH=4`, `OTP_TTL=5min`, `ATTEMPT_LIMIT=5`,
`ATTEMPT_LOCK_SECONDS=30min`, `RESEND_COOLDOWN_SECONDS=60s` — all match
`api/utils/web_utils.py`.
- **Email body**: matches `RESET_CODE_EMAIL_TMPL` byte-for-byte.
## Files
### New
| File | Purpose |
|---|---|
| `internal/utility/otp.go` | OTP/captcha constants, Redis key builders
(`CaptchaRedisKey`, `OTPRedisKeys`, `OTPVerifiedRedisKey`),
`HashOTPCode`, `GenerateOTPCode` / `GenerateCaptchaCode` /
`GenerateOTPSalt` via `crypto/rand`, and `EncodeOTPStorageValue` /
`DecodeOTPStorageValue` matching Python's storage shape. |
| `internal/utility/smtp.go` | Minimal stdlib `net/smtp` sender.
`SendResetCodeEmail(to, otp, ttlMin)` builds an RFC 5322 plain-text
message and dispatches via implicit TLS / STARTTLS / plain — same
selectors as Python `aiosmtplib`. Returns `SMTPNotConfiguredError` if
the config block is empty. |
### Modified
| File | Change |
|---|---|
| `internal/server/config.go` | New `SMTPConfig` struct + `Config.SMTP`
field. Field names mirror the `smtp:` keys in `common/settings.py`
(`mail_server`, `mail_port`, `mail_use_ssl`, `mail_use_tls`,
`mail_username`, `mail_password`, `mail_from_name`, `mail_from_address`,
`mail_frontend_url`) so a single `conf/service_conf.yaml` powers both
backends. |
| `internal/service/user.go` | Four methods — `ForgotIssueCaptcha`,
`ForgotSendOTP`, `ForgotVerifyOTP`, `ForgotResetPassword`. Reuses the
existing `decryptPassword`, `HashPassword`, `userDAO.Update`, and
`utility.GenerateToken` so the reset+auto-login path is identical to
`LoginByEmail`. |
| `internal/handler/user.go` | Four handlers in the same `c.JSON` shape
as `LoginByEmail`. The reset handler rotates the access token and emits
an `Authorization` header for auto-login (matches Python
`construct_response(auth=user.get_id())`). |
| `internal/router/router.go` | Routes registered under `apiNoAuth`,
with an explanatory comment on why they sit outside the auth middleware.
|
## Known divergence — captcha rendering
The Python endpoint returns a rendered `image/JPEG` from the
`python-captcha` library. The Go side has **no image-captcha dependency
vendored** in `go.mod`, and hand-rolling a raster generator was out of
scope for this PR.
This commit returns JSON `{captcha: "<text>"}` instead. Implications:
- **Backend gate is identical** — the OTP step still verifies the
user-submitted captcha string against the Redis value, so the security
model is unchanged.
- **Frontend impact**: the password-reset page rendering needs a small
tweak (text display instead of `<img>`) until a Go captcha library is
wired in.
- The handler comments call this out explicitly so the next PR knows
what to swap.
Possible follow-ups (any one closes the gap):
1. Add `github.com/mojocn/base64Captcha` or `github.com/dchest/captcha`
to `go.mod` and replace the JSON response with an `image/JPEG`.
2. Hand-roll a 5x7 bitmap font + `image/png` writer using only the
stdlib.
3. Render a server-side SVG (cheap, but trivially OCR-able — only useful
as a UI shim).
## Test plan
- [ ] **Captcha**: `POST
/api/v1/auth/password/forgot/captcha?email=<existing>` returns `{code:
0, data: {captcha: "ABCD"}}`. Redis shows `captcha:<email>` with that
value and ~60s TTL. Unknown email returns `code: CodeDataError`.
- [ ] **OTP send**: `POST /api/v1/auth/password/forgot/otp` with the
right captcha mints an OTP, stores `<hash>:<salt>` under `otp:<email>`
for 5 min, sends an email, returns success. With a wrong captcha returns
`CodeAuthenticationError`. Hitting it again within 60s returns "you
still have to wait …" with `CodeNotEffective`.
- [ ] **OTP verify**: correct OTP → `code: 0`, OTP keys cleared,
`otp:verified:<email>` = `"1"`. Wrong OTP → `code:
CodeAuthenticationError`, attempt counter bumped; after 5 wrong tries
`otp_lock:<email>` is set and further attempts hit `CodeNotEffective`.
- [ ] **Reset**: with the verified flag set, supply a new password
(RSA-encrypted+base64, same as `LoginByEmail`). Returns `code: 0`,
`Authorization` header set, verified flag deleted. Without the verified
flag returns `CodeAuthenticationError`.
- [ ] **Wire-compat smoke**: mint an OTP from the Python backend, verify
it via the Go endpoint, and vice versa. Should both succeed.
- [ ] **SMTP misconfigured**: drop `smtp.mail_server` from
`conf/service_conf.yaml`. The OTP-send endpoint should now return
"failed to send email" without panicking; check the log for the
`SMTPNotConfiguredError` warning.
- [ ] **End-to-end FE**: hit the password-reset flow from
`web/src/pages/login-next/`. Confirm the text-captcha shim works after
the FE tweak.
- [ ] `go build ./...` and `go vet ./...` — I could not run these in the
sandbox; please confirm a clean build before merging.
- [ ] `uv run pytest` to confirm no Python regressions (shared Redis
schema).
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
## Summary
- Implemented the Go API endpoint for Memory message forgetting:
- `DELETE /api/v1/messages/{memory_id}:{message_id}`
- Added route registration for the Memory message DELETE endpoint only.
- Added request path validation for `memory_id:message_id`.
- Added service logic to mark a message as forgotten by setting
`forget_at`.
- Preserved Python-compatible response behavior:
- Success returns `code: 0`, `message: true`, `data: null`.
- Added focused unit tests for message path parsing and invalid message
ID handling.
- Fixed Linux cgo linker config to use the installed shared PCRE2
library so Go tests/builds can run in this environment.
## Related Issue
Closes: #15240
## Change Type
- [x] Feature
- [x] Test
- [x] Build / CI compatibility
## Implemented API
- `DELETE /api/v1/messages/{memory_id}:{message_id}`
## Real Behavior Proof
Validated with targeted Go tests:
```bash
/tmp/go1.25.0/bin/go test ./internal/handler ./internal/router
```
Result:
```text
ok ragflow/internal/handler
? ragflow/internal/router [no test files]
```
Validated server entrypoint build:
```bash
/tmp/go1.25.0/bin/go build -o /tmp/ragflow-server-main ./cmd/server_main.go
```
Result:
```text
build succeeded
```
Validated patch formatting:
```bash
git diff --check
```
Result:
```text
no whitespace errors
```
## Checklist
- [x] Implemented only `DELETE
/api/v1/messages/{memory_id}:{message_id}`.
- [x] Did not implement unrelated Memory message APIs.
- [x] Added route registration.
- [x] Added handler validation.
- [x] Added service-level memory access check.
- [x] Added tests.
- [x] Ran targeted Go tests.
- [x] Ran server build validation.
- [x] Ran `git diff --check`.
### What problem does this PR solve?
1. Fix go test, some cases still failed.
2. Remove unused code.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
---------
Signed-off-by: Jin Hai <haijin.chn@gmail.com>
## Summary
- Add official Qwen models to `conf/all_models.json` with `qwen/`
canonical names
- Include verified aliases from official Qwen/Hugging Face model IDs and
common provider naming
- Add metadata for context length, model types, thinking support, and
embedding dimensions
## Details
- Added Qwen model families from the official Hugging Face Qwen
organization
- Normalized canonical model names to the `qwen/...` format
- Preserved official HF IDs and lowercase/common aliases for lookup
compatibility
- Added `dimension` for Qwen embedding models
- Added or corrected `max_tokens` for Qwen model families, including:
- Qwen2.5 Instruct variants
- Qwen3 original, 2507, VL, Coder, Coder-Next, Next, Embedding, and
Reranker models
- Qwen3.5 and Qwen3.6 models
- QwQ models
- Added verified `thinking` metadata where officially supported
- Corrected `model_types` for Qwen Image, Omni, Audio, VL, embedding,
reranker, benchmark, and tokenizer entries
## Summary
- Normalize model alias index keys to lowercase
- Detect lowercase alias collisions during provider manager
initialization
- Fix ListModels metadata mapping for mixed-case provider aliases
### What problem does this PR solve?
Fix: add image2text/speech2text/ocr support
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
### What problem does this PR solve?
Fix: add thin scrollbar styling for x-spreadsheet component
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
### What problem does this PR solve?
Fix: The dataset retrieval test returned an incorrect total number.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
---------
Co-authored-by: balibabu <assassin_cike@163.com>
### What problem does this PR solve?
Ensure agent components with image inputs route to `image2text` models
instead of staying on the chat path, so visual requests use the CV
wrapper when supported.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
## Summary
LocalAI exposes two API surfaces with conflicting naming conventions:
- `GET /api/tags` returns model names with `:latest` suffix (Ollama
format)
- `POST /v1/chat/completions` expects names without `:latest` (OpenAI
format)
RAGFlow discovered models via `/api/tags` and stored the tagged name,
then used it with `/v1/chat/completions`, causing a 404 error because
LocalAI didn't recognize `model:latest`.
## Fix
In `LocalAI.get_model_list()`, strip the tag suffix from model names
using `model["name"].rsplit(":", 1)[0]`, so stored names match what the
OpenAI-compatible endpoints expect.
### What problem does this PR solve?
Set OpenDataLoader and call in parser and naive
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
## What does this PR do?
This PR migrates the Agent Temporary File Download endpoint (`GET
/api/v1/agents/download`) from the Python backend to the Go backend,
optimizing the data retrieval flow and maintaining strict functional
parity. It also fixes a persistent parsing error in the Sandbox code
execution node.
## Checklist
- [x] Code logic matches Python implementation
- [x] All local unit tests passed
- [x] No breaking changes to existing router interfaces
Co-authored-by: Yingfeng <yingfeng.zhang@gmail.com>
### What problem does this PR solve?
1. remove unused code
2. fix login issue
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
- [x] Refactoring
Signed-off-by: Jin Hai <haijin.chn@gmail.com>
### What problem does this PR solve?
cohere api call failing because of missing prefix
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
### What problem does this PR solve?
Support factory models with multiple model types, so visual chat models
can be exposed as both image2text and chat while preserving the database
model-type-per-record design.
This also updates the SILICONFLOW model list and adds a helper script to
refresh SiliconFlow models from the provider API.
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
### What problem does this PR solve?
fix: rename ark_api_key to api_key for volcengine provider config
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
### What problem does this PR solve?
Fix: Fix some model provider-related UI issues
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
This PR expands conf/all_models.json with DeepSeek model entries and
provider aliases.
Changes:
- Added DeepSeek model entries across `V4`, `V3.2`, `V3.1`, `V3`, `R1`,
`Coder`, `Math`, `VL`, `OCR`, `Prover`, `MoE`, and `LLM` series.
- Normalized model name values to lowercase canonical IDs.
- Added alias values for official DeepSeek/Hugging Face names and
provider-specific names from OpenRouter, VolcEngine, SiliconFlow,
HuaweiCloud, and QiniuCloud.
- Preserved model metadata such as max_tokens, model_types, and thinking
where applicable.
- Added Gitee ListModels tests to verify DeepSeek aliases map back to
model metadata from all_models.json.
- Added an optional Gitee integration test gated by
GITEE_LIST_MODELS_INTEGRATION=1.
Test:
/usr/local/go/bin/go clean -cache
/usr/local/go/bin/go test ./internal/entity/models -run
'TestGiteeListModels(MapsAllDeepSeekAliasesToModelMetadata|KeepsOwnedBySuffixAfterAliasMetadataLookup|
Integration)'
### What problem does this PR solve?
Fix QWen rerank error handling so DashScope error responses without a
text attribute do not raise a secondary KeyError and hide the real
provider error.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
### What problem does this PR solve?
The RAGFlow Docker image was 9.06 GB with build-only compiler packages
leaking into the runtime, duplicate frontend source shipped alongside
compiled assets, and no .dockerignore causing ~6 GB of unnecessary
context transfer per build.
### Type of change
- [x] Performance Improvement
## Summary
- Add knowledge-base retrieval support to Go chat completions.
## What changed
- Routes KB-backed chat sessions through the Go retrieval service
instead of falling back to solo chat.
- Resolves embedding and rerank models, validates accessible knowledge
bases, and preserves tenant-aware retrieval.
- Rejects mixed embedding models across selected knowledge bases before
retrieval to avoid incompatible vector dimensions.
- Threads the HTTP request context into streaming retrieval so cancelled
requests can stop downstream retrieval work.
- Applies metadata filters and message-level `doc_ids` before retrieval.
- Expands parent/child chunks before building references and prompt
context.
- Injects retrieved knowledge through a copied dialog prompt config so
the caller's original dialog is not mutated.
- Honors configured empty responses when no chunks are found.
- Names the metadata no-match sentinel and reuses it across
retrieval/handler paths.
- Adds a defensive content cast while appending streamed answers.
- Adds focused unit coverage for retrieval, metadata filtering,
authorization, multimodal messages, references, empty-response behavior,
prompt immutability, and mixed embedding models.
---------
Co-authored-by: Yingfeng <yingfeng.zhang@gmail.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
### What problem does this PR solve?
```
RAGFlow(api/default)> use admin
SUCCESS
RAGFlow(api/default)> use api 'abc';
SUCCESS
```
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
---------
Signed-off-by: Jin Hai <haijin.chn@gmail.com>
### What problem does this PR solve?
Updated supported model providers and the corresponding URLs.
~~Synced supported model providers and base URLs with
**llm_factories.json**, while keeping the AI Badgr configuration example
via the OpenAI-API-Compatible provider.~~
### Type of change
- [x] Documentation Update
### What problem does this PR solve?
As title
```
/api/v1/datasets/<dataset_id>/documents/<document_id>/metadata/config PUT
```
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
Co-authored-by: Yingfeng <yingfeng.zhang@gmail.com>
### What problem does this PR solve?
Add two API in go
```
/api/v1/agents/test_db_connection POST
/api/v1/agents/<agent_id>/sessions DELETE
```
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
---------
Co-authored-by: Yingfeng <yingfeng.zhang@gmail.com>
Implements POST /api/v1/searchbots/ask in Go with streaming SSE,
citations, and think-tag processing. 23 files, 90+ unit tests.
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
## Summary
This PR re-enables the Go test steps in CI that were previously
commented out, and fixes all compilation errors that have accumulated in
`internal/entity/models/` since the `ListModels` return type was changed
from `[]string` to `[]ListModelResponse`.
## Changes
### CI (`.github/workflows/tests.yml`)
- Re-enable **Prepare test resources** step (clones resource repo with
WordNet data)
- Re-enable **Test Go packages** step (runs `go test ./internal/...`)
- Fix resource path race condition by using
`/tmp/resource-${GITHUB_RUN_ID}` instead of `/tmp/resource`
- Exclude `/cli` package from Go tests (contains `main` redeclarations)
### Test fixes (16 model provider test files)
All errors were caused by the upstream change from `[]string` to
`[]ListModelResponse` in the `ListModels` interface:
- Add `joinModelNames` test helper to extract `.Name` from
`[]ListModelResponse` slices
- `strings.Join(models, ",")` → `joinModelNames(models, ",")` (11 files)
- `ids[i] != "..."` → `ids[i].Name != "..."` (cometapi, mistral)
- `got[i] != want[i]` → `got[i].Name != want[i]` (bedrock)
- `[]string` return types → `[]ListModelResponse` (google)
### Pre-existing bugs in model_test.go
Bugs introduced by the upstream `entity/` → `entity/models/` directory
rename:
- Add missing `pm := GetProviderManager()` calls in 3 test functions
- Fix `InitProviderManager` signature (`_, err :=` → `err :=`)
- Fix `MaxTokens` `*int` dereference (6 comparisons)
- Fix `readProviderConfig` relative path (3 levels up instead of 2)
### model.go
- Add `findRepoRoot()` to make `conf/all_models.json` resolution work
from any CWD, fixing `TestSiliconFlowProviderConfigLoadsLatestProModels`
### Test validation
```bash
go build ./internal/... # ✅
go test ./internal/entity/models/... -count=1 # ✅ all pass
```
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
### Summary
Closes#15795
Knowledge-graph queries rank entities by `pagerank * sim` in `KGSearch`,
but the entity chunks written at index time stopped carrying the values
that ranking depends on. `graph_node_to_chunk` only stored
`entity_type`, `description`, and `source_id`, dropping the node
`pagerank` and the n-hop neighbour paths, while `search.py` still read
them back as `rank_flt` and `n_hop_with_weight`.
The producer of these fields, `update_nodes_pagerank_nhop_neighbour`,
was removed in #6513, but the read side in `KGSearch` was never updated.
The result is that on every knowledge-graph query:
- `pagerank` resolves to `0`, so the `pagerank * sim` sort key is `0`
for every entity and selection falls back to arbitrary order.
- Every displayed entity score is `0.00`.
- The n-hop relation-enrichment block is dead code because `n_hop_ents`
is always empty, leaving `merge_tuples` and `is_continuous_subsequence`
orphaned.
This PR restores the missing index-time fields so the documented `P(E|Q)
= pagerank * sim` ranking and the n-hop enrichment work again.
What changed:
- `graph_node_to_chunk` now writes `rank_flt` from the node pagerank and
`n_hop_with_weight` from the recomputed n-hop neighbour paths.
- Reintroduced the n-hop path computation (`n_neighbor`) in
`rag/graphrag/utils.py`, reusing the previously orphaned `merge_tuples`
/ `is_continuous_subsequence` helpers, with a direction-agnostic
edge-weight lookup for undirected graphs. `set_graph` computes the paths
per added or updated node and passes them through.
- `KGSearch` now selects `n_hop_with_weight` in the entity keyword
search so Infinity and OceanBase return it (Elasticsearch and OpenSearch
already read it from `_source`), and the read is hardened against
missing keys or empty strings before `json.loads`.
- Added the `n_hop_with_weight` column to OceanBase, including the
`EXTRA_COLUMNS` migration entry so existing tables get it. The other
engines already map both fields via dynamic templates or the Infinity
mapping.
Scope note: pagerank and n-hop are re-indexed for the added or updated
nodes in each pass, consistent with the existing incremental indexing
design.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
### Testing
Added unit tests in
`test/unit_test/rag/graphrag/test_graphrag_utils.py`:
- `n_neighbor`: path and weight shape, one-hop vs two-hop, isolated
nodes, missing weights, and direction-agnostic lookup.
- `graph_node_to_chunk`: `rank_flt` populated from pagerank and
defaulting to `0`, `n_hop_with_weight` serialized and defaulting to an
empty list.
```
uv run pytest test/unit_test/rag/graphrag/ # 106 passed
uv run ruff check rag/graphrag/ rag/utils/ob_conn.py
```
## Summary
- Complete the Go Elasticsearch result functions that remained stubbed
after #15160.
- Add focused unit coverage for field mapping, aggregation, IDs, and
highlighting behavior.
- Update a stale query-builder test type import discovered during
validation.
## What changed
- Keep the Elasticsearch Go implementation merged in #15160 and fill in
`GetFields`, `GetAggregation`, `GetHighlight`, and `GetDocIDs` in
`internal/engine/elasticsearch/chunk.go`.
- Add regression and invariant coverage in
`internal/engine/elasticsearch/chunk_helpers_test.go`.
- Update `internal/service/nlp/query_builder_test.go` to use the current
`types.MatchTextExpr` type.
## Why
- #15160 implemented the main Go Elasticsearch surface, but
retrieval/tag flows still call result functions that returned stubs.
- Completing these functions keeps Elasticsearch result processing
aligned with the expected document-engine behavior for field extraction,
tag aggregation, doc ID extraction, and snippet highlighting.
## Validation
- `go test ./internal/engine/elasticsearch`
- `GOARCH=arm64 CGO_ENABLED=1 go test ./internal/service/nlp -run
TestQueryBuilder`
- `git diff --check`
- CodeRabbit review reported 0 issues after follow-up fixes.
- Codex Security diff scan found no reportable issues.
## Notes
- This PR is now a follow-up to #15160 rather than a competing
implementation.
- A full local `go test ./internal/service/nlp` run is blocked by local
WordNet resource prerequisites; the query-builder tests touched by this
PR pass with the arm64 CGO path.
### What problem does this PR solve?
Fix: The variables in the Visual Input File of the agent operator are
not displayed.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
### What problem does this PR solve?
Fixes four Go paths that dereference a pointer with no prior nil check,
each
causing a **runtime panic**. Closes#15814.
| # | File | Bug | Fix |
|---|------|-----|-----|
| 1 | `internal/entity/models/deepseek.go` | streaming path runs `switch
*chatModelConfig.Effort` inside `if *Thinking`; panics when
`Thinking=true` and `Effort==nil` | nil-check with default `"high"`,
matching the non-streaming path in the same file |
| 2 | `internal/entity/models/volcengine.go` | identical oversight:
`switch *modelConfig.Effort` with no guard | nil-check with default
`"medium"`, matching its non-streaming path |
| 3 | `internal/handler/auth.go` | `AuthMiddleware` does `if
*user.IsSuperuser`; panics on every authenticated request when the DB
column is `NULL` | guard with `user.IsSuperuser != nil &&`, matching
every other call site (`admin/handler.go`, `admin/service.go`,
`user.go`) |
| 4 | `internal/service/heartbeat_sender.go` |
`responseBody["code"].(float64)` panics on any non-200 response lacking
a numeric `code`; the upstream `recover()` calls `Fatal()` →
`os.Exit(1)`, taking down the whole server | comma-ok assertion (`code,
ok := ...`); return an error instead of panicking |
- [x] Bug Fix (non-breaking change which fixes an issue)
### What problem does this PR solve?
Fixes four panic / spurious-error paths in the Go model layer. Closes
#15818.
| # | File | Bug | Fix |
|---|------|-----|-----|
| 1 | | Thinking-mode streaming path: accessed unconditionally; Gemini
emits usage-only chunks with an empty slice, causing a runtime panic |
Guard each step: , , before indexing |
| 2 | | is a plain for ordinary requests; the cast to silently returns ,
then panics immediately | Switch on concrete type; handle both and |
| 3 | | Identical panic on the streaming path | Same switch-on-type fix
|
| 4 | | The field is optional (absent for non-thinking models) but the
code returned an error when it was missing, breaking every ordinary
Ollama completion | Change to a silent comma-ok assertion; is empty
string when the field is absent |
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
### What problem does this PR solve?
feat: support custom editing for model list
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
## Problem
When users configure auto-metadata for a dataset, parsing crashes with:
```
KeyError: 'properties' in gen_metadata → schema["properties"]
```
## Root Cause
Pydantic `AutoMetadataField` defaults `enum` and `description` to `None`
when the frontend omits these fields:
```python
class AutoMetadataField(Base):
enum: Annotated[list[str] | None, Field(default=None)]
description: Annotated[str | None, Field(default=None)]
```
These `None` values propagate through the call chain and cause two
crashes:
### What problem does this PR solve?
Dedup api_key and migrate `is_tools `in migration.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
### What problem does this PR solve?
When setting the API key for the BaiduYiyan provider, all model
validations fail with the error "Fail to access model using this api
key. No valid response received".
**Root cause:**
1. `BaiduYiyanChat` in `rag/llm/chat_model.py` does not override
`async_chat_streamly()`. The `verify_api_key()` function uses
`mdl.async_chat_streamly()` to validate, but `BaiduYiyanChat` inherits
`Base.async_chat_streamly()` which uses the OpenAI client, not the Baidu
Qianfan SDK (qianfan). Since BaiduYiyan has no OpenAI-compatible
base_url, validation always fails.
2. `verify_api_key()` in `provider_api_service.py` does not format the
raw API key string into the JSON format (`{"yiyan_ak": "...",
"yiyan_sk": "..."}`) that `BaiduYiyanChat.__init__()` expects via
`json.loads(key)`.
**Fix:**
1. Add `async_chat_streamly()` method to `BaiduYiyanChat` using the
qianfan SDK, consistent with the existing `chat_streamly()` method.
2. Add BaiduYiyan API key formatting in `provider_api_service.py`
`verify_api_key()` to match the format expected by
`BaiduYiyanChat.__init__()`.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
- [ ] New Feature (non-breaking change which adds functionality)
- [ ] Documentation Update
- [ ] Refactoring
- [ ] Performance Improvement
- [ ] Other (please describe):
### What problem does this PR solve?
Force image parser runtime output format to JSON so downstream chunking
reads OCR results from the JSON output and image parser chunks can be
displayed.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
Co-authored-by: Wang Qi <wangq8@outlook.com>
### What problem does this PR solve?
The last column definition `INDEX idx_instance_id (instance_id),` in the
`CREATE TABLE tenant_model` statement has a trailing comma, which causes
a MySQL syntax error during deployment.
Closes#15832
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
### How was this tested?
- [x] Visual inspection: the trailing comma on line 837 has been removed
### What problem does this PR solve?
Propagate `tenant_id` from memory task messages into task collection so
refactored task execution can build a valid context.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
### Description
This PR ports the `GET /api/v1/agents/prompts` endpoint from the Python
backend to the Go backend.
### Changes Made
- **Handler**: Added `GetPrompts` method to `internal/handler/agent.go`.
- **Router**: Registered the `agents.GET("/prompts")` route in
`internal/router/router.go`.
- **Logic**: Leveraged the existing `service.LoadPrompt` utility to read
`analyze_task_system`, `analyze_task_user`, `next_step`, `reflect`, and
`citation_prompt` templates directly from the `rag/prompts` directory.
- **Unit Test**: Added `TestGetPrompts_Success` to
`internal/handler/agent_test.go` to mock the HTTP context and validate
the JSON response structure.
### Motivation
This is part of the ongoing effort to port the Agent API surface to Go.
Since this specific endpoint only serves static prompt templates and
does not require the complex DAG/Canvas execution engine, it can be
seamlessly and safely handled by the Go backend.
### Testing
- [x] Added automated unit test `TestGetPrompts_Success` (verified
passing).
- [x] Tested locally via `curl` against the Go server (port 9380) and
Python server (port 9384).
- [x] Verified that the Go JSON response structure and loaded prompt
text are logically 100% identical to the Python implementation.
### What problem does this PR solve?
Fix: An error message appears when accessing the agent's launch page:
"pagesize exceeds maximum value".
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
---------
Co-authored-by: balibabu <assassin_cike@163.com>
## Summary
This PR adds checkpoint/resume support for the GraphRAG
`extract_community` and `resolve_entities` stages.
The implementation stores successful intermediate results in the
document store so interrupted ingestion can resume without repeating
already-completed LLM work. Checkpoints are loaded before each stage,
reused when available, saved after successful batch/community
processing, and cleaned up after the stage completes successfully.
## Related Issue
Closes: #15518
## Change Type
- [x] Feature
- [x] Bug fix
- [x] Test
- [ ] Refactor
- [ ] Documentation
- [ ] Breaking change
## Real Behavior Proof
Validation commands run locally:
```bash
uv run python -m py_compile \
rag/graphrag/checkpoints.py \
rag/graphrag/general/community_reports_extractor.py \
rag/graphrag/entity_resolution.py \
rag/graphrag/general/index.py \
test/unit_test/rag/graphrag/test_checkpoints.py
```
Result:
```text
Passed
```
```bash
uv run pytest test/unit_test/rag/graphrag/test_checkpoints.py
```
Result:
```text
4 passed
```
```bash
uv run pytest \
test/unit_test/rag/graphrag/test_phase_markers.py \
test/unit_test/rag/graphrag/test_graphrag_utils.py \
test/unit_test/rag/graphrag/test_checkpoints.py
```
Result:
```text
95 passed
```
```bash
git diff --check
```
Result:
```text
Passed
```
## Checklist
- [x] Implemented checkpoint/resume support for `extract_community`.
- [x] Implemented checkpoint/resume support for `resolve_entities`.
- [x] Avoided touching unrelated API behavior.
- [x] Added unit tests for the new checkpoint helper logic.
- [x] Verified Python syntax compilation.
- [x] Ran related GraphRAG unit tests successfully.
- [x] Ran `git diff --check`.
- [ ] Ran full project test suite.
---------
Co-authored-by: Wang Qi <wangq8@outlook.com>
### What problem does this PR solve?
Display intl `base_url` for Tongyi-Qianwen
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
### What problem does this PR solve?
fix: resolve issue where some models do not use modelInfo parameter
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
### What problem does this PR solve?
OceanBase could start without the `ragflow` tenant, so RAGFlow failed to
connect with `root@ragflow`. This PR adds a safe startup reconcile step
and documents the required host limits before using OceanBase.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
- [x] Documentation Update
## Summary
The embedding branch of `verify_api_key` was missing `await` on
`asyncio.wait_for(...)`, so valid embedding-only providers always failed
API-key verification.
## Root cause
`arr, tc = asyncio.wait_for(...)` (no `await`) returns a coroutine;
unpacking it raises `TypeError: cannot unpack non-iterable coroutine`,
which the `except` swallows as a failure. The chat branch (`await
asyncio.wait_for(check_streamly())`) and rerank branch (`arr, tc = await
asyncio.wait_for(...)`) already `await` correctly.
## Fix
```diff
- arr, tc = asyncio.wait_for(
+ arr, tc = await asyncio.wait_for(
asyncio.to_thread(mdl.encode, ["Test if the api key is available"]),
timeout=timeout_seconds,
)
```
## Files changed
- `api/apps/services/provider_api_service.py`
## Verification
- `ruff check` — clean
- Fix mirrors the already-correct chat/rerank branches in the same
function. Local full pytest not run (heavy RAG deps); CI validates.
## Note
Implemented with LLM assistance (model: claude-opus-4-8).
Closes#15619
Co-authored-by: dearsishs <MCarter112116@outlook.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
## Summary
`SearchService.get_detail` crashed with `AttributeError` (HTTP 500) when
no matching row existed, because it called `.first().to_dict()` before
the `if not search` guard — making that guard dead code.
## Root cause
`.first()` returns `None` when the query matches nothing (deleted search
app, or joined `User` not `VALID`). `None.to_dict()` raises before the
guard runs.
## Fix
```diff
.first()
- .to_dict()
)
if not search:
return {}
- return search
+ return search.to_dict()
```
Guard the `None` first, then serialize — restoring the intended `{}`
"not found" return that every caller (`search_api`, `bot_api`,
`chat_api`, `dataset_api_service`) already handles.
## Files changed
- `api/db/services/search_service.py`
## Verification
- `ruff check` — clean
- Logic: `.first()` → `None` now hits `return {}` instead of
`None.to_dict()`. Local full pytest not run (heavy RAG deps); CI
validates.
## Note
Implemented with LLM assistance (model: claude-opus-4-8).
Closes#15621
Co-authored-by: dearsishs <MCarter112116@outlook.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
## Problem
`TestXiaomiNewModelWithCustomDefaultTransport` panics on Go 1.25:
```
panic: interface conversion: http.RoundTripper is models.roundTripperFunc, not *http.Transport
```
In Go 1.25, `http.DefaultTransport` is no longer `*http.Transport`, so
the unchecked type assertion in `NewXiaomiModel` panics when the test
replaces it with a `roundTripperFunc`.
## Fix
Use a safe type assertion with fallback to a new `http.Transport`,
matching the pattern already used in `modelscope.go`.
## Verification
```bash
go test -run TestXiaomiNewModelWithCustomDefaultTransport ./internal/entity/models/...
# PASS
```
Internal contributors only.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
### What problem does this PR solve?
`is_english()` in `rag/nlp/__init__.py` compiles a **single-character**
regex class and `fullmatch`es it against each item:
```python
pattern = re.compile(r"[`a-zA-Z0-9\s.,':;/\"?<>!\(\)\-]") # no quantifier
...
eng = sum(1 for t in texts if pattern.fullmatch(t.strip()))
```
For a **string** argument the text is first split into single characters
(`texts = list(texts)`), so each `fullmatch` sees one character and
works. But for a **list** argument each item is a whole multi-character
string, and `fullmatch` of a one-character pattern against a
multi-character string always fails — so `is_english()` returns `False`
for **any** list, regardless of content.
```python
is_english("This is English") # True (ok)
is_english(["The quick brown fox jumps.", "Hello world."]) # False (bug — should be True)
is_english(["这是中文。"]) # False (right answer, wrong reason)
```
Many call sites pass lists and were therefore silently always-`False`,
e.g.:
- `rag/llm/chat_model.py:1088`, `rag/llm/cv_model.py:168,1155` —
`is_english([ans])` when an answer is truncated at `max_tokens`, so an
English reply gets the Chinese "······由于长度的原因，回答被截断了，要继续吗？" continuation
suffix instead of the English one.
- `rag/app/book.py` — `remove_contents_table(...,
eng=is_english([...sections...]))`, so English books have their contents
table stripped in Chinese mode.
- `common/doc_store/es_conn_base.py:339`,
`rag/utils/opensearch_conn.py:733` — `is_english(txt.split())` in
highlight handling.
- plus `rag/app/qa.py`, `rag/flow/parser/utils.py`,
`common/doc_store/infinity_conn_base.py`.
### Fix
Add a `+` quantifier so an all-English multi-character item matches:
```python
pattern = re.compile(r"[`a-zA-Z0-9\s.,':;/\"?<>!\(\)\-]+")
```
The string path is unchanged (single characters still match) and
non-English lists still return `False`. Adds
`test/unit_test/rag/test_is_english.py`; the two list cases fail before
this change and pass after.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
Used the Claude CLI while working on this.
### What problem does this PR solve?
Support model list for VolcEngine.
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
## What problem does this PR solve?
Go test files are never compiled in CI — only production binaries via
`go build`. This allowed a missing `"sort"` import in
`metadata_filter_test.go` to be merged without detection.
## Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
- [x] New Feature (non-breaking change which adds functionality)
## Changes
- Add `go test -count=1 ./internal/...` step after Go build in CI
workflow
- Fix missing `"sort"` import in `metadata_filter_test.go` (pre-existing
compile error)
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
## Summary
Normalizes Qwen model-family names before reasoning extraction so
provider-prefixed Qwen models use the existing `<think>...</think>`
fallback.
## Summary
- keep MiniMax chat calls in non-streaming mode and streaming calls in
SSE mode
- make MiniMax model listing and connection checks use a bodyless GET
/v1/models
- add focused MiniMax request/response regression tests
### What problem does this PR solve?
**Verified from CLI**
```
RAGFlow(user)> chat with 'mimo-v2.5@test@xiaomi' message 'who r u'
Answer: Hello! I'm MiMo-v2.5, a large language model developed by Xiaomi's LLM Core Team. You can think of me as a friendly AI assistant ready to help you answer questions, have conversations, or work on creative tasks. My context window can handle up to 1 million tokens, so we can dive into pretty long discussions or documents if you'd like. What can I help you with today?
Time: 3.831830
RAGFlow(user)> stream chat with 'mimo-v2.5@test@xiaomi' message 'who r u'
Answer: there! I'm MiMo-v2.5, an AI assistant created by the Xiaomi LLM Core Team. I'm here to chat, help out, answer questions, or just have a friendly conversation. Think of me as a helpful buddy with a pretty big memory (1 million tokens worth!). What can I do for you today?😊
Time: 2.421630
RAGFlow(user)> think chat with 'mimo-v2.5@test@xiaomi' message 'who r u'
Thinking: The user is asking a simple question about who I am. According to my system prompt, I should:
- Identify myself as **MiMo-v2.5**
- State that I was developed by the **Xiaomi LLM Core Team**
- Answer in first person and be warm and conversational
Answer: Hey there! 👋
I'm **MiMo**, an AI assistant created by the **Xiaomi LLM Core Team**. Think of me as a friendly chat buddy who's here to help you with all sorts of questions and tasks!
I love having conversations, answering questions, brainstorming ideas, and helping people figure things out. Whether you want to chat, need help with something specific, or just want to explore ideas together — I'm here for it! 😊
What can I help you with today?
Time: 6.651589
RAGFlow(user)> tts with 'mimo-v2.5-tts@test@xiaomi' text 'hello? show yourself' play format 'wav' param '{"voice": "Chloe"}'
SUCCESS
RAGFlow(user)> asr with 'mimo-v2.5-asr@test@xiaomi' audio './internal/test.wav' param '{"language": "zh"}'
+------------------------------------------------------------------------------------------------------------------------+
| text |
+------------------------------------------------------------------------------------------------------------------------+
| 1 The examination and testimony of the experts enabled the commission to conclude that five shots may have been fired. |
+------------------------------------------------------------------------------------------------------------------------+
```
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
- [x] Refactoring
## Summary
- keep Moonshot chat calls in non-streaming mode and streaming calls in
SSE mode
- make Moonshot model listing and balance checks use bodyless GET
requests
- add focused Moonshot request/response regression tests
### What problem does this PR solve?
Fix LM-Studio provider connection verification so embedding checks await
the async wrapper correctly and log the full traceback on failures.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
### What problem does this PR solve?
Fix: Add a waiting status to the messages on the chat page.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
### What problem does this PR solve?
fix: Resolve error when checking pipeline parsing result
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
### What problem does this PR solve?
Table parser metadata aggregation previously only ran when
`table_column_mode` was set to `manual`. In auto mode (default), all
columns default to `"both"` role, meaning they should also be aggregated
into document-level metadata for UI/chat filters. Additionally, the task
snapshot could be stale — `table_column_names` are written to KB
`parser_config` during `chunk()` but the task may have been created
before that.
Changes:
- Renames `aggregate_table_manual_doc_metadata` →
`aggregate_table_doc_metadata`
- Supports both `"manual"` and `"auto"` `table_column_mode` (defaults to
`"auto"`)
- Reloads `table_column_names` from KB DB when missing from task
snapshot
- Removes the manual-only guard in `task_executor` and refactored
`post_processor`
- Updates all tests with new function name and adds auto mode test cases
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
### What problem does this PR solve?
This PR adds `ModelMeta` implementations for four additional LLM/RAG
ecosystem platforms, building on the ModelMeta infrastructure introduced
in #15711.
Currently, only `Ollama` and `VolcEngine` have `ModelMeta` classes that
enable remote model list fetching. This PR extends that support to four
more platforms.
### Changes
Added four new `ModelMeta` subclasses in `rag/llm/model_meta.py`:
| Platform | `_FACTORY_NAME` | Has model list | Has full model info |
Approach |
|----------|-----------------|----------------|---------------------|----------|
| **Xinference** | `"Xinference"` | ✅ | ✅ | Parses `model_type` and
`context_length` from `/v1/models` response. Maps 6 model types
(LLM/embedding/rerank/image/TTS/speech2text). |
| **LocalAI** | `"LocalAI"` | ✅ | ✅ | Uses Ollama-compatible `GET
/api/tags` + `POST /api/show` endpoints. Returns capabilities
(completion/embedding/vision/tools/thinking) and
`general.context_length`. |
| **BaiduYiyan** | `"BaiduYiyan"` | ✅ | ✅ | Uses Qianfan SDK static
model catalog + `get_model_info()` for `max_input_tokens`. Returns 60
models (56 chat + 4 embedding) with real context lengths. |
| **Tencent Cloud** | `"Tencent Cloud"` | ❌ | ❌ | `NotImplementedError`
— uses SDK-based SID/SK HMAC signing, no model list REST API available.
|
All classes are automatically discovered and registered via the existing
`__init__.py` mechanism — no additional configuration needed.
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
### What problem does this PR solve?
Browser parsed sys.query from prompts but never called set_input_value,
so node_finished inputs displayed null in the agent orchestration run
log.
Additionally, Browser’s tenant-model path could trigger unsupported
structured-output modes (response_format/tool_choice) for some
OpenAI-compatible providers (notably DeepSeek thinking models), causing
step failures.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
Fixes#15413
### What problem does this PR solve?
In **Settings → Model providers**, the *Available models* panel lets you
filter
providers by model type (All, LLM, Embedding, Rerank, TTS, ASR, VLM, …),
but the
filter tags gave no hint of how many providers fall under each type.
Users had to
click a tag to find out, and an empty category looked identical to a
populated one.
This PR adds a count to each filter tag in `AvailableModels`:
- The **All** tag shows the total number of providers currently listed.
- Each model-type tag shows how many providers offer that model type.
- Counts respect the active search term, so the badge always matches the
number of
cards shown once that tag is selected.
- Each provider is counted once per model type (deduplicated via a
`Set`), so a
provider that lists the same type more than once isn't double-counted.
Counts are rendered with `tabular-nums` for stable width and dimmed via
`opacity-60`
so they read as secondary to the label. No API changes; the existing
filter logic is
untouched — this is purely an additive UI affordance.
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
### What problem does this PR solve?
Follow-up to #15393. After #15393 fixed the OpenSearch `search()`
signature and
the doc-meta mapping, document metadata still renders as **"0 fields"**
for every
document on the OpenSearch backend (`DOC_ENGINE=opensearch`).
**Root cause.** `OSConnection.insert()` pops `id` out of the document
before
indexing:
meta_id = d_copy.pop("id", "") # id used as _id, then DROPPED from
_source
so the stored `_source` never contains an `id` field. But the doc-meta
read path
filters and sorts on that field:
- `DocMetadataService.get_metadata_for_documents()` builds
`condition = {"kb_id": kb_id, "id": doc_ids}` -> `OSConnection.search()`
emits
`Q("terms", id=doc_ids)` (a term query on the `id` field), and
- `_search_metadata()` sorts with `order_by.asc("id")`.
With `id` absent from `_source`, the terms filter matches nothing, so
`get_metadata_for_documents()` returns an empty map and the UI shows "0
fields"
-- even though the metadata was written correctly (it is visible via a
kb_id-only query).
`ESConnection.insert()` already keeps `id` (`d_copy.get("id", "")`) with
the
comment *"also keep 'id' as a regular field for sorting"*. This is a
plain
OpenSearch-only divergence (`pop()` vs `get()`).
### Fix
Mirror Elasticsearch: use `get("id")` instead of `pop("id")` so `id`
survives in
`_source`. The doc-meta mapping already declares `id` as `keyword`, so
the field
is searchable/sortable once populated.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
### Affected backends
OpenSearch only. Elasticsearch already keeps `id`; Infinity / OceanBase
unaffected.
### How to reproduce
1. `DOC_ENGINE=opensearch`, create a KB, upload/parse a document, set
metadata.
2. Open the document list -> every document shows "0 fields" (the
metadata exists
in the `ragflow_doc_meta_*` index but its `_source` has no `id` field).
### Risk & backward compatibility
`insert()` is shared with the main chunk index; keeping `id` in
`_source` brings
OpenSearch in line with Elasticsearch (which already does this), so it
is parity,
not new behavior. No default / ES / Infinity / OceanBase behavior
change.
Note: affects new inserts only. Existing `ragflow_doc_meta_*` indices
created
before this change have no `id` in `_source`; re-sync metadata, or
backfill once
with `_update_by_query` (`ctx._source.id = ctx._id`).
### Test plan
- [ ] OpenSearch: after the fix the document list shows correct metadata
field
counts (not "0 fields"); metadata filter/sort by id works.
- [ ] Elasticsearch regression: unchanged.
### What problem does this PR solve?
`RAGFlowExcelParser.html()` iterates `(len(rows) - 1) // chunk_rows + 1`
times. `rows[0]` is the header, so `len(rows) - 1` is the data-row
count. When that count is an exact multiple of `chunk_rows`, the `+ 1`
over-counts by one: the final iteration's data slice is empty, but the
header row is still appended — producing a chunk that contains only the
table header and no data.
This is reachable via `rag/app/naive.py` (`html4excel`, `chunk_rows=12`)
and `rag/app/one.py`. A sheet with 12/24/36… data rows (or 256/512… with
the default `chunk_rows=256`) produces an extra
`<table><caption>…</caption><tr><th>…</th></tr></table>` chunk. It is
non-empty, so it passes the `if _` filter and gets indexed as a real
(empty) chunk.
| data rows (chunk_rows=12) | before | after |
|---|---|---|
| 12 | 2 chunks (1 header-only) | 1 |
| 24 | 3 chunks (1 header-only) | 2 |
| 13 | 2 (unchanged) | 2 |
### Fix
Iterate `ceil(n_data / chunk_rows)` times instead of `n_data //
chunk_rows + 1`. Adds
`test/unit_test/deepdoc/parser/test_excel_parser.py`; the
header-only-chunk cases fail before this change and pass after.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
Used the Claude CLI while working on this.
## Summary
Closes#15720
`FulltextQueryer.paragraph` normalized its `content_tks` token string
with `[c.strip() for c in content_tks.strip() ...]`, which iterates the
string **character by character** — `"machine learning model"` becomes
20 single characters instead of 3 tokens. Those single chars are fed to
`tw.weights(..., preprocess=False)`, producing meaningless term weights
and a garbage `MatchTextExpr`.
`paragraph()` backs `Dealer.tag_content` (the KB auto-tagging feature),
so tag retrieval/scoring is silently broken for tag-enabled knowledge
bases. Every other method in this file tokenizes with `.split()` — this
is a `.strip()`-vs-`.split()` typo.
## Change
- `rag/nlp/query.py` — change `content_tks.strip()` to
`content_tks.split()` in the `paragraph` token-normalization line.
## Why it's safe
- The caller passes a space-separated token string; `.split()` recovers
the real tokens, matching the contract of `tw.weights` and the
`.split()` tokenization used by the sibling methods (`similarity`,
`question`).
- No behavior depends on the per-character expansion.
## Verification
- `python -m py_compile rag/nlp/query.py` — OK.
- Demonstrated: `"machine learning model"` → 20 single-character entries
before, 3 real tokens after. No test references `paragraph`.
Co-authored-by: seekmistar01 <seekmistar01@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
## What problem does this PR solve?
Implements `FetchChunkVectors` — the infrastructure needed to hydrate
chunk embedding vectors on demand. This is a prerequisite for
`insert_citations` (citation insertion in the `searchbots/ask`
endpoint), matching the Python `Dealer.fetch_chunk_vectors` pattern.
Without this, citation insertion cannot compute answer-vs-chunk vector
similarity.
## Type of change
- [x] New Feature (non-breaking change which adds functionality)
## Changes
### New Function
- `FetchChunkVectors(engine, chunkIDs, tenantIDs, kbIDs, dim)` — fetches
embedding vectors for a set of chunk IDs
- Consumer-side `vectorFetcher` interface with only `Search` + `GetType`
methods
- Both `*elasticsearchEngine` and `*infinityEngine` implicitly satisfy
the interface
### Engine Behavior
- **ES**: queries by chunk ID list via `Search` with filter `{"id":
chunkIDs}`, parses tab-separated `q_N_vec` string format
- **Infinity / OceanBase**: skips the round-trip (vectors already
shipped with chunks)
- **Degrades gracefully**: engine errors return zero vectors — citation
insertion will use placeholders instead of failing
### Vector Parsing
- Handles ES tab-separated string format (`"0.1\t0.2\t0.3"`)
- Handles `[]float64` and `[]interface{}` formats
- Returns zero vector for wrong-dimension or unparseable input
### Bug Fix
- `metadata_filter_test.go`: add missing `"sort"` import (pre-existing
build break)
### Tests
- 12 unit tests: empty input, Infinity/OceanBase skip, ES string vector,
ES float slice, ES interface slice, search error degradation, missing
chunk → zero, wrong dimension → zero, parse edge cases
## Files Changed
| File | Change |
|------|--------|
| `internal/service/chunk_vector.go` | New — FetchChunkVectors + parse
helpers |
| `internal/service/chunk_vector_test.go` | New — 12 tests |
| `internal/service/metadata_filter_test.go` | Fix missing `"sort"`
import |
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
### What problem does this PR solve?
Refine the stream parsing for `<think>` / `</think>` so MiniMax and
DeepSeek-style chunking both flush in the right order without mixing
think and answer buffers.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
### What problem does this PR solve?
feat：Unify the 'Add Model Provider' modal
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
- [x] Refactoring
### What problem does this PR solve?
As Title
Codes were tested by Postman
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
### What problem does this PR solve?
Fixes the OpenSearch side of #10747: hybrid search drops the keyword
(BM25) leg and
ends up doing plain vector search.
When a search has both a text and a vector leg, `OSConnection.search()`
throws the text
query away:
del q["query"]
q["query"] = {"knn": knn_query}
The text clause only stays on as a filter inside the knn query, so it
narrows the
candidate set but doesn't count towards scoring. So hybrid search on
OpenSearch behaves
like plain vector search, unlike the Elasticsearch backend.
What I changed:
- when both legs are present, send a real hybrid query
`{"hybrid": {"queries": [bm25, {"knn": ...}]}}` and let a
normalization-processor
search pipeline score and combine the two legs
- only the actual filters (kb_id, available_int, ...) go in the knn
filter, not the
text must clause
- create the pipeline on startup if it's missing, so there's no separate
provisioning
step. name and weights can be set under `os:` in service_conf.yaml, or
via
`OS_HYBRID_PIPELINE`; defaults are `ragflow_hybrid_pipeline` and `[0.5,
0.5]`
- normalization-processor needs OpenSearch 2.10+. on older clusters, or
when the
pipeline can't be created, log a warning and fall back to vector-only
instead of
pointing at a pipeline that doesn't exist
This is only the hybrid-search fix; `create_doc_meta_idx` is already on
main.
Testing (there's no OpenSearch path in CI): added a unit test
(`test/unit_test/rag/utils/test_opensearch_hybrid_search.py`, no
services needed) that
checks the query built in each case — hybrid + pipeline param for
text+vector, plain knn
for vector-only, plain bool for text-only, the knn filter never carrying
the text
query_string, and the vector-only fallback when the pipeline isn't
available. Also ran
it against a real OpenSearch 2.19.1 container with a doc that matches
the keyword but
sits outside the knn top-k: pure knn returns `['D1','D2','D5']` (keyword
doc missing),
the hybrid query returns `['A','D1','D2','D5']` (keyword doc present).
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
Signed-off-by: Danut Matei <matei.danut.dm@gmail.com>
## What problem does this PR solve?
Implements `POST /api/v1/searchbots/retrieval_test` in the Go API
server, aligning with the Python `bot_api.py` counterpart. Also applies
security hardening and consistency fixes discovered during CTO-level
code review:
- **Missing endpoint**: `retrieval_test` was not available in Go,
requiring Python fallback
- **Security**: Both `chunkHandler` and `searchBotHandler` leaked
`err.Error()` to API consumers
- **Python alignment**: Default values, empty question handling, and
`top_k <= 0` validation differed from Python behavior
- **Test gaps**: `chunkHandler.RetrievalTest` had zero unit tests;
several edge cases uncovered
## Type of change
- [x] New Feature (non-breaking change which adds functionality)
- [x] Bug Fix (non-breaking change which fixes an issue)
- [x] Refactoring
## Summary
### New Endpoint
- `POST /api/v1/searchbots/retrieval_test` — retrieval test with full
field support (page, size, top_k, use_kg, cross_languages, keyword,
similarity_threshold, vector_similarity_weight)
### New Type
- `common.StringSlice` — JSON type that accepts both `"kb1"` and
`["kb1", "kb2"]`, matching Python API flexibility
### Security
- Both `searchBotHandler` and `chunkHandler` now use `common.Warn()` +
generic error messages instead of leaking `err.Error()` to API consumers
- All error responses include consistent `"data": nil` shape
- `chunkHandler.RetrievalTest` uses interface-based DI (`chunkService`)
to enable testability
### Python Alignment
- Handler-level defaults align with Python `bot_api.py` (page=1,
size=30, top_k=1024, similarity_threshold=0.0,
vector_similarity_weight=0.3)
- `top_k <= 0` validation matching Python behavior
- Empty/whitespace question returns 200 + empty result (matches
`chunk_api.py`)
- `chunkHandler` `Datasets` field uses `common.StringSlice` for
string-or-array flexibility
### Refactoring
- `ChunkServiceIface` → `ChunkRetriever`, `chunkSvcIface` →
`chunkService` (Go-conventional naming)
- Extracted `applyRetrievalDefaults`, `toRetrievalServiceRequest` from
handler body
- Regex moved to package-level var in `parseRelatedQuestions`
- `service.RetrievalTestRequest.Datasets` type changed to
`common.StringSlice`
- `chunkHandler` now uses consumer-side interface for DI
### Tests
- 37 unit tests across both handlers: auth, validation, defaults,
StringSlice edge cases, empty/whitespace KbID, service errors, JSON
format, `top_k <= 0`, field mapping verification
## Files Changed
| File | Change |
|------|--------|
| `cmd/server_main.go` | Wire new handler + chunkService +
difyRetrievalHandler |
| `internal/common/json_types.go` | New StringSlice type |
| `internal/common/json_types_test.go` | StringSlice tests |
| `internal/handler/chunk.go` | Interface-based DI, security, Python
alignment, defaults |
| `internal/handler/chunk_test.go` | New — 9 comprehensive tests |
| `internal/handler/searchbot.go` | New endpoint + refactoring + `top_k
<= 0` validation |
| `internal/handler/searchbot_test.go` | 18 tests covering all edge
cases |
| `internal/router/router.go` | Register new route +
difyRetrievalHandler |
| `internal/service/chunk.go` | Datasets type → StringSlice, Question
binding relaxed |
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
### What problem does this PR solve?
Fix: The embedded website floating component on the agent page does not
display citations.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
### What problem does this PR solve?
Normalize agent session chunk references so they are mapped through a
dedicated helper instead of duplicating the field extraction inline.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
### What problem does this PR solve?
Fix: The time zone is not displayed on the personal profile page.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
### What problem does this PR solve?
Closes#15611.
RAGFlow's fallback reasoning parser only recognized the exact model
family `qwen3`. For provider-prefixed Qwen model names such as
SiliconFlow's `qwen/qwen3-8b`, the derived model class can be
`qwen/qwen3`, so inline `<think>...</think>` content was not split from
the visible answer when `reasoning_content` was absent.
This PR normalizes model-family detection before fallback reasoning
extraction, keeps the parser nil-safe, and adds focused tests for Qwen3
variants plus Gitee and SiliconFlow chat responses.
It also makes SiliconFlow propagate `ChatConfig.Thinking` into the chat
request body, matching the existing Gitee behavior, so Qwen thinking
mode is actually enabled when requested.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
- [x] Refactoring
### Validation
- `/root/go/bin/gofmt -l internal/entity/models/common.go
internal/entity/models/common_test.go
internal/entity/models/reasoning_family_provider_test.go
internal/entity/models/siliconflow.go`
- `git diff --check`
- `/root/go/bin/go test ./internal/entity/models -run
'Test(NormalizeModelFamily|GetThinkingAndAnswer|GiteeChatExtractsQwenThinkingFromInlineContent|SiliconflowChatExtractsProviderPrefixedQwenThinkingFromInlineContent)'
-vet=off -count=1`
Note: the full package command `/root/go/bin/go test
./internal/entity/models -vet=off -count=1` now runs locally, but it
currently fails on an unrelated existing
`TestAstraflowEmbedReturnsNoSuchMethod` panic in
`internal/entity/models/astraflow.go:482`.
### What problem does this PR solve?
Closes#15428
The hybrid score in `rag/nlp/search.py` (`rerank_by_model`) blends
reranker similarity with token similarity on a fixed `[0, 1]` scale:
```python
return tkweight * np.array(tksim) + vtweight * vtsim + rank_fea # tkweight=0.3, vtweight=0.7
```
The reranker implementations did not agree on that scale. Only three of
roughly 17 providers normalized their output, and `NvidiaRerank`
returned raw, unbounded logits. Weighted at `0.7`, a negative logit
could push a genuinely relevant chunk below pure keyword matches, and
its magnitude swamped `tksim`, which lives in `[0, 1]`. The practical
effect was that the same query produced differently scaled scores
depending on the configured reranker, and logit based providers degraded
retrieval quality instead of improving it.
This PR enforces a single scoring contract in one place:
- `Base.similarity` is now the only public entry point. It
short-circuits empty input and guarantees a normalized result. Each
provider implements its raw scoring in `_compute_rank`, which removes
sixteen duplicated empty input guards and the three scattered
normalization calls.
- Normalization is range aware. Providers that already return calibrated
`[0, 1]` relevance scores (Cohere, Jina, Voyage, and others) keep their
absolute magnitudes, so `similarity_threshold` filtering and the
reported `vector_similarity` stay meaningful. Only out-of-range output
such as NVIDIA logits is min-max rescaled into `[0, 1]`.
- The twelve leftover `[DEBUG ...]` prints in `rerank_by_model`,
introduced in #14231, are removed. They ran on every retrieval, added
per chunk overhead, and leaked queries, keywords, and document content
to stdout and logs.
A new regression suite in
`test/unit_test/rag/llm/test_rerank_normalization.py` covers logit
rescaling (positive, negative, and flat batches), preservation of
already calibrated scores, ordering, empty input handling, and the per
provider HTTP path. It also asserts that no provider overrides
`similarity()`, so the contract cannot silently drift.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
## Summary
Port the Python `GET /v1/plugin/tools` endpoint to the Go API server.
Listed in the Go-API port checklist of #15240.
Returns the metadata of every embedded LLM tool plugin in the same JSON
shape the Python endpoint emits (camelCase keys preserved), so existing
frontends bind to the Go server without changes.
### What problem does this PR solve?
Closes#15433
Reranked retrieval drops results and returns short pages once pagination
crosses the first candidate block, for the common page sizes 10 and 30.
In `rag/nlp/search.py`, the candidate window (`RERANK_LIMIT`) is rounded
up to a multiple of `page_size` to keep block based pagination aligned,
and then clamped back to 64:
```python
RERANK_LIMIT = math.ceil(64 / page_size) * page_size if page_size > 1 else 1 # e.g. 70 for page_size=10
RERANK_LIMIT = max(30, RERANK_LIMIT)
if rerank_mdl and top > 0:
RERANK_LIMIT = min(RERANK_LIMIT, top, 64) # clamps back to 64, breaking the multiple
```
`RERANK_LIMIT` is used both as the backend block size (`page =
global_offset // RERANK_LIMIT`) and as the modulus that slices a page
out of a reranked block (`begin = global_offset % RERANK_LIMIT`). When
it stops being a multiple of `page_size`, the block that gets fetched
and the slice taken from it no longer agree. With `page_size=10` and
`top=1024`, page 7 returns only 4 of 10 results and the head of the next
block is never shown on any page. This happens whenever the result set
spans more than one block, which is the default.
**Fix**
The window math is moved into a small reusable helper,
`Dealer._rerank_window`, which:
- targets a pool of about 64 candidates,
- bounds it by `top` when a reranker is active, and
- always rounds to a whole number of pages, so the window stays an exact
multiple of `page_size`.
The call site becomes a single line, and the alignment invariant now
lives in one documented place. Behavior is unchanged on every path that
was already aligned (the non reranked path and any `top` that already
produced a page multiple).
**Verification**
A simulation of the full retrieval path (per block rerank, similarity
threshold filter, and the exact `page // window` and `offset % window`
math) confirms the fix loses nothing where the old code lost real
results:
```
ps=10 top=1024: new window=70 dropped_valid=0 | old window=64 dropped_valid=16
ps=30 top=1024: new window=90 dropped_valid=0 | old window=64 dropped_valid=66
```
New unit tests in `test/unit_test/rag/test_search_pagination.py` cover
the alignment invariant, cross block pagination (every candidate
surfaced once, in order, no gaps, no short interior pages), the reported
regression, and parity with the old window on the previously correct
paths. All 114 cases pass and `ruff check` is clean.
Fixes the reranked deep pagination data loss described above.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
### Description
This PR syncs the `documentServiceIface` interface and introduces
handler methods for document preview, artifact fetching, and downloading
in the Go API. It also ensures that strict dataset alignment and access
checks are enforced when retrieving or downloading documents.
Furthermore, this PR introduces comprehensive unit tests for both the
newly added Handler and Service methods to ensure robustness and prevent
future regressions.
### Key Changes
* **Router & Handler Integration**:
* Added and wired new API endpoints in `internal/router/router.go`.
* Synchronized the `documentServiceIface` with `GetDocumentArtifact`,
`GetDocumentPreview`, and `DownloadDocument`.
* Implemented handlers for these endpoints in
`internal/handler/document.go`.
* **Access & Validation Enforcement**:
* Refactored `internal/service/document.go` to strictly check if a
document belongs to the requested dataset before allowing downloads or
previews.
* Added robust artifact file sanitization (`sanitizeArtifactFilename`)
and attachment handling (`shouldForceArtifactAttachment`).
* **Comprehensive Unit Testing**:
* **Handler Layer (`internal/handler/document_test.go`)**: Added mock
service implementations and Gin router tests covering success,
not-found, and internal error states for all 3 new endpoints.
* **Service Layer (`internal/service/document_test.go`)**: Added
extensive business logic tests including dataset mismatch checks,
non-existent document checks, and artifact file validation.
### What problem does this PR solve?
Feat：
- Get model list from remote provider.
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
Prepend a leading slash and reject `..` segments so scoped OneDrive
delta queries use `root:/path:/delta` instead of `root:path:/delta`.
Fixes#15500
### What problem does this PR solve?
The OneDrive connector builds Microsoft Graph delta URLs from optional
`config.folder_path`. When users enter a path without a leading slash
(e.g. `Documents/Reports` instead of `/Documents/Reports`), the
connector produces a malformed URL such as
`root:Documents/Reports:/delta`. Per [Microsoft Graph path-based
addressing](https://learn.microsoft.com/en-us/graph/onedrive-addressing-driveitems),
the segment after `root:` must start with `/` (e.g.
`root:/Documents/Reports:/delta`). Sync and validation then fail or
return no documents, which is hard to diagnose from the UI because the
optional folder field does not enforce the format.
This PR normalizes `folder_path` at connector construction time (prepend
`/`, trim whitespace and trailing slashes) and rejects `..` segments
before any Graph request is made.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
## Summary
This PR improves the Go CLI in two areas:
1. It adds batch model management support, allowing multiple models to
be added or removed in a single command.
2. It makes the `dimension` argument optional for the `embed text`
command.
These changes keep the existing single-model and explicit-dimension
behaviors compatible while making the CLI more convenient for common
workflows.
## What Changed
### 1. Batch model add/remove support
The CLI now supports operating on multiple model names provided in a
single quoted string.
Supported commands include:
```
add model 'x1 x2 x3' to provider 'vllm' instance 'test' with tokens 1024 chat think vision, token 2048 chat, token 1024 think vision;
drop model 'x1 x2 x3' from 'vllm' 'test';
remove model 'x1 x2 x3' from 'vllm' 'test';
```
For add model, each config segment after with is matched to the
corresponding model name by position.
Example mapping:
- x1 -> tokens 1024, chat + vision, thinking=true
- x2 -> tokens 2048, chat
- x3 -> tokens 1024, vision, thinking=true
The existing single-model syntax remains supported.
### 2. Optional embedding dimension
Previously, the Go CLI required dimension to be explicitly provided for
embed text.
Before:
embed text 'what is rag' 'who are you' with 'model@test@provider'
dimension 8192;
Now both forms are supported:
embed text 'what is rag' 'who are you' with 'model@test@provider'
dimension 8192;
embed text 'what is rag' 'who are you' with 'model@test@provider';
When omitted, the CLI leaves dimension unset and relies on
provider/backend behavior.
## Tests
Added parser tests covering:
- Multiple models with multiple config segments
- Model type deduplication
- Model/config count mismatch
- Drop/remove multiple models
- Optional embedding dimension parsing
### What problem does this PR solve?
Fix: When adding a chat in the main interface, a warning will
automatically pop up (even if embedding and LLM model have already been
configured).
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
## What
- make `Switch` ignore conditions that have no evaluable items
- add a regression for blank `cpn_id` items falling through to the else
branch
- keep the existing non-empty `and` condition behavior covered
Fixes#15643.
## Verified
- `python -m py_compile agent\component\switch.py
test\unit_test\agent\component\test_switch.py`
- `python -m pytest test\unit_test\agent\component\test_switch.py -q` ->
`2 passed`
- `python -m ruff check agent\component\switch.py
test\unit_test\agent\component\test_switch.py`
- `git diff --check`
I also checked `python -m ruff format --check` on the touched files. It
would reformat pre-existing style in `agent/component/switch.py` beyond
this bug fix, so I kept the patch scoped instead of reformatting the
whole file.
### What problem does this PR solve?
Update Dockerfile and release workflow to use GitHub mirror instead of
Gitee
### Type of change
- [x] Other (please describe): CI
### What problem does this PR solve?
Fixes#15542.
AWS Bedrock support for the Go model provider layer was added in #15166,
but embedding support was intentionally left out of scope and
`BedrockModel.Embed(...)` still returned the `no such method` sentinel.
This PR implements Bedrock text embeddings under the umbrella provider
tracker #14736.
### What this PR includes
- `internal/entity/models/bedrock.go`: implement
`BedrockModel.Embed(...)` through Bedrock Runtime `InvokeModel` with
existing SigV4 auth, region resolution, and runtime URL helpers.
- Titan embeddings: supports `amazon.titan-embed-text-v1` and
`amazon.titan-embed-text-v2:0`; v2 forwards `EmbeddingConfig.Dimension`
as `dimensions` when provided, while v1 keeps the payload minimal.
- Cohere embeddings: supports `cohere.embed-english-v3`,
`cohere.embed-multilingual-v3`, and `cohere.embed-v4:0`; batches input
texts and maps returned vectors to RAGFlow `EmbeddingData` in input
order.
- `conf/models/bedrock.json`: adds the `embedding` URL suffix (`invoke`)
and Bedrock embedding model entries.
- `internal/entity/models/bedrock_test.go`: adds unit tests for Titan,
Cohere, typed Cohere responses, validation, empty input, unsupported
models, and HTTP error propagation.
Reference docs:
- Bedrock InvokeModel API:
https://docs.aws.amazon.com/bedrock/latest/APIReference/API_runtime_InvokeModel.html
- Titan Text Embeddings:
https://docs.aws.amazon.com/bedrock/latest/userguide/titan-embedding-models.html
- Cohere Embed models on Bedrock:
https://docs.aws.amazon.com/bedrock/latest/userguide/model-parameters-embed.html
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
### How was this tested?
- [x] `jq empty conf/models/bedrock.json`
- [x] `git diff --check`
- [x] `go test ./internal/entity/models/... -run Bedrock -count=1`
- [x] `go test ./internal/entity/models/... -run '^$' -count=1`
- [x] `go test ./internal/entity/models/... -run Bedrock -race -count=1`
Note: `go test ./internal/entity/models/... -count=1` currently fails in
unrelated existing Astraflow coverage
(`TestAstraflowEmbedReturnsNoSuchMethod` panics in
`internal/entity/models/astraflow.go`). The Bedrock-specific tests and
compile-only package check pass.
## Summary
Ports the MCP (Model Context Protocol) server management endpoints that
power `web/src/pages/user-setting/mcp/` from Python
(`api/apps/restful_apis/mcp_api.py`) to Go. There were no MCP routes in
the Go server before this change.
Closes#15275 (subtask of #15240).
## Endpoints implemented (base path `/api/v1`)
| Method | Path | Description |
|--------|------|-------------|
| GET | `/mcp/servers` | List tenant servers (keyword / order /
pagination) |
| POST | `/mcp/servers` | Create a server |
| GET | `/mcp/servers/{mcp_id}` | Get one (`?mode=download` exports
config) |
| PUT | `/mcp/servers/{mcp_id}` | Update a server |
| DELETE | `/mcp/servers/{mcp_id}` | Delete a server |
| POST | `/mcp/import` | Bulk import from JSON config |
| POST | `/mcp/servers/{mcp_id}/test` | Connect + list tools (see notes)
|
## Implementation
Follows the existing `handler → service → dao` layering (per PR #14790):
- **entity** (`internal/entity/mcp.go`): added `MCPServerType` constants
and `IsValidMCPServerType` over the existing `MCPServer` model.
- **dao** (`internal/dao/mcp.go`): new `MCPServerDAO` with tenant-scoped
CRUD, a keyword filter, and a **whitelisted order-column map** (guards
against SQL injection via the caller-supplied `orderby`).
- **service** (`internal/service/mcp.go`): new `MCPService` —
list/get/export/create/update/delete/import/test — mirroring
`MCPServerService` and the `mcp_api` request validation, with sentinel
errors for clean code mapping.
- **handler** (`internal/handler/mcp.go`): new `MCPHandler` with the
seven handlers and Python-compatible response codes.
- **router / server_main**: registered the `/mcp` group and wired the
handler.
## Deviations from Python (documented in code)
1. **Bulk import is at `POST /mcp/import`, not `/mcp/servers/import`.**
gin (v1.9.1) cannot register a static segment and a path param at the
same tree node, so `/mcp/servers/import` would collide with
`/mcp/servers/:mcp_id` and panic at startup. The frontend should call
`/mcp/import`.
2. **No live tool discovery on create/update/import.** The Python path
runs `get_mcp_tools` over SSE / streamable-HTTP and stores
`variables.tools`. The Go server has no MCP client yet, so these persist
`variables`/`headers` but leave `variables.tools` unpopulated.
3. **`/test` returns a data error (`ErrMCPTestUnsupported`)** until a Go
MCP client lands. Per the issue, the live-connection path is scoped as a
follow-up; the handler still validates `url` + `server_type`.
## Testing
- Added `internal/service/mcp_test.go` covering `IsValidMCPServerType`
and the `TestServer` validation/short-circuit paths (no DB required).
- No Go toolchain was available in the dev environment, so `go build
./...` / `go vet ./...` verification is left to CI.
## Follow-ups
- Go MCP client (SSE / streamable-HTTP) to enable live tool discovery
and the real `/test` behavior.
- Reconcile the `/mcp/import` vs `/mcp/servers/import` path with the
frontend.
---------
### What problem does this PR solve?
Closes#15461.
RAGFlow had no way to ingest Salesforce CRM data, so support / sales
teams couldn't ground responses on live Accounts, Contacts,
Opportunities, Cases, or Knowledge articles. This adds a first-class
Salesforce data source connector that authenticates against a Connected
App via OAuth 2.0 client-credentials, queries selected SObjects via
SOQL, and turns each record into an indexable document with incremental
sync.
**Highlights**
- `common/data_source/salesforce_connector.py`: new
`SalesforceConnector` (`CheckpointedConnectorWithPermSync` +
`SlimConnectorWithPermSync`).
- OAuth 2.0 client-credentials flow; canonical `instance_url` from the
token response so multi-pod orgs route correctly.
- Per-object `SystemModstamp` cursor stored in
`SalesforceCheckpoint.cursors` — a failure mid-object doesn't rewind
sibling objects, and re-syncs only fetch changed rows.
- Deterministic record-to-text formatter (sorted keys) so SOQL field
reordering on the server doesn't mark every row "changed" on each poll.
- `_get_json` raises on non-2xx so 429 / 5xx never silently advance the
checkpoint past missing data.
- `Knowledge__kav` is in the default object set but is skipped silently
when the org doesn't have Salesforce Knowledge enabled (404 on
describe).
- Slim-doc IDs are scoped as `<Object>/<Id>` so prune deletes can't
collide across object types.
- `common/constants.py`, `common/data_source/config.py`,
`common/data_source/__init__.py`: register `salesforce` in `FileSource`
/ `DocumentSource` and export `SalesforceConnector`.
- `rag/svr/sync_data_source.py`: new `Salesforce(SyncBase)` class routed
through `load_from_checkpoint` (poll_source would re-walk every object
each run) and added to `func_factory`.
- Frontend:
- `web/src/pages/user-setting/data-source/constant/index.tsx`: new
`DataSourceKey.SALESFORCE`, form fields (instance URL, client ID/secret,
objects, api_version, batch size), `syncDeletedFiles` capability,
default form values, and tile entry with the new icon.
- `web/src/locales/{en,zh}.ts`: description + per-field tooltips.
- `web/src/assets/svg/data-source/salesforce.svg`: 48x48 brand-style
icon to match the other Microsoft / cloud tiles.
**Verification**
- `npm run build` (vite + esbuild) passes (1m 26s).
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
### What problem does this PR solve?
This fixes duplicated post-think text in streamed chat responses. When
the model emits text immediately after `</think>`, the stream state now
advances its cursor correctly so the same visible prefix is not emitted
twice.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
## Summary
This PR fixes case-sensitivity regressions introduced in #15656 and
consolidates the metadata filtering pipeline by removing the duplicate
`applySingleCondition` adapter layer.
### Bug fixes
1. **contains / not contains**: restored case-insensitive matching (was
lost when `applySingleCondition` was replaced by
`common.MetaFilter.matchValue` which lacked `strings.ToLower`)
2. **not in**: restored case-insensitive matching (was lost for same
reason; uses `strings.EqualFold`)
3. **!= with date filter values**: non-date metadata values now
correctly match the `≠` operator (a non-date value IS not equal to any
date, but was returning false)
### Architecture
4. **Removed `applySingleCondition`** (65 lines) — the inline switch was
a duplicate of `common.MetaFilter` logic. `ApplyMetaFilter` now converts
conditions and delegates to `common.MetaFilter` once per filter set,
eliminating ~25 lines of duplicate AND/OR merge logic.
5. **Added `filterSet`** — O(n+m) hash-map fast path for `in`/`not in`
operators, replacing the O(n*m) linear scan in `matchValue`.
6. **Exported `NormalizeOperator`** from `common` for consistent
operator alias handling.
### Cleanup
7. Removed 18 lines of dead code (`matchValue`'s `in`/`not in` branches
already bypassed by `filterOut` delegation)
8. Fixed orphaned godoc comment for `convertOperator`
9. Fixed incorrect `filterSet` doc comment (claimed "matching EqualFold"
but used `strings.ToLower`)
10. Completed `convertToMetaCondition` operator normalization
documentation
### Testing
- 60 tests (24 service + 36 common), all passing
- New tests: `==`, `≠`, `>`, `<`, `≥`, `≤`, `empty`, `not empty` through
`ApplyMetaFilter`
- New tests: `<`, `≤`, `≠` through `MetaFilter`; `not-in-empty-list`
through `filterSet`
- All 18 `MetaFilter` tests pass; all 10 `filterSet` unit tests pass
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
## Summary
Add `EnrichChunksWithDocMetadata` as a method on `MetadataService` that
attaches document metadata to retrieval chunks in-place. Equivalent to
Python's `enrich_chunks_with_document_metadata()` from
`api/utils/reference_metadata_utils.py`.
### Usage
```go
metadataSvc.EnrichChunksWithDocMetadata(chunks, tenantID, metadataFields)
```
### Changes
- **`service/metadata.go`**: Added `EnrichChunksWithDocMetadata` method
- **`service/enrich_metadata_test.go`** (new): 7 test cases
### Algorithm
1. Collect unique `(kb_id, doc_id)` pairs from chunks
2. Fetch metadata from ES via `SearchMetadata(kbID, tenantID, docIDs)`
3. Attach `document_metadata` field to each matching chunk
4. Optionally filter to specified `metadataFields`
### Testing
All 7 tests pass:
```
=== RUN TestEnrichChunksWithDocMetadata_NoChunks --- PASS
=== RUN TestEnrichChunksWithDocMetadata_EmptyChunks --- PASS
=== RUN TestEnrichChunksWithDocMetadata_EmptyDocID --- PASS
=== RUN TestEnrichChunksWithDocMetadata_DuplicateDocIDs --- PASS
=== RUN TestEnrichChunksWithDocMetadata_MultipleKBs --- PASS
=== RUN TestEnrichChunksWithDocMetadata_WithMetadataFields --- PASS
=== RUN TestEnrichChunksWithDocMetadata_MixedFields --- PASS
```
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
### What problem does this PR solve?
Markdown extraction can split tables row by row when delimiter-based
extraction uses a newline delimiter. That loses table structure during
chunking even though delimiters should still split normally outside
tables.
This PR keeps the follow-up to #15482 intentionally narrow:
- preserve Markdown pipe tables during delimiter-based extraction
- preserve borderless pipe tables during delimiter-based extraction
- preserve multiline HTML tables during delimiter-based extraction
- keep delimiter splitting unchanged outside protected table ranges
Refs #15482
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
### Testing
- `ruff check deepdoc/parser/markdown_parser.py
test/unit_test/deepdoc/parser/test_markdown_parser.py`
- `python3 run_tests.py -t
test/unit_test/deepdoc/parser/test_markdown_parser.py`
- `git diff --check`
## Summary
Port the canvas-template catalogue endpoint to the Go API server. Listed
in the Go-API port checklist of #15240.
Mirrors `list_agent_template` in `api/apps/restful_apis/agent_api.py`:
returns every row from the `canvas_template` table so that the UI can
render the template gallery on the New-Agent screen.
## What
- `internal/dao/canvas_template.go` — new `CanvasTemplateDAO.GetAll()`
ordered by `create_time desc` (newest templates first).
- `internal/service/agent.go` — wire the new DAO into `AgentService` and
expose `ListTemplates() ([]*entity.CanvasTemplate, error)`.
- `internal/handler/agent.go` — new `AgentHandler.ListTemplates` HTTP
handler (auth-gated, mirrors Python `@login_required`).
- `internal/router/router.go` — `agents.GET("/templates",
r.agentHandler.ListTemplates)` registered alongside the existing `GET
/agents`.
- `internal/handler/agent_test.go` — three new tests covering: success
path, empty-list → JSON array (not `null`), and the auth gate.
## Notes
- `CanvasTemplate` entity, GORM tags, and DB migration already exist in
`internal/entity/canvas.go` and `internal/dao/database.go` — no schema
change required.
- The handler coerces a `nil` slice to `[]*entity.CanvasTemplate{}` so
the JSON payload is always an array (the frontend does `data.map(...)`
on it).
## Test plan
- [x] `go vet ./internal/handler ./internal/service ./internal/dao
./internal/router` clean
- [x] Three unit tests added; existing `TestListAgents_Success`
untouched
- [ ] CI runs `go test ./internal/handler` with cgo binding linked
## Related
- Tracker: #15240
QueryRewrite prompt builder and response parser. Zero external
dependencies.
### Functions
- `BuildQueryRewritePrompt`: Renders `minirag_query2kwd` prompt with
query and type pool
- `ParseQueryRewriteResponse`: Parses LLM JSON response with fallback
for markdown and extra text
### Testing
```
=== RUN TestBuildQueryRewritePrompt --- PASS
=== RUN TestParseQueryRewriteResponse_ValidJSON --- PASS
=== RUN TestParseQueryRewriteResponse_MarkdownBlock --- PASS
=== RUN TestParseQueryRewriteResponse_ExtraText --- PASS
=== RUN TestParseQueryRewriteResponse_Invalid --- PASS
=== RUN TestParseQueryRewriteResponse_EmptyEntities --- PASS
```
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
## Summary
- Infer `Content-Type` from the stored document filename on SDK download
routes.
- Covers `GET /api/v1/datasets/<dataset_id>/documents/<document_id>` and
`GET /api/v1/documents/<document_id>`.
- Aligns with REST preview/download via `CONTENT_TYPE_MAP`.
## Test plan
- [x] `pytest
test/testcases/test_http_api/test_file_management_within_dataset/test_doc_sdk_routes_unit.py::TestDocRoutesUnit::test_download_mimetype_from_filename`
- [x] Manual: `curl -sSI` on SDK dataset document download for a PDF;
expect `Content-Type: application/pdf`
Fixes#15112.
### What problem does this PR solve?
Fix:
- VolcEngine adapt to new api_key format
- Save dict api_key as json
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
### What problem does this PR solve?
On some Linux hosts (e.g. x86_64 with enforced POSIX ACL on overlay
storage), the official `elasticsearch` Docker image cannot start because
`docker-entrypoint.sh` needs to create temporary files under `/tmp` for
bash here-documents, while the image ACL grants `user:elasticsearch`
only `r-x` on `/tmp`:
```
/usr/local/bin/docker-entrypoint.sh: line 73/84: cannot create temp file for here-document: Permission denied
```
RAGFlow users hit this when running `docker compose` with the default
`es01` service. See also Refs #284.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
## Summary
Mount a writable `tmpfs` at `/tmp` for the `es01` service so
Elasticsearch entrypoint scripts can run on ACL-enforced environments.
Closes the startup failure described in #284 for non-ARM deployments.
## Changes
- Add `tmpfs: /tmp:mode=1777,size=512m` to `es01` in
`docker/docker-compose-base.yml`
- Document why the mount is required (ES image `/tmp` ACL vs entrypoint
here-documents)
## Test plan
- [x] Verified on Linux (x86_64): `docker run --rm elasticsearch:8.11.3
bash -c 'mktemp'` fails without tmpfs and succeeds with `--tmpfs
/tmp:mode=1777,size=512m`
- [x] Verified `es01` becomes healthy after `docker compose up -d es01`
with this change
- [ ] Upstream maintainers: `docker compose -f
docker/docker-compose-base.yml --profile elasticsearch up -d es01` on a
host where ACL is enforced
Made with [Cursor](https://cursor.com)
Co-authored-by: Cursor <cursoragent@cursor.com>
Add `ResolveReferenceMetadata` to parse `include_metadata` /
`metadata_fields` from request and config payloads.
### Changes
- **New**: `internal/common/reference_metadata.go` — pure function, zero
dependencies
- **New**: `internal/common/reference_metadata_test.go` — 8 test cases
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
## Summary
Change `GetFlattedMetaByKBs` return type from `map[string]interface{}`
to strongly-typed `common.MetaData`.
**Depends on**: #15648 (provides `MetaData`, `MetaValueDocs` types)
### Changes
- `service/metadata.go`: Changed return type, removed type assertions
- `service/metadata_filter.go`: Updated all metadata function signatures
- `service/metadata_filter_test.go` (new): 12 test cases
### Bug fix
`applySingleCondition` used `.([]interface{})` assertions on `[]string`
data, silently breaking operators like `!=`, `contains`, `start with`,
etc.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
### What problem does this PR solve?
Closes#15465.
RAGFlow supports S3, Google Cloud Storage, R2, and OCI as data sources
but not Azure Blob Storage, leaving Azure users without a way to index
container objects into a knowledge base. This adds a first-class Azure
Blob Storage data-source connector — distinct from RAGFlow's existing
Azure storage *backends* (`rag/utils/azure_sas_conn.py`,
`rag/utils/azure_spn_conn.py`) which store RAGFlow's own files.
**Highlights**
- `common/data_source/azure_blob_connector.py`: new `AzureBlobConnector`
(`CheckpointedConnectorWithPermSync` + `SlimConnectorWithPermSync`).
- Uses the existing `azure-storage-blob` dependency (already in
`pyproject.toml`).
- Three auth modes, tried in order of precedence:
1. **Account key** — `account_name` + `account_key` + `container_name`.
2. **Connection string** — `connection_string` + `container_name`.
3. **SAS token** — `container_url` + `sas_token` (same shape as
`RAGFlowAzureSasBlob`).
- ETag fingerprint stored per blob in `AzureBlobCheckpoint.etags` —
unchanged blobs (same ETag as last run) are skipped without a download.
Only new/modified blobs are fetched.
- Optional `prefix` scopes indexing to a virtual folder.
- `validate_connector_settings()` probes `get_container_properties()`
and maps `AuthenticationFailed / 403 / ContainerNotFound` to typed
connector exceptions.
- Slim-doc IDs are blob names so prune reconciles correctly.
- `common/constants.py`, `common/data_source/config.py`,
`common/data_source/__init__.py`: register `azure_blob` in `FileSource`
/ `DocumentSource` and export `AzureBlobConnector`.
- `rag/svr/sync_data_source.py`: new `AzureBlob(SyncBase)` class routed
through `load_from_checkpoint` (ETag fingerprint owns change-detection)
and added to `func_factory`.
- Frontend:
- `web/src/pages/user-setting/data-source/constant/index.tsx`: new
`DataSourceKey.AZURE_BLOB`, auth-mode selector (account key / connection
string / SAS token), all credential fields, prefix + batch-size,
`syncDeletedFiles` capability, default form values, tile entry with
icon.
- `web/src/locales/{en,zh}.ts`: description + per-field tooltips for all
9 new keys.
- `web/src/assets/svg/data-source/azure-blob.svg`: Azure-branded
stacked-cylinders icon.
**Verification**
- `npm run build` (vite + esbuild) passes (37 s).
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
## Summary
`DocumentDAO.GetByIDs()` generated `WHERE id IN ()` for empty/nil ID
slices, which is invalid SQL and would fail on most databases. This PR
adds a nil guard and comprehensive tests.
### Changes
- **Modified**: `internal/dao/document.go` — Added `len(ids) == 0` guard
to `GetByIDs`
- **New**: `internal/dao/document_test.go` — 4 test cases covering
success, empty IDs, nil IDs, and no-match
### Testing
```
=== RUN TestDocumentGetByIDs_Success --- PASS
=== RUN TestDocumentGetByIDs_EmptyIDs --- PASS
=== RUN TestDocumentGetByIDs_NilIDs --- PASS
=== RUN TestDocumentGetByIDs_NoMatch --- PASS
```
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
## Summary
Migrate the metadata filtering utilities `meta_filter` and
`convert_conditions` from `common/metadata_utils.py` to Go as pure
functions with zero external dependencies.
These functions are used by `dify/retrieval`, `openai/chat/completions`,
`document_api`, and `chunk_api` for filtering documents by metadata
conditions.
### Changes
- **New**: `internal/common/metadata_utils.go` — `ConvertConditions()`
and `MetaFilter()` with full operator support
- **New**: `internal/common/metadata_utils_test.go` — 18 test cases
covering all operators and edge cases
### Supported Operators
`=`, `≠`, `>`, `<`, `≥`, `≤`, `contains`, `not contains`, `in`, `not
in`, `start with`, `end with`, `empty`, `not empty`
### Design
- Numeric comparison via `strconv.ParseFloat`
- Date comparison via YYYY-MM-DD format detection
- Case-insensitive string comparison fallback
- `and` / `or` logic support for multiple conditions
- Zero external dependencies — pure functions only
## Summary
Implement the `GET /api/v1/agents/<agent_id>/versions/<version_id>`
endpoint in Go, returning full version details including DSL.
Depends on #15629 which introduced the version list endpoint and
`UserCanvasVersionDAO` infrastructure.
### Changes
- **Modified**: `internal/handler/agent.go` — Added `GetAgentVersion`
handler with auth check and ownership verification
- **Modified**: `internal/router/router.go` — Registered `GET
/:agent_id/versions/:version_id` route
- **New/Modified tests**: Service and handler tests for the version
detail endpoint
### Testing
```
=== RUN TestGetVersion_Success --- PASS
=== RUN TestGetVersion_WrongCanvas --- PASS
=== RUN TestGetVersion_NotFound --- PASS
=== RUN TestGetAgentVersionHandler_Success --- PASS
=== RUN TestGetAgentVersionHandler_VersionNotFound --- PASS
```
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
### What problem does this PR solve?
When a document is rerun or updated concurrently, the previous
unconditional update could overwrite a newer task state.
This change adds an `update_time`-based optimistic lock so the update
only succeeds if the record has not been modified by another flow in the
meantime.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
### What problem does this PR solve?
This change ensures `/searchbots/ask` receives `search_id` from the
frontend, so the backend can load the matching search configuration when
the shared search flow invokes the endpoint.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
### What problem does this PR solve?
Not display `success` when check not passed.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
## Summary
- keep the native Docling chunking path when it returns usable chunks
- fall back to the standard Docling response parser when a chunked
request gets HTTP 200 but returns no usable chunks
- add a regression test for older Docling servers that accept the
chunking request but return a standard conversion payload
## Why
Older external Docling servers can accept a request containing
`do_chunking: true` and still return the standard conversion response
shape. The current code treats any HTTP 200 from the chunked request as
a native chunk response, finds no chunk entries, and returns zero
sections without trying the standard response parser.
Fixes#15569.
## Validation
- `python -m pytest
test\\unit_test\\deepdoc\\parser\\test_docling_parser_remote.py -q`
- `python -m py_compile deepdoc\\parser\\docling_parser.py
test\\unit_test\\deepdoc\\parser\\test_docling_parser_remote.py`
- `python -m ruff check deepdoc\\parser\\docling_parser.py
test\\unit_test\\deepdoc\\parser\\test_docling_parser_remote.py`
- `git diff --check`
### What problem does this PR solve?
Markdown extraction currently applies custom delimiters before
respecting fenced code blocks. When a delimiter such as a newline is
configured, fenced code can be split into separate chunks, and longer
outer fences can be closed incorrectly by shorter nested fences.
This PR keeps the fix intentionally narrow for the Markdown chunking
discussion in #15482:
- preserve fenced code blocks when delimiter-based extraction is used
- support both backtick and tilde fences
- respect fence length so longer outer fences can contain shorter inner
fences
- keep delimiter splitting unchanged outside fenced blocks
Refs #15482
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
### Testing
- `ruff check deepdoc/parser/markdown_parser.py
test/unit_test/deepdoc/parser/test_markdown_parser.py`
- `python3 run_tests.py -t
test/unit_test/deepdoc/parser/test_markdown_parser.py`
### What problem does this PR solve?
remove duplicate document preview access check
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
### What problem does this PR solve?
Fix:
- Verify provider with empty llm list in llm_factories.json
- Set search bot's chat_llm_name, use tenant default chat model as
default
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
Restore `DocumentService.accessible` on `GET
/api/v1/documents/{doc_id}/preview` so cross-tenant users cannot stream
documents by UUID.
Fixes#15501
### What problem does this PR solve?
PR #15146 (`71a52d579`) moved the agent attachment download route and
accidentally removed the `DocumentService.accessible(doc_id,
current_user.id)` guard from the REST preview handler. The endpoint
still requires login, but any authenticated user who knows another
tenant's `doc_id` can download the raw file bytes.
This restores the same authorization check that existed before #15146,
returning a generic `"Document not found!"` when access is denied (no
cross-tenant ID enumeration). SDK download routes tracked in #15125 are
unchanged.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
## Summary
Restore the `DocumentService.accessible(doc_id, current_user.id)` check
that PR #15146 dropped from the REST document preview handler. Any
authenticated caller could download any tenant's document bytes by
guessing/knowing the `doc_id`.
## Root cause
`api/apps/restful_apis/document_api.py` — the `GET
/documents/<doc_id>/preview` handler called `DocumentService.get_by_id`
and went straight to `File2DocumentService.get_storage_address` +
`STORAGE_IMPL.get`, with no tenant check between the lookup and the
read. The handler's docstring even promises "user must belong to the
tenant that owns the document's knowledge base" — the code didn't
enforce it.
## Fix
- Add `current_user` to the existing `api.apps` import.
- Immediately after `get_by_id`, call
`DocumentService.accessible(doc_id, current_user.id)`; on denial, return
the **same** `get_data_error_result(message="Document not found!")`
shape used for the missing-doc branch. That makes a cross-tenant probe
indistinguishable from a missing-doc probe, preventing ID enumeration
(the issue body calls this out explicitly).
- Emit `logging.warning` with caller user + doc_id for audit.
- Restores symmetry with peer routes that already call
`accessible(doc_id, user_id)` (e.g. `_run_sync` at
`document_api.py:1380`).
## Test plan
Adds
`test/unit_test/api/apps/restful_apis/test_document_preview_accessible.py`:
- **`test_cross_tenant_preview_is_denied`** — owner tenant ≠ caller
tenant; asserts the response shape is `Document not found!` and the
storage backend (`thread_pool_exec(STORAGE_IMPL.get, ...)`) is **never**
invoked.
- **`test_missing_doc_returns_not_found`** — missing-doc behaviour
unchanged.
Stub-loader pattern mirrors
`test/unit_test/api/apps/sdk/test_dify_retrieval.py` (added in #15028,
passing in CI).
## Provenance — how this fix was produced
This PR was authored against a small cited knowledge base committed in
the working tree as a `.vouch/` (see
[vouchdev/vouch](https://github.com/vouchdev/vouch)). The loop used
here:
1. **Grounding first.** Before reading the handler, queried the KB for
prior context: `vouch context "tenant scoped accessible authorization"`
→ retrieved a cited claim distilled from PR #15028 (which restored the
same `accessible()` check on `/dify/retrieval`). The retrieved rule:
> *ragflow REST endpoints that load by tenant-scoped id must call
`<Service>.accessible(id, tenant_id)` after `get_by_id` and before
storage/DB read; deny with code 109 'No authorization.' and log a
warning. Established by PR #15028.*
2. **Applied the pattern with a domain refinement.** For an API/JSON
endpoint, `No authorization.` is the right denial shape. For a
**byte-streaming, browser-facing** endpoint like `/preview`, leaking
*existence* itself enables enumeration — so per the issue's expected
behaviour, this PR denies with `Document not found!` (indistinguishable
from missing) instead. Same auth check, narrower response.
3. **Recorded the refinement back into the KB** as a new cited claim, so
the next IDOR-class issue starts already grounded in both the general
pattern and the byte-route nuance.
Net effect of the workflow: the fix replicates a known-good pattern
instead of reinventing it, *and* the place where the pattern was nuanced
is now retrievable for the next pass. Mechanism is fully independent of
this PR — it's not a runtime dependency, just process discipline.
Closes#15501
### What problem does this PR solve?
### Problem
On the Model Providers page, the Embedding Model dropdown in System
Model Settings shows empty (no default selected), even though a default
embedding model is configured in `service_conf.yaml`.
### Root Cause
Two issues were identified:
1. **Backend: `_get_model_info` fails for unregistered providers**
The tenant's `embd_id` is set to `bge-m3@xxxx` during initialization
(from the placeholder config `factory: 'xxxx'`). The `_get_model_info`
function requires the provider to exist in `tenant_model_provider`
table, but `xxxx` is never a real provider. Even after the user adds a
real provider (e.g., ZHIPU-AI), the stale `embd_id` still references the
non-existent one, causing the function to return `None`.
2. **Frontend: default models cache not invalidated after adding
provider**
`useAddProviderInstance` only invalidates `addedProviders` and
`allModels` caches after adding a provider instance, but does **not**
invalidate the `defaultModels` cache. This means the default model list
is not re-fetched until the user manually refreshes the page.
### Fix
**`api/apps/services/models_api_service.py`**
- Added `_resolve_model_from_tenant_providers()` helper: when the
default model's provider doesn't exist (e.g., placeholder `xxxx`), it
searches through the tenant's actually registered providers for a model
of the same type and returns the first match.
- When an instance name doesn't match (e.g., `"default"` vs actual name
`"1"`), the function now auto-resolves to the first real instance under
that provider.
- Falls back to `FACTORY_LLM_INFOS` validation when neither provider nor
instance exists.
**`web/src/hooks/use-llm-request.tsx`**
- Added `queryClient.invalidateQueries({ queryKey:
LlmKeys.defaultModels() })` to `useAddProviderInstance` so that the
default model list is re-fetched immediately after a provider instance
is added, eliminating the need for a manual page refresh.
### Testing
- Verified with a tenant whose `embd_id=bge-m3@xxxx` and only provider
is ZHIPU-AI (instance `1`): `_resolve_model_from_tenant_providers`
correctly resolves to `embedding-2@1@ZHIPU-AI`.
- After adding a provider via the UI, the embedding model dropdown now
immediately shows the resolved default without requiring a page refresh.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
---------
Signed-off-by: noob <yixiao121314@outlook.com>
### What problem does this PR solve?
The following API is available in go
> /api/v1/connectors/google/oauth/web/start POST
> /api/v1/connectors/gmail/oauth/web/callback GET
> /api/v1/connectors/google-drive/oauth/web/callback GET
> /api/v1/connectors/google/oauth/web/result POST
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
### What problem does this PR solve?
Adds a shared safe default implementation for unsupported Go
model-driver capability methods and migrates the confirmed panic-stub
providers to use it.
The Go `ModelDriver` interface requires providers to implement many
capability methods even when the provider does not support them. XunFei
had unsupported capability methods implemented as `panic("implement
me")`, Mistral still had a panic in `ParseFile`, and HuaweiCloud carried
an unreachable `panic("implement me")` after a normal chat return.
### Type of change
- [x] Refactoring
Co-authored-by: Haruko386 <tryeverypossible@163.com>
## Summary
Fixes#15534 — `update_document_name_only()` crashes with
`AttributeError` when `File2Document` exists but the linked `File` row
was deleted.
`update_document_name_only()` in `document_api_service.py` called
`FileService.get_by_id()` when a `File2Document` row existed, then
accessed `file.id` without checking the lookup result. An orphan
`File2Document` link (file deleted, mapping left behind) caused document
rename via `PATCH /api/v1/datasets/{dataset_id}/documents/{document_id}`
to return HTTP 500.
This PR mirrors guards used in `file2document_api.py` and
`file_api_service.py`: skip the optional file rename when the file is
missing, and still update the document record and search index.
## Changes
- `api/apps/services/document_api_service.py` — check `e and file`
before `FileService.update_by_id`
- `test/unit_test/api/apps/services/test_update_document_name_only.py` —
regression tests (orphan link + happy path)
## Test plan
- [x] `pytest
test/unit_test/api/apps/services/test_update_document_name_only.py -v`
- [ ] Manual: PATCH document `name` when `File2Document` points to a
non-existent `file_id` → 200, document/index renamed, no 500
### What problem does this PR solve?
1. Add license announcement
2. Add sanity check on API config
3. Add base class: BaseModel
4. Add GetBaseURL
### Type of change
- [x] Refactoring
---------
Signed-off-by: Jin Hai <haijin.chn@gmail.com>
### What problem does this PR solve?
Fix: Switching pagesize on a chunk page did not reset the current page.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
### What problem does this PR solve?
remove old and add latest cohere models
### Type of change
- [x] Refactoring
- [x] Other (please describe): update models
### What problem does this PR solve?
Fix: Model provider add verify and fixed form in modal not resetting
issue
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
### What problem does this PR solve?
Closes#15388.
Chat completion routes did not reliably honor per-request generation
settings:
- `/api/v1/chat/completions` copied generation settings with a
truthiness check, so valid zero values such as `temperature: 0`, `top_p:
0`, `frequency_penalty: 0`, `presence_penalty: 0`, and `max_tokens: 0`
were dropped.
- `/api/v1/openai/{chat_id}/chat/completions` did not forward standard
generation settings into the request-specific dialog LLM settings before
calling `async_chat`.
This PR preserves explicitly supplied generation parameters, including
zero values, and merges request-level overrides into existing dialog
settings where appropriate.
The supported generation parameter keys and merge behavior live in a
shared REST API helper to keep both completion routes aligned.
Validation:
- `git diff --check`
- `python3 -m py_compile api/apps/restful_apis/_generation_params.py
api/apps/restful_apis/chat_api.py api/apps/restful_apis/openai_api.py
test/testcases/test_http_api/test_session_management/test_session_sdk_routes_unit.py`
- `uv run ruff check api/apps/restful_apis/_generation_params.py
api/apps/restful_apis/chat_api.py api/apps/restful_apis/openai_api.py
test/testcases/test_http_api/test_session_management/test_session_sdk_routes_unit.py`
- `ZHIPU_AI_API_KEY=dummy uv run pytest
test/testcases/test_http_api/test_session_management/test_session_sdk_routes_unit.py
-q -k generation_params`
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
## Problem
When uploading `.md` files with `parser=naive` and `delimiter="\n"`,
markdown headers (e.g., `## Quick Travel`) become separate chunks with
very short content (16-18 characters). This causes retrieval issues:
when the header is matched, the corresponding body text is not included
in the chunk.
## Related Issues
Closes#15487
## Checklist
- [x] Code changes are minimal and focused
- [x] Unit tests added (12/12 passed)
- [x] No breaking changes
### What problem does this PR solve?
Fix:
- Handle siliconflow and siliconflow_intl api_key
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
add the newanthropic and voyage models. Strip opus 4.7 and 4.8 of
certain usnspported keys
Co-authored-by: Idriss Sbaaoui <112825897+6ba3i@users.noreply.github.com>
## What
#15240
implementation for PUT /api/v1/mcp/servers/:mcp_id
## Changes
- Adds the Go implementation for `PUT /api/v1/mcp/servers/:mcp_id`.
- Wires MCP service and handler into the Go server/router for the update
route.
- Preserves Python-style behavior for ownership checks, partial update
fields, MCP type/name/URL validation, `headers`/`variables`
normalization, and tool metadata scrubbing.
### What problem does this PR solve?
Closes#15379
Around 29 Go model providers in `internal/entity/models/` share an
`http.Client` configured with `Timeout: 120 * time.Second`, and reuse
that same client for `ChatStreamlyWithSender`. Go's
`http.Client.Timeout` is a hard ceiling on the whole request that also
covers reading the response body, so it behaves as a wall clock on
streaming. Any streamed chat response that lasts longer than 120 seconds
gets cut off in the middle with a timeout error. Long generations,
reasoning model outputs, and slow or overloaded upstreams are the common
victims.
The providers that already behave correctly (`groq`, `mistral`,
`voyage`, `anthropic`) set no client `Timeout` and instead wrap each
request in a `context.WithTimeout`. This change converges the affected
providers onto that same pattern.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
---------
Co-authored-by: Jin Hai <haijin.chn@gmail.com>
### What problem does this PR solve?
mark mysql migrations as applied
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
## Summary
- Harden `NewN1NModel` to avoid panics when `http.DefaultTransport` is a
custom non-`*http.Transport` RoundTripper.
- Fallback to a safe transport (`ProxyFromEnvironment`) while preserving
existing pooling/timeout settings.
- Add `n1n_test.go` with coverage for name/factory plus
`TestN1NNewModelWithCustomDefaultTransport`.
Co-authored-by: Cursor <cursoragent@cursor.com>
### What problem does this PR solve?
This PR aligns `POST /api/v1/system/tokens` in Go with the Python
implementation.
### Type of change
- Keep the token creation flow under the system API route.
- Preserve the owner-tenant authorization check.
- Generate and persist API tokens consistently with the current Go
service flow.
- Return the created token payload in the standard API response format.
Co-authored-by: Jin Hai <haijin.chn@gmail.com>
### What problem does this PR solve?
Fix:
- Use @ to avoid split by `_` in model_name.
- Verify api_key when add instance.
- Pop api_key in list intances response.
- Remove useless index.
- Sort providers, instances and models by name.
- Get `is_tools` from llm_factories.json
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
### What problem does this PR solve?
implement /api/v1/datasets/<dataset_id>/metadata/config
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
### What problem does this PR solve?
This PR fixes the issue where Qwen3.5/Qwen3.6 series models may spend
excessive time on simple document-parsing tasks, such as Auto Metadata
extraction, keyword extraction, question generation, and image
description when using the MinerU parser.
For these tasks, Qwen3.5/Qwen3.6 models may perform unnecessary
reasoning by default, which can lead to very long response times, high
token consumption, and, in some cases, potential infinite output loops.
Since Qwen3.5/Qwen3.6 multimodal models are instantiated as `CvModel`
when configured as `image2text`, the existing `enable_thinking=False`
logic in `chat_model.py` does not apply to them. This PR adds the
corresponding handling for the CV/image-to-text model path as well.
This helps reduce unnecessary thinking time, avoid potential infinite
loops, and improve parsing efficiency without noticeably affecting
output quality for these simple extraction and image-description tasks.
Fixes#15083.
### What problem does this PR solve?
Fixes#15286.
When calling `/api/v1/openai/<chat_id>/chat/completions` with `"stream":
true`, the response contains the answer **twice** — the final message
repeats everything that was already streamed.
#### Root cause
RAGFlow's `async_chat` streams the body as incremental `delta.content`
chunks, then emits a terminating `final` event whose `answer` is the
**complete** (decorated) message. The handler re-emitted that full
answer as one more `delta.content` chunk:
```python
if ans.get("final"):
if ans.get("answer"):
full_content = ans["answer"]
response["choices"][0]["delta"]["content"] = full_content # <-- whole answer again
yield ...
```
So a client accumulating `delta.content` ends up with the message
duplicated.
#### Fix
Drop the re-emission. The complete answer from the `final` event is now
surfaced **only** through the trailing chunk's `final_content` and
`reference` fields, which matches OpenAI streaming semantics: deltas are
incremental, and the final chunk carries only `finish_reason` / `usage`
(plus RAGFlow's `reference` / `final_content` extensions).
This matches the expected behavior described in the issue: "The stream
should only yield content chunks once, and the final message should only
contain reference, usage, and finish_reason."
#### Testability refactor
The streaming SSE assembly was a closure inside the request handler, so
it could only be exercised against a live server + real LLM. I extracted
it into a module-level `_stream_chat_completion_sse` async generator
(behavior-preserving) so it can be unit-tested with a fake event stream.
#### Tests
Adds
`test/unit_test/api/apps/restful_apis/test_openai_stream_no_duplicate.py`
(same import-stub pattern as the existing `test_get_agent_session.py`):
- body is streamed exactly once (the regression);
- the complete answer is never re-emitted as a content chunk;
- the terminating chunk has `finish_reason="stop"`, `content=None`, and
correct `usage`;
- `final_content` / `reference` are present on the trailing chunk;
- reasoning (`think`) deltas stream separately and are not duplicated.
> Note: this is unrelated to #15442, which only changes the `stream`
default — it does not touch the duplication logic.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
- [x] Added test cases
---------
Co-authored-by: Wang Qi <wangq8@outlook.com>
### What problem does this PR solve?
Feature: #14961
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
- [x] Refactoring
### What problem does this PR solve?
Fixes#15115.
`GET /api/v1/documents/images/<image_id>` returned **Image not found**
when the thumbnail storage object key contained hyphens (e.g.
`page-1.png`). Document APIs build URLs as `{dataset_id}-{thumbnail}`,
but `get_document_image()` used `image_id.split("-")` and required
exactly two segments, so keys like `<kb_id>-page-1.png` were rejected
even though the blob existed.
This PR splits only on the first hyphen (`split("-", 1)`) and sets
`Content-Type` from the object key extension via `CONTENT_TYPE_MAP`
instead of hardcoding `image/JPEG`.
### What problem does this PR solve?
Fixes#15117.
Chunk images are stored with `img_id = f"{bucket}-{objname}"` in
`image2id()` (`rag/utils/base64_image.py`). When loading via
`id2image()`, the code used `image_id.split("-")` and required exactly
two segments. Object keys that contain hyphens (e.g. `page-1.jpg`)
produce more than two segments, so `id2image` returns `None` and chunk
image previews fail even though the blob exists.
This is the same parsing issue as #15115 (HTTP thumbnail route); this PR
fixes the indexing/retrieval path.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
- [ ] New Feature (non-breaking change which adds functionality)
- [ ] Documentation Update
- [ ] Refactoring
- [ ] Performance Improvement
- [ ] Other (please describe):
### Test plan
- [x] `pytest test/unit_test/rag/utils/test_base64_image.py`
- [ ] Manual: index a chunk with an `objname` containing hyphens and
confirm `img_id` resolves to an image in retrieval
Fixes#15117.
### Related issues
Closes#15312
### What problem does this PR solve?
`tools/scripts/mysql_migration.py` built batch INSERT SQL for the
`tenant_model_provider` stage using f-strings with raw `llm_factory` and
`tenant_id` values. If either value contained a single quote, migration
SQL could fail; this also created unnecessary SQL-injection risk in the
migration path.
This PR replaces string interpolation with parameterized SQL
placeholders in `TenantModelProviderStage.execute()`. The migration now
safely handles quoted values and executes deterministically across
existing tenant data.
### Related issues
Closes#15358
<!-- After filing upstream, replace XXXX with your issue number. -->
---
### What problem does this PR solve?
`POST /api/v1/openai/<chat_id>/chat/completions` forwards `messages` to
`async_chat` without normalizing `content`. Downstream, `dialog_service`
assumes string content:
```python
re.sub(r"##\d+\$\$", "", m["content"])
```
OpenAI-compatible clients may send `content` as an **array** of parts
(text, `image_url`, etc.), including text-only arrays. That causes
`TypeError` and HTTP **500** instead of a valid response or a clear
**400**.
`openai_api.py` also reads `messages[-1]["content"]` directly for
`prompt` without handling list-shaped content.
This PR normalizes array `content` to a string (concatenating `type:
text` parts) before calling `async_chat`, matching a minimal
OpenAI-compat path. Image parts can be documented as unsupported or
handled in a follow-up if vision integration is required.
### What problem does this PR solve?
Fix auto metadata type issue
https://github.com/infiniflow/ragflow/issues/15323
Type information is missing at frontend - backend correctly store the
type information for the auto metadata type.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
## Summary
Fixes#15245 — `POST /api/v1/chat/completions` with `stream=true`
intermittently returns 500:
```
data:{"code": 500, "message": "failed to encode response: json:
unsupported value: NaN (status code: 500)", "data": {...}}
```
…even though "the same question" works on retry.
## Root cause
The streaming path serialized the answer with bare `json.dumps(...)`
(`api/apps/restful_apis/chat_api.py:1221`). `json.dumps` defaults to
`allow_nan=True` and emits the literal token `NaN` for NaN /
Infinity float values. That is valid Python-flavored JSON but
**invalid per RFC 8259**, so downstream consumers reject it. The
reporter's gateway is Go-based and the error wording
(`failed to encode response: json: unsupported value: NaN`) is
straight from Go's `encoding/json`.
How NaN gets into the payload: retrieval scoring in
`rag/nlp/search.py` runs `np.mean(...)` over aggregations that can
be empty, and similarity denominators can be zero. Reference chunk
fields like `similarity`, `vector_similarity`, `term_similarity`
can therefore be NaN depending on which chunks a given query
retrieves — which is exactly why the failure is intermittent for
the same question.
The non-streaming branch (`get_json_result(data=answer)`,
`chat_api.py:1243`) has the same vulnerability — Quart's `jsonify`
also defaults to `allow_nan=True` and the same retrieval pipeline
feeds both branches.
`agent/tools/exesql.py:88-102` already has the same NaN/Inf guard
for SQL results. This PR brings the chat completions path up to
parity.
## Fix
Add a small `_sanitize_json_floats(obj)` helper near the top of
`api/apps/restful_apis/chat_api.py`. It walks `dict` / `list` /
`tuple` and replaces any `float` that is `NaN` or `±Infinity` with
`None`. Apply it at the two serialization boundaries:
- **Streaming branch** (`stream()`): sanitize the SSE payload before
`json.dumps`.
- **Non-streaming branch**: sanitize the `answer` dict before
`get_json_result(data=...)`.
The terminal `data:True` frame and the `code:500` error frame carry
no scores and are left untouched.
Added `import math` to the existing alphabetical import block.
No change to retrieval logic — replacing NaN with `null` at the
serialization boundary is conservative: clients still parse the
JSON, a missing-score chunk is a strictly better failure mode than
a 500 that kills the whole reply.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
### What problem does this PR solve?
Fixes#15427.
All LiteLLM-routed chats fail with:
- Anthropic: `litellm.BadRequestError: AnthropicException -
{"type":"invalid_request_error","message":"model_type: Extra inputs are
not permitted"}`
- OpenAI: `litellm.BadRequestError: OpenAIException - Unknown parameter:
'model_type'`
This is a regression from v0.25.4.
#### Root cause
A chat assistant's `llm_setting` is forwarded to the model as
`gen_conf`. `llm_setting` can legitimately carry RAGFlow-internal
metadata such as `model_type` (the chat REST APIs in
`api/apps/restful_apis/` read it back out of `llm_setting`), so that key
ends up inside `gen_conf`.
`Base._clean_conf` (OpenAI-compatible providers) already **whitelists**
the keys it forwards, so direct-OpenAI providers were unaffected.
`LiteLLMBase._clean_conf` only dropped `max_tokens` and passed
everything else straight through to `litellm.acompletion`, which
forwarded `model_type` to the upstream provider — and Anthropic / OpenAI
reject it. Because both Claude and GPT route through LiteLLM, every chat
broke.
#### Fix
- Extract the allowed-key set into a shared `ALLOWED_GEN_CONF_KEYS`
constant and reuse it in `Base._clean_conf`.
- Apply the same whitelist in `LiteLLMBase._clean_conf`, plus the
LiteLLM-specific reasoning params (`thinking`, `reasoning_effort`,
`extra_body`) that the model-family policies inject for reasoning
models.
This covers all four LiteLLM completion paths (`async_chat`,
`async_chat_streamly`, `async_chat_with_tools`,
`async_chat_streamly_with_tools`), since they all route through
`_clean_conf`.
#### Tests
Adds `test/unit_test/rag/llm/test_clean_conf_whitelist.py` covering both
backends: `model_type` (and other stray keys) are dropped, genuine
generation params and `thinking` survive, `max_tokens` is removed, and
the whitelist invariants hold.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
- [x] Added test cases
## What
#15240
Implements `GET /api/v1/mcp/servers` in the Go API server.
## Changes
- Added MCP server DAO list query with tenant scoping.
- Added MCP service response wrapper.
- Added MCP handler for list request parsing and response formatting.
- Wired `GET /api/v1/mcp/servers` under authenticated `/api/v1` routes.
- Initialized MCP service and handler in the Go server startup.
- update_time and update_date now both map to update_date
- create_time and create_date now both map to create_date
- default ordering now returns create_date
## API Behavior
Matches the Python endpoint behavior:
- Requires authenticated user.
- Lists MCP servers for the current user tenant.
- Supports `keywords`.
- Supports `mcp_id` and repeated/comma-separated `mcp_ids`.
- Supports `page`, `page_size`, `orderby`, and `desc`.
- Returns:
```json
{
"code": 0,
"message": "success",
"data": {
"mcp_servers": [],
"total": 0
}
}
```
## Summary
Add Tigris configuration to the Configuration and Backup & migration
pages, using the existing AWS_S3 backend — no code changes required.
Fix `region` → `region_name` in the existing S3 config example in
`backup_and_migration.md`. The code in `s3_conn.py` reads `region_name`,
so the previous field name was silently ignored.
##Context
With MinIO's open-source repository archived (#13840 on
infiniflow/ragflow), users need documented alternatives for object
storage. Tigris is S3-compatible and works with RAGFlow's existing
AWS_S3 backend out of the box.
## Changes
`configurations.md`: Added `### s3 (Tigris)` section after `### minio`,
matching the existing reference style. Includes config block, field
descriptions, and a pointer to `service_conf.yaml.template` for other
S3-compatible backends.
`backup_and_migration.md`: Added Tigris config block under single-bucket
mode. Fixed region → region_name in the existing S3 example. Added
Tigris to the supported backends list.
##Notes
No new files — edits to existing docs only.
Config field names (`access_key`, `secret_key`, `region_name`,
`endpoint_url`, `bucket`, `prefix_path`, `signature_version`,
`addressing_style`) verified against `rag/utils/s3_conn.py`.
## Summary
- Skip MinerU `header`, `footer`, and `page_number` blocks when
converting `content_list.json` into sections.
- Ignore unsupported block types explicitly so future MinerU output
types cannot re-emit the previous text block.
Fixes duplicate text in General/naive chunks when parsing PDFs via
MinerU (reported with repeated page headers and body text in slices).
Closes#15335
## Test plan
- [x] `pytest test/unit_test/deepdoc/parser/test_mineru_parser.py -v`
(4/4 passed)
## Summary
- Add custom `base_url` support to the Google Go model driver.
- Preserve Google URL suffix configuration when creating custom base URL
driver instances.
- Validate Google chat/stream request inputs before constructing the SDK
client.
- Cover Google model listing, connection checks, base URL resolution,
and request validation with focused tests.
## What changed
- `GoogleModel.NewInstance` now returns a Google driver configured with
the supplied base URL map.
- Google SDK client creation now resolves configured base URLs through
`genai.HTTPOptions.BaseURL`.
- Base URL lookup supports configured regions, empty-region keys, and
`default` fallback.
- Google chat, streaming chat, embeddings, and model listing now reject
blank API keys before creating SDK clients.
- Google chat and streaming chat now reject blank model names locally,
and streaming chat rejects a nil sender.
- Existing message handling, embeddings, pagination, and provider errors
are preserved.
## Why
Google custom model instances could not use configured base URLs because
`NewInstance` returned `nil` and the SDK client path ignored the driver
base URL map. The request validation keeps invalid Google calls from
reaching SDK client construction with blank credentials or incomplete
chat inputs.
### What problem does this PR solve?
## Problem
When parsing PDFs containing English figure/table captions (e.g. "Fig.
20", "Figure 20", "Table 20"), the `is_caption` method in
`TableStructureRecognizer` failed to recognize them as captions. This
caused figure numbering gaps in the parsed output (e.g. Fig. 19 → Fig.
21, skipping Fig. 20).
## Root Cause
The `is_caption` regex only matched Chinese caption formats:
```python
patt = [r"[图表]+[ 0-9:：]{2,}"]
```
When the layout recognizer also failed to assign a `caption` layout type
to a given text block, English captions were entirely missed.
## Fix
Added three case-insensitive English caption patterns to `is_caption` in
`deepdoc/vision/table_structure_recognizer.py`:
- `(?i)Fig\.?\s*\d+` — matches `Fig. 20`, `Fig 20`, `FIG. 20`, etc.
- `(?i)Figure\s+\d+` — matches `Figure 20`, `FIGURE 20`, etc.
- `(?i)Table\s+\d+` — matches `Table 20`, `TABLE 20`, etc.
## Files Changed
- `deepdoc/vision/table_structure_recognizer.py` — extended `is_caption`
regex patterns
- [x] Bug Fix (non-breaking change which fixes an issue)
Signed-off-by: noob <yixiao121314@outlook.com>
### What problem does this PR solve?
Fix: The newly added model did not appear in the drop-down menu.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
fix: restore TitleChunker output for json/chunks upstream formats
## Summary
The refactor commit e194027b (#14247) introduced two regressions that
caused `TitleChunker` to produce zero chunks when the upstream Parser
node outputs `json` or `chunks` format (e.g. PDF parsing).
## Root Cause
### 1. Dead code in `extract_line_records` (critical)
After refactor, when `payload` is `None` (which is the case for `json`
and `chunks` output formats), the method returns an empty list
immediately via `return []`, so no records are ever extracted from
structured upstream output. The original `json`/`chunks` handling code
became unreachable dead code.
### 2. Unconditional overwrite in `build_chunks_from_record_groups`
The `chunks` variable assigned in the `if` branch for markdown/text/html
formats was unconditionally overwritten by the statement below it, due
to a missing `else` keyword.
## Fix
- Remove the premature `return []` so the `json`/`chunks` branch is
reachable again.
- Add `else` branch in `build_chunks_from_record_groups` so the two
format families are handled independently.
## Test Plan
- [x] Verified no lint errors on the changed file
- [ ] Tested with a PDF document parsed via DeepDOC → TitleChunker
pipeline
- [ ] Tested with markdown input through TitleChunker
- [ ] Tested hierarchy and group chunking modes
## Impact
- Fixes the regression where documents parsed with `json`/`chunks`
output format produced no chunks from `TitleChunker`.
- No API or configuration changes. Fully backward compatible.
Signed-off-by: noob <yixiao121314@outlook.com>
### What problem does this PR solve?
Part of the Python → Go API server rewrite tracked in #15240 (Dataset
ingestion section). This PR implements the three dataset ingestion
endpoints in the Go API server, mirroring the existing Python
`dataset_api_service` behaviour:
- `GET /api/v1/datasets/<dataset_id>/ingestions/summary`
- `GET /api/v1/datasets/<dataset_id>/ingestions`
- `GET /api/v1/datasets/<dataset_id>/ingestions/<log_id>`
### Type of change
- [x] Refactoring
- [x] New Feature (non-breaking change which adds functionality)
Co-authored-by: sxxtony <sxxtony@users.noreply.github.com>
### What problem does this PR solve?
The Go GPUStack driver returned a stub error for `Embed()` even though
GPUStack exposes OpenAI-compatible embeddings on the **v1-openai** route
(not `v1/embeddings`).
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
### What problem does this PR solve?
As title
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
- [x] New Feature (non-breaking change which adds functionality
### What problem does this PR solve?
extend restful api suite
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
- [x] Other (please describe): test
### What problem does this PR solve?
Fix: If the filename is too long, it overflows the confirmation box for
deleting the file.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
### What problem does this PR solve?
Closes#15332.
RAGFlow can index Gmail and generic IMAP mailboxes but had no native
connector for Outlook / Microsoft 365 mail. Organisations on Microsoft
365 had no way to bring mailbox content into a knowledge base through
Microsoft Graph.
This PR adds a net-new Outlook data source that:
- Authenticates against Microsoft Graph with the same MSAL
client-credentials flow already used by the SharePoint and Teams
connectors (no new auth primitives).
- Pages over `/users/{id}/mailFolders/{folder}/messages/delta` per
mailbox and persists `@odata.deltaLink` values in
`OutlookCheckpoint.delta_links`, so incremental syncs only fetch changed
messages.
- Supports two scoping modes:
- **Tenant-wide** (default): enumerates every user in the tenant via
`/users` and syncs each mailbox. Requires `User.Read.All`.
- **Targeted**: when `user_ids` is provided (comma-separated UPNs or
object IDs), only those mailboxes are synced. `User.Read.All` is not
needed in this mode.
- Lets the caller pick the mail folder (`inbox`, `sentitems`, `archive`,
...). Defaults to `inbox`.
- Maps each message to a `Document` shaped after the Gmail connector:
one `TextSection` carrying `From/To/Cc/Subject` headers + body, with
HTML bodies stripped to text inline (no extra dependency).
- Surfaces typed errors on the validation probe:
401 → `ConnectorMissingCredentialError`, 403 →
`InsufficientPermissionsError` (with `Mail.Read` / `User.Read.All`
hint), 404 on a configured mailbox → `ConnectorValidationError`, 5xx →
`UnexpectedValidationError`.
- Skips messages flagged `@removed` by the delta semantics and messages
whose `receivedDateTime` is older than `poll_range_start`.
#### Files
| File | Change |
|------|--------|
| `common/data_source/outlook_connector.py` | **New** —
`OutlookConnector` (`CheckpointedConnectorWithPermSync` +
`SlimConnectorWithPermSync`) + `OutlookCheckpoint` + tiny `_strip_html`
helper. |
| `common/data_source/config.py` | `DocumentSource.OUTLOOK = "outlook"`.
|
| `common/constants.py` | `FileSource.OUTLOOK = "outlook"`. |
| `common/data_source/__init__.py` | Export `OutlookConnector`. |
| `rag/svr/sync_data_source.py` | `Outlook(SyncBase)` with `batch_size`
normalisation, CSV/list parsing of `user_ids`; registered in
`func_factory`. |
| `web/src/pages/user-setting/data-source/constant/index.tsx` |
`DataSourceKey.OUTLOOK`, visibility map (`syncDeletedFiles: true`), info
entry, form fields (tenant_id, client_id, client_secret, folder,
user_ids, batch_size), default values. |
| `web/src/locales/en.ts`, `web/src/locales/zh.ts` |
`outlookDescription` + 5 tooltip keys (EN + ZH). |
| `test/unit_test/data_source/test_outlook_connector_unit.py` | **New**
— 19 unit tests (`p1`/`p2`/`p3`) covering auth, validation (tenant-wide
vs specific user vs error paths), checkpoint helpers, user enumeration
pagination, message filtering, HTML body stripping. |
#### Required Azure AD permissions
- `Mail.Read` (Application, admin-granted) — always.
- `User.Read.All` (Application, admin-granted) — only when `user_ids` is
left blank so the connector can enumerate mailboxes.
#### Out of scope
- **Attachment indexing.** The current connector emits message body +
headers; binary attachments are flagged via `metadata.has_attachments`
but not pulled. Adding attachment hydration is straightforward but
scoped out per the issue's "decide whether attachments are indexed in
the first version" note.
- **Delegated (per-user) OAuth.** The connector uses app-only
credentials, consistent with the SharePoint / Teams precedent in this
codebase.
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
### What problem does this PR solve?
Document metadata is completely broken on the OpenSearch backend
(`DOC_ENGINE=opensearch`). Both failures were introduced by #14577,
which added
a doc-metadata dispatch surface but only validated it against
Elasticsearch.
**1. Index creation rejected (`mapper_parsing_exception`).**
`OSConnection.create_doc_meta_idx` feeds `conf/doc_meta_es_mapping.json`
verbatim to OpenSearch. That file declares a top-level `"dynamic":
"runtime"`.
Runtime fields are Elasticsearch-only; OpenSearch cannot parse the
value:
mapper_parsing_exception: Could not convert [dynamic.dynamic] to boolean
(400)
**2. `search()` signature mismatch (`TypeError`).**
`DocMetadataService` (added by #14577) calls `docStoreConn.search(...)`
with
snake_case kwargs (`select_fields=`, `index_names=`,
`knowledgebase_ids=`, …),
matching `ESConnection.search`. But `OSConnection.search` still uses
camelCase
parameters (`selectFields`, `indexNames`, `knowledgebaseIds`, …):
TypeError: OSConnection.search() got an unexpected keyword argument
'select_fields'
The UI then shows "0 fields" for every document on OpenSearch.
### Fix
1. In `OSConnection.create_doc_meta_idx`, normalize a top-level
`"dynamic": "runtime"` to `True` **for the OpenSearch request only**.
The
shared mapping file is left untouched, so the Elasticsearch backend
keeps its
runtime-field behavior. Dynamic field discovery is preserved on
OpenSearch.
2. Rename the `OSConnection.search()` parameters (and their in-method
local
uses) from camelCase to snake_case so they match `ESConnection.search()`
and
the `DocMetadataService` call sites. The change is confined to
`search()`;
`get/insert/update/delete` keep their existing positional signatures
(they
are called positionally from `rag/nlp/search.py`).
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
### Affected backends
OpenSearch only. Elasticsearch, Infinity and OceanBase are untouched.
### How to reproduce
1. `DOC_ENGINE=opensearch`, restart the stack.
2. Upload/parse a document, then open the dataset's document list / set
metadata.
- Before: index creation 400s (`Could not convert [dynamic.dynamic]`),
and/or
`TypeError ... 'select_fields'`; document metadata shows 0 fields.
### Risk & backward compatibility
- ES default deployment: no change. `doc_meta_es_mapping.json` is not
modified,
so ES still receives `"dynamic": "runtime"`.
- `search()` rename is internal; the only kwarg caller
(`DocMetadataService`)
already uses the snake_case names this PR aligns to.
### Test plan
- [ ] `DOC_ENGINE=opensearch`: per-tenant `ragflow_doc_meta_*` index is
created
(no `mapper_parsing_exception`); document metadata reads/writes work.
- [ ] `DOC_ENGINE=elasticsearch` regression: doc-meta index still
created with
runtime mapping; metadata unchanged.
### What problem does this PR solve?
On the OpenSearch backend (`DOC_ENGINE=opensearch`), every retrieval
that
performs the KNN second-pass scoring crashes with:
AttributeError: 'OSConnection' object has no attribute 'get_scores'
**Root cause.** #14970 ("Refactor: Drop the vector fetch for ES") added
a
`get_scores()` helper to `ESConnectionBase`
(`common/doc_store/es_conn_base.py`)
and introduced `Dealer._knn_scores()` in `rag/nlp/search.py`, which
calls
`self.dataStore.get_scores(res)`. `search.py` routes Infinity and
OceanBase to
their own similarity paths via `DOC_ENGINE_INFINITY` /
`DOC_ENGINE_OCEANBASE`,
but OpenSearch sets neither flag, so it falls into the Elasticsearch
branch and
calls `get_scores`. `OSConnection` (which subclasses
`DocStoreConnection`
directly, not `ESConnectionBase`) never received that method, so any
vector-search hit triggers the crash. It reproduces with any normal
embedding
(e.g. 1024-dim mistral-embed) as soon as a KNN query returns hits.
### Fix
Add `OSConnection.get_scores()`, mirroring
`ESConnectionBase.get_scores()`.
OpenSearch hit headers expose `_score` exactly like Elasticsearch (the
existing
`OSConnection.__getSource` already reads `d["_score"]`), so the
implementation
is identical.
Scope note: Infinity and OceanBase deliberately do not use `get_scores`
(#14970 routes them elsewhere), so this fix is intentionally limited to
the
OpenSearch backend, which is the only one reaching the ES KNN-score
path.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
### Affected backends
OpenSearch only. Elasticsearch already implements `get_scores`; Infinity
/
OceanBase are routed away from it.
### How to reproduce
1. `DOC_ENGINE=opensearch` (docker `.env`), restart the stack.
2. Create a knowledge base with any dense embedding model and parse a
document.
3. Run a retrieval / chat over that KB -> 500 with the AttributeError
above.
### Risk & backward compatibility
None for the default Elasticsearch deployment -- the change only adds a
method
to `OSConnection`. No default values or ES/Infinity/OceanBase behavior
change.
### Test plan
- [ ] With `DOC_ENGINE=opensearch`, retrieval over a KB returns scored
chunks
(no AttributeError).
- [ ] `DOC_ENGINE=elasticsearch` regression: retrieval unchanged.
- [ ] Empty-result path: `_knn_scores` early-returns `{}` (guarded),
get_scores
handles an empty `hits` list gracefully.
### Related issues
Closes#15310
### What problem does this PR solve?
`/api/v1/dify/retrieval` had duplicate `GET` route registrations in
`dify_retrieval_api.py`: one for authenticated retrieval and another for
unauthenticated health checks. Sharing the same path and method created
ambiguous routing behavior and an unstable API contract for Dify
external knowledge base integration.
This PR separates concerns by moving the health-check endpoint to `GET
/api/v1/dify/retrieval/health`, while keeping retrieval on
`/api/v1/dify/retrieval`. This makes auth behavior deterministic and
prevents route shadowing/conflicts.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
## Summary
Upgrade crawl4ai from 0.7.6 to 0.8.0 to fix CVE-2026-26217.
## Vulnerability
| Field | Value |
|-------|-------|
| **ID** | CVE-2026-26217 |
| **Severity** | CRITICAL |
| **Scanner** | trivy |
| **Rule** | `CVE-2026-26217` |
| **File** | `uv.lock` |
| **Assessment** | Likely exploitable |
**Description**: Crawl4AI Has Local File Inclusion in Docker API via
file:// URLs
## Evidence
**Scanner confirmation**: trivy rule `CVE-2026-26217` flagged this
pattern.
**Production code**: This file is in the production codebase, not
test-only code.
## Threat Model Context
This is a web service - vulnerabilities in request handlers are directly
exploitable by remote attackers.
## Changes
- `pyproject.toml`
- `uv.lock`
## Verification
- [x] Build passes
- [x] Scanner re-scan confirms fix
- [x] LLM code review passed
---
*This change addresses a pattern flagged by static analysis. The code
path handles user-influenced input and the fix reduces the attack
surface against both manual and automated exploitation.*
---
*Automated security fix by [OrbisAI Security](https://orbisappsec.com)*
This PR introduces a `.rooignore` file to the root of the repository to
optimize how AI coding assistants (like Roo) interact with the RAGFlow
codebase.
Currently, when AI agents index the workspace, they can waste tokens and
processing time reading through generated files, caches, large
dependency artifacts, and runtime logs. This `.rooignore` file provides
a standard configuration to exclude these irrelevant directories and
files (such as `.venv/`, `node_modules/`, `__pycache__/`, logs, and
large binaries). This significantly reduces indexing noise, prevents
accidental reads of sensitive or bulky local data, and ensures AI coding
agents remain focused strictly on relevant source code.
### Type of change
- [x] Other (please describe): Developer Experience (DX) / AI Tooling
configuration
### What problem does this PR solve?
This PR updates `SystemService.ListAPITokens` to lazily backfill missing
`beta` values for API tokens, matching the Python behavior of
`/api/v1/system/tokens`.
### Type of change
- When an API token has an empty `beta`, generate a new one.
- Persist the generated `beta` back to the `api_token` table.
- Keep the handler/routing unchanged.
- `GET /api/v1/system/tokens` now returns tokens with `beta` filled in
for older records that were missing it.
- This aligns Go behavior with the Python implementation.
## Summary
- Validate Hunyuan embedding model name and API key before building
requests.
- Reuse region-aware base URL validation for embedding requests.
- Replace the stale unsupported Embed test with happy-path and
validation coverage.
## What changed
- Added early Hunyuan Embed validation for missing model names and API
keys.
- Routed Embed through the same base URL region guard used by the other
Hunyuan methods.
- Updated Hunyuan tests to configure the embedding suffix and cover
Embed success plus invalid inputs.
## Why
Hunyuan Embed is implemented, but the existing test still expected it to
be unsupported and could panic before returning a normal validation
error. This keeps the implemented embedding path aligned with the
current driver behavior and prevents nil input panics.
Closes#15087
Refs #14736
### What problem does this PR solve?
Closes#15180.
`OIDCClient.parse_id_token` in `api/apps/auth/oidc.py` read the JWT
signing
algorithm from the **unverified** JWT header and passed it through to
`jwt.decode(..., algorithms=[alg], ...)` as the trust anchor. This is
the
textbook JWT algorithm-confusion vulnerability (CWE-345 / CWE-347). Any
unauthenticated client capable of reaching the OIDC callback could take
over
an arbitrary account on any RAGFlow deployment with OIDC login enabled:
1. **`alg: "none"`** — present a JWT with `{"alg": "none"}` and no
signature segment → `jwt.decode(..., algorithms=["none"])` → PyJWT's
`NoneAlgorithm` accepts the token without verification → login as any
user.
2. **RSA / HMAC confusion** — fetch the public RSA key from the
provider's
JWKS (it's public), forge a JWT with `{"alg": "HS256"}` HMAC-signed
using the public-key bytes as the secret → `jwt.decode(...,
algorithms=["HS256"], key=public_key)` → verifier accepts → login as
any user. (Modern PyJWT independently refuses to use a PEM-formatted
key as an HMAC secret, which mitigates this leg for PEM key formats;
the fix here is the only mitigation for raw / DER / JWK octet keys and
for older PyJWT versions.)
### What changed
**`api/apps/auth/oidc.py`:**
- New module constants `_ALLOWED_OIDC_SIGNING_ALGS` (asymmetric-only:
`RS*`, `ES*`, `PS*`, `EdDSA` — explicitly excludes `none` and `HS*`)
and `_DEFAULT_OIDC_SIGNING_ALGS = ("RS256",)` (the OIDC Core 1.0 §2
spec default).
- New helper `_resolve_id_token_signing_algs(metadata)` — intersects the
provider's advertised `id_token_signing_alg_values_supported` from
`/.well-known/openid-configuration` with the safe allowlist; falls back
to RS256 when the field is missing or contains only unsafe values.
- `OIDCClient.__init__` now stores the resolved allowlist on
`self.id_token_signing_algs` — pinned once, from a trusted source, at
construction time.
- `parse_id_token` no longer calls `jwt.get_unverified_header` and no
longer reads `alg` from the JWT header. It passes
`self.id_token_signing_algs` to `jwt.decode(..., algorithms=...)`.
`PyJWKClient.get_signing_key_from_jwt` still reads the `kid` from the
header internally for JWKS lookup — that's fine, `kid` is not a
security decision; the signature still proves which key was actually
used.
**`test/testcases/test_web_api/test_auth_app/test_oidc_client_unit.py`:**
- Existing `test_parse_id_token_success_and_error` drops its
`jwt.get_unverified_header` mock (no longer called by `parse_id_token`).
- `_metadata` and `_make_client` helpers grew an optional `signing_algs`
parameter so tests can configure what the discovery document advertises.
- New `TestSSRFValidation` / algorithm-confusion regression block (7
tests):
- `test_id_token_signing_algs_default_to_rs256_when_metadata_missing`
- `test_id_token_signing_algs_intersect_metadata_with_safe_allowlist`
- `test_id_token_signing_algs_fall_back_when_only_unsafe_advertised`
- `test_id_token_signing_algs_ignores_non_string_entries`
- `test_id_token_signing_algs_handles_non_list_metadata_field`
- `test_parse_id_token_passes_pinned_algorithms_to_jwt_decode` —
sabotages `jwt.get_unverified_header` to raise on call, proving the
verification path never consults the unverified header.
- `test_parse_id_token_rejects_alg_none` — uses real PyJWT to encode an
`alg: "none"` token; `parse_id_token` raises `ValueError("Error
parsing ID Token: …")` instead of accepting it.
- `test_parse_id_token_rejects_hs256_when_allowlist_is_asymmetric` —
uses real PyJWT to forge an `alg: "HS256"` token with a non-PEM
shared secret (so PyJWT's incidental PEM-as-HMAC refusal isn't what
blocks it); `parse_id_token` raises because `HS256` is not in the
pinned allowlist.
Sanity-checked end-to-end with real PyJWT outside the project test
runner:
- `alg=none` forged token + `algorithms=["RS256"]` →
`InvalidAlgorithmError` ✓
- `alg=HS256` forged token + `algorithms=["RS256"]` →
`InvalidAlgorithmError` ✓
- Same `alg=HS256` token + `algorithms=["HS256"]` → **accepted**
({'sub': 'admin'})
— confirming the attack path was real before the fix.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
Co-authored-by: galuis116 <contact@duerrimports.com>
### Summary
Closes#15381
Every provider in `internal/entity/models/` reads its streaming response
with `bufio.NewScanner(resp.Body)` and iterates over `scanner.Scan()`.
The default `bufio.Scanner` maximum token size is 64KB, so when an
upstream sends a single SSE `data:` line larger than 64KB (long content
deltas, large tool or function call argument blobs, bundled
`reasoning_content`, or providers that emit a whole message in one
event) `scanner.Scan()` returns `false` and `scanner.Err()` returns
`bufio.ErrTooLong`. Streaming chat then ends with an error partway
through the response.
This change adds `scanner.Buffer(make([]byte, 64*1024), 1024*1024)`
immediately after every SSE scanner that was still bare, raising the cap
to 1MB. 1MB is the value already used for streaming chat in `openai.go`,
`modelscope.go`, `groq.go`, `mistral.go`, `xai.go` and the other already
patched providers (the 8MB cap in the repo is reserved for TTS and
embedding paths), so this simply converges the remaining providers onto
the established pattern. Nothing else changes: line parsing, `data:`
prefix handling, `[DONE]` detection, JSON unmarshalling, error handling,
and the existing `scanner.Err()` checks all stay the same.
Providers covered (23 scanners across 22 files): 302ai, aliyun,
baichuan, baidu, cohere, deepinfra, deepseek, gitee, huggingface,
lmstudio, minimax (the chat scanner, whose TTS scanner was already
bumped), moonshot, nvidia, ollama, openrouter, orcarouter, paddleocr,
siliconflow, tokenhub, vllm, volcengine, xunfei, zhipu-ai. `jiekouai.go`
is excluded because it is covered by the in flight #15337.
A table driven regression test (`sse_scanner_buffer_test.go`) streams a
single 128KB `data:` content delta followed by `data: [DONE]` through an
`httptest` server and asserts that `ChatStreamlyWithSender` delivers the
full content with no error across a representative subset of providers.
Without the buffer fix the test fails with `bufio.Scanner: token too
long`.
This PR also removes three duplicate declarations of the package level
`roundTripperFunc` test helper that several recently merged provider PRs
each added independently, which had left the `internal/entity/models`
test package unable to compile. The helper now lives in a single place
and is shared.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
## Summary
- Fix `meta_filter()` AND logic so an empty result from an early
condition is not overwritten when a later condition matches.
- Add regression tests for empty-first AND, successful AND intersection,
and OR behavior after an empty first condition.
Fixes incorrect `/retrieval` metadata filtering when multiple AND
conditions are used and the first condition matches no documents.
Closes#15360
## Test plan
- [x] `pytest test/unit_test/common/test_metadata_filter_operators.py
-v` (19/19 passed)
### What problem does this PR solve?
Fixes custom `base_url` resolution when a model instance has no
configured region.
Some drivers read custom base URLs from `BaseURL[""]` when
`apiConfig.Region` is empty, while others normalize empty region to
`"default"` and read `BaseURL["default"]`. This PR adds the `"default"`
alias only for empty-region custom base URLs while preserving the
existing empty-region key.
Closes#15042
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
### What this PR fixes
This PR fixes an issue in the Python backend where user logout did not
reliably persist the invalidated access_token to the database.
Although the logout endpoint returned success and logged that the token
had been invalidated, the user.access_token value could remain
unchanged in the database, which meant the previous login token could
stay valid longer than expected.
### What changed
- Resolve the real user object before updating the token
- Persist the invalidated access_token before calling logout_user()
- Return a server error if the token update is not written successfully
### Impact
- Logging out now correctly replaces the stored access_token with an
INVALID_... value
- The previous login session is properly invalidated
- The change is limited to the logout flow and is intentionally small in
scope
### What problem does this PR solve?
This PR fixes several behavior gaps in the Go implementation of the user
registration API.
### Type of change
- Make `nickname` required for user registration.
- Align registration error messages and response data with expected API
behavior.
- Handle password decryption errors for registration more consistently.
- Generate UUID v1-style IDs for new users, access tokens, tenants,
user-tenant records, and root files.
- Initialize default user fields during registration, including:
- language
- color schema
- timezone
- last login time
- Create user, tenant, user-tenant relation, tenant LLM records, and
root folder in a single DB transaction.
- Initialize default tenant LLM records from configured default models.
- Avoid partial registration data when one creation step fails.
- Use locale-based default language fallback for user profile responses.
### What problem does this PR solve?
Added 4 new models:
deepseek-ai/DeepSeek-V4-Pro
deepseek-ai/DeepSeek-V4-Flash
Pro/moonshotai/Kimi-K2.6
Pro/zai-org/GLM-5.1
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
### What problem does this PR solve?
Closes#15330.
RAGFlow had no connector for OneDrive / OneDrive for Business. Users who
store working documents in OneDrive could not index them into a
knowledge base without manually downloading and re-uploading files.
This PR adds a net-new OneDrive data source that:
- Authenticates against Microsoft Graph with the same MSAL
client-credentials flow already used by the SharePoint and Teams
connectors (no new auth primitives).
- Enumerates every drive visible to the service principal and pages
through `/drives/{id}/root/delta`, persisting `@odata.deltaLink` values
per drive so subsequent syncs only fetch changed items.
- Optionally narrows ingestion to a sub-folder (`folder_path`) without
needing a separate code path.
- Surfaces typed errors on the validation probe (`GET /drives?$top=1`):
401 → `ConnectorMissingCredentialError`, 403 →
`InsufficientPermissionsError` (with a `Files.Read.All` hint), 5xx →
`UnexpectedValidationError`.
- Filters folders, soft-deleted items, and unsupported extensions (`.pdf
.docx .doc .xlsx .xls .pptx .ppt .txt .md .csv`).
#### Files
| File | Change |
|------|--------|
| `common/data_source/onedrive_connector.py` | **New** —
`OneDriveConnector` + `OneDriveCheckpoint`. |
| `common/data_source/config.py` | `DocumentSource.ONEDRIVE =
"onedrive"`. |
| `common/constants.py` | `FileSource.ONEDRIVE = "onedrive"`. |
| `common/data_source/__init__.py` | Export `OneDriveConnector`. |
| `rag/svr/sync_data_source.py` | `OneDrive(SyncBase)` with `batch_size`
normalisation; registered in `func_factory`. |
| `web/src/pages/user-setting/data-source/constant/index.tsx` |
`DataSourceKey.ONEDRIVE`, visibility map (`syncDeletedFiles: true`),
info entry, form fields (tenant_id, client_id, client_secret,
folder_path, batch_size), default values. |
| `web/src/locales/en.ts`, `web/src/locales/zh.ts` |
`onedriveDescription` + 4 tooltip keys (EN + ZH). |
| `test/unit_test/data_source/test_onedrive_connector_unit.py` | **New**
— 13 unit tests (`p1`/`p2`) covering auth, validation, checkpoint
helpers, and document filtering. |
#### Required Azure AD permission
`Files.Read.All` (Application, admin-granted).
#### Out of scope
- Interactive end-user OAuth (delegated permissions) — the connector
uses app-only credentials, consistent with the SharePoint / Teams
precedent.
- Binary download of file contents — the sync layer emits `Document`s
carrying `webUrl` + metadata; bytes are hydrated downstream by the parse
pipeline.
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
### What problem does this PR solve?
default OpenAI chat completions to non-stream when `stream` is omitted
https://github.com/infiniflow/ragflow/issues/15356
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
### What problem does this PR solve?
Python implementation of the Go-based model_provider API suite.
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
---------
Co-authored-by: bill <yibie_jingnian@163.com>
## Summary
- Harden `NewNovitaModel` to avoid panics when `http.DefaultTransport`
is a custom non-`*http.Transport` RoundTripper.
- Fallback to a safe transport (`ProxyFromEnvironment`) while preserving
existing pooling/timeout settings.
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Jin Hai <haijin.chn@gmail.com>
### What problem does this PR solve?
Closes#15199.
The add-custom-model endpoint is routed through
`/api/v1/providers/:provider_name/instances/:instance_name/models`, but
the handler previously trusted `provider_name` and `instance_name` from
the JSON body instead of the path target. A request could therefore hit
one provider/instance URL while operating on a different body
provider/instance.
The same handler only rejected `model_types` when the slice was nil. An
empty array passed validation and reached
`ModelProviderService.AddCustomModel`, where `request.ModelTypes[0]`
could panic.
This PR makes the path provider/instance authoritative, rejects
mismatched body values, rejects missing or empty `model_types`, and adds
a service-level guard so direct service callers cannot hit the same
panic path.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
## Summary
Ports the Python `tenant_api` team/member management endpoints to Go,
adding 4 endpoints under `/api/v1/tenants/:tenant_id/`:
- `GET /tenants/:tenant_id/users` — list non-owner members with user
details (owner only)
- `POST /tenants/:tenant_id/users` — invite a user by email; creates
invite-role join record (owner only)
- `DELETE /tenants/:tenant_id/users` — remove a member by `user_id`;
owner can remove anyone, members can remove themselves
- `PATCH /tenants/:tenant_id` — accept a pending invitation,
transitioning role `invite → normal`
Closes#15294
### What problem does this PR solve?
As title
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
- [x] New Feature (non-breaking change which adds functionality)
- [x] Refactoring
## Summary
- Harden JieKouAI request validation before outbound provider calls
- Force non-streaming and streaming chat methods to use their expected
stream modes
- Make model listing use a bodyless GET and parse model responses
without panics
Closes#14736
---------
Co-authored-by: Jin Hai <haijin.chn@gmail.com>
## Summary
Closes#15268.
The `UpdateMetadataSetting` handler at `internal/handler/kb.go:126`
retrieved the authenticated user via `GetUser(c)` but discarded the user
object (`_, errorCode, errorMessage := GetUser(c)`), then forwarded the
caller-supplied `kb_id` straight to the service layer with no ownership
check. Any authenticated user could mutate the `parser_config` /
metadata of any knowledge base in the system by guessing or harvesting a
`kb_id` — a classic IDOR (CWE-284, OWASP A01).
This is the only handler in `internal/handler/kb.go` missing the check;
every sibling (`ListTags`, `ListTagsFromKbs`, `RenameTag`,
`KnowledgeGraph`, `DeleteKnowledgeGraph`, `GetMeta`, `GetBasicInfo`)
already calls `h.kbService.Accessible(kbID, user.ID)`. The same
defensive check on the document preview endpoint was added in PR #14625
— this PR closes the matching gap on the KB metadata endpoint.
---------
Co-authored-by: Jin Hai <haijin.chn@gmail.com>
## Summary
- Harden `NewModelScopeModel` to avoid panics when
`http.DefaultTransport` is a custom non-`*http.Transport` RoundTripper.
- Fallback to a safe transport (`ProxyFromEnvironment`) while preserving
existing pooling/timeout settings.
- Add `TestModelScopeNewModelWithCustomDefaultTransport` regression
coverage.
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Jin Hai <haijin.chn@gmail.com>
## Summary
Closes: #15328
- Implements `GET /api/v1/agents` — the agent/canvas listing endpoint
needed to complete the Home dashboard tile in `web/src/pages/home/`.
- Mirrors Python `api/apps/restful_apis/agent_api.py::list_agents`
exactly: tenant-join auth, optional `owner_ids` guard, keyword filter,
pagination, ordering, and `canvas_category` filter (default:
`agent_canvas`).
- **Scope:** read-only list only. Full agent CRUD and canvas runtime are
explicitly out of scope (separate slice of #15240).
## Summary
Ports the connector (data source) management endpoints that power
`web/src/pages/user-setting/data-source/` from Python
(`api/apps/restful_apis/connector_api.py`) to Go. Previously only `GET
/connectors` (list) was implemented in Go; this adds the rest of the
lifecycle.
Closes#15273 (subtask of #15240).
## Endpoints implemented
All under base path `/api/v1` (mirrors the Python routes):
| Method | Path | Description |
|--------|------|-------------|
| POST | `/connectors/{connector_id}/test` | Validate stored credentials
|
`GET /connectors` (list) was already present and is unchanged.
---------
Co-authored-by: Jin Hai <haijin.chn@gmail.com>
### What problem does this PR solve?
Closes#15191.
RAGFlow shipped a Microsoft Teams connector stub
(`common/data_source/teams_connector.py`) whose document-loading methods
all returned `[]`, `Teams._generate()` was a `pass`, and Teams was
commented out of the data-source settings UI. As a result there was no
way to index Teams channel conversations into a knowledge base.
This PR implements the connector end to end on top of Microsoft Graph
(Office365-REST-Python-Client). It shares the MSAL client-credentials
auth shape with the SharePoint connector.
**Backend**
- `common/data_source/teams_connector.py`
- `load_credentials()` now builds the Graph client using an MSAL
client-credentials **token callback** — the form `GraphClient` actually
expects. (The previous stub passed a raw access-token string to
`GraphClient(...)`, which is not how that client is driven.) Token
acquisition is lazy, so credential loading performs no network call.
- `validate_connector_settings()` lists teams via Graph.
- `load_from_checkpoint()` is now a generator that pages teams →
channels → messages, flattens each top-level post together with its
replies into one blob-based `Document` (`extension` `.txt`/`.html`,
`blob`, `size_bytes`, `doc_updated_at`). Incremental syncs are bounded
by message `lastModifiedDateTime` (falling back to `createdDateTime`).
Per-message errors surface as `ConnectorFailure` instead of aborting the
run.
- `retrieve_all_slim_docs_perm_sync()` yields id-only `SlimDocument`
batches and the checkpoint helpers return proper `TeamsCheckpoint`s.
- ACL → `ExternalAccess` mapping is intentionally left best-effort
(`load_from_checkpoint_with_perm_sync` delegates to the standard load)
because the sync pipeline does not currently persist `ExternalAccess`.
- `rag/svr/sync_data_source.py`
- Implemented `Teams._generate()` using the existing
`CheckpointOutputWrapper` pattern (same shape as Confluence/Jira/Google
Drive), supporting full reindex and incremental polling from
`poll_range_start`.
- `TeamsConnector` is already exported from
`common/data_source/__init__.py`.
**Frontend (`web/`)**
- Enabled the `TEAMS` data-source enum and added its form fields
(`tenant_id`, `client_id`, `client_secret`), default values, display
metadata, and a Teams icon.
- Added `teamsDescription` / `teamsTenantIdTip` to `en.ts` and `zh.ts`.
**Tests**
- `test/unit_test/data_source/test_teams_connector_unit.py`: mock-based
unit tests covering credential loading (incomplete creds raise, happy
path sets the Graph client, fetch-without-creds raises), post/reply
flattening (incl. the HTML vs text extension), incremental
`lastModifiedDateTime` filtering, and slim-doc listing. All 6 pass;
`ruff check` is clean.
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
## Summary
- Harden `NewVoyageModel` to avoid panics when `http.DefaultTransport`
is a custom non-`*http.Transport` RoundTripper.
- Fallback to a safe transport (`ProxyFromEnvironment`) while preserving
existing pooling/timeout settings.
- Add `TestVoyageNewModelWithCustomDefaultTransport` regression
coverage.
Co-authored-by: Cursor <cursoragent@cursor.com>
## Summary
- Harden `NewLongCatModel` to avoid panics when `http.DefaultTransport`
is a custom non-`*http.Transport` RoundTripper.
- Fallback to a safe transport (`ProxyFromEnvironment`) while preserving
existing pooling/timeout settings.
- Add `TestLongCatNewModelWithCustomDefaultTransport` regression
coverage.
Co-authored-by: Cursor <cursoragent@cursor.com>
### What problem does this PR solve?
implement delete, rebuild api for connector
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
### What problem does this PR solve?
Closes#15187.
RAGFlow shipped a Slack connector
(`common/data_source/slack_connector.py`) but it was never usable:
`Slack._generate()` in the sync worker was a `pass` stub, the
connector's document-generating code was incompatible with the current
data model,
and Slack was commented out of the data-source settings UI. As a result,
teams had no way to index Slack channels/threads into a knowledge base.
This PR completes the connector end to end.
**Backend**
- `common/data_source/slack_connector.py`
- Rewrote `thread_to_doc` to produce a blob-based `Document`
(`extension`/`blob`/`size_bytes`). The previous implementation built the
doc with a `sections=[...]` argument and omitted the now-required
`blob`/`extension`/ `size_bytes` fields, so it raised a validation error
against the current `Document` model. Thread messages are now cleaned
and flattened into a single UTF-8 text blob.
- Added `load_from_state()` / `poll_source(start, end)` generators. The
connector's checkpoint interface is a no-op stub, so both full and
incremental syncs run through a single channel-iterating generator built
on the existing module helpers (`get_channels`, `filter_channels`,
`get_channel_messages`, `_process_message`), with per-channel thread
de-duplication.
- `rag/svr/sync_data_source.py`
- Implemented `Slack._generate()`. Credentials are loaded via
`StaticCredentialsProvider` (the connector requires `slack_bot_token`
and does not support `load_credentials`). Supports full reindex and
incremental polling from `poll_range_start`, plus the optional channel
filter. Modeled on the Confluence/Dropbox wrappers.
- `SlackConnector` was already exported from
`common/data_source/__init__.py`.
**Frontend (`web/`)**
- Enabled the `SLACK` data-source enum and added its form fields (Slack
bot token + optional channel filter), default values, display metadata,
and a Slack icon.
- Added `slackDescription` / `slackBotTokenTip` / `slackChannelsTip`
strings to `en.ts` and `zh.ts`.
**Tests**
- `test/unit_test/data_source/test_slack_connector_unit.py`: unit tests
covering credential loading (`load_credentials` raises,
`set_credentials_provider` initializes clients, missing credentials
raises) and document generation (standalone message + flattened thread,
blob/extension/size_bytes/metadata, and the incremental poll time
window). All 5 pass; `ruff check` is clean.
Required Slack scopes: `channels:read`, `channels:history`,
`users:read`.
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
## Summary
- Harden the 302.AI model driver request validation and response parsing
paths.
- Add focused tests for chat request mode, model listing, malformed
provider responses, and input validation.
## What changed
- Validate API keys, model names, rerank queries, ASR file paths, OCR
inputs, parse URLs, task IDs, and model-list IDs before use.
- Keep chat and streaming methods from accepting conflicting `stream`
values in request payloads.
- Send `ListModels` as a bodyless GET and parse the response with typed
JSON structs instead of unchecked assertions.
- Remove raw SSE event logging from stream handling.
## Why
The driver could panic or send inconsistent requests when optional
config fields were nil, empty, malformed, or contradicted the method
path. This keeps provider-driver behavior explicit while preserving the
existing supported 302.AI flows.
Closes#14736
### What problem does this PR solve?
Fixes the `/user/me` response so it returns the current user's model
settings correctly.
### Type of change
- Added model settings data to the `/user/me` response.
- Kept the response structure compatible with existing user profile
fields.
- Avoided changing unrelated user/session behavior.
## Summary
- Add Go REST support for `GET /api/v1/system/healthz`.
- Return Python-compatible `ok`/`nok` dependency fields for DB, Redis,
document engine, and storage.
- Return HTTP 200 only when all checks pass; otherwise return HTTP 500
with `_meta` failure details.
- Add focused service coverage for the unhealthy dependency response
when Go dependencies are not initialized.
## Scope
This is a small, isolated slice of #15240. It avoids current open
connector PRs (#15274, #15300, #15265, #15264), tenant/member PRs
(#15295, #15301, #15276), MCP PRs (#15281, #15253, #15254, #15260,
#15261, #15262), and the memory-message PR (#15256).
Refs #15240
### What problem does this PR solve?
Closes#15189.
RAGFlow shipped a SharePoint connector stub
(`common/data_source/sharepoint_connector.py`) whose document-loading
methods all returned `[]`, `SharePoint._generate()` was a `pass`, and
SharePoint was commented out of the data-source settings UI. As a result
there was no way to index files stored in SharePoint document libraries.
This PR implements the connector end to end on top of Microsoft Graph
(Office365-REST-Python-Client).
**Backend**
- `common/data_source/sharepoint_connector.py`
- `load_credentials()` now builds the Graph client using an MSAL
client-credentials **token callback** — the form `GraphClient` actually
expects. (The previous stub passed a raw access-token string to
`GraphClient(...)`, which is not how that client is driven.) Token
acquisition is lazy, so credential loading does no network call.
- `validate_connector_settings()` resolves the configured site via
Graph.
- `load_from_checkpoint()` is now a generator that enumerates every
document library under the site, walks folders depth-first, downloads
each file, and yields blob-based `Document` objects (`extension` /
`blob` / `size_bytes` / `doc_updated_at`). Incremental syncs are bounded
by file `lastModifiedDateTime`. Per-file errors are surfaced as
`ConnectorFailure` rather than aborting the run.
- `retrieve_all_slim_docs_perm_sync()` yields id-only `SlimDocument`
batches (no downloads) and the checkpoint helpers return proper
checkpoints.
- ACL → `ExternalAccess` mapping is intentionally left best-effort
(`load_from_checkpoint_with_perm_sync` delegates to the standard load)
because the sync pipeline does not currently persist `ExternalAccess`;
this can be extended once that plumbing exists.
- `rag/svr/sync_data_source.py`
- Implemented `SharePoint._generate()` using the existing
`CheckpointOutputWrapper` pattern (same shape as Confluence/Jira/Google
Drive), supporting full reindex and incremental polling from
`poll_range_start`.
- `SharePointConnector` is already exported from
`common/data_source/__init__.py`.
**Frontend (`web/`)**
- Enabled the `SHAREPOINT` data-source enum and added its form fields
`site_url`, `tenant_id`, `client_id`, `client_secret`), default values,
display metadata, and a SharePoint icon.
- Added `sharepointDescription` / `sharepointSiteUrlTip` to `en.ts` and
`zh.ts`.
**Tests**
- `test/unit_test/data_source/test_sharepoint_connector_unit.py`:
mock-based unit tests covering credential loading (incomplete creds
raise, happy path sets the Graph client, fetch-without-creds raises),
drive traversal + file download, incremental `lastModifiedDateTime`
filtering, and slim-doc listing. All 6 pass; `ruff check` is clean.
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
### What problem does this PR solve?
extend restful api suite
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
- [x] Other (please describe): test
### What problem does this PR solve?
- Add Go implementation parity for `PATCH /api/v1/users/me`.
- This updates the Go user settings endpoint to match the Python
behavior for updating the current user's profile settings.
### Changes
- Route `PATCH /api/v1/users/me` through the authenticated current user
from middleware.
- Add `password` and `new_password` support to `UpdateSettingsRequest`.
- Prevent `email` from being updated through this endpoint, matching the
Python blacklist behavior.
- Support updating:
- `nickname`
- `avatar`
- `language`
- `color_schema`
- `timezone`
- `password`
- Align password handling with Python:
- invalid plaintext password payload returns `CodeExceptionError`
- wrong old password returns `Password error!`
- successful update returns `{ code: 0, data: true, message: "success"
}`
### Test
Tested manually with Python and Go backends using the same request
bodies:
- `PATCH /api/v1/users/me` with nickname/timezone update
- plaintext password payload returns Python-compatible `Incorrect
padding`
- wrong old password returns `Password error!`
### What problem does this PR solve?
1. Break huge function into smaller pieces
2. Add unit test for the smaller pieces function
3. Layer-ed design
a. infra layer - task_context.py, recording_context.py,
write_operation_interceptor.py, ...
b. service layer - *_service.py
c. business layer - task_handler.py
4. Default behavior: use "refactor-ed version" - can switch to original
version by change env variable
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
- [x] Refactoring
- [x] Performance Improvement
---------
Co-authored-by: Liu An <asiro@qq.com>
Co-authored-by: Zhichang Yu <yuzhichang@gmail.com>
Wrap per-statement execution in both the generic and IBM DB2 loops so a
failing statement reports a friendly "SQL Execution Failed" message and
continues, instead of letting a raw driver exception abort the node and
discard results from statements that already succeeded.
Rolls back after a failure so PostgreSQL's aborted-transaction state
does not cascade into every subsequent statement in the batch.
### What problem does this PR solve?
Closes#14737
The **ExeSQL** agent node splits its input on `;` and runs each
statement in a loop. Both execution loops — the generic one
(`cursor.execute`) and the IBM DB2 one (`ibm_db.exec_immediate`) — were
wrapped only in a `try/finally` for resource cleanup, with **no
`except`** around statement execution.
As a result, when any single statement failed (e.g. the reporter's MSSQL
`('42S02', "[42S02] ... 对象名 'ASSET_AUDIT' 无效")`):
- The raw, unformatted driver exception bubbled up and the node failed
with an ugly `_ERROR` instead of friendly information.
- **The whole node aborted** — results from statements that had already
succeeded were discarded, and the remaining statements in the batch
never ran. The reporter confirmed this was the real pain point: *"after
reporting an exception, the previous normal query cannot be executed
properly … Do not interrupt the workflow for any issues."*
Connection-level failures were already wrapped with a friendly
`"Database Connection Failed!"` prefix — only per-statement execution
errors were missed.
**This PR** wraps per-statement execution in `try/except` in both loops.
A failing statement now:
- records a friendly `SQL Execution Failed: <sql>\n<error>` entry into
the `json` and `formalized_content` outputs (the actual DB error is kept
so the user can see *what* failed), and
- `continue`s to the next statement — so earlier results survive and
later statements still run.
After a failure in the generic loop, the connection is rolled back so
PostgreSQL's aborted-transaction state does not cascade into every
subsequent statement in the batch. The node returns normally (no
`_ERROR` raised), so the agent workflow proceeds instead of halting.
Connection failures remain fatal (correct — nothing can run without a
connection). The pre-existing `break` on `cursor.rowcount == 0` is
intentionally left unchanged; it is out of scope for this fix.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
### What problem does this PR solve?
implement create_connector API
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
### What problem does this PR solve?
Fix: Uploading TSV format documents to the knowledge base did not
generate any error messages.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
## Summary
- centralize TokenHub chat request validation for chat and streaming
calls
- reject blank TokenHub model names before sending provider requests
- send TokenHub model listing requests as bodyless GET requests
## What changed
- Added shared TokenHub chat request validation for API key, model name,
and messages.
- Updated `ListModels` to call `GET /models` without a request body.
- Added focused tests for blank model names and accidental GET request
bodies.
- Replaced an httptest handler callback `t.Fatalf` with `t.Errorf` plus
an HTTP error and return.
## Why
TokenHub chat requests should fail locally for invalid model names
instead of sending avoidable malformed requests upstream. Model listing
should also match normal GET semantics and avoid sending an empty JSON
body.
Closes#14736
Co-authored-by: Jin Hai <haijin.chn@gmail.com>
### What problem does this PR solve?
As title
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
- [x] New Feature (non-breaking change which adds functionality)
- [x] Refactoring
## Summary
- Add Go REST support for `GET /api/v1/connectors/:connector_id`.
- Reuse the Python API behavior by returning the connector only when the
current user can access its tenant.
- Add focused handler coverage for success and unauthorized responses.
Co-authored-by: Jin Hai <haijin.chn@gmail.com>
### What problem does this PR solve?
`ReplicateModel.Rerank` in `internal/entity/models/replicate.go` was a
`"replicate, no such method"` stub. The chat path landed in #14958 and
the embed path in #15073; rerank is the last major retrieval surface
still missing on this provider.
Until this PR, a tenant who selected a Replicate reranker model got the
sentinel error on every rerank call.
Co-authored-by: sxxtony <sxxtony@users.noreply.github.com>
Co-authored-by: Jin Hai <haijin.chn@gmail.com>
### What problem does this PR solve?
Closes#15090.
Adds GiteeAI support to the Go model-provider layer so GiteeAI chat
models can be routed through the Go API server using the same
OpenAI-compatible chat, streaming, model listing, and connection-check
flow used by other SaaS providers.
GiteeAI is implemented as a separate provider from the existing `gitee`
provider.
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
## Summary
- Added a GiteeAI Go model driver.
- Added the GiteeAI provider catalog with default base URL
`https://ai.gitee.com/v1`.
- Registered `giteeai` in the model factory separately from `gitee`.
- Added focused provider tests for sync chat, streaming chat, model
listing, connection checks, base URL override, SSE parsing, `[DONE]`
handling, and unsupported methods.
## What changed
- Implemented `ChatWithMessages` for `POST /chat/completions`.
- Implemented `ChatStreamlyWithSender` with SSE parsing, `delta`
extraction, `finish_reason`, `[DONE]`, and `<think>` tag handling.
- Implemented `ListModels` for `GET /models`.
- Implemented `CheckConnection` by delegating to `ListModels`.
- Returned standard `no such method` errors for unsupported embedding,
rerank, image-to-text, ASR, and TTS paths.
## Tests
```bash
go test -vet=off ./internal/entity/models -run 'TestGiteeAI' -count=1
go test -vet=off ./internal/entity -run 'Test.*Provider|Test.*Model' -count=1
```
---------
Co-authored-by: Jin Hai <haijin.chn@gmail.com>
### What problem does this PR solve?
This PR adds Qiniu provider integration for the Go model driver layer in
RAGFlow.
Supported capabilities:
- [X] Chat
- [X] Think Chat
- [X] Stream Chat
- [X] Stream Think Chat
- [X] Model listing
- [X] Provider configuration and factory registration
Verified examples from the CLI:
```
login user '***' password '***';
ADD PROVIDER 'qiniu';
CREATE PROVIDER 'qiniu' INSTANCE 'test' KEY '***';
chat with 'deepseek/deepseek-v3.1-terminus-thinking@test@qiniu' message
'hello';
think chat with 'deepseek/deepseek-v3.1-terminus-thinking@test@qiniu'
message 'hello';
stream chat with 'deepseek/deepseek-v3.1-terminus-thinking@test@qiniu'
message 'hello, what are you';
stream think chat with
'deepseek/deepseek-v3.1-terminus-thinking@test@qiniu' message 'hello,
what are you';
stream think chat with 'qwen3-max-2026-01-23@test@qiniu' message 'hello,
what are you';
LIST MODELS FROM 'qiniu' 'test';
```
### Type of change
- [X] New Feature
- [X] Provider integration
## Summary
- add the VolcEngine `models` URL suffix used by the existing Go
`ListModels` implementation
- return a clear error when the VolcEngine models suffix is missing
- add focused VolcEngine model-listing regression tests
## What changed
- Added `url_suffix.models` to `conf/models/volcengine.json`.
- Normalized the configured models suffix before building the request
URL.
- Covered config loading, successful model listing, upstream errors, and
missing suffix handling.
## Why
`VolcEngine.ListModels` already builds requests from `URLSuffix.Models`,
but the bundled VolcEngine config did not define that suffix. That left
the model-listing path unable to call the documented `/models` endpoint
from the existing provider config.
Fixes#14701
Co-authored-by: Jin Hai <haijin.chn@gmail.com>
### What problem does this PR solve?
extend restful api suite
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
- [x] Other (please describe): test
### What problem does this PR solve?
Fix: The Creativity parameter of chat was not saved.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
## Summary
- route hosted MinerU.Net and PaddleOCR.Net provider names to their
existing Go drivers
- add regression coverage for loading the hosted OCR provider configs
through ProviderManager
## What changed
- Added canonical provider-name aliases for the hosted OCR provider
display names.
- Covered both bundled configs with a focused provider-manager test.
## Why
The hosted provider configs use display names with `.Net`, while model
factory dispatch lowercases the provider name. Without aliases, those
configs fall through to `DummyModel` instead of using the existing
MinerU and PaddleOCR drivers.
---------
Co-authored-by: Jin Hai <haijin.chn@gmail.com>
## Summary
- Add LongCat model-list support through the documented
OpenAI-compatible models endpoint.
## What changed
- Add the LongCat `models` URL suffix for `/openai/v1/models`.
- Implement `ListModels` for the LongCat Go driver.
- Delegate `CheckConnection` to the lightweight model-list request.
- Add focused regression coverage for successful, malformed, oversized,
and missing-key responses.
## Why
LongCat documents a models endpoint under the OpenAI-compatible API
surface, but the Go driver still returned `no such method` for model
listing and connection checks.
## Validation
- `go test ./internal/entity/models -run TestLongCat -count=1`
- `go test -race ./internal/entity/models -run TestLongCat -count=1`
- `go test ./internal/entity -count=1`
- `git diff --check`
## Notes
- Related to the broader Go model provider tracking in #14736, but this
PR only handles LongCat model listing.
- `go test ./internal/entity/models -count=1` is currently blocked by an
unrelated Astraflow test panic outside this LongCat change.
---------
Co-authored-by: Jin Hai <haijin.chn@gmail.com>
## Summary
- add the xAI `models` URL suffix used by the existing Go `ListModels`
implementation
- return a clear error when the xAI models suffix is missing
- add focused xAI model-listing and connection-check regression tests
## What changed
- Added `url_suffix.models` to `conf/models/xai.json`.
- Normalized the configured models suffix before building the request
URL.
- Covered config loading, successful model listing, upstream errors,
API-key validation, missing suffix handling, and `CheckConnection`
delegation.
## Why
`XAIModel.ListModels` already builds requests from `URLSuffix.Models`,
and `CheckConnection` delegates to that method. The bundled xAI config
did not define that suffix, which left the model-listing path unable to
call the provider `/models` endpoint from the existing provider config.
## Validation
- `go test ./internal/entity/models -run TestXAI -count=1`
- `go test ./internal/entity -count=1`
- `git diff HEAD~1..HEAD --check`
## Notes
- `go test ./internal/entity/models -count=1` currently fails in
unchanged Astraflow coverage: `TestAstraflowEmbedReturnsNoSuchMethod`
panics before reaching any xAI assertions.
---------
Co-authored-by: Jin Hai <haijin.chn@gmail.com>
### What problem does this PR solve?
- Update version tags in README files (including translations) from
v0.25.5 to v0.25.6
- Modify Docker image references and documentation to reflect new
version
- Update version badges and image descriptions
- Maintain consistency across all language variants of README files
### Type of change
- [x] Documentation Update
### What problem does this PR solve?
implement provider `OrcaRouter`
**The following functionalities are now supported:**
**Cohere:**
- [x] Chat / Think Chat / Stream Chat / Stream Think Chat
- [x] Model listing
- [x] TTS
- [ ] Balance
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
---------
Co-authored-by: Jin Hai <haijin.chn@gmail.com>
Closes#15040.
ModelScope was listed unchecked in the Go-rewrite tracker #14736 and
already had an llm_factories.json entry (tags: LLM) but no Go driver, so
the new Go API server could not route ModelScope instances. The Python
side has supported it through the OpenAI-compatible base at
rag/llm/chat_model.py:618 (ModelScopeChat), which requires a
user-supplied base URL and appends /v1.
This adds:
- internal/entity/models/modelscope.go: self-hosted OpenAI-compatible
driver with chat (sync + SSE stream with idle-timeout cancellation),
list_models, and check_connection. Auth header is optional, matching the
xinference pattern, so deployments without auth and auth-enabled
deployments both work. Base URL is normalized so users can configure
either the root endpoint or the /v1 endpoint.
- internal/entity/models/modelscope_test.go: 12 tests covering name, URL
normalization, factory routing, chat happy path / auth header /
reasoning_content extraction, stream happy path / stream=false rejection
/ idle cancellation, list_models + check_connection, missing-base-URL
clear error, and the no-such-method sentinels.
- conf/models/modelscope.json: shipped config (class: "local",
url_suffix v1/chat/completions and v1/models).
- internal/entity/models/factory.go: case "modelscope" →
ModelScopeModel.
- internal/service/llm.go: ModelScope added to the selfDeployed map
alongside Ollama, Xinference, LocalAI, LM-Studio, GPUStack — the Python
side requires user-supplied URL with no default, so the Go side
classifies it the same way.
Follow-on issues will add Embed and Rerank, in line with how Novita,
NVIDIA, TogetherAI, and other providers landed method-by-method.
---------
Co-authored-by: Jin Hai <haijin.chn@gmail.com>
### What problem does this PR solve?
This PR adds HuaweiCloud provider integration in RAGFlow.
Supported capabilities:
- [x] Chat / Think Chat / Stream Chat / Stream Think Chat
- [x] Embedding
- [x] Rerank
- [x] Model listing
- [x] Provider connection checking
Verified examples from the CLI:
```
check instance 'test' from 'HuaweiCloud';
chat with 'deepseek-v4-flash@test@HuaweiCloud' message 'hello';
think chat with 'deepseek-v4-flash@test@HuaweiCloud' message 'hello';
stream chat with 'deepseek-v4-flash@test@HuaweiCloud' message 'hello';
stream think chat with 'deepseek-v4-flash@test@HuaweiCloud' message
'hello';
embed text 'what is rag' 'who are you' with 'bge-m3@test@HuaweiCloud'
dimension 1024;
rerank query 'what is rag' document 'rag is retrieval augmented
generation' 'rag need llm' 'famous rag
project includes ragflow' with 'bge-reranker-v2-m3@test@HuaweiCloud' top
3;
list supported models from 'HuaweiCloud' 'test';
LIST MODELS FROM 'HuaweiCloud' 'test';
```
### Type of change
- [x] New Feature
- [x] Provider integration
## Summary
- Wire the Go TokenHub provider through the model factory.
- Harden TokenHub request handling for chat, streaming, embeddings, and
model listing.
- Add focused TokenHub unit coverage for factory wiring and provider
behavior.
## Notes
- Refs #14736.
- Follows up #15159.
Co-authored-by: Jin Hai <haijin.chn@gmail.com>
## Summary
Closes#15165.
Implements the AWS Bedrock model provider for the Go API server, tracked
under #14736. Adds Converse + Converse-Stream chat and foundation-model
listing, with SigV4 signing over a hand-rolled `net/http` path that
matches the established pattern in `internal/entity/models/` (no new
direct `go.mod` deps).
## Linked tracker
Tracked under #14736 (Implement model providers of RAGFlow API server in
Go). Closes#15165.
### What problem does this PR solve?
extend restful api suite
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
- [x] Other (please describe): test
### What problem does this PR solve?
Fix: Fixed metadata issue
- The dataset's built-in metadata is now active, but it appears to be
disabled in the individual file configuration.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
### What problem does this PR solve?
Creating or updating an agent via `POST /api/v1/agents` and `PUT
/api/v1/agents/{agent_id}` did not persist `canvas_type` because the
handler `req` dict never assigned the field before
`UserCanvasService.save` / `update_by_id`.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
- [ ] New Feature (non-breaking change which adds functionality)
- [ ] Documentation Update
- [ ] Refactoring
- [ ] Performance Improvement
- [ ] Other (please describe):
Co-authored-by: Cursor <cursoragent@cursor.com>
### What problem does this PR solve?
The Go DeepInfra driver returned a stub error for `Rerank()` even though
DeepInfra serves reranker models at `POST /v1/inference/{model}` with
`query`, `documents`, and a `scores[]` response.
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
Co-authored-by: Cursor <cursoragent@cursor.com>
### What problem does this PR solve?
Add a Go driver for **FuturMix** (https://futurmix.ai/docs), one of the
unchecked providers on the umbrella tracking issue #14736. FuturMix is
documented as an "OpenAI-compatible API" aggregator over Claude / GPT /
Gemini / DeepSeek (~22 models per their `/models` page).
Until this PR, a tenant who configured `futurmix` as a model provider in
the Go layer fell through to the default branch of
`internal/entity/models/factory.go` and got the dummy driver.
---------
Co-authored-by: sxxtony <sxxtony@users.noreply.github.com>
Co-authored-by: Jin Hai <haijin.chn@gmail.com>
### What problem does this PR solve?
Closes#15167.
The Baidu Go provider advertises OCR support through
`paddleocr-vl-0.9b`, but `BaiduModel.OCRFile` dereferenced required
inputs before validating them. Calling OCR with a missing API config,
API key, or model name could panic instead of returning a normal error.
This PR adds explicit input validation for those required values.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
---------
Co-authored-by: Jin Hai <haijin.chn@gmail.com>
### What problem does this PR solve?
Closes#15142.
ZhipuAI lists `glm-ocr` as an OCR model, but the Go driver still
returned `no such method` from `OCRFile`. This wires the advertised
model to Z.AI's documented `layout_parsing` endpoint and returns the
`md_results` Markdown output through the existing `OCRFileResponse.Text`
field.
This PR also adds focused tests for URL input, raw file-content base64
input, and validation errors.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
- [ ] New Feature (non-breaking change which adds functionality)
- [ ] Documentation Update
- [ ] Refactoring
- [ ] Performance Improvement
- [ ] Other (please describe):
### Test
- [x] `go test -vet=off ./internal/entity/models -run
'TestZhipuAIOCRFile'`
### What problem does this PR solve?
extend restful api suite
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
- [x] Other (please describe): test
### What problem does this PR solve?
Fix [Bug]: Save parser configs in dataset configuration page is not
working #15175
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
## Summary
Fixes the confirmed asyncio anti-patterns from #14755. Only the three
verified bugs are addressed; patterns already correctly using
`asyncio.new_event_loop()` in a fresh thread are left untouched.
### Changes
**`api/apps/restful_apis/tenant_api.py` — fire-and-forget
`send_invite_email`**
`asyncio.create_task()` was called without storing the `Task` reference.
CPython's GC can collect an unfinished task, silently cancelling it and
swallowing exceptions. Fixed by storing the task in a module-level
`_background_tasks: set[Task]` with a `done_callback` to discard it on
completion — the standard Python idiom for safe background tasks.
**`api/apps/restful_apis/agent_api.py` — fire-and-forget
`background_run`**
Same root cause in the webhook "Immediately" execution path. Same fix
applied.
**`rag/llm/chat_model.py` (`LocalLLM._stream_response`) —
`asyncio.get_event_loop()` on running loop**
`asyncio.get_event_loop()` returns Quart's running event loop when
called from an async context.
Calling `loop.run_until_complete()` on it raises `RuntimeError`.
Replaced with `asyncio.new_event_loop()` so the generator
uses a dedicated fresh loop, closed in a `finally` block.
## What was NOT changed
- `llm_service._sync_from_async_stream` and
`evaluation_service._sync_from_async_gen`: both already correctly use
`asyncio.new_event_loop()` inside a fresh thread.
- `llm_service._run_coroutine_sync`: only caller is `rag/app/resume.py`
(sync context), so `thread.join()` is correct there.
- `requests` in agent tools: sync methods dispatched through thread
pools; httpx migration is a separate, larger refactor.
## Test plan
- [ ] Invite a team member and confirm the email is sent with no task
warnings in logs.
- [ ] Trigger a webhook agent in "Immediately" mode; confirm canvas
state is persisted after background run.
- [ ] Verify `LocalLLM` (Jina backend) chat and streaming work
end-to-end.
Closes#14755
---------
Co-authored-by: Zhichang Yu <yuzhichang@gmail.com>
### What problem does this PR solve?
Fix: The prompt variable for the agent operator disappears after input.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
### What problem does this PR solve?
extend restful api suite
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
- [x] Other (please describe): test
### What problem does this PR solve?
Feat: Enable agent messages to display base64 images
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
### What problem does this PR solve?
IDK how to implement **`Ollama`** on #14580 but it's totally wrong.
This is the rewrite version for **`Ollama`**
**Verified from CLI**
```
# Embed
RAGFlow(user)> embed text 'what is rag' 'who are you' with 'nomic-embed-text:latest@test12@ollama' dimension 1024;
+-----------+-------+
| dimension | index |
+-----------+-------+
| 768 | 0 |
| 768 | 1 |
+-----------+-------+
# Chat
RAGFlow(user)> think chat with 'qwen3:0.6b@test12@ollama' message 'who r u'
Thinking: Okay, the user asked, "Who r u?" I need to respond appropriately. First, I should acknowledge their question. Since I'm an AI, I don't have a physical form, but I can confirm that I'm a large language model. I should keep the response friendly and offer help. Let me make sure I'm not making up any information and that the response is natural. Also, I should check for any typos and ensure clarity. Alright, that should cover it.
Answer: I'm an AI language model, and I don't have a physical form. However, I can tell you that I'm designed to assist with questions and tasks. How can I help you today?
Time: 2.914285
RAGFlow(user)> stream think chat with 'qwen3:0.6b@test12@ollama' message 'who r u'
Thinking: , the user asked, "Who are you?" I need to respond appropriately. Since I'm an AI assistant, I should mention that I don't have a physical form or a mind. I should also clarify that I can help with various tasks like answering questions or providing information. It's important to keep the response friendly and informative while maintaining the correct tone.
Answer: don't have a physical form or a mind, but I'm here to help with your questions or tasks! What can I do for you today?
Time: 1.740047
# LisyModels
RAGFlow(user)> list supported models from 'ollama' 'test12'
+-------------------------+
| model_name |
+-------------------------+
| nomic-embed-text:latest |
| qwen3:0.6b |
+-------------------------+
```
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
- [x] Refactoring
### What problem does this PR solve?
Fix: Replace the red highlight at the top of the PDF document with
yellow.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
Follow on PR #15146 to reslove the backwad compatability issue.
1. /agents/<attachment_id>/download ->
/agents/attachments/<attachment_id>/download
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
## Summary
This change fixes ingestion quality issues where MinerU parser output
may contain HTML fragments (for example, table-related tags like `<tr>`,
`<td>`, `<br>`), which were previously passed directly into
chunking/tokenization and degraded chunk quality.
The fix adds a sanitization step in the MinerU parser path so parsed
sections are normalized to clean text before chunking.
## Change Type (select all)
- [x] Bug fix
- [x] Ingestion pipeline improvement
- [x] Parser/chunking quality fix
## Related Issue
- https://github.com/infiniflow/ragflow/issues/14831
### What problem does this PR solve?
This PR improves the table upload flow for CSV/Excel files by allowing
table column role configuration at upload time.
Previously, users had to:
1. Upload and parse a table file.
2. Open parser settings and manually set table column roles.
3. Re-parse the file for the roles to take effect.
This was inefficient and required an unnecessary second parse.
With this change:
1. When the knowledge base uses table parsing, the upload dialog
extracts CSV/Excel headers client-side.
2. Users can choose Auto mode or Manual mode.
3. In Manual mode, users can assign per-column roles before upload.
4. The selected parser config is sent with the upload request and
applied server-side during document creation.
Result: configured table column roles are applied from the first parse.
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
Co-authored-by: Ahmad Intisar <ahmadintisar@Ahmads-MacBook-M4-Pro.local>
Declare doc_id, filename, mime_type, and size as separate outputs on the
Document Generation component so downstream nodes (e.g., the Code
component) can consume them via the variable picker. The existing
download JSON blob is preserved unchanged for the Message component's
download-chip rendering.
### What problem does this PR solve?
The Document Generation component previously exposed only a single
`download` output —
a JSON-encoded blob containing the file's `doc_id`, `filename`,
`mime_type`, `size`,
and base64 payload. On top of that, the variable picker actively hides
this `download`
entry from every consumer except the Message component (because the
embedded base64 is
too heavy to splat into arbitrary downstream nodes).
The combined effect: users wiring the Doc Generator's output into a Code
component had
no way to retrieve basic file info such as `file_name` or `doc_id` from
the picker,
blocking workflows that need to post-process the generated file (e.g.,
registering it
elsewhere, custom delivery, follow-up API calls).
This PR declares `doc_id`, `filename`, `mime_type`, and `size` as
**discrete outputs**
on the Document Generation component, alongside the existing `download`
blob. The new
fields:
- Appear in the variable picker for **all** downstream nodes, including
the Code
component, so users can bind them directly to script arguments.
- Are cheap scalars only — no base64 payload leaks into other
components.
- Leave the existing `download` JSON blob completely untouched, so the
Message
component's download-chip rendering (which parses that blob via
`_is_download_info`)
keeps working with no behavior change.
Changes:
- `agent/component/docs_generator.py` — declare the four new outputs in
`DocGeneratorParam` and emit them via `set_output(...)` in `_invoke`.
- `web/src/pages/agent/constant/index.tsx` — extend
`initialDocGeneratorValues.outputs`
with the new keys.
- `web/src/pages/agent/form/doc-generator-form/index.tsx` — mirror the
new outputs in
the zod schema so the form is valid.
No changes needed to the picker's existing `download`-hiding filter — it
matches only
on the literal output name `download`, so the new metadata entries fall
through
naturally.
Reported in: https://github.com/infiniflow/ragflow/issues/14461.
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
### What problem does this PR solve?
Implement embed for Tencent Hunyuan
**Verified from CLI**
```
RAGFlow(user)> embed text 'what is rag' 'who are you' with 'hunyuan-embedding@test1@hunyuan' dimension 16;
+-----------+-------+
| dimension | index |
+-----------+-------+
| 1024 | 0 |
| 1024 | 1 |
+-----------+-------+
```
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
- [x] Refactoring
### What problem does this PR solve?
1. Fix /chat/completions to send only the latest message
2. Allo chat stream=False
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
### What problem does this PR solve?
Go: implement provider: PaddleOCR_Local
**Verified from CLI**
```
RAGFlow(user)> ocr with 'PaddleOCR-VL@test@paddleocr_local' file './internal/test1.jpg'
+----------------------+
| text |
+----------------------+
| ## Parallel to these |
+----------------------+
```
### Type of change
- [X] Bug Fix (non-breaking change which fixes an issue)
- [X] New Feature (non-breaking change which adds functionality)
- [X] Refactoring
## Summary
- Adds a `Hunyuan` Go driver so the new API server can route Tencent
Hunyuan chat instances (registered in `conf/llm_factories.json:3830` as
`Tencent Hunyuan`). Follows the same SaaS-driver shape used for
Astraflow, Avian, Novita, TogetherAI, Replicate, DeepInfra, Upstage, and
LongCat.
Closes#15087
---------
Co-authored-by: Jin Hai <haijin.chn@gmail.com>
### What problem does this PR solve?
Fix /chat/completions not aware of conversation_id
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
### What problem does this PR solve?
Closes#15102.
OpenAI's Go provider config advertises `whisper-1` as ASR and `tts-1` as
TTS, but the Go driver returned `openai, no such method` for both audio
paths and did not define `url_suffix.asr` / `url_suffix.tts`.
This PR:
- adds OpenAI audio URL suffixes for `audio/transcriptions` and
`audio/speech`
- implements non-streaming `TranscribeAudio` using multipart form
uploads
- implements non-streaming `AudioSpeech` using the OpenAI speech JSON
request shape
- keeps streaming TTS explicitly unsupported instead of sending binary
audio through the text SSE sender
- adds focused tests for config coverage, ASR/TTS request shape,
required TTS voice validation, and unsupported streaming TTS
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
---------
Co-authored-by: Jin Hai <haijin.chn@gmail.com>
### What problem does this PR solve?
Fix: /openai/<chat_id>/chat/completions not aware of session_id
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
### What problem does this PR solve?
Go: implement reasoning_chat, TTS, ASR for Groq
**Verify from CLI**
```
RAGFlow(user)> think chat with 'qwen/qwen3-32b@test@groq' message 'who r u'
Thinking: Okay, the user asked, who r u. I need to determine what the user is asking. They may be asking about my identity. I should introduce my name and basic functions. The user might want to know what I can do, so I should list some common use cases, such as answering questions, creating writing, coding, and expressing opinions. The user may be curious about how they can interact with me, so they can be advised to ask any questions or provide instructions. Keep your answers conversational, avoid overly technical terms, keep answers concise, and encourage further interaction. Check if there's any ambiguity in the answer and make sure it's accurate and meets the user's needs. Also consider if there are other aspects the user may be interested in, such as my training data or performance. But since the question is basic, I'll focus on the essentials first and invite the user to ask more. In summary, respond to the user's questions by introducing yourself, your functions, and encouraging further interaction.
Answer: Hello! I'm Qwen. I am a large-scale language model developed by Tongyi Lab, designed to assist you in various ways, such as answering questions, creating text, logical reasoning, programming, and more. I aim to provide clear, accurate, and helpful information and support. How can I assist you today? Feel free to ask any questions or give me tasks! 😊
Time: 2.199908
RAGFlow(user)> stream think chat with 'openai/gpt-oss-20b@test@groq' message 'who r u'
Thinking: to respond politely.
Answer: ’m ChatGPT—an AI language model created by OpenAI. I’m here to answer questions, offer explanations, and help with a wide range of topics. How can I assist you today?
RAGFlow(user)> tts with 'canopylabs/orpheus-arabic-saudi@test@groq' text 'hello? show yourself' play format 'wav' param '{"voice": "fahad"}'
SUCCESS
RAGFlow(user)> asr with 'whisper-large-v3-turbo@test@groq' audio './internal/test.wav' param '{"language": "en"}'
+----------------------------------------------------------------------------------------------------------------------+
| text |
+----------------------------------------------------------------------------------------------------------------------+
| The examination and testimony of the experts enabled the Commission to conclude that five shots may have been fired |
+----------------------------------------------------------------------------------------------------------------------+
```
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
### What problem does this PR solve?
Closes#15088.
Adds Groq support to the Go model-provider layer so Groq instances can
be routed through the Go API server with the same OpenAI-compatible
chat, streaming, model listing, and connection-check flow used by other
SaaS providers.
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
## Summary
- Added a Groq Go model driver.
- Added the Groq provider catalog and default OpenAI-compatible API URL.
- Registered Groq in the model factory.
- Added focused provider tests.
## What changed
- Implemented chat completions, SSE streaming, ListModels, and
CheckConnection for Groq.
- Covered request shape, stream termination, reasoning fallback, model
listing, custom base URLs, safe transport setup, and unsupported
methods.
- Kept the provider catalog scoped to current Groq chat-capable model
IDs.
- Cleaned up pre-existing Go model package validation blockers so the
package can be tested normally with vet enabled.
## Why
The existing Python/provider catalog path includes Groq, but the Go
model-provider layer did not have a Groq driver, so the Go API server
could not instantiate or use Groq as requested in #15088.
## Notes
The model package now validates without disabling vet.
---------
Co-authored-by: Jin Hai <haijin.chn@gmail.com>
### What problem does this PR solve?
move agent attachment download api to the correct route and update
frontend callers
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
### Notes
- Move the attachment download endpoint from document routes to agent
routes.
- Update frontend download callers to use the agent attachment endpoint.
- Reuse the shared file response header helper instead of duplicating it
in `agent_api.py`.
## Summary
- Adds a `TokenPony` Go driver so the new API server can route TokenPony
chat instances, matching the existing Python `TokenPonyChat`
(`rag/llm/chat_model.py:1210`). Follows the same SaaS-driver shape used
for Astraflow, Avian, Novita, TogetherAI, Replicate, DeepInfra, Upstage,
and LongCat.
Closes#15086
---------
Co-authored-by: Jin Hai <haijin.chn@gmail.com>
## Summary
Implements the TODO in `evaluation_service.py`: **Track token usage** in
evaluation results.
## Changes
- **Import** `num_tokens_from_string` from `common.token_utils`
- **Prompt tokens**: Use the full prompt returned by `async_chat` when
available (includes system prompt + knowledge base + query), otherwise
fall back to the question token count
- **Completion tokens**: Count tokens in the generated answer
- **Storage**: Store `token_usage` as `{prompt_tokens,
completion_tokens, total_tokens}` in each `EvaluationResult` instead of
`None`
## Why
The evaluation pipeline previously saved `token_usage: None` for every
result. This change allows downstream consumers (e.g. evaluation
dashboards, cost tracking) to see approximate token usage per test case
using the same tokenizer (tiktoken cl100k_base) used elsewhere in
RAGFlow.
## Testing
- No new tests added; existing evaluation flow unchanged
- Token counting uses existing `num_tokens_from_string` utility
---------
Co-authored-by: kiannidev <kiannidev@users.noreply.github.com>
### What problem does this PR solve?
Fixes#15066
OpenRouter now exposes an official speech-to-text endpoint at `POST
/api/v1/audio/transcriptions`, but the Go model driver still returned
`openrouter, no such method` from `TranscribeAudio`. This left
OpenRouter ASR models unavailable through the Go API server even though
the provider already has OpenRouter audio support for TTS.
Related provider-tracking context: #14736
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
Co-authored-by: Jin Hai <haijin.chn@gmail.com>
### What problem does this PR solve?
The agent API currently does not pass chat_template_kwargs to the
underlying LLM call path, so clients cannot control template-level model
behavior (such as thinking-mode toggles) when invoking
/agents/chat/completion. This PR adds passthrough support for
chat_template_kwargs across agent execution flows (session and
non-session, streaming and non-streaming) by propagating it through
canvas runtime state and into LLM invocation kwargs. This addresses the
feature gap raised in [Issue
#14182](https://github.com/infiniflow/ragflow/issues/14182).
Closes#14182
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
Closes#14789
### What problem does this PR solve?
User API endpoints (`login`, `user_profile`, `user_add`,
`forget_reset_password`) were returning full user objects via
`to_json()` / `to_dict()`, which included sensitive fields like
`password` and `access_token` in the response body. This leaks
credentials to the client.
This PR adds a `to_safe_dict()` method on the `User` model that strips
sensitive fields (`password`, `access_token`) and replaces all affected
call sites to use it.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
### What problem does this PR solve?
1. Enhance retry and timeout, and adjust the default timeout
2. NER: spacy do not batch chunks
3. extract _has_cancel_and_exit
4. enhance log messages
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
- [x] Refactoring
Closes#4310
### What problem does this PR solve?
Issue #4310 requests practical examples for the RAGFlow SDK and HTTP API
to help developers get started faster. The existing `example/sdk/`
folder only contains `dataset_example.py`. This PR fills the remaining
gaps by adding examples for three key API areas not yet covered in
`main` or by other open PRs (#13904, #13284):
- **Chunk management** — add, list, update, delete, and retrieve chunks
within a dataset
- **Chat assistant** — create a chat assistant, open a session, send
messages (streaming and non-streaming), and clean up
- **Retrieval** — perform semantic retrieval across one or multiple
datasets
### Type of change
- [x] Documentation Update
- [x] New Feature (non-breaking change which adds functionality)
### What problem does this PR solve?
Closes#14865
`download_img` in `common/misc_utils.py` is used for OAuth avatar URLs.
The previous implementation called `async_request` from
`common.http_client`, which followed redirects without re-validating
each hop and did not apply the same SSRF protections as this path needs.
That made it possible to reach non-public or disallowed targets (for
example via redirects or unsafe URLs) when fetching avatars.
This change replaces that flow with an explicit, bounded fetch: each URL
(including every redirect target) is checked with
`common.ssrf_guard.assert_url_is_safe`, DNS is pinned with
`pin_dns_global`, `httpx` streams the body with `follow_redirects=False`
and a manual redirect loop (capped by
`RAGFLOW_OAUTH_AVATAR_MAX_REDIRECTS`), and total response size is capped
(`RAGFLOW_OAUTH_AVATAR_MAX_BYTES`). Timeouts, proxy, and user agent
align with `HTTP_CLIENT_*` env vars without importing `http_client`, so
lightweight tests stay simple.
Unit tests cover empty/None URLs, loopback, cloud metadata-style
addresses, and disallowed schemes so SSRF regressions are caught early.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
---------
Co-authored-by: Kevin Hu <kevinhu.sh@gmail.com>
### What problem does this PR solve?
This PR implements ASR and TTS support for the ZhipuAI Go driver.
The ZhipuAI model config already advertises `glm-asr-2512` as an ASR
model, but the Go driver returned `zhipu, no such method` from
`TranscribeAudio`. This adds the documented audio transcription endpoint
suffix and sends multipart transcription requests with `model`,
`stream=false`, and `file` fields.
Per maintainer review, this also adds the ZhipuAI TTS endpoint suffix
and implements `AudioSpeech` / `AudioSpeechWithSender` for `glm-tts`.
Closes#15133
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
- [x] New Feature (non-breaking change which adds functionality)
### What problem does this PR solve?
Closes#15089.
Adds PPIO support to the Go model-provider layer so PPIO instances can
be routed through the Go API server with the same OpenAI-compatible
chat, streaming, model listing, and connection-check flow used by other
SaaS providers.
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
## Summary
- Added a PPIO Go model driver.
- Added the PPIO provider catalog and default OpenAI-compatible API URL.
- Registered PPIO in the model factory.
- Added focused provider and provider-manager tests.
## What changed
- Implemented chat completions, SSE streaming, ListModels, and
CheckConnection for PPIO.
- Covered request shape, stream termination, reasoning fallback, model
listing, custom base URLs, safe transport setup, unsupported methods,
and provider config loading.
- Kept the provider catalog aligned with the existing RAGFlow PPIO
factory model set.
- Cleaned up pre-existing Go model package validation blockers so the
scoped provider tests can run normally with vet enabled.
## Why
The existing Python/provider catalog path includes PPIO, but the Go
model-provider layer did not have a PPIO driver, so the Go API server
could not instantiate or use PPIO as requested in #15089.
### What problem does this PR solve?
implement rerank, asr, tts for TogetherAI
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
### What problem does this PR solve?
1. update python version to 3.13
2. upgrade ormsgpack to 1.6.0
### Type of change
- [x] Refactoring
---------
Signed-off-by: Jin Hai <haijin.chn@gmail.com>
### What problem does this PR solve?
implement ASR and TTS for Xinference
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
- [x] Refactoring
## Summary
Fixes 10 unguarded `response.choices[0]` accesses that cause
`IndexError` or `AttributeError` when the LLM returns an empty `choices`
list — the scenario described in #14711.
- `rag/llm/cv_model.py`
- `rag/llm/chat_model.py`
Each access site is now guarded with:
```python
if not response.choices:
raise ValueError("LLM returned empty response")
```
## Verification
Detected and verified by [pact](https://github.com/qizwiz/pact) — a
sheaf-cohomological LLM contract checker using Z3 as a local theory
solver.
**pact sheaf-cohomological proof status after fix:**
| File | Ȟ¹ (after) | Z3 |
|------|-----------|-----|
| `rag/llm/cv_model.py` | 0 | UNSAT ✓ |
| `rag/llm/chat_model.py` | 0 | UNSAT ✓ |
All access sites proven safe (Z3 UNSAT certificate).
The checker was also used to verify the autogen streaming-None fix in
[microsoft/autogen#7711](https://github.com/microsoft/autogen/pull/7711).
## Test plan
- [ ] Existing test suite passes
- [ ] Manually test with a provider that returns empty `choices` under
load (e.g. Vertex AI)
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Signed-off-by: Jonathan Hill <jonathan.f.hill@gmail.com>
`GET /agents/<agent_id>/sessions/<session_id>` crashed with
`AttributeError: 'NoneType' object has no attribute 'to_dict'` when the
session lookup failed: `_, conv =
API4ConversationService.get_by_id(...)` returned `(False, None)`, then
`conv.to_dict()` was called unconditionally.
This is reachable in multi-instance deployments: the session row may not
yet be visible on the node servicing the immediate follow-up GET after a
session is created on a different node.
Add the same `if not exists` guard already used by every other call site
of `API4ConversationService.get_by_id` (see agent_api.py:1147,
sdk/session.py:179, conversation_service.py:248, canvas_service.py:323).
Closes#14989
### What problem does this PR solve?
_Briefly describe what this PR aims to solve. Include background context
that will help reviewers understand the purpose of the PR._
### Type of change
- [ ] Bug Fix (non-breaking change which fixes an issue)
- [ ] New Feature (non-breaking change which adds functionality)
- [ ] Documentation Update
- [ ] Refactoring
- [ ] Performance Improvement
- [ ] Other (please describe):
Replace the RuntimeError with a warning + first-address fallback so a
single email whose From header contains multiple addresses no longer
crashes the entire IMAP sync task. Also add regression tests covering:
- #14963: RFC 5322 quoted display names with commas (e.g. "Schlüter,
Sabine" <s@x>) parsed as one address, not two.
- #14964: multi-address headers warn instead of raising.
Closes#14964
Refs #14963
## Summary
- Bump pinned nginx in `Dockerfile` from `1.29.5-1~noble` (vulnerable)
to `1.31.0-1~noble` to remediate **CVE-2026-42945**.
## Root Cause
`Dockerfile:58` pinned `ARG NGINX_VERSION=1.29.5-1~noble`. Per the
official nginx security advisory, **CVE-2026-42945** is a buffer
overflow in `ngx_http_rewrite_module` triggered via the `rewrite` and
`set` directives, affecting nginx **0.6.27 through 1.30.0**. `1.29.5`
falls inside that range, so the shipped image is vulnerable.
References:
- nginx security advisories:
https://nginx.org/en/security_advisories.html
- Vendor advisory: https://my.f5.com/manage/s/article/K000161019
- Fixed versions: `1.31.0` (mainline) and `1.30.1` (stable)
## Fix
Single-line change in `Dockerfile:58`:
```diff
-ARG NGINX_VERSION=1.29.5-1~noble
+ARG NGINX_VERSION=1.31.0-1~noble
### What problem does this PR solve?
Fixes#14997.
RAPTOR builds on the Infinity backend have been broken since v0.25.2
introduced the `extra` field in code (`rag/svr/task_executor.py:1011`)
without declaring it in `conf/infinity_mapping.json`. Every RAPTOR job
fails with:
```
infinity.common.InfinityException: (3013, 'Fail to bind the expression: extra@src/planner/expression_binder_impl.cpp:99')
```
The auto-migration in
`common/doc_store/infinity_conn_base.py:_migrate_db()` adds any columns
it finds in the mapping JSON to existing tables — so the only thing
standing between users and a working RAPTOR build is that one missing
declaration. OceanBase, ES, and OpenSearch were unaffected because they
store `extra` as a native JSON type; only Infinity (which has a strict
`varchar`/`integer`/`float` schema) needed the addition.
### The fix
Two-part change:
1. **`conf/infinity_mapping.json`**: declare `"extra": {"type":
"varchar", "default": ""}`. On next startup, `_migrate_db()` adds the
column to all existing chunk tables — no manual DDL needed for upgrading
installations.
2. **`rag/utils/infinity_conn.py` `insert()`**: serialize the `extra`
dict to a JSON string at write time, since Infinity's `varchar` can't
store a Python dict directly. Modelled on the existing `chunk_data`
handling a few lines above.
The read path (`rag/utils/raptor_utils.py:_as_extra_dict`) already
normalises both dict and JSON-string inputs, so no read-side change is
needed. Other backends are untouched — `task_executor.py` still writes
the dict, and the OceanBase/ES/OpenSearch insert paths handle dicts
natively.
### Verification
Tested on a v0.25.4 deployment with the Infinity backend by applying the
same two changes via mounted-volume override:
- Confirmed `_migrate_db()` adds the `extra` column to all pre-existing
chunk tables on startup (column visible via Infinity's
`show_columns()`).
- Triggered RAPTOR builds on four datasets (~21k chunks total) via `POST
/api/v1/datasets/<id>/index?type=raptor`.
- All four progressed past the previously-failing
`get_raptor_chunk_methods()` call into actual entity-extraction and
clustering work without the (3013) error.
- GraphRAG builds (which can trigger the same path indirectly via
`task_executor.py:857`) also progressed cleanly.
### Type of change
- [X] Bug Fix (non-breaking change which fixes an issue)
### What problem does this PR solve?
`UpstageModel.ChatStreamlyWithSender` (in the driver merged via #14819)
only extracted `delta.content` from each SSE event. For the `solar-pro3`
reasoning family (and any future Upstage model that follows the same
wire shape), the chain-of-thought is streamed in a **separate
`delta.reasoning` field**, and the driver was silently dropping all of
it.
The non-streaming path already extracts `message.reasoning` into
`ChatResponse.ReasonContent` (added earlier in this PR's history), so
the same model produced **inconsistent behavior** between streaming and
non-streaming: a tenant calling `solar-pro3` with `reasoning_effort:
high` would see the reasoning trace if they used `ChatWithMessages` but
not if they used `ChatStreamlyWithSender`.
### Live evidence
Probed against `api.upstage.ai/v1/chat/completions` with `solar-pro3` +
`reasoning_effort: high` + `stream: true` (8000-token budget so the
reasoning has room to finish):
```
$ curl -sN -H "Authorization: Bearer <key>" -H "Content-Type: application/json" \
-X POST https://api.upstage.ai/v1/chat/completions \
-d '{"model":"solar-pro3","messages":[{"role":"user","content":"Compute 15% of 80."}],
"max_tokens":8000,"stream":true,"reasoning_effort":"high"}'
# across 168 SSE events:
# delta keys seen: [content reasoning role]
# delta.content total len: 121 chars (the visible answer)
# delta.reasoning total len: 159 chars (the chain-of-thought) <- driver dropped this
```
A representative event showing both fields side by side:
```json
data: {"choices":[{"index":0,"delta":{"reasoning":"15% = 0.15."}}]}
data: {"choices":[{"index":0,"delta":{"content":"15% of 80 is "}}]}
```
The 159 chars of reasoning were arriving on the wire and being thrown
away. `solar-pro2` was also probed (625 events); it does **not** emit
`delta.reasoning` — its reasoning is inlined into `delta.content` — so
this change is a no-op for it and for `solar-mini`.
### What this PR includes
- `internal/entity/models/upstage.go`: in the SSE scanner loop, extract
`delta.reasoning` before `delta.content` and forward each non-empty
chunk via the sender's second arg (the existing `reasonContent` channel
the non-stream path already populates).
The ordering contract is documented inline: reasoning chunks within a
single SSE event are emitted before content chunks, so a UI that pipes
both sees the chain-of-thought start before the answer for that token,
matching the wire order Upstage emits.
- `internal/entity/models/upstage_test.go`: three new tests pinning the
new behavior:
- `TestUpstageStreamExtractsReasoningDelta` — reasoning + content
forwarded to the right sender args; one-of invariant per call
- `TestUpstageStreamReasoningChunksArriveBeforeContent` — ordering
pinned within a single SSE event that carries both fields
- `TestUpstageStreamWithoutReasoningStillWorks` — regression net:
non-reasoning models (`solar-mini`, `solar-pro2`) continue to work; the
reason callback never fires
No interface change. No factory change. No config change.
### How was this tested?
```
$ go test -vet=off -run TestUpstage -count=1 -v ./internal/entity/models/...
... (existing tests 1..9 still pass) ...
=== RUN TestUpstageStreamExtractsReasoningDelta
--- PASS: TestUpstageStreamExtractsReasoningDelta (0.01s)
=== RUN TestUpstageStreamReasoningChunksArriveBeforeContent
--- PASS: TestUpstageStreamReasoningChunksArriveBeforeContent (0.01s)
=== RUN TestUpstageStreamWithoutReasoningStillWorks
--- PASS: TestUpstageStreamWithoutReasoningStillWorks (0.00s)
PASS
ok ragflow/internal/entity/models 0.034s
```
12/12 Upstage tests pass on go 1.25. `go build
./internal/entity/models/...` exits 0.
**Live integration test** (smoke test not committed) — the patched
driver was run directly against `api.upstage.ai/v1` with the same prompt
that produced the curl evidence above:
```
=== RUN TestUpstageStreamReasoningLiveSmoke
[OK] visible content: 50 chunks, 84 chars
[OK] reasoning: 39 chunks, 90 chars
content head 200: "\\(15\\% = \\frac{15}{100}=0.15\\).\n\n\\[\n0.15 \\times 80 = 12.\n\\]\n\n**15 % of 80 is 12.**"
reasoning head 200: "We need to compute 15% of 80. That's 0.15 * 80 = 12. So answer is 12. Provide explanation."
UPSTAGE STREAM REASONING SMOKE PASSED
--- PASS: TestUpstageStreamReasoningLiveSmoke (1.97s)
```
Before this fix, the same call would have produced **0 reasoning
chunks**. The 90 chars of reasoning that the patched driver now surfaces
are the chain-of-thought solar-pro3 emits when reasoning_effort is high.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
### What problem does this PR solve?
`MistralModel.ChatWithMessages` (in the driver merged via #14807)
assumes that `choices[0].message.content` from `/v1/chat/completions` is
always a string and falls through to `return nil, fmt.Errorf("invalid
content format")` on anything else.
That assumption breaks for the **magistral reasoning family**
(`magistral-small-*`, `magistral-medium-*`). When the model needs a
chain-of-thought to answer, Mistral returns `content` as a **structured
array of typed parts**:
```json
"content": [
{"type": "thinking",
"thinking": [{"type": "text", "text": "Combined speed is 150 mph. 300 / 150 = 2 hours."}],
"closed": true},
{"type": "text", "text": "They will meet after **2 hours**."}
]
```
Concretely, this is what the live API returns today (probed against
`api.mistral.ai/v1`):
```
$ curl -H "Authorization: Bearer <key>" -H "Content-Type: application/json" \
-X POST https://api.mistral.ai/v1/chat/completions \
-d '{"model":"magistral-medium-latest",
"messages":[{"role":"user","content":"two trains 60mph and 90mph, 300mi apart, when do they meet? step by step."}],
"max_tokens":1024}'
HTTP 200
{ "choices":[{"message":{
"role":"assistant",
"content":[
{"type":"thinking","thinking":[{"type":"text","text":"Okay, let's see..."}],"closed":true},
{"type":"text","text":"To determine when the two trains meet..."}
]}}] }
```
With the current driver, every call like that returns the generic
`"invalid content format"` error. Trivial prompts that happen to fit in
a string answer still succeed, so the breakage is **non-deterministic
from the tenant's POV**: same model, same provider, sometimes works,
sometimes 500s with no useful error.
A secondary issue: `conf/models/mistral.json` does not include any
magistral model. The picker hid the broken path, which is why this
wasn't caught during #14807's review.
### What this PR includes
- New helper `extractMistralContent(raw interface{}) (answer,
reasonContent string, err error)` in
`internal/entity/models/mistral.go`, which normalizes both shapes
Mistral can return:
- `string` → historical path. `Answer = content`, `ReasonContent = ""`.
Preserves behavior for every non-reasoning model (`mistral-large-*`,
`mistral-small-*`, `ministral-*`, `codestral-*`, `pixtral-*`,
`open-mistral-nemo`).
- `[]interface{}` → walk the parts. Concatenate every `{"type":"text",
"text":...}` part into `Answer`; concatenate the inner text inside every
`{"type":"thinking", "thinking":[...]}` part into `ReasonContent`.
- `ChatWithMessages` now calls the helper instead of doing the raw
`.(string)` cast.
- Unknown part types are **skipped, not failed**. Mistral has been
adding new content variants quickly (audio chunks, citations, etc.);
this driver should not 500 every call when a new part type appears.
- `conf/models/mistral.json`: add `magistral-medium-latest` and
`magistral-small-latest`. Both are visible in `/v1/models` today.
No interface change. No factory change. No new dependencies.
### How was this tested?
**Unit tests** — 5 new tests in `internal/entity/models/mistral_test.go`
on top of the 27 already shipped via #14807:
- `TestMistralChatHandlesStringContent` — regression net for the
historical path
- `TestMistralChatExtractsReasoningFromStructuredContent` — the fixture
body is a trimmed copy of the actual `magistral-medium-latest` response
captured above; asserts both `Answer` and `ReasonContent` are populated
correctly
- `TestMistralChatHandlesStructuredContentWithoutThinking` —
`magistral-*` with a trivial answer returns a structured shape that has
only a `text` part; `ReasonContent` must stay empty
- `TestMistralChatIgnoresUnknownContentPartTypes` — `audio_url` and
`future_part_type` parts are skipped, `text` parts still flow through
- `TestExtractMistralContent` — table-driven unit coverage of the helper
for string, empty string, nil, empty array, text-only, thinking+text,
unsupported root type
```
$ go test -vet=off -run "TestMistral|TestExtractMistralContent" -count=1 -v ./internal/entity/models/...
=== RUN TestMistralChatHandlesStringContent
--- PASS: TestMistralChatHandlesStringContent (0.00s)
=== RUN TestMistralChatExtractsReasoningFromStructuredContent
--- PASS: TestMistralChatExtractsReasoningFromStructuredContent (0.00s)
=== RUN TestMistralChatHandlesStructuredContentWithoutThinking
--- PASS: TestMistralChatHandlesStructuredContentWithoutThinking (0.00s)
=== RUN TestMistralChatIgnoresUnknownContentPartTypes
--- PASS: TestMistralChatIgnoresUnknownContentPartTypes (0.00s)
=== RUN TestExtractMistralContent
=== RUN TestExtractMistralContent/plain_string
=== RUN TestExtractMistralContent/empty_string
=== RUN TestExtractMistralContent/nil
=== RUN TestExtractMistralContent/empty_array
=== RUN TestExtractMistralContent/text_only
=== RUN TestExtractMistralContent/thinking_then_text
=== RUN TestExtractMistralContent/unknown_root_type
--- PASS: TestExtractMistralContent (0.00s)
PASS
ok ragflow/internal/entity/models 0.046s
```
All 32 Mistral tests pass on go 1.25. `go build
./internal/entity/models/...` exits 0.
**Live integration test** — driver exercised against `api.mistral.ai/v1`
with the patched code:
```
=== RUN TestMistralMagistralSmoke
[OK] "magistral-small-latest" present upstream
[OK] "magistral-medium-latest" present upstream
[OK trivial] Answer="7" ReasonContent=""
[OK reasoning] Answer len=797 head="To determine when the two trains meet, we can follow these steps:\n\n1. **Identify..."
ReasonContent len=1069 head="Okay, let's see. There are two trains, one going 60 mph and the other going 90 mph. They're moving towards each other, s..."
MAGISTRAL SMOKE PASSED
--- PASS: TestMistralMagistralSmoke (18.09s)
PASS
ok ragflow/internal/entity/models 18.112s
```
What the live run proves on the wire:
- `magistral-small-latest` with a trivial prompt still uses the
string-content shape; the regression-net path is exercised against the
real server, not just the mock.
- `magistral-medium-latest` with a reasoning prompt uses the
structured-array shape; the new code path extracts a 1069-character
reasoning trace into `ChatResponse.ReasonContent` and a 797-character
visible answer into `ChatResponse.Answer`. Before this fix, the same
call returned `"invalid content format"` and the caller saw nothing.
The smoke-test file itself is not committed (live tests live outside the
PR diff, same convention used for prior provider PRs).
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
## Problem
The Go server build pipeline (`build.sh` + CMake + CGO bindings) was
tested on Ubuntu only. On macOS arm64 with Homebrew it fails in five
orthogonal places. None of these require platform-specific code paths —
the same source builds on both Linux and Darwin after these fixes.
## Reproduction (before)
```
$ uname -a
Darwin … 25.4.0 arm64
$ brew install cmake pcre2 simde
$ bash build.sh
…
error: 'simde/x86/sse4.1.h' file not found
error: implicit instantiation of undefined template 'std::basic_istringstream<char>'
error: no matching function for call to 'Join'
…
clang: error: no such file or directory: '/usr/local/lib/libpcre2-8.a'
```
## Fix (5 small, orthogonal changes)
### 1. `internal/cpp/CMakeLists.txt` — find Homebrew + libpcre2-8
portably
- Detect Apple platforms via `if(APPLE)`, call `brew --prefix` once, add
`${HOMEBREW_PREFIX}/include` and `${HOMEBREW_PREFIX}/lib`. No effect on
Linux.
- Replace the literal `libpcre2-8.a` link token (which only the Linux
linker finds in `/usr/local/lib` by default) with
`find_library(PCRE2_LIB NAMES pcre2-8 REQUIRED)`. Works on
`/usr/lib/x86_64-linux-gnu` (Debian/Ubuntu), `/usr/local/lib` (Intel Mac
& legacy Linux), `/opt/homebrew/lib` (Apple Silicon).
### 2. `internal/cpp/wordnet_lemmatizer.cpp` +
`internal/cpp/rag_analyzer.cpp` — explicit `#include <sstream>`
libstdc++ (Linux) pulls `<sstream>` in transitively via `<fstream>`;
libc++ (Apple Clang) doesn't, so the existing `std::istringstream` /
`std::ostringstream` uses fail to compile on macOS. One-line include in
each file.
### 3. `internal/cpp/rag_analyzer.cpp` — `Join` template overload fix
`Join(tokens, start, tokens.size(), delim)` at line 146 passes `size_t`
to an `int` parameter. C++23 strict mode in Apple Clang refuses the
implicit narrowing and reports the 4-arg overload as a substitution
failure, leaving the call ambiguous between the 3-arg and 4-arg
templates. Fix: explicit `static_cast<int>(tokens.size())`. Behaviour
identical on libstdc++ — the narrowing was always intentional.
### 4. `internal/binding/rag_analyzer.go` — split darwin CGO LDFLAGS
The existing `#cgo darwin LDFLAGS: ... /usr/local/lib/libpcre2-8.a` only
matches Intel Macs. Apple Silicon Homebrew installs to `/opt/homebrew`.
Split into `darwin,arm64` and `darwin,amd64` build constraints with the
right absolute path on each.
### 5. `build.sh` — accept Homebrew path in the pcre2 sanity check
The sanity check looked at two Linux paths only and then fell through to
`sudo apt -y install libpcre2-dev` on failure. Added
`/opt/homebrew/lib/libpcre2-8.a`, and on Darwin failure now exits
cleanly with the right `brew install pcre2` hint instead of trying
`apt`.
## Verified
- `bash build.sh` now completes on macOS arm64 (Apple Silicon, brew 4.x,
cmake 4.x, Apple Clang 17, Go 1.25, pcre2 10.x, simde 0.8.x).
- Produced binaries: `bin/server_main`, `bin/admin_server`,
`bin/ragflow_cli`.
- `bin/server_main` boots, connects MySQL, runs migrations, loads the 64
model provider configs cleanly.
- Still builds on Linux — the CMake additions are inside an `if(APPLE)`
guard, the `find_library` call matches Linux paths too, the build.sh
check still tries `apt` when not on Darwin.
## Out of scope
The Go server itself currently fails at runtime when not pointing at
Elasticsearch (`Failed to initialize doc engine: failed to ping
Elasticsearch`), but that's the placeholder Infinity engine documented
in `internal/engine/README.md` — unrelated to this build patchset.
---
Happy to split this into smaller PRs if you'd prefer (one per file). The
five changes are independent.
## What
- Add Perplexity as a chat and embedding provider backed by its
OpenAI-compatible `/chat/completions` and `/v1/embeddings` APIs
- Register Perplexity in the Go model factory and provider config
- Support non-streaming chat, SSE streaming chat, embeddings, model
listing, and connection checks
Refs #14736
---------
Co-authored-by: Jin Hai <haijin.chn@gmail.com>
- Adds an `Astraflow` Go driver so the new API server can route
Astraflow (UCloud ModelVerse) chat instances, matching the existing
Python `AstraflowChat` (`rag/llm/chat_model.py:1237`). Follows the same
SaaS-driver shape used for Avian, Novita, TogetherAI, Replicate,
DeepInfra, Upstage, and LongCat.
Closes#15062
---------
Co-authored-by: Jin Hai <haijin.chn@gmail.com>
Closes#15044.
Avian was listed unchecked in the Go-rewrite tracker #14736 and already
had an llm_factories.json entry with 4 preconfigured chat models
(deepseek-v3.2, kimi-k2.5, glm-5, minimax-m2.5), but the Go API server
had no driver to route them. The Python side has supported Avian at
rag/llm/chat_model.py:1220 (AvianChat) via the LiteLLM openai/ provider
with default base https://api.avian.io/v1.
Co-authored-by: Jin Hai <haijin.chn@gmail.com>
### What problem does this PR solve?
`ReplicateModel.Embed` in `internal/entity/models/replicate.go` was a
`"replicate, no such method"` stub. Tracking issue #14736 lists
Replicate's embedding surface as not implemented. This PR wires it up
against Replicate's documented embedding schema.
Until this PR, a tenant who selected a Replicate embedding model got the
sentinel error on every embed call.
Co-authored-by: sxxtony <sxxtony@users.noreply.github.com>
Co-authored-by: Jin Hai <haijin.chn@gmail.com>
### What problem does this PR solve?
This PR adds a new `Browser` operator to Agent workflows, enabling
prompt-driven browser automation in RAGFlow.Technically based
‘Browser-Use’
It includes:
- Backend browser component execution with tenant LLM integration
- Upload source support (file IDs, URLs, variables, CSV/JSON array)
- Downloaded file persistence to RAGFlow storage
- Frontend node/operator integration, form config, icon, and i18n
updates
- Unit tests for upload/download and ID parsing logic
- Dependency and Docker updates for browser-use runtime support
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
## Summary
- Adds a lightweight `@tool` decorator and `FunctionToolSession` adapter
in `rag/llm/tool_decorator.py` that let callers register plain Python
functions as LLM tools without hand-writing OpenAI function schemas or
building an MCP-style session.
- Refactors `Base.bind_tools` and `LiteLLMBase.bind_tools` in
`rag/llm/chat_model.py` to accept either the new decorator form
`bind_tools(tools=[fn1, fn2])` or the existing `(toolcall_session,
tools_schemas)` form, so existing agent/dialog call-sites in
`agent/component/agent_with_tools.py`, `api/db/services/llm_service.py`,
and `api/db/services/dialog_service.py` are unaffected.
- Adds 8 unit tests in `test/unit_test/rag/llm/test_tool_decorator.py`
covering schema shape, required/optional inference, sync + async
dispatch, and bad-input rejection.
## Usage
```python
from rag.llm.tool_decorator import tool
@tool
def get_weather(city: str) -> str:
"""Get current weather for a city.
:param city: City name to look up.
"""
return f"{city}: 21 C, partly cloudy"
chat_mdl.bind_tools(tools=[get_weather])
ans, tk = await chat_mdl.async_chat_with_tools(system, history)
```
The decorator introspects `inspect.signature` + type hints + the
docstring (`:param name:` style) and attaches an OpenAI-format
`openai_schema` to the callable. `FunctionToolSession` duck-types the
existing `ToolCallSession` protocol, dispatching async callables
directly and sync ones through `thread_pool_exec` so the event loop is
never blocked.
## Design notes
- `tool_decorator.py` deliberately does **not** live inside
`rag/llm/__init__.py` to avoid forcing every consumer through the heavy
provider auto-discovery loop and to sidestep a circular import
(`__init__.py` imports `chat_model`, which would otherwise need symbols
from `__init__.py`).
- `FunctionToolSession` is duck-typed against
`common.mcp_tool_call_conn.ToolCallSession` rather than explicitly
inheriting from it, so importing the decorator doesn't pull the MCP
client SDK into the import graph.
- Docstring parsing is intentionally minimal (`:param name:` only) to
keep this dependency-free; Google/NumPy styles can be added later via
`docstring_parser` if needed.
## Test plan
- [x] `python -m pytest test/unit_test/rag/llm/test_tool_decorator.py
-v` — 8 passed
- [x] `python -m pytest test/unit_test/rag/llm/
--ignore=test/unit_test/rag/llm/test_perplexity_embed.py` — 11 passed
(the ignored test has a pre-existing `numpy` import that's unrelated)
- [ ] Reviewer: smoke-test the new path end-to-end with a live model via
`chat_mdl.bind_tools(tools=[my_fn])` to confirm the OpenAI-format
schemas pass through unchanged
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
### What problem does this PR solve?
Closes#15048.
Several SDK session routes in `api/apps/sdk/session.py` called
`.split()` directly on `request.headers.get("Authorization")`. When
clients omitted the header, the handlers raised `AttributeError` before
returning the existing `Authorization is not valid!` response.
This PR centralizes SDK Authorization parsing in a small helper and
keeps the existing error response for missing, empty, or malformed
headers.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
### Tests
- `ZHIPU_AI_API_KEY=dummy uv run --python 3.13 --group test pytest
test/testcases/test_http_api/test_session_management/test_session_sdk_routes_unit.py::test_sdk_session_routes_missing_authorization_unit
-q`
- `uv run --python 3.13 --group test ruff check api/apps/sdk/session.py
test/testcases/test_http_api/test_session_management/test_session_sdk_routes_unit.py`
- `python3 -m py_compile api/apps/sdk/session.py
test/testcases/test_http_api/test_session_management/test_session_sdk_routes_unit.py`
- `git diff --check`
### What problem does this PR solve?
Remove duplicate function definitions in
`api/db/services/dialog_service.py`.
**Problem:** Two helper functions were defined twice in the same file,
but with different parameter orders:
- First definition (line 57):
`_resolve_reference_metadata(request_payload=None, config=None)`
- Second definition (line 136): `_resolve_reference_metadata(config,
request_payload=None)`
**Solution:** Keep the second definition (which is actually used by
other modules) and remove the first one to avoid confusion.
Additionally, remove duplicate `_enrich_chunks_with_document_metadata`
definition (keep line 140 version).
<img width="1584" height="313" alt="image"
src="https://github.com/user-attachments/assets/7daee832-244f-4bb2-8488-e3b65012a3f9"
/>
<img width="1672" height="359" alt="image"
src="https://github.com/user-attachments/assets/4fd2f523-273c-4b20-a7c9-ab35740b7834"
/>
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
- [ ] New Feature (non-breaking change which adds functionality)
- [ ] Documentation Update
- [x] Refactoring
- [ ] Performance Improvement
- [ ] Other (please describe):
## Summary
- Align **GET `/api/v1/documents/<doc_id>/download`** with
**`/preview`**: resolve extension and MIME type from the stored document
name when the **`ext` query parameter is omitted**, instead of
defaulting to `markdown`.
- When **`?ext=`** is present, behavior stays the same as before
(explicit extension / `Content-Type` mapping).
- Enforce the same access + document lookup pattern as preview
(**`accessible`** + **`get_by_id`**).
- Extend unit tests for the no-`ext` PDF filename case.
## Test plan
- [x] `uv run pytest
test/testcases/test_web_api/test_document_app/test_document_metadata.py::TestDocumentMetadataUnit::test_download_attachment_success_and_exception_unit`
- [x] Optional: `curl -sSI` against
`/api/v1/documents/<pdf_doc_id>/download` without `ext` and confirm
`Content-Type: application/pdf`
Fixes#15052.
POST /api/v1/dify/retrieval resolved the caller via @apikey_required
(injecting tenant_id) but then fetched the requested knowledge_id with
no tenant filter and ran the full retrieval pipeline against
kb.tenant_id (the owner). Any valid Dify-compatible API key could
retrieve chunks from any tenant whose KB UUID was known. Adds the
missing ownership check.
## Root Cause
api/apps/sdk/dify_retrieval.py line 253:
KnowledgebaseService.get_by_id(kb_id) fetched the KB by id alone, then
the handler used kb.tenant_id (the OWNER) to build the embedding model
and call the retriever. The caller tenant_id was only used downstream at
line 278 for retrieval_by_children, well after cross-tenant data was
already retrieved.
grep confirmed there was no KnowledgebaseService.accessible call
anywhere in the handler.
## Fix
Two-line guard immediately after the existing get_by_id lookup,
mirroring the pattern PR #14749 lands for the sibling sdk/doc.py routes
(download, parse, stop_parsing, retrieval_test):
e, kb = KnowledgebaseService.get_by_id(kb_id)
if not e:
return build_error_result(message="Knowledgebase not found!",
code=RetCode.NOT_FOUND)
+ if not KnowledgebaseService.accessible(kb_id, tenant_id):
+ return build_error_result(message="No authorization.",
code=RetCode.AUTHENTICATION_ERROR)
if kb.tenant_embd_id:
...
KnowledgebaseService.accessible already handles solo-tenant ownership,
team membership via TenantService.get_joined_tenants_by_user_id, and the
permission=ME distinction. No behavior change for legitimate callers;
cross-tenant callers now receive RetCode.AUTHENTICATION_ERROR (109).
## Test Plan
- [x] Regression test added:
test/unit_test/api/apps/sdk/test_dify_retrieval.py
- test_cross_tenant_request_is_rejected -- attacker tenant calling owner
tenant KB gets 109; retriever is not invoked
- test_same_tenant_request_succeeds -- owner tenant gets the records
back
- test_missing_knowledge_base_returns_not_found -- missing KB returns
404 BEFORE the access check fires (legit callers see the clearer
message)
- [x] All 3 tests pass after the fix
- [x] Cross-tenant test FAILS on pre-fix main (KeyError on result[code]
because handler leaks records dict instead of returning auth error)
- [x] ruff check clean on both changed files
- [x] No drive-by reformatting in dify_retrieval.py -- only the 2 added
lines
### Post-fix output
test_cross_tenant_request_is_rejected PASSED [ 33%]
test_same_tenant_request_succeeds PASSED [ 66%]
test_missing_knowledge_base_returns_not_found PASSED [100%]
============================== 3 passed in 0.04s
===============================
Closes#15027
### What problem does this PR solve?
Closes#15076
Two endpoints in `api/apps/restful_apis/chat_api.py` accepted a
`user_id` field from the request body and used it directly when creating
a session:
```python
# before (vulnerable)
"user_id": req.get("user_id", current_user.id) # create_session
conv = await _create_session_for_completion(chat_id, dia, req.get("user_id", current_user.id)) # session_completion
```
Any authenticated caller could supply an arbitrary `user_id` and have
the new session attributed to a different user — effectively spoofing
session ownership. Both call sites are now fixed to always use
`current_user.id`, which is set by the authentication middleware and
cannot be tampered with via the request payload.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
### Changes
| File | Change |
|------|--------|
| `api/apps/restful_apis/chat_api.py` | Remove `req.get("user_id", ...)`
fallback in `create_session` and `session_completion`; always use
`current_user.id` |
|
`test/testcases/test_http_api/test_session_management/test_session_sdk_routes_unit.py`
| Add `test_create_session_user_id_not_spoofable` and
`test_session_completion_user_id_not_spoofable` (both `@pytest.mark.p2`)
|
### Testing
Two new unit tests assert that a `user_id` value supplied in the request
body is silently ignored and the session is always owned by the
authenticated user:
```
test_create_session_user_id_not_spoofable
test_session_completion_user_id_not_spoofable
```
Run with:
```bash
uv run pytest test/testcases/test_http_api/test_session_management/test_session_sdk_routes_unit.py -k "spoofable" -v
```
## What problem does this PR solve?
Closes#15021.
The Go model-provider layer had no support for **Azure OpenAI**. Azure
OpenAI is *not* a drop-in base-URL swap of the OpenAI driver — it
differs in authentication, endpoint structure, and how models are listed
— so it needs its own `ModelDriver` implementation.
## Type of change
- [x] New feature (non-breaking change which adds functionality)
Co-authored-by: Jin Hai <haijin.chn@gmail.com>
### What problem does this PR solve?
Fixes#15023
GPUStack is listed as unchecked in the Go-rewrite tracker #14736, and
`internal/service/llm.go:171` already classifies it as a self-deployed
provider alongside Ollama, Xinference, LocalAI, and LM Studio — but
`internal/entity/models/` had no `gpustack.go` driver, so the new Go API
server could not route GPUStack instances. This PR adds the chat surface
for GPUStack so it lines up with the existing self-hosted Go drivers.
Co-authored-by: Jin Hai <haijin.chn@gmail.com>
## Summary
- Replaces the `"no such method"` stub on `XinferenceModel.Embed`
(`internal/entity/models/xinference.go`) with a real implementation
against Xinference's OpenAI-compatible `/v1/embeddings` endpoint.
- Adds the `"embedding": "v1/embeddings"` URL suffix to
`conf/models/xinference.json`.
- Mirrors the Python `XinferenceEmbed` class in
`rag/llm/embedding_model.py:407` for payload shape (OpenAI-compatible
`model + input` → `data[*].index + data[*].embedding`) and tolerates the
same no-auth default Xinference deployments use. Authorization is only
sent when a non-empty API key is configured, via the existing
`setXinferenceAuth` helper.
- Reuses the existing `normalizeXinferenceBaseURL` + `baseURLForRegion`
helpers so both `http://127.0.0.1:9997` and `http://127.0.0.1:9997/v1`
resolve to the same `/v1/embeddings` target without doubled `/v1`.
- Validates response indices — duplicate, missing, or out-of-range
`data[*].index` values fail with a clear error rather than silently
producing misaligned vectors.
- Returns `[]EmbeddingData` in original input order (placed by `Index`)
so downstream callers can index positionally without re-sorting.
- Forwards `EmbeddingConfig.Dimension` as `dimensions` when `> 0`,
matching the OpenAI cluster pattern.
Closes#14810
Co-authored-by: Jin Hai <haijin.chn@gmail.com>
### What problem does this PR solve?
Fixes#15012
The Novita Go driver landed in #14850 and shipped a stub `Rerank` method
that returned `"novita, no such method"`, so Novita could not be used as
a rerank provider in RAGFlow. This PR fills that gap, in the same way
#14895 filled the Embed gap on the same driver.
Novita exposes a public rerank endpoint at `POST
https://api.novita.ai/openai/v1/rerank` that accepts the
Cohere-compatible request shape (`{model, query, documents, top_n}`)
with `Authorization: Bearer <api_key>`. `baai/bge-reranker-v2-m3` is
documented in Novita's model library with a 1024-token limit.
### What problem does this PR solve?
Fixes#14816
The Xinference Go driver landed chat in #14938 and Embed is in review in
#14932, but `Rerank` shipped as a stub that returns `"xinference, no
such method"`. Tenants who launch a rerank model with `--model-type
rerank` on their Xinference instance cannot route it through the Go API
server. This PR fills the gap.
Xinference exposes an OpenAI-compatible REST API. The rerank endpoint is
at `POST <base>/v1/rerank` and accepts the Cohere-shaped body `{model,
query, documents, top_n}`, returning `{results: [{index,
relevance_score}]}` — the same wire shape used by the merged NVIDIA
(#14778), Aliyun (#14676), Gitee (#14656), ZhipuAI (#14608), Novita
(#15014), and LocalAI (#14813) Rerank implementations. Documented in
[Xinference rerank
docs](https://inference.readthedocs.io/en/v1.6.1/models/model_abilities/rerank.html);
the [builtin rerank model
catalog](https://inference.readthedocs.io/en/stable/models/builtin/rerank/)
lists `bge-reranker-base`, `bge-reranker-large`, `bge-reranker-v2-m3`,
and others.
### What problem does this PR solve?
Add a Go driver for **n1n.ai** (https://docs.n1n.ai), one of the
unchecked providers on the umbrella tracking issue #14736. n1n.ai is an
OpenAI-compatible aggregator hosting a 450+ model catalog (GPT, Claude,
Gemini, DeepSeek, Kimi, Qwen, embedding + reranker families) under
`https://api.n1n.ai/v1`.
Until this PR, a tenant who configured `n1n` as a model provider in the
Go layer fell through to the default branch of
`internal/entity/models/factory.go` and got the dummy driver.
---------
Co-authored-by: sxxtony <sxxtony@users.noreply.github.com>
### What problem does this PR solve?
Fixes#15015
The TogetherAI Go driver in `internal/entity/models/togetherai.go`
shipped a stub `Embed` method that returned `"TogetherAI, no such
method"`, so TogetherAI could not be used as an embedding provider in
RAGFlow. This PR fills that gap.
TogetherAI exposes a public OpenAI-compatible embeddings endpoint at
`POST https://api.together.ai/v1/embeddings` that accepts the standard
`{model, input}` shape with `Authorization: Bearer <api_key>` (confirmed
in TogetherAI's official docs:
https://docs.together.ai/docs/embeddings-overview). Documented embedding
models include `intfloat/multilingual-e5-large-instruct`,
`BAAI/bge-large-en-v1.5`, and `BAAI/bge-base-en-v1.5`.
### Changes
- `internal/entity/models/togetherai.go`: implement
`TogetherAIModel.Embed`.
- Validate inputs (api key, model name) and short-circuit on empty
texts.
- Resolve region with the existing `baseURLForRegion` helper.
- Build URL from `URLSuffix.Embedding`.
- Send `{model, input}` POST body, add `dimensions` when
`embeddingConfig.Dimension > 0` (matches the pattern in #14735).
- Bearer auth + JSON content type, mirroring the chat path.
- Parse `{data: [{embedding, index}]}` and reorder by `index`, rejecting
out-of-range indices, duplicates, and missing entries so the output
always lines up with the input. Same shape as the merged Mistral,
Upstage, and Novita Embed implementations.
- `conf/models/togetherai.json`:
- Add `"embedding": "embeddings"` to `url_suffix`.
- Add default embedding model entries for
`intfloat/multilingual-e5-large-instruct`, `BAAI/bge-large-en-v1.5`, and
`BAAI/bge-base-en-v1.5`.
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
### What problem does this PR solve?
Fix: The logs on the data source details page are not fully displayed.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
description: Go naming conventions and best practices. Use this skill when working with Go code and need to name packages, files, directories, structs, interfaces, functions, variables, or constants. Provides comprehensive naming guidelines following Go community standards.
---
Strictly follow the naming conventions in [rules/named.md](rules/named.md)
Strictly follow the naming conventions in [rules/named.md](../../rules/named.md)
# Persist computed ports into .env so docker-compose uses the correct host bindings.
# Remove previous CI overrides first; docker compose uses the last duplicate key.
sed -i '/^ES_PORT=/d;/^OS_PORT=/d;/^INFINITY_THRIFT_PORT=/d;/^INFINITY_HTTP_PORT=/d;/^INFINITY_PSQL_PORT=/d;/^EXPOSE_MYSQL_PORT=/d;/^MINIO_PORT=/d;/^MINIO_CONSOLE_PORT=/d;/^REDIS_PORT=/d;/^TEI_PORT=/d;/^KIBANA_PORT=/d;/^SVR_HTTP_PORT=/d;/^ADMIN_SVR_HTTP_PORT=/d;/^SVR_MCP_PORT=/d;/^GO_HTTP_PORT=/d;/^GO_ADMIN_PORT=/d;/^SANDBOX_EXECUTOR_MANAGER_PORT=/d;/^SVR_WEB_HTTP_PORT=/d;/^SVR_WEB_HTTPS_PORT=/d;/^NATS_PORT=/d;/^COMPOSE_PROFILES=/d;/^TEI_MODEL=/d;/^RAGFLOW_IMAGE=/d;/^DOC_ENGINE=/d' .env
if:always() # always run this step even if previous steps failed
run:|
# Sometimes `docker compose down` fail due to hang container, heavy load etc. Need to remove such containers to release resources(for example, listen ports).
# Persist computed ports into .env so docker-compose uses the correct host bindings.
# Remove previous CI overrides first; docker compose uses the last duplicate key.
sed -i '/^ES_PORT=/d;/^OS_PORT=/d;/^INFINITY_THRIFT_PORT=/d;/^INFINITY_HTTP_PORT=/d;/^INFINITY_PSQL_PORT=/d;/^EXPOSE_MYSQL_PORT=/d;/^MINIO_PORT=/d;/^MINIO_CONSOLE_PORT=/d;/^REDIS_PORT=/d;/^TEI_PORT=/d;/^KIBANA_PORT=/d;/^SVR_HTTP_PORT=/d;/^ADMIN_SVR_HTTP_PORT=/d;/^SVR_MCP_PORT=/d;/^GO_HTTP_PORT=/d;/^GO_ADMIN_PORT=/d;/^SANDBOX_EXECUTOR_MANAGER_PORT=/d;/^SVR_WEB_HTTP_PORT=/d;/^SVR_WEB_HTTPS_PORT=/d;/^NATS_PORT=/d;/^COMPOSE_PROFILES=/d;/^TEI_MODEL=/d;/^RAGFLOW_IMAGE=/d;/^DOC_ENGINE=/d' .env
- name:Generate server coverage report Elasticsearch
if:${{ !cancelled() }}
@@ -569,7 +997,7 @@ jobs:
else
echo ".coverage file not found!"
fi
- name:Collect ragflow log Elasticsearch
if:${{ !cancelled() }}
run:|
@@ -585,8 +1013,11 @@ jobs:
if:always() # always run this step even if previous steps failed
run:|
# Sometimes `docker compose down` fail due to hang container, heavy load etc. Need to remove such containers to release resources(for example, listen ports).
This file provides context, build instructions, and coding standards for the RAGFlow project.
It is structured to follow GitHub Copilot's [customization guidelines](https://docs.github.com/en/copilot/concepts/prompting/response-customization).
Use this file as the local operating guide for the current codebase. Prefer the code and the current CLAUDE.md over any older convention or remembered project shape.
## 1. Project Overview
RAGFlow is an open-source RAG (Retrieval-Augmented Generation) engine based on deep document understanding. It is a full-stack application with a Python backend and a React/TypeScript frontend.
## Core stance
- Treat legacy code as liability, not as a compatibility target.
- Prefer deletion over shims, deprecated branches, wrapper APIs, and dual-track migration notes.
- If old and new implementations coexist, converge to one path unless an external contract forces compatibility.
- Remove dead tests, commented-out code, stale docs, and "move later" notes instead of preserving them.
- Reduce public surface area when a helper can be made private or internal.
- Keep refactors centered on the owning abstraction, not on adjacent compatibility layers.
- **Backend**: Python 3.10+ (Flask/Quart)
-**Frontend**: TypeScript, React, UmiJS
-**Architecture**: Microservices based on Docker.
-`api/`: Backend API server.
-`rag/`: Core RAG logic (indexing, retrieval).
-`deepdoc/`: Document parsing and OCR.
-`web/`: Frontend application.
## Current stack
-Backend: Python 3.13+, Quart-based API server, Peewee ORM, async workers.
-Frontend: React + TypeScript + Vite in `web/`.
-Go: the repository also has a substantial Go module for servers, ingestion, parser/runtime, CLI, and supporting services.
-Runtime services commonly include MySQL/PostgreSQL, Redis, MinIO, and Elasticsearch/Infinity/OpenSearch depending on configuration.
## 2. Directory Structure
-`api/`: Backend API server (Flask/Quart).
-`apps/`: API Blueprints (Knowledge Base, Chat, etc.).
-`db/`: Database models and services.
-`rag/`: Core RAG logic.
-`llm/`: LLM, Embedding, and Rerank model abstractions.
-`deepdoc/`: Document parsing and OCR modules.
-`agent/`: Agentic reasoning components.
-`web/`: Frontend application (React + UmiJS).
-`docker/`: Docker deployment configurations.
-`sdk/`: Python SDK.
-`test/`: Backend tests.
## Code layout to expect
-`api/`: Python API server entrypoints, blueprints, services, and database code.
-`rag/`: ingestion, retrieval, LLM integration, and graph RAG logic.
-`deepdoc/`: parsing and OCR.
-`agent/`: workflow canvas, components, tools, and templates.
-`cmd/`: Go entrypoints. `ragflow_main` is the main server/admin/ingestor binary surface; `ragflow-cli` is the CLI entrypoint.
-`internal/`: main Go application code. Important subtrees:
-`internal/parser/`: parser and chunk libraries used by ingestion and other Go paths.
-`internal/parser/parser/`: typed parse-result parsers for markdown/html/pdf/docx/xlsx/text and related families.
-`internal/parser/chunk/`: chunk operator library and DSL/typed execution helpers.
-`internal/service/`: higher-level business services used by handlers and server flows.
-`internal/storage/`: storage backends and in-memory test doubles.
-`internal/router/`: HTTP route registration.
-`internal/server/`: server bootstrap/config wiring.
-`internal/cpp/`: C++ sources used by native-backed Go features.
-`web/`: frontend application.
-`docker/`: local and production compose files.
-`sdk/` and `test/`: SDK and automated tests.
## 3. Build Instructions
## Go-specific rules
- Treat `internal/ingestion`, `internal/parser`, and `internal/deepdoc` as actively refactored code. Prefer collapsing duplicate paths over preserving transitional wrappers.
- Do not add or preserve deprecated Go APIs just to ease migration inside the repo.
- Remove commented-out Go code instead of leaving recovery notes in place.
- Keep package comments and doc comments aligned with the current runtime path, not with migration history.
### Backend (Python)
The project uses **uv** for dependency management.
## Working rules
- Before editing, inspect the nearest code path that actually owns the behavior.
- Keep changes small and local unless the task is explicitly a broader refactor.
- Prefer one implementation path instead of preserving old and new versions side by side.
- Preserve behavior with focused tests when the behavior is still valid; do not keep tests that protect obsolete behavior.
- If a surface is only there for compatibility, remove it unless the user asks to keep it.
- Do not add new compatibility wording in comments or docs.
- When a maintainer takes over a community PR, a new commit generated by rewriting history (e.g. `merge`, `rebase -i`) must preserve the original author and add the maintainer as co-author (via a `Co-authored-by:` trailer) instead of overwriting the author with the maintainer alone.
docker compose -f docker/docker-compose-base.yml up -d
```
- **Launch**:
```bash
source .venv/bin/activate
export PYTHONPATH=$(pwd)
bash docker/launch_backend_service.sh
```
### Frontend (TypeScript/React)
Located in `web/`.
1. **Install Dependencies**:
```bash
cd web
npm install
```
2. **Run Dev Server**:
```bash
npm run dev
```
Runs on port 8000 by default.
### Docker Deployment
To run the full stack using Docker:
## Commands
### Backend
```bash
cd docker
docker compose -f docker-compose.yml up -d
uv sync --python 3.13 --all-extras
uv run python3 ragflow_deps/download_deps.py
docker compose -f docker/docker-compose-base.yml up -d
source .venv/bin/activate
exportPYTHONPATH=$(pwd)
bash docker/launch_backend_service.sh
uv run pytest
ruff check
ruff format
```
## 4. Testing Instructions
### Frontend
```bash
cd web
npm install
npm run dev
npm run build
npm run lint
npm run test
npm run type-check
```
### Backend Tests
- **Run All Tests**:
```bash
uv run pytest
```
- **Run Specific Test**:
```bash
uv run pytest test/test_api.py
```
### Go
```bash
uv run ragflow_deps/download_deps.py
bash build.sh --test ./path/to/package/...
bash build.sh --go
# or build specific binaries:
bash build.sh --all
```
### Frontend Tests
- **Run Tests**:
```bash
cd web
npm run test
```
## 5. Coding Standards & Guidelines
- **Python Formatting**: Use `ruff` for linting and formatting.
```bash
ruff check
ruff format
```
- **Frontend Linting**:
```bash
cd web
npm run lint
```
- **Pre-commit**: Ensure pre-commit hooks are installed.
```bash
pre-commit install
pre-commit run --all-files
```
## Validation preference
- Run the narrowest relevant test, lint, or build command after a change.
- For backend changes, prefer targeted pytest or ruff checks over full-suite runs.
- For frontend changes, prefer the touched-package lint, type-check, or test command.
- For Go changes, prefer package-scoped `bash build.sh --test ...` first.
- Do not default to raw `go test`, `go build`, or IDE Run/Debug for Go in this repo. They often miss the required CGO flags and native static libraries (`office_oxide`, `pdfium-static`, `pdf_oxide`) that `build.sh` wires correctly.
- If Go native builds fail, inspect `build.sh` and `internal/development.md` before changing code. Common environment issues are missing downloaded native deps and missing `lld` on Linux.
## Default review checklist
- Remove instead of retaining `deprecated`, `legacy`, or compatibility-only code.
- Collapse duplicate implementations to one path.
- Drop stale comments and documentation that describe a superseded design.
- Keep exported APIs only when the current code actually needs them.
@@ -87,6 +87,7 @@ Try our cloud service at [https://cloud.ragflow.io](https://cloud.ragflow.io).
## 🔥 Latest Updates
- 2026-06-15 Support multiple chat channels such as Feishu, Discord, Telegram, Line, etc.
- 2026-04-24 Supports DeepSeek v4.
- 2026-03-24 [RAGFlow Skill on OpenClaw](https://clawhub.ai/yingfeng/ragflow-skill) — Provides an official skill for accessing RAGFlow datasets via OpenClaw.
- 2025-12-26 Supports 'Memory' for AI agent.
@@ -97,7 +98,6 @@ Try our cloud service at [https://cloud.ragflow.io](https://cloud.ragflow.io).
- 2025-08-08 Supports OpenAI's latest GPT-5 series models.
- 2025-08-01 Supports agentic workflow and MCP.
- 2025-05-23 Adds a Python/JavaScript code executor component to Agent.
- 2025-05-05 Supports cross-language query.
- 2025-03-19 Supports using a multi-modal model to make sense of images within PDF or DOCX files.
## 🎉 Stay Tuned
@@ -152,6 +152,7 @@ releases! 🌟
- RAM >= 16 GB
- Disk >= 50 GB
- Docker >= 24.0.0 & Docker Compose >= v2.26.1
- Python >= 3.13
- [gVisor](https://gvisor.dev/docs/user_guide/install/): Required only if you intend to use the code executor (sandbox) feature of RAGFlow.
> [!TIP]
@@ -192,12 +193,12 @@ releases! 🌟
> All Docker images are built for x86 platforms. We don't currently offer Docker images for ARM64.
> If you are on an ARM64 platform, follow [this guide](https://ragflow.io/docs/dev/build_docker_image) to build a Docker image compatible with your system.
> The command below downloads the `v0.25.5` edition of the RAGFlow Docker image. See the following table for descriptions of different RAGFlow editions. To download a RAGFlow edition different from `v0.25.5`, update the `RAGFLOW_IMAGE` variable accordingly in **docker/.env** before using `docker compose` to start the server.
> The command below downloads the `v0.26.4` edition of the RAGFlow Docker image. See the following table for descriptions of different RAGFlow editions. To download a RAGFlow edition different from `v0.26.4`, update the `RAGFLOW_IMAGE` variable accordingly in **docker/.env** before using `docker compose` to start the server.
```bash
$ cd ragflow/docker
# git checkout v0.25.5
git checkout v0.26.4
# Optional: use a stable tag (see releases: https://github.com/infiniflow/ragflow/releases)
# This step ensures the **entrypoint.sh** file in the code matches the Docker image version.
1. Install `uv` and `pre-commit`, or skip this step if they are already installed:
> [!IMPORTANT]
> After cloning the repository for the first time, run `git config --local --unset core.hooksPath`, `uv tool install lefthook` and `lefthook install` once from the repo root to enable local Git hooks.
1. Install `uv`, or skip this step if it is already installed:
```bash
pipx install uv pre-commit
pipx install uv
```
2. Clone the source code and install Python dependencies:
- 23-05-2025 تمت إضافة مكون منفذ كود Python/JavaScript إلى Agent.
- 05-05-2025 يدعم الاستعلام بين اللغات.
- 19-03-2025 يدعم استخدام نموذج متعدد الوسائط لفهم الصور داخل ملفات PDF أو DOCX.
## 🎉 تابعونا
@@ -152,6 +152,7 @@
- الرام >= 16 جيجا
- القرص >= 50 جيجا بايت
- Docker >= 24.0.0 & Docker Compose >= v2.26.1
- بايثون >= 3.13
- [gVisor](https://gvisor.dev/docs/user_guide/install/): مطلوب فقط إذا كنت تنوي استخدام ميزة منفذ التعليمات البرمجية (وضع الحماية) لـ RAGFlow.
> [!TIP]
@@ -192,12 +193,12 @@
> جميع الصور Docker مصممة لمنصات x86. لا نعرض حاليًا صور Docker لـ ARM64.
> إذا كنت تستخدم نظامًا أساسيًا ARM64، فاتبع [هذا الدليل](https://ragflow.io/docs/dev/build_docker_image) لإنشاء صورة Docker متوافقة مع نظامك.
> يقوم الأمر أدناه بتنزيل إصدار `v0.25.5` من الصورة RAGFlow Docker. راجع الجدول التالي للحصول على أوصاف لإصدارات RAGFlow المختلفة. لتنزيل إصدار RAGFlow مختلف عن `v0.25.5`، قم بتحديث المتغير `RAGFLOW_IMAGE` وفقًا لذلك في **docker/.env** قبل استخدام `docker compose` لبدء تشغيل الخادم.
> يقوم الأمر أدناه بتنزيل إصدار `v0.26.4` من الصورة RAGFlow Docker. راجع الجدول التالي للحصول على أوصاف لإصدارات RAGFlow المختلفة. لتنزيل إصدار RAGFlow مختلف عن `v0.26.4`، قم بتحديث المتغير `RAGFLOW_IMAGE` وفقًا لذلك في **docker/.env** قبل استخدام `docker compose` لبدء تشغيل الخادم.
```bash
$ cd ragflow/docker
# git checkout v0.25.5
git checkout v0.26.4
# Optional: use a stable tag (see releases: https://github.com/infiniflow/ragflow/releases)
# This step ensures the **entrypoint.sh** file in the code matches the Docker image version.
@@ -87,6 +87,7 @@ Essayez notre service cloud sur [https://cloud.ragflow.io](https://cloud.ragflow
## 🔥 Dernières mises à jour
- 15-06-2026 Prise en charge de plusieurs canaux de discussion tels que Feishu, Discord, Telegram, Line, etc.
- 24-04-2026 Prise en charge de DeepSeek v4.
- 24-03-2026 [RAGFlow Skill on OpenClaw](https://clawhub.ai/yingfeng/ragflow-skill) — Fournit un skill officiel pour accéder aux datasets RAGFlow via OpenClaw.
- 26-12-2025 Prise en charge de la « Mémoire » pour l'agent IA.
@@ -97,7 +98,6 @@ Essayez notre service cloud sur [https://cloud.ragflow.io](https://cloud.ragflow
- 08-08-2025 Prise en charge des derniers modèles de la série GPT-5 d'OpenAI.
- 01-08-2025 Prise en charge du flux de travail agentique et de MCP.
- 23-05-2025 Ajout d'un composant exécuteur de code Python/JavaScript à l'Agent.
- 05-05-2025 Prise en charge des requêtes inter-langues.
- 19-03-2025 Prise en charge de l'utilisation d'un modèle multi-modal pour analyser les images dans les fichiers PDF ou DOCX.
## 🎉 Restez informé
@@ -150,6 +150,7 @@ Essayez notre service cloud sur [https://cloud.ragflow.io](https://cloud.ragflow
- RAM >= 16 Go
- Disque >= 50 Go
- Docker >= 24.0.0 & Docker Compose >= v2.26.1
- Python >= 3.13
- [gVisor](https://gvisor.dev/docs/user_guide/install/) : Requis uniquement si vous souhaitez utiliser la fonctionnalité d'exécuteur de code (sandbox) de RAGFlow.
> [!TIP]
@@ -189,12 +190,12 @@ Essayez notre service cloud sur [https://cloud.ragflow.io](https://cloud.ragflow
> Toutes les images Docker sont construites pour les plateformes x86. Nous ne proposons pas actuellement d'images Docker pour ARM64.
> Si vous êtes sur une plateforme ARM64, suivez [ce guide](https://ragflow.io/docs/dev/build_docker_image) pour construire une image Docker compatible avec votre système.
> La commande ci-dessous télécharge l'édition `v0.25.5` de l'image Docker RAGFlow. Consultez le tableau suivant pour les descriptions des différentes éditions de RAGFlow. Pour télécharger une édition de RAGFlow différente de `v0.25.5`, mettez à jour la variable `RAGFLOW_IMAGE` dans **docker/.env** avant d'utiliser `docker compose` pour démarrer le serveur.
> La commande ci-dessous télécharge l'édition `v0.26.4` de l'image Docker RAGFlow. Consultez le tableau suivant pour les descriptions des différentes éditions de RAGFlow. Pour télécharger une édition de RAGFlow différente de `v0.26.4`, mettez à jour la variable `RAGFLOW_IMAGE` dans **docker/.env** avant d'utiliser `docker compose` pour démarrer le serveur.
```bash
$ cd ragflow/docker
# git checkout v0.25.5
git checkout v0.26.4
# Optionnel : utiliser un tag stable (voir les versions : https://github.com/infiniflow/ragflow/releases)
# Cette étape garantit que le fichier **entrypoint.sh** dans le code correspond à la version de l'image Docker.
@@ -87,6 +87,7 @@ Coba layanan cloud kami di [https://cloud.ragflow.io](https://cloud.ragflow.io).
## 🔥 Pembaruan Terbaru
- 2026-06-15 Mendukung berbagai saluran obrolan seperti Feishu, Discord, Telegram, Line, dll.
- 2026-04-24 Mendukung DeepSeek v4.
- 2026-03-24 [RAGFlow Skill on OpenClaw](https://clawhub.ai/yingfeng/ragflow-skill) — Menyediakan skill resmi untuk mengakses dataset RAGFlow melalui OpenClaw.
- 2025-12-26 Mendukung 'Memori' untuk agen AI.
@@ -97,10 +98,7 @@ Coba layanan cloud kami di [https://cloud.ragflow.io](https://cloud.ragflow.io).
- 2025-08-08 Mendukung model seri GPT-5 terbaru dari OpenAI.
- 2025-08-01 Mendukung alur kerja agen dan MCP.
- 2025-05-23 Menambahkan komponen pelaksana kode Python/JS ke Agen.
- 2025-05-05 Mendukung kueri lintas bahasa.
- 2025-03-19 Mendukung penggunaan model multi-modal untuk memahami gambar di dalam file PDF atau DOCX.
- 2024-12-18 Meningkatkan model Analisis Tata Letak Dokumen di DeepDoc.
- 2024-08-22 Dukungan untuk teks ke pernyataan SQL melalui RAG.
## 🎉 Tetap Terkini
@@ -152,6 +150,7 @@ Coba layanan cloud kami di [https://cloud.ragflow.io](https://cloud.ragflow.io).
- RAM >= 16 GB
- Disk >= 50 GB
- Docker >= 24.0.0 & Docker Compose >= v2.26.1
- Python >= 3.13
- [gVisor](https://gvisor.dev/docs/user_guide/install/): Hanya diperlukan jika Anda ingin menggunakan fitur eksekutor kode (sandbox) dari RAGFlow.
> [!TIP]
@@ -192,12 +191,12 @@ Coba layanan cloud kami di [https://cloud.ragflow.io](https://cloud.ragflow.io).
> Semua gambar Docker dibangun untuk platform x86. Saat ini, kami tidak menawarkan gambar Docker untuk ARM64.
> Jika Anda menggunakan platform ARM64, [silakan gunakan panduan ini untuk membangun gambar Docker yang kompatibel dengan sistem Anda](https://ragflow.io/docs/dev/build_docker_image).
> Perintah di bawah ini mengunduh edisi v0.25.5 dari gambar Docker RAGFlow. Silakan merujuk ke tabel berikut untuk deskripsi berbagai edisi RAGFlow. Untuk mengunduh edisi RAGFlow yang berbeda dari v0.25.5, perbarui variabel RAGFLOW_IMAGE di docker/.env sebelum menggunakan docker compose untuk memulai server.
> Perintah di bawah ini mengunduh edisi v0.26.4 dari gambar Docker RAGFlow. Silakan merujuk ke tabel berikut untuk deskripsi berbagai edisi RAGFlow. Untuk mengunduh edisi RAGFlow yang berbeda dari v0.26.4, perbarui variabel RAGFLOW_IMAGE di docker/.env sebelum menggunakan docker compose untuk memulai server.
```bash
$ cd ragflow/docker
# git checkout v0.25.5
git checkout v0.26.4
# Opsional: gunakan tag stabil (lihat releases: https://github.com/infiniflow/ragflow/releases)
# This steps ensures the **entrypoint.sh** file in the code matches the Docker image version.
- 2026-06-15 Feishu, Discord, Telegram, Line 등 다양한 채팅 채널을 지원합니다.
- 2026-04-24 DeepSeek v4를 지원합니다.
- 2026-03-24 [RAGFlow Skill on OpenClaw](https://clawhub.ai/yingfeng/ragflow-skill) — OpenClaw를 통해 RAGFlow 데이터셋에 접근하는 공식 스킬 제공.
- 2025-12-26 AI 에이전트의 '메모리' 기능 지원.
@@ -79,10 +80,8 @@
- 2025-08-08 OpenAI의 최신 GPT-5 시리즈 모델을 지원합니다.
- 2025-08-01 에이전트 워크플로우와 MCP를 지원합니다.
- 2025-05-23 Agent에 Python/JS 코드 실행기 구성 요소를 추가합니다.
- 2025-05-05 언어 간 쿼리를 지원합니다.
- 2025-03-19 PDF 또는 DOCX 파일 내의 이미지를 이해하기 위해 다중 모드 모델을 사용하는 것을 지원합니다.
- 2024-12-18 DeepDoc의 문서 레이아웃 분석 모델 업그레이드.
- 2024-08-22 RAG를 통해 SQL 문에 텍스트를 지원합니다.
## 🎉 계속 지켜봐 주세요
@@ -134,6 +133,7 @@
- RAM >= 16 GB
- Disk >= 50 GB
- Docker >= 24.0.0 & Docker Compose >= v2.26.1
- Python >= 3.13
- [gVisor](https://gvisor.dev/docs/user_guide/install/): RAGFlow의 코드 실행기(샌드박스) 기능을 사용하려는 경우에만 필요합니다.
> [!TIP]
@@ -174,12 +174,12 @@
> 모든 Docker 이미지는 x86 플랫폼을 위해 빌드되었습니다. 우리는 현재 ARM64 플랫폼을 위한 Docker 이미지를 제공하지 않습니다.
> ARM64 플랫폼을 사용 중이라면, [시스템과 호환되는 Docker 이미지를 빌드하려면 이 가이드를 사용해 주세요](https://ragflow.io/docs/dev/build_docker_image).
> 아래 명령어는 RAGFlow Docker 이미지의 v0.25.5 버전을 다운로드합니다. 다양한 RAGFlow 버전에 대한 설명은 다음 표를 참조하십시오. v0.25.5와 다른 RAGFlow 버전을 다운로드하려면, docker/.env 파일에서 RAGFLOW_IMAGE 변수를 적절히 업데이트한 후 docker compose를 사용하여 서버를 시작하십시오.
> 아래 명령어는 RAGFlow Docker 이미지의 v0.26.4 버전을 다운로드합니다. 다양한 RAGFlow 버전에 대한 설명은 다음 표를 참조하십시오. v0.26.4와 다른 RAGFlow 버전을 다운로드하려면, docker/.env 파일에서 RAGFLOW_IMAGE 변수를 적절히 업데이트한 후 docker compose를 사용하여 서버를 시작하십시오.
```bash
$ cd ragflow/docker
# git checkout v0.25.5
git checkout v0.26.4
# Optional: use a stable tag (see releases: https://github.com/infiniflow/ragflow/releases)
# 이 단계는 코드의 entrypoint.sh 파일이 Docker 이미지 버전과 일치하도록 보장합니다.
@@ -88,6 +88,7 @@ Experimente o nosso serviço na nuvem em [https://cloud.ragflow.io](https://clou
## 🔥 Últimas Atualizações
- 15-06-2026 Suporte a múltiplos canais de chat, como Feishu, Discord, Telegram, Line, etc..
- 24-04-2026 Suporta DeepSeek v4.
- 24-03-2026 [RAGFlow Skill on OpenClaw](https://clawhub.ai/yingfeng/ragflow-skill) — Fornece um skill oficial para acessar datasets do RAGFlow via OpenClaw.
- 26-12-2025 Suporte à função 'Memória' para agentes de IA.
@@ -98,10 +99,7 @@ Experimente o nosso serviço na nuvem em [https://cloud.ragflow.io](https://clou
- 08-08-2025 Suporta a mais recente série GPT-5 da OpenAI.
- 01-08-2025 Suporta fluxo de trabalho agente e MCP.
- 23-05-2025 Adicione o componente executor de código Python/JS ao Agente.
- 05-05-2025 Suporte a consultas entre idiomas.
- 19-03-2025 Suporta o uso de um modelo multi-modal para entender imagens dentro de arquivos PDF ou DOCX.
- 18-12-2024 Atualiza o modelo de Análise de Layout de Documentos no DeepDoc.
- 22-08-2024 Suporta conversão de texto para comandos SQL via RAG.
## 🎉 Fique Ligado
@@ -153,6 +151,7 @@ Experimente o nosso serviço na nuvem em [https://cloud.ragflow.io](https://clou
- RAM >= 16 GB
- Disco >= 50 GB
- Docker >= 24.0.0 & Docker Compose >= v2.26.1
- Python >= 3.13
- [gVisor](https://gvisor.dev/docs/user_guide/install/): Necessário apenas se você pretende usar o recurso de executor de código (sandbox) do RAGFlow.
> [!TIP]
@@ -192,12 +191,12 @@ Experimente o nosso serviço na nuvem em [https://cloud.ragflow.io](https://clou
> Todas as imagens Docker são construídas para plataformas x86. Atualmente, não oferecemos imagens Docker para ARM64.
> Se você estiver usando uma plataforma ARM64, por favor, utilize [este guia](https://ragflow.io/docs/dev/build_docker_image) para construir uma imagem Docker compatível com o seu sistema.
> O comando abaixo baixa a edição`v0.25.5` da imagem Docker do RAGFlow. Consulte a tabela a seguir para descrições de diferentes edições do RAGFlow. Para baixar uma edição do RAGFlow diferente da `v0.25.5`, atualize a variável `RAGFLOW_IMAGE` conforme necessário no **docker/.env** antes de usar `docker compose` para iniciar o servidor.
> O comando abaixo baixa a edição`v0.26.4` da imagem Docker do RAGFlow. Consulte a tabela a seguir para descrições de diferentes edições do RAGFlow. Para baixar uma edição do RAGFlow diferente da `v0.26.4`, atualize a variável `RAGFLOW_IMAGE` conforme necessário no **docker/.env** antes de usar `docker compose` para iniciar o servidor.
```bash
$ cd ragflow/docker
# git checkout v0.25.5
git checkout v0.26.4
# Opcional: use uma tag estável (veja releases: https://github.com/infiniflow/ragflow/releases)
# Esta etapa garante que o arquivo entrypoint.sh no código corresponda à versão da imagem do Docker.
- 2026-06-15 Feishu, Discord, Telegram, Line vb. gibi birden fazla sohbet kanalını destekleyin.
- 2026-04-24 DeepSeek v4 desteği.
- 2026-03-24 [RAGFlow Skill on OpenClaw](https://clawhub.ai/yingfeng/ragflow-skill) — OpenClaw üzerinden RAGFlow veri setlerine erişmek için resmi bir skill sağlar.
- 2025-12-26 Yapay zeka ajanı için 'Bellek' desteği eklendi.
- [gVisor](https://gvisor.dev/docs/user_guide/install/): Yalnızca RAGFlow'un kod çalıştırıcı (sandbox) özelliğini kullanmayı planlıyorsanız gereklidir.
> Tüm Docker imajları x86 platformları için oluşturulmuştur. Şu anda ARM64 için Docker imajı sunmuyoruz.
> ARM64 platformundaysanız, sisteminizle uyumlu bir Docker imajı oluşturmak için [bu kılavuzu](https://ragflow.io/docs/dev/build_docker_image) takip edin.
> Aşağıdaki komut RAGFlow Docker imajının `v0.25.5` sürümünü indirir. Farklı RAGFlow sürümleri için aşağıdaki tabloya bakın. `v0.25.5` dışında bir sürüm indirmek için, `docker compose` ile sunucuyu başlatmadan önce **docker/.env** dosyasındaki `RAGFLOW_IMAGE` değişkenini güncelleyin.
> Aşağıdaki komut RAGFlow Docker imajının `v0.26.4` sürümünü indirir. Farklı RAGFlow sürümleri için aşağıdaki tabloya bakın. `v0.26.4` dışında bir sürüm indirmek için, `docker compose` ile sunucuyu başlatmadan önce **docker/.env** dosyasındaki `RAGFLOW_IMAGE` değişkenini güncelleyin.
```bash
$ cd ragflow/docker
# git checkout v0.25.5
git checkout v0.26.4
# İsteğe bağlı: Kararlı bir etiket kullanın (sürümler: https://github.com/infiniflow/ragflow/releases)
# Bu adım, koddaki **entrypoint.sh** dosyasının Docker imaj sürümüyle eşleşmesini sağlar.
description="Admin Service's client of [RAGFlow](https://github.com/infiniflow/ragflow). The Admin Service provides user management and system monitoring. "
pub="-----BEGIN PUBLIC KEY-----\nMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEArq9XTUSeYr2+N1h3Afl/z8Dse/2yD0ZGrKwx+EEEcdsBLca9Ynmx3nIB5obmLlSfmskLpBo0UACBmB5rEjBp2Q2f3AG3Hjd4B+gNCG6BDaawuDlgANIhGnaTLrIqWrrcm4EMzJOnAOI1fgzJRsOOUEfaS318Eq9OVO3apEyCCt0lOQK6PuksduOjVxtltDav+guVAA068NrPYmRNabVKRNLJpL8w4D44sfth5RvZ3q9t+6RTArpEtc5sh5ChzvqPOzKGMXW83C95TxmXqpbK6olN4RevSfVjEAgCydH6HN6OhtOQEcnrU97r9H0iZOWwbw3pVrZiUkuRD1R56Wzs2wIDAQAB\n-----END PUBLIC KEY-----"
"system":"You are an intelligent assistant. Your primary function is to answer questions based strictly on the provided knowledge base.\n\n **Essential Rules:**\n - Your answer must be derived **solely** from this knowledge base: `{knowledge}`.\n - **When information is available**: Summarize the content to give a detailed answer.\n - **When information is unavailable**: Your response must contain this exact sentence: \"The answer you are looking for is not found in the knowledge base!\"\n - **Always consider** the entire conversation history.",
"system":'You are an intelligent assistant. Your primary function is to answer questions based strictly on the provided knowledge base.\n\n **Essential Rules:**\n - Your answer must be derived **solely** from this knowledge base: `{knowledge}`.\n - **When information is available**: Summarize the content to give a detailed answer.\n - **When information is unavailable**: Your response must contain this exact sentence: "The answer you are looking for is not found in the knowledge base!"\n - **Always consider** the entire conversation history.',
pub="-----BEGIN PUBLIC KEY-----\nMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEArq9XTUSeYr2+N1h3Afl/z8Dse/2yD0ZGrKwx+EEEcdsBLca9Ynmx3nIB5obmLlSfmskLpBo0UACBmB5rEjBp2Q2f3AG3Hjd4B+gNCG6BDaawuDlgANIhGnaTLrIqWrrcm4EMzJOnAOI1fgzJRsOOUEfaS318Eq9OVO3apEyCCt0lOQK6PuksduOjVxtltDav+guVAA068NrPYmRNabVKRNLJpL8w4D44sfth5RvZ3q9t+6RTArpEtc5sh5ChzvqPOzKGMXW83C95TxmXqpbK6olN4RevSfVjEAgCydH6HN6OhtOQEcnrU97r9H0iZOWwbw3pVrZiUkuRD1R56Wzs2wIDAQAB\n-----END PUBLIC KEY-----"
f"Added tenant for email: {user_info['email']}, A default tenant has been set; changing the default models after login is strongly recommended.")
# TenantLLMService.insert_many(tenant_llm)
logging.info(f"Added tenant for email: {user_info['email']}, A default tenant has been set; changing the default models after login is strongly recommended.")
defcheck_admin_auth(func):
@@ -162,13 +152,13 @@ def check_admin_auth(func):
deflogin_admin(email:str,password:str):
"""
:param email: admin email
:param password: string before decrypt
:param password: string before decrypt (RSA encrypted + base64 encoded)
"""
users=UserService.query(email=email)
ifnotusers:
raiseUserNotFoundError(email)
psw=decrypt(password)
user=UserService.query_user(email,psw)
decrypted=decrypt(password)
user=UserService.query_user(email,decrypted)
ifnotuser:
raiseAdminException("Email and password do not match!")
_logger.debug("[Agent] Built user prompt with length=%d, reasoning=%s, context=%s",len(usr_pmt),bool(kwargs.get("reasoning")),bool(kwargs.get("context")))
ifnotself.tools:
ifself.check_if_canceled("Agent processing"):
return
_logger.debug("[Agent] No tools configured. Delegating to LLM._invoke_async. prompt_count=%d",len(self._param.prompts)ifself._param.promptselse0)
logging.warning("Browser no local browser executable found. Set BROWSER_USE_EXECUTABLE_PATH or preinstall chromium in image to avoid runtime playwright install.")
ifprofile_dir:
browser_kwargs["user_data_dir"]=profile_dir
# browser-use expects profile_directory to be a profile name
# such as "Default" / "Profile 1", not an absolute path.
You are an advanced classification system that categorizes user questions into specific types. Analyze the input question and classify it into ONE of the following categories:
@@ -83,10 +78,7 @@ Here's description of each category:
- Return only the category name without explanations
return"⌛Give me a moment—starting from: \n\n"+re.sub(r"(User's query:|[\\]+)",'',msg[-1]['content'],flags=re.DOTALL)+"\n\nI’ll figure out our best next move."
_,msg,_=self._prepare_prompt_variables()
return"⌛Give me a moment—starting from: \n\n"+re.sub(r"(User's query:|[\\]+)","",msg[-1]["content"],flags=re.DOTALL)+"\n\nI’ll figure out our best next move."
@@ -130,9 +124,7 @@ class E2BProvider(SandboxProvider):
# POST /sandbox/{sandboxID}/execute
raiseRuntimeError(
"E2B provider is not yet fully implemented. "
"Please use the self-managed provider or implement the E2B API integration. "
"See https://github.com/e2b-dev/e2b for API documentation."
"E2B provider is not yet fully implemented. Please use the self-managed provider or implement the E2B API integration. See https://github.com/e2b-dev/e2b for API documentation."
)
defdestroy_instance(self,instance_id:str)->bool:
@@ -208,7 +200,7 @@ class E2BProvider(SandboxProvider):
"min":5,
"max":300,
"description":"API request timeout for code execution",
"de":"CAJAL-Agent für wissenschaftliche Arbeiten",
"zh":"CAJAL 科学论文助手"
},
"description":{
"en":"A local-first scientific paper generation agent for RAGFlow. It is preconfigured for Agnuxo/CAJAL-4B-P2PCLAW through Ollama, retrieves knowledge-base evidence, and drafts citation-grounded LaTeX-ready academic sections.",
"de":"Ein lokal ausgerichteter Agent zur Erstellung wissenschaftlicher Arbeiten in RAGFlow. Er ist für Agnuxo/CAJAL-4B-P2PCLAW über Ollama vorkonfiguriert, ruft Evidenz aus der Wissensdatenbank ab und erstellt zitationsgestützte, LaTeX-fähige akademische Abschnitte.",
"content":"# Research request\n{sys.query}\n\nUse the configured knowledge base retrieval tool before drafting. If no evidence is retrieved, state that limitation clearly."
}
],
"sys_prompt":"## Role & Task\nYou are **CAJAL**, a local-first scientific writing agent specialized in citation-grounded paper generation.\n\nUse RAGFlow retrieval results as the source of truth. Write precise academic content that can be pasted into a paper draft, technical report, or LaTeX manuscript. You are optimized for the local GGUF model `Agnuxo/CAJAL-4B-P2PCLAW` served through Ollama as `agnuxo/cajal-4b-p2pclaw`.\n\n## Operating Rules\n1. Decompose the user request into research goals, target section type, expected citation style, and missing evidence.\n2. Retrieve relevant knowledge-base passages before drafting factual claims.\n3. Ground every substantive claim in retrieved evidence. If evidence is missing, mark it as a limitation instead of inventing a citation.\n4. Prefer structured scientific writing: abstract, introduction, related work, methodology, experiments, results, limitations, and conclusion.\n5. Use LaTeX formatting for equations, symbols, algorithms, tables, and section headings when it helps the manuscript.\n6. Preserve traceability: cite source titles, document names, page numbers, or chunk identifiers when available in retrieved context.\n7. Keep language technical, concise, and reproducible. Avoid marketing language and vague generalizations.\n\n## Output Contract\nReturn one of the following, depending on the user request:\n- A complete paper section with citation markers and a short evidence map.\n- A literature review organized by themes, methods, findings, and gaps.\n- A methodology or experiment section with reproducible steps, assumptions, and limitations.\n- A LaTeX-ready abstract, introduction, or conclusion.\n\nAlways include:\n- **Draft**: the requested scientific content.\n- **Evidence used**: concise bullets mapping claims to retrieved sources.\n- **Limitations**: missing evidence, weak support, or assumptions that require verification.\n",
"temperature":0.2,
"temperatureEnabled":true,
"tools":[
{
"component_name":"Retrieval",
"name":"Retrieval",
"params":{
"cross_languages":[],
"description":"Retrieve papers, reports, datasets, and notes that ground CAJAL scientific writing outputs.",
"empty_response":"",
"kb_ids":[],
"keywords_similarity_weight":0.7,
"outputs":{
"formalized_content":{
"type":"string",
"value":""
}
},
"rerank_id":"",
"similarity_threshold":0.2,
"top_k":1024,
"top_n":10,
"use_kg":false
}
}
],
"topPEnabled":false,
"top_p":0.75,
"user_prompt":"",
"visual_files_var":""
}
},
"upstream":[
"begin"
]
},
"Message:OrangeYearsShine":{
"downstream":[],
"obj":{
"component_name":"Message",
"params":{
"content":[
"{Agent:NewPumasLick@content}"
]
}
},
"upstream":[
"Agent:NewPumasLick"
]
},
"begin":{
"downstream":[
"Agent:NewPumasLick"
],
"obj":{
"component_name":"Begin",
"params":{
"enablePrologue":true,
"inputs":{},
"mode":"conversational",
"prologue":"Hi, I am CAJAL in RAGFlow. Add research papers or datasets to your knowledge base, then ask me to draft a citation-grounded paper section, literature review, methodology, or LaTeX-ready abstract."
"prologue":"Hi, I am CAJAL in RAGFlow. Add research papers or datasets to your knowledge base, then ask me to draft a citation-grounded paper section, literature review, methodology, or LaTeX-ready abstract."
},
"label":"Begin",
"name":"begin"
},
"dragging":false,
"id":"begin",
"measured":{
"height":48,
"width":200
},
"position":{
"x":-9.569875358221438,
"y":205.84018385864917
},
"selected":false,
"sourcePosition":"left",
"targetPosition":"right",
"type":"beginNode"
},
{
"data":{
"form":{
"content":[
"{Agent:NewPumasLick@content}"
]
},
"label":"Scientific Draft",
"name":"Response"
},
"dragging":false,
"id":"Message:OrangeYearsShine",
"measured":{
"height":56,
"width":200
},
"position":{
"x":734.4061285881053,
"y":199.9706031723009
},
"selected":false,
"sourcePosition":"right",
"targetPosition":"left",
"type":"messageNode"
},
{
"data":{
"form":{
"delay_after_error":1,
"description":"",
"exception_comment":"",
"exception_default_value":"",
"exception_goto":[],
"exception_method":null,
"frequencyPenaltyEnabled":false,
"frequency_penalty":0.5,
"llm_id":"agnuxo/cajal-4b-p2pclaw@Ollama",
"maxTokensEnabled":true,
"max_retries":3,
"max_rounds":3,
"max_tokens":32768,
"mcp":[],
"message_history_window_size":12,
"outputs":{
"content":{
"type":"string",
"value":""
}
},
"parameter":"Precise",
"presencePenaltyEnabled":false,
"presence_penalty":0.5,
"prompts":[
{
"role":"user",
"content":"# Research request\n{sys.query}\n\nUse the configured knowledge base retrieval tool before drafting. If no evidence is retrieved, state that limitation clearly."
}
],
"sys_prompt":"## Role & Task\nYou are **CAJAL**, a local-first scientific writing agent specialized in citation-grounded paper generation.\n\nUse RAGFlow retrieval results as the source of truth. Write precise academic content that can be pasted into a paper draft, technical report, or LaTeX manuscript. You are optimized for the local GGUF model `Agnuxo/CAJAL-4B-P2PCLAW` served through Ollama as `agnuxo/cajal-4b-p2pclaw`.\n\n## Operating Rules\n1. Decompose the user request into research goals, target section type, expected citation style, and missing evidence.\n2. Retrieve relevant knowledge-base passages before drafting factual claims.\n3. Ground every substantive claim in retrieved evidence. If evidence is missing, mark it as a limitation instead of inventing a citation.\n4. Prefer structured scientific writing: abstract, introduction, related work, methodology, experiments, results, limitations, and conclusion.\n5. Use LaTeX formatting for equations, symbols, algorithms, tables, and section headings when it helps the manuscript.\n6. Preserve traceability: cite source titles, document names, page numbers, or chunk identifiers when available in retrieved context.\n7. Keep language technical, concise, and reproducible. Avoid marketing language and vague generalizations.\n\n## Output Contract\nReturn one of the following, depending on the user request:\n- A complete paper section with citation markers and a short evidence map.\n- A literature review organized by themes, methods, findings, and gaps.\n- A methodology or experiment section with reproducible steps, assumptions, and limitations.\n- A LaTeX-ready abstract, introduction, or conclusion.\n\nAlways include:\n- **Draft**: the requested scientific content.\n- **Evidence used**: concise bullets mapping claims to retrieved sources.\n- **Limitations**: missing evidence, weak support, or assumptions that require verification.\n",
"temperature":0.2,
"temperatureEnabled":true,
"tools":[
{
"component_name":"Retrieval",
"name":"Retrieval",
"params":{
"cross_languages":[],
"description":"Retrieve papers, reports, datasets, and notes that ground CAJAL scientific writing outputs.",
File diff suppressed because one or more lines are too long
Some files were not shown because too many files have changed in this diff
Show More
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
