- Add Go `EmailParser` support for Outlook `.msg` (OLE2/CFB) files via
the `gomsg` library, in addition to the existing `.eml` (RFC 5322)
support. The `.msg` hard-error is gone; emails with `.msg` attachments
are now ingested end-to-end.
- Re-chunk email attachments into retrievable text (user-oriented). Each
attachment is re-parsed by its file extension through the shared parser
registry and folded back into the same document, so attachment content
becomes searchable. This mirrors Python's legacy `rag/app/email.py`.
Binary attachments (images/audio/video/folders) are skipped by design.
- Restore a corrupted `sample.msg` test fixture and add guards so binary
fixtures are never mangled again (`.gitattributes` marks `*.msg` binary;
`check_files.py` skips NUL-byte files). Also made `check_files.py`
ruff-clean.
This PR fixes **#18193** — the Python `rag/app/book.py` naive-branch `split("@")` bug that destroys PDF coordinate (`@@`) tags, so chunks lose their clickable page highlight.
### Summary
Closes#17885.
MinerU figure enrichment now passes the resolved dataset language to
`vision_llm_figure_describe_prompt`. Missing and empty language values
use `English`, matching the other figure-description paths.
This change is limited to MinerU. PR #18021 already fixed the Mistral
path.
### Summary
Add Querit Contents as a built-in page content tool for RAGFlow Agents
and Canvas workflows.
Querit Contents crawls one or more known URLs and returns their page
content and optional metadata. It complements the existing Querit Search
tool: Search discovers relevant pages, while Contents retrieves pages
already selected by an Agent or workflow.
This integration supports two usage modes:
- A standalone `QueritContents` node in Canvas workflows.
- An embedded content tool available to RAGFlow Agents.
### Summary
`format_document_soup` tracks "am I inside a table" and "am I inside a
link" with sticky flags that are meant to be reset by `elif e.name ==
"/table"` and `elif e.name == "/a"`. BeautifulSoup's `.descendants` only
yields opening tags — a `Tag` named `/table` or `/a` never exists — so
both branches are dead code and neither flag is ever cleared.
Everything after the first `<table>` on a page is therefore formatted as
if it were still table content: paragraphs lose their newline, list
items lose their `- ` marker, headings lose their break, and the text is
glued onto the last table cell. Under
`HTML_BASED_CONNECTOR_TRANSFORM_LINKS_STRATEGY=markdown` the same bug
leaks a link's `href` into everything that follows it, including whole
subsequent paragraphs. The Confluence connector
(`confluence_connector.py:948`) goes through this path.
Real output for a Confluence-shaped page (heading, intro, spec table,
then the body) via the public `parse_html_page_basic`:
**Before**
```
prod us-east-1 Rollback procedure If the canary fails, run the rollback script immediately. Drain the load balancer Revert the deployment Escalate to the on-call rota if the rollback stalls. Do not skip the post-mortem.
```
**After**
```
prod us-east-1
Rollback procedure
If the canary fails, run the rollback script immediately.
- Drain the load balancer
- Revert the deployment
Escalate to [the on-call rota](http://oncall.example.com) if the rollback stalls.
Do not skip the post-mortem.
```
Every heading, paragraph and list marker after the table is lost, and
the whole body is indexed as one run-on line hanging off a table cell.
### Fix
Derive both scopes from each element's **ancestors** instead of from
flags that nothing can clear, and drop the two dead branches plus the
two that become redundant.
The scopes are resolved in one up-front pass into `id`-keyed maps
(`table_scope`, `href_scope`) and looked up in O(1) per element. Probing
per element with `find_parent` instead is O(depth) each, which measured
12–13× slower on table-heavy pages and up to 103× on deeply nested
markup; the map version costs a depth-independent 1.13–1.35× over
`main`. Numbers and method are in the round-2 comment below.
This also changes one adjacent behaviour worth calling out explicitly: a
link **inside** a table cell now renders as markdown, where before it
rendered as plain text. That previous behaviour was not by design — it
only held when no link preceded the table. With a link before the table,
`main` stamps the stale href onto every cell:
```
main: '[pre](http://STALE.com)\n\t[cellA](http://STALE.com)\t[cellB](http://STALE.com)'
branch: '[pre](http://STALE.com)\n\tcellA\tcellB'
```
Those cells are not links. Both symptoms are the same sticky-state bug,
so they are fixed together rather than left half-done.
### Testing
`test/unit_test/data_source/test_html_utils.py` is new —
`format_document_soup` had no test coverage. 11 tests: 8 fail on `main`
and pass on this branch, 3 are controls that pass on both (the table
itself still separates rows and cells, anchor text is still linkified,
the default `strip` strategy still strips).
Representative failures on `main`:
```
assert '\nAfter' in 'Before\n\tA\tB After'
assert '\n- item1' in 'Before\n\tA\tB item1 item2'
assert 'see [link](http://x.com) [ after](http://x.com)' == 'see [link](http://x.com) after'
assert '[next paragraph]' not in '[link](http://x.com)\n[next paragraph](http://x.com)'
```
Reverting each clause of the fix independently keeps the anchors honest:
reverting only the table clause fails exactly the 4 table tests and
leaves the link tests green; reverting only the link clause fails
exactly the 3 link tests and leaves the table tests green.
(`test_link_inside_a_table_cell_is_linkified` needs both clauses broken
to fail, so it appears in neither single-clause revert — it is covered
by the 8-fail run against `main`.)
Full `test/unit_test/data_source/` suite: **3 failed, 199 passed**, and
the failure set is byte-identical to clean `main` (**3 failed, 188
passed**) — the 3 are `TestSSRFValidation::*`, which resolve
`api.example.com` against real DNS and are unrelated to this change.
`ruff check` and `ruff format --check` are clean on both touched files.
---
This PR was drafted with AI assistance (Claude). I reviewed the change,
independently reproduced both symptoms against `main`, and take
responsibility for it.
### Summary
Fixes#18107.
`editdistance==0.8.1` (the only recent release on PyPI) has no cp313
wheels for any platform. Since this project requires exactly Python
3.13, `uv`/`pip`/`poetry` fall back to building it from source (Cython),
which fails on Windows for anyone without a working C build toolchain —
that's the PEP 517 build error in the issue.
Swapped `editdistance` for `rapidfuzz`, which ships full cp313 wheels
(win32/win_amd64/win_arm64 included) and has no build-from-source step
on any of our target platforms. The only call site was
`EntityResolution.is_similarity` in `rag/graphrag/entity_resolution.py`,
using `editdistance.eval(a, b)` to get the unweighted Levenshtein
distance between two entity names.
`rapidfuzz.distance.Levenshtein.distance(a, b)` computes the same thing
(verified identical output on several string pairs) and is used as a
direct replacement.
### Summary
Refs #17885.
Mistral figure enrichment now receives the dataset language through the
production parsing path. `by_mistral_ocr` forwards `lang` to
`MistralParser.parse_pdf`; the parser stores the normalized language and
passes it to the figure-description prompt. Empty or missing values
still fall back to English.
### Summary
Brings both halves of the Tenki sandbox provider onto current SDKs and
removes `project_id`, which Tenki deleted from its API.
**Go:** `github.com/LuxorLabs/tenki-sdk-go/sandbox` `v0.5.2` → `v0.7.0`
(current latest).
**Python:** the provider's SDK was renamed on PyPI — `tenki-sandbox` is
frozen at 0.4.0 and everything from 0.5 ships as
[`tenki`](https://pypi.org/project/tenki/). The docs told operators to
`pip install tenki-sandbox`, which installs a stale SDK that no longer
matches this provider's expectations.
**`project_id` is gone.** Tenki removed project scoping from the sandbox
API in 0.5.x: `Client.create()` no longer accepts `project_id`, so the
current code path would raise `TypeError` against a current SDK. It was
also marked `required: True` in the config schema, so the Admin >
Sandbox Settings form asked for a value that no longer exists.
### What problem does this PR solve?
Incremental Wiki compilation could lose provenance for claim-light
entities, produce unstable page groups across embedding models, route
entities to unrelated pages, and assign topics without sufficient
page-level context. Document removals and page membership changes could
also leave stale Wiki state.
This PR:
- preserves source document and chunk provenance throughout entity
matching, reduction, page generation, and deletion;
- uses embeddings to retrieve candidates and the LLM to make final page
grouping and incremental routing decisions;
- batches embedding and LLM operations with bounded concurrency and
deterministic fallbacks;
- selects source-scoped topic candidates with embeddings before the page
LLM chooses the final topic;
- rebuilds Wiki state when the compilation mode or embedding model
changes;
- normalizes Wiki array fields returned by the API and retains entities
without relations in graph responses.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
## What
This pull request adds **MWS GPT Model Hub** as a built-in model
provider in RAGFlow.
The integration allows users to configure an MWS project endpoint and
token, discover the models available to that project, and use supported
MWS models for chat completion, embeddings, and reranking.
Co-authored-by: ilarionov_n <ilarionov_n@promis.ru>
## Summary
- Extends unit test coverage for
pi/apps/restful_apis/dify_retrieval_api.py (the Dify external knowledge
base endpoint).
Co-authored-by: zjm11902 <zjm11902@users.noreply.github.com>
Co-authored-by: Jin Hai <haijin.chn@gmail.com>
Unifies the Go TokenChunker merge path on a single `mergeUnits` core and
fixes coordinate-tag drift in the Python JSON merge at `overlap > 0`.
Rebased on top of #17979 (delimiter_mode convergence).
Ports dataset knowledge compilation (wiki/graph/tree/mindmap) to the Go
scheduler with a status contract, aligns wiki storage/retrieval with
Python, sizes prompts by content_length, and resolves embedding batch
size from provider capability.
Net effect: inline prose stays on one line (`Hello World`), real `<br>` boundaries survive (including before tags and repeated breaks), and source formatting whitespace no longer over-splits.
Ports the dataset knowledge compilation (wiki/graph/tree/mindmap) to the
Go scheduler with a status contract, aligns wiki storage/retrieval with
Python, and sizes prompts by content_length.
## Summary
Fixes a regression introduced by #17203 (strict-cap atom-split) and a
secondary delimiter-handling bug from #17723.
**Root cause:**
- #17203 added `_split_oversized_unit` / `_compute_chunk_update`, which
split oversize units into ≤ token_size pieces. This collapsed
`token_size=1` into 1-token chunks and set the cap at 512, mismatching
the model-layer truncation boundary (embedding ~8191 / rerank
500/4096/8192/2048). Atom-split is unnecessary: oversize units stay
whole and the model layer truncates.
- #17723's delimiter handling dropped consecutive delimiters (`A####B`
-> `A##B`), glued JSON items with `"".join`, ignored
`children_delimiters`, and stripped whitespace delimiters.
## Changes
- New pure helper `merge_paragraphs(paragraphs, token_size, strategy)`
with a `MergeStrategy` enum (`UNDER_CAP` / `OVER_CAP`); **default
`OVER_CAP`**. `UNDER_CAP` is a strict cap (never overflows
`token_size`); `OVER_CAP` greedily accumulates adjacent paragraphs while
the projected total stays within `token_size`, merging one
boundary-overflow paragraph before closing. Oversize paragraphs stand
alone.
- `naive_merge` / `naive_merge_with_images` /
`RAGFlowTxtParser.parser_txt` now use `merge_paragraphs`; atom-split
removed. `naive_merge` / `naive_merge_with_images` always split a
section on the delimiter whenever one is present (even when the section
already fits `token_size`), so delimiter text never leaks into a chunk.
Only the empty-delimiter (size-only) mode skips splitting.
- `token_chunker`: delimiter text is dropped (not stripped); JSON flush
joins buffered items with `"\n"`; `children_delimiters` and
`PDF_POSITIONS_KEY` are preserved on the delimiter path. PDF positions
are now attributed **per segment** — each split chunk carries only the
positions of the item(s) that contributed to it — fixing a leak where
page-N coordinates were attached to page-M chunks and all segments
shared one preview image.
- `test_txt_parser.py` rewritten to assert the new contract (not the old
strict cap); `naive_merge` and delimiter-case-sensitive matrices
updated.
## Contract (refs #17799)
- user specified delimiter = chunk boundary; user specified delimiter
text never enters a chunk.
- `token_size` = soft target + merge strategy; no atom-split.
- Default strategy = `OVER_CAP`; migration can switch to `UNDER_CAP`
(strict cap).
- `OVER_CAP` has no hard cap; the model layer truncates oversize units.
`UNDER_CAP` enforces a strict cap.
## Notes
- Closes the wrong-object revert in #17774 (revert #17723 would
re-introduce delimiter-in-chunk and the strict cap).
- Go-side alignment (`internal/ingestion/component/chunker/token.go`) is
a follow-up PR.
---------
Co-authored-by: CodeBuddy <noreply@tencent.com>
## Summary
Migrates `mistralai` from `==0.4.2` to `>=2.7.2,<3.0.0` to unblock the
orjson CVE fix. The old SDK pinned `orjson>=3.9.10,<3.11`, preventing
upgrade to the patched version.
| CVE | Severity | Package | Installed | Fixed in |
|---|---|---|---|---|
| CVE-2025-67221 | HIGH | orjson | 3.10.18 | 3.11.6 |
`mistralai` 2.x (the current maintained version) drops the orjson
dependency entirely. Added `orjson>=3.11.6` to `constraint-dependencies`
to pin the floor for remaining parent packages (`langgraph-sdk`,
`langsmith`, `ranx`).
## What
Adds [**SereneDB**](https://serenedb.com) as a selectable doc-store
engine on **both** RAGFlow paths:
- the **Go** `DocEngine` (`internal/engine/serenedb`), alongside
Elasticsearch and Infinity;
- the **Python** `DocStoreConnection` (`rag/utils/serenedb_conn.py`) +
`DOC_ENGINE=serenedb` registration.
SereneDB is a PostgreSQL-wire engine (DuckDB execution) whose single
inverted index carries **both** a scored text column (`@@`, BM25) and an
IVF vector column (`<#>`, inner product), so hybrid search is one SQL
statement. The Go engine connects with `database/sql` + `lib/pq`
(already a dependency, no new module); the Python connector uses
psycopg2 (already a dependency).
## Storage model
One table per tenant with `kb_id` as a filter column - the
**Elasticsearch / OceanBase** model, not Infinity's per-dataset tables.
This keeps BM25 statistics (IDF, avgdl) computed over the whole tenant
corpus (global IDF). Both connectors use this identical layout, so they
are storage- and retrieval-compatible: `hybrid` proxy routing and
Python↔Go switching are safe. On the Python side the connector is wired
as OceanBase's plain-SQL sibling (chunk_data JSON metadata, inline chunk
vectors, verbatim ES field names); the ES tokenizer path is unchanged.
Metadata stays one table per tenant (`ragflow_doc_meta_<tenant>`).
The query shapes mirror the Python connector, including the five
empirically-found landmines: the scored dictionary needs `frequency +
norm` (else `BM25()` silently returns 0.0), the `@@` query is the
tokenized query, the scored lexical branch matches one column, vectors
use an L2-normalized shadow column with `ip`/`sq8`, and the similarity
threshold goes directly in the ANN scan's `WHERE`. **Minimum engine
version: SereneDB 26.07.4.**
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
## Summary
Six sites used to read the same `parser_config.delimiter` field with
divergent grammars:
- `rag.nlp.get_delimiters` (PDF/DOCX/HTML/EPUB/JSON/CSV/XLSX/email/book)
- `rag.nlp.naive_merge` (custom-delimiter branch)
- `rag.nlp.naive_merge_with_images`
- `rag.nlp._build_cks`
- `deepdoc.parser.txt_parser.parser_txt` (.txt, code)
-
`deepdoc.parser.markdown_parser.MarkdownElementExtractor.get_delimiters`
The six implementations disagreed on bare-vs-wrapped chars, dedupe, sort
order, CRLF normalization, and `re.I` (#17384). The shipped default ``
`\n!?;。；！？` `` was a no-op for `.md` because the markdown path only
matched backtick-wrapped tokens.
## Changes
- **new:** `rag/nlp/delim.py` with `parse_delimiter_field` and
`compile_delimiter_pattern`. Single source of truth. CRLF normalization
at the top; longest-first stable sort; insertion-ordered dedupe; no
`re.I`.
- **refactor:** all six call sites delegate to the helper.
- `rag/nlp/__init__.py::get_delimiters` becomes a thin shim.
- `deepdoc/parser/txt_parser.py::parser_txt` drops the
`[encode/decode/unicode_escape]` round-trip.
- `deepdoc/parser/markdown_parser.py::get_delimiters` honors bare chars
(fixes [1]).
- **tests:** `test/unit_test/rag/test_delim.py` (85 tests) — helper,
acceptance table, frontend parity, static guard against re-inlining.
- **tests:** `test/unit_test/rag/test_delimiter_case_sensitive.py` (from
#17386) updated to retarget the static check at the new helper +
AST-based broader scan.
## Acceptance criteria
- All six sites produce the same regex pattern for the same input.
- Shipped default keeps working for `.txt` / `.pdf` / `.docx`.
- Shipped default for `.md` now splits (was a silent no-op).
- Tooltip example `` `\n##;` `` produces three effective delimiters
regardless of file type.
- Bare whitespace inputs split on every occurrence.
- Backtick-wrapped whitespace splits only on the exact N-char sequence.
- CRLF-line-ending documents split identically to LF-line-ending
documents.
- 123 tests pass (85 new + 38 existing).
## Rebase protocol
As #17385 and #17386 evolve, this branch will be rebased on top. The
only overlap between this PR's diff and the other two is
`test_delimiter_case_sensitive.py`, where #17383 modifies the static
check to point at the new helper location.
---------
Co-authored-by: kiloconnect[bot] <240665456+kiloconnect[bot]@users.noreply.github.com>
