## Background
Issue #17889 asks that, when merging adjacent segments, the chunker
first
checks each segment's type and only merges **text** segments —
**table**,
**image**, and any other non-text type must each remain a standalone
chunk
and must never be merged with a neighbouring segment.
## Why this PR closes#17889 (no Go code change required)
After tracing the Go TokenChunker, the requirement is **already
satisfied**
on the structured (JSON / chunks) path. The type-aware rule is enforced
at
three layers in `internal/ingestion/component/chunker/`:
- `common.go:138` `itemDocType` derives the type from `doc_type_kwd`
(`"table"` -> `"table"`, `"image"` -> `"image"`, anything else ->
`"text"`).
It does **not** depend on the `ck_type` field being populated, so the
type
survives even when only `doc_type_kwd` is set (e.g. upstream
Title/Group/Hierarchy chunks).
- `token.go:756` `chunkFromItem` emits a non-text item as a single
standalone
chunk before the merge loop ever runs.
- `token.go:1050` `mergeByTokenSizeFromJSON` forces any non-text chunk
standalone (`if ck.CKType != "text"`); and `token.go:991` starts a
*fresh*
text chunk after a non-text chunk, so text on either side of a
table/image is never merged across it.
The only path without type information is the raw markdown/text/html
string
path (`PayloadFormatMarkdown/Text/HTML`), where the input is by contract
an
untyped string and `applyChildrenDelim` hard-codes `CKType: "text"` so
merging is correct. There is no non-text segment to merge there, so this
is
out of #17889's scope (which is about the merge logic).
## Why the Python side is deferred
The Python `naive` parser path does not thread a `ck_type` through to
`merge_paragraphs` / `naive_merge` / `naive_merge_with_images`
(`rag/nlp/__init__.py`): its parsers emit flat `(text, pos)` sections
plus a
parallel `section_images` list, and the type-aware `_merge_cks` rule
(`rag/nlp/__init__.py:1749`) is only wired into the docx path.
Propagating
`ck_type` end-to-end across every Python parser is a large refactor, so
it is
intentionally **not** part of this PR. The Go engine is the active
ingestion
path, and it already honors the rule.
## This PR
Adds a regression-lock (characterization) test, not a fix:
- `TestTokenChunker_InvokeJSONPayload_KeepsNonTextStandalone` feeds a
`[text, table, text, image, text]` structured payload and asserts it
produces exactly five standalone chunks in the order
`text, table, text, image, text` — proving tables/images stay standalone
and text on either side is not merged across them.
Verified green:
```
bash build.sh --test -run TestTokenChunker_InvokeJSONPayload_KeepsNonTextStandalone ./internal/ingestion/component/chunker/...
--- PASS: TestTokenChunker_InvokeJSONPayload_KeepsNonTextStandalone (0.07s)
```
## Related
- Issue #17889
- PR #17808 (chunking refactor, merged)
- Contract doc #17799
## Summary
Remediates CVE-2026-26209 (HIGH) in `cbor2` by adding `cbor2>=5.9.0` to
`constraint-dependencies` in `pyproject.toml`.
| CVE | Severity | Package | Installed | Fixed in |
|---|---|---|---|---|
| CVE-2026-26209 | HIGH | cbor2 | 5.8.0 | 5.9.0 |
`cbor2` is a transitive dependency pulled in by `ranx` with no version
constraint, stuck at 5.8.0 because lockfile hadn't been re-resolved.
Net effect: inline prose stays on one line (`Hello World`), real `<br>` boundaries survive (including before tags and repeated breaks), and source formatting whitespace no longer over-splits.
## Summary
- `api/apps/services/provider_api_service.py` hardcoded the DashScope
international base URL for Tongyi-Qianwen as `.../compatible-model/v1`
instead of `.../compatible-mode/v1`, in two places (`list_providers`,
lines ~93 and ~116).
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
Ports the dataset knowledge compilation (wiki/graph/tree/mindmap) to the
Go scheduler with a status contract, aligns wiki storage/retrieval with
Python, and sizes prompts by content_length.
## Summary
The generic `buildRequestBody` in `internal/entity/models/base_model.go`
unconditionally forwarded `ChatConfig.MaxTokens` as `"max_tokens"` for
every OpenAI-compatible provider.
Providers that need a different token field already delete or override
it after the call (e.g. Xiaomi uses `max_completion_tokens`, Replicate
uses `max_new_tokens`). This change stops setting `max_tokens` in the
shared builder so it only forwards the parameters common across
providers, and each provider remains free to set its own token limit
field.
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The Go `TokenChunker` text/markdown path (`mergeByTokenSize`)
unconditionally
called `splitOversizedUnit` on any unit that exceeded
`chunk_token_size`,
emitting Go-only sub-chunks. Python's `naive_merge`
(`_merge_paragraph_groups`,
`rag/nlp/__init__.py`) never atom-splits an oversize unit under either
`OVER_CAP` or `UNDER_CAP`: a paragraph larger than the budget becomes
its own
standalone chunk and the model layer truncates it later.
This aligns the text/markdown path with the **structured JSON path**
(`invokeJSONPayload` → `mergeByTokenSizeFromJSON(...,
subSplitOversize=false)`,
#17739). It completes the OVER_CAP alignment started in #17835.
## Summary
- Pin `werkzeug>=3.1.7,<4` and refresh `uv.lock` to **3.1.8**.
- Fixes intermittent corruption of uploaded file bodies: when TCP
segments split right after multipart part headers, Werkzeug **3.1.5**
can include a leading `\r\n` in the file content
([pallets/werkzeug#3088](https://github.com/pallets/werkzeug/issues/3088);
fixed in 3.1.7).
- In RAGFlow this commonly breaks `.xlsx` parsing: ZIP/OOXML magic
(`PK\x03\x04`) no longer matches, the Excel parser falls back to CSV,
then fails with UTF-8 decode errors such as `invalid start/continuation
byte`.
Made with [Cursor](https://cursor.com)
Co-authored-by: zhangjiangshan1 <zhangjiangshan1@kingsoft.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Jin Hai <haijin.chn@gmail.com>
Go's `TokenChunker` kept the captured delimiter glued to the preceding
segment on **both** the primary (`chunkFromItem`) and secondary
(`children_delimiters`) split paths, while Python's reference
`token_chunker` drops it via `_split_text_by_pattern`
(`token_chunker.py:79-93`, used by both `_build_json_chunks` and
`_split_chunk_docs_by_children`). The divergence leaked the delimiter
into every emitted chunk's `text`.
