mirror of
https://github.com/infiniflow/ragflow.git
synced 2026-08-08 16:38:01 +08:00
Go's `TokenChunker` kept the captured delimiter glued to the preceding segment on **both** the primary (`chunkFromItem`) and secondary (`children_delimiters`) split paths, while Python's reference `token_chunker` drops it via `_split_text_by_pattern` (`token_chunker.py:79-93`, used by both `_build_json_chunks` and `_split_chunk_docs_by_children`). The divergence leaked the delimiter into every emitted chunk's `text`.