mirror of
https://github.com/infiniflow/ragflow.git
synced 2026-08-08 16:38:01 +08:00
Converge `TokenChunker.delimiter_mode` from three values (`token_size`, `delimiter`, `one`) to two (`delimiter`, `one`). The unified `delimiter` mode now carries the old `token_size` semantics: when no active (backtick) delimiter is present, text/JSON chunks are merged up to `chunk_token_size`; when a backtick delimiter is present, the text is split by it and not merged. `one` continues to be handled by the separate `OneChunker`.