mirror of
https://github.com/infiniflow/ragflow.git
synced 2026-08-04 23:00:30 +08:00
1f8e9af644
## Summary
`TokenChunker._invoke` discarded the user's configured `delimiters`
whenever no backtick-wrapped delimiter was present: it passed a
hardcoded `""` to `naive_merge`, so the configured delimiters (including
the default `["\n"]`) were never forwarded. `naive_merge` then ignored
the newline sentence boundary and cut chunks **mid-sentence** once the
token budget was exceeded.
## Root cause
`token_chunker.py:326` called `naive_merge(payload, chunk_token_size,
"", overlapped_percent)`. `_compile_delimiter_pattern` intentionally
returns `""` for bare (non-backtick) delimiters — that return value is a
*path selector* (empty → token-budget merge; non-empty → hard
`_split_text_by_pattern` split). The bug was not in that selector but in
the `else` branch, which threw away `self._param.delimiters` instead of
forwarding it.
## Fix
Forward the configured delimiters as a soft boundary:
```python
else naive_merge(
payload,
self._param.chunk_token_size,
"".join(self._param.delimiters),
overlapped_percent,
)
```
`naive_merge` already parses the string via the canonical
`parse_delimiter_field`, so bare and backtick-wrapped delimiters are
honored as soft boundaries while the token budget is still respected.
The path-selection role of `_compile_delimiter_pattern` is untouched:
backtick-wrapped delimiters still select the hard
`_split_text_by_pattern` path; bare delimiters still take the
token-budget merge path. No regression for any previously-working
(wrapped-delimiter) configuration.
## Test
Adds `rag/flow/tests/test_token_chunker_delimiter.py`:
- `test_token_chunker_token_size_mode_does_not_split_sentences` — fails
on the old code (3 sentences cut mid-stream), passes after the fix.
- `test_naive_merge_empty_delimiter_ignores_newline_break` — root-cause
companion asserting `naive_merge("")` cuts while `naive_merge("\n")`
preserves boundaries.
Both tests skip when the tokenizer is unavailable (dead-tokenizer
guard).
## Note
This branch contains only this one-line fix on top of `main`; it is
intentionally independent of the unrelated tokenizer/offline-BPE work.
Co-authored-by: CodeBuddy <noreply@codebuddy.ai>