Files
ragflow/internal/ingestion/component/chunker
Jack 2fcc34904b fix(chunker): keep oversize text/markdown unit whole (OVER_CAP alignment) (#17854)
The Go `TokenChunker` text/markdown path (`mergeByTokenSize`)
unconditionally
called `splitOversizedUnit` on any unit that exceeded
`chunk_token_size`,
emitting Go-only sub-chunks. Python's `naive_merge`
(`_merge_paragraph_groups`,
`rag/nlp/__init__.py`) never atom-splits an oversize unit under either
`OVER_CAP` or `UNDER_CAP`: a paragraph larger than the budget becomes
its own
standalone chunk and the model layer truncates it later.

This aligns the text/markdown path with the **structured JSON path**
(`invokeJSONPayload` → `mergeByTokenSizeFromJSON(...,
subSplitOversize=false)`,
#17739). It completes the OVER_CAP alignment started in #17835.
2026-08-05 18:53:59 +08:00
..
2026-07-27 10:20:16 +08:00
2026-08-03 21:30:01 +08:00
2026-07-27 10:20:16 +08:00
2026-07-27 10:20:16 +08:00
2026-07-27 10:20:16 +08:00