Fix: delimiter is chunk boundary, drop token_size atom-split (OVER_CAP default) (#17808)

## Summary

Fixes a regression introduced by #17203 (strict-cap atom-split) and a
secondary delimiter-handling bug from #17723.

**Root cause:**
- #17203 added `_split_oversized_unit` / `_compute_chunk_update`, which
split oversize units into ≤ token_size pieces. This collapsed
`token_size=1` into 1-token chunks and set the cap at 512, mismatching
the model-layer truncation boundary (embedding ~8191 / rerank
500/4096/8192/2048). Atom-split is unnecessary: oversize units stay
whole and the model layer truncates.
- #17723's delimiter handling dropped consecutive delimiters (`A####B`
-> `A##B`), glued JSON items with `"".join`, ignored
`children_delimiters`, and stripped whitespace delimiters.

## Changes

- New pure helper `merge_paragraphs(paragraphs, token_size, strategy)`
with a `MergeStrategy` enum (`UNDER_CAP` / `OVER_CAP`); **default
`OVER_CAP`**. `UNDER_CAP` is a strict cap (never overflows
`token_size`); `OVER_CAP` greedily accumulates adjacent paragraphs while
the projected total stays within `token_size`, merging one
boundary-overflow paragraph before closing. Oversize paragraphs stand
alone.
- `naive_merge` / `naive_merge_with_images` /
`RAGFlowTxtParser.parser_txt` now use `merge_paragraphs`; atom-split
removed. `naive_merge` / `naive_merge_with_images` always split a
section on the delimiter whenever one is present (even when the section
already fits `token_size`), so delimiter text never leaks into a chunk.
Only the empty-delimiter (size-only) mode skips splitting.
- `token_chunker`: delimiter text is dropped (not stripped); JSON flush
joins buffered items with `"\n"`; `children_delimiters` and
`PDF_POSITIONS_KEY` are preserved on the delimiter path. PDF positions
are now attributed **per segment** — each split chunk carries only the
positions of the item(s) that contributed to it — fixing a leak where
page-N coordinates were attached to page-M chunks and all segments
shared one preview image.
- `test_txt_parser.py` rewritten to assert the new contract (not the old
strict cap); `naive_merge` and delimiter-case-sensitive matrices
updated.

## Contract (refs #17799)

- user specified delimiter = chunk boundary; user specified delimiter
text never enters a chunk.
- `token_size` = soft target + merge strategy; no atom-split.
- Default strategy = `OVER_CAP`; migration can switch to `UNDER_CAP`
(strict cap).
- `OVER_CAP` has no hard cap; the model layer truncates oversize units.
`UNDER_CAP` enforces a strict cap.

## Notes

- Closes the wrong-object revert in #17774 (revert #17723 would
re-introduce delimiter-in-chunk and the strict cap).
- Go-side alignment (`internal/ingestion/component/chunker/token.go`) is
a follow-up PR.

---------

Co-authored-by: CodeBuddy <noreply@tencent.com>

deepdoc/parser/txt_parser.py

@@ -17,8 +17,8 @@
 import logging
 import re
 
-from common.token_utils import num_tokens_from_string
 from deepdoc.parser.utils import get_text
+from rag.nlp import MergeStrategy, merge_paragraphs
 from rag.nlp.delim import (
     compile_delimiter_pattern,
     normalize_text_newlines,
@@ -35,22 +35,6 @@ class RAGFlowTxtParser:
     def parser_txt(cls, txt, chunk_token_num=128, delimiter="\n!?;。；！？", keep_delimiters=False):
         if not isinstance(txt, str):
             raise TypeError("txt type should be str!")
-        cks = [""]
-        tk_nums = [0]
-
-        def add_chunk(t):
-            nonlocal cks, tk_nums
-            tnum = num_tokens_from_string(t)
-            if tk_nums[-1] > chunk_token_num:
-                cks.append(t)
-                tk_nums.append(tnum)
-            else:
-                if cks[-1]:
-                    cks[-1] += "\n" + t
-                else:
-                    cks[-1] += t
-                tk_nums[-1] += tnum
-
         txt = normalize_text_newlines(txt)
         parsed_dels = parse_delimiter_field(delimiter)
         dels = compile_delimiter_pattern(parsed_dels)
@@ -60,6 +44,7 @@ class RAGFlowTxtParser:
             bool(dels),
         )
         secs = re.split(r"(%s)" % dels, txt) if dels else [txt]
+        paragraphs = []
         for index, sec in enumerate(secs):
             if dels and re.match(f"^{dels}$", sec):
                 continue
@@ -67,7 +52,12 @@ class RAGFlowTxtParser:
                 continue
             if keep_delimiters and index + 1 < len(secs) and re.match(f"^{dels}$", secs[index + 1]):
                 sec += secs[index + 1]
-            add_chunk(sec)
+            paragraphs.append(sec)
 
+        # Group delimiter-split paragraphs with the OVER_CAP merge strategy: no
+        # atom-split, delimiter text never enters a chunk. A paragraph larger
+        # than chunk_token_num stands alone; the model layer truncates it.
+        groups = merge_paragraphs(paragraphs, chunk_token_num, MergeStrategy.OVER_CAP)
+        cks = ["\n".join(g) for g in groups]
         logging.debug("parser_txt: %d sections -> %d chunks (chunk_token_num=%d)", len(secs), len(cks), chunk_token_num)
         return [[c, ""] for c in cks]