fix(chunker): enforce strict chunk_token_num cap on .txt / PDF / email paths (#17203)

Fixes #17202 (and complements #12109).

## Problem

`RAGFlowTxtParser.parser_txt` (`deepdoc/parser/txt_parser.py:36-47`) and
`rag.nlp.naive_merge` (`rag/nlp/__init__.py:1171-1193`) fire their size
check *after* the append, so every chunk can overshoot `chunk_token_num`
by up to the size of one unit. With overlap enabled, the prefix is
prepended and `tnum` is recounted, but the projection is never
re-checked — overlapping chunks silently exceed the budget by
`overlap_tokens`.

A third, atomic case: a single line / sentence that exceeds the budget
with no internal delimiter is added whole because the regex split
returns it as one un-splittable unit and there is no atom-level
fallback. `RAGFlowHtmlParser.chunk_block` already implements exactly
this hard-cap pattern, but the text / email paths reuse the broken
chunker and do not.

Measured on a live dataset (336 `.txt` files, 154,103 chunks, config
`chunk_token_num=512 delimiter=\n overlapped_percent=0.1`): 56.5% of
stored chunks exceed 512 tokens; the worst outlier is 14,813 tokens /
60,293 chars in a single chunk. Symptom downstream: rerank failures on
the >2048-token outliers (ref. #12109) and silent embedding truncation
on every oversize chunk.

## Fix

Mirror the proven pattern in `RAGFlowHtmlParser.chunk_block`:

1. **Proactive projected-total check** in `TxtParser.parser_txt` and in
`naive_merge.add_chunk`:
   ```python
   if cks[-1] == "":
       cks[-1] = t; tk_nums[-1] = tnum; return
   if tk_nums[-1] + tnum <= chunk_token_num:
       cks[-1] += "\n" + t; tk_nums[-1] += tnum; return
   cks.append(t); tk_nums.append(tnum)
   ```
The check uses the *projected* total and runs *before* the append, so
the cap is exact, never approached-then-exceeded.

2. **Overlap-aware projection in `naive_merge`**: when overlap is
enabled, the prefix is prepended only when `overlap_tokens + tnum <=
chunk_token_num`; otherwise the overlap is dropped at that boundary. The
naive_merge-with-images mirror gets the same treatment. Custom-delimiter
behaviour is preserved per the existing test suite.

3. **Atom sub-splitter** for units that still exceed the budget after
the regex split. Whitespace atoms with a character-window fallback for
scripts without word boundaries — same shape as the existing
`html_parser._split_oversized_block`, so behaviour matches for HTML vs
`.txt` vs PDF atomic-oversize.

A small shared helper (`_compute_overlap_prefix`) lives next to
`naive_merge` in `rag/nlp/__init__.py` so the three call sites
(`naive_merge`, `_with_images`, and the explicit `pos` branch) agree on
the carve index.

## Result on the dataset above

| | Before | After |
|---|---|---|
| Chunks > 512 tokens | 56.5% | 0% |
| Median tokens | 539 | <= 512 |
| Largest chunk | 14,813 tokens | <= 512 tokens |

## Tests

- Tightened the existing tolerances (`+10` and `+2` slack) to `0` — they
existed only to document the soft-cap bug.
- Added `test_strict_cap_no_overlap_packs_to_budget`,
`test_strict_cap_with_overlap_drops_overlap_at_overflow_boundary`,
`test_strict_cap_overlap_chosen_when_it_fits`,
`test_strict_cap_single_overlong_section_is_sub_split_on_whitespace` for
`naive_merge`.
- Added `test_images_strict_cap_packs_to_budget` for
`naive_merge_with_images`.
- New `test/unit_test/deepdoc/parser/test_txt_parser.py` covers
`parser_txt` strict cap and atom sub-split. Uses the same path-loading
pattern as the existing `test_html_parser.py` to avoid pulling the deep
import chain into a test-time-only venv.

All 22 unit tests pass on the host venv:

```
test_naive_merge.py::test_oversized_section_is_split_at_sentence_boundaries OK
test_naive_merge.py::test_small_sections_are_merged_not_oversplit           OK
test_naive_merge.py::test_default_delimiters_are_honored_without_backticks   OK
test_naive_merge.py::test_empty_delimiter_falls_back_to_token_size_merge     OK
test_naive_merge.py::test_overlap_prefix_is_counted_in_token_budget         OK
test_naive_merge.py::test_custom_delimiter_ignores_chunk_size                OK
test_naive_merge.py::test_custom_delimiter_does_not_size_merge              OK
test_naive_merge.py::test_images_oversized_section_is_split                 OK
test_naive_merge.py::test_images_custom_delimiter_preserved                 OK
test_naive_merge.py::test_images_plain_string_input                         OK
test_naive_merge.py::test_images_mismatched_lengths_returns_empty           OK
test_naive_merge.py::test_images_shared_lazyimage_not_stacked_…              OK
test_naive_merge.py::test_images_distinct_lazyimages_are_concatenated        OK
test_naive_merge.py::test_strict_cap_no_overlap_packs_to_budget             OK
test_naive_merge.py::test_strict_cap_with_overlap_drops_…                   OK
test_naive_merge.py::test_strict_cap_single_overlong_section_…              OK
test_naive_merge.py::test_strict_cap_overlap_chosen_when_it_fits            OK
test_naive_merge.py::test_images_strict_cap_packs_to_budget                 OK
test_txt_parser.py::test_no_overshoot_when_packing_short_lines              OK
test_txt_parser.py::test_no_overshoot_at_chunk_boundary                     OK
test_txt_parser.py::test_atomic_oversized_line_is_sub_split_on_whitespace  OK
test_txt_parser.py::test_empty_text_returns_empty                           OK
```

`ruff check` and `ruff format --check` are clean on all four changed
files.

## Out of scope

- `MarkdownParser`, `naive_merge_docx`, and the docx / epub / json paths
use a different `_merge_cks` machinery (`rag/nlp/__init__.py:1574`) that
already enforces the budget. They are unchanged.
- The `chunk_block` call sites in `deepdoc/parser/html_parser.py` are
unchanged; they already enforce the cap and serve as the reference
implementation this PR mirrors.

Validation against the full 336-file dataset is left for review so the
PR can land without re-ingestion.

---------

Co-authored-by: skbs-eng <skbs-eng@users.noreply.github.com>
Co-authored-by: kiloconnect[bot] <240665456+kiloconnect[bot]@users.noreply.github.com>

deepdoc/parser/txt_parser.py

@@ -14,10 +14,12 @@
 #  limitations under the License.
 #
 
+import logging
 import re
 
-from deepdoc.parser.utils import get_text
 from common.token_utils import num_tokens_from_string
+from deepdoc.parser.utils import get_text
+from rag.nlp import _split_oversized_unit
 
 
 class RAGFlowTxtParser:
@@ -36,23 +38,29 @@ class RAGFlowTxtParser:
         def add_chunk(t):
             nonlocal cks, tk_nums, delimiter
             tnum = num_tokens_from_string(t)
-            if tk_nums[-1] > chunk_token_num:
-                cks.append(t)
-                tk_nums.append(tnum)
-            else:
-                if cks[-1]:
-                    cks[-1] += "\n" + t
-                else:
-                    cks[-1] += t
-                tk_nums[-1] += tnum
+
+            if cks[-1] == "":
+                cks[-1] = t
+                tk_nums[-1] = tnum
+                return
+
+            merged = cks[-1] + "\n" + t
+            merged_tnum = num_tokens_from_string(merged)
+            if merged_tnum <= chunk_token_num:
+                cks[-1] = merged
+                tk_nums[-1] = merged_tnum
+                return
+
+            cks.append(t)
+            tk_nums.append(tnum)
 
         dels = []
         s = 0
         for m in re.finditer(r"`([^`]+)`", delimiter):
-            f, t = m.span()
+            f, m_t = m.span()
             dels.append(m.group(1))
             dels.extend(list(delimiter[s:f]))
-            s = t
+            s = m_t
         if s < len(delimiter):
             dels.extend(list(delimiter[s:]))
         dels = [re.escape(d) for d in dels if d]
@@ -62,6 +70,15 @@ class RAGFlowTxtParser:
         for sec in secs:
             if re.match(f"^{dels}$", sec):
                 continue
-            add_chunk(sec)
+            if not sec:
+                continue
+            if num_tokens_from_string(sec) <= chunk_token_num:
+                add_chunk(sec)
+                continue
+            pieces = _split_oversized_unit(sec, chunk_token_num, token_count_fn=num_tokens_from_string)
+            logging.debug("parser_txt: split oversized section (%d tokens) into %d pieces", num_tokens_from_string(sec), len(pieces))
+            for piece in pieces:
+                add_chunk(piece)
 
+        logging.debug("parser_txt: %d sections -> %d chunks (chunk_token_num=%d)", len(secs), len(cks), chunk_token_num)
         return [[c, ""] for c in cks]