refactor(nlp): collapse 6 delimiter-parser implementations into one canonical helper (#17383) (#17387)

## Summary

Six sites used to read the same `parser_config.delimiter` field with
divergent grammars:

- `rag.nlp.get_delimiters` (PDF/DOCX/HTML/EPUB/JSON/CSV/XLSX/email/book)
- `rag.nlp.naive_merge` (custom-delimiter branch)
- `rag.nlp.naive_merge_with_images`
- `rag.nlp._build_cks`
- `deepdoc.parser.txt_parser.parser_txt` (.txt, code)
-
`deepdoc.parser.markdown_parser.MarkdownElementExtractor.get_delimiters`

The six implementations disagreed on bare-vs-wrapped chars, dedupe, sort
order, CRLF normalization, and `re.I` (#17384). The shipped default ``
`\n!?;。；！？` `` was a no-op for `.md` because the markdown path only
matched backtick-wrapped tokens.

## Changes

- **new:** `rag/nlp/delim.py` with `parse_delimiter_field` and
`compile_delimiter_pattern`. Single source of truth. CRLF normalization
at the top; longest-first stable sort; insertion-ordered dedupe; no
`re.I`.
- **refactor:** all six call sites delegate to the helper.
  - `rag/nlp/__init__.py::get_delimiters` becomes a thin shim.
- `deepdoc/parser/txt_parser.py::parser_txt` drops the
`[encode/decode/unicode_escape]` round-trip.
- `deepdoc/parser/markdown_parser.py::get_delimiters` honors bare chars
(fixes [1]).
- **tests:** `test/unit_test/rag/test_delim.py` (85 tests) — helper,
acceptance table, frontend parity, static guard against re-inlining.
- **tests:** `test/unit_test/rag/test_delimiter_case_sensitive.py` (from
#17386) updated to retarget the static check at the new helper +
AST-based broader scan.

## Acceptance criteria

- All six sites produce the same regex pattern for the same input.
- Shipped default keeps working for `.txt` / `.pdf` / `.docx`.
- Shipped default for `.md` now splits (was a silent no-op).
- Tooltip example `` `\n##;` `` produces three effective delimiters
regardless of file type.
- Bare whitespace inputs split on every occurrence.
- Backtick-wrapped whitespace splits only on the exact N-char sequence.
- CRLF-line-ending documents split identically to LF-line-ending
documents.
- 123 tests pass (85 new + 38 existing).

## Rebase protocol

As #17385 and #17386 evolve, this branch will be rebased on top. The
only overlap between this PR's diff and the other two is
`test_delimiter_case_sensitive.py`, where #17383 modifies the static
check to point at the new helper location.

---------

Co-authored-by: kiloconnect[bot] <240665456+kiloconnect[bot]@users.noreply.github.com>

test/unit_test/rag/conftest.py

@@ -14,7 +14,10 @@
 #  limitations under the License.
 #
 
-"""Restore the real ``common.data_source`` package before importing rag unit tests.
+"""Shared fixtures for ``rag`` unit tests.
+
+Also restores the real ``common.data_source`` package before importing rag
+unit tests.
 
 ``test/unit_test/data_source/conftest.py`` registers a lightweight
 ``sys.modules["common.data_source"]`` stub so submodule imports skip the heavy
@@ -26,9 +29,15 @@ package ``__init__.py``. Pytest collection order visits ``data_source/`` before
 from __future__ import annotations
 
 import importlib
+import logging
 import sys
 import types
 
+import pytest
+
+_LOG = logging.getLogger(__name__)
+_PDF_PARSER_KEY = "deepdoc.parser.pdf_parser"
+
 
 def _restore_common_data_source_package() -> None:
     mod = sys.modules.get("common.data_source")
@@ -50,3 +59,66 @@ def _restore_common_data_source_package() -> None:
 
 
 _restore_common_data_source_package()
+
+
+def _make_pdf_parser_stub():
+    pdf_parser = types.ModuleType(_PDF_PARSER_KEY)
+
+    class _StubPdfParser:
+        @staticmethod
+        def remove_tag(text):
+            return text
+
+    pdf_parser.RAGFlowPdfParser = _StubPdfParser
+    return pdf_parser
+
+
+def _install_pdf_parser_stub() -> None:
+    """Install a lightweight stub so ``rag.nlp`` imports without deepdoc/infinity.
+
+    Must run at conftest import time: delimiter and naive_merge tests import
+    ``rag.nlp`` at module scope, which is before any fixture runs.
+    """
+    if _PDF_PARSER_KEY in sys.modules:
+        _LOG.debug(
+            "pdf_parser_stub: retaining existing module for %s",
+            _PDF_PARSER_KEY,
+        )
+        return
+    sys.modules[_PDF_PARSER_KEY] = _make_pdf_parser_stub()
+    _LOG.debug("pdf_parser_stub: installed stub for %s", _PDF_PARSER_KEY)
+
+
+_install_pdf_parser_stub()
+
+
+@pytest.fixture
+def pdf_parser_stub():
+    """Ensure the pdf_parser stub is installed for the duration of a test.
+
+    Saves and restores any pre-existing ``sys.modules`` entry so tests that
+    opt into this fixture do not permanently replace a real module loaded
+    earlier in the session.
+    """
+    previous = sys.modules.get(_PDF_PARSER_KEY)
+    stub = _make_pdf_parser_stub()
+    sys.modules[_PDF_PARSER_KEY] = stub
+    _LOG.debug("pdf_parser_stub fixture: installed stub for %s", _PDF_PARSER_KEY)
+    try:
+        yield stub
+    finally:
+        if previous is None:
+            # Keep a stub in place so later module-level imports still work;
+            # only restore when a real prior module existed.
+            if sys.modules.get(_PDF_PARSER_KEY) is stub:
+                pass
+            _LOG.debug(
+                "pdf_parser_stub fixture: left stub in place for %s",
+                _PDF_PARSER_KEY,
+            )
+        else:
+            sys.modules[_PDF_PARSER_KEY] = previous
+            _LOG.debug(
+                "pdf_parser_stub fixture: restored prior module for %s",
+                _PDF_PARSER_KEY,
+            )