zlei6/ragflow
1
0
Fork 0
mirror of https://github.com/infiniflow/ragflow.git synced 2026-08-04 14:50:30 +08:00
Code Issues Packages Projects Releases Wiki Activity

refactor(nlp): collapse 6 delimiter-parser implementations into one canonical helper (#17383) (#17387)

Browse Source 
## Summary

Six sites used to read the same `parser_config.delimiter` field with
divergent grammars:

- `rag.nlp.get_delimiters` (PDF/DOCX/HTML/EPUB/JSON/CSV/XLSX/email/book)
- `rag.nlp.naive_merge` (custom-delimiter branch)
- `rag.nlp.naive_merge_with_images`
- `rag.nlp._build_cks`
- `deepdoc.parser.txt_parser.parser_txt` (.txt, code)
-
`deepdoc.parser.markdown_parser.MarkdownElementExtractor.get_delimiters`

The six implementations disagreed on bare-vs-wrapped chars, dedupe, sort
order, CRLF normalization, and `re.I` (#17384). The shipped default ``
`\n!?;。；！？` `` was a no-op for `.md` because the markdown path only
matched backtick-wrapped tokens.

## Changes

- **new:** `rag/nlp/delim.py` with `parse_delimiter_field` and
`compile_delimiter_pattern`. Single source of truth. CRLF normalization
at the top; longest-first stable sort; insertion-ordered dedupe; no
`re.I`.
- **refactor:** all six call sites delegate to the helper.
  - `rag/nlp/__init__.py::get_delimiters` becomes a thin shim.
- `deepdoc/parser/txt_parser.py::parser_txt` drops the
`[encode/decode/unicode_escape]` round-trip.
- `deepdoc/parser/markdown_parser.py::get_delimiters` honors bare chars
(fixes [1]).
- **tests:** `test/unit_test/rag/test_delim.py` (85 tests) — helper,
acceptance table, frontend parity, static guard against re-inlining.
- **tests:** `test/unit_test/rag/test_delimiter_case_sensitive.py` (from
#17386) updated to retarget the static check at the new helper +
AST-based broader scan.

## Acceptance criteria

- All six sites produce the same regex pattern for the same input.
- Shipped default keeps working for `.txt` / `.pdf` / `.docx`.
- Shipped default for `.md` now splits (was a silent no-op).
- Tooltip example `` `\n##;` `` produces three effective delimiters
regardless of file type.
- Bare whitespace inputs split on every occurrence.
- Backtick-wrapped whitespace splits only on the exact N-char sequence.
- CRLF-line-ending documents split identically to LF-line-ending
documents.
- 123 tests pass (85 new + 38 existing).

## Rebase protocol

As #17385 and #17386 evolve, this branch will be rebased on top. The
only overlap between this PR's diff and the other two is
`test_delimiter_case_sensitive.py`, where #17383 modifies the static
check to point at the new helper location.

---------

Co-authored-by: kiloconnect[bot] <240665456+kiloconnect[bot]@users.noreply.github.com>
This commit is contained in:
S
2026-08-02 14:37:14 +05:30
committed by GitHub
parent 01d667296d
commit d4ceeee4ed
14 changed files with 1551 additions and 321 deletions
Download Patch File Download Diff File Expand all files Collapse all files

107
rag/nlp/__init__.py
View File

@@ -28,6 +28,15 @@ from word2number import w2n
from common.token_utils import num_tokens_from_string
# Re-exported below for backwards compatibility; the canonical parser lives
# in ``rag.nlp.delim``.
from rag.nlp.delim import (
compile_delimiter_pattern,
has_wrapped_delimiter,
normalize_text_newlines,
parse_delimiter_field,
)
__all__ = ["rag_tokenizer"]
all_codecs = [
@@ -1290,7 +1299,7 @@ def naive_merge(sections: str | list, chunk_token_num=128, delimiter="\n。；
if isinstance(sections[0], str):
sections = [(s, "") for s in sections]
# Normalize line endings so delimiter ``\n`` matches ``\r\n`` and standalone ``\r``.
sections = [(s.replace("\r\n", "\n").replace("\r", "\n"), pos) for s, pos in sections]
sections = [(normalize_text_newlines(s), pos) for s, pos in sections]
cks = [""]
tk_nums = [0]
@@ -1304,16 +1313,22 @@ def naive_merge(sections: str | list, chunk_token_num=128, delimiter="\n。；
cks.append(text)
tk_nums.append(tk_num)
custom_delimiters = [m.group(1) for m in re.finditer(r"`([^`]+)`", delimiter)]
has_custom = bool(custom_delimiters)
# Parse the delimiter field once, via the canonical helper (#17383).
# `has_custom` means the field contains a backtick-wrapped token — the
# historical signal that chunk_token_num should be bypassed. Splitting
# itself uses every parsed delimiter (bare and wrapped).
parsed_dels = parse_delimiter_field(delimiter)
has_custom = has_wrapped_delimiter(delimiter)
if has_custom:
# Custom delimiters ignore chunk_token_num: each segment is its own chunk.
custom_pattern = "|".join(re.escape(t) for t in sorted(set(custom_delimiters), key=len, reverse=True))
custom_pattern = compile_delimiter_pattern(parsed_dels)
cks, tk_nums = [], []
for sec, pos in sections:
split_sec = re.split(r"(%s)" % custom_pattern, sec, flags=re.DOTALL)
split_sec = re.split(r"(%s)" % custom_pattern, sec, flags=re.DOTALL) if custom_pattern else [sec]
for sub_sec in split_sec:
if re.fullmatch(custom_pattern, sub_sec or ""):
if not sub_sec:
continue
if custom_pattern and re.fullmatch(custom_pattern, sub_sec):
continue
text = "\n" + sub_sec
local_pos = pos
@@ -1329,7 +1344,7 @@ def naive_merge(sections: str | list, chunk_token_num=128, delimiter="\n。；
# Units that exceed the budget after the regex split (a single long line with
# no delimiter, e.g. PDF / .txt runs of unbroken text) are sub-split on
# whitespace atoms with a character-window fallback, mirroring the html path.
dels = get_delimiters(delimiter)
dels = compile_delimiter_pattern(parsed_dels)
for sec, pos in sections:
sec_text = "\n" + sec
if num_tokens_from_string(sec_text) <= chunk_token_num:
@@ -1386,20 +1401,26 @@ def naive_merge_with_images(texts, images, chunk_token_num=128, delimiter="\n。
result_images.append(image)
tk_nums.append(tk_num)
custom_delimiters = [m.group(1) for m in re.finditer(r"`([^`]+)`", delimiter)]
has_custom = bool(custom_delimiters)
# Parse the delimiter field once, via the canonical helper (#17383).
# See the matching block in ``naive_merge`` for the rationale on
# `has_custom` (backtick-wrapped tokens opt into chunk-token-num bypass).
parsed_dels = parse_delimiter_field(delimiter)
has_custom = has_wrapped_delimiter(delimiter)
if has_custom:
# Custom delimiters ignore chunk_token_num: each segment is its own chunk.
custom_pattern = "|".join(re.escape(t) for t in sorted(set(custom_delimiters), key=len, reverse=True))
custom_pattern = compile_delimiter_pattern(parsed_dels)
cks, result_images, tk_nums = [], [], []
for text, image in zip(texts, images):
text_str = text[0] if isinstance(text, tuple) else text
if text_str is None:
text_str = ""
text_str = normalize_text_newlines(text_str)
text_pos = text[1] if isinstance(text, tuple) and len(text) > 1 else ""
split_sec = re.split(r"(%s)" % custom_pattern, text_str)
split_sec = re.split(r"(%s)" % custom_pattern, text_str) if custom_pattern else [text_str]
for sub_sec in split_sec:
if re.fullmatch(custom_pattern, sub_sec or ""):
if not sub_sec:
continue
if custom_pattern and re.fullmatch(custom_pattern, sub_sec):
continue
text_seg = "\n" + sub_sec
local_pos = text_pos
@@ -1416,7 +1437,7 @@ def naive_merge_with_images(texts, images, chunk_token_num=128, delimiter="\n。
# along on every piece (concat_img dedupes when pieces re-merge into a chunk).
# Units still exceeding the budget after the regex split are sub-split on
# whitespace atoms so they cannot blow past the token cap.
dels = get_delimiters(delimiter)
dels = compile_delimiter_pattern(parsed_dels)
for text, image in zip(texts, images):
# if text is tuple, unpack it
if isinstance(text, tuple):
@@ -1425,7 +1446,7 @@ def naive_merge_with_images(texts, images, chunk_token_num=128, delimiter="\n。
else:
text_str = text or ""
text_pos = ""
text_str = normalize_text_newlines(text_str)
text_seg = "\n" + text_str
if num_tokens_from_string(text_seg) <= chunk_token_num:
add_chunk(text_seg, image, text_pos)
@@ -1524,15 +1545,14 @@ def _build_cks(sections, delimiter):
tables = []
images = []
# extract custom delimiters wrapped by backticks: `##`, `---`, etc.
custom_delimiters = [m.group(1) for m in re.finditer(r"`([^`]+)`", delimiter)]
has_custom = bool(custom_delimiters)
if has_custom:
# escape delimiters and build alternation pattern, longest first
custom_pattern = "|".join(re.escape(t) for t in sorted(set(custom_delimiters), key=len, reverse=True))
# capture delimiters so they appear in re.split results
pattern = r"(%s)" % custom_pattern
# Parse the delimiter field once, via the canonical helper (#17383).
# Split on every parsed delimiter (bare and wrapped). `has_custom`
# only controls whether _merge_cks bypasses chunk_token_num (wrapped
# token present in the original field).
parsed_dels = parse_delimiter_field(delimiter)
has_custom = has_wrapped_delimiter(delimiter)
split_pattern = compile_delimiter_pattern(parsed_dels)
pattern = r"(%s)" % split_pattern if split_pattern else ""
seg = ""
for text, image, table in sections:
@@ -1540,7 +1560,7 @@ def _build_cks(sections, delimiter):
if not text:
text = ""
else:
text = "\n" + str(text)
text = "\n" + normalize_text_newlines(str(text))
if table:
# table chunk
@@ -1571,12 +1591,16 @@ def _build_cks(sections, delimiter):
images.append(idx)
continue
# pure text chunk(s)
if has_custom:
# pure text chunk(s) — split on every parsed delimiter when present
if split_pattern:
split_sec = re.split(pattern, text)
for sub_sec in split_sec:
# ① empty or whitespace-only segment → flush current buffer
if not sub_sec or not sub_sec.strip():
if not sub_sec:
continue
# ① matched delimiter (exact capture; do not strip — wrapped
# whitespace delimiters such as `` ` ` `` or `\n` must match here)
if re.fullmatch(split_pattern, sub_sec):
if seg and seg.strip():
s = seg.strip()
cks.append(
@@ -1590,8 +1614,8 @@ def _build_cks(sections, delimiter):
seg = ""
continue
# ② matched custom delimiter (allow surrounding whitespace)
if re.fullmatch(custom_pattern, sub_sec.strip()):
# ② empty or whitespace-only ordinary segment → flush current buffer
if not sub_sec.strip():
if seg and seg.strip():
s = seg.strip()
cks.append(
@@ -1619,8 +1643,8 @@ def _build_cks(sections, delimiter):
}
)
# final flush after loop (only when custom delimiters are used)
if has_custom and seg and seg.strip():
# final flush after loop (only when delimiters were used for splitting)
if split_pattern and seg and seg.strip():
s = seg.strip()
cks.append(
{
@@ -1765,25 +1789,6 @@ def extract_between(text: str, start_tag: str, end_tag: str) -> list[str]:
return re.findall(pattern, text, flags=re.DOTALL)
def get_delimiters(delimiters: str):
dels = []
s = 0
for m in re.finditer(r"`([^`]+)`", delimiters):
f, t = m.span()
dels.append(m.group(1))
dels.extend(list(delimiters[s:f]))
s = t
if s < len(delimiters):
dels.extend(list(delimiters[s:]))
dels.sort(key=lambda x: -len(x))
dels = [re.escape(d) for d in dels if d]
dels = [d for d in dels if d]
dels_pattern = "|".join(dels)
return dels_pattern
class Node:
def __init__(self, level, depth=-1, texts=None):
self.level = level

170
rag/nlp/delim.py Normal file
View File

@@ -0,0 +1,170 @@
#
# Copyright 2025 The InfiniFlow Authors. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
"""Canonical parser for the ``parser_config.delimiter`` field.
Background
----------
The single string field ``parser_config.delimiter`` is consumed by several
parser implementations depending only on the file extension. Before this
module existed, six implementations diverged on:
* whether bare (non-backtick) characters are honored
* dedupe behavior
* sort order
* CRLF / CR normalization
* whether ``re.I`` is applied
* the ``re.escape`` round-trip dance in ``txt_parser``
This module owns the canonical parsing rule. All six implementations now
call :func:`parse_delimiter_field` and :func:`compile_delimiter_pattern`.
Parsing rule
------------
A "delimiter field" is a string with the following grammar::
delimiter_field := token*
token := backtick_wrapped | bare_char
backtick_wrapped := "`" bare_char+ "`"
bare_char := any single Unicode character except "`"
Semantics:
1. Any character(s) between matching backticks is one multi-character
delimiter.
2. Any character outside backticks is its own single-character
delimiter.
3. The two are combined, deduplicated, and sorted longest-first so
``##`` matches before ``#``.
4. ``\\r\\n`` and standalone ``\\r`` are normalized to ``\\n`` at the
top of :func:`parse_delimiter_field` so Windows-line-ending
documents produce identical splits to Unix-line-ending ones.
5. No ``re.I`` is used. Delimiter matching is case-sensitive.
Returns
-------
:func:`parse_delimiter_field` returns a ``list[str]`` of raw delimiter
strings (sorted longest-first, deduplicated, CRLF-normalized).
:func:`compile_delimiter_pattern` takes that list and returns a regex
alternation pattern with ``re.escape`` applied, ready for
``re.split(r"(%s)" % pattern, ...)``.
Frontend parity
---------------
The web UI preview in ``web/src/utils/delimiter-preview.ts``
(``parseDelimitersForDisplay``) follows the same parsing rule
(normalization, dedupe, longest-first order) and applies whitespace
glyph substitution only for display.
"""
from __future__ import annotations
import logging
import re
# Match a backtick-wrapped token. Case-sensitive on purpose (see #17384).
_BACKTICK_RE = re.compile(r"`([^`]+)`")
def normalize_text_newlines(text: str) -> str:
"""Normalize CRLF and standalone CR to LF in source text."""
if not text:
return text
return text.replace("\r\n", "\n").replace("\r", "\n")
def has_wrapped_delimiter(s: str) -> bool:
"""True when the delimiter field contains at least one backtick-wrapped token.
Used to decide the historical "custom delimiter" mode that bypasses
``chunk_token_num``. Separate from whether any delimiter is present
after parsing (bare single-character delimiters still split).
"""
if not s:
return False
return _BACKTICK_RE.search(s) is not None
def parse_delimiter_field(s: str) -> list[str]:
"""Parse the delimiter field into a list of delimiter strings.
Returns an empty list for an empty field. Whitespace characters are
treated as valid single-character delimiters.
The output is sorted longest-first and deduplicated while
preserving the first-occurrence order for equal-length items (the
sort is stable). CRLF / CR line endings inside the field are
normalized to LF so a user typing ``"\\r\\n"`` and a user typing
``"\\n"`` get the same effective delimiter (a single newline).
"""
if not s:
return []
# CRLF normalization: \r\n → \n, then standalone \r → \n. We do this
# before parsing so the parser never sees a \r in either bare-char
# position or backtick-wrapped content.
normalized = normalize_text_newlines(s)
# Insertion-ordered dedupe so equal-length items keep their first-
# occurrence order, which is then preserved by the stable sort below.
delimiters: list[str] = []
seen: set[str] = set()
cursor = 0
for match in _BACKTICK_RE.finditer(normalized):
start, end = match.span()
# Bare characters before this backtick-wrapped token.
for ch in normalized[cursor:start]:
if ch not in seen:
seen.add(ch)
delimiters.append(ch)
# The backtick-wrapped token (verbatim, except CRLF normalization).
token = match.group(1)
if token and token not in seen:
seen.add(token)
delimiters.append(token)
cursor = end
# Bare characters after the last token (or the whole string if no
# backticks were present).
for ch in normalized[cursor:]:
if ch not in seen:
seen.add(ch)
delimiters.append(ch)
# Stable sort by length, longest-first.
result = sorted(delimiters, key=len, reverse=True)
logging.debug(
"parse_delimiter_field: parsed %d delimiters with lengths %s",
len(result),
[len(delimiter) for delimiter in result],
)
return result
def compile_delimiter_pattern(delimiters: list[str]) -> str:
"""Build an alternation regex pattern from a list of delimiter strings.
Each delimiter is ``re.escape``'d so that whitespace and regex
metacharacters are matched literally. The returned pattern is empty
when ``delimiters`` is empty.
The pattern is intended for use with
``re.split(r"(%s)" % pattern, text)`` (capture group so delimiters
appear in the split output) or with
``re.compile(pattern).finditer(text)``.
"""
if not delimiters:
return ""
return "|".join(re.escape(d) for d in delimiters if d)