## Summary
Six sites used to read the same `parser_config.delimiter` field with
divergent grammars:
- `rag.nlp.get_delimiters` (PDF/DOCX/HTML/EPUB/JSON/CSV/XLSX/email/book)
- `rag.nlp.naive_merge` (custom-delimiter branch)
- `rag.nlp.naive_merge_with_images`
- `rag.nlp._build_cks`
- `deepdoc.parser.txt_parser.parser_txt` (.txt, code)
-
`deepdoc.parser.markdown_parser.MarkdownElementExtractor.get_delimiters`
The six implementations disagreed on bare-vs-wrapped chars, dedupe, sort
order, CRLF normalization, and `re.I` (#17384). The shipped default ``
`\n!?;。；！？` `` was a no-op for `.md` because the markdown path only
matched backtick-wrapped tokens.
## Changes
- **new:** `rag/nlp/delim.py` with `parse_delimiter_field` and
`compile_delimiter_pattern`. Single source of truth. CRLF normalization
at the top; longest-first stable sort; insertion-ordered dedupe; no
`re.I`.
- **refactor:** all six call sites delegate to the helper.
- `rag/nlp/__init__.py::get_delimiters` becomes a thin shim.
- `deepdoc/parser/txt_parser.py::parser_txt` drops the
`[encode/decode/unicode_escape]` round-trip.
- `deepdoc/parser/markdown_parser.py::get_delimiters` honors bare chars
(fixes [1]).
- **tests:** `test/unit_test/rag/test_delim.py` (85 tests) — helper,
acceptance table, frontend parity, static guard against re-inlining.
- **tests:** `test/unit_test/rag/test_delimiter_case_sensitive.py` (from
#17386) updated to retarget the static check at the new helper +
AST-based broader scan.
## Acceptance criteria
- All six sites produce the same regex pattern for the same input.
- Shipped default keeps working for `.txt` / `.pdf` / `.docx`.
- Shipped default for `.md` now splits (was a silent no-op).
- Tooltip example `` `\n##;` `` produces three effective delimiters
regardless of file type.
- Bare whitespace inputs split on every occurrence.
- Backtick-wrapped whitespace splits only on the exact N-char sequence.
- CRLF-line-ending documents split identically to LF-line-ending
documents.
- 123 tests pass (85 new + 38 existing).
## Rebase protocol
As #17385 and #17386 evolve, this branch will be rebased on top. The
only overlap between this PR's diff and the other two is
`test_delimiter_case_sensitive.py`, where #17383 modifies the static
check to point at the new helper location.
---------
Co-authored-by: kiloconnect[bot] <240665456+kiloconnect[bot]@users.noreply.github.com>
Fixes#17202 (and complements #12109).
## Problem
`RAGFlowTxtParser.parser_txt` (`deepdoc/parser/txt_parser.py:36-47`) and
`rag.nlp.naive_merge` (`rag/nlp/__init__.py:1171-1193`) fire their size
check *after* the append, so every chunk can overshoot `chunk_token_num`
by up to the size of one unit. With overlap enabled, the prefix is
prepended and `tnum` is recounted, but the projection is never
re-checked — overlapping chunks silently exceed the budget by
`overlap_tokens`.
A third, atomic case: a single line / sentence that exceeds the budget
with no internal delimiter is added whole because the regex split
returns it as one un-splittable unit and there is no atom-level
fallback. `RAGFlowHtmlParser.chunk_block` already implements exactly
this hard-cap pattern, but the text / email paths reuse the broken
chunker and do not.
Measured on a live dataset (336 `.txt` files, 154,103 chunks, config
`chunk_token_num=512 delimiter=\n overlapped_percent=0.1`): 56.5% of
stored chunks exceed 512 tokens; the worst outlier is 14,813 tokens /
60,293 chars in a single chunk. Symptom downstream: rerank failures on
the >2048-token outliers (ref. #12109) and silent embedding truncation
on every oversize chunk.
## Fix
Mirror the proven pattern in `RAGFlowHtmlParser.chunk_block`:
1. **Proactive projected-total check** in `TxtParser.parser_txt` and in
`naive_merge.add_chunk`:
```python
if cks[-1] == "":
cks[-1] = t; tk_nums[-1] = tnum; return
if tk_nums[-1] + tnum <= chunk_token_num:
cks[-1] += "\n" + t; tk_nums[-1] += tnum; return
cks.append(t); tk_nums.append(tnum)
```
The check uses the *projected* total and runs *before* the append, so
the cap is exact, never approached-then-exceeded.
2. **Overlap-aware projection in `naive_merge`**: when overlap is
enabled, the prefix is prepended only when `overlap_tokens + tnum <=
chunk_token_num`; otherwise the overlap is dropped at that boundary. The
naive_merge-with-images mirror gets the same treatment. Custom-delimiter
behaviour is preserved per the existing test suite.
3. **Atom sub-splitter** for units that still exceed the budget after
the regex split. Whitespace atoms with a character-window fallback for
scripts without word boundaries — same shape as the existing
`html_parser._split_oversized_block`, so behaviour matches for HTML vs
`.txt` vs PDF atomic-oversize.
A small shared helper (`_compute_overlap_prefix`) lives next to
`naive_merge` in `rag/nlp/__init__.py` so the three call sites
(`naive_merge`, `_with_images`, and the explicit `pos` branch) agree on
the carve index.
## Result on the dataset above
| | Before | After |
|---|---|---|
| Chunks > 512 tokens | 56.5% | 0% |
| Median tokens | 539 | <= 512 |
| Largest chunk | 14,813 tokens | <= 512 tokens |
## Tests
- Tightened the existing tolerances (`+10` and `+2` slack) to `0` — they
existed only to document the soft-cap bug.
- Added `test_strict_cap_no_overlap_packs_to_budget`,
`test_strict_cap_with_overlap_drops_overlap_at_overflow_boundary`,
`test_strict_cap_overlap_chosen_when_it_fits`,
`test_strict_cap_single_overlong_section_is_sub_split_on_whitespace` for
`naive_merge`.
- Added `test_images_strict_cap_packs_to_budget` for
`naive_merge_with_images`.
- New `test/unit_test/deepdoc/parser/test_txt_parser.py` covers
`parser_txt` strict cap and atom sub-split. Uses the same path-loading
pattern as the existing `test_html_parser.py` to avoid pulling the deep
import chain into a test-time-only venv.
All 22 unit tests pass on the host venv:
```
test_naive_merge.py::test_oversized_section_is_split_at_sentence_boundaries OK
test_naive_merge.py::test_small_sections_are_merged_not_oversplit OK
test_naive_merge.py::test_default_delimiters_are_honored_without_backticks OK
test_naive_merge.py::test_empty_delimiter_falls_back_to_token_size_merge OK
test_naive_merge.py::test_overlap_prefix_is_counted_in_token_budget OK
test_naive_merge.py::test_custom_delimiter_ignores_chunk_size OK
test_naive_merge.py::test_custom_delimiter_does_not_size_merge OK
test_naive_merge.py::test_images_oversized_section_is_split OK
test_naive_merge.py::test_images_custom_delimiter_preserved OK
test_naive_merge.py::test_images_plain_string_input OK
test_naive_merge.py::test_images_mismatched_lengths_returns_empty OK
test_naive_merge.py::test_images_shared_lazyimage_not_stacked_… OK
test_naive_merge.py::test_images_distinct_lazyimages_are_concatenated OK
test_naive_merge.py::test_strict_cap_no_overlap_packs_to_budget OK
test_naive_merge.py::test_strict_cap_with_overlap_drops_… OK
test_naive_merge.py::test_strict_cap_single_overlong_section_… OK
test_naive_merge.py::test_strict_cap_overlap_chosen_when_it_fits OK
test_naive_merge.py::test_images_strict_cap_packs_to_budget OK
test_txt_parser.py::test_no_overshoot_when_packing_short_lines OK
test_txt_parser.py::test_no_overshoot_at_chunk_boundary OK
test_txt_parser.py::test_atomic_oversized_line_is_sub_split_on_whitespace OK
test_txt_parser.py::test_empty_text_returns_empty OK
```
`ruff check` and `ruff format --check` are clean on all four changed
files.
## Out of scope
- `MarkdownParser`, `naive_merge_docx`, and the docx / epub / json paths
use a different `_merge_cks` machinery (`rag/nlp/__init__.py:1574`) that
already enforces the budget. They are unchanged.
- The `chunk_block` call sites in `deepdoc/parser/html_parser.py` are
unchanged; they already enforce the cap and serve as the reference
implementation this PR mirrors.
Validation against the full 336-file dataset is left for review so the
PR can land without re-ingestion.
---------
Co-authored-by: skbs-eng <skbs-eng@users.noreply.github.com>
Co-authored-by: kiloconnect[bot] <240665456+kiloconnect[bot]@users.noreply.github.com>
Closes#17384.
## Summary
Drops a dead `re.I` flag from two outlier delimiter-parsing sites and
adds regression tests so the inconsistency can't creep back.
## What's wrong
Two of the six delimiter-parsing implementations pass `re.I` to
`re.finditer`:
- `rag/nlp/__init__.py::get_delimiters` (line 1633)
- `deepdoc/parser/txt_parser.py::parser_txt` (line 51)
The other four implementations correctly omit `re.I`:
- `rag/nlp/__init__.py::naive_merge` custom-delimiter path (line 1195)
- `rag/nlp/__init__.py::naive_merge_with_images` custom-delimiter path
(line 1269)
- `rag/nlp/__init__.py::_build_cks` (line 1389)
- `rag/flow/chunker/token_chunker.py` (line 73)
## Why this matters (and why it doesn't break anything)
The flag is **dead code** today. Verified empirically with a Python
REPL:
```python
>>> import re
>>> for m in re.finditer(r"`([^`]+)`", "`end`", re.I):
... print(repr(m.group(1)))
'end' # plain string, no flag attached
>>> re.split("(a)", "Class A is a Sample")
['Cl', 'a', '', 's', ' A i', 's', ' a Sample']
# Case-sensitive: only lowercase 'a' splits. Uppercase 'A' is preserved.
```
`re.I` does not propagate from `re.finditer` to `m.group(1)` or to
downstream `re.split` / `re.match` calls (which all omit `re.I`). So the
actual splitting behavior has always been case-sensitive — removing the
flag is a **defensive cleanup**, not a behavioral fix.
So why bother?
1. **Consistency** — the two sites were the only outliers in a six-way
implementation cluster. The three sibling sites in `rag/nlp/__init__.py`
already omit `re.I`, which strongly suggests the flag was accidental.
2. **Future-proofing** — a refactor could easily propagate the flag to a
downstream `re.split` call where it *would* change behavior. The tests
added here pin the case-sensitive semantics so that regression fails
loudly.
3. **Reader clarity** — the flag is misleading. Anyone reading
`re.finditer(..., re.I)` reasonably assumes case-insensitive matching,
then has to trace all downstream calls to discover it's a no-op.
## Changes
- `rag/nlp/__init__.py` — drop `re.I` from `get_delimiters` (line 1633).
- `deepdoc/parser/txt_parser.py` — drop `re.I` from `parser_txt` (line
51).
- `test/unit_test/rag/test_delimiter_case_sensitive.py` — new test file
with:
- 4 behavioral tests on `get_delimiters` (pattern output + `re.split`
round-trip).
- 3 end-to-end tests through `naive_merge` (bare-char +
backtick-wrapped, both cases).
- 2 parametrized static checks that `re.I` / `re.IGNORECASE` is not
present at either of the two `re.finditer` sites.
## Testing
```
$ pytest test/unit_test/rag/test_delimiter_case_sensitive.py -v
============================= 9 passed in 0.19s ==============================
```
All tests pass on the patched code. Before the patch, the 2 static
checks fail with a clear assertion message (the 7 behavioral tests pass
either way, confirming `re.I` was dead code).
## Related
- #17384 — the issue this PR closes. Note the issue's reproduction code
(`re.split(..., flags=re.I)`) doesn't actually match what the production
code does — the production `re.split` calls all omit `re.I`, which is
why current behavior is already case-sensitive. The fix here is still
valuable as a defensive cleanup + test coverage, but it's not a
behavioral fix per se.
- #17383 — broader parser consolidation (six implementations → one). The
fix here is independent and small enough to land first.
- #17385 — sibling UX PR (tooltip + live preview). Files are disjoint
(`web/src/**` vs `rag/nlp/**` + `deepdoc/parser/**`), so no interaction.
---------
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: kiloconnect[bot] <240665456+kiloconnect[bot]@users.noreply.github.com>
### What problem does this PR solve?
When multiple columns are used as content columns in RDBMS connector,
the generated document text gets chunked by TxtParser which strips
newline delimiters during merge. This causes field names and values from
different columns to be concatenated without any separator, making the
content unreadable.
Changes:
- txt_parser.py: restore newline separator when merging adjacent text
segments within a chunk, so that split sections are not directly
concatenated
- rdbms_connector.py: use double newline between fields and place field
value on a new line after the field name bracket, giving TxtParser
clearer boundaries to work with
Closes#13001
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
Co-authored-by: tunsuytang <tunsuytang@tencent.com>
### What problem does this PR solve?
Fix special delimiter parsing issue #5382
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
### What problem does this PR solve?
Add get_txt function to reduce duplicate code
### Type of change
- [x] Refactoring
---------
Co-authored-by: Kevin Hu <kevinhu.sh@gmail.com>
### What problem does this PR solve?
### Type of change
- [ ] Bug Fix (non-breaking change which fixes an issue)
- [x] New Feature (non-breaking change which adds functionality)
### What problem does this PR solve?
add support for eml file parser
#1363
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
---------
Co-authored-by: Zhedong Cen <cenzhedong2@126.com>
Co-authored-by: Kevin Hu <kevinhu.sh@gmail.com>
