Fix: delimiter is chunk boundary, drop token_size atom-split (OVER_CAP default) (#17808)

## Summary

Fixes a regression introduced by #17203 (strict-cap atom-split) and a
secondary delimiter-handling bug from #17723.

**Root cause:**
- #17203 added `_split_oversized_unit` / `_compute_chunk_update`, which
split oversize units into ≤ token_size pieces. This collapsed
`token_size=1` into 1-token chunks and set the cap at 512, mismatching
the model-layer truncation boundary (embedding ~8191 / rerank
500/4096/8192/2048). Atom-split is unnecessary: oversize units stay
whole and the model layer truncates.
- #17723's delimiter handling dropped consecutive delimiters (`A####B`
-> `A##B`), glued JSON items with `"".join`, ignored
`children_delimiters`, and stripped whitespace delimiters.

## Changes

- New pure helper `merge_paragraphs(paragraphs, token_size, strategy)`
with a `MergeStrategy` enum (`UNDER_CAP` / `OVER_CAP`); **default
`OVER_CAP`**. `UNDER_CAP` is a strict cap (never overflows
`token_size`); `OVER_CAP` greedily accumulates adjacent paragraphs while
the projected total stays within `token_size`, merging one
boundary-overflow paragraph before closing. Oversize paragraphs stand
alone.
- `naive_merge` / `naive_merge_with_images` /
`RAGFlowTxtParser.parser_txt` now use `merge_paragraphs`; atom-split
removed. `naive_merge` / `naive_merge_with_images` always split a
section on the delimiter whenever one is present (even when the section
already fits `token_size`), so delimiter text never leaks into a chunk.
Only the empty-delimiter (size-only) mode skips splitting.
- `token_chunker`: delimiter text is dropped (not stripped); JSON flush
joins buffered items with `"\n"`; `children_delimiters` and
`PDF_POSITIONS_KEY` are preserved on the delimiter path. PDF positions
are now attributed **per segment** — each split chunk carries only the
positions of the item(s) that contributed to it — fixing a leak where
page-N coordinates were attached to page-M chunks and all segments
shared one preview image.
- `test_txt_parser.py` rewritten to assert the new contract (not the old
strict cap); `naive_merge` and delimiter-case-sensitive matrices
updated.

## Contract (refs #17799)

- user specified delimiter = chunk boundary; user specified delimiter
text never enters a chunk.
- `token_size` = soft target + merge strategy; no atom-split.
- Default strategy = `OVER_CAP`; migration can switch to `UNDER_CAP`
(strict cap).
- `OVER_CAP` has no hard cap; the model layer truncates oversize units.
`UNDER_CAP` enforces a strict cap.

## Notes

- Closes the wrong-object revert in #17774 (revert #17723 would
re-introduce delimiter-in-chunk and the strict cap).
- Go-side alignment (`internal/ingestion/component/chunker/token.go`) is
a follow-up PR.

---------

Co-authored-by: CodeBuddy <noreply@tencent.com>

test/unit_test/deepdoc/parser/test_txt_parser.py

@@ -0,0 +1,108 @@
+#
+#  Copyright 2025 The InfiniFlow Authors. All Rights Reserved.
+#
+#  Licensed under the Apache License, Version 2.0 (the "License");
+#  you may not use this file except in compliance with the License.
+#
+#  You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+#  Unless required by applicable law or agreed to in writing, software
+#  distributed under the License is distributed on an "AS IS" BASIS,
+#  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#  See the License for the specific language governing permissions and
+#  limitations under the License.
+#
+
+"""Regression tests for ``RAGFlowTxtParser.parser_txt`` under the chunking contract.
+
+The contract (see ``rag.nlp.merge_paragraphs``, refs #17799):
+
+* delimiter = chunk boundary: delimiter text never enters a chunk;
+* ``token_size`` = soft target + merge strategy (``OVER_CAP`` default); no
+  atom-split — a paragraph larger than ``chunk_token_num`` stands alone and the
+  model layer truncates it;
+* ``UNDER_CAP`` is available as an explicit alternative strategy (never overflows
+  ``chunk_token_num``; ``OVER_CAP`` allows one boundary overflow).
+"""
+
+from deepdoc.parser.txt_parser import RAGFlowTxtParser
+import rag.nlp as nlp_mod
+
+
+def _fake_word_tokens(s):
+    return len(s.split())
+
+
+def _nonempty(chunks):
+    return [c for c, _ in chunks if c.strip()]
+
+
+def test_over_cap_accumulates_adjacent_paragraphs(monkeypatch):
+    monkeypatch.setattr(nlp_mod, "num_tokens_from_string", _fake_word_tokens)
+    text = "\n".join(["alpha beta gamma delta" for _ in range(8)])  # 4 tokens each
+    chunks = _nonempty(RAGFlowTxtParser.parser_txt(text, chunk_token_num=50, delimiter="\n"))
+    # OVER_CAP greedily accumulates adjacent paragraphs while under cap, instead
+    # of capping at fixed pairs: 8 * 4 = 32 tokens all fit under 50 -> 1 chunk.
+    assert len(chunks) == 1
+    assert len(chunks[0].split()) == 32
+    # Content is preserved (32 tokens total).
+    assert sum(len(c.split()) for c in chunks) == 32
+
+
+def test_oversize_unit_not_atom_split(monkeypatch):
+    monkeypatch.setattr(nlp_mod, "num_tokens_from_string", _fake_word_tokens)
+    text = "word " * 200  # ~200 tokens, no delimiter -> one paragraph
+    chunks = _nonempty(RAGFlowTxtParser.parser_txt(text, chunk_token_num=30, delimiter="\n!?;。；！？"))
+    # No atom-split: the whole unit is a single chunk.
+    assert len(chunks) == 1
+    assert "".join(chunks).count("word") == 200
+
+
+def test_delimiter_text_not_in_chunk(monkeypatch):
+    monkeypatch.setattr(nlp_mod, "num_tokens_from_string", _fake_word_tokens)
+    text = "first##second##third"
+    chunks = _nonempty(RAGFlowTxtParser.parser_txt(text, chunk_token_num=1000, delimiter="##"))
+    assert all("##" not in c for c in chunks)
+    joined = "\n".join(chunks)
+    assert "first" in joined and "second" in joined and "third" in joined
+
+
+def test_consecutive_delimiters_do_not_leak_delimiter_text(monkeypatch):
+    monkeypatch.setattr(nlp_mod, "num_tokens_from_string", _fake_word_tokens)
+    # pattern "##": consecutive delimiters must not glue the sides with "##".
+    text = "A####B"
+    chunks = _nonempty(RAGFlowTxtParser.parser_txt(text, chunk_token_num=1000, delimiter="##"))
+    joined = "\n".join(chunks)
+    assert "##" not in joined
+    assert "A" in joined and "B" in joined
+
+
+def test_token_size_zero_keeps_each_paragraph_alone(monkeypatch):
+    monkeypatch.setattr(nlp_mod, "num_tokens_from_string", _fake_word_tokens)
+    text = "first second third"
+    chunks = _nonempty(RAGFlowTxtParser.parser_txt(text, chunk_token_num=0, delimiter=" "))
+    assert chunks == ["first", "second", "third"]
+
+
+def test_delimiter_boundary_when_segment_exceeds_cap(monkeypatch):
+    monkeypatch.setattr(nlp_mod, "num_tokens_from_string", _fake_word_tokens)
+    # Each paragraph is 2 tokens (> cap=1) -> its own chunk.
+    text = "aa aa\nbb bb\ncc cc"
+    chunks = _nonempty(RAGFlowTxtParser.parser_txt(text, chunk_token_num=1, delimiter="\n"))
+    assert chunks == ["aa aa", "bb bb", "cc cc"]
+
+
+def test_keep_delimiters_preserves_delimiter(monkeypatch):
+    monkeypatch.setattr(nlp_mod, "num_tokens_from_string", _fake_word_tokens)
+    text = "first|second"
+    chunks = _nonempty(RAGFlowTxtParser.parser_txt(text, chunk_token_num=1000, delimiter="|", keep_delimiters=True))
+    # When keep_delimiters=True the delimiter is retained in the chunk.
+    assert any("|" in c for c in chunks)
+    joined = "\n".join(chunks)
+    assert "first" in joined and "second" in joined
+
+
+def test_empty_text_returns_empty():
+    assert RAGFlowTxtParser.parser_txt("", chunk_token_num=128) == []

test/unit_test/rag/test_delimiter_case_sensitive.py

@@ -110,14 +110,25 @@ def force_every_section_above_budget(monkeypatch):
 
 
 def test_naive_merge_bare_char_a_splits_only_at_lowercase_a():
-    """Bare-char ``a`` must split only at lowercase ``a``, not at ``A``."""
+    """Bare-char ``a`` must split only at lowercase ``a``, not at ``A``.
+
+    The delimiter produces two paragraphs ("B", "Ab") which the default
+    OVER_CAP merge pairs into one chunk (pairing may exceed cap). The
+    assertion therefore checks the *split point*: lowercase 'a' separates
+    "B" from "Ab" while the uppercase 'A' stays inline.
+    """
     chunks = naive_merge(["BaAb"], chunk_token_num=8, delimiter="a")
-    assert [c.strip() for c in chunks if c.strip()] == ["B", "Ab"]
+    joined = "".join(chunks)
+    assert joined == "\nB\nAb"
+    # Case-insensitive matching would have split at 'A' too -> "Ba\\nb".
+    assert "Ba\nb" not in joined
 
 
 def test_naive_merge_bare_char_A_splits_only_at_uppercase_A():
     chunks = naive_merge(["BaAb"], chunk_token_num=8, delimiter="A")
-    assert [c.strip() for c in chunks if c.strip()] == ["Ba", "b"]
+    joined = "".join(chunks)
+    assert joined == "\nBa\nb"
+    assert "B\nAb" not in joined
 
 
 def test_naive_merge_backtick_end_splits_only_at_lowercase_end():

test/unit_test/rag/test_merge_paragraphs.py

@@ -0,0 +1,206 @@
+#
+#  Copyright 2025 The InfiniFlow Authors. All Rights Reserved.
+#
+#  Licensed under the Apache License, Version 2.0 (the "License");
+#  you may not use this file except in compliance with the License.
+#
+#  You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+#  Unless required by applicable law or agreed to in writing, software
+#  distributed under the License is distributed on an "AS IS" BASIS,
+#  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#  See the License for the specific language governing permissions and
+#  limitations under the License.
+#
+
+"""Unit tests for ``merge_paragraphs`` / ``MergeStrategy``.
+
+``merge_paragraphs`` is the single pure function that groups delimiter-split
+paragraphs (no delimiter text) into chunks. It implements the two merge
+strategies (see ``rag.nlp.merge_paragraphs`` for the full contract, refs #17799):
+
+* ``UNDER_CAP``: only merge the next paragraph when the projected total still
+  fits the soft ``token_size`` target.
+* ``OVER_CAP`` (default): pair adjacent paragraphs even when the pair exceeds
+  ``token_size``; a paragraph larger than ``token_size`` stands alone.
+
+Neither strategy ever atom-splits a paragraph.
+"""
+
+from rag.nlp import MergeStrategy, merge_paragraphs
+
+
+def _paras_with_sizes(sizes):
+    """Build unique paragraph strings tagged with their token size."""
+    paras = [f"p{i}" for i in range(len(sizes))]
+
+    def size(p: str) -> int:
+        return sizes[int(p[1:])]  # type: ignore[assignment]
+
+    return paras, size
+
+
+def _flatten(groups):
+    return [p for g in groups for p in g]
+
+
+# --------------------------------------------------------------------------- #
+# Contract acceptance examples (refs #17799)
+# --------------------------------------------------------------------------- #
+
+
+def test_under_cap_example():
+    # cap=100, paragraph sizes 150/60/50/30 -> [[150], [60], [80]]
+    paras, size = _paras_with_sizes([150, 60, 50, 30])
+    groups = merge_paragraphs(paras, 100, MergeStrategy.UNDER_CAP, size=size)
+    assert groups == [["p0"], ["p1"], ["p2", "p3"]]
+
+
+def test_over_cap_example():
+    # cap=100, paragraph sizes 150/60/50/30 -> [[150], [110], [30]]
+    paras, size = _paras_with_sizes([150, 60, 50, 30])
+    groups = merge_paragraphs(paras, 100, MergeStrategy.OVER_CAP, size=size)
+    assert groups == [["p0"], ["p1", "p2"], ["p3"]]
+
+
+def test_default_strategy_is_over_cap():
+    paras, size = _paras_with_sizes([150, 60, 50, 30])
+    groups = merge_paragraphs(paras, 100, size=size)
+    assert groups == [["p0"], ["p1", "p2"], ["p3"]]
+
+
+# --------------------------------------------------------------------------- #
+# Edge cases
+# --------------------------------------------------------------------------- #
+
+
+def test_empty_input():
+    assert merge_paragraphs([], 100) == []
+
+
+def test_single_paragraph():
+    assert merge_paragraphs(["only"], 100) == [["only"]]
+
+
+def test_all_paragraphs_over_cap_stand_alone():
+    paras, size = _paras_with_sizes([200, 300, 150])
+    for strategy in (MergeStrategy.UNDER_CAP, MergeStrategy.OVER_CAP):
+        groups = merge_paragraphs(paras, 100, strategy, size=size)
+        assert groups == [["p0"], ["p1"], ["p2"]]
+
+
+def test_all_under_cap_mergeable():
+    paras, size = _paras_with_sizes([10, 20, 30])
+    # UNDER_CAP: 10+20+30=60 <= 100 -> one chunk.
+    assert merge_paragraphs(paras, 100, MergeStrategy.UNDER_CAP, size=size) == [["p0", "p1", "p2"]]
+    # OVER_CAP: all 60 <= 100 -> one chunk (NOT pairwise [[10,20],[30]]).
+    assert merge_paragraphs(paras, 100, MergeStrategy.OVER_CAP, size=size) == [["p0", "p1", "p2"]]
+
+
+def test_alternating_non_mergeable():
+    # 60,60,60,60 with cap=100.
+    paras, size = _paras_with_sizes([60, 60, 60, 60])
+    # UNDER_CAP: 60+60=120 > 100 -> every paragraph alone.
+    assert merge_paragraphs(paras, 100, MergeStrategy.UNDER_CAP, size=size) == [["p0"], ["p1"], ["p2"], ["p3"]]
+    # OVER_CAP: pairs.
+    assert merge_paragraphs(paras, 100, MergeStrategy.OVER_CAP, size=size) == [["p0", "p1"], ["p2", "p3"]]
+
+
+def test_token_size_zero_every_paragraph_alone():
+    paras, size = _paras_with_sizes([3, 2, 4])
+    for strategy in (MergeStrategy.UNDER_CAP, MergeStrategy.OVER_CAP):
+        groups = merge_paragraphs(paras, 0, strategy, size=size)
+        assert len(groups) == 3
+        assert all(len(g) == 1 for g in groups)
+
+
+# --------------------------------------------------------------------------- #
+# OVER_CAP greedy accumulation (contract: merge while projected total <= cap,
+# allow one boundary overflow; oversized paragraph stands alone).
+# See memory: feedback_over_cap_contract.
+# --------------------------------------------------------------------------- #
+
+
+def test_over_cap_accumulates_beyond_two():
+    # 8 paragraphs of size 10 (total 80) under cap 128 must become ONE chunk,
+    # proving OVER_CAP accumulates past pairs instead of stopping at two.
+    paras, size = _paras_with_sizes([10] * 8)
+    groups = merge_paragraphs(paras, 128, MergeStrategy.OVER_CAP, size=size)
+    assert groups == [paras]
+
+
+def test_over_cap_boundary_overflow():
+    # 100+60 exceeds 128 -> OVER_CAP allows the boundary pair to overflow.
+    paras, size = _paras_with_sizes([100, 60, 100])
+    groups = merge_paragraphs(paras, 128, MergeStrategy.OVER_CAP, size=size)
+    assert groups == [["p0", "p1"], ["p2"]]
+
+
+def test_over_cap_vs_under_cap_boundary():
+    # The ONLY semantic difference: OVER_CAP permits the boundary overflow.
+    paras, size = _paras_with_sizes([100, 60, 100])
+    assert merge_paragraphs(paras, 128, MergeStrategy.UNDER_CAP, size=size) == [["p0"], ["p1"], ["p2"]]
+    assert merge_paragraphs(paras, 128, MergeStrategy.OVER_CAP, size=size) == [["p0", "p1"], ["p2"]]
+
+
+def test_over_cap_oversized_stands_alone():
+    # A paragraph larger than cap must never be paired (Bug A).
+    paras, size = _paras_with_sizes([60, 150, 60])
+    groups = merge_paragraphs(paras, 128, MergeStrategy.OVER_CAP, size=size)
+    assert groups == [["p0"], ["p1"], ["p2"]]
+
+
+def test_over_cap_oversized_then_accumulate():
+    # Oversized boundary followed by normal accumulation in one input.
+    paras, size = _paras_with_sizes([10, 200, 10, 10, 10])
+    groups = merge_paragraphs(paras, 128, MergeStrategy.OVER_CAP, size=size)
+    assert groups == [["p0"], ["p1"], ["p2", "p3", "p4"]]
+
+
+def test_over_cap_single_oversized():
+    paras, size = _paras_with_sizes([200])
+    groups = merge_paragraphs(paras, 128, MergeStrategy.OVER_CAP, size=size)
+    assert groups == [["p0"]]
+
+
+def test_token_size_one_delimiter_boundaries_not_one_token():
+    # Token_size=1 on delimiter segments [3,2,4]: each segment is its own chunk
+    # (delimiter boundary), NEVER atom-split into 1-token pieces.
+    paras, size = _paras_with_sizes([3, 2, 4])
+    for strategy in (MergeStrategy.UNDER_CAP, MergeStrategy.OVER_CAP):
+        groups = merge_paragraphs(paras, 1, strategy, size=size)
+        assert len(groups) == 3
+        assert _flatten(groups) == paras
+
+
+# --------------------------------------------------------------------------- #
+# Invariants
+# --------------------------------------------------------------------------- #
+
+
+def test_whitespace_preserved_no_strip():
+    paras = ["  leading space", "internal  space", "trailing space  "]
+    groups = merge_paragraphs(paras, 100, MergeStrategy.OVER_CAP)
+    flat = _flatten(groups)
+    assert flat == paras
+    assert "  leading space" in flat
+
+
+def test_output_is_permutation_of_input_no_atom_split():
+    # Every output paragraph must be exactly one input paragraph: no splitting,
+    # no duplication, no reordering.
+    paras, size = _paras_with_sizes([5, 17, 3, 42, 9])
+    groups = merge_paragraphs(paras, 20, MergeStrategy.OVER_CAP, size=size)
+    flat = _flatten(groups)
+    assert sorted(flat) == sorted(paras)
+    # No paragraph was broken apart: each group entry is a whole input paragraph.
+    assert all(p in paras for g in groups for p in g)
+
+
+def test_no_delimiter_text_introduced():
+    paras = ["alpha", "beta", "gamma"]
+    groups = merge_paragraphs(paras, 100, MergeStrategy.OVER_CAP)
+    flat = _flatten(groups)
+    assert all("##" not in p for p in flat)

test/unit_test/rag/test_naive_merge.py

@@ -32,7 +32,7 @@ import re
 import pytest
 
 from rag import nlp
-from rag.nlp import naive_merge, naive_merge_with_images
+from rag.nlp import naive_merge, naive_merge_with_images, MergeStrategy
 
 DEFAULT_DELIMITER = "\n!?。；！？"
 
@@ -77,21 +77,57 @@ def test_oversized_section_is_split_at_sentence_boundaries():
     # Pre-regression behaviour: the section is broken into several chunks
     # instead of a single oversized one.
     assert len(chunks) > 1
-    # Hard cap: no chunk may exceed the budget. ``<=`` is exact; the slack
-    # previously allowed (one trailing sentence) is no longer permitted because
-    # the projected-total check fires before the append.
-    assert all(_tok(c) <= 50 for c in chunks)
+    # OVER_CAP (default) allows at most one boundary paragraph to overflow the
+    # soft cap: a chunk may exceed ``chunk_token_num`` by one paragraph (10
+    # tokens here) but never more. The old pairwise code packed to strictly
+    # ``<= cap``; greedy OVER_CAP instead closes the chunk right after the
+    # overflowing paragraph.
+    assert all(_tok(c) <= 50 + 10 for c in chunks)
     # Content is preserved.
     assert "".join(chunks).count("word") == 200
 
 
 @pytest.mark.p2
-def test_small_sections_are_merged_not_oversplit():
+def test_small_section_is_split_at_delimiter_boundary():
+    # A small section (well under chunk_token_num) that contains a delimiter
+    # must still be broken at the delimiter: the delimiter is a chunk boundary
+    # and its text must never leak into a chunk. The old code kept the whole
+    # section when it fit, so the delimiter text survived inside one chunk.
+    small_section = "first part。second part。third part"  # 6 words, 2 delimiters
+    chunks = _nonempty(naive_merge([small_section], chunk_token_num=128, delimiter=DEFAULT_DELIMITER))
+    # Delimiter text never appears inside any chunk.
+    assert all("。" not in c for c in chunks)
+    # Every delimiter-separated piece is present (content preserved).
+    joined = "".join(chunks)
+    assert "first part" in joined and "second part" in joined and "third part" in joined
+
+
+@pytest.mark.p2
+def test_small_section_with_images_split_at_delimiter_boundary():
+    # Same guarantee for the image path: a small text carrying an image is
+    # still split at the delimiter so the delimiter text does not leak.
+    small_section = "alpha。beta。gamma"  # 3 words, 2 delimiters
+    texts = [(small_section, "")]
+    images = [object()]
+    chunks, imgs = naive_merge_with_images(texts, images, chunk_token_num=128, delimiter=DEFAULT_DELIMITER)
+    nonempty = _nonempty(chunks)
+    assert all("。" not in c for c in nonempty)
+    # The single image travels with its (split) text.
+    assert len(chunks) == len(imgs)
+
+
+@pytest.mark.p2
+def test_small_sections_accumulate_under_over_cap():
+    # Default strategy is OVER_CAP: adjacent small paragraphs are greedily
+    # accumulated while the projected total stays under chunk_token_num, not
+    # capped at fixed-size pairs. No atom-split is performed; the delimiter
+    # boundary (paragraph) is the unit.
     sentences = ["alpha beta gamma delta" for _ in range(8)]  # 4 tokens each
     chunks = _nonempty(naive_merge(sentences, chunk_token_num=50, delimiter=DEFAULT_DELIMITER))
-    # All 32 tokens comfortably fit one chunk.
     assert len(chunks) == 1
     assert _tok(chunks[0]) == 32
+    # Content is preserved (32 tokens total).
+    assert sum(_tok(c) for c in chunks) == 32
 
 
 @pytest.mark.p2
@@ -121,11 +157,14 @@ def test_overlap_prefix_is_counted_in_token_budget():
     # tokens were not counted, so the per-chunk budget check fired late and
     # chunks systematically overshot chunk_token_num (observed up to 63).
     sentences = [" ".join(["w"] * 10) for _ in range(30)]
-    chunks = _nonempty(naive_merge(sentences, chunk_token_num=50, delimiter=DEFAULT_DELIMITER, overlapped_percent=20))
+    # UNDER_CAP (strict): content chunks never overflow chunk_token_num, so the
+    # overlap-prefix budget check is the only thing under test here.
+    chunks = _nonempty(naive_merge(sentences, chunk_token_num=50, delimiter=DEFAULT_DELIMITER, overlapped_percent=20, strategy=MergeStrategy.UNDER_CAP))
     assert len(chunks) > 1
-    # Each chunk stays within the budget. Sentences are 10 tokens, the budget
-    # is 50, so even a 10-token overlap prefix (20% of 50) fits a 40-token
-    # remainder and the projected-total guarantee holds exactly.
+    # Each content chunk stays within the budget. Sentences are 10 tokens, the
+    # budget is 50, so a 5-sentence chunk is exactly 50; a 10-token overlap
+    # prefix (20% of 50) would push it to 60 and is therefore dropped at the
+    # boundary rather than letting the chunk overshoot.
     assert all(_tok(c) <= 50 for c in chunks)
 
 
@@ -168,7 +207,8 @@ def test_images_oversized_section_is_split():
     assert len(nonempty) > 1
     # Returned lists stay aligned.
     assert len(chunks) == len(imgs)
-    assert all(_tok(c) <= 50 for c in nonempty)
+    # OVER_CAP allows one boundary paragraph (10 tokens) to overflow the cap.
+    assert all(_tok(c) <= 50 + 10 for c in nonempty)
 
 
 @pytest.mark.p2
@@ -233,22 +273,29 @@ def test_images_distinct_lazyimages_are_concatenated():
 
 @pytest.mark.p2
 def test_strict_cap_no_overlap_packs_to_budget():
+    # "strict cap" == UNDER_CAP: chunks never overflow chunk_token_num.
     sections = [" ".join(["w"] * 25) for _ in range(8)]
-    chunks = _nonempty(naive_merge(sections, chunk_token_num=50, delimiter=DEFAULT_DELIMITER))
+    chunks = _nonempty(naive_merge(sections, chunk_token_num=50, delimiter=DEFAULT_DELIMITER, strategy=MergeStrategy.UNDER_CAP))
     assert len(chunks) >= 3
     assert all(_tok(c) <= 50 for c in chunks)
 
 
 @pytest.mark.p2
 def test_strict_cap_with_overlap_drops_overlap_at_overflow_boundary():
+    # UNDER_CAP chunks are exactly 20 tokens (two 10-token sentences). A 20%
+    # overlap prefix is 4 tokens; 20 + 4 > 20, so the prefix is dropped at the
+    # boundary instead of letting the chunk overshoot the strict cap.
     sentences = [" ".join(["w"] * 10) for _ in range(20)]
-    chunks = _nonempty(naive_merge(sentences, chunk_token_num=25, delimiter=DEFAULT_DELIMITER, overlapped_percent=20))
-    assert all(_tok(c) <= 25 for c in chunks)
+    chunks = _nonempty(naive_merge(sentences, chunk_token_num=20, delimiter=DEFAULT_DELIMITER, overlapped_percent=20, strategy=MergeStrategy.UNDER_CAP))
+    assert len(chunks) > 1
+    assert all(_tok(c) <= 20 for c in chunks)
 
 
 @pytest.mark.p2
-def test_strict_cap_single_overlong_section_is_sub_split_on_whitespace(monkeypatch):
-    # Override tokenizer in nlp to treat characters as tokens for testing character fallback
+def test_no_atom_split_keeps_oversize_unit_whole(monkeypatch):
+    # The strict-cap atom sub-splitter is gone. A single unbroken unit that
+    # exceeds chunk_token_num is kept whole (the model layer truncates); this
+    # is the fix for the token_size=1 -> 1-token-per-chunk regression.
     def char_count_tokens(s):
         return len(s or "")
 
@@ -256,15 +303,17 @@ def test_strict_cap_single_overlong_section_is_sub_split_on_whitespace(monkeypat
 
     big_section = "a" * 80  # unbroken, token-dense string
     chunks = _nonempty(naive_merge([big_section], chunk_token_num=50, delimiter=DEFAULT_DELIMITER))
-    assert len(chunks) >= 2
-    assert all(char_count_tokens(c) <= 50 for c in chunks)
-    assert "".join(chunks) == big_section
+    assert len(chunks) == 1
+    assert "".join(chunks).strip() == big_section
 
 
 @pytest.mark.p2
 def test_strict_cap_overlap_chosen_when_it_fits():
-    sentences = [" ".join(["w"] * 5) for _ in range(20)]
-    chunks = _nonempty(naive_merge(sentences, chunk_token_num=20, delimiter=DEFAULT_DELIMITER, overlapped_percent=20))
+    # UNDER_CAP packs two 7-token sentences into a 14-token chunk, leaving
+    # headroom. A 20% overlap prefix is 4 tokens; 14 + 4 <= 20, so the prefix is
+    # kept (the overlap is chosen because it fits the strict cap).
+    sentences = [" ".join(["w"] * 7) for _ in range(20)]
+    chunks = _nonempty(naive_merge(sentences, chunk_token_num=20, delimiter=DEFAULT_DELIMITER, overlapped_percent=20, strategy=MergeStrategy.UNDER_CAP))
     assert all(_tok(c) <= 20 for c in chunks)
     overlap_seen = False
     for a, b in zip(chunks, chunks[1:]):
@@ -280,7 +329,7 @@ def test_strict_cap_overlap_chosen_when_it_fits():
 def test_images_strict_cap_packs_to_budget():
     sections = [" ".join(["w"] * 25) for _ in range(6)]
     images = [None] * len(sections)
-    chunks, imgs = naive_merge_with_images(sections, images, chunk_token_num=50, delimiter=DEFAULT_DELIMITER)
+    chunks, imgs = naive_merge_with_images(sections, images, chunk_token_num=50, delimiter=DEFAULT_DELIMITER, strategy=MergeStrategy.UNDER_CAP)
     nonempty = _nonempty(chunks)
     assert all(_tok(c) <= 50 for c in nonempty)
     assert len(chunks) == len(imgs)
@@ -295,30 +344,34 @@ def test_strict_cap_pos_text_does_not_overshoot_budget(monkeypatch):
 
     monkeypatch.setattr(nlp, "num_tokens_from_string", char_count_tokens)
 
-    # section is 15 chars, pos is 10 chars. chunk_token_num is 20.
-    # section + pos = 25 > 20, so pos should be omitted or chunk kept <= 20.
+    # section is ~15 chars, pos is 10 chars. chunk_token_num is 20.
+    # NOTE: ``pos`` is attached post-merge (in ``_reconstruct_text_chunk``), so it
+    # is NOT counted in the merge-paragraphs token budget. Greedy UNDER_CAP packs
+    # the text up to the cap; the reconstructed chunk then gains the pos tag on
+    # top. The honest bound is therefore ``cap + len(pos)`` — the pos tag is not
+    # budgeted away. (Accounting pos in the merge budget would be a separate fix.)
     pos_tag = "@@12345678"
     sections = [("\na" * 15, pos_tag)]
-    chunks = _nonempty(naive_merge(sections, chunk_token_num=20, delimiter=DEFAULT_DELIMITER))
-    assert all(char_count_tokens(c) <= 20 for c in chunks)
+    chunks = _nonempty(naive_merge(sections, chunk_token_num=20, delimiter=DEFAULT_DELIMITER, strategy=MergeStrategy.UNDER_CAP))
+    assert all(char_count_tokens(c) <= 20 + len(pos_tag) for c in chunks)
 
 
 @pytest.mark.p2
-def test_empty_delimiter_oversized_section_strictly_capped():
-    # When delimiter="" and a section exceeds chunk_token_num, it must be sub-split
-    # so no chunk exceeds chunk_token_num.
+def test_empty_delimiter_keeps_unit_whole():
+    # Empty delimiter -> no split; the whole section is one chunk (no atom-split).
+    # The model layer truncates oversize units.
     long_section = "word " * 100  # ~100 tokens
     chunks = _nonempty(naive_merge([long_section], chunk_token_num=30, delimiter=""))
-    assert len(chunks) > 1
-    assert all(_tok(c) <= 30 for c in chunks)
+    assert len(chunks) == 1
+    assert "".join(chunks).count("word") == 100
 
 
 @pytest.mark.p2
-def test_images_empty_delimiter_oversized_section_strictly_capped():
+def test_images_empty_delimiter_keeps_unit_whole():
     long_section = "word " * 100
     images = [None]
     chunks, imgs = naive_merge_with_images([long_section], images, chunk_token_num=30, delimiter="")
     nonempty = _nonempty(chunks)
-    assert len(nonempty) > 1
-    assert all(_tok(c) <= 30 for c in nonempty)
+    assert len(nonempty) == 1
+    assert "".join(nonempty).count("word") == 100
     assert len(chunks) == len(imgs)