mirror of
https://github.com/infiniflow/ragflow.git
synced 2026-07-15 01:18:26 +08:00
e54e7ec7ef
Fixes #16812 ### Problem In the `rag/flow` ingestion pipeline, when `TitleChunker` feeds `TokenChunker`, the chapter-aware chunks are silently discarded and the parser's raw flat json is re-chunked instead. `TitleChunker` emits `output_format="chunks"` and writes its chapter-aware output to the `chunks` field (`rag/flow/chunker/title_chunker/common.py`, `set_output("output_format", "chunks")`). But `TokenChunker._invoke` only handles `output_format` in `["markdown", "text", "html"]`, then falls through to the `# json` path which reads `from_upstream.json_result`. There is no branch for `"chunks"`, so `from_upstream.chunks` is never read. Downstream effects reported in #16812: PageIndex/TOC extraction receives flat line-level text instead of structured chapter blocks (incorrect/duplicate/missing chapters), and retrieval quality degrades because chunks are no longer aligned to document structure. ### Fix Select the source list based on `output_format`, mirroring the exact pattern already used in `title_chunker/common.py`: ```python json_result = (from_upstream.chunks if from_upstream.output_format == "chunks" else from_upstream.json_result) or [] ``` `chunks` items share the same dict shape as `json_result` items (both consumed via `.get("text")`, `.get("doc_type_kwd")`, etc.), so they flow through the existing token-sizing path unchanged. One-line change, no behavior change for the `json`/`markdown`/`text`/`html` paths. ### Test Adds `rag/flow/tests/test_token_chunker.py`, an isolated unit test that runs the real `TokenChunker._invoke` (heavy deps stubbed; real pydantic schema used when available) and asserts that with `output_format="chunks"` the upstream `chunks` are consumed rather than the raw parser `json`. Verified RED -> GREEN: the test fails against the current code (reads the raw json) and passes with the fix. Signed-off-by: Yash Raj Pandey <yashpn62@gmail.com>