## Summary
`TokenChunker._invoke` discarded the user's configured `delimiters`
whenever no backtick-wrapped delimiter was present: it passed a
hardcoded `""` to `naive_merge`, so the configured delimiters (including
the default `["\n"]`) were never forwarded. `naive_merge` then ignored
the newline sentence boundary and cut chunks **mid-sentence** once the
token budget was exceeded.
## Root cause
`token_chunker.py:326` called `naive_merge(payload, chunk_token_size,
"", overlapped_percent)`. `_compile_delimiter_pattern` intentionally
returns `""` for bare (non-backtick) delimiters — that return value is a
*path selector* (empty → token-budget merge; non-empty → hard
`_split_text_by_pattern` split). The bug was not in that selector but in
the `else` branch, which threw away `self._param.delimiters` instead of
forwarding it.
## Fix
Forward the configured delimiters as a soft boundary:
```python
else naive_merge(
payload,
self._param.chunk_token_size,
"".join(self._param.delimiters),
overlapped_percent,
)
```
`naive_merge` already parses the string via the canonical
`parse_delimiter_field`, so bare and backtick-wrapped delimiters are
honored as soft boundaries while the token budget is still respected.
The path-selection role of `_compile_delimiter_pattern` is untouched:
backtick-wrapped delimiters still select the hard
`_split_text_by_pattern` path; bare delimiters still take the
token-budget merge path. No regression for any previously-working
(wrapped-delimiter) configuration.
## Test
Adds `rag/flow/tests/test_token_chunker_delimiter.py`:
- `test_token_chunker_token_size_mode_does_not_split_sentences` — fails
on the old code (3 sentences cut mid-stream), passes after the fix.
- `test_naive_merge_empty_delimiter_ignores_newline_break` — root-cause
companion asserting `naive_merge("")` cuts while `naive_merge("\n")`
preserves boundaries.
Both tests skip when the tokenizer is unavailable (dead-tokenizer
guard).
## Note
This branch contains only this one-line fix on top of `main`; it is
intentionally independent of the unrelated tokenizer/offline-BPE work.
Co-authored-by: CodeBuddy <noreply@codebuddy.ai>
Fixes#16812
### Problem
In the `rag/flow` ingestion pipeline, when `TitleChunker` feeds
`TokenChunker`, the chapter-aware chunks are silently discarded and the
parser's raw flat json is re-chunked instead.
`TitleChunker` emits `output_format="chunks"` and writes its
chapter-aware output to the `chunks` field
(`rag/flow/chunker/title_chunker/common.py`,
`set_output("output_format", "chunks")`). But `TokenChunker._invoke`
only handles `output_format` in `["markdown", "text", "html"]`, then
falls through to the `# json` path which reads
`from_upstream.json_result`. There is no branch for `"chunks"`, so
`from_upstream.chunks` is never read.
Downstream effects reported in #16812: PageIndex/TOC extraction receives
flat line-level text instead of structured chapter blocks
(incorrect/duplicate/missing chapters), and retrieval quality degrades
because chunks are no longer aligned to document structure.
### Fix
Select the source list based on `output_format`, mirroring the exact
pattern already used in `title_chunker/common.py`:
```python
json_result = (from_upstream.chunks if from_upstream.output_format == "chunks" else from_upstream.json_result) or []
```
`chunks` items share the same dict shape as `json_result` items (both
consumed via `.get("text")`, `.get("doc_type_kwd")`, etc.), so they flow
through the existing token-sizing path unchanged. One-line change, no
behavior change for the `json`/`markdown`/`text`/`html` paths.
### Test
Adds `rag/flow/tests/test_token_chunker.py`, an isolated unit test that
runs the real `TokenChunker._invoke` (heavy deps stubbed; real pydantic
schema used when available) and asserts that with
`output_format="chunks"` the upstream `chunks` are consumed rather than
the raw parser `json`.
Verified RED -> GREEN: the test fails against the current code (reads
the raw json) and passes with the fix.
Signed-off-by: Yash Raj Pandey <yashpn62@gmail.com>
### What problem does this PR solve?
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
- [x] Refactoring
---------
Co-authored-by: Zhichang Yu <yuzhichang@gmail.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
### What problem does this PR solve?
Dataflow support audio. And fix giteeAI's sequence2text model.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
- [x] New Feature (non-breaking change which adds functionality)
### What problem does this PR solve?
support parse image by OCR or VLM.
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
### What problem does this PR solve?
Dataflow supports markdown.
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
---------
Co-authored-by: Kevin Hu <kevinhu.sh@gmail.com>
### What problem does this PR solve?
Dataflow supports Spreadsheet and Word processor document
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
### What problem does this PR solve?
Refine dataflow and initialize dataflow app.
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
