ragflow

zlei6/ragflow

Fork 0

mirror of https://github.com/infiniflow/ragflow.git synced 2026-08-25 17:42:24 +08:00

Commit Graph

Author	SHA1	Message	Date
Jack	ebd6be09c7	feat(title-chunker): configurable token-count cap with sentence-boundary re-split (#18455 ) ## Summary The TitleChunker (both `hierarchy` and `group` methods) had no token-size ceiling, so a long section without sub-headings became one giant chunk. This adds a configurable `chunk_token_cap` that guarantees every text chunk stays within a token budget.	2026-08-18 21:03:51 +08:00
Jack	423c8489b5	fix(chunker): carry overlap-head PDF positions into new chunk (#18148 ) (#18227 ) When `TokenChunker` starts a fresh chunk with an overlap prefix (Go `computeOverlapPrefix` / Python visible-text cut), the previous chunk's tail PDF coordinates were dropped. As a result, the overlap head of a PDF chunk is displayed but not highlighted — the highlight box is shifted/truncated relative to the displayed span (infiniflow/ragflow#18148).	2026-08-13 22:18:33 +08:00

Author

SHA1

Message

Date

Jack

ebd6be09c7

feat(title-chunker): configurable token-count cap with sentence-boundary re-split (#18455 )

## Summary

The TitleChunker (both `hierarchy` and `group` methods) had no
token-size ceiling, so a long section without sub-headings became one
giant chunk. This adds a configurable `chunk_token_cap` that guarantees
every text chunk stays within a token budget.

2026-08-18 21:03:51 +08:00

Jack

423c8489b5

fix(chunker): carry overlap-head PDF positions into new chunk (#18148 ) (#18227 )

When `TokenChunker` starts a fresh chunk with an overlap prefix (Go `computeOverlapPrefix` / Python visible-text cut), the previous chunk's **tail PDF coordinates were dropped**. As a result, the overlap head of a PDF chunk is displayed but **not highlighted** — the highlight box is shifted/truncated relative to the displayed span (infiniflow/ragflow#18148).

2026-08-13 22:18:33 +08:00

2 Commits