mirror of
https://github.com/infiniflow/ragflow.git
synced 2026-08-08 16:38:01 +08:00
109b74e410
## Summary - Remove `splitOversizedUnit`, `splitAtomByTokenBudget` and `atomRE` from `internal/ingestion/component/chunker/token.go`. - Delete `split_oversized_guard_test.go` (added by #17740), which guarded the removed atom-split behaviour. - Drop the now-unused `wordCount`/`charCount` helpers from `token_strict_cap_test.go`. - Add `TestMergeByTokenSize_OversizedUnitStaysWhole` to pin the #17799 contract invariant (over-budget unit stays whole, never atom-split) on the **text path**. The JSON path is already covered by `TestMergeByTokenSizeFromJSON_OversizedUnitStaysWhole`. ## Why The production merge path (`mergeByTokenSize` / `mergeByTokenSizeFromJSON`) keeps over-budget units whole and relies on the embedding/rerank layer to truncate them, per the TokenChunker contract (#17799: remove atom-split, no hard_cap). The deleted helpers implemented the opposite behaviour and had **no production caller**, so they contradicted the contract and misled readers into thinking atom-split was active. ## Parser vs chunker layering Python's `_split_oversized_unit` lives at the **parser layer** (pre-split before `naive_merge`), not in the chunker. Go's parser backends are currently skeletons, so there is no parser-side equivalent yet; if added later it belongs in `internal/parser/parser/*`, not the chunker. ## Test plan `bash build.sh --test ./internal/ingestion/component/chunker/...` passes; the new text-path test passes and the orphaned atom-split tests are gone. ## Changes - 3 files changed, 32 insertions(+), 250 deletions(-)