mirror of
https://github.com/infiniflow/ragflow.git
synced 2026-08-04 23:00:30 +08:00
3e86227a14
## Summary Align Go `splitOversizedUnitWith` with Python `rag/nlp._split_oversized_unit` so the whitespace-atom sub-split produces byte-identical chunk boundaries. ### Root cause of the divergence cl100k token counting is **not additive across whitespace joins** (`token(a)+token(b) != token(a+b)`). Go previously used the exact joined-string fit check `countFn(current+atom) > budget`, while Python accumulates a running sum `current_tokens + a_tokens > budget`. The two formulas disagree by one atom at the boundary, so Go and Python emitted the same chunk *count* but shifted *text*. ### Changes - `splitOversizedUnitWith` (`token.go`): replace the exact joined-string fit check with the running-sum check (mirroring Python's `current_tokens` accumulator), and after a flush keep the overflow whitespace atom (`current += atom`) instead of dropping it. - `token_strict_cap_test.go`: relax `TestMergeByTokenSizeFromJSON_OversizedUnitIsSubSplit` to allow the same cl100k non-additive +1 overshoot Python exhibits (the invariant — an oversized unit is sub-split, not collapsed — is preserved). ### Test plan `bash build.sh --test ./internal/ingestion/component/chunker/...` — green. ## Note Test infrastructure for this change (golden parity harness, `split_oversized_test.go`, `testdata/parity/**`, `known_diffs.json`, `capture_golden.py`/`live_chunk.py`, and the `go-cmp` dependency promotion) is split into a separate, stacked PR #17735 so this PR stays minimal (production code only). This PR is **independent of #17712** (the offline BPE loader). It is based on `upstream/main` and contains only this change; no BPE-loader code is included. Co-authored-by: CodeBuddy <noreply@cnb.cool>