mirror of
https://github.com/infiniflow/ragflow.git
synced 2026-08-01 13:33:48 +08:00
de17dda5f8
## Summary - The `eng` flag passed to `removeTOCWord`/`removeContentsTable` was hardcoded `false`, so English documents used the CJK 3-char TOC prefix instead of Python's 2-word English prefix. This caused English tables of contents to be under-deleted (left in indexed text) or over-deleted (body paragraphs sharing the 3-char prefix dropped). - `eng` is now derived from the parsed item content via `isEnglishItems` (a port of Python `is_english`/`_is_english`), mirroring Python's content heuristic. ## Changes - `internal/parser/parser/text_toc.go`: add `isEnglishItems`/`isEnglishTexts` (ASCII-ratio >80% ⇒ English, fullmatch-anchored regex). - `internal/parser/parser/docx_parser.go`: pass `isEnglishItems(sections)` / `isEnglishItems(lineItems)` to `removeTOCWord` (json + markdown paths). - `internal/parser/parser/html_parser.go`: pass `isEnglishItems(items)` to `removeContentsTable`. - `docs/migration_python_go_diff.md`: close the 1.9/1.10 `eng` residual and resolve the contradictory Parser 2.11 "Partially fixed" note. ## Notes - `remove_toc` is a Parser-stage, DSL-configured feature (`remove_toc: true/false` in the parser family setup). This change only fixes the internal English/CJK prefix decision; it does not alter the pipeline or the DSL contract. `remove_header_footer` (precise match) is unaffected. - `eng` is auto-derived from content rather than exposed as a new config knob, to stay faithful to Python behavior. ## Test plan - `CGO_ENABLED=0 go test ./internal/parser/parser/ -run 'TestRemoveContentsTable|TestIsEnglishTexts|TestRemoveTOCWordEnglishDetection'` - Added `TestIsEnglishTexts` (ASCII-ratio cases) and `TestRemoveTOCWordEnglishDetection` (English TOC no longer over-deletes "Chapter 2 Method"). --------- Co-authored-by: CodeBuddy <noreply@codebuddy.ai>