Jack 39ba8ae0bb test(chunker): self-contained guard for splitOversizedUnit running-sum flush (#17740) ...

## Summary

Adds a self-contained regression guard for `splitOversizedUnitWith`
after PR #17729
aligned it with Python's `rag/nlp._split_oversized_unit` running-sum
flush.

#17729 shipped a `slack=1` relaxation in `token_strict_cap_test.go` (the
oversized
unit is now sub-split with the same running-sum flush Python uses, which
can leave a
piece one token over the nominal budget due to cl100k non-additivity).
This PR adds
the missing positive proof that the sub-split boundaries are correct, so
the relaxed
assertion is no longer unguarded.

## Changes

- `split_oversized_guard_test.go` (new, self-contained — no harness
loader, no
  `testdata`; the Python oracle is inlined):
- `TestSplitOversizedUnitRunningSumMatchesPython`: asserts the exact
piece
boundaries (live tokenizer) match Python's `_split_oversized_unit`,
compensating
    the `slack=1` relaxation from #17729.
- `TestSplitOversizedUnitDeadTokenizerCollapses`: asserts a
zero-counting tokenizer
collapses the B1 paragraph into exactly one chunk, catching a silently
dead encoder
    that a non-empty-result check would miss.

## Notes

- Test function names are deliberately distinct from PR #17735's
`TestSplitOversizedUnitMatchesPython`, so the two PRs verify
independently and do not
  conflict at merge time.
- PR #17735 (golden parity harness) is intentionally left unchanged per
the split plan.

## Test plan

`bash build.sh --test ./internal/ingestion/component/chunker/` — green,
including both
new tests.

Co-authored-by: CodeBuddy <noreply@cnb.cool>

2026-08-03 22:15:41 +08:00

..

common.go

refactor(nlp): collapse 6 delimiter-parser implementations into one canonical helper (#17383) (#17387)

2026-08-02 17:07:14 +08:00

crop_title_test.go

fix: Go ingestion migration batch 5 (Parser 1.1/1.7/2.11, Chunker 1.7/1.8/2.6/2.7, Tokenizer 6x fixes) (#17419)

2026-07-28 11:12:52 +08:00

delimiter_case_sensitive_test.go

refactor(nlp): collapse 6 delimiter-parser implementations into one canonical helper (#17383) (#17387)

2026-08-02 17:07:14 +08:00

group_test.go

Go: add context, part5 (#17392)

2026-07-27 10:20:16 +08:00

group.go

Go: fix warnings (#17738)

2026-08-03 21:30:01 +08:00

hierarchy_test.go

Go: add context, part5 (#17392)

2026-07-27 10:20:16 +08:00

hierarchy.go

fix: Go ingestion migration batch 5 (Parser 1.1/1.7/2.11, Chunker 1.7/1.8/2.6/2.7, Tokenizer 6x fixes) (#17419)

2026-07-28 11:12:52 +08:00

image_upload_test.go

Feat(ingestion): add canvas pipeline debug (dry-run) mode with View result log (#17538)

2026-07-31 13:10:08 +08:00

image_upload.go

Feat(ingestion): align image to MinIO upload, unify chunk-id computation and add PPT parsing support (#17111)

2026-07-20 19:33:51 +08:00

image.go

Implement builtin chunk method as ingestion pipeline in GO (#16822)

2026-07-13 13:51:40 +08:00

one_test.go

fix: Go ingestion migration batch 5 (Parser 1.1/1.7/2.11, Chunker 1.7/1.8/2.6/2.7, Tokenizer 6x fixes) (#17419)

2026-07-28 11:12:52 +08:00

one.go

fix: Go ingestion migration batch 5 (Parser 1.1/1.7/2.11, Chunker 1.7/1.8/2.6/2.7, Tokenizer 6x fixes) (#17419)

2026-07-28 11:12:52 +08:00

pdfcrop_cgo_test.go

Fix(go): align ingestion pipeline with Python (parser/media dispatch + PDF coordinate chain + Chunker) (#17349)

2026-07-24 21:06:38 +08:00

pdfcrop_cgo.go

Go: add context, part5 (#17392)

2026-07-27 10:20:16 +08:00

pdfcrop_nocgo.go

Feat(ingestion): add canvas pipeline debug (dry-run) mode with View result log (#17538)

2026-07-31 13:10:08 +08:00

presentation_test.go

Go: add context, part5 (#17392)

2026-07-27 10:20:16 +08:00

presentation.go

fix(ingestion): improve Extractor/LLM robustness and Python->Go parity (#17470)

2026-07-28 19:22:18 +08:00

python_parity_test.go

Go: add context, part5 (#17392)

2026-07-27 10:20:16 +08:00

qa_batch2_test.go

fix: Go ingestion migration batch 5 (Parser 1.1/1.7/2.11, Chunker 1.7/1.8/2.6/2.7, Tokenizer 6x fixes) (#17419)

2026-07-28 11:12:52 +08:00

qa_test.go

fix: Go ingestion migration batch 5 (Parser 1.1/1.7/2.11, Chunker 1.7/1.8/2.6/2.7, Tokenizer 6x fixes) (#17419)

2026-07-28 11:12:52 +08:00

qa.go

fix: Go ingestion migration batch 5 (Parser 1.1/1.7/2.11, Chunker 1.7/1.8/2.6/2.7, Tokenizer 6x fixes) (#17419)

2026-07-28 11:12:52 +08:00

register_test.go

Implement builtin chunk method as ingestion pipeline in GO (#16822)

2026-07-13 13:51:40 +08:00

register.go

Feat(ingestion): add canvas pipeline debug (dry-run) mode with View result log (#17538)

2026-07-31 13:10:08 +08:00

split_oversized_guard_test.go

test(chunker): self-contained guard for splitOversizedUnit running-sum flush (#17740)

2026-08-03 22:15:41 +08:00

table_test.go

Go: add context, part5 (#17392)

2026-07-27 10:20:16 +08:00

table.go

Go: add context, part5 (#17392)

2026-07-27 10:20:16 +08:00

tag_test.go

fix: Go ingestion migration batch 5 (Parser 1.1/1.7/2.11, Chunker 1.7/1.8/2.6/2.7, Tokenizer 6x fixes) (#17419)

2026-07-28 11:12:52 +08:00

tag.go

fix: Go ingestion migration batch 5 (Parser 1.1/1.7/2.11, Chunker 1.7/1.8/2.6/2.7, Tokenizer 6x fixes) (#17419)

2026-07-28 11:12:52 +08:00

title_test.go

Chunker: port Python overlapped_percent normalization and BULLET_PATTERN title fallback (#17396)

2026-07-27 14:42:44 +08:00

title.go

fix(ingestion): clamp unconfigured embedding limit; document chunker title parity (comment-only) (#17539)

2026-07-29 21:10:19 +08:00

token_batch1_test.go

fix(chunker): enforce strict chunk_token_num cap on .txt / PDF / email paths (#17203)

2026-08-01 22:48:51 +08:00

token_pdfpos_test.go

Fix(go): align ingestion pipeline with Python (parser/media dispatch + PDF coordinate chain + Chunker) (#17349)

2026-07-24 21:06:38 +08:00

token_strict_cap_test.go

fix(chunker): align splitOversizedUnitWith with Python running-sum flush (#17729)

2026-08-03 17:48:39 +08:00

token_tag_test.go

fix: Go ingestion migration batch 5 (Parser 1.1/1.7/2.11, Chunker 1.7/1.8/2.6/2.7, Tokenizer 6x fixes) (#17419)

2026-07-28 11:12:52 +08:00

token_test.go

fix(common,internal): round instead of truncate in normalize_overlapped_percent (closes #17418) (#17452)

2026-07-31 11:51:21 +08:00

token.go

fix(chunker): align splitOversizedUnitWith with Python running-sum flush (#17729)

2026-08-03 17:48:39 +08:00