Jack 39ba8ae0bb test(chunker): self-contained guard for splitOversizedUnit running-sum flush (#17740) ...

## Summary

Adds a self-contained regression guard for `splitOversizedUnitWith`
after PR #17729
aligned it with Python's `rag/nlp._split_oversized_unit` running-sum
flush.

#17729 shipped a `slack=1` relaxation in `token_strict_cap_test.go` (the
oversized
unit is now sub-split with the same running-sum flush Python uses, which
can leave a
piece one token over the nominal budget due to cl100k non-additivity).
This PR adds
the missing positive proof that the sub-split boundaries are correct, so
the relaxed
assertion is no longer unguarded.

## Changes

- `split_oversized_guard_test.go` (new, self-contained — no harness
loader, no
  `testdata`; the Python oracle is inlined):
- `TestSplitOversizedUnitRunningSumMatchesPython`: asserts the exact
piece
boundaries (live tokenizer) match Python's `_split_oversized_unit`,
compensating
    the `slack=1` relaxation from #17729.
- `TestSplitOversizedUnitDeadTokenizerCollapses`: asserts a
zero-counting tokenizer
collapses the B1 paragraph into exactly one chunk, catching a silently
dead encoder
    that a non-empty-result check would miss.

## Notes

- Test function names are deliberately distinct from PR #17735's
`TestSplitOversizedUnitMatchesPython`, so the two PRs verify
independently and do not
  conflict at merge time.
- PR #17735 (golden parity harness) is intentionally left unchanged per
the split plan.

## Test plan

`bash build.sh --test ./internal/ingestion/component/chunker/` — green,
including both
new tests.

Co-authored-by: CodeBuddy <noreply@cnb.cool>

2026-08-03 22:15:41 +08:00

..

chunker

test(chunker): self-contained guard for splitOversizedUnit running-sum flush (#17740)

2026-08-03 22:15:41 +08:00

globals

Port PR14140 and PR16881 to GO (#17102)

2026-07-21 17:46:23 +08:00

knowledge_compiler

Port Python agentic search to Go (nav service, harness, tools) (#17702)

2026-08-03 11:16:16 +08:00

schema

fix(common,internal): round instead of truncate in normalize_overlapped_percent (closes #17418) (#17452)

2026-07-31 11:51:21 +08:00

dispatch_model.go

Go: fix warnings (#17738)

2026-08-03 21:30:01 +08:00

document_storage.go

Go: add context to storage (#17690)

2026-08-03 15:08:21 +08:00

docx_vision_dispatch_test.go

Go: add context, part12 (#17435)

2026-07-27 19:30:41 +08:00

docx_vision_dispatch.go

Go: add context, part12 (#17435)

2026-07-27 19:30:41 +08:00

extractor_tag_test.go

fix(ingestion): improve Extractor/LLM robustness and Python->Go parity (#17470)

2026-07-28 19:22:18 +08:00

extractor_tag.go

Go: add context to storage (#17690)

2026-08-03 15:08:21 +08:00

extractor_test.go

Go port: doc-level metadata extraction and knowledge compiler (#17536)

2026-07-29 21:06:48 +08:00

extractor.go

Go: add context to redis client (#17689)

2026-08-02 22:50:54 +08:00

file_test.go

Go: add context to storage (#17690)

2026-08-03 15:08:21 +08:00

file.go

Go: fix warnings (#17738)

2026-08-03 21:30:01 +08:00

idempotency.go

Align Go ingestion boundaries with Python (#16647)

2026-07-05 20:43:52 +08:00

image_uploader.go

Go: add context to storage (#17690)

2026-08-03 15:08:21 +08:00

markdown_vision_dispatch.go

Go: add context, part12 (#17435)

2026-07-27 19:30:41 +08:00

media_dispatch_test.go

fix: Go ingestion migration batch 5 (Parser 1.1/1.7/2.11, Chunker 1.7/1.8/2.6/2.7, Tokenizer 6x fixes) (#17419)

2026-07-28 11:12:52 +08:00

media_dispatch.go

fix(ingestion): improve Extractor/LLM robustness and Python->Go parity (#17470)

2026-07-28 19:22:18 +08:00

parser_check_test.go

Go: fix warnings (#17738)

2026-08-03 21:30:01 +08:00

parser_dispatch_pdf_vision_cgo_test.go

Go: add context, part12 (#17435)

2026-07-27 19:30:41 +08:00

parser_dispatch_test.go

Go: fix SSRF (#17641)

2026-07-31 19:15:38 +08:00

parser_dispatch.go

Go: fix warnings (#17738)

2026-08-03 21:30:01 +08:00

parser_test.go

Go: add context to storage (#17690)

2026-08-03 15:08:21 +08:00

parser.go

Go: fix warnings (#17738)

2026-08-03 21:30:01 +08:00

pdf_vision_dispatch_cgo.go

Align Go parser backends and PDF pipeline with Python (#16676)

2026-07-06 19:50:54 +08:00

pdf_vision_dispatch_nocgo.go

Align Go parser backends and PDF pipeline with Python (#16676)

2026-07-06 19:50:54 +08:00

pdf_vision_dispatch_test.go

Fix(go): align ingestion pipeline with Python (parser/media dispatch + PDF coordinate chain + Chunker) (#17349)

2026-07-24 21:06:38 +08:00

pdf_vision_dispatch.go

fix: Go ingestion migration batch 5 (Parser 1.1/1.7/2.11, Chunker 1.7/1.8/2.6/2.7, Tokenizer 6x fixes) (#17419)

2026-07-28 11:12:52 +08:00

slice3_integration_test.go

Go: add context, part5 (#17392)

2026-07-27 10:20:16 +08:00

slice3_test.go

Refactor: refactor dataflow_service.go and guard nats message re-delivery (#16826)

2026-07-13 11:08:04 +08:00

ssrf_test_helper_test.go

Go: fix SSRF (#17641)

2026-07-31 19:15:38 +08:00

tokenizer_nonpersist_test.go

Feat(ingestion): add canvas pipeline debug (dry-run) mode with View result log (#17538)

2026-07-31 13:10:08 +08:00

tokenizer_test.go

Feat(ingestion): add canvas pipeline debug (dry-run) mode with View result log (#17538)

2026-07-31 13:10:08 +08:00

tokenizer_unit_test.go

Feat(ingestion): add canvas pipeline debug (dry-run) mode with View result log (#17538)

2026-07-31 13:10:08 +08:00

tokenizer.go

Go: fix warnings (#17738)

2026-08-03 21:30:01 +08:00