Jack a399b93143 Test(chunker): add golden parity harness, fixtures, and live Go<->Python tool (#17735) ...

Golden-parity test infrastructure for the **Go `TokenChunker` ↔ Python alignment**.
It runs the Go chunker over a committed case set (`testdata/parity/cases/`) and diffs each output against a captured Python golden (`testdata/parity/golden/`), honoring a `known_diffs.json` ratchet (`extra_fields` / `chunk_count` / `chunk_text`) so accepted divergences are tracked rather than silently widening.

2026-08-11 17:46:34 +08:00

..

testdata/parity

Test(chunker): add golden parity harness, fixtures, and live Go<->Python tool (#17735)

2026-08-11 17:46:34 +08:00

tool-py

Test(chunker): add golden parity harness, fixtures, and live Go<->Python tool (#17735)

2026-08-11 17:46:34 +08:00

common.go

Go: fix context and warnings (#18097)

2026-08-11 16:18:49 +08:00

crop_title_test.go

fix: Go ingestion migration batch 5 (Parser 1.1/1.7/2.11, Chunker 1.7/1.8/2.6/2.7, Tokenizer 6x fixes) (#17419)

2026-07-28 11:12:52 +08:00

custom_delim_test.go

Go: fix context and warnings (#18097)

2026-08-11 16:18:49 +08:00

delimiter_case_sensitive_test.go

fix(parser/chunk): dedupe CompileDelimiterPatternList active entries (#17939)

2026-08-07 10:06:04 +08:00

golden_parity_test.go

Test(chunker): add golden parity harness, fixtures, and live Go<->Python tool (#17735)

2026-08-11 17:46:34 +08:00

group_test.go

Go: add context, part5 (#17392)

2026-07-27 10:20:16 +08:00

group.go

Go: fix context and warnings (#18097)

2026-08-11 16:18:49 +08:00

hierarchy_test.go

Go: add context, part5 (#17392)

2026-07-27 10:20:16 +08:00

hierarchy.go

fix: Go ingestion migration batch 5 (Parser 1.1/1.7/2.11, Chunker 1.7/1.8/2.6/2.7, Tokenizer 6x fixes) (#17419)

2026-07-28 11:12:52 +08:00

image_upload_test.go

Feat(ingestion): add canvas pipeline debug (dry-run) mode with View result log (#17538)

2026-07-31 13:10:08 +08:00

image_upload.go

Go: fix context and warnings (#18097)

2026-08-11 16:18:49 +08:00

json_global_merge_test.go

fix(chunker): JSON path merges globally and keeps over-budget items whole (#17739)

2026-08-03 22:17:59 +08:00

live_parity_test.go

Test(chunker): add golden parity harness, fixtures, and live Go<->Python tool (#17735)

2026-08-11 17:46:34 +08:00

one_tag_test.go

fix(chunker): unify TokenChunker merge and strip coord tags in Python JSON path (#18002)

2026-08-07 21:55:07 +08:00

one_test.go

fix: Go ingestion migration batch 5 (Parser 1.1/1.7/2.11, Chunker 1.7/1.8/2.6/2.7, Tokenizer 6x fixes) (#17419)

2026-07-28 11:12:52 +08:00

one.go

fix(chunker): unify TokenChunker merge and strip coord tags in Python JSON path (#18002)

2026-08-07 21:55:07 +08:00

pdfcrop_cgo_test.go

Fix(go): align ingestion pipeline with Python (parser/media dispatch + PDF coordinate chain + Chunker) (#17349)

2026-07-24 21:06:38 +08:00

pdfcrop_cgo.go

Go: add context, part5 (#17392)

2026-07-27 10:20:16 +08:00

pdfcrop_nocgo.go

Feat(ingestion): add canvas pipeline debug (dry-run) mode with View result log (#17538)

2026-07-31 13:10:08 +08:00

presentation_test.go

Go: add context, part5 (#17392)

2026-07-27 10:20:16 +08:00

presentation.go

fix(ingestion): improve Extractor/LLM robustness and Python->Go parity (#17470)

2026-07-28 19:22:18 +08:00

python_parity_test.go

Go: add context, part5 (#17392)

2026-07-27 10:20:16 +08:00

qa_batch2_test.go

fix: Go ingestion migration batch 5 (Parser 1.1/1.7/2.11, Chunker 1.7/1.8/2.6/2.7, Tokenizer 6x fixes) (#17419)

2026-07-28 11:12:52 +08:00

qa_test.go

fix: Go ingestion migration batch 5 (Parser 1.1/1.7/2.11, Chunker 1.7/1.8/2.6/2.7, Tokenizer 6x fixes) (#17419)

2026-07-28 11:12:52 +08:00

qa.go

fix: Go ingestion migration batch 5 (Parser 1.1/1.7/2.11, Chunker 1.7/1.8/2.6/2.7, Tokenizer 6x fixes) (#17419)

2026-07-28 11:12:52 +08:00

register_test.go

…

register.go

refactor[Go]: remove chunker dead code & fix stale/false descriptors (L1/L2/L3) (#17960)

2026-08-07 11:07:45 +08:00

table_test.go

Go: add context, part5 (#17392)

2026-07-27 10:20:16 +08:00

table.go

Go: add context, part5 (#17392)

2026-07-27 10:20:16 +08:00

tag_test.go

fix: Go ingestion migration batch 5 (Parser 1.1/1.7/2.11, Chunker 1.7/1.8/2.6/2.7, Tokenizer 6x fixes) (#17419)

2026-07-28 11:12:52 +08:00

tag.go

fix: Go ingestion migration batch 5 (Parser 1.1/1.7/2.11, Chunker 1.7/1.8/2.6/2.7, Tokenizer 6x fixes) (#17419)

2026-07-28 11:12:52 +08:00

title_test.go

Go: fix context and warnings (#18097)

2026-08-11 16:18:49 +08:00

title.go

fix(ingestion): clamp unconfigured embedding limit; document chunker title parity (comment-only) (#17539)

2026-07-29 21:10:19 +08:00

token_batch1_test.go

fix(chunker): decide token merge on running sum, not re-tokenized join (#17948)

2026-08-07 09:57:01 +08:00

token_children_delim_test.go

refactor(go): clean up TokenChunker delimiter routing and children appliers (#17954)

2026-08-07 11:07:21 +08:00

token_delim_contract_test.go

refactor[Go]: remove chunker dead code & fix stale/false descriptors (L1/L2/L3) (#17960)

2026-08-07 11:07:45 +08:00

token_json_overlap_test.go

fix(chunker): decide token merge on running sum, not re-tokenized join (#17948)

2026-08-07 09:57:01 +08:00

token_merge_parity_test.go

fix(chunker): unify TokenChunker merge and strip coord tags in Python JSON path (#18002)

2026-08-07 21:55:07 +08:00

token_merge_units_test.go

fix(chunker): unify TokenChunker merge and strip coord tags in Python JSON path (#18002)

2026-08-07 21:55:07 +08:00

token_overlap_test.go

fix(chunker): align Go token merge with Python OVER_CAP and delimiter boundary (#17835)

2026-08-05 13:58:45 +08:00

token_oversize_whole_test.go

fix(chunker): unify TokenChunker merge and strip coord tags in Python JSON path (#18002)

2026-08-07 21:55:07 +08:00

token_pdfpos_test.go

fix(chunker): drop dead atom-split path in mergeByTokenSizeFromJSON (#17873)

2026-08-05 19:58:51 +08:00

token_strict_cap_test.go

fix(chunker): unify TokenChunker merge and strip coord tags in Python JSON path (#18002)

2026-08-07 21:55:07 +08:00

token_tag_test.go

refactor(chunker): converge delimiter_mode to {delimiter, one}, drop token_size (#17979)

2026-08-07 16:11:42 +08:00

token_test.go

refactor(chunker): converge delimiter_mode to {delimiter, one}, drop token_size (#17979)

2026-08-07 16:11:42 +08:00

token.go

Go: fix context and warnings (#18097)

2026-08-11 16:18:49 +08:00