## Summary
Closes two Chunker migration gaps documented in
`docs/migration_python_go_diff.md`
(diffs **2.6** and **1.7**), improving parity with the Python ingestion
pipeline.
This is the code portion of commit `261e1fd0b` on branch
`fix/batch-3-4`; the
migration document itself is tracked separately (untracked in this
commit).
### Diff 2.6 — `overlapped_percent` missing Python normalization
Python's `normalize_overlapped_percent` (`common/float_utils.py:50-58`)
accepts a
`[0,1)` fraction (the flow-canvas UI validates `[0,1)`), multiplies it
by 100,
`int()`-truncates, and clamps to `[0,90]`. Go previously only accepted a
raw
`[0,90]` percentage and **rejected** out-of-range input, so a Python
config
passing `0.1` (meaning 10%) silently produced ~0% overlap.
Added `normalizeOverlappedPercent`
(`internal/ingestion/component/chunker/common.go`) mirroring the Python
helper:
- parses numbers and numeric strings (mirrors Python `float()`; bad /
`NaN` / `Inf` → `0`),
- `0 < v < 1 → v *= 100`,
- `int()` truncation,
- clamps to `[0,90]`.
Wired into `tokenChunkerParam.Update` (`token.go`); the merge math
(`token.go:705`, `(100-x)/100`) already matched Python.
`TokenChunkerParam.Validate`
(`schema/chunker.go`) now only guards direct struct construction.
Tests: `TestNormalizeOverlappedPercent`, extended
`TestTokenChunker_NewAcceptsPythonOverlappedRange` (adds fraction/clamp
inputs),
new `TestTokenChunker_NormalizesOverlappedPercent`. The existing
reject-test cases
for `<0` / `>90` were removed because they are now normalized/clamped
(Python parity).
### Diff 1.7 — missing `BULLET_PATTERN` title-level fallback
`resolveTitleLevels` (`title.go`) now applies a 4th-level fallback: when
outline +
regex + layout all yield body level, `bulletsCategory` selects the
best-matching
bullet-pattern group (Chinese legal / numbering / Chinese numbering /
English legal
— mirroring `rag/nlp/__init__.py:258-320`) and assigns structural
levels. Guarded by
`allBodyLevel` so it never overrides an existing outline/regex level.
Tests: `TestResolveTitleLevels_BulletFallback` (4 subtests).
### Incidental test adjustments included in the commit
- `token_batch1_test.go`: overlap input changed `0.3` → `30.0` to
reflect the
post-normalization 30% semantics.
- `real_consumer_test.go`: updated `LoadFromIngestionTask(task)` →
`LoadFromIngestionTask(ctx, task)` for the new context-first signature.
## Verification
`bash build.sh --test ./internal/ingestion/component/...` — chunker +
schema suites
pass, no regression (CGO build).
## Migration doc reference
`docs/migration_python_go_diff.md` §Chunker 1.7 and 2.6 are marked
**Fixed** for
these changes.
## Summary
Three groups of changes across the Go ingestion pipeline:
### 1. DOCX parsing improvements
- **docx_parser.go**: Enhanced DOCX parsing with better structure
extraction and media handling
- **docx_parser_cgo_test.go**, **docx_parser_test.go**: Companion tests
- **office_parsers_no_cgo.go**: Stub sync for non-CGO builds
### 2. Email (.eml) parsing: base64 Content-Transfer-Encoding decoding
- **email_parser.go** (`decodeCTE`): Added Content-Transfer-Encoding
decoding for base64 and quoted-printable. Go's `mime/multipart.Reader`
does not decode Content-Transfer-Encoding automatically, so attachments
with `Content-Transfer-Encoding: base64` remained base64-encoded in the
output. The new `decodeCTE` helper is called after reading each
multipart part's raw bytes in `readMailBody`, mirroring Python's
`part.get_payload(decode=True)`.
- **email_parser_test.go**: Two new tests — simple base64 attachment and
nested multipart/alternative with base64 attachment.
### 3. Extractor LLM driver fix + ModelDriver consolidation
- **extractor.go**: Fixed a bug where the Extractor component used
`ModelFactory.CreateModelDriver()`, which creates bare model instances
without API keys or provider configuration. Switched to
`models.GetPreconfiguredDriver()` which resolves the actual
pre-configured driver from `ProviderManager`, matching the codepath used
by `llm.go`. This fixes auto keyword/question extraction in DSL
pipelines that require LLM calls.
- **get_driver.go** (new): Extracted shared `GetPreconfiguredDriver()`
from `llm.go:newChatModelDriver()` so both `llm.go` and `extractor.go`
use the same codepath.
- **get_driver_test.go** (new): Tests for the shared driver resolution.
- **llm.go**: Replaced inline driver resolution with
`models.GetPreconfiguredDriver()`.
### 4. Chunker fixes and observability
- **group.go** (`extractLineRecords`): Fixed to also read `markdown` and
`html` payload keys — previously it only read `text`/`content`, causing
GroupTitleChunker to silently return empty results for markdown-format
parser output.
- **common.go** (`compileDelimPattern`): Aligned with Python's
`_compile_delimiter_pattern` — only backtick-wrapped delimiters produce
an active regex pattern; plain delimiters are not compiled into the
split regex.
- **token.go** (`applyChildrenDelim`): Set `DocType` and `CKType` to
`"text"` on created ChunkDocs so the token-size merge path correctly
identifies and merges text segments.
- **parser.go**, **extractor.go**, **tokenizer.go**, **group.go**,
**hierarchy.go**: Added debug-level logging for pipeline diagnostics.
- **parser_dispatch_test.go**, **group_test.go**: New tests.
## Verification
- All Go tests pass: `bash build.sh --test ./internal/parser/parser/...`
and `bash build.sh --test ./internal/ingestion/component/...`
- Build succeeds: `bash build.sh --go`
Two refactors on the Go port (agent-go-port):
- Remove the dead per-tenant canvas runtime selector (write-only Redis
scaffolding with no runtime callers) and its dependent metrics/admin
code.
- Move the tokenizer embedding-model id from the shared ingestion
globals into a Tokenizer-scoped setup, and wire the production embedder
resolver in the ingestion task package.
32 files changed, 861 insertions, 1228 deletions.
