Crystora 4effc242fd feat(go): add length split strategy with overlap to chunk pipeline (#16047) ...

### What problem does this PR solve?

The Go ingestion chunk pipeline's `SplitOperator`
(`internal/ingestion/chunk/split.go`) supported only `sentence`, `char`,
and `paragraph` strategies, but not **fixed-size (length) chunking with
overlap** — the canonical RAG strategy for bounding chunk length while
preserving cross-boundary context.

This adds a `length` strategy alongside the existing ones, configurable
via DSL `params`:

- `chunk_size` — target window size in **runes** (rune-aware:
multi-byte/CJK text is windowed by character, never split mid-rune).
- `overlap` — runes carried from the end of each window into the next.

The window advances by `chunk_size - overlap`. `chunk_size` falls back
to a default (256) when unset/non-positive, and `overlap` is clamped to
`[0, chunk_size-1]` so the window always advances and the operation
terminates. Implementation follows the existing
`splitByChar`/`splitByParagraph` pattern and reuses `DetectLanguage` for
chunk metadata.

It also adds `split_test.go` — the first unit tests for the `chunk`
package — covering basic windowing, overlap, overlap
clamping/termination, rune-awareness (CJK), default sizing, no-overlap
reconstruction, empty input, and DSL param parsing.

Validation: `gofmt` clean, `go vet ./internal/ingestion/chunk/` clean,
`go build` ok, `go test ./internal/ingestion/chunk/` — all tests pass.

Closes #16046

### Type of change

- [x] New Feature (non-breaking change which adds functionality)

Co-authored-by: bittoby <218712309+bittoby@users.noreply.github.com>

2026-07-06 14:14:21 +08:00

..

chunk

feat(go): add length split strategy with overlap to chunk pipeline (#16047)

2026-07-06 14:14:21 +08:00

parser

Align Go ingestion boundaries with Python (#16647)

2026-07-05 20:43:52 +08:00