Crystora b633ff0435 feat(go): drop empty and duplicate chunks in postprocess filter (#16049) ...

### What problem does this PR solve?

The Go chunk pipeline's `PostprocessOperator` `filter` stage
(`internal/ingestion/chunk/postprocess.go`) only filtered by length
(`min_length`/`max_length`). It could not drop empty/whitespace-only
chunks or duplicate chunks — both standard RAG post-processing steps
(blank chunks shouldn't be indexed; identical chunks waste embedding
compute and add redundant retrieval results).

This adds two optional, default-off booleans to the `filter` config:

- `drop_empty` — drop chunks whose content is empty or whitespace-only.
- `drop_duplicates` — drop chunks whose exact content already appeared
(order-preserving; the first occurrence is kept).

They compose with the existing length bounds and are reflected in
`String()` for plan explainability. Also adds the first unit tests for
the postprocess filter (length bounds, drop_empty, drop_duplicates,
combined, exact-content matching, and config parsing).

Validation: `gofmt` clean, `go vet ./internal/ingestion/chunk/` clean,
`go build` ok, `go test ./internal/ingestion/chunk/` — all tests pass.

Closes #16048

### Type of change

- [x] New Feature (non-breaking change which adds functionality)

Co-authored-by: Ling Qin <qinling0210@163.com>

2026-07-06 14:14:31 +08:00

..

chunk

feat(go): drop empty and duplicate chunks in postprocess filter (#16049)

2026-07-06 14:14:31 +08:00

parser

Align Go ingestion boundaries with Python (#16647)

2026-07-05 20:43:52 +08:00