Ports the dataset knowledge compilation (wiki/graph/tree/mindmap) to the
Go scheduler with a status contract, aligns wiki storage/retrieval with
Python, and sizes prompts by content_length.
The Go `TokenChunker` text/markdown path (`mergeByTokenSize`)
unconditionally
called `splitOversizedUnit` on any unit that exceeded
`chunk_token_size`,
emitting Go-only sub-chunks. Python's `naive_merge`
(`_merge_paragraph_groups`,
`rag/nlp/__init__.py`) never atom-splits an oversize unit under either
`OVER_CAP` or `UNDER_CAP`: a paragraph larger than the budget becomes
its own
standalone chunk and the model layer truncates it later.
This aligns the text/markdown path with the **structured JSON path**
(`invokeJSONPayload` → `mergeByTokenSizeFromJSON(...,
subSplitOversize=false)`,
#17739). It completes the OVER_CAP alignment started in #17835.
Go's `TokenChunker` kept the captured delimiter glued to the preceding
segment on **both** the primary (`chunkFromItem`) and secondary
(`children_delimiters`) split paths, while Python's reference
`token_chunker` drops it via `_split_text_by_pattern`
(`token_chunker.py:79-93`, used by both `_build_json_chunks` and
`_split_chunk_docs_by_children`). The divergence leaked the delimiter
into every emitted chunk's `text`.
## Summary
Follow-up to #17835 (merged). The OVER_CAP / UNDER_CAP merge strategy
was threaded through `mergeDecision` and `mergeByTokenSizeFromJSON` as
an inlined `allowBoundaryOverflow bool` derived from `!c.param.UnderCap`
at three call sites. This replaces that with a named
`schema.MergeStrategy` enum.
## Why
- The `!c.param.UnderCap` inversion was hand-written in three places, so
a future strategy addition could silently drift between the JSON path
(`invokeTextPayload` / `invokeJSONPayload`) and the text path
(`mergeByTokenSize`) — no compile error, and the existing tests don't
cover all three sites with both strategies.
- The strategy concept was never named; `allowBoundaryOverflow` (true =
OVER_CAP) is a double-negation of `UnderCap` and reads opaquely at the
5th positional argument.
## What changed
- Add `schema.MergeStrategy` (`MergeOverCap` / `MergeUnderCap`)
mirroring Python's `rag/nlp/__init__.py` `MergeStrategy`, so Go and
Python stay on the same vocabulary.
- Expose `TokenChunkerParam.MergeStrategy()` derived from the
wire-facing `UnderCap bool` (existing `"under_cap"` configs keep working
— no schema break).
- `mergeDecision` and `mergeByTokenSizeFromJSON` now take
`schema.MergeStrategy` instead of `allowBoundaryOverflow bool`; the
three call sites pass `c.param.MergeStrategy()` (no `!`).
- Tests updated to pass the enum; added a guard test for the `UnderCap`
-> `MergeStrategy` mapping and an end-to-end test for UNDER_CAP on the
JSON path.
No behavior change: default remains OVER_CAP, `under_cap=true` still
selects UNDER_CAP.
## Test plan
`bash build.sh --test ./internal/ingestion/component/chunker/...
./internal/ingestion/component/schema/...` — all green, including
`TestMergeByTokenSizeFromJSON_UnderCapNoOverflow`,
`TestMergeByTokenSize_UnderCapNoOverflow`,
`TestInvokeJSONPayload_UnderCapEndToEnd`, and
`TestTokenChunkerParamMergeStrategy`.
## Related issues
- Relates to #17835 — wired UNDER_CAP as a tested merge-strategy seam
(merged)
- Relates to #17799 — contract doc for token-chunker cap/delimiter
alignment
- Relates to #17808 — related chunker alignment work
---------
Co-authored-by: CodeBuddy <noreply@cnb.cool>
## Problem
During ingestion, `indexdoc.ProcessChunksForPipeline` stamped
`ck["kb_id"]`
on every chunk. This was both:
- **a dead write** — `elasticsearch.InsertChunks` unconditionally
overwrites
the value with `datasetID` (`chunk.go:211`), so the producer's value
never
reached the index;
- **the wrong shape** — it was emitted as `[]string`, while both engines
actually need a single string.
This is the `kb_id` slice of the ingestion -> engine schema leak tracked
in
#17371: ingestion was carrying index-physical schema knowledge it should
not
own.
## Fix
Make the search engines the single owner of `kb_id` at the write
boundary,
and stop ingestion from emitting it:
- **Elasticsearch** (`chunk.go:211`) already sets `docCopy["kb_id"] =
datasetID`
— unchanged.
- **Infinity** (`chunk.go`) `InsertChunks` now stamps
`insertChunks[i]["kb_id"] = datasetID` right after
`transformChunkFields`
(previously it only *read/normalized* the producer value, which forced
ingestion to supply it). Both engines are now consistent.
- `ProcessChunksForPipeline` no longer stamps `kb_id` and the now-leaky
`kbID` parameter is removed. The same removal is propagated to
`ProcessPipelineOutputForGolden` and the `compare_pipeline_golden` dev
tool
(its `-kb-id` flag is dropped).
The stored `kb_id` value is byte-for-byte unchanged: `datasetID` passed
to
`InsertChunks` is `taskCtx.Doc.KbID`, i.e. the same id that was
previously
set on the producer chunk.
## Verification
- `bash build.sh --test ./internal/ingestion/task/indexdoc/...
./internal/engine/infinity/...`
— both green.
- `internal/ingestion/task` has **two pre-existing** failures
(`TestPipelineExecutor_Run_RealCanvasDSL_UsesGeneralPipeline`,
`TestRunPipeline_RealPipelineOutput_ProducesIndexFields`) that assert
`inserted chunk count = 1, want 2` — a parser/assertion mismatch (the Go
parser merges the 2-paragraph fixture into 1 chunk). They are unrelated
to
this change, which never touches chunk counting. The `kb_id`-related
test
failure this change would otherwise introduce is fixed by updating the
tests
below.
- Updated the pinning unit test:
`TestProcessChunksForPipeline_SetsDocID`
(formerly `...SetsDocIDAndKBID`) now asserts `kb_id` is **not** set by
the
producer. Removed the `kb_id` assertion and the now-dead
`taskChunkFieldEqualsStr` helper from
`pipeline_real_integration_test.go`.
## Scope
This closes only the `kb_id` portion of #17371. The remaining
index-physical
fields (`docnm_kwd`, `create_timestamp_flt`, `page_num_int`/`top_int`/
`position_int`, etc.) are intentionally left for a follow-up (P2).
## Summary
- `ProcessChunksForPipeline` now sets `kb_id` to a plain string instead
of `[]string{kbID}`, removing an index-physical array shape from the
ingestion domain.
- Stored documents are byte-identical: Elasticsearch overrides `kb_id`
with `datasetID` on write, and Infinity's `transformChunkFields` already
accepts a plain string.
- Infinity is intentionally left unchanged — `service/chunk` paths still
feed `kb_id` as `[]string`, and Infinity handles both forms. The
`dataset` artifact merge (`dataset_artifact_service.go`) is out of scope
for this step.
- Unit assertion updated to expect a string.
## Scope / non-goals
This is the smallest first step (T1) of the index-schema leak cleanup
tracked in #17371. It does **not** move the other leaks (`docnm_kwd`,
`create_timestamp_flt`, position ints) to the engine boundary — those
are later steps behind a read-back golden test.
## Test plan
- `go test ./internal/ingestion/task/indexdoc/...` passes.
- The two `task` "Real" integration tests fail identically on a clean
tree (environment lacks real embedding/parsing); they are pre-existing,
unrelated to this change.
🤖 Generated with [CodeBuddy Code](https://cnb.cool/codebuddy)
## Summary
Extract the pipeline-output → search-engine index document mapping
helpers out of the `task` package into a dedicated, dependency-light
leaf package `internal/ingestion/task/indexdoc`.
These functions are pure transforms (they only depend on
`common`/`utility`) and are not task-orchestration concerns:
- `NormalizeChunks`, `DeepCopyChunks` (was unexported `deepCopyChunks`),
`toChunkMaps` → `indexdoc/normalize.go`
- `ProcessChunksForPipeline`, `RenameTextToContentWithWeight`,
`GetEmbeddingTokenConsumption`, `cleanupConsumedChunkFields`,
`mergeChunkMetadata`, `processChunkPositions`,
`AggregateTableDocMetadata`, `resolveTableColumnConfig` →
`indexdoc/process.go`
- `AddPositions` → `indexdoc/position.go`
- `EmbeddingTokenConsumptionKey` constant → `indexdoc/constants.go`
(task/constants.go keeps only `GRAPH_RAPTOR_FAKE_DOC_ID`)
Call sites in `pipeline_executor.go` and `golden_compare.go` now
reference the `indexdoc` package; package-task tests qualify the moved
symbols.
## Why
The `task` package had grown into a "orchestration + pure mapping +
debug" mix. Splitting the pure mapping helpers into a leaf package
sharpens package boundaries, removes a misleading top-level
`ingestion/chunk` candidate (there are already `parser/chunk` and
`service/chunk`), and lets the golden tool / future reuse pull in the
mapping logic without dragging in `task`'s `dao`/`engine`/`service`
dependency graph (Go subpackage import does not pull in the parent).
## Test plan
- `build.sh --test ./internal/ingestion/task/...` — **green** (task
4.7s, indexdoc 0.007s), matching the pre-change baseline.
- `gofmt` clean; `build.sh` builds both `ragflow-cli` and
`ragflow_server` successfully.
- Integration/E2E tiers are delegated to CI (need real MySQL/MinIO/ES
services).
Note: `pipeline_e2e_test.go` has a **pre-existing** compile error
(`server.ElasticsearchConfig` / `server.InfinityConfig` are now defined
under `internal/server/config/`, not re-exported by `internal/server`).
This is unrelated to this change — the diff to that file is only the
added `indexdoc` import and the qualified `EmbeddingTokenConsumptionKey`
reference.
## Summary
Adds a self-contained regression guard for `splitOversizedUnitWith`
after PR #17729
aligned it with Python's `rag/nlp._split_oversized_unit` running-sum
flush.
#17729 shipped a `slack=1` relaxation in `token_strict_cap_test.go` (the
oversized
unit is now sub-split with the same running-sum flush Python uses, which
can leave a
piece one token over the nominal budget due to cl100k non-additivity).
This PR adds
the missing positive proof that the sub-split boundaries are correct, so
the relaxed
assertion is no longer unguarded.
## Changes
- `split_oversized_guard_test.go` (new, self-contained — no harness
loader, no
`testdata`; the Python oracle is inlined):
- `TestSplitOversizedUnitRunningSumMatchesPython`: asserts the exact
piece
boundaries (live tokenizer) match Python's `_split_oversized_unit`,
compensating
the `slack=1` relaxation from #17729.
- `TestSplitOversizedUnitDeadTokenizerCollapses`: asserts a
zero-counting tokenizer
collapses the B1 paragraph into exactly one chunk, catching a silently
dead encoder
that a non-empty-result check would miss.
## Notes
- Test function names are deliberately distinct from PR #17735's
`TestSplitOversizedUnitMatchesPython`, so the two PRs verify
independently and do not
conflict at merge time.
- PR #17735 (golden parity harness) is intentionally left unchanged per
the split plan.
## Test plan
`bash build.sh --test ./internal/ingestion/component/chunker/` — green,
including both
new tests.
Co-authored-by: CodeBuddy <noreply@cnb.cool>
## Summary
Align Go `splitOversizedUnitWith` with Python
`rag/nlp._split_oversized_unit` so the whitespace-atom sub-split
produces byte-identical chunk boundaries.
### Root cause of the divergence
cl100k token counting is **not additive across whitespace joins**
(`token(a)+token(b) != token(a+b)`). Go previously used the exact
joined-string fit check `countFn(current+atom) > budget`, while Python
accumulates a running sum `current_tokens + a_tokens > budget`. The two
formulas disagree by one atom at the boundary, so Go and Python emitted
the same chunk *count* but shifted *text*.
### Changes
- `splitOversizedUnitWith` (`token.go`): replace the exact joined-string
fit check with the running-sum check (mirroring Python's
`current_tokens` accumulator), and after a flush keep the overflow
whitespace atom (`current += atom`) instead of dropping it.
- `token_strict_cap_test.go`: relax
`TestMergeByTokenSizeFromJSON_OversizedUnitIsSubSplit` to allow the same
cl100k non-additive +1 overshoot Python exhibits (the invariant — an
oversized unit is sub-split, not collapsed — is preserved).
### Test plan
`bash build.sh --test ./internal/ingestion/component/chunker/...` —
green.
## Note
Test infrastructure for this change (golden parity harness,
`split_oversized_test.go`, `testdata/parity/**`, `known_diffs.json`,
`capture_golden.py`/`live_chunk.py`, and the `go-cmp` dependency
promotion) is split into a separate, stacked PR #17735 so this PR stays
minimal (production code only).
This PR is **independent of #17712** (the offline BPE loader). It is
based on `upstream/main` and contains only this change; no BPE-loader
code is included.
Co-authored-by: CodeBuddy <noreply@cnb.cool>
Port Python rag/advanced_rag agentic search to Go: ES-backed dataset-nav
service, agentic-search harness, and agent tools.
Includes agentic-search port plan and self-review docs.
## Summary
Six sites used to read the same `parser_config.delimiter` field with
divergent grammars:
- `rag.nlp.get_delimiters` (PDF/DOCX/HTML/EPUB/JSON/CSV/XLSX/email/book)
- `rag.nlp.naive_merge` (custom-delimiter branch)
- `rag.nlp.naive_merge_with_images`
- `rag.nlp._build_cks`
- `deepdoc.parser.txt_parser.parser_txt` (.txt, code)
-
`deepdoc.parser.markdown_parser.MarkdownElementExtractor.get_delimiters`
The six implementations disagreed on bare-vs-wrapped chars, dedupe, sort
order, CRLF normalization, and `re.I` (#17384). The shipped default ``
`\n!?;。；！？` `` was a no-op for `.md` because the markdown path only
matched backtick-wrapped tokens.
## Changes
- **new:** `rag/nlp/delim.py` with `parse_delimiter_field` and
`compile_delimiter_pattern`. Single source of truth. CRLF normalization
at the top; longest-first stable sort; insertion-ordered dedupe; no
`re.I`.
- **refactor:** all six call sites delegate to the helper.
- `rag/nlp/__init__.py::get_delimiters` becomes a thin shim.
- `deepdoc/parser/txt_parser.py::parser_txt` drops the
`[encode/decode/unicode_escape]` round-trip.
- `deepdoc/parser/markdown_parser.py::get_delimiters` honors bare chars
(fixes [1]).
- **tests:** `test/unit_test/rag/test_delim.py` (85 tests) — helper,
acceptance table, frontend parity, static guard against re-inlining.
- **tests:** `test/unit_test/rag/test_delimiter_case_sensitive.py` (from
#17386) updated to retarget the static check at the new helper +
AST-based broader scan.
## Acceptance criteria
- All six sites produce the same regex pattern for the same input.
- Shipped default keeps working for `.txt` / `.pdf` / `.docx`.
- Shipped default for `.md` now splits (was a silent no-op).
- Tooltip example `` `\n##;` `` produces three effective delimiters
regardless of file type.
- Bare whitespace inputs split on every occurrence.
- Backtick-wrapped whitespace splits only on the exact N-char sequence.
- CRLF-line-ending documents split identically to LF-line-ending
documents.
- 123 tests pass (85 new + 38 existing).
## Rebase protocol
As #17385 and #17386 evolve, this branch will be rebased on top. The
only overlap between this PR's diff and the other two is
`test_delimiter_case_sensitive.py`, where #17383 modifies the static
check to point at the new helper location.
---------
Co-authored-by: kiloconnect[bot] <240665456+kiloconnect[bot]@users.noreply.github.com>
## Summary
This PR refactors the Go knowledge-compilation ingestion pipeline
(`internal/ingestion/knowledge_compile` +
`internal/ingestion/component/knowledge_compiler`) with three related
changes:
- **Token-budget batching for LLM merge decisions.**
`LLMMergeDecider.DecideBatch` previously stuffed every `(existing,
candidate)` pair into a single LLM call, risking `max_token` overflow.
It now splits pairs into token-bounded sub-batches (budget =
`llmMaxTokens * 0.85`) via `tokenizer.NumTokensFromString`, runs them
concurrently while preserving the global pair index, and never
reindexes.
- **Process-level global compile pool.** Introduces a single vCPU-sized
goroutine pool (`pool.go`, env `KC_COMPILE_CONCURRENCY`) dedicated to
*all* knowledge-compilation stages. KNN search loop, `DecideBatch`
sub-batches, `WriteMerged`/`DeleteMerged` internals, and the
component-level (structure/mindmap) per-call pools are all unified into
it via an injected submitter. No more per-job short-lived goroutines in
`runCompilerJobs` (futures are collected then awaited on the caller).
Fan-out stays bounded by the pool worker count; these stages are
docengine-bounded / LLM-bounded, not CPU-bounded.
- **DocEngine-only deletion.** `Consumer.processBatch` deletion no
longer loads the deleted docs' products into memory. Two sequential
DocEngine calls replace the old in-memory surgery:
- `DeleteDocLevelForDocs` — one `DeleteChunks` over `doc_id IN
deletedDocIDs` (merged rows carry `doc_id == kb`, so only per-doc
products match).
- `StripMergedSources` — one `Search` of `kc_merged=1` rows filtered by
`source_doc_ids IN deletedDocIDs` (intersection pushed down to the
engine), `UpdateChunks` the source array of survivors, and
`DeleteChunks` the rows whose array became empty.
## Changes
- `internal/ingestion/knowledge_compile/pool.go` (new): global
`compilerPool` +
`runCompilerJobs`/`SubmitCompilerJob`/`SubmitCompilerJobs`.
- `internal/ingestion/knowledge_compile/consumer.go`: deletion rewritten
to the two DocEngine calls;
`mergedBase`/`toDelete`/`stripDeletedSources` removed.
- `internal/ingestion/knowledge_compile/writer.go`:
`DeleteDocLevelForDocs` + `StripMergedSources` replace
`DeleteMergedForDoc`/`DeleteMerged`.
- `internal/ingestion/knowledge_compile/reader.go`: drop
`LoadMergedBySourceDoc` + `containsString` (keep `LoadDocProducts` for
the completion branch).
- `internal/ingestion/knowledge_compile/dedup.go`: `NewLLMDeduper` takes
`llmMaxTokens`; wires `SetMaxBatchTokens`/`SetSubmitter`.
- `internal/ingestion/knowledge_compiler/{structure,merge}.go`,
`mindmap/mindmap.go`, `pool_wiring.go`: token-budget split + submitter
injection.
- Tests: `structure_test.go` (token-budget split), `dedup_test.go`,
`consumer_test.go` (tombstone + DocEngine deletion assertions) updated.
## Validation
`bash build.sh --test -race ./internal/ingestion/knowledge_compile/...
./internal/ingestion/component/knowledge_compiler/...` passes (unit
tier, no external services).
🤖 Generated with [CodeBuddy](https://www.codebuddy.ai)
---------
Co-authored-by: yuzhichang <yuzhichang@infiniflow.ai>
Fixes#17202 (and complements #12109).
## Problem
`RAGFlowTxtParser.parser_txt` (`deepdoc/parser/txt_parser.py:36-47`) and
`rag.nlp.naive_merge` (`rag/nlp/__init__.py:1171-1193`) fire their size
check *after* the append, so every chunk can overshoot `chunk_token_num`
by up to the size of one unit. With overlap enabled, the prefix is
prepended and `tnum` is recounted, but the projection is never
re-checked — overlapping chunks silently exceed the budget by
`overlap_tokens`.
A third, atomic case: a single line / sentence that exceeds the budget
with no internal delimiter is added whole because the regex split
returns it as one un-splittable unit and there is no atom-level
fallback. `RAGFlowHtmlParser.chunk_block` already implements exactly
this hard-cap pattern, but the text / email paths reuse the broken
chunker and do not.
Measured on a live dataset (336 `.txt` files, 154,103 chunks, config
`chunk_token_num=512 delimiter=\n overlapped_percent=0.1`): 56.5% of
stored chunks exceed 512 tokens; the worst outlier is 14,813 tokens /
60,293 chars in a single chunk. Symptom downstream: rerank failures on
the >2048-token outliers (ref. #12109) and silent embedding truncation
on every oversize chunk.
## Fix
Mirror the proven pattern in `RAGFlowHtmlParser.chunk_block`:
1. **Proactive projected-total check** in `TxtParser.parser_txt` and in
`naive_merge.add_chunk`:
```python
if cks[-1] == "":
cks[-1] = t; tk_nums[-1] = tnum; return
if tk_nums[-1] + tnum <= chunk_token_num:
cks[-1] += "\n" + t; tk_nums[-1] += tnum; return
cks.append(t); tk_nums.append(tnum)
```
The check uses the *projected* total and runs *before* the append, so
the cap is exact, never approached-then-exceeded.
2. **Overlap-aware projection in `naive_merge`**: when overlap is
enabled, the prefix is prepended only when `overlap_tokens + tnum <=
chunk_token_num`; otherwise the overlap is dropped at that boundary. The
naive_merge-with-images mirror gets the same treatment. Custom-delimiter
behaviour is preserved per the existing test suite.
3. **Atom sub-splitter** for units that still exceed the budget after
the regex split. Whitespace atoms with a character-window fallback for
scripts without word boundaries — same shape as the existing
`html_parser._split_oversized_block`, so behaviour matches for HTML vs
`.txt` vs PDF atomic-oversize.
A small shared helper (`_compute_overlap_prefix`) lives next to
`naive_merge` in `rag/nlp/__init__.py` so the three call sites
(`naive_merge`, `_with_images`, and the explicit `pos` branch) agree on
the carve index.
## Result on the dataset above
| | Before | After |
|---|---|---|
| Chunks > 512 tokens | 56.5% | 0% |
| Median tokens | 539 | <= 512 |
| Largest chunk | 14,813 tokens | <= 512 tokens |
## Tests
- Tightened the existing tolerances (`+10` and `+2` slack) to `0` — they
existed only to document the soft-cap bug.
- Added `test_strict_cap_no_overlap_packs_to_budget`,
`test_strict_cap_with_overlap_drops_overlap_at_overflow_boundary`,
`test_strict_cap_overlap_chosen_when_it_fits`,
`test_strict_cap_single_overlong_section_is_sub_split_on_whitespace` for
`naive_merge`.
- Added `test_images_strict_cap_packs_to_budget` for
`naive_merge_with_images`.
- New `test/unit_test/deepdoc/parser/test_txt_parser.py` covers
`parser_txt` strict cap and atom sub-split. Uses the same path-loading
pattern as the existing `test_html_parser.py` to avoid pulling the deep
import chain into a test-time-only venv.
All 22 unit tests pass on the host venv:
```
test_naive_merge.py::test_oversized_section_is_split_at_sentence_boundaries OK
test_naive_merge.py::test_small_sections_are_merged_not_oversplit OK
test_naive_merge.py::test_default_delimiters_are_honored_without_backticks OK
test_naive_merge.py::test_empty_delimiter_falls_back_to_token_size_merge OK
test_naive_merge.py::test_overlap_prefix_is_counted_in_token_budget OK
test_naive_merge.py::test_custom_delimiter_ignores_chunk_size OK
test_naive_merge.py::test_custom_delimiter_does_not_size_merge OK
test_naive_merge.py::test_images_oversized_section_is_split OK
test_naive_merge.py::test_images_custom_delimiter_preserved OK
test_naive_merge.py::test_images_plain_string_input OK
test_naive_merge.py::test_images_mismatched_lengths_returns_empty OK
test_naive_merge.py::test_images_shared_lazyimage_not_stacked_… OK
test_naive_merge.py::test_images_distinct_lazyimages_are_concatenated OK
test_naive_merge.py::test_strict_cap_no_overlap_packs_to_budget OK
test_naive_merge.py::test_strict_cap_with_overlap_drops_… OK
test_naive_merge.py::test_strict_cap_single_overlong_section_… OK
test_naive_merge.py::test_strict_cap_overlap_chosen_when_it_fits OK
test_naive_merge.py::test_images_strict_cap_packs_to_budget OK
test_txt_parser.py::test_no_overshoot_when_packing_short_lines OK
test_txt_parser.py::test_no_overshoot_at_chunk_boundary OK
test_txt_parser.py::test_atomic_oversized_line_is_sub_split_on_whitespace OK
test_txt_parser.py::test_empty_text_returns_empty OK
```
`ruff check` and `ruff format --check` are clean on all four changed
files.
## Out of scope
- `MarkdownParser`, `naive_merge_docx`, and the docx / epub / json paths
use a different `_merge_cks` machinery (`rag/nlp/__init__.py:1574`) that
already enforces the budget. They are unchanged.
- The `chunk_block` call sites in `deepdoc/parser/html_parser.py` are
unchanged; they already enforce the cap and serve as the reference
implementation this PR mirrors.
Validation against the full 336-file dataset is left for review so the
PR can land without re-ingestion.
---------
Co-authored-by: skbs-eng <skbs-eng@users.noreply.github.com>
Co-authored-by: kiloconnect[bot] <240665456+kiloconnect[bot]@users.noreply.github.com>
Closes#17384.
## Summary
Drops a dead `re.I` flag from two outlier delimiter-parsing sites and
adds regression tests so the inconsistency can't creep back.
## What's wrong
Two of the six delimiter-parsing implementations pass `re.I` to
`re.finditer`:
- `rag/nlp/__init__.py::get_delimiters` (line 1633)
- `deepdoc/parser/txt_parser.py::parser_txt` (line 51)
The other four implementations correctly omit `re.I`:
- `rag/nlp/__init__.py::naive_merge` custom-delimiter path (line 1195)
- `rag/nlp/__init__.py::naive_merge_with_images` custom-delimiter path
(line 1269)
- `rag/nlp/__init__.py::_build_cks` (line 1389)
- `rag/flow/chunker/token_chunker.py` (line 73)
## Why this matters (and why it doesn't break anything)
The flag is **dead code** today. Verified empirically with a Python
REPL:
```python
>>> import re
>>> for m in re.finditer(r"`([^`]+)`", "`end`", re.I):
... print(repr(m.group(1)))
'end' # plain string, no flag attached
>>> re.split("(a)", "Class A is a Sample")
['Cl', 'a', '', 's', ' A i', 's', ' a Sample']
# Case-sensitive: only lowercase 'a' splits. Uppercase 'A' is preserved.
```
`re.I` does not propagate from `re.finditer` to `m.group(1)` or to
downstream `re.split` / `re.match` calls (which all omit `re.I`). So the
actual splitting behavior has always been case-sensitive — removing the
flag is a **defensive cleanup**, not a behavioral fix.
So why bother?
1. **Consistency** — the two sites were the only outliers in a six-way
implementation cluster. The three sibling sites in `rag/nlp/__init__.py`
already omit `re.I`, which strongly suggests the flag was accidental.
2. **Future-proofing** — a refactor could easily propagate the flag to a
downstream `re.split` call where it *would* change behavior. The tests
added here pin the case-sensitive semantics so that regression fails
loudly.
3. **Reader clarity** — the flag is misleading. Anyone reading
`re.finditer(..., re.I)` reasonably assumes case-insensitive matching,
then has to trace all downstream calls to discover it's a no-op.
## Changes
- `rag/nlp/__init__.py` — drop `re.I` from `get_delimiters` (line 1633).
- `deepdoc/parser/txt_parser.py` — drop `re.I` from `parser_txt` (line
51).
- `test/unit_test/rag/test_delimiter_case_sensitive.py` — new test file
with:
- 4 behavioral tests on `get_delimiters` (pattern output + `re.split`
round-trip).
- 3 end-to-end tests through `naive_merge` (bare-char +
backtick-wrapped, both cases).
- 2 parametrized static checks that `re.I` / `re.IGNORECASE` is not
present at either of the two `re.finditer` sites.
## Testing
```
$ pytest test/unit_test/rag/test_delimiter_case_sensitive.py -v
============================= 9 passed in 0.19s ==============================
```
All tests pass on the patched code. Before the patch, the 2 static
checks fail with a clear assertion message (the 7 behavioral tests pass
either way, confirming `re.I` was dead code).
## Related
- #17384 — the issue this PR closes. Note the issue's reproduction code
(`re.split(..., flags=re.I)`) doesn't actually match what the production
code does — the production `re.split` calls all omit `re.I`, which is
why current behavior is already case-sensitive. The fix here is still
valuable as a defensive cleanup + test coverage, but it's not a
behavioral fix per se.
- #17383 — broader parser consolidation (six implementations → one). The
fix here is independent and small enough to land first.
- #17385 — sibling UX PR (tooltip + live preview). Files are disjoint
(`web/src/**` vs `rag/nlp/**` + `deepdoc/parser/**`), so no interaction.
---------
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: kiloconnect[bot] <240665456+kiloconnect[bot]@users.noreply.github.com>
## Summary
Refactor the Go `KnowledgeCompilerComponent` so its parameter is a
**single string template id** instead of a DSL-level `variant` (or
plural group id list). The `variant` is no longer in the DSL — it is now
**derived at runtime from the resolved compilation template's `kind`
field**.
This aligns the Go ingestion port with the frontend Compiler operator,
which emits a singular `compilation_template_group_id` and does not
write `variant` into the generated `compiler.json`.
## Summary
Adds a side-effect-free DataFlow canvas **debug (dry-run) mode** plus a
**debug run log with a "View result" panel**, so a canvas can be
executed synchronously and inspected end-to-end (per-component progress
and parsed chunks) without persisting anything.
### Dry-run execution (inline parsed chunks)
- `task/debug.go`: `NewDebugTaskContext` builds an in-memory
`TaskContext` with **`KB.ID == ""` — the single debug signal used across
the ingestion pipeline**. A canvas debug run has no knowledgebase, and
production ingestion always supplies one, so `kb_id == ""` occurs ONLY
in debug mode. Components gate their own side effects on this signal
without any dedicated debug vocabulary (the former `CANVAS_DEBUG_DOC_ID`
marker constant is removed).
- `task/pipeline_executor.go`: `validateTaskContext` no longer requires
a KB when `KB.ID == ""` (debug); debug runs return `collectDebugOutput`
(chunks) instead of a no-op; uploaded bytes are delivered as
`inputs['binary']` for doc-less runs; `injectDebugPageCap` caps the
parser to the first pages for a fast preview via the production
`override_params` channel (Parser cpnID + family).
- `component/tokenizer.go`: `shouldHaveEmbedding` skips embedding when
`kb_id == ""` — the embedder is configured on the knowledgebase, so a
debug run has nothing to resolve against and stays side-effect free.
- `chunker/register.go`: chunk images are uploaded to MinIO only when a
KB is present (persist run). **In debug mode the raw image bytes are
intentionally dropped (`delete(ck, "image")`)** — the debug preview does
not render chunk images, and dropping the bytes keeps them out of memory
and out of the Redis-stored debug log. This is a deliberate trade-off,
not an oversight.
- `component/file.go`: pass through in-memory binary bytes, skipping
`doc_id` -> storage resolution.
- `handler/agent.go` + `agent_webhook.go`: detect `dataflow_canvas` and
run a sync debug returning chunks inline on the existing
chat/completions endpoint; reject DataFlow canvases from webhooks (fixes
the previously dead `== "DataFlow"` check; mirrors Python
`agent_api.py`).
- `parser_dispatch.go`: export `ParserFileFamily` for the executor's
page-cap injection.
### Debug run log + "View result"
Mirrors Python's debug-log contract so the front-end can replay each
component's progress and parsed output:
- `task/debug_log_sink.go`: a `DebugLogSink` records every component's
lifecycle into a `[{component_id, trace}]` array (each trace entry
carries `message`, `progress`, `timestamp`, `elapsed_time`). `Flush`
appends a terminal `END` marker whose first trace message is non-empty
so the front-end detects completion. On failure the END marker is
prefixed `[ERROR]` yet still carries the run, so the failure timeline
renders instead of being stuck empty. Timestamps and `elapsed_time` are
in seconds (matching the rest of the app).
- `task/debug_result_dsl.go`: `BuildDebugResultDSL` builds the `dsl` the
END marker carries — the Go analogue of Python's `Graph.__str__` +
END-marker `dsl` in `rag/flow/pipeline.py`. It combines the static DSL
structure (component_name / downstream / params / graph.nodes) with the
run output map (`output["state"][<id>]`) to emit, per component,
`obj.params.outputs[<format>].value` (chunks / text / json / html /
markdown) — the exact keys the front-end `dataflow-result` page reads to
render each step's parsed chunks. Raw embedding vectors (including the
dimension-scoped `q_<dim>_vec` keys) are stripped so the stored log
stays Python-scale.
- `task/pipeline_executor.go`: after the run, attach the built `dsl` to
the END marker via the `ResultSink` capability.
- `handler/agent.go`: `runCanvasPipelineDebug` generates a stable
`message_id` up-front and always flushes the log (success or failure);
`respondWithDebugResult` returns `message_id` in **both** the success
and the error envelope so the front-end can poll the log. The debug-log
endpoint `GET /agents/:id/logs/:message_id` serves the array.
- `web/src/pages/agent/hooks/use-run-dataflow.ts`: on a run failure,
also surface `message_id` via `setMessageId` so the log sheet renders
the failure timeline (the `[ERROR]` END marker is already written).
Guarded by `if (msgId)`, so it is a safe no-op when the back-end does
not return an id.
## Behavioral notes
- Debug parses only the first pages (`debugPageCapPages`) for a fast
preview; an explicit `pages` cap already present in the ParserConfig is
respected.
- Debug mode does not keep chunk images (see above) and does not compute
embeddings — it exercises parse + chunk only.
## Test plan
- Go: `debug_test.go`, `debug_log_sink_test.go` (trace pairing, END
marker, `[ERROR]` prefix, fractional-second timestamp/elapsed_time, size
caps, and a real-pipeline test asserting the END-marker `dsl` carries
non-empty per-component `params.outputs` with chunks),
`debug_result_dsl_test.go` (flat and real nested `output["state"]`
shapes, vector stripping, format priority),
`debug_pages_integration_test.go`, `pipeline_executor_persist_test.go`,
`handler/agent_pipeline_debug_test.go`, `handler/agent_logs_test.go`
(incl. `TestRunCanvasPipelineDebug_ErrorStillExposesMessageID` /
`TestRespondWithDebugResult_ErrorCarriesMessageID` locking `message_id`
on failure), plus updates to `agent_test.go` / `agent_webhook_test.go` /
`chunker/image_upload_test.go` / `tokenizer*.go`.
- `go build ./...` and `./build.sh --test` for affected packages.
🤖 Generated with [CodeBuddy Code](https://cnb.cool/codebuddy)
---------
Co-authored-by: CodeBuddy Code <noreply@tencent.com>
## Summary
Classify the Go test suite by dependency level using build tags so the
default
`go test ./...` run stays self-contained, and add local convenience
commands
plus a documented convention.
- Add build tags to 5 real-service tests that were previously un-tagged
and only
soft-isolated via `t.Skip`: `kg_test.go` (integration), `minio_test.go`
(integration), `template_integration_test.go` (integration),
`stagehand_runtime_integration_test.go` (integration),
`pipeline_e2e_test.go` (e2e).
The default unit run no longer compiles/attempts these.
- Reclassify the full-pipeline `real_consumer` tests from `integration`
to `e2e`.
- Add `build.sh` shortcuts: `--test-integration`, `--test-e2e`,
`--test-manual`,
`--test-all` (integration + e2e; `manual` is excluded and is local
opt-in only,
never run in CI).
- Document the tier scheme (unit / integration / e2e / manual +
orthogonal cgo) in
`AGENTS.md`.
## Tier definitions
| Tier | Build tag | Runs by default? |
|---|---|---|
| Unit | (none) | Yes — in-memory SQLite / miniredis / httptest stubs |
| Integration | `integration` | No (`-tags integration`) — single real
service |
| E2E | `e2e` | No (`-tags e2e`) — full ingest→index→retrieve pipeline |
| Manual | `manual` | No (`-tags manual`) — very slow; never in CI |
## Verification
- `gofmt -l` clean on all changed files; `bash -n build.sh` OK.
- `go list` confirms the default set excludes the tagged files, and
`-tags integration` / `-tags e2e` include them.
- Full regression: unit / integration / e2e each **97 ok, 0 FAIL**.
- Fixed a regression where `pipeline_knowledge_compiler_test.go` relied
on a
transitive import side-effect from `template_integration_test.go` to
register
the `File`/`Parser`/`TokenChunker` components; it now blank-imports the
component packages directly.
## Test plan
- [ ] `./build.sh --test` (unit) passes
- [ ] `./build.sh --test-integration` passes (needs real services; skips
otherwise)
- [ ] `./build.sh --test-e2e` passes (needs real services; skips
otherwise)
---------
Co-authored-by: CodeBuddy <noreply@cnb.cool>
Two regressions from #17536: (1) MySQL 8.0 AutoMigrate crash on
knowledge_compile_docs (Error 1101) from a TEXT default; (2) ingestor
Start() sync.Once re-entrant deadlock. Both fixed with regression tests.
## Summary
Fixes embedding inputs being emptied and documents chunker
heading-detection parity.
- `component/tokenizer.go`: `truncateForEmbedding` now mirrors Python
`common/token_utils.py:183-185` `truncate(string, max_len)` (keep the
first `max_len` tokens). For an unconfigured embedder reporting
`maxTokens <= 0`, instead of mirroring Python's `truncate` (which
returns `""` for `max_len <= 0` and would make the embeddings API reject
the batch with "inputs cannot be empty"), Go **clamps the limit to a
safe default** (`defaultEmbeddingTokenLimit = 8192`) so truncation stays
active and never produces empty inputs. The previous code returned `""`
for `maxTokens <= 10`, which emptied non-empty chunks into the batch and
triggered the API error.
- A **10-token safety margin** is reserved (mirroring Python's embedding
path `rag/svr/task_executor.py` which uses `mdl.max_length - 10`) so Go
does not over-send tokens to hard-capped embedding APIs; it is only
applied when the limit `> 10` so small limits remain non-empty.
- The clamp is applied **centrally inside `truncateForEmbedding`**,
covering both Builtin and generic callers — no separate
`model_service.go` change is required.
- `component/chunker/title.go`: comment-only — documents the
already-ported PDF-outline detection (omission 1.5 / Gap C) and the
hardcoded `BULLET_PATTERN` fallback (omission 1.7 / Gap C), porting
`common.py:_outline_similarity` and `rag/nlp` `BULLET_PATTERN`.
- `tokenizer_unit_test.go`: update truncation expectations to the new
positive-keeps-tokens / non-positive-clamps-to-default behaviour.
## Test plan
- `tokenizer_unit_test.go` updated for the new truncation behaviour
(`TestTruncateForEmbedding_SmallMaxTokens`,
`TestTruncateForEmbedding_UnconfiguredClampsToDefault`).
- `./build.sh --test ./internal/ingestion/component/` passes.
🤖 Generated with [CodeBuddy Code](https://cnb.cool/codebuddy)
Ports doc-level auto-metadata extraction to Go and adds the
knowledge_compiler component with scheduler/routing. Fixes Extractor
metadata injection type assertion and enable_metadata default-on.
## Summary
This PR hardens the Go ingestion **Extractor** and **LLM retry** paths
and closes several Python->Go parity gaps in the keyword/question/tag
extraction flow.
- **Generic retry utility** (`internal/common/retry.go`):
`RetryWithBackoff` with exponential backoff (default 3 retries, 2s
initial delay, capped at 1m), context-aware sleep, and a `maxRetries<=0`
fast path. Covered by `internal/common/retry_test.go`.
- **LLM retry reuse**: `agent/component/llm_retry.go` now delegates to
`common.RetryWithBackoff` instead of an inline loop (behavior preserved:
ctx cancellation short-circuits the backoff).
- **Extractor LLM calls** (`extractor.go`):
- `call()` now retries transient LLM failures via `RetryWithBackoff`
(retry exhaustion fails the chunk instead of silently skipping).
- Sets `temperature = 0.2`, matching Python `generator.py:230,245`.
- Runs keyword and question extraction **concurrently** per chunk when
both are enabled (`task_executor.py:444-448`), with mutex-guarded map
writes to avoid data races.
- Substitutes `{field_name}` placeholders (including `{chunks}` -> chunk
text) in `prompt`/`system_prompt` before the call, mirroring Python
`string_format` (`extractor.py:102-103`); unmatched placeholders are
left as-is.
- Falls back to the **tenant default chat model** when `llm_id` is empty
(`task_executor.py:573-574`).
- Strips `` **greedily** (`strings.LastIndex`) in
`cleanExtractionResult`.
- **Auto-tagging** (`extractor_tag.go`): drops the `in.llmID != ""`
guards so an empty `llm_id` no longer skips tagging (uses the tenant
default model), and strips `` greedily in `parseTaggerResponse`.
- **Docs**: fixes a misleading `PresentationChunker` docstring that
claimed per-slide `image`/`position` output (the PPTX path emits none —
unlike PDF), and removes a stale `docs/migration_python_go_diff.md`
reference in `media_dispatch.go`.
## Test plan
- `bash build.sh --test ./internal/common/...` — passes (new retry
utility + tests).
- `bash build.sh --test ./internal/ingestion/component/...` — passes
(extractor/chunker/schema).
- `gofmt` and lefthook pre-commit checks pass.
Note: the personal `docs/migration_python_go_diff.md` working notebook
in the tree is intentionally **not** part of this PR.
---------
Co-authored-by: CodeBuddy <noreply@codebuddy.ai>
## Problem
`TestMergeByTokenSizeFromJSON_ClampsOverlappedPct` reused a single
`items` fixture across four calls to `mergeByTokenSizeFromJSON`.
`mergeByTokenSizeFromJSON` mutates its `perItem` argument in place and
returns the same backing array (token.go: `perItem[idx] = merged`). As a
result:
- The 2nd and later calls merged already-merged chunks instead of the
original input.
- Because the returned slice aliases the input, later calls silently
overwrote the earlier results (`at100`, `at0`, ...).
- `reflect.DeepEqual(at100, at150)` was therefore vacuously true — the
test was a **false positive** that never actually exercised the clamp.
It would still pass even if the clamp were broken.
This is exactly the defect flagged in the review comment on PR #17396.
## Fix
Add a `clampOverlapFixture()` factory and build a fresh fixture for
every call, so that 150 / 1e300 / -5 / -1e300 are applied to the
original input.
## Verification
- The test passes after the fix.
- When the clamp logic was temporarily disabled, the test **failed**
(panic at token.go:768 — the negative index produced by an out-of-range
pct), proving the fixed test is no longer a false positive and can catch
a clamp regression.
## Scope
Test file only. No production code change (verified `token.go` is
identical to `upstream/main`).
Refs: review comment on PR #17396.
## Summary
Closes two Chunker migration gaps documented in
`docs/migration_python_go_diff.md`
(diffs **2.6** and **1.7**), improving parity with the Python ingestion
pipeline.
This is the code portion of commit `261e1fd0b` on branch
`fix/batch-3-4`; the
migration document itself is tracked separately (untracked in this
commit).
### Diff 2.6 — `overlapped_percent` missing Python normalization
Python's `normalize_overlapped_percent` (`common/float_utils.py:50-58`)
accepts a
`[0,1)` fraction (the flow-canvas UI validates `[0,1)`), multiplies it
by 100,
`int()`-truncates, and clamps to `[0,90]`. Go previously only accepted a
raw
`[0,90]` percentage and **rejected** out-of-range input, so a Python
config
passing `0.1` (meaning 10%) silently produced ~0% overlap.
Added `normalizeOverlappedPercent`
(`internal/ingestion/component/chunker/common.go`) mirroring the Python
helper:
- parses numbers and numeric strings (mirrors Python `float()`; bad /
`NaN` / `Inf` → `0`),
- `0 < v < 1 → v *= 100`,
- `int()` truncation,
- clamps to `[0,90]`.
Wired into `tokenChunkerParam.Update` (`token.go`); the merge math
(`token.go:705`, `(100-x)/100`) already matched Python.
`TokenChunkerParam.Validate`
(`schema/chunker.go`) now only guards direct struct construction.
Tests: `TestNormalizeOverlappedPercent`, extended
`TestTokenChunker_NewAcceptsPythonOverlappedRange` (adds fraction/clamp
inputs),
new `TestTokenChunker_NormalizesOverlappedPercent`. The existing
reject-test cases
for `<0` / `>90` were removed because they are now normalized/clamped
(Python parity).
### Diff 1.7 — missing `BULLET_PATTERN` title-level fallback
`resolveTitleLevels` (`title.go`) now applies a 4th-level fallback: when
outline +
regex + layout all yield body level, `bulletsCategory` selects the
best-matching
bullet-pattern group (Chinese legal / numbering / Chinese numbering /
English legal
— mirroring `rag/nlp/__init__.py:258-320`) and assigns structural
levels. Guarded by
`allBodyLevel` so it never overrides an existing outline/regex level.
Tests: `TestResolveTitleLevels_BulletFallback` (4 subtests).
### Incidental test adjustments included in the commit
- `token_batch1_test.go`: overlap input changed `0.3` → `30.0` to
reflect the
post-normalization 30% semantics.
- `real_consumer_test.go`: updated `LoadFromIngestionTask(task)` →
`LoadFromIngestionTask(ctx, task)` for the new context-first signature.
## Verification
`bash build.sh --test ./internal/ingestion/component/...` — chunker +
schema suites
pass, no regression (CGO build).
## Migration doc reference
`docs/migration_python_go_diff.md` §Chunker 1.7 and 2.6 are marked
**Fixed** for
these changes.
