## Summary
Six sites used to read the same `parser_config.delimiter` field with
divergent grammars:
- `rag.nlp.get_delimiters` (PDF/DOCX/HTML/EPUB/JSON/CSV/XLSX/email/book)
- `rag.nlp.naive_merge` (custom-delimiter branch)
- `rag.nlp.naive_merge_with_images`
- `rag.nlp._build_cks`
- `deepdoc.parser.txt_parser.parser_txt` (.txt, code)
-
`deepdoc.parser.markdown_parser.MarkdownElementExtractor.get_delimiters`
The six implementations disagreed on bare-vs-wrapped chars, dedupe, sort
order, CRLF normalization, and `re.I` (#17384). The shipped default ``
`\n!?;。；！？` `` was a no-op for `.md` because the markdown path only
matched backtick-wrapped tokens.
## Changes
- **new:** `rag/nlp/delim.py` with `parse_delimiter_field` and
`compile_delimiter_pattern`. Single source of truth. CRLF normalization
at the top; longest-first stable sort; insertion-ordered dedupe; no
`re.I`.
- **refactor:** all six call sites delegate to the helper.
- `rag/nlp/__init__.py::get_delimiters` becomes a thin shim.
- `deepdoc/parser/txt_parser.py::parser_txt` drops the
`[encode/decode/unicode_escape]` round-trip.
- `deepdoc/parser/markdown_parser.py::get_delimiters` honors bare chars
(fixes [1]).
- **tests:** `test/unit_test/rag/test_delim.py` (85 tests) — helper,
acceptance table, frontend parity, static guard against re-inlining.
- **tests:** `test/unit_test/rag/test_delimiter_case_sensitive.py` (from
#17386) updated to retarget the static check at the new helper +
AST-based broader scan.
## Acceptance criteria
- All six sites produce the same regex pattern for the same input.
- Shipped default keeps working for `.txt` / `.pdf` / `.docx`.
- Shipped default for `.md` now splits (was a silent no-op).
- Tooltip example `` `\n##;` `` produces three effective delimiters
regardless of file type.
- Bare whitespace inputs split on every occurrence.
- Backtick-wrapped whitespace splits only on the exact N-char sequence.
- CRLF-line-ending documents split identically to LF-line-ending
documents.
- 123 tests pass (85 new + 38 existing).
## Rebase protocol
As #17385 and #17386 evolve, this branch will be rebased on top. The
only overlap between this PR's diff and the other two is
`test_delimiter_case_sensitive.py`, where #17383 modifies the static
check to point at the new helper location.
---------
Co-authored-by: kiloconnect[bot] <240665456+kiloconnect[bot]@users.noreply.github.com>
## Summary
This PR refactors the Go knowledge-compilation ingestion pipeline
(`internal/ingestion/knowledge_compile` +
`internal/ingestion/component/knowledge_compiler`) with three related
changes:
- **Token-budget batching for LLM merge decisions.**
`LLMMergeDecider.DecideBatch` previously stuffed every `(existing,
candidate)` pair into a single LLM call, risking `max_token` overflow.
It now splits pairs into token-bounded sub-batches (budget =
`llmMaxTokens * 0.85`) via `tokenizer.NumTokensFromString`, runs them
concurrently while preserving the global pair index, and never
reindexes.
- **Process-level global compile pool.** Introduces a single vCPU-sized
goroutine pool (`pool.go`, env `KC_COMPILE_CONCURRENCY`) dedicated to
*all* knowledge-compilation stages. KNN search loop, `DecideBatch`
sub-batches, `WriteMerged`/`DeleteMerged` internals, and the
component-level (structure/mindmap) per-call pools are all unified into
it via an injected submitter. No more per-job short-lived goroutines in
`runCompilerJobs` (futures are collected then awaited on the caller).
Fan-out stays bounded by the pool worker count; these stages are
docengine-bounded / LLM-bounded, not CPU-bounded.
- **DocEngine-only deletion.** `Consumer.processBatch` deletion no
longer loads the deleted docs' products into memory. Two sequential
DocEngine calls replace the old in-memory surgery:
- `DeleteDocLevelForDocs` — one `DeleteChunks` over `doc_id IN
deletedDocIDs` (merged rows carry `doc_id == kb`, so only per-doc
products match).
- `StripMergedSources` — one `Search` of `kc_merged=1` rows filtered by
`source_doc_ids IN deletedDocIDs` (intersection pushed down to the
engine), `UpdateChunks` the source array of survivors, and
`DeleteChunks` the rows whose array became empty.
## Changes
- `internal/ingestion/knowledge_compile/pool.go` (new): global
`compilerPool` +
`runCompilerJobs`/`SubmitCompilerJob`/`SubmitCompilerJobs`.
- `internal/ingestion/knowledge_compile/consumer.go`: deletion rewritten
to the two DocEngine calls;
`mergedBase`/`toDelete`/`stripDeletedSources` removed.
- `internal/ingestion/knowledge_compile/writer.go`:
`DeleteDocLevelForDocs` + `StripMergedSources` replace
`DeleteMergedForDoc`/`DeleteMerged`.
- `internal/ingestion/knowledge_compile/reader.go`: drop
`LoadMergedBySourceDoc` + `containsString` (keep `LoadDocProducts` for
the completion branch).
- `internal/ingestion/knowledge_compile/dedup.go`: `NewLLMDeduper` takes
`llmMaxTokens`; wires `SetMaxBatchTokens`/`SetSubmitter`.
- `internal/ingestion/knowledge_compiler/{structure,merge}.go`,
`mindmap/mindmap.go`, `pool_wiring.go`: token-budget split + submitter
injection.
- Tests: `structure_test.go` (token-budget split), `dedup_test.go`,
`consumer_test.go` (tombstone + DocEngine deletion assertions) updated.
## Validation
`bash build.sh --test -race ./internal/ingestion/knowledge_compile/...
./internal/ingestion/component/knowledge_compiler/...` passes (unit
tier, no external services).
🤖 Generated with [CodeBuddy](https://www.codebuddy.ai)
---------
Co-authored-by: yuzhichang <yuzhichang@infiniflow.ai>
Fixes#17202 (and complements #12109).
## Problem
`RAGFlowTxtParser.parser_txt` (`deepdoc/parser/txt_parser.py:36-47`) and
`rag.nlp.naive_merge` (`rag/nlp/__init__.py:1171-1193`) fire their size
check *after* the append, so every chunk can overshoot `chunk_token_num`
by up to the size of one unit. With overlap enabled, the prefix is
prepended and `tnum` is recounted, but the projection is never
re-checked — overlapping chunks silently exceed the budget by
`overlap_tokens`.
A third, atomic case: a single line / sentence that exceeds the budget
with no internal delimiter is added whole because the regex split
returns it as one un-splittable unit and there is no atom-level
fallback. `RAGFlowHtmlParser.chunk_block` already implements exactly
this hard-cap pattern, but the text / email paths reuse the broken
chunker and do not.
Measured on a live dataset (336 `.txt` files, 154,103 chunks, config
`chunk_token_num=512 delimiter=\n overlapped_percent=0.1`): 56.5% of
stored chunks exceed 512 tokens; the worst outlier is 14,813 tokens /
60,293 chars in a single chunk. Symptom downstream: rerank failures on
the >2048-token outliers (ref. #12109) and silent embedding truncation
on every oversize chunk.
## Fix
Mirror the proven pattern in `RAGFlowHtmlParser.chunk_block`:
1. **Proactive projected-total check** in `TxtParser.parser_txt` and in
`naive_merge.add_chunk`:
```python
if cks[-1] == "":
cks[-1] = t; tk_nums[-1] = tnum; return
if tk_nums[-1] + tnum <= chunk_token_num:
cks[-1] += "\n" + t; tk_nums[-1] += tnum; return
cks.append(t); tk_nums.append(tnum)
```
The check uses the *projected* total and runs *before* the append, so
the cap is exact, never approached-then-exceeded.
2. **Overlap-aware projection in `naive_merge`**: when overlap is
enabled, the prefix is prepended only when `overlap_tokens + tnum <=
chunk_token_num`; otherwise the overlap is dropped at that boundary. The
naive_merge-with-images mirror gets the same treatment. Custom-delimiter
behaviour is preserved per the existing test suite.
3. **Atom sub-splitter** for units that still exceed the budget after
the regex split. Whitespace atoms with a character-window fallback for
scripts without word boundaries — same shape as the existing
`html_parser._split_oversized_block`, so behaviour matches for HTML vs
`.txt` vs PDF atomic-oversize.
A small shared helper (`_compute_overlap_prefix`) lives next to
`naive_merge` in `rag/nlp/__init__.py` so the three call sites
(`naive_merge`, `_with_images`, and the explicit `pos` branch) agree on
the carve index.
## Result on the dataset above
| | Before | After |
|---|---|---|
| Chunks > 512 tokens | 56.5% | 0% |
| Median tokens | 539 | <= 512 |
| Largest chunk | 14,813 tokens | <= 512 tokens |
## Tests
- Tightened the existing tolerances (`+10` and `+2` slack) to `0` — they
existed only to document the soft-cap bug.
- Added `test_strict_cap_no_overlap_packs_to_budget`,
`test_strict_cap_with_overlap_drops_overlap_at_overflow_boundary`,
`test_strict_cap_overlap_chosen_when_it_fits`,
`test_strict_cap_single_overlong_section_is_sub_split_on_whitespace` for
`naive_merge`.
- Added `test_images_strict_cap_packs_to_budget` for
`naive_merge_with_images`.
- New `test/unit_test/deepdoc/parser/test_txt_parser.py` covers
`parser_txt` strict cap and atom sub-split. Uses the same path-loading
pattern as the existing `test_html_parser.py` to avoid pulling the deep
import chain into a test-time-only venv.
All 22 unit tests pass on the host venv:
```
test_naive_merge.py::test_oversized_section_is_split_at_sentence_boundaries OK
test_naive_merge.py::test_small_sections_are_merged_not_oversplit OK
test_naive_merge.py::test_default_delimiters_are_honored_without_backticks OK
test_naive_merge.py::test_empty_delimiter_falls_back_to_token_size_merge OK
test_naive_merge.py::test_overlap_prefix_is_counted_in_token_budget OK
test_naive_merge.py::test_custom_delimiter_ignores_chunk_size OK
test_naive_merge.py::test_custom_delimiter_does_not_size_merge OK
test_naive_merge.py::test_images_oversized_section_is_split OK
test_naive_merge.py::test_images_custom_delimiter_preserved OK
test_naive_merge.py::test_images_plain_string_input OK
test_naive_merge.py::test_images_mismatched_lengths_returns_empty OK
test_naive_merge.py::test_images_shared_lazyimage_not_stacked_… OK
test_naive_merge.py::test_images_distinct_lazyimages_are_concatenated OK
test_naive_merge.py::test_strict_cap_no_overlap_packs_to_budget OK
test_naive_merge.py::test_strict_cap_with_overlap_drops_… OK
test_naive_merge.py::test_strict_cap_single_overlong_section_… OK
test_naive_merge.py::test_strict_cap_overlap_chosen_when_it_fits OK
test_naive_merge.py::test_images_strict_cap_packs_to_budget OK
test_txt_parser.py::test_no_overshoot_when_packing_short_lines OK
test_txt_parser.py::test_no_overshoot_at_chunk_boundary OK
test_txt_parser.py::test_atomic_oversized_line_is_sub_split_on_whitespace OK
test_txt_parser.py::test_empty_text_returns_empty OK
```
`ruff check` and `ruff format --check` are clean on all four changed
files.
## Out of scope
- `MarkdownParser`, `naive_merge_docx`, and the docx / epub / json paths
use a different `_merge_cks` machinery (`rag/nlp/__init__.py:1574`) that
already enforces the budget. They are unchanged.
- The `chunk_block` call sites in `deepdoc/parser/html_parser.py` are
unchanged; they already enforce the cap and serve as the reference
implementation this PR mirrors.
Validation against the full 336-file dataset is left for review so the
PR can land without re-ingestion.
---------
Co-authored-by: skbs-eng <skbs-eng@users.noreply.github.com>
Co-authored-by: kiloconnect[bot] <240665456+kiloconnect[bot]@users.noreply.github.com>
Closes#17384.
## Summary
Drops a dead `re.I` flag from two outlier delimiter-parsing sites and
adds regression tests so the inconsistency can't creep back.
## What's wrong
Two of the six delimiter-parsing implementations pass `re.I` to
`re.finditer`:
- `rag/nlp/__init__.py::get_delimiters` (line 1633)
- `deepdoc/parser/txt_parser.py::parser_txt` (line 51)
The other four implementations correctly omit `re.I`:
- `rag/nlp/__init__.py::naive_merge` custom-delimiter path (line 1195)
- `rag/nlp/__init__.py::naive_merge_with_images` custom-delimiter path
(line 1269)
- `rag/nlp/__init__.py::_build_cks` (line 1389)
- `rag/flow/chunker/token_chunker.py` (line 73)
## Why this matters (and why it doesn't break anything)
The flag is **dead code** today. Verified empirically with a Python
REPL:
```python
>>> import re
>>> for m in re.finditer(r"`([^`]+)`", "`end`", re.I):
... print(repr(m.group(1)))
'end' # plain string, no flag attached
>>> re.split("(a)", "Class A is a Sample")
['Cl', 'a', '', 's', ' A i', 's', ' a Sample']
# Case-sensitive: only lowercase 'a' splits. Uppercase 'A' is preserved.
```
`re.I` does not propagate from `re.finditer` to `m.group(1)` or to
downstream `re.split` / `re.match` calls (which all omit `re.I`). So the
actual splitting behavior has always been case-sensitive — removing the
flag is a **defensive cleanup**, not a behavioral fix.
So why bother?
1. **Consistency** — the two sites were the only outliers in a six-way
implementation cluster. The three sibling sites in `rag/nlp/__init__.py`
already omit `re.I`, which strongly suggests the flag was accidental.
2. **Future-proofing** — a refactor could easily propagate the flag to a
downstream `re.split` call where it *would* change behavior. The tests
added here pin the case-sensitive semantics so that regression fails
loudly.
3. **Reader clarity** — the flag is misleading. Anyone reading
`re.finditer(..., re.I)` reasonably assumes case-insensitive matching,
then has to trace all downstream calls to discover it's a no-op.
## Changes
- `rag/nlp/__init__.py` — drop `re.I` from `get_delimiters` (line 1633).
- `deepdoc/parser/txt_parser.py` — drop `re.I` from `parser_txt` (line
51).
- `test/unit_test/rag/test_delimiter_case_sensitive.py` — new test file
with:
- 4 behavioral tests on `get_delimiters` (pattern output + `re.split`
round-trip).
- 3 end-to-end tests through `naive_merge` (bare-char +
backtick-wrapped, both cases).
- 2 parametrized static checks that `re.I` / `re.IGNORECASE` is not
present at either of the two `re.finditer` sites.
## Testing
```
$ pytest test/unit_test/rag/test_delimiter_case_sensitive.py -v
============================= 9 passed in 0.19s ==============================
```
All tests pass on the patched code. Before the patch, the 2 static
checks fail with a clear assertion message (the 7 behavioral tests pass
either way, confirming `re.I` was dead code).
## Related
- #17384 — the issue this PR closes. Note the issue's reproduction code
(`re.split(..., flags=re.I)`) doesn't actually match what the production
code does — the production `re.split` calls all omit `re.I`, which is
why current behavior is already case-sensitive. The fix here is still
valuable as a defensive cleanup + test coverage, but it's not a
behavioral fix per se.
- #17383 — broader parser consolidation (six implementations → one). The
fix here is independent and small enough to land first.
- #17385 — sibling UX PR (tooltip + live preview). Files are disjoint
(`web/src/**` vs `rag/nlp/**` + `deepdoc/parser/**`), so no interaction.
---------
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: kiloconnect[bot] <240665456+kiloconnect[bot]@users.noreply.github.com>
### What problem does this PR solve?
`common/misc_utils.once` set `executed=True` before invoking the wrapped
function, so an exception on the first call permanently disabled future
calls and returned the cached `None`.
This change marks `executed=True` only after a successful call, allowing
retries after transient failures while preserving once-only behavior
after success. It also adds regression tests for retry-after-exception
and thread-safe single execution.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
---------
Co-authored-by: Harsh Kashyap <harshkashyap@Harshs-MacBook-Pro.local>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Jin Hai <haijin.chn@gmail.com>
### What problem does this PR solve?
Fixes#17623 by updating the retrieval API response example to use
`dataset_id`, matching the actual `POST /api/v1/retrieval` response
field and the implementation mapping from internal `kb_id`.
Co-authored-by: Codex <codex@openai.com>
## Summary
Remediates CVE-2026-59939 (HIGH) in `httplib2` by adding
`httplib2>=0.32.0` to `constraint-dependencies` in `pyproject.toml`.
| CVE | Severity | Package | Installed | Fixed in |
|---|---|---|---|---|
| CVE-2026-59939 | HIGH | httplib2 | 0.31.0 | 0.32.0 |
`httplib2` is a transitive dependency pulled in by
`google-api-python-client` and `google-auth-httplib2`. Both declare
`httplib2<1.0.0,>=0.19.0`, allowing 0.32.0 without conflict.
## Summary
Refactor the Go `KnowledgeCompilerComponent` so its parameter is a
**single string template id** instead of a DSL-level `variant` (or
plural group id list). The `variant` is no longer in the DSL — it is now
**derived at runtime from the resolved compilation template's `kind`
field**.
This aligns the Go ingestion port with the frontend Compiler operator,
which emits a singular `compilation_template_group_id` and does not
write `variant` into the generated `compiler.json`.
### Summary
As title, this can not be tested for now
Close#17520
---------
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
## Summary
- Align Go agent logs access with the Python behavior by checking tenant
access to the target agent instead of requiring the token to be bound
to that agent.
- Marshal canvas lifecycle event payloads safely so non-serializable
inputs do not break Thinking expansion in the UI.
## Testing
- `CGO_ENABLED=0 go test -count=1 -v -run
"Handler|Attachment|Logs|Chatbot|Agentbot|BuildWorkflow|NodeLifecycle|VarRef|Resolve|Extract"./internal/handler/
./internal/agent/canvas/...` PASS
### Summary
`TestSearchHandlerUpdateRejectsInvalidSearchID` fails on `main`:
```
--- FAIL: TestSearchHandlerUpdateRejectsInvalidSearchID (0.00s)
search_handler_test.go:98: expected 'no authorization' in message, got No authorization.
```
`UpdateSearch` answers an unauthorized `search_id` with `"No
authorization."` (`internal/handler/search.go:337`), which mirrors the
Python API's message verbatim, but the assertion searched for the
lowercase `"no authorization"`.
The fix lowercases the response before the substring check rather than
changing the handler's message: the capitalized text is the contract the
Python API exposes, so the test — not the handler — was wrong.
`bash build.sh --test -run TestSearchHandler ./internal/handler/`
passes.
## Summary
Adds a side-effect-free DataFlow canvas **debug (dry-run) mode** plus a
**debug run log with a "View result" panel**, so a canvas can be
executed synchronously and inspected end-to-end (per-component progress
and parsed chunks) without persisting anything.
### Dry-run execution (inline parsed chunks)
- `task/debug.go`: `NewDebugTaskContext` builds an in-memory
`TaskContext` with **`KB.ID == ""` — the single debug signal used across
the ingestion pipeline**. A canvas debug run has no knowledgebase, and
production ingestion always supplies one, so `kb_id == ""` occurs ONLY
in debug mode. Components gate their own side effects on this signal
without any dedicated debug vocabulary (the former `CANVAS_DEBUG_DOC_ID`
marker constant is removed).
- `task/pipeline_executor.go`: `validateTaskContext` no longer requires
a KB when `KB.ID == ""` (debug); debug runs return `collectDebugOutput`
(chunks) instead of a no-op; uploaded bytes are delivered as
`inputs['binary']` for doc-less runs; `injectDebugPageCap` caps the
parser to the first pages for a fast preview via the production
`override_params` channel (Parser cpnID + family).
- `component/tokenizer.go`: `shouldHaveEmbedding` skips embedding when
`kb_id == ""` — the embedder is configured on the knowledgebase, so a
debug run has nothing to resolve against and stays side-effect free.
- `chunker/register.go`: chunk images are uploaded to MinIO only when a
KB is present (persist run). **In debug mode the raw image bytes are
intentionally dropped (`delete(ck, "image")`)** — the debug preview does
not render chunk images, and dropping the bytes keeps them out of memory
and out of the Redis-stored debug log. This is a deliberate trade-off,
not an oversight.
- `component/file.go`: pass through in-memory binary bytes, skipping
`doc_id` -> storage resolution.
- `handler/agent.go` + `agent_webhook.go`: detect `dataflow_canvas` and
run a sync debug returning chunks inline on the existing
chat/completions endpoint; reject DataFlow canvases from webhooks (fixes
the previously dead `== "DataFlow"` check; mirrors Python
`agent_api.py`).
- `parser_dispatch.go`: export `ParserFileFamily` for the executor's
page-cap injection.
### Debug run log + "View result"
Mirrors Python's debug-log contract so the front-end can replay each
component's progress and parsed output:
- `task/debug_log_sink.go`: a `DebugLogSink` records every component's
lifecycle into a `[{component_id, trace}]` array (each trace entry
carries `message`, `progress`, `timestamp`, `elapsed_time`). `Flush`
appends a terminal `END` marker whose first trace message is non-empty
so the front-end detects completion. On failure the END marker is
prefixed `[ERROR]` yet still carries the run, so the failure timeline
renders instead of being stuck empty. Timestamps and `elapsed_time` are
in seconds (matching the rest of the app).
- `task/debug_result_dsl.go`: `BuildDebugResultDSL` builds the `dsl` the
END marker carries — the Go analogue of Python's `Graph.__str__` +
END-marker `dsl` in `rag/flow/pipeline.py`. It combines the static DSL
structure (component_name / downstream / params / graph.nodes) with the
run output map (`output["state"][<id>]`) to emit, per component,
`obj.params.outputs[<format>].value` (chunks / text / json / html /
markdown) — the exact keys the front-end `dataflow-result` page reads to
render each step's parsed chunks. Raw embedding vectors (including the
dimension-scoped `q_<dim>_vec` keys) are stripped so the stored log
stays Python-scale.
- `task/pipeline_executor.go`: after the run, attach the built `dsl` to
the END marker via the `ResultSink` capability.
- `handler/agent.go`: `runCanvasPipelineDebug` generates a stable
`message_id` up-front and always flushes the log (success or failure);
`respondWithDebugResult` returns `message_id` in **both** the success
and the error envelope so the front-end can poll the log. The debug-log
endpoint `GET /agents/:id/logs/:message_id` serves the array.
- `web/src/pages/agent/hooks/use-run-dataflow.ts`: on a run failure,
also surface `message_id` via `setMessageId` so the log sheet renders
the failure timeline (the `[ERROR]` END marker is already written).
Guarded by `if (msgId)`, so it is a safe no-op when the back-end does
not return an id.
## Behavioral notes
- Debug parses only the first pages (`debugPageCapPages`) for a fast
preview; an explicit `pages` cap already present in the ParserConfig is
respected.
- Debug mode does not keep chunk images (see above) and does not compute
embeddings — it exercises parse + chunk only.
## Test plan
- Go: `debug_test.go`, `debug_log_sink_test.go` (trace pairing, END
marker, `[ERROR]` prefix, fractional-second timestamp/elapsed_time, size
caps, and a real-pipeline test asserting the END-marker `dsl` carries
non-empty per-component `params.outputs` with chunks),
`debug_result_dsl_test.go` (flat and real nested `output["state"]`
shapes, vector stripping, format priority),
`debug_pages_integration_test.go`, `pipeline_executor_persist_test.go`,
`handler/agent_pipeline_debug_test.go`, `handler/agent_logs_test.go`
(incl. `TestRunCanvasPipelineDebug_ErrorStillExposesMessageID` /
`TestRespondWithDebugResult_ErrorCarriesMessageID` locking `message_id`
on failure), plus updates to `agent_test.go` / `agent_webhook_test.go` /
`chunker/image_upload_test.go` / `tokenizer*.go`.
- `go build ./...` and `./build.sh --test` for affected packages.
🤖 Generated with [CodeBuddy Code](https://cnb.cool/codebuddy)
---------
Co-authored-by: CodeBuddy Code <noreply@tencent.com>
