## What
Adds [**SereneDB**](https://serenedb.com) as a selectable doc-store
engine on **both** RAGFlow paths:
- the **Go** `DocEngine` (`internal/engine/serenedb`), alongside
Elasticsearch and Infinity;
- the **Python** `DocStoreConnection` (`rag/utils/serenedb_conn.py`) +
`DOC_ENGINE=serenedb` registration.
SereneDB is a PostgreSQL-wire engine (DuckDB execution) whose single
inverted index carries **both** a scored text column (`@@`, BM25) and an
IVF vector column (`<#>`, inner product), so hybrid search is one SQL
statement. The Go engine connects with `database/sql` + `lib/pq`
(already a dependency, no new module); the Python connector uses
psycopg2 (already a dependency).
## Storage model
One table per tenant with `kb_id` as a filter column - the
**Elasticsearch / OceanBase** model, not Infinity's per-dataset tables.
This keeps BM25 statistics (IDF, avgdl) computed over the whole tenant
corpus (global IDF). Both connectors use this identical layout, so they
are storage- and retrieval-compatible: `hybrid` proxy routing and
Python↔Go switching are safe. On the Python side the connector is wired
as OceanBase's plain-SQL sibling (chunk_data JSON metadata, inline chunk
vectors, verbatim ES field names); the ES tokenizer path is unchanged.
Metadata stays one table per tenant (`ragflow_doc_meta_<tenant>`).
The query shapes mirror the Python connector, including the five
empirically-found landmines: the scored dictionary needs `frequency +
norm` (else `BM25()` silently returns 0.0), the `@@` query is the
tokenized query, the scored lexical branch matches one column, vectors
use an L2-normalized shadow column with `ip`/`sq8`, and the similarity
threshold goes directly in the ANN scan's `WHERE`. **Minimum engine
version: SereneDB 26.07.4.**
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
## Summary
Relate to #17284.
Completes the migration of the four non-OpenAI-compatible model drivers
(`anthropic`, `cohere`, `google`, `bedrock`) onto the shared
usage-recording path. Earlier batches (#17634, #17643, #17696–#17700)
covered only the OpenAI-compatible cluster; these four providers ship
wire formats that do not fit the OpenAI `choices[0].delta` / `usage`
block template and so were left for a separate pass.
Per the maintainer's guidance for this batch, each driver is migrated on
its own terms rather than forced through a single template. The shared
machinery used is intentionally small: `recordResponseUsage`,
`parseChatCompletionResponse`, `BaseModel.newJSONPostRequest`, and the
existing `authHeader` hook for non-Bearer auth.
Co-authored-by: Haruko386 <tryeverypossible@163.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
## Summary
DeepSeek and Azure OpenAI require `stream_options.include_usage=true` to
return token usage in streaming responses. Without it, all streaming
calls report zero usage to ClickHouse and the UI shows no token stats.
- [x] Verify DeepSeek streaming calls now report usage
- [x] Verify Azure OpenAI streaming calls now report usage
## Summary
- Normalize Canvas component names before resolving Go Agent tools and
parameters.
- Add regression coverage for CodeExec and other Canvas tool mappings.
## Testing
- `CGO_ENABLED=0 go test -count=1 ./internal/agent/tool
./internal/agent/component`
## Summary
Extract the pipeline-output → search-engine index document mapping
helpers out of the `task` package into a dedicated, dependency-light
leaf package `internal/ingestion/task/indexdoc`.
These functions are pure transforms (they only depend on
`common`/`utility`) and are not task-orchestration concerns:
- `NormalizeChunks`, `DeepCopyChunks` (was unexported `deepCopyChunks`),
`toChunkMaps` → `indexdoc/normalize.go`
- `ProcessChunksForPipeline`, `RenameTextToContentWithWeight`,
`GetEmbeddingTokenConsumption`, `cleanupConsumedChunkFields`,
`mergeChunkMetadata`, `processChunkPositions`,
`AggregateTableDocMetadata`, `resolveTableColumnConfig` →
`indexdoc/process.go`
- `AddPositions` → `indexdoc/position.go`
- `EmbeddingTokenConsumptionKey` constant → `indexdoc/constants.go`
(task/constants.go keeps only `GRAPH_RAPTOR_FAKE_DOC_ID`)
Call sites in `pipeline_executor.go` and `golden_compare.go` now
reference the `indexdoc` package; package-task tests qualify the moved
symbols.
## Why
The `task` package had grown into a "orchestration + pure mapping +
debug" mix. Splitting the pure mapping helpers into a leaf package
sharpens package boundaries, removes a misleading top-level
`ingestion/chunk` candidate (there are already `parser/chunk` and
`service/chunk`), and lets the golden tool / future reuse pull in the
mapping logic without dragging in `task`'s `dao`/`engine`/`service`
dependency graph (Go subpackage import does not pull in the parent).
## Test plan
- `build.sh --test ./internal/ingestion/task/...` — **green** (task
4.7s, indexdoc 0.007s), matching the pre-change baseline.
- `gofmt` clean; `build.sh` builds both `ragflow-cli` and
`ragflow_server` successfully.
- Integration/E2E tiers are delegated to CI (need real MySQL/MinIO/ES
services).
Note: `pipeline_e2e_test.go` has a **pre-existing** compile error
(`server.ElasticsearchConfig` / `server.InfinityConfig` are now defined
under `internal/server/config/`, not re-exported by `internal/server`).
This is unrelated to this change — the diff to that file is only the
added `indexdoc` import and the qualified `EmbeddingTokenConsumptionKey`
reference.
## Summary
Adds a self-contained regression guard for `splitOversizedUnitWith`
after PR #17729
aligned it with Python's `rag/nlp._split_oversized_unit` running-sum
flush.
#17729 shipped a `slack=1` relaxation in `token_strict_cap_test.go` (the
oversized
unit is now sub-split with the same running-sum flush Python uses, which
can leave a
piece one token over the nominal budget due to cl100k non-additivity).
This PR adds
the missing positive proof that the sub-split boundaries are correct, so
the relaxed
assertion is no longer unguarded.
## Changes
- `split_oversized_guard_test.go` (new, self-contained — no harness
loader, no
`testdata`; the Python oracle is inlined):
- `TestSplitOversizedUnitRunningSumMatchesPython`: asserts the exact
piece
boundaries (live tokenizer) match Python's `_split_oversized_unit`,
compensating
the `slack=1` relaxation from #17729.
- `TestSplitOversizedUnitDeadTokenizerCollapses`: asserts a
zero-counting tokenizer
collapses the B1 paragraph into exactly one chunk, catching a silently
dead encoder
that a non-empty-result check would miss.
## Notes
- Test function names are deliberately distinct from PR #17735's
`TestSplitOversizedUnitMatchesPython`, so the two PRs verify
independently and do not
conflict at merge time.
- PR #17735 (golden parity harness) is intentionally left unchanged per
the split plan.
## Test plan
`bash build.sh --test ./internal/ingestion/component/chunker/` — green,
including both
new tests.
Co-authored-by: CodeBuddy <noreply@cnb.cool>
## Summary
Relate to #17284. Completes the batch 5 migration of 7 OpenAI-compatible
drivers (`vllm`, `volcengine`, `xai`, `xiaomi`, `xinference`, `xunfei`,
`zhipu-ai`) onto the unified request/response helpers
(`doRequest`/`doStreamRequest` +
`HandleNonStreamingResponse`/`HandleStreamingResponse` +
`ParserConfig`), established by `deepseek` in #17634.
This branch is rebased on the current `pr/migrate-models-batch5` and
fixes the issues in the previous state of the PR.
Co-authored-by: Haruko386 <tryeverypossible@163.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
## Summary
RAGFlow's Go tokenizer silently returned **0 tokens for every string**
whenever the `cl100k_base` BPE table could not be loaded — which is the
normal case for an offline/air-gapped Go server. This PR makes the
loader resolve the table from disk (where RAGFlow actually ships it) and
fail loudly when it is genuinely missing.
## Root cause
`tiktoken-go`'s stock loader downloads the encoding table over HTTP and
caches it under `TIKTOKEN_CACHE_DIR`. That does not work for RAGFlow:
- `TIKTOKEN_CACHE_DIR` is exported **only inside the Python process**
(`common/token_utils.py`). `docker/entrypoint.sh` launches the Go binary
(`bin/ragflow_server`) from a shell, so the Go process never inherits
the variable.
- The Dockerfile *does* ship the table (under its sha1 name in the
working directory), but nothing told the Go side to look there.
- Reaching `openaipublic.blob.core.windows.net` at runtime is not an
option for air-gapped installs, and is unreliable where that host is
blocked.
The failure was **silent**: `NumTokensFromString` returns `0` when the
encoder fails to build, and a `sync.Once` memoizes that error for the
process lifetime. Every token count became `0`, so chunk merging never
crossed its token budget and an entire document collapsed into a single
chunk. Python has no such failure mode because its encoder is built at
import time (a missing table aborts startup instead of degrading).
## Fix
Register a local-only `BpeLoader` via `tiktoken.SetBpeLoader`
(`internal/tokenizer/bpe_loader.go`) that resolves the table from disk
**only**, in priority order:
1. `TIKTOKEN_CACHE_DIR` / `DATA_GYM_CACHE_DIR` (honored so operators who
already configured one keep working).
2. The working directory, the executable's directory, and all of their
ancestors — matching the Dockerfile layout (table under its sha1 name in
the install root).
3. A `ragflow_deps/<basename>` checkout produced by
`ragflow_deps/download_deps.py`.
It **never performs network I/O**. When nothing is found it returns an
error listing every path it tried (pointing at `download_deps.py` or
`TIKTOKEN_CACHE_DIR`), so a genuinely missing table fails loudly instead
of degrading to zero.
## Test plan
- `internal/tokenizer/bpe_loader_test.go` (unit tier, runs under `bash
build.sh --test ./internal/tokenizer/...`):
- Loader reads from `TIKTOKEN_CACHE_DIR`, `DATA_GYM_CACHE_DIR`, the
sha1-named file in the working dir, and the bundled `ragflow_deps/`
name.
- Explicit cache dir wins over the bundled vocab.
- A malformed table is reported as an error rather than skipped.
- A genuinely missing table reports the candidates it tried (no network
attempt).
- `NumTokensFromString` matches Python-derived anchors (`""`→0,
`"hello"`→1, `"hello world"`→2, `"hello, world!"`→4, `"世界"`→3, `"Hello
世界 🌍"`→8, `"RAGFlow"`→3).
## Notes
- `.github/workflows/tests.yml` currently excludes `internal/tokenizer`
from `go test`, so these tests do not run in CI. The tokenizer fix is
exercised in CI indirectly via the chunker package once a
token-count-sensitive parity case lands (tracked separately). Consider
including `internal/tokenizer` in CI as a follow-up.
- Supported deployments already ship the table (`download_deps.py` →
`ragflow_deps/cl100k_base.tiktoken`; Dockerfile → `<sha1>` in cwd), so
no `ENV` change is required for the fix to take effect. Setting `ENV
TIKTOKEN_CACHE_DIR` in the Dockerfile remains a cheap
belt-and-suspenders hardening that can be done separately.
🤖 Generated with [CodeBuddy Code](https://cnb.cool/codebuddy)
---------
Co-authored-by: CodeBuddy <noreply@codebuddy.ai>
Co-authored-by: CodeBuddy Code <noreply@cnb.cool>
Co-authored-by: CodeBuddy <noreply@tencent.com>
## Summary
Relate to #17284
Migrate 10 OpenAI-compatible drivers (`orcarouter`, `perplexity`,
`ppio`, `qiniu`, `ragcon`, `stepfun`, `togetherai`, `tokenhub`,
`tokenpony`, `upstage`) to use the unified response handlers
(`HandleNonStreamingResponse` / `HandleStreamingResponse`), following
the same pattern established by `deepseek` in #17634.
- Cut ~150 lines per driver (1436 lines removed, 62 added across 10
files).
- No functional changes — pure deduplication of HTTP plumbing.
- Each driver now routes through `baseModel.doRequest()` and
`HandleNonStreamingResponse()`.
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
## Summary
Align Go `splitOversizedUnitWith` with Python
`rag/nlp._split_oversized_unit` so the whitespace-atom sub-split
produces byte-identical chunk boundaries.
### Root cause of the divergence
cl100k token counting is **not additive across whitespace joins**
(`token(a)+token(b) != token(a+b)`). Go previously used the exact
joined-string fit check `countFn(current+atom) > budget`, while Python
accumulates a running sum `current_tokens + a_tokens > budget`. The two
formulas disagree by one atom at the boundary, so Go and Python emitted
the same chunk *count* but shifted *text*.
### Changes
- `splitOversizedUnitWith` (`token.go`): replace the exact joined-string
fit check with the running-sum check (mirroring Python's
`current_tokens` accumulator), and after a flush keep the overflow
whitespace atom (`current += atom`) instead of dropping it.
- `token_strict_cap_test.go`: relax
`TestMergeByTokenSizeFromJSON_OversizedUnitIsSubSplit` to allow the same
cl100k non-additive +1 overshoot Python exhibits (the invariant — an
oversized unit is sub-split, not collapsed — is preserved).
### Test plan
`bash build.sh --test ./internal/ingestion/component/chunker/...` —
green.
## Note
Test infrastructure for this change (golden parity harness,
`split_oversized_test.go`, `testdata/parity/**`, `known_diffs.json`,
`capture_golden.py`/`live_chunk.py`, and the `go-cmp` dependency
promotion) is split into a separate, stacked PR #17735 so this PR stays
minimal (production code only).
This PR is **independent of #17712** (the offline BPE loader). It is
based on `upstream/main` and contains only this change; no BPE-loader
code is included.
Co-authored-by: CodeBuddy <noreply@cnb.cool>
## Summary
Relate to #17284
Migrate 10 OpenAI-compatible drivers (`minimax`, `mistral`,
`modelscope`, `moonshot`, `n1n`, `novita`, `ollama`, `openai`,
`openai_api_compatible`, `openrouter`) to use the unified response
handlers (`HandleNonStreamingResponse` / `HandleStreamingResponse`),
following the same pattern established by `deepseek` in #17634.
- Cut ~150 lines per driver (1692 lines removed, 172 added across 10
files).
- `openai_api_compatible` gains `ChatWithMessages` +
`ChatStreamlyWithSender` required by the unified handler infrastructure.
- `openai` driver preserves `reasoning_content` extraction for o-series
models.
- All drivers: pure deduplication of HTTP plumbing.
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
## Summary
Relate to #17284
Migrate 10 OpenAI-compatible drivers (`gitee`, `gpustack`, `greenpt`,
`huaweicloud`, `huggingface`, `hunyuan`, `jiekouai`, `jina`, `lmstudio`,
`localai`) to use the unified response handlers
(`HandleNonStreamingResponse` / `HandleStreamingResponse`), following
the same pattern established by `deepseek` in #17634.
- Cut ~150 lines per driver (1398 lines removed, 155 added across 10
files).
- `greenpt` gains `ChatWithMessages` + `ChatStreamlyWithSender` required
by the unified handler infrastructure.
- All other drivers: pure deduplication of HTTP plumbing.
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthoric.com>
## Summary
Relate to #17284
Migrate 10 OpenAI-compatible drivers (`302ai`, `aliyun`, `astraflow`,
`avian`, `azure_openai`, `baichuan`, `baidu`, `cometapi`, `deepinfra`,
`futurmix`) to use the unified response handlers
(`HandleNonStreamingResponse` / `HandleStreamingResponse`), following
the same pattern established by `deepseek` in #17634.
- Cut ~150 lines per driver (1507 lines removed, 144 added across 10
files).
- No functional changes — pure deduplication of HTTP plumbing.
- Each driver now routes through `baseModel.doRequest()` and
`HandleNonStreamingResponse()`.
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Port Python rag/advanced_rag agentic search to Go: ES-backed dataset-nav
service, agentic-search harness, and agent tools.
Includes agentic-search port plan and self-review docs.
## Summary
Relate to #17284
Migrate four OpenAI-compatible drivers (`nvidia`, `siliconflow`, `groq`,
`longcat`) to use the unified response handlers
(`HandleNonStreamingResponse` / `HandleStreamingResponse`), following
the same pattern established by `deepseek` in #17634.
- Cut ~100 lines per driver (541 lines removed, 19 added across 4
files).
- `nvidia` now extracts token usage (previously had none).
- All four drivers produce the unified `StreamUsage` log.
## Summary
Six sites used to read the same `parser_config.delimiter` field with
divergent grammars:
- `rag.nlp.get_delimiters` (PDF/DOCX/HTML/EPUB/JSON/CSV/XLSX/email/book)
- `rag.nlp.naive_merge` (custom-delimiter branch)
- `rag.nlp.naive_merge_with_images`
- `rag.nlp._build_cks`
- `deepdoc.parser.txt_parser.parser_txt` (.txt, code)
-
`deepdoc.parser.markdown_parser.MarkdownElementExtractor.get_delimiters`
The six implementations disagreed on bare-vs-wrapped chars, dedupe, sort
order, CRLF normalization, and `re.I` (#17384). The shipped default ``
`\n!?;。；！？` `` was a no-op for `.md` because the markdown path only
matched backtick-wrapped tokens.
## Changes
- **new:** `rag/nlp/delim.py` with `parse_delimiter_field` and
`compile_delimiter_pattern`. Single source of truth. CRLF normalization
at the top; longest-first stable sort; insertion-ordered dedupe; no
`re.I`.
- **refactor:** all six call sites delegate to the helper.
- `rag/nlp/__init__.py::get_delimiters` becomes a thin shim.
- `deepdoc/parser/txt_parser.py::parser_txt` drops the
`[encode/decode/unicode_escape]` round-trip.
- `deepdoc/parser/markdown_parser.py::get_delimiters` honors bare chars
(fixes [1]).
- **tests:** `test/unit_test/rag/test_delim.py` (85 tests) — helper,
acceptance table, frontend parity, static guard against re-inlining.
- **tests:** `test/unit_test/rag/test_delimiter_case_sensitive.py` (from
#17386) updated to retarget the static check at the new helper +
AST-based broader scan.
## Acceptance criteria
- All six sites produce the same regex pattern for the same input.
- Shipped default keeps working for `.txt` / `.pdf` / `.docx`.
- Shipped default for `.md` now splits (was a silent no-op).
- Tooltip example `` `\n##;` `` produces three effective delimiters
regardless of file type.
- Bare whitespace inputs split on every occurrence.
- Backtick-wrapped whitespace splits only on the exact N-char sequence.
- CRLF-line-ending documents split identically to LF-line-ending
documents.
- 123 tests pass (85 new + 38 existing).
## Rebase protocol
As #17385 and #17386 evolve, this branch will be rebased on top. The
only overlap between this PR's diff and the other two is
`test_delimiter_case_sensitive.py`, where #17383 modifies the static
check to point at the new helper location.
---------
Co-authored-by: kiloconnect[bot] <240665456+kiloconnect[bot]@users.noreply.github.com>
## Summary
This PR refactors the Go knowledge-compilation ingestion pipeline
(`internal/ingestion/knowledge_compile` +
`internal/ingestion/component/knowledge_compiler`) with three related
changes:
- **Token-budget batching for LLM merge decisions.**
`LLMMergeDecider.DecideBatch` previously stuffed every `(existing,
candidate)` pair into a single LLM call, risking `max_token` overflow.
It now splits pairs into token-bounded sub-batches (budget =
`llmMaxTokens * 0.85`) via `tokenizer.NumTokensFromString`, runs them
concurrently while preserving the global pair index, and never
reindexes.
- **Process-level global compile pool.** Introduces a single vCPU-sized
goroutine pool (`pool.go`, env `KC_COMPILE_CONCURRENCY`) dedicated to
*all* knowledge-compilation stages. KNN search loop, `DecideBatch`
sub-batches, `WriteMerged`/`DeleteMerged` internals, and the
component-level (structure/mindmap) per-call pools are all unified into
it via an injected submitter. No more per-job short-lived goroutines in
`runCompilerJobs` (futures are collected then awaited on the caller).
Fan-out stays bounded by the pool worker count; these stages are
docengine-bounded / LLM-bounded, not CPU-bounded.
- **DocEngine-only deletion.** `Consumer.processBatch` deletion no
longer loads the deleted docs' products into memory. Two sequential
DocEngine calls replace the old in-memory surgery:
- `DeleteDocLevelForDocs` — one `DeleteChunks` over `doc_id IN
deletedDocIDs` (merged rows carry `doc_id == kb`, so only per-doc
products match).
- `StripMergedSources` — one `Search` of `kc_merged=1` rows filtered by
`source_doc_ids IN deletedDocIDs` (intersection pushed down to the
engine), `UpdateChunks` the source array of survivors, and
`DeleteChunks` the rows whose array became empty.
## Changes
- `internal/ingestion/knowledge_compile/pool.go` (new): global
`compilerPool` +
`runCompilerJobs`/`SubmitCompilerJob`/`SubmitCompilerJobs`.
- `internal/ingestion/knowledge_compile/consumer.go`: deletion rewritten
to the two DocEngine calls;
`mergedBase`/`toDelete`/`stripDeletedSources` removed.
- `internal/ingestion/knowledge_compile/writer.go`:
`DeleteDocLevelForDocs` + `StripMergedSources` replace
`DeleteMergedForDoc`/`DeleteMerged`.
- `internal/ingestion/knowledge_compile/reader.go`: drop
`LoadMergedBySourceDoc` + `containsString` (keep `LoadDocProducts` for
the completion branch).
- `internal/ingestion/knowledge_compile/dedup.go`: `NewLLMDeduper` takes
`llmMaxTokens`; wires `SetMaxBatchTokens`/`SetSubmitter`.
- `internal/ingestion/knowledge_compiler/{structure,merge}.go`,
`mindmap/mindmap.go`, `pool_wiring.go`: token-budget split + submitter
injection.
- Tests: `structure_test.go` (token-budget split), `dedup_test.go`,
`consumer_test.go` (tombstone + DocEngine deletion assertions) updated.
## Validation
`bash build.sh --test -race ./internal/ingestion/knowledge_compile/...
./internal/ingestion/component/knowledge_compiler/...` passes (unit
tier, no external services).
🤖 Generated with [CodeBuddy](https://www.codebuddy.ai)
---------
Co-authored-by: yuzhichang <yuzhichang@infiniflow.ai>
Fixes#17202 (and complements #12109).
## Problem
`RAGFlowTxtParser.parser_txt` (`deepdoc/parser/txt_parser.py:36-47`) and
`rag.nlp.naive_merge` (`rag/nlp/__init__.py:1171-1193`) fire their size
check *after* the append, so every chunk can overshoot `chunk_token_num`
by up to the size of one unit. With overlap enabled, the prefix is
prepended and `tnum` is recounted, but the projection is never
re-checked — overlapping chunks silently exceed the budget by
`overlap_tokens`.
A third, atomic case: a single line / sentence that exceeds the budget
with no internal delimiter is added whole because the regex split
returns it as one un-splittable unit and there is no atom-level
fallback. `RAGFlowHtmlParser.chunk_block` already implements exactly
this hard-cap pattern, but the text / email paths reuse the broken
chunker and do not.
Measured on a live dataset (336 `.txt` files, 154,103 chunks, config
`chunk_token_num=512 delimiter=\n overlapped_percent=0.1`): 56.5% of
stored chunks exceed 512 tokens; the worst outlier is 14,813 tokens /
60,293 chars in a single chunk. Symptom downstream: rerank failures on
the >2048-token outliers (ref. #12109) and silent embedding truncation
on every oversize chunk.
## Fix
Mirror the proven pattern in `RAGFlowHtmlParser.chunk_block`:
1. **Proactive projected-total check** in `TxtParser.parser_txt` and in
`naive_merge.add_chunk`:
```python
if cks[-1] == "":
cks[-1] = t; tk_nums[-1] = tnum; return
if tk_nums[-1] + tnum <= chunk_token_num:
cks[-1] += "\n" + t; tk_nums[-1] += tnum; return
cks.append(t); tk_nums.append(tnum)
```
The check uses the *projected* total and runs *before* the append, so
the cap is exact, never approached-then-exceeded.
2. **Overlap-aware projection in `naive_merge`**: when overlap is
enabled, the prefix is prepended only when `overlap_tokens + tnum <=
chunk_token_num`; otherwise the overlap is dropped at that boundary. The
naive_merge-with-images mirror gets the same treatment. Custom-delimiter
behaviour is preserved per the existing test suite.
3. **Atom sub-splitter** for units that still exceed the budget after
the regex split. Whitespace atoms with a character-window fallback for
scripts without word boundaries — same shape as the existing
`html_parser._split_oversized_block`, so behaviour matches for HTML vs
`.txt` vs PDF atomic-oversize.
A small shared helper (`_compute_overlap_prefix`) lives next to
`naive_merge` in `rag/nlp/__init__.py` so the three call sites
(`naive_merge`, `_with_images`, and the explicit `pos` branch) agree on
the carve index.
## Result on the dataset above
| | Before | After |
|---|---|---|
| Chunks > 512 tokens | 56.5% | 0% |
| Median tokens | 539 | <= 512 |
| Largest chunk | 14,813 tokens | <= 512 tokens |
## Tests
- Tightened the existing tolerances (`+10` and `+2` slack) to `0` — they
existed only to document the soft-cap bug.
- Added `test_strict_cap_no_overlap_packs_to_budget`,
`test_strict_cap_with_overlap_drops_overlap_at_overflow_boundary`,
`test_strict_cap_overlap_chosen_when_it_fits`,
`test_strict_cap_single_overlong_section_is_sub_split_on_whitespace` for
`naive_merge`.
- Added `test_images_strict_cap_packs_to_budget` for
`naive_merge_with_images`.
- New `test/unit_test/deepdoc/parser/test_txt_parser.py` covers
`parser_txt` strict cap and atom sub-split. Uses the same path-loading
pattern as the existing `test_html_parser.py` to avoid pulling the deep
import chain into a test-time-only venv.
All 22 unit tests pass on the host venv:
```
test_naive_merge.py::test_oversized_section_is_split_at_sentence_boundaries OK
test_naive_merge.py::test_small_sections_are_merged_not_oversplit OK
test_naive_merge.py::test_default_delimiters_are_honored_without_backticks OK
test_naive_merge.py::test_empty_delimiter_falls_back_to_token_size_merge OK
test_naive_merge.py::test_overlap_prefix_is_counted_in_token_budget OK
test_naive_merge.py::test_custom_delimiter_ignores_chunk_size OK
test_naive_merge.py::test_custom_delimiter_does_not_size_merge OK
test_naive_merge.py::test_images_oversized_section_is_split OK
test_naive_merge.py::test_images_custom_delimiter_preserved OK
test_naive_merge.py::test_images_plain_string_input OK
test_naive_merge.py::test_images_mismatched_lengths_returns_empty OK
test_naive_merge.py::test_images_shared_lazyimage_not_stacked_… OK
test_naive_merge.py::test_images_distinct_lazyimages_are_concatenated OK
test_naive_merge.py::test_strict_cap_no_overlap_packs_to_budget OK
test_naive_merge.py::test_strict_cap_with_overlap_drops_… OK
test_naive_merge.py::test_strict_cap_single_overlong_section_… OK
test_naive_merge.py::test_strict_cap_overlap_chosen_when_it_fits OK
test_naive_merge.py::test_images_strict_cap_packs_to_budget OK
test_txt_parser.py::test_no_overshoot_when_packing_short_lines OK
test_txt_parser.py::test_no_overshoot_at_chunk_boundary OK
test_txt_parser.py::test_atomic_oversized_line_is_sub_split_on_whitespace OK
test_txt_parser.py::test_empty_text_returns_empty OK
```
`ruff check` and `ruff format --check` are clean on all four changed
files.
## Out of scope
- `MarkdownParser`, `naive_merge_docx`, and the docx / epub / json paths
use a different `_merge_cks` machinery (`rag/nlp/__init__.py:1574`) that
already enforces the budget. They are unchanged.
- The `chunk_block` call sites in `deepdoc/parser/html_parser.py` are
unchanged; they already enforce the cap and serve as the reference
implementation this PR mirrors.
Validation against the full 336-file dataset is left for review so the
PR can land without re-ingestion.
---------
Co-authored-by: skbs-eng <skbs-eng@users.noreply.github.com>
Co-authored-by: kiloconnect[bot] <240665456+kiloconnect[bot]@users.noreply.github.com>
Closes#17384.
## Summary
Drops a dead `re.I` flag from two outlier delimiter-parsing sites and
adds regression tests so the inconsistency can't creep back.
## What's wrong
Two of the six delimiter-parsing implementations pass `re.I` to
`re.finditer`:
- `rag/nlp/__init__.py::get_delimiters` (line 1633)
- `deepdoc/parser/txt_parser.py::parser_txt` (line 51)
The other four implementations correctly omit `re.I`:
- `rag/nlp/__init__.py::naive_merge` custom-delimiter path (line 1195)
- `rag/nlp/__init__.py::naive_merge_with_images` custom-delimiter path
(line 1269)
- `rag/nlp/__init__.py::_build_cks` (line 1389)
- `rag/flow/chunker/token_chunker.py` (line 73)
## Why this matters (and why it doesn't break anything)
The flag is **dead code** today. Verified empirically with a Python
REPL:
```python
>>> import re
>>> for m in re.finditer(r"`([^`]+)`", "`end`", re.I):
... print(repr(m.group(1)))
'end' # plain string, no flag attached
>>> re.split("(a)", "Class A is a Sample")
['Cl', 'a', '', 's', ' A i', 's', ' a Sample']
# Case-sensitive: only lowercase 'a' splits. Uppercase 'A' is preserved.
```
`re.I` does not propagate from `re.finditer` to `m.group(1)` or to
downstream `re.split` / `re.match` calls (which all omit `re.I`). So the
actual splitting behavior has always been case-sensitive — removing the
flag is a **defensive cleanup**, not a behavioral fix.
So why bother?
1. **Consistency** — the two sites were the only outliers in a six-way
implementation cluster. The three sibling sites in `rag/nlp/__init__.py`
already omit `re.I`, which strongly suggests the flag was accidental.
2. **Future-proofing** — a refactor could easily propagate the flag to a
downstream `re.split` call where it *would* change behavior. The tests
added here pin the case-sensitive semantics so that regression fails
loudly.
3. **Reader clarity** — the flag is misleading. Anyone reading
`re.finditer(..., re.I)` reasonably assumes case-insensitive matching,
then has to trace all downstream calls to discover it's a no-op.
## Changes
- `rag/nlp/__init__.py` — drop `re.I` from `get_delimiters` (line 1633).
- `deepdoc/parser/txt_parser.py` — drop `re.I` from `parser_txt` (line
51).
- `test/unit_test/rag/test_delimiter_case_sensitive.py` — new test file
with:
- 4 behavioral tests on `get_delimiters` (pattern output + `re.split`
round-trip).
- 3 end-to-end tests through `naive_merge` (bare-char +
backtick-wrapped, both cases).
- 2 parametrized static checks that `re.I` / `re.IGNORECASE` is not
present at either of the two `re.finditer` sites.
## Testing
```
$ pytest test/unit_test/rag/test_delimiter_case_sensitive.py -v
============================= 9 passed in 0.19s ==============================
```
All tests pass on the patched code. Before the patch, the 2 static
checks fail with a clear assertion message (the 7 behavioral tests pass
either way, confirming `re.I` was dead code).
## Related
- #17384 — the issue this PR closes. Note the issue's reproduction code
(`re.split(..., flags=re.I)`) doesn't actually match what the production
code does — the production `re.split` calls all omit `re.I`, which is
why current behavior is already case-sensitive. The fix here is still
valuable as a defensive cleanup + test coverage, but it's not a
behavioral fix per se.
- #17383 — broader parser consolidation (six implementations → one). The
fix here is independent and small enough to land first.
- #17385 — sibling UX PR (tooltip + live preview). Files are disjoint
(`web/src/**` vs `rag/nlp/**` + `deepdoc/parser/**`), so no interaction.
---------
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: kiloconnect[bot] <240665456+kiloconnect[bot]@users.noreply.github.com>
## Summary
Refactor the Go `KnowledgeCompilerComponent` so its parameter is a
**single string template id** instead of a DSL-level `variant` (or
plural group id list). The `variant` is no longer in the DSL — it is now
**derived at runtime from the resolved compilation template's `kind`
field**.
This aligns the Go ingestion port with the frontend Compiler operator,
which emits a singular `compilation_template_group_id` and does not
write `variant` into the generated `compiler.json`.
