## Problem
`kg_test.go`'s `getTestConfig()` returned `map[string]interface{}`,
which no longer matches `NewEngine`'s `config.ElasticsearchConfig`
signature after the config package moved to `internal/server/config`.
This broke compilation of the `elasticsearch` package under `-tags
integration` (kg_test.go:38).
## Fix
Make `getTestConfig()` return the typed `config.ElasticsearchConfig` so
the integration tier builds again.
## Scope
Fixes test compilation only; no production code changed. Unrelated to PR
#17802 (kb_id single-string + T0 read-back baseline), so it is tracked
in its own PR to keep that refactor focused.
🤖 Generated with [CodeBuddy Code](https://cnb.cool/codebuddy)
## Summary
- `ProcessChunksForPipeline` now sets `kb_id` to a plain string instead
of `[]string{kbID}`, removing an index-physical array shape from the
ingestion domain.
- Stored documents are byte-identical: Elasticsearch overrides `kb_id`
with `datasetID` on write, and Infinity's `transformChunkFields` already
accepts a plain string.
- Infinity is intentionally left unchanged — `service/chunk` paths still
feed `kb_id` as `[]string`, and Infinity handles both forms. The
`dataset` artifact merge (`dataset_artifact_service.go`) is out of scope
for this step.
- Unit assertion updated to expect a string.
## Scope / non-goals
This is the smallest first step (T1) of the index-schema leak cleanup
tracked in #17371. It does **not** move the other leaks (`docnm_kwd`,
`create_timestamp_flt`, position ints) to the engine boundary — those
are later steps behind a read-back golden test.
## Test plan
- `go test ./internal/ingestion/task/indexdoc/...` passes.
- The two `task` "Real" integration tests fail identically on a clean
tree (environment lacks real embedding/parsing); they are pre-existing,
unrelated to this change.
🤖 Generated with [CodeBuddy Code](https://cnb.cool/codebuddy)
Relate to #17284.
## Summary
Batch 5/6 migrated the rest of the Go model drivers onto the shared HTTP
helpers (`doRequest`, `doStreamRequest`, `applyAuth`). This PR completes
the batch for the remaining OpenAI-compatible chat-streaming drivers
that were still hand-writing HTTP requests:
- **7 drop-in migrations**: deepseek, gpustack, groq, longcat, moonshot,
openai, siliconflow
- **1 adapter migration**: minimax (relocated its `io.Pipe`
error-interception into the `doStreamRequest` handler)
- **1 full migration**: azure_openai (all four paths: chat, streaming,
embeddings, list-models) plus the auth header hook
- **1 receiver fix**: nvidia `NewInstance` value → pointer
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- Start the Agent model-stream collector before ReAct execution.
- Preserve all streamed reasoning/content deltas and drain the collector
on errors.
- Add coverage for delayed thinking streams and tool-call execution.
Relate to #17284.
## Problem
`novitaHandleStream` guarded usage recording with `if found &&
chatConfig != nil`. When a caller passes a nil `*ChatConfig` — common in
the service layer (`model_chat.go`, `chat_pipeline.go`) — the streamed
token usage is dropped entirely.
The shared `HandleStreamingResponse` only uses `chatConfig` to expose
`UsageResult` and records usage whenever the stream carries it. Novita's
bespoke handler diverged from every other OpenAI-compatible streaming
driver.
## Fix
Record usage whenever the stream carries a usage event, mirroring
`HandleStreamingResponse`. `applyStreamUsage` already handles a nil
`chatConfig` internally (it only writes `chatConfig.UsageResult` when
non-nil), so the extra guard was doing nothing but dropping usage.
## Test
`TestNovitaStreamRecordsUsageWithoutChatConfig`:
- nil `chatConfig` + usage event → stream completes without error (guard
removed safely)
- non-nil `chatConfig` + usage event → `UsageResult` populated with the
streamed tokens
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
## What
Adds [**SereneDB**](https://serenedb.com) as a selectable doc-store
engine on **both** RAGFlow paths:
- the **Go** `DocEngine` (`internal/engine/serenedb`), alongside
Elasticsearch and Infinity;
- the **Python** `DocStoreConnection` (`rag/utils/serenedb_conn.py`) +
`DOC_ENGINE=serenedb` registration.
SereneDB is a PostgreSQL-wire engine (DuckDB execution) whose single
inverted index carries **both** a scored text column (`@@`, BM25) and an
IVF vector column (`<#>`, inner product), so hybrid search is one SQL
statement. The Go engine connects with `database/sql` + `lib/pq`
(already a dependency, no new module); the Python connector uses
psycopg2 (already a dependency).
## Storage model
One table per tenant with `kb_id` as a filter column - the
**Elasticsearch / OceanBase** model, not Infinity's per-dataset tables.
This keeps BM25 statistics (IDF, avgdl) computed over the whole tenant
corpus (global IDF). Both connectors use this identical layout, so they
are storage- and retrieval-compatible: `hybrid` proxy routing and
Python↔Go switching are safe. On the Python side the connector is wired
as OceanBase's plain-SQL sibling (chunk_data JSON metadata, inline chunk
vectors, verbatim ES field names); the ES tokenizer path is unchanged.
Metadata stays one table per tenant (`ragflow_doc_meta_<tenant>`).
The query shapes mirror the Python connector, including the five
empirically-found landmines: the scored dictionary needs `frequency +
norm` (else `BM25()` silently returns 0.0), the `@@` query is the
tokenized query, the scored lexical branch matches one column, vectors
use an L2-normalized shadow column with `ip`/`sq8`, and the similarity
threshold goes directly in the ANN scan's `WHERE`. **Minimum engine
version: SereneDB 26.07.4.**
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
## Summary
Relate to #17284.
Completes the migration of the four non-OpenAI-compatible model drivers
(`anthropic`, `cohere`, `google`, `bedrock`) onto the shared
usage-recording path. Earlier batches (#17634, #17643, #17696–#17700)
covered only the OpenAI-compatible cluster; these four providers ship
wire formats that do not fit the OpenAI `choices[0].delta` / `usage`
block template and so were left for a separate pass.
Per the maintainer's guidance for this batch, each driver is migrated on
its own terms rather than forced through a single template. The shared
machinery used is intentionally small: `recordResponseUsage`,
`parseChatCompletionResponse`, `BaseModel.newJSONPostRequest`, and the
existing `authHeader` hook for non-Bearer auth.
Co-authored-by: Haruko386 <tryeverypossible@163.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
## Summary
DeepSeek and Azure OpenAI require `stream_options.include_usage=true` to
return token usage in streaming responses. Without it, all streaming
calls report zero usage to ClickHouse and the UI shows no token stats.
- [x] Verify DeepSeek streaming calls now report usage
- [x] Verify Azure OpenAI streaming calls now report usage
## Summary
- Normalize Canvas component names before resolving Go Agent tools and
parameters.
- Add regression coverage for CodeExec and other Canvas tool mappings.
## Testing
- `CGO_ENABLED=0 go test -count=1 ./internal/agent/tool
./internal/agent/component`
## Summary
Extract the pipeline-output → search-engine index document mapping
helpers out of the `task` package into a dedicated, dependency-light
leaf package `internal/ingestion/task/indexdoc`.
These functions are pure transforms (they only depend on
`common`/`utility`) and are not task-orchestration concerns:
- `NormalizeChunks`, `DeepCopyChunks` (was unexported `deepCopyChunks`),
`toChunkMaps` → `indexdoc/normalize.go`
- `ProcessChunksForPipeline`, `RenameTextToContentWithWeight`,
`GetEmbeddingTokenConsumption`, `cleanupConsumedChunkFields`,
`mergeChunkMetadata`, `processChunkPositions`,
`AggregateTableDocMetadata`, `resolveTableColumnConfig` →
`indexdoc/process.go`
- `AddPositions` → `indexdoc/position.go`
- `EmbeddingTokenConsumptionKey` constant → `indexdoc/constants.go`
(task/constants.go keeps only `GRAPH_RAPTOR_FAKE_DOC_ID`)
Call sites in `pipeline_executor.go` and `golden_compare.go` now
reference the `indexdoc` package; package-task tests qualify the moved
symbols.
## Why
The `task` package had grown into a "orchestration + pure mapping +
debug" mix. Splitting the pure mapping helpers into a leaf package
sharpens package boundaries, removes a misleading top-level
`ingestion/chunk` candidate (there are already `parser/chunk` and
`service/chunk`), and lets the golden tool / future reuse pull in the
mapping logic without dragging in `task`'s `dao`/`engine`/`service`
dependency graph (Go subpackage import does not pull in the parent).
## Test plan
- `build.sh --test ./internal/ingestion/task/...` — **green** (task
4.7s, indexdoc 0.007s), matching the pre-change baseline.
- `gofmt` clean; `build.sh` builds both `ragflow-cli` and
`ragflow_server` successfully.
- Integration/E2E tiers are delegated to CI (need real MySQL/MinIO/ES
services).
Note: `pipeline_e2e_test.go` has a **pre-existing** compile error
(`server.ElasticsearchConfig` / `server.InfinityConfig` are now defined
under `internal/server/config/`, not re-exported by `internal/server`).
This is unrelated to this change — the diff to that file is only the
added `indexdoc` import and the qualified `EmbeddingTokenConsumptionKey`
reference.
## Summary
Adds a self-contained regression guard for `splitOversizedUnitWith`
after PR #17729
aligned it with Python's `rag/nlp._split_oversized_unit` running-sum
flush.
#17729 shipped a `slack=1` relaxation in `token_strict_cap_test.go` (the
oversized
unit is now sub-split with the same running-sum flush Python uses, which
can leave a
piece one token over the nominal budget due to cl100k non-additivity).
This PR adds
the missing positive proof that the sub-split boundaries are correct, so
the relaxed
assertion is no longer unguarded.
## Changes
- `split_oversized_guard_test.go` (new, self-contained — no harness
loader, no
`testdata`; the Python oracle is inlined):
- `TestSplitOversizedUnitRunningSumMatchesPython`: asserts the exact
piece
boundaries (live tokenizer) match Python's `_split_oversized_unit`,
compensating
the `slack=1` relaxation from #17729.
- `TestSplitOversizedUnitDeadTokenizerCollapses`: asserts a
zero-counting tokenizer
collapses the B1 paragraph into exactly one chunk, catching a silently
dead encoder
that a non-empty-result check would miss.
## Notes
- Test function names are deliberately distinct from PR #17735's
`TestSplitOversizedUnitMatchesPython`, so the two PRs verify
independently and do not
conflict at merge time.
- PR #17735 (golden parity harness) is intentionally left unchanged per
the split plan.
## Test plan
`bash build.sh --test ./internal/ingestion/component/chunker/` — green,
including both
new tests.
Co-authored-by: CodeBuddy <noreply@cnb.cool>
## Summary
Relate to #17284. Completes the batch 5 migration of 7 OpenAI-compatible
drivers (`vllm`, `volcengine`, `xai`, `xiaomi`, `xinference`, `xunfei`,
`zhipu-ai`) onto the unified request/response helpers
(`doRequest`/`doStreamRequest` +
`HandleNonStreamingResponse`/`HandleStreamingResponse` +
`ParserConfig`), established by `deepseek` in #17634.
This branch is rebased on the current `pr/migrate-models-batch5` and
fixes the issues in the previous state of the PR.
Co-authored-by: Haruko386 <tryeverypossible@163.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
## Summary
RAGFlow's Go tokenizer silently returned **0 tokens for every string**
whenever the `cl100k_base` BPE table could not be loaded — which is the
normal case for an offline/air-gapped Go server. This PR makes the
loader resolve the table from disk (where RAGFlow actually ships it) and
fail loudly when it is genuinely missing.
## Root cause
`tiktoken-go`'s stock loader downloads the encoding table over HTTP and
caches it under `TIKTOKEN_CACHE_DIR`. That does not work for RAGFlow:
- `TIKTOKEN_CACHE_DIR` is exported **only inside the Python process**
(`common/token_utils.py`). `docker/entrypoint.sh` launches the Go binary
(`bin/ragflow_server`) from a shell, so the Go process never inherits
the variable.
- The Dockerfile *does* ship the table (under its sha1 name in the
working directory), but nothing told the Go side to look there.
- Reaching `openaipublic.blob.core.windows.net` at runtime is not an
option for air-gapped installs, and is unreliable where that host is
blocked.
The failure was **silent**: `NumTokensFromString` returns `0` when the
encoder fails to build, and a `sync.Once` memoizes that error for the
process lifetime. Every token count became `0`, so chunk merging never
crossed its token budget and an entire document collapsed into a single
chunk. Python has no such failure mode because its encoder is built at
import time (a missing table aborts startup instead of degrading).
## Fix
Register a local-only `BpeLoader` via `tiktoken.SetBpeLoader`
(`internal/tokenizer/bpe_loader.go`) that resolves the table from disk
**only**, in priority order:
1. `TIKTOKEN_CACHE_DIR` / `DATA_GYM_CACHE_DIR` (honored so operators who
already configured one keep working).
2. The working directory, the executable's directory, and all of their
ancestors — matching the Dockerfile layout (table under its sha1 name in
the install root).
3. A `ragflow_deps/<basename>` checkout produced by
`ragflow_deps/download_deps.py`.
It **never performs network I/O**. When nothing is found it returns an
error listing every path it tried (pointing at `download_deps.py` or
`TIKTOKEN_CACHE_DIR`), so a genuinely missing table fails loudly instead
of degrading to zero.
## Test plan
- `internal/tokenizer/bpe_loader_test.go` (unit tier, runs under `bash
build.sh --test ./internal/tokenizer/...`):
- Loader reads from `TIKTOKEN_CACHE_DIR`, `DATA_GYM_CACHE_DIR`, the
sha1-named file in the working dir, and the bundled `ragflow_deps/`
name.
- Explicit cache dir wins over the bundled vocab.
- A malformed table is reported as an error rather than skipped.
- A genuinely missing table reports the candidates it tried (no network
attempt).
- `NumTokensFromString` matches Python-derived anchors (`""`→0,
`"hello"`→1, `"hello world"`→2, `"hello, world!"`→4, `"世界"`→3, `"Hello
世界 🌍"`→8, `"RAGFlow"`→3).
## Notes
- `.github/workflows/tests.yml` currently excludes `internal/tokenizer`
from `go test`, so these tests do not run in CI. The tokenizer fix is
exercised in CI indirectly via the chunker package once a
token-count-sensitive parity case lands (tracked separately). Consider
including `internal/tokenizer` in CI as a follow-up.
- Supported deployments already ship the table (`download_deps.py` →
`ragflow_deps/cl100k_base.tiktoken`; Dockerfile → `<sha1>` in cwd), so
no `ENV` change is required for the fix to take effect. Setting `ENV
TIKTOKEN_CACHE_DIR` in the Dockerfile remains a cheap
belt-and-suspenders hardening that can be done separately.
🤖 Generated with [CodeBuddy Code](https://cnb.cool/codebuddy)
---------
Co-authored-by: CodeBuddy <noreply@codebuddy.ai>
Co-authored-by: CodeBuddy Code <noreply@cnb.cool>
Co-authored-by: CodeBuddy <noreply@tencent.com>
## Summary
Relate to #17284
Migrate 10 OpenAI-compatible drivers (`orcarouter`, `perplexity`,
`ppio`, `qiniu`, `ragcon`, `stepfun`, `togetherai`, `tokenhub`,
`tokenpony`, `upstage`) to use the unified response handlers
(`HandleNonStreamingResponse` / `HandleStreamingResponse`), following
the same pattern established by `deepseek` in #17634.
- Cut ~150 lines per driver (1436 lines removed, 62 added across 10
files).
- No functional changes — pure deduplication of HTTP plumbing.
- Each driver now routes through `baseModel.doRequest()` and
`HandleNonStreamingResponse()`.
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
## Summary
Align Go `splitOversizedUnitWith` with Python
`rag/nlp._split_oversized_unit` so the whitespace-atom sub-split
produces byte-identical chunk boundaries.
### Root cause of the divergence
cl100k token counting is **not additive across whitespace joins**
(`token(a)+token(b) != token(a+b)`). Go previously used the exact
joined-string fit check `countFn(current+atom) > budget`, while Python
accumulates a running sum `current_tokens + a_tokens > budget`. The two
formulas disagree by one atom at the boundary, so Go and Python emitted
the same chunk *count* but shifted *text*.
### Changes
- `splitOversizedUnitWith` (`token.go`): replace the exact joined-string
fit check with the running-sum check (mirroring Python's
`current_tokens` accumulator), and after a flush keep the overflow
whitespace atom (`current += atom`) instead of dropping it.
- `token_strict_cap_test.go`: relax
`TestMergeByTokenSizeFromJSON_OversizedUnitIsSubSplit` to allow the same
cl100k non-additive +1 overshoot Python exhibits (the invariant — an
oversized unit is sub-split, not collapsed — is preserved).
### Test plan
`bash build.sh --test ./internal/ingestion/component/chunker/...` —
green.
## Note
Test infrastructure for this change (golden parity harness,
`split_oversized_test.go`, `testdata/parity/**`, `known_diffs.json`,
`capture_golden.py`/`live_chunk.py`, and the `go-cmp` dependency
promotion) is split into a separate, stacked PR #17735 so this PR stays
minimal (production code only).
This PR is **independent of #17712** (the offline BPE loader). It is
based on `upstream/main` and contains only this change; no BPE-loader
code is included.
Co-authored-by: CodeBuddy <noreply@cnb.cool>
## Summary
Relate to #17284
Migrate 10 OpenAI-compatible drivers (`minimax`, `mistral`,
`modelscope`, `moonshot`, `n1n`, `novita`, `ollama`, `openai`,
`openai_api_compatible`, `openrouter`) to use the unified response
handlers (`HandleNonStreamingResponse` / `HandleStreamingResponse`),
following the same pattern established by `deepseek` in #17634.
- Cut ~150 lines per driver (1692 lines removed, 172 added across 10
files).
- `openai_api_compatible` gains `ChatWithMessages` +
`ChatStreamlyWithSender` required by the unified handler infrastructure.
- `openai` driver preserves `reasoning_content` extraction for o-series
models.
- All drivers: pure deduplication of HTTP plumbing.
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
## Summary
Relate to #17284
Migrate 10 OpenAI-compatible drivers (`gitee`, `gpustack`, `greenpt`,
`huaweicloud`, `huggingface`, `hunyuan`, `jiekouai`, `jina`, `lmstudio`,
`localai`) to use the unified response handlers
(`HandleNonStreamingResponse` / `HandleStreamingResponse`), following
the same pattern established by `deepseek` in #17634.
- Cut ~150 lines per driver (1398 lines removed, 155 added across 10
files).
- `greenpt` gains `ChatWithMessages` + `ChatStreamlyWithSender` required
by the unified handler infrastructure.
- All other drivers: pure deduplication of HTTP plumbing.
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthoric.com>
## Summary
Relate to #17284
Migrate 10 OpenAI-compatible drivers (`302ai`, `aliyun`, `astraflow`,
`avian`, `azure_openai`, `baichuan`, `baidu`, `cometapi`, `deepinfra`,
`futurmix`) to use the unified response handlers
(`HandleNonStreamingResponse` / `HandleStreamingResponse`), following
the same pattern established by `deepseek` in #17634.
- Cut ~150 lines per driver (1507 lines removed, 144 added across 10
files).
- No functional changes — pure deduplication of HTTP plumbing.
- Each driver now routes through `baseModel.doRequest()` and
`HandleNonStreamingResponse()`.
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Port Python rag/advanced_rag agentic search to Go: ES-backed dataset-nav
service, agentic-search harness, and agent tools.
Includes agentic-search port plan and self-review docs.
## Summary
Relate to #17284
Migrate four OpenAI-compatible drivers (`nvidia`, `siliconflow`, `groq`,
`longcat`) to use the unified response handlers
(`HandleNonStreamingResponse` / `HandleStreamingResponse`), following
the same pattern established by `deepseek` in #17634.
- Cut ~100 lines per driver (541 lines removed, 19 added across 4
files).
- `nvidia` now extracts token usage (previously had none).
- All four drivers produce the unified `StreamUsage` log.
## Summary
Six sites used to read the same `parser_config.delimiter` field with
divergent grammars:
- `rag.nlp.get_delimiters` (PDF/DOCX/HTML/EPUB/JSON/CSV/XLSX/email/book)
- `rag.nlp.naive_merge` (custom-delimiter branch)
- `rag.nlp.naive_merge_with_images`
- `rag.nlp._build_cks`
- `deepdoc.parser.txt_parser.parser_txt` (.txt, code)
-
`deepdoc.parser.markdown_parser.MarkdownElementExtractor.get_delimiters`
The six implementations disagreed on bare-vs-wrapped chars, dedupe, sort
order, CRLF normalization, and `re.I` (#17384). The shipped default ``
`\n!?;。；！？` `` was a no-op for `.md` because the markdown path only
matched backtick-wrapped tokens.
## Changes
- **new:** `rag/nlp/delim.py` with `parse_delimiter_field` and
`compile_delimiter_pattern`. Single source of truth. CRLF normalization
at the top; longest-first stable sort; insertion-ordered dedupe; no
`re.I`.
- **refactor:** all six call sites delegate to the helper.
- `rag/nlp/__init__.py::get_delimiters` becomes a thin shim.
- `deepdoc/parser/txt_parser.py::parser_txt` drops the
`[encode/decode/unicode_escape]` round-trip.
- `deepdoc/parser/markdown_parser.py::get_delimiters` honors bare chars
(fixes [1]).
- **tests:** `test/unit_test/rag/test_delim.py` (85 tests) — helper,
acceptance table, frontend parity, static guard against re-inlining.
- **tests:** `test/unit_test/rag/test_delimiter_case_sensitive.py` (from
#17386) updated to retarget the static check at the new helper +
AST-based broader scan.
## Acceptance criteria
- All six sites produce the same regex pattern for the same input.
- Shipped default keeps working for `.txt` / `.pdf` / `.docx`.
- Shipped default for `.md` now splits (was a silent no-op).
- Tooltip example `` `\n##;` `` produces three effective delimiters
regardless of file type.
- Bare whitespace inputs split on every occurrence.
- Backtick-wrapped whitespace splits only on the exact N-char sequence.
- CRLF-line-ending documents split identically to LF-line-ending
documents.
- 123 tests pass (85 new + 38 existing).
## Rebase protocol
As #17385 and #17386 evolve, this branch will be rebased on top. The
only overlap between this PR's diff and the other two is
`test_delimiter_case_sensitive.py`, where #17383 modifies the static
check to point at the new helper location.
---------
Co-authored-by: kiloconnect[bot] <240665456+kiloconnect[bot]@users.noreply.github.com>
## Summary
This PR refactors the Go knowledge-compilation ingestion pipeline
(`internal/ingestion/knowledge_compile` +
`internal/ingestion/component/knowledge_compiler`) with three related
changes:
- **Token-budget batching for LLM merge decisions.**
`LLMMergeDecider.DecideBatch` previously stuffed every `(existing,
candidate)` pair into a single LLM call, risking `max_token` overflow.
It now splits pairs into token-bounded sub-batches (budget =
`llmMaxTokens * 0.85`) via `tokenizer.NumTokensFromString`, runs them
concurrently while preserving the global pair index, and never
reindexes.
- **Process-level global compile pool.** Introduces a single vCPU-sized
goroutine pool (`pool.go`, env `KC_COMPILE_CONCURRENCY`) dedicated to
*all* knowledge-compilation stages. KNN search loop, `DecideBatch`
sub-batches, `WriteMerged`/`DeleteMerged` internals, and the
component-level (structure/mindmap) per-call pools are all unified into
it via an injected submitter. No more per-job short-lived goroutines in
`runCompilerJobs` (futures are collected then awaited on the caller).
Fan-out stays bounded by the pool worker count; these stages are
docengine-bounded / LLM-bounded, not CPU-bounded.
- **DocEngine-only deletion.** `Consumer.processBatch` deletion no
longer loads the deleted docs' products into memory. Two sequential
DocEngine calls replace the old in-memory surgery:
- `DeleteDocLevelForDocs` — one `DeleteChunks` over `doc_id IN
deletedDocIDs` (merged rows carry `doc_id == kb`, so only per-doc
products match).
- `StripMergedSources` — one `Search` of `kc_merged=1` rows filtered by
`source_doc_ids IN deletedDocIDs` (intersection pushed down to the
engine), `UpdateChunks` the source array of survivors, and
`DeleteChunks` the rows whose array became empty.
## Changes
- `internal/ingestion/knowledge_compile/pool.go` (new): global
`compilerPool` +
`runCompilerJobs`/`SubmitCompilerJob`/`SubmitCompilerJobs`.
- `internal/ingestion/knowledge_compile/consumer.go`: deletion rewritten
to the two DocEngine calls;
`mergedBase`/`toDelete`/`stripDeletedSources` removed.
- `internal/ingestion/knowledge_compile/writer.go`:
`DeleteDocLevelForDocs` + `StripMergedSources` replace
`DeleteMergedForDoc`/`DeleteMerged`.
- `internal/ingestion/knowledge_compile/reader.go`: drop
`LoadMergedBySourceDoc` + `containsString` (keep `LoadDocProducts` for
the completion branch).
- `internal/ingestion/knowledge_compile/dedup.go`: `NewLLMDeduper` takes
`llmMaxTokens`; wires `SetMaxBatchTokens`/`SetSubmitter`.
- `internal/ingestion/knowledge_compiler/{structure,merge}.go`,
`mindmap/mindmap.go`, `pool_wiring.go`: token-budget split + submitter
injection.
- Tests: `structure_test.go` (token-budget split), `dedup_test.go`,
`consumer_test.go` (tombstone + DocEngine deletion assertions) updated.
## Validation
`bash build.sh --test -race ./internal/ingestion/knowledge_compile/...
./internal/ingestion/component/knowledge_compiler/...` passes (unit
tier, no external services).
🤖 Generated with [CodeBuddy](https://www.codebuddy.ai)
---------
Co-authored-by: yuzhichang <yuzhichang@infiniflow.ai>
