Ports the dataset knowledge compilation (wiki/graph/tree/mindmap) to the
Go scheduler with a status contract, aligns wiki storage/retrieval with
Python, and sizes prompts by content_length.
## Summary
The generic `buildRequestBody` in `internal/entity/models/base_model.go`
unconditionally forwarded `ChatConfig.MaxTokens` as `"max_tokens"` for
every OpenAI-compatible provider.
Providers that need a different token field already delete or override
it after the call (e.g. Xiaomi uses `max_completion_tokens`, Replicate
uses `max_new_tokens`). This change stops setting `max_tokens` in the
shared builder so it only forwards the parameters common across
providers, and each provider remains free to set its own token limit
field.
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The Go `TokenChunker` text/markdown path (`mergeByTokenSize`)
unconditionally
called `splitOversizedUnit` on any unit that exceeded
`chunk_token_size`,
emitting Go-only sub-chunks. Python's `naive_merge`
(`_merge_paragraph_groups`,
`rag/nlp/__init__.py`) never atom-splits an oversize unit under either
`OVER_CAP` or `UNDER_CAP`: a paragraph larger than the budget becomes
its own
standalone chunk and the model layer truncates it later.
This aligns the text/markdown path with the **structured JSON path**
(`invokeJSONPayload` → `mergeByTokenSizeFromJSON(...,
subSplitOversize=false)`,
#17739). It completes the OVER_CAP alignment started in #17835.
Go's `TokenChunker` kept the captured delimiter glued to the preceding
segment on **both** the primary (`chunkFromItem`) and secondary
(`children_delimiters`) split paths, while Python's reference
`token_chunker` drops it via `_split_text_by_pattern`
(`token_chunker.py:79-93`, used by both `_build_json_chunks` and
`_split_chunk_docs_by_children`). The divergence leaked the delimiter
into every emitted chunk's `text`.
- Prefer canonical `dataset_ids` over legacy `kb_ids` in Canvas
retrieval components.
- Cover selected and explicitly cleared dataset IDs with regression
tests.
- Add cross-language retrieval with dataset-specific embedding and
rerank models.
- Support vector and keyword similarity controls, metadata filtering,
TOC enhancement, and child-chunk expansion.
- Route Canvas retrieval across datasets and memories with compatible
embedding validation.
## Summary
Update `conf/all_models.json`: replace legacy `max_tokens` with
`content_length` + `max_output` for all 2,178 chat/vision models, with
values verified against official vendor documentation.
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
## Summary
- Verify and populate `content_length` (context window) and `max_output`
(max generation tokens) for all **478 chat/vision models** across **47
provider configs**
- Data sourced from **official API documentation** via 12 parallel
agents + targeted web verification
- Update Go test assertions to match verified values
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
## Summary
Follow-up to #17835 (merged). The OVER_CAP / UNDER_CAP merge strategy
was threaded through `mergeDecision` and `mergeByTokenSizeFromJSON` as
an inlined `allowBoundaryOverflow bool` derived from `!c.param.UnderCap`
at three call sites. This replaces that with a named
`schema.MergeStrategy` enum.
## Why
- The `!c.param.UnderCap` inversion was hand-written in three places, so
a future strategy addition could silently drift between the JSON path
(`invokeTextPayload` / `invokeJSONPayload`) and the text path
(`mergeByTokenSize`) — no compile error, and the existing tests don't
cover all three sites with both strategies.
- The strategy concept was never named; `allowBoundaryOverflow` (true =
OVER_CAP) is a double-negation of `UnderCap` and reads opaquely at the
5th positional argument.
## What changed
- Add `schema.MergeStrategy` (`MergeOverCap` / `MergeUnderCap`)
mirroring Python's `rag/nlp/__init__.py` `MergeStrategy`, so Go and
Python stay on the same vocabulary.
- Expose `TokenChunkerParam.MergeStrategy()` derived from the
wire-facing `UnderCap bool` (existing `"under_cap"` configs keep working
— no schema break).
- `mergeDecision` and `mergeByTokenSizeFromJSON` now take
`schema.MergeStrategy` instead of `allowBoundaryOverflow bool`; the
three call sites pass `c.param.MergeStrategy()` (no `!`).
- Tests updated to pass the enum; added a guard test for the `UnderCap`
-> `MergeStrategy` mapping and an end-to-end test for UNDER_CAP on the
JSON path.
No behavior change: default remains OVER_CAP, `under_cap=true` still
selects UNDER_CAP.
## Test plan
`bash build.sh --test ./internal/ingestion/component/chunker/...
./internal/ingestion/component/schema/...` — all green, including
`TestMergeByTokenSizeFromJSON_UnderCapNoOverflow`,
`TestMergeByTokenSize_UnderCapNoOverflow`,
`TestInvokeJSONPayload_UnderCapEndToEnd`, and
`TestTokenChunkerParamMergeStrategy`.
## Related issues
- Relates to #17835 — wired UNDER_CAP as a tested merge-strategy seam
(merged)
- Relates to #17799 — contract doc for token-chunker cap/delimiter
alignment
- Relates to #17808 — related chunker alignment work
---------
Co-authored-by: CodeBuddy <noreply@cnb.cool>
### Summary
Remove the Go Report Card badge from `internal/harness/README.md`
because the service has been retired and no longer provides a repository
grade.
This is a documentation-only change. Validated with `git diff --check`.
## Problem
During ingestion, `indexdoc.ProcessChunksForPipeline` stamped
`ck["kb_id"]`
on every chunk. This was both:
- **a dead write** — `elasticsearch.InsertChunks` unconditionally
overwrites
the value with `datasetID` (`chunk.go:211`), so the producer's value
never
reached the index;
- **the wrong shape** — it was emitted as `[]string`, while both engines
actually need a single string.
This is the `kb_id` slice of the ingestion -> engine schema leak tracked
in
#17371: ingestion was carrying index-physical schema knowledge it should
not
own.
## Fix
Make the search engines the single owner of `kb_id` at the write
boundary,
and stop ingestion from emitting it:
- **Elasticsearch** (`chunk.go:211`) already sets `docCopy["kb_id"] =
datasetID`
— unchanged.
- **Infinity** (`chunk.go`) `InsertChunks` now stamps
`insertChunks[i]["kb_id"] = datasetID` right after
`transformChunkFields`
(previously it only *read/normalized* the producer value, which forced
ingestion to supply it). Both engines are now consistent.
- `ProcessChunksForPipeline` no longer stamps `kb_id` and the now-leaky
`kbID` parameter is removed. The same removal is propagated to
`ProcessPipelineOutputForGolden` and the `compare_pipeline_golden` dev
tool
(its `-kb-id` flag is dropped).
The stored `kb_id` value is byte-for-byte unchanged: `datasetID` passed
to
`InsertChunks` is `taskCtx.Doc.KbID`, i.e. the same id that was
previously
set on the producer chunk.
## Verification
- `bash build.sh --test ./internal/ingestion/task/indexdoc/...
./internal/engine/infinity/...`
— both green.
- `internal/ingestion/task` has **two pre-existing** failures
(`TestPipelineExecutor_Run_RealCanvasDSL_UsesGeneralPipeline`,
`TestRunPipeline_RealPipelineOutput_ProducesIndexFields`) that assert
`inserted chunk count = 1, want 2` — a parser/assertion mismatch (the Go
parser merges the 2-paragraph fixture into 1 chunk). They are unrelated
to
this change, which never touches chunk counting. The `kb_id`-related
test
failure this change would otherwise introduce is fixed by updating the
tests
below.
- Updated the pinning unit test:
`TestProcessChunksForPipeline_SetsDocID`
(formerly `...SetsDocIDAndKBID`) now asserts `kb_id` is **not** set by
the
producer. Removed the `kb_id` assertion and the now-dead
`taskChunkFieldEqualsStr` helper from
`pipeline_real_integration_test.go`.
## Scope
This closes only the `kb_id` portion of #17371. The remaining
index-physical
fields (`docnm_kwd`, `create_timestamp_flt`, `page_num_int`/`top_int`/
`position_int`, etc.) are intentionally left for a follow-up (P2).
## Problem
`kg_test.go`'s `getTestConfig()` returned `map[string]interface{}`,
which no longer matches `NewEngine`'s `config.ElasticsearchConfig`
signature after the config package moved to `internal/server/config`.
This broke compilation of the `elasticsearch` package under `-tags
integration` (kg_test.go:38).
## Fix
Make `getTestConfig()` return the typed `config.ElasticsearchConfig` so
the integration tier builds again.
## Scope
Fixes test compilation only; no production code changed. Unrelated to PR
#17802 (kb_id single-string + T0 read-back baseline), so it is tracked
in its own PR to keep that refactor focused.
🤖 Generated with [CodeBuddy Code](https://cnb.cool/codebuddy)
## Summary
- `ProcessChunksForPipeline` now sets `kb_id` to a plain string instead
of `[]string{kbID}`, removing an index-physical array shape from the
ingestion domain.
- Stored documents are byte-identical: Elasticsearch overrides `kb_id`
with `datasetID` on write, and Infinity's `transformChunkFields` already
accepts a plain string.
- Infinity is intentionally left unchanged — `service/chunk` paths still
feed `kb_id` as `[]string`, and Infinity handles both forms. The
`dataset` artifact merge (`dataset_artifact_service.go`) is out of scope
for this step.
- Unit assertion updated to expect a string.
## Scope / non-goals
This is the smallest first step (T1) of the index-schema leak cleanup
tracked in #17371. It does **not** move the other leaks (`docnm_kwd`,
`create_timestamp_flt`, position ints) to the engine boundary — those
are later steps behind a read-back golden test.
## Test plan
- `go test ./internal/ingestion/task/indexdoc/...` passes.
- The two `task` "Real" integration tests fail identically on a clean
tree (environment lacks real embedding/parsing); they are pre-existing,
unrelated to this change.
🤖 Generated with [CodeBuddy Code](https://cnb.cool/codebuddy)
Relate to #17284.
## Summary
Batch 5/6 migrated the rest of the Go model drivers onto the shared HTTP
helpers (`doRequest`, `doStreamRequest`, `applyAuth`). This PR completes
the batch for the remaining OpenAI-compatible chat-streaming drivers
that were still hand-writing HTTP requests:
- **7 drop-in migrations**: deepseek, gpustack, groq, longcat, moonshot,
openai, siliconflow
- **1 adapter migration**: minimax (relocated its `io.Pipe`
error-interception into the `doStreamRequest` handler)
- **1 full migration**: azure_openai (all four paths: chat, streaming,
embeddings, list-models) plus the auth header hook
- **1 receiver fix**: nvidia `NewInstance` value → pointer
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- Start the Agent model-stream collector before ReAct execution.
- Preserve all streamed reasoning/content deltas and drain the collector
on errors.
- Add coverage for delayed thinking streams and tool-call execution.
Relate to #17284.
## Problem
`novitaHandleStream` guarded usage recording with `if found &&
chatConfig != nil`. When a caller passes a nil `*ChatConfig` — common in
the service layer (`model_chat.go`, `chat_pipeline.go`) — the streamed
token usage is dropped entirely.
The shared `HandleStreamingResponse` only uses `chatConfig` to expose
`UsageResult` and records usage whenever the stream carries it. Novita's
bespoke handler diverged from every other OpenAI-compatible streaming
driver.
## Fix
Record usage whenever the stream carries a usage event, mirroring
`HandleStreamingResponse`. `applyStreamUsage` already handles a nil
`chatConfig` internally (it only writes `chatConfig.UsageResult` when
non-nil), so the extra guard was doing nothing but dropping usage.
## Test
`TestNovitaStreamRecordsUsageWithoutChatConfig`:
- nil `chatConfig` + usage event → stream completes without error (guard
removed safely)
- non-nil `chatConfig` + usage event → `UsageResult` populated with the
streamed tokens
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
## What
Adds [**SereneDB**](https://serenedb.com) as a selectable doc-store
engine on **both** RAGFlow paths:
- the **Go** `DocEngine` (`internal/engine/serenedb`), alongside
Elasticsearch and Infinity;
- the **Python** `DocStoreConnection` (`rag/utils/serenedb_conn.py`) +
`DOC_ENGINE=serenedb` registration.
SereneDB is a PostgreSQL-wire engine (DuckDB execution) whose single
inverted index carries **both** a scored text column (`@@`, BM25) and an
IVF vector column (`<#>`, inner product), so hybrid search is one SQL
statement. The Go engine connects with `database/sql` + `lib/pq`
(already a dependency, no new module); the Python connector uses
psycopg2 (already a dependency).
## Storage model
One table per tenant with `kb_id` as a filter column - the
**Elasticsearch / OceanBase** model, not Infinity's per-dataset tables.
This keeps BM25 statistics (IDF, avgdl) computed over the whole tenant
corpus (global IDF). Both connectors use this identical layout, so they
are storage- and retrieval-compatible: `hybrid` proxy routing and
Python↔Go switching are safe. On the Python side the connector is wired
as OceanBase's plain-SQL sibling (chunk_data JSON metadata, inline chunk
vectors, verbatim ES field names); the ES tokenizer path is unchanged.
Metadata stays one table per tenant (`ragflow_doc_meta_<tenant>`).
The query shapes mirror the Python connector, including the five
empirically-found landmines: the scored dictionary needs `frequency +
norm` (else `BM25()` silently returns 0.0), the `@@` query is the
tokenized query, the scored lexical branch matches one column, vectors
use an L2-normalized shadow column with `ip`/`sq8`, and the similarity
threshold goes directly in the ANN scan's `WHERE`. **Minimum engine
version: SereneDB 26.07.4.**
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
## Summary
Relate to #17284.
Completes the migration of the four non-OpenAI-compatible model drivers
(`anthropic`, `cohere`, `google`, `bedrock`) onto the shared
usage-recording path. Earlier batches (#17634, #17643, #17696–#17700)
covered only the OpenAI-compatible cluster; these four providers ship
wire formats that do not fit the OpenAI `choices[0].delta` / `usage`
block template and so were left for a separate pass.
Per the maintainer's guidance for this batch, each driver is migrated on
its own terms rather than forced through a single template. The shared
machinery used is intentionally small: `recordResponseUsage`,
`parseChatCompletionResponse`, `BaseModel.newJSONPostRequest`, and the
existing `authHeader` hook for non-Bearer auth.
Co-authored-by: Haruko386 <tryeverypossible@163.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
## Summary
DeepSeek and Azure OpenAI require `stream_options.include_usage=true` to
return token usage in streaming responses. Without it, all streaming
calls report zero usage to ClickHouse and the UI shows no token stats.
- [x] Verify DeepSeek streaming calls now report usage
- [x] Verify Azure OpenAI streaming calls now report usage
## Summary
- Normalize Canvas component names before resolving Go Agent tools and
parameters.
- Add regression coverage for CodeExec and other Canvas tool mappings.
## Testing
- `CGO_ENABLED=0 go test -count=1 ./internal/agent/tool
./internal/agent/component`
## Summary
Extract the pipeline-output → search-engine index document mapping
helpers out of the `task` package into a dedicated, dependency-light
leaf package `internal/ingestion/task/indexdoc`.
These functions are pure transforms (they only depend on
`common`/`utility`) and are not task-orchestration concerns:
- `NormalizeChunks`, `DeepCopyChunks` (was unexported `deepCopyChunks`),
`toChunkMaps` → `indexdoc/normalize.go`
- `ProcessChunksForPipeline`, `RenameTextToContentWithWeight`,
`GetEmbeddingTokenConsumption`, `cleanupConsumedChunkFields`,
`mergeChunkMetadata`, `processChunkPositions`,
`AggregateTableDocMetadata`, `resolveTableColumnConfig` →
`indexdoc/process.go`
- `AddPositions` → `indexdoc/position.go`
- `EmbeddingTokenConsumptionKey` constant → `indexdoc/constants.go`
(task/constants.go keeps only `GRAPH_RAPTOR_FAKE_DOC_ID`)
Call sites in `pipeline_executor.go` and `golden_compare.go` now
reference the `indexdoc` package; package-task tests qualify the moved
symbols.
## Why
The `task` package had grown into a "orchestration + pure mapping +
debug" mix. Splitting the pure mapping helpers into a leaf package
sharpens package boundaries, removes a misleading top-level
`ingestion/chunk` candidate (there are already `parser/chunk` and
`service/chunk`), and lets the golden tool / future reuse pull in the
mapping logic without dragging in `task`'s `dao`/`engine`/`service`
dependency graph (Go subpackage import does not pull in the parent).
## Test plan
- `build.sh --test ./internal/ingestion/task/...` — **green** (task
4.7s, indexdoc 0.007s), matching the pre-change baseline.
- `gofmt` clean; `build.sh` builds both `ragflow-cli` and
`ragflow_server` successfully.
- Integration/E2E tiers are delegated to CI (need real MySQL/MinIO/ES
services).
Note: `pipeline_e2e_test.go` has a **pre-existing** compile error
(`server.ElasticsearchConfig` / `server.InfinityConfig` are now defined
under `internal/server/config/`, not re-exported by `internal/server`).
This is unrelated to this change — the diff to that file is only the
added `indexdoc` import and the qualified `EmbeddingTokenConsumptionKey`
reference.
## Summary
Adds a self-contained regression guard for `splitOversizedUnitWith`
after PR #17729
aligned it with Python's `rag/nlp._split_oversized_unit` running-sum
flush.
#17729 shipped a `slack=1` relaxation in `token_strict_cap_test.go` (the
oversized
unit is now sub-split with the same running-sum flush Python uses, which
can leave a
piece one token over the nominal budget due to cl100k non-additivity).
This PR adds
the missing positive proof that the sub-split boundaries are correct, so
the relaxed
assertion is no longer unguarded.
## Changes
- `split_oversized_guard_test.go` (new, self-contained — no harness
loader, no
`testdata`; the Python oracle is inlined):
- `TestSplitOversizedUnitRunningSumMatchesPython`: asserts the exact
piece
boundaries (live tokenizer) match Python's `_split_oversized_unit`,
compensating
the `slack=1` relaxation from #17729.
- `TestSplitOversizedUnitDeadTokenizerCollapses`: asserts a
zero-counting tokenizer
collapses the B1 paragraph into exactly one chunk, catching a silently
dead encoder
that a non-empty-result check would miss.
## Notes
- Test function names are deliberately distinct from PR #17735's
`TestSplitOversizedUnitMatchesPython`, so the two PRs verify
independently and do not
conflict at merge time.
- PR #17735 (golden parity harness) is intentionally left unchanged per
the split plan.
## Test plan
`bash build.sh --test ./internal/ingestion/component/chunker/` — green,
including both
new tests.
Co-authored-by: CodeBuddy <noreply@cnb.cool>
## Summary
Relate to #17284. Completes the batch 5 migration of 7 OpenAI-compatible
drivers (`vllm`, `volcengine`, `xai`, `xiaomi`, `xinference`, `xunfei`,
`zhipu-ai`) onto the unified request/response helpers
(`doRequest`/`doStreamRequest` +
`HandleNonStreamingResponse`/`HandleStreamingResponse` +
`ParserConfig`), established by `deepseek` in #17634.
This branch is rebased on the current `pr/migrate-models-batch5` and
fixes the issues in the previous state of the PR.
Co-authored-by: Haruko386 <tryeverypossible@163.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
