Relate to #17284.
## Problem
`novitaHandleStream` guarded usage recording with `if found &&
chatConfig != nil`. When a caller passes a nil `*ChatConfig` — common in
the service layer (`model_chat.go`, `chat_pipeline.go`) — the streamed
token usage is dropped entirely.
The shared `HandleStreamingResponse` only uses `chatConfig` to expose
`UsageResult` and records usage whenever the stream carries it. Novita's
bespoke handler diverged from every other OpenAI-compatible streaming
driver.
## Fix
Record usage whenever the stream carries a usage event, mirroring
`HandleStreamingResponse`. `applyStreamUsage` already handles a nil
`chatConfig` internally (it only writes `chatConfig.UsageResult` when
non-nil), so the extra guard was doing nothing but dropping usage.
## Test
`TestNovitaStreamRecordsUsageWithoutChatConfig`:
- nil `chatConfig` + usage event → stream completes without error (guard
removed safely)
- non-nil `chatConfig` + usage event → `UsageResult` populated with the
streamed tokens
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
## What
Adds [**SereneDB**](https://serenedb.com) as a selectable doc-store
engine on **both** RAGFlow paths:
- the **Go** `DocEngine` (`internal/engine/serenedb`), alongside
Elasticsearch and Infinity;
- the **Python** `DocStoreConnection` (`rag/utils/serenedb_conn.py`) +
`DOC_ENGINE=serenedb` registration.
SereneDB is a PostgreSQL-wire engine (DuckDB execution) whose single
inverted index carries **both** a scored text column (`@@`, BM25) and an
IVF vector column (`<#>`, inner product), so hybrid search is one SQL
statement. The Go engine connects with `database/sql` + `lib/pq`
(already a dependency, no new module); the Python connector uses
psycopg2 (already a dependency).
## Storage model
One table per tenant with `kb_id` as a filter column - the
**Elasticsearch / OceanBase** model, not Infinity's per-dataset tables.
This keeps BM25 statistics (IDF, avgdl) computed over the whole tenant
corpus (global IDF). Both connectors use this identical layout, so they
are storage- and retrieval-compatible: `hybrid` proxy routing and
Python↔Go switching are safe. On the Python side the connector is wired
as OceanBase's plain-SQL sibling (chunk_data JSON metadata, inline chunk
vectors, verbatim ES field names); the ES tokenizer path is unchanged.
Metadata stays one table per tenant (`ragflow_doc_meta_<tenant>`).
The query shapes mirror the Python connector, including the five
empirically-found landmines: the scored dictionary needs `frequency +
norm` (else `BM25()` silently returns 0.0), the `@@` query is the
tokenized query, the scored lexical branch matches one column, vectors
use an L2-normalized shadow column with `ip`/`sq8`, and the similarity
threshold goes directly in the ANN scan's `WHERE`. **Minimum engine
version: SereneDB 26.07.4.**
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
## Summary
Relate to #17284.
Completes the migration of the four non-OpenAI-compatible model drivers
(`anthropic`, `cohere`, `google`, `bedrock`) onto the shared
usage-recording path. Earlier batches (#17634, #17643, #17696–#17700)
covered only the OpenAI-compatible cluster; these four providers ship
wire formats that do not fit the OpenAI `choices[0].delta` / `usage`
block template and so were left for a separate pass.
Per the maintainer's guidance for this batch, each driver is migrated on
its own terms rather than forced through a single template. The shared
machinery used is intentionally small: `recordResponseUsage`,
`parseChatCompletionResponse`, `BaseModel.newJSONPostRequest`, and the
existing `authHeader` hook for non-Bearer auth.
Co-authored-by: Haruko386 <tryeverypossible@163.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
## Summary
DeepSeek and Azure OpenAI require `stream_options.include_usage=true` to
return token usage in streaming responses. Without it, all streaming
calls report zero usage to ClickHouse and the UI shows no token stats.
- [x] Verify DeepSeek streaming calls now report usage
- [x] Verify Azure OpenAI streaming calls now report usage
## Summary
- Normalize Canvas component names before resolving Go Agent tools and
parameters.
- Add regression coverage for CodeExec and other Canvas tool mappings.
## Testing
- `CGO_ENABLED=0 go test -count=1 ./internal/agent/tool
./internal/agent/component`
## Summary
Extract the pipeline-output → search-engine index document mapping
helpers out of the `task` package into a dedicated, dependency-light
leaf package `internal/ingestion/task/indexdoc`.
These functions are pure transforms (they only depend on
`common`/`utility`) and are not task-orchestration concerns:
- `NormalizeChunks`, `DeepCopyChunks` (was unexported `deepCopyChunks`),
`toChunkMaps` → `indexdoc/normalize.go`
- `ProcessChunksForPipeline`, `RenameTextToContentWithWeight`,
`GetEmbeddingTokenConsumption`, `cleanupConsumedChunkFields`,
`mergeChunkMetadata`, `processChunkPositions`,
`AggregateTableDocMetadata`, `resolveTableColumnConfig` →
`indexdoc/process.go`
- `AddPositions` → `indexdoc/position.go`
- `EmbeddingTokenConsumptionKey` constant → `indexdoc/constants.go`
(task/constants.go keeps only `GRAPH_RAPTOR_FAKE_DOC_ID`)
Call sites in `pipeline_executor.go` and `golden_compare.go` now
reference the `indexdoc` package; package-task tests qualify the moved
symbols.
## Why
The `task` package had grown into a "orchestration + pure mapping +
debug" mix. Splitting the pure mapping helpers into a leaf package
sharpens package boundaries, removes a misleading top-level
`ingestion/chunk` candidate (there are already `parser/chunk` and
`service/chunk`), and lets the golden tool / future reuse pull in the
mapping logic without dragging in `task`'s `dao`/`engine`/`service`
dependency graph (Go subpackage import does not pull in the parent).
## Test plan
- `build.sh --test ./internal/ingestion/task/...` — **green** (task
4.7s, indexdoc 0.007s), matching the pre-change baseline.
- `gofmt` clean; `build.sh` builds both `ragflow-cli` and
`ragflow_server` successfully.
- Integration/E2E tiers are delegated to CI (need real MySQL/MinIO/ES
services).
Note: `pipeline_e2e_test.go` has a **pre-existing** compile error
(`server.ElasticsearchConfig` / `server.InfinityConfig` are now defined
under `internal/server/config/`, not re-exported by `internal/server`).
This is unrelated to this change — the diff to that file is only the
added `indexdoc` import and the qualified `EmbeddingTokenConsumptionKey`
reference.
## Summary
Adds a self-contained regression guard for `splitOversizedUnitWith`
after PR #17729
aligned it with Python's `rag/nlp._split_oversized_unit` running-sum
flush.
#17729 shipped a `slack=1` relaxation in `token_strict_cap_test.go` (the
oversized
unit is now sub-split with the same running-sum flush Python uses, which
can leave a
piece one token over the nominal budget due to cl100k non-additivity).
This PR adds
the missing positive proof that the sub-split boundaries are correct, so
the relaxed
assertion is no longer unguarded.
## Changes
- `split_oversized_guard_test.go` (new, self-contained — no harness
loader, no
`testdata`; the Python oracle is inlined):
- `TestSplitOversizedUnitRunningSumMatchesPython`: asserts the exact
piece
boundaries (live tokenizer) match Python's `_split_oversized_unit`,
compensating
the `slack=1` relaxation from #17729.
- `TestSplitOversizedUnitDeadTokenizerCollapses`: asserts a
zero-counting tokenizer
collapses the B1 paragraph into exactly one chunk, catching a silently
dead encoder
that a non-empty-result check would miss.
## Notes
- Test function names are deliberately distinct from PR #17735's
`TestSplitOversizedUnitMatchesPython`, so the two PRs verify
independently and do not
conflict at merge time.
- PR #17735 (golden parity harness) is intentionally left unchanged per
the split plan.
## Test plan
`bash build.sh --test ./internal/ingestion/component/chunker/` — green,
including both
new tests.
Co-authored-by: CodeBuddy <noreply@cnb.cool>
## Summary
Relate to #17284. Completes the batch 5 migration of 7 OpenAI-compatible
drivers (`vllm`, `volcengine`, `xai`, `xiaomi`, `xinference`, `xunfei`,
`zhipu-ai`) onto the unified request/response helpers
(`doRequest`/`doStreamRequest` +
`HandleNonStreamingResponse`/`HandleStreamingResponse` +
`ParserConfig`), established by `deepseek` in #17634.
This branch is rebased on the current `pr/migrate-models-batch5` and
fixes the issues in the previous state of the PR.
Co-authored-by: Haruko386 <tryeverypossible@163.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
## Summary
RAGFlow's Go tokenizer silently returned **0 tokens for every string**
whenever the `cl100k_base` BPE table could not be loaded — which is the
normal case for an offline/air-gapped Go server. This PR makes the
loader resolve the table from disk (where RAGFlow actually ships it) and
fail loudly when it is genuinely missing.
## Root cause
`tiktoken-go`'s stock loader downloads the encoding table over HTTP and
caches it under `TIKTOKEN_CACHE_DIR`. That does not work for RAGFlow:
- `TIKTOKEN_CACHE_DIR` is exported **only inside the Python process**
(`common/token_utils.py`). `docker/entrypoint.sh` launches the Go binary
(`bin/ragflow_server`) from a shell, so the Go process never inherits
the variable.
- The Dockerfile *does* ship the table (under its sha1 name in the
working directory), but nothing told the Go side to look there.
- Reaching `openaipublic.blob.core.windows.net` at runtime is not an
option for air-gapped installs, and is unreliable where that host is
blocked.
The failure was **silent**: `NumTokensFromString` returns `0` when the
encoder fails to build, and a `sync.Once` memoizes that error for the
process lifetime. Every token count became `0`, so chunk merging never
crossed its token budget and an entire document collapsed into a single
chunk. Python has no such failure mode because its encoder is built at
import time (a missing table aborts startup instead of degrading).
## Fix
Register a local-only `BpeLoader` via `tiktoken.SetBpeLoader`
(`internal/tokenizer/bpe_loader.go`) that resolves the table from disk
**only**, in priority order:
1. `TIKTOKEN_CACHE_DIR` / `DATA_GYM_CACHE_DIR` (honored so operators who
already configured one keep working).
2. The working directory, the executable's directory, and all of their
ancestors — matching the Dockerfile layout (table under its sha1 name in
the install root).
3. A `ragflow_deps/<basename>` checkout produced by
`ragflow_deps/download_deps.py`.
It **never performs network I/O**. When nothing is found it returns an
error listing every path it tried (pointing at `download_deps.py` or
`TIKTOKEN_CACHE_DIR`), so a genuinely missing table fails loudly instead
of degrading to zero.
## Test plan
- `internal/tokenizer/bpe_loader_test.go` (unit tier, runs under `bash
build.sh --test ./internal/tokenizer/...`):
- Loader reads from `TIKTOKEN_CACHE_DIR`, `DATA_GYM_CACHE_DIR`, the
sha1-named file in the working dir, and the bundled `ragflow_deps/`
name.
- Explicit cache dir wins over the bundled vocab.
- A malformed table is reported as an error rather than skipped.
- A genuinely missing table reports the candidates it tried (no network
attempt).
- `NumTokensFromString` matches Python-derived anchors (`""`→0,
`"hello"`→1, `"hello world"`→2, `"hello, world!"`→4, `"世界"`→3, `"Hello
世界 🌍"`→8, `"RAGFlow"`→3).
## Notes
- `.github/workflows/tests.yml` currently excludes `internal/tokenizer`
from `go test`, so these tests do not run in CI. The tokenizer fix is
exercised in CI indirectly via the chunker package once a
token-count-sensitive parity case lands (tracked separately). Consider
including `internal/tokenizer` in CI as a follow-up.
- Supported deployments already ship the table (`download_deps.py` →
`ragflow_deps/cl100k_base.tiktoken`; Dockerfile → `<sha1>` in cwd), so
no `ENV` change is required for the fix to take effect. Setting `ENV
TIKTOKEN_CACHE_DIR` in the Dockerfile remains a cheap
belt-and-suspenders hardening that can be done separately.
🤖 Generated with [CodeBuddy Code](https://cnb.cool/codebuddy)
---------
Co-authored-by: CodeBuddy <noreply@codebuddy.ai>
Co-authored-by: CodeBuddy Code <noreply@cnb.cool>
Co-authored-by: CodeBuddy <noreply@tencent.com>
## Summary
Relate to #17284
Migrate 10 OpenAI-compatible drivers (`orcarouter`, `perplexity`,
`ppio`, `qiniu`, `ragcon`, `stepfun`, `togetherai`, `tokenhub`,
`tokenpony`, `upstage`) to use the unified response handlers
(`HandleNonStreamingResponse` / `HandleStreamingResponse`), following
the same pattern established by `deepseek` in #17634.
- Cut ~150 lines per driver (1436 lines removed, 62 added across 10
files).
- No functional changes — pure deduplication of HTTP plumbing.
- Each driver now routes through `baseModel.doRequest()` and
`HandleNonStreamingResponse()`.
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
## Summary
Align Go `splitOversizedUnitWith` with Python
`rag/nlp._split_oversized_unit` so the whitespace-atom sub-split
produces byte-identical chunk boundaries.
### Root cause of the divergence
cl100k token counting is **not additive across whitespace joins**
(`token(a)+token(b) != token(a+b)`). Go previously used the exact
joined-string fit check `countFn(current+atom) > budget`, while Python
accumulates a running sum `current_tokens + a_tokens > budget`. The two
formulas disagree by one atom at the boundary, so Go and Python emitted
the same chunk *count* but shifted *text*.
### Changes
- `splitOversizedUnitWith` (`token.go`): replace the exact joined-string
fit check with the running-sum check (mirroring Python's
`current_tokens` accumulator), and after a flush keep the overflow
whitespace atom (`current += atom`) instead of dropping it.
- `token_strict_cap_test.go`: relax
`TestMergeByTokenSizeFromJSON_OversizedUnitIsSubSplit` to allow the same
cl100k non-additive +1 overshoot Python exhibits (the invariant — an
oversized unit is sub-split, not collapsed — is preserved).
### Test plan
`bash build.sh --test ./internal/ingestion/component/chunker/...` —
green.
## Note
Test infrastructure for this change (golden parity harness,
`split_oversized_test.go`, `testdata/parity/**`, `known_diffs.json`,
`capture_golden.py`/`live_chunk.py`, and the `go-cmp` dependency
promotion) is split into a separate, stacked PR #17735 so this PR stays
minimal (production code only).
This PR is **independent of #17712** (the offline BPE loader). It is
based on `upstream/main` and contains only this change; no BPE-loader
code is included.
Co-authored-by: CodeBuddy <noreply@cnb.cool>
