## Summary
- `api/apps/services/provider_api_service.py` hardcoded the DashScope
international base URL for Tongyi-Qianwen as `.../compatible-model/v1`
instead of `.../compatible-mode/v1`, in two places (`list_providers`,
lines ~93 and ~116).
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
Ports the dataset knowledge compilation (wiki/graph/tree/mindmap) to the
Go scheduler with a status contract, aligns wiki storage/retrieval with
Python, and sizes prompts by content_length.
## Summary
The generic `buildRequestBody` in `internal/entity/models/base_model.go`
unconditionally forwarded `ChatConfig.MaxTokens` as `"max_tokens"` for
every OpenAI-compatible provider.
Providers that need a different token field already delete or override
it after the call (e.g. Xiaomi uses `max_completion_tokens`, Replicate
uses `max_new_tokens`). This change stops setting `max_tokens` in the
shared builder so it only forwards the parameters common across
providers, and each provider remains free to set its own token limit
field.
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The Go `TokenChunker` text/markdown path (`mergeByTokenSize`)
unconditionally
called `splitOversizedUnit` on any unit that exceeded
`chunk_token_size`,
emitting Go-only sub-chunks. Python's `naive_merge`
(`_merge_paragraph_groups`,
`rag/nlp/__init__.py`) never atom-splits an oversize unit under either
`OVER_CAP` or `UNDER_CAP`: a paragraph larger than the budget becomes
its own
standalone chunk and the model layer truncates it later.
This aligns the text/markdown path with the **structured JSON path**
(`invokeJSONPayload` → `mergeByTokenSizeFromJSON(...,
subSplitOversize=false)`,
#17739). It completes the OVER_CAP alignment started in #17835.
## Summary
- Pin `werkzeug>=3.1.7,<4` and refresh `uv.lock` to **3.1.8**.
- Fixes intermittent corruption of uploaded file bodies: when TCP
segments split right after multipart part headers, Werkzeug **3.1.5**
can include a leading `\r\n` in the file content
([pallets/werkzeug#3088](https://github.com/pallets/werkzeug/issues/3088);
fixed in 3.1.7).
- In RAGFlow this commonly breaks `.xlsx` parsing: ZIP/OOXML magic
(`PK\x03\x04`) no longer matches, the Excel parser falls back to CSV,
then fails with UTF-8 decode errors such as `invalid start/continuation
byte`.
Made with [Cursor](https://cursor.com)
Co-authored-by: zhangjiangshan1 <zhangjiangshan1@kingsoft.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Jin Hai <haijin.chn@gmail.com>
Go's `TokenChunker` kept the captured delimiter glued to the preceding
segment on **both** the primary (`chunkFromItem`) and secondary
(`children_delimiters`) split paths, while Python's reference
`token_chunker` drops it via `_split_text_by_pattern`
(`token_chunker.py:79-93`, used by both `_build_json_chunks` and
`_split_chunk_docs_by_children`). The divergence leaked the delimiter
into every emitted chunk's `text`.
- Prefer canonical `dataset_ids` over legacy `kb_ids` in Canvas
retrieval components.
- Cover selected and explicitly cleared dataset IDs with regression
tests.
- Add cross-language retrieval with dataset-specific embedding and
rerank models.
- Support vector and keyword similarity controls, metadata filtering,
TOC enhancement, and child-chunk expansion.
- Route Canvas retrieval across datasets and memories with compatible
embedding validation.
## Summary
Update `conf/all_models.json`: replace legacy `max_tokens` with
`content_length` + `max_output` for all 2,178 chat/vision models, with
values verified against official vendor documentation.
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
## Summary
- Verify and populate `content_length` (context window) and `max_output`
(max generation tokens) for all **478 chat/vision models** across **47
provider configs**
- Data sourced from **official API documentation** via 12 parallel
agents + targeted web verification
- Update Go test assertions to match verified values
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
## Summary
Follow-up to #17835 (merged). The OVER_CAP / UNDER_CAP merge strategy
was threaded through `mergeDecision` and `mergeByTokenSizeFromJSON` as
an inlined `allowBoundaryOverflow bool` derived from `!c.param.UnderCap`
at three call sites. This replaces that with a named
`schema.MergeStrategy` enum.
## Why
- The `!c.param.UnderCap` inversion was hand-written in three places, so
a future strategy addition could silently drift between the JSON path
(`invokeTextPayload` / `invokeJSONPayload`) and the text path
(`mergeByTokenSize`) — no compile error, and the existing tests don't
cover all three sites with both strategies.
- The strategy concept was never named; `allowBoundaryOverflow` (true =
OVER_CAP) is a double-negation of `UnderCap` and reads opaquely at the
5th positional argument.
## What changed
- Add `schema.MergeStrategy` (`MergeOverCap` / `MergeUnderCap`)
mirroring Python's `rag/nlp/__init__.py` `MergeStrategy`, so Go and
Python stay on the same vocabulary.
- Expose `TokenChunkerParam.MergeStrategy()` derived from the
wire-facing `UnderCap bool` (existing `"under_cap"` configs keep working
— no schema break).
- `mergeDecision` and `mergeByTokenSizeFromJSON` now take
`schema.MergeStrategy` instead of `allowBoundaryOverflow bool`; the
three call sites pass `c.param.MergeStrategy()` (no `!`).
- Tests updated to pass the enum; added a guard test for the `UnderCap`
-> `MergeStrategy` mapping and an end-to-end test for UNDER_CAP on the
JSON path.
No behavior change: default remains OVER_CAP, `under_cap=true` still
selects UNDER_CAP.
## Test plan
`bash build.sh --test ./internal/ingestion/component/chunker/...
./internal/ingestion/component/schema/...` — all green, including
`TestMergeByTokenSizeFromJSON_UnderCapNoOverflow`,
`TestMergeByTokenSize_UnderCapNoOverflow`,
`TestInvokeJSONPayload_UnderCapEndToEnd`, and
`TestTokenChunkerParamMergeStrategy`.
## Related issues
- Relates to #17835 — wired UNDER_CAP as a tested merge-strategy seam
(merged)
- Relates to #17799 — contract doc for token-chunker cap/delimiter
alignment
- Relates to #17808 — related chunker alignment work
---------
Co-authored-by: CodeBuddy <noreply@cnb.cool>
## Summary
Fixes a regression introduced by #17203 (strict-cap atom-split) and a
secondary delimiter-handling bug from #17723.
**Root cause:**
- #17203 added `_split_oversized_unit` / `_compute_chunk_update`, which
split oversize units into ≤ token_size pieces. This collapsed
`token_size=1` into 1-token chunks and set the cap at 512, mismatching
the model-layer truncation boundary (embedding ~8191 / rerank
500/4096/8192/2048). Atom-split is unnecessary: oversize units stay
whole and the model layer truncates.
- #17723's delimiter handling dropped consecutive delimiters (`A####B`
-> `A##B`), glued JSON items with `"".join`, ignored
`children_delimiters`, and stripped whitespace delimiters.
## Changes
- New pure helper `merge_paragraphs(paragraphs, token_size, strategy)`
with a `MergeStrategy` enum (`UNDER_CAP` / `OVER_CAP`); **default
`OVER_CAP`**. `UNDER_CAP` is a strict cap (never overflows
`token_size`); `OVER_CAP` greedily accumulates adjacent paragraphs while
the projected total stays within `token_size`, merging one
boundary-overflow paragraph before closing. Oversize paragraphs stand
alone.
- `naive_merge` / `naive_merge_with_images` /
`RAGFlowTxtParser.parser_txt` now use `merge_paragraphs`; atom-split
removed. `naive_merge` / `naive_merge_with_images` always split a
section on the delimiter whenever one is present (even when the section
already fits `token_size`), so delimiter text never leaks into a chunk.
Only the empty-delimiter (size-only) mode skips splitting.
- `token_chunker`: delimiter text is dropped (not stripped); JSON flush
joins buffered items with `"\n"`; `children_delimiters` and
`PDF_POSITIONS_KEY` are preserved on the delimiter path. PDF positions
are now attributed **per segment** — each split chunk carries only the
positions of the item(s) that contributed to it — fixing a leak where
page-N coordinates were attached to page-M chunks and all segments
shared one preview image.
- `test_txt_parser.py` rewritten to assert the new contract (not the old
strict cap); `naive_merge` and delimiter-case-sensitive matrices
updated.
## Contract (refs #17799)
- user specified delimiter = chunk boundary; user specified delimiter
text never enters a chunk.
- `token_size` = soft target + merge strategy; no atom-split.
- Default strategy = `OVER_CAP`; migration can switch to `UNDER_CAP`
(strict cap).
- `OVER_CAP` has no hard cap; the model layer truncates oversize units.
`UNDER_CAP` enforces a strict cap.
## Notes
- Closes the wrong-object revert in #17774 (revert #17723 would
re-introduce delimiter-in-chunk and the strict cap).
- Go-side alignment (`internal/ingestion/component/chunker/token.go`) is
a follow-up PR.
---------
Co-authored-by: CodeBuddy <noreply@tencent.com>
