Add a shared `dao.ResolveModelContentLength` that resolves a chat model's context window (`content_length`) from a `tenant_model` UUID or a composite `model@provider` reference, with an optional `driver + modelName` catalog fallback for the no-database path.
The agent LLM component and the ingestion Extractor component both need
to trim prompts to the model's context window before calling the
provider. Each previously did (or would do) this with its own copy of
the logic. This PR adds the shared primitive; follow-up PRs wire it into
the agent LLM component (#18092) and the ingestion Extractor/tagger
(#18095).
### Summary
- Propagate the dataset language through Go DOCX, Markdown, PDF
figure-enhancement, and standalone-image vision paths.
- Explicitly render the shared figure prompt's `{{ language }}`
placeholder in Go.
- Use English when the dataset language is empty.
- Make the default standalone-image prompt request the dataset language
while preserving visible text in its original language.
- Add focused tests for caller propagation, language fallback, prompt
rendering, and prompt-cache isolation.
Stop flattening `<table>` into a single text blob. A `<table>` now emits:
1. an inlined `doc_type_kwd:"text"` item keeping the `<table>…</table>`
markup (row/column structure survives for embedding/retrieval/LLM rendering),
2. a structured `doc_type_kwd:"table"` / `ck_type:"table"` item appended
after the walk, consumed by the downstream chunker.
Port the wiki_incremental dataset-level merge and make its rewrite
barrier durable and concurrency-safe. Wiki pages merge replace-only; the
barrier persists a monotonic numeric generation, and a scheduler-backed
per-dataset lock closes the cross-process TOCTOU window. Adds the
Compiler Plan toggle (frontend) with Mode A grouping.
## What
This pull request adds **MWS GPT Model Hub** as a built-in model
provider in RAGFlow.
The integration allows users to configure an MWS project endpoint and
token, discover the models available to that project, and use supported
MWS models for chat completion, embeddings, and reranking.
Co-authored-by: ilarionov_n <ilarionov_n@promis.ru>
Centralize the shared golden-doc + alignment helpers in `align_test.go` so the format-specific PRs (text&code, markdown golden, HTML) reuse one implementation instead of each carrying their own copy of the scaffolding.
Golden-parity test infrastructure for the **Go `TokenChunker` ↔ Python alignment**.
It runs the Go chunker over a committed case set (`testdata/parity/cases/`) and diffs each output against a captured Python golden (`testdata/parity/golden/`), honoring a `known_diffs.json` ratchet (`extra_fields` / `chunk_count` / `chunk_text`) so accepted divergences are tracked rather than silently widening.
Port dataset-level wiki incremental compile and refactor splitByTokens
token budgeting. Includes replace-only wiki merge, KNN dedup routing,
and template/config wiring.
## What changed
- add an OceanBase/SeekDB Go document engine using `database/sql` and
the existing MySQL driver
- preserve the Python connector's configuration, physical table names,
schema, index names, and ARRAY/JSON/VECTOR encodings
- implement chunk, memory, document metadata, skill, SQL, full-text,
vector, and fusion search paths
- support `DBMS_HYBRID_SEARCH.SEARCH` behind the existing feature flag,
with SQL fallback only when the package is unavailable
- wire the engine into retrieval, memory, metadata, vector hydration,
and SQL chat flows
- add Python/Go compatibility contracts, SQL mock tests, and an
integration-tagged round-trip test
---------
Co-authored-by: Jin Hai <haijin.chn@gmail.com>
Aligns the Go EML parser (`internal/parser/parser/email_parser.go`) with the Python flow parser (`rag/flow/parser/parser.py:_email`) on two structural points so the Go path is a faithful drop-in for the Python path on `.eml` inputs.
Unifies the Go TokenChunker merge path on a single `mergeUnits` core and
fixes coordinate-tag drift in the Python JSON merge at `overlap > 0`.
Rebased on top of #17979 (delimiter_mode convergence).
Re-materialize wiki page graph from merged wiki_page rows after each
batch merge. Adds ProjectWikiGraph/DropWikiGraph, full page_type/slug
identity, delete-then-insert, tests.
Converge `TokenChunker.delimiter_mode` from three values (`token_size`,
`delimiter`, `one`) to two (`delimiter`, `one`). The unified `delimiter`
mode now carries the old `token_size` semantics: when no active
(backtick) delimiter is present, text/JSON chunks are merged up to
`chunk_token_size`; when a backtick delimiter is present, the text is
split by it and not merged. `one` continues to be handled by the
separate `OneChunker`.
Restore the deduplication that was dropped when #17926 was merged.
`CompileDelimiterPatternList` now keeps a `seen` set and collapses
equivalent active entries (both backtick-inner and bare) into a single
alternation. This PR also removes the dead code that the re-review
surfaced.
Collapses three duplicated/dead code smells in the canvas debug-result
path of `internal/ingestion/task` (remaining findings #2/#3/#4 from the
package CTO review):
- **#2 — vector-key stripping duplication.** `isVectorKey` (the full
stripper: fixed legacy keys `vector`/`embedding`/`feature`/`q_vec` plus
the `q_<dim>_vec` pattern) was re-implemented as a weaker inline copy in
the golden-compare tool (`tool/compare_pipeline_golden.go`) that only
matched `q_<dim>_vec` and let real vectors leak into the diff. Exported
as `IsVectorKey` and reused by the tool.
- **#3 — near-duplicate deep copy.** `deepCopy` and `deepCopyStrip` were
identical walkers differing only in vector stripping. Parameterized
`deepCopy(v any, stripVector bool) any` and deleted `deepCopyStrip`.
- **#4 — redundant switch.** `lookupComponentOutput` had two switch
cases with identical bodies (both the `map[string]map[string]any` and
`map[string]any` state shapes). Unified into a single `found`/`ok`
resolution; nested-state and flat-fallback semantics unchanged.
## Summary
- Add embedding batch-size metadata to model responses and tenant
overrides.
- Validate embedding dimensions and batch limits across provider
verification and embedding requests.
- Expand validation tests for defaults, limits, and missing metadata.
---------
Signed-off-by: Jin Hai <haijin.chn@gmail.com>
Co-authored-by: Jin Hai <haijin.chn@gmail.com>
`compileChildrenPattern` re-implemented the delimiter-list compile
inline with two divergences from the shared
`CompileDelimiterListPattern`:
- It never stripped backticks, so a backtick-wrapped
`children_delimiter` like `` `###` `` matched the **literal wrapped
token** rather than the inner `###`.
- It sorted by **byte length** instead of rune count (`sortSlice`), so
multi-byte delimiters could be ordered incorrectly and a longer
delimiter could fail to win over a shorter prefix.
Ports dataset knowledge compilation (wiki/graph/tree/mindmap) to the Go
scheduler with a status contract, aligns wiki storage/retrieval with
Python, sizes prompts by content_length, and resolves embedding batch
size from provider capability.
## Summary
Port the DSL tokenizer's `important_kwd` splitting into the Go
`Tokenizer` component so the indexed keyword array is byte-compatible
with the Python DSL pipeline and with the keyword-extraction prompt
contract.
- **Problem:** The Go component split `keywords` on the full ASCII+CJK
delimiter set (`utility.SplitKeywords`, regex `[,，;；、\r\n]+`), while the
DSL baseline `rag/flow/tokenizer/tokenizer.py:153` uses
`keywords.split(",")`, and `rag/prompts/keyword_prompt.md` instructs the
LLM to delimit keywords by **ENGLISH COMMA**. For a dataflow canvas that
includes the Tokenizer component, this divergence made Go's indexed
`important_kwd` differ from the Python-DSL-built index (CJK
commas/semicolons were split in Go but kept whole in Python).
- **Fix:** Use `strings.Split(kw, ",")` at `tokenizer.go:701`,
preserving empty middle elements to match Python's `"a,,b".split(",") ==
["a","","b"]`. The indexing fallback layer
(`internal/ingestion/task/indexdoc/process.go`) already mirrors the
Python multi-delimiter fallback (`dataflow_service.py:322`), so only the
component layer diverged and only it is changed.
## Test plan
- `TestTokenizerComponent_ImportantKwd_CommaOnly` (no build tag, default
`go test ./...`): switches the tokenizer to the identity engine (no CGo
pool needed) and asserts `"kw1,kw2;kw3，kw4"` → `["kw1","kw2;kw3，kw4"]`;
also asserts `important_tks` still tokenizes the full keyword string.
- `TestTokenizerComponent_Invoke_KeywordSplitCommaOnly` (`integration`
tag, real CGo analyzer): covers comma-split, CJK/semicolon-not-split,
and empty-middle preservation.
- Both tiers pass (unit `ok`, integration `ok`).
## Regression notes
- Intentional behavior change for canvases that include the Tokenizer
component: keywords containing `;`/`、`/newlines now stay as one keyword
(matching Python DSL) instead of being split. Re-indexing existing
Go-built data will change the `important_kwd` set — expected parity
cost, documented in code comments and commit message.
- Canvases without a Tokenizer component are unaffected (they hit the
unchanged multi-delimiter fallback).
- Other fields (`important_tks`, `questions`, `summary`, `text`) are
untouched; the `utility` import was removed cleanly.
