Some NVIDIA hosted models (e.g. meta/llama-3.2-11b-vision-instruct )
expose a full
endpoint URL per model that does not follow the normal base_url +
url_suffix
assembly. Previously the Go driver always called {base}/chat/completions
, so chat
requests for these vision models hit the wrong endpoint and failed.
This PR adds an optional per-model url field in conf/models/nvidia.json
. When
present, every NVIDIA driver request (chat, streaming chat, embedding,
rerank, model
listing) uses it directly; otherwise the standard assembly is unchanged.
Add a shared `dao.ResolveModelContentLength` that resolves a chat model's context window (`content_length`) from a `tenant_model` UUID or a composite `model@provider` reference, with an optional `driver + modelName` catalog fallback for the no-database path.
The agent LLM component and the ingestion Extractor component both need
to trim prompts to the model's context window before calling the
provider. Each previously did (or would do) this with its own copy of
the logic. This PR adds the shared primitive; follow-up PRs wire it into
the agent LLM component (#18092) and the ingestion Extractor/tagger
(#18095).
### Summary
- Propagate the dataset language through Go DOCX, Markdown, PDF
figure-enhancement, and standalone-image vision paths.
- Explicitly render the shared figure prompt's `{{ language }}`
placeholder in Go.
- Use English when the dataset language is empty.
- Make the default standalone-image prompt request the dataset language
while preserving visible text in its original language.
- Add focused tests for caller propagation, language fallback, prompt
rendering, and prompt-cache isolation.
Stop flattening `<table>` into a single text blob. A `<table>` now emits:
1. an inlined `doc_type_kwd:"text"` item keeping the `<table>…</table>`
markup (row/column structure survives for embedding/retrieval/LLM rendering),
2. a structured `doc_type_kwd:"table"` / `ck_type:"table"` item appended
after the walk, consumed by the downstream chunker.
Port the wiki_incremental dataset-level merge and make its rewrite
barrier durable and concurrency-safe. Wiki pages merge replace-only; the
barrier persists a monotonic numeric generation, and a scheduler-backed
per-dataset lock closes the cross-process TOCTOU window. Adds the
Compiler Plan toggle (frontend) with Mode A grouping.
## What
This pull request adds **MWS GPT Model Hub** as a built-in model
provider in RAGFlow.
The integration allows users to configure an MWS project endpoint and
token, discover the models available to that project, and use supported
MWS models for chat completion, embeddings, and reranking.
Co-authored-by: ilarionov_n <ilarionov_n@promis.ru>
Centralize the shared golden-doc + alignment helpers in `align_test.go` so the format-specific PRs (text&code, markdown golden, HTML) reuse one implementation instead of each carrying their own copy of the scaffolding.
Golden-parity test infrastructure for the **Go `TokenChunker` ↔ Python alignment**.
It runs the Go chunker over a committed case set (`testdata/parity/cases/`) and diffs each output against a captured Python golden (`testdata/parity/golden/`), honoring a `known_diffs.json` ratchet (`extra_fields` / `chunk_count` / `chunk_text`) so accepted divergences are tracked rather than silently widening.
Port dataset-level wiki incremental compile and refactor splitByTokens
token budgeting. Includes replace-only wiki merge, KNN dedup routing,
and template/config wiring.
## What changed
- add an OceanBase/SeekDB Go document engine using `database/sql` and
the existing MySQL driver
- preserve the Python connector's configuration, physical table names,
schema, index names, and ARRAY/JSON/VECTOR encodings
- implement chunk, memory, document metadata, skill, SQL, full-text,
vector, and fusion search paths
- support `DBMS_HYBRID_SEARCH.SEARCH` behind the existing feature flag,
with SQL fallback only when the package is unavailable
- wire the engine into retrieval, memory, metadata, vector hydration,
and SQL chat flows
- add Python/Go compatibility contracts, SQL mock tests, and an
integration-tagged round-trip test
---------
Co-authored-by: Jin Hai <haijin.chn@gmail.com>
Aligns the Go EML parser (`internal/parser/parser/email_parser.go`) with the Python flow parser (`rag/flow/parser/parser.py:_email`) on two structural points so the Go path is a faithful drop-in for the Python path on `.eml` inputs.
Unifies the Go TokenChunker merge path on a single `mergeUnits` core and
fixes coordinate-tag drift in the Python JSON merge at `overlap > 0`.
Rebased on top of #17979 (delimiter_mode convergence).
Re-materialize wiki page graph from merged wiki_page rows after each
batch merge. Adds ProjectWikiGraph/DropWikiGraph, full page_type/slug
identity, delete-then-insert, tests.
Converge `TokenChunker.delimiter_mode` from three values (`token_size`,
`delimiter`, `one`) to two (`delimiter`, `one`). The unified `delimiter`
mode now carries the old `token_size` semantics: when no active
(backtick) delimiter is present, text/JSON chunks are merged up to
`chunk_token_size`; when a backtick delimiter is present, the text is
split by it and not merged. `one` continues to be handled by the
separate `OneChunker`.
Restore the deduplication that was dropped when #17926 was merged.
`CompileDelimiterPatternList` now keeps a `seen` set and collapses
equivalent active entries (both backtick-inner and bare) into a single
alternation. This PR also removes the dead code that the re-review
surfaced.
Collapses three duplicated/dead code smells in the canvas debug-result
path of `internal/ingestion/task` (remaining findings #2/#3/#4 from the
package CTO review):
- **#2 — vector-key stripping duplication.** `isVectorKey` (the full
stripper: fixed legacy keys `vector`/`embedding`/`feature`/`q_vec` plus
the `q_<dim>_vec` pattern) was re-implemented as a weaker inline copy in
the golden-compare tool (`tool/compare_pipeline_golden.go`) that only
matched `q_<dim>_vec` and let real vectors leak into the diff. Exported
as `IsVectorKey` and reused by the tool.
- **#3 — near-duplicate deep copy.** `deepCopy` and `deepCopyStrip` were
identical walkers differing only in vector stripping. Parameterized
`deepCopy(v any, stripVector bool) any` and deleted `deepCopyStrip`.
- **#4 — redundant switch.** `lookupComponentOutput` had two switch
cases with identical bodies (both the `map[string]map[string]any` and
`map[string]any` state shapes). Unified into a single `found`/`ok`
resolution; nested-state and flat-fallback semantics unchanged.
## Summary
- Add embedding batch-size metadata to model responses and tenant
overrides.
- Validate embedding dimensions and batch limits across provider
verification and embedding requests.
- Expand validation tests for defaults, limits, and missing metadata.
---------
Signed-off-by: Jin Hai <haijin.chn@gmail.com>
Co-authored-by: Jin Hai <haijin.chn@gmail.com>
`compileChildrenPattern` re-implemented the delimiter-list compile
inline with two divergences from the shared
`CompileDelimiterListPattern`:
- It never stripped backticks, so a backtick-wrapped
`children_delimiter` like `` `###` `` matched the **literal wrapped
token** rather than the inner `###`.
- It sorted by **byte length** instead of rune count (`sortSlice`), so
multi-byte delimiters could be ordered incorrectly and a longer
delimiter could fail to win over a shorter prefix.
