When `TokenChunker` starts a fresh chunk with an overlap prefix (Go `computeOverlapPrefix` / Python visible-text cut), the previous chunk's **tail PDF coordinates were dropped**. As a result, the overlap head of a PDF chunk is displayed but **not highlighted** — the highlight box is shifted/truncated relative to the displayed span (infiniflow/ragflow#18148).
Collapse the nine identical **"cut through the last `</think>`"**
implementations — mirroring Python's `re.sub(r"^.*</think>", "", s,
re.DOTALL)` — into one shared helper `common.StripThinkTrailing`,
preventing future behavior drift between copies.
Fix compilation template config validation for JSONMap; merge template groups into agents list ordered by category/name; install nav service in ingestor; write readable nav cluster/doc names and emit nav_doc leaves;
port tree-to-graph projection and full document structure graph endpoint parity.
Go's `TokenChunker` previously ignored **bare (non-backtick)
delimiters** such as `::`, `.`, or `；`. `compileDelimPattern` compiled
them with `keepBare=false`, so a bare delimiter produced a `nil` pattern
and the payload was routed to the single-section merge that never split
on it — diverging from Python's `naive_merge`, which splits on bare
delimiters and then merges by token size.
### Summary
Add Querit Contents as a built-in page content tool for RAGFlow Agents
and Canvas workflows.
Querit Contents crawls one or more known URLs and returns their page
content and optional metadata. It complements the existing Querit Search
tool: Search discovers relevant pages, while Contents retrieves pages
already selected by an Agent or workflow.
This integration supports two usage modes:
- A standalone `QueritContents` node in Canvas workflows.
- An embedded content tool available to RAGFlow Agents.
Adds a characterization / regression test
(`pdf_postprocess_measure_test.go`) that pins the **figure-aware reading
order** produced by the PDF adapter layer
(`internal/parser/parser/pdf_postprocess.go`).
Make `ResolveModelContentLength` honor the per-model custom **context window length** (`content_length`) — stored in the Python-legacy `tenant_model.extra["max_tokens"]` field, whose semantic meaning is the context window, NOT the generation cap — **before** any provider-catalog read, and remove the parallel service-layer implementation so every consumer shares one resolution path.
Incremental follow-up to #18023 (gap + balance-gate hybrid). Adds two
complementary column detectors to `AssignColumn` that run **only after**
the gap detector and the balance gate both fail, so already-correct
pages are never touched.
Trim Extractor call prompts and the automatic tagger prompt to the chat model's context window (`content_length`) before sending, so oversized chunks or tag files are trimmed instead of rejected by the provider with a context-length error.
Switch the agent LLM component's `fitMessages` from the local map-based `messageFitInRaw`/`countAllTokens`/`stringContent`/`setContent` helpers to the new shared `internal/component/messagefit` package (#18091).
Some NVIDIA hosted models (e.g. meta/llama-3.2-11b-vision-instruct )
expose a full
endpoint URL per model that does not follow the normal base_url +
url_suffix
assembly. Previously the Go driver always called {base}/chat/completions
, so chat
requests for these vision models hit the wrong endpoint and failed.
This PR adds an optional per-model url field in conf/models/nvidia.json
. When
present, every NVIDIA driver request (chat, streaming chat, embedding,
rerank, model
listing) uses it directly; otherwise the standard assembly is unchanged.
Add a shared `dao.ResolveModelContentLength` that resolves a chat model's context window (`content_length`) from a `tenant_model` UUID or a composite `model@provider` reference, with an optional `driver + modelName` catalog fallback for the no-database path.
The agent LLM component and the ingestion Extractor component both need
to trim prompts to the model's context window before calling the
provider. Each previously did (or would do) this with its own copy of
the logic. This PR adds the shared primitive; follow-up PRs wire it into
the agent LLM component (#18092) and the ingestion Extractor/tagger
(#18095).
### Summary
- Propagate the dataset language through Go DOCX, Markdown, PDF
figure-enhancement, and standalone-image vision paths.
- Explicitly render the shared figure prompt's `{{ language }}`
placeholder in Go.
- Use English when the dataset language is empty.
- Make the default standalone-image prompt request the dataset language
while preserving visible text in its original language.
- Add focused tests for caller propagation, language fallback, prompt
rendering, and prompt-cache isolation.
Stop flattening `<table>` into a single text blob. A `<table>` now emits:
1. an inlined `doc_type_kwd:"text"` item keeping the `<table>…</table>`
markup (row/column structure survives for embedding/retrieval/LLM rendering),
2. a structured `doc_type_kwd:"table"` / `ck_type:"table"` item appended
after the walk, consumed by the downstream chunker.
Port the wiki_incremental dataset-level merge and make its rewrite
barrier durable and concurrency-safe. Wiki pages merge replace-only; the
barrier persists a monotonic numeric generation, and a scheduler-backed
per-dataset lock closes the cross-process TOCTOU window. Adds the
Compiler Plan toggle (frontend) with Mode A grouping.
## What
This pull request adds **MWS GPT Model Hub** as a built-in model
provider in RAGFlow.
The integration allows users to configure an MWS project endpoint and
token, discover the models available to that project, and use supported
MWS models for chat completion, embeddings, and reranking.
Co-authored-by: ilarionov_n <ilarionov_n@promis.ru>
Centralize the shared golden-doc + alignment helpers in `align_test.go` so the format-specific PRs (text&code, markdown golden, HTML) reuse one implementation instead of each carrying their own copy of the scaffolding.
Golden-parity test infrastructure for the **Go `TokenChunker` ↔ Python alignment**.
It runs the Go chunker over a committed case set (`testdata/parity/cases/`) and diffs each output against a captured Python golden (`testdata/parity/golden/`), honoring a `known_diffs.json` ratchet (`extra_fields` / `chunk_count` / `chunk_text`) so accepted divergences are tracked rather than silently widening.
