Go's `TokenChunker` previously ignored **bare (non-backtick)
delimiters** such as `::`, `.`, or `；`. `compileDelimPattern` compiled
them with `keepBare=false`, so a bare delimiter produced a `nil` pattern
and the payload was routed to the single-section merge that never split
on it — diverging from Python's `naive_merge`, which splits on bare
delimiters and then merges by token size.
### Summary
Add Querit Contents as a built-in page content tool for RAGFlow Agents
and Canvas workflows.
Querit Contents crawls one or more known URLs and returns their page
content and optional metadata. It complements the existing Querit Search
tool: Search discovers relevant pages, while Contents retrieves pages
already selected by an Agent or workflow.
This integration supports two usage modes:
- A standalone `QueritContents` node in Canvas workflows.
- An embedded content tool available to RAGFlow Agents.
Adds a characterization / regression test
(`pdf_postprocess_measure_test.go`) that pins the **figure-aware reading
order** produced by the PDF adapter layer
(`internal/parser/parser/pdf_postprocess.go`).
Second ragflow PR (#17639 merged 2026-07-31). Docstrings only, no
behavior change.
---
🤖 Written with [Claude Code](https://claude.com/claude-code). Each site
was opened and read, and
each description was checked against the line that consumes the value.
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Make `ResolveModelContentLength` honor the per-model custom **context window length** (`content_length`) — stored in the Python-legacy `tenant_model.extra["max_tokens"]` field, whose semantic meaning is the context window, NOT the generation cap — **before** any provider-catalog read, and remove the parallel service-layer implementation so every consumer shares one resolution path.
Incremental follow-up to #18023 (gap + balance-gate hybrid). Adds two
complementary column detectors to `AssignColumn` that run **only after**
the gap detector and the balance gate both fail, so already-correct
pages are never touched.
Trim Extractor call prompts and the automatic tagger prompt to the chat model's context window (`content_length`) before sending, so oversized chunks or tag files are trimmed instead of rejected by the provider with a context-length error.
Switch the agent LLM component's `fitMessages` from the local map-based `messageFitInRaw`/`countAllTokens`/`stringContent`/`setContent` helpers to the new shared `internal/component/messagefit` package (#18091).
Some NVIDIA hosted models (e.g. meta/llama-3.2-11b-vision-instruct )
expose a full
endpoint URL per model that does not follow the normal base_url +
url_suffix
assembly. Previously the Go driver always called {base}/chat/completions
, so chat
requests for these vision models hit the wrong endpoint and failed.
This PR adds an optional per-model url field in conf/models/nvidia.json
. When
present, every NVIDIA driver request (chat, streaming chat, embedding,
rerank, model
listing) uses it directly; otherwise the standard assembly is unchanged.
Add a shared `dao.ResolveModelContentLength` that resolves a chat model's context window (`content_length`) from a `tenant_model` UUID or a composite `model@provider` reference, with an optional `driver + modelName` catalog fallback for the no-database path.
The agent LLM component and the ingestion Extractor component both need
to trim prompts to the model's context window before calling the
provider. Each previously did (or would do) this with its own copy of
the logic. This PR adds the shared primitive; follow-up PRs wire it into
the agent LLM component (#18092) and the ingestion Extractor/tagger
(#18095).
### Summary
`format_document_soup` tracks "am I inside a table" and "am I inside a
link" with sticky flags that are meant to be reset by `elif e.name ==
"/table"` and `elif e.name == "/a"`. BeautifulSoup's `.descendants` only
yields opening tags — a `Tag` named `/table` or `/a` never exists — so
both branches are dead code and neither flag is ever cleared.
Everything after the first `<table>` on a page is therefore formatted as
if it were still table content: paragraphs lose their newline, list
items lose their `- ` marker, headings lose their break, and the text is
glued onto the last table cell. Under
`HTML_BASED_CONNECTOR_TRANSFORM_LINKS_STRATEGY=markdown` the same bug
leaks a link's `href` into everything that follows it, including whole
subsequent paragraphs. The Confluence connector
(`confluence_connector.py:948`) goes through this path.
Real output for a Confluence-shaped page (heading, intro, spec table,
then the body) via the public `parse_html_page_basic`:
**Before**
```
prod us-east-1 Rollback procedure If the canary fails, run the rollback script immediately. Drain the load balancer Revert the deployment Escalate to the on-call rota if the rollback stalls. Do not skip the post-mortem.
```
**After**
```
prod us-east-1
Rollback procedure
If the canary fails, run the rollback script immediately.
- Drain the load balancer
- Revert the deployment
Escalate to [the on-call rota](http://oncall.example.com) if the rollback stalls.
Do not skip the post-mortem.
```
Every heading, paragraph and list marker after the table is lost, and
the whole body is indexed as one run-on line hanging off a table cell.
### Fix
Derive both scopes from each element's **ancestors** instead of from
flags that nothing can clear, and drop the two dead branches plus the
two that become redundant.
The scopes are resolved in one up-front pass into `id`-keyed maps
(`table_scope`, `href_scope`) and looked up in O(1) per element. Probing
per element with `find_parent` instead is O(depth) each, which measured
12–13× slower on table-heavy pages and up to 103× on deeply nested
markup; the map version costs a depth-independent 1.13–1.35× over
`main`. Numbers and method are in the round-2 comment below.
This also changes one adjacent behaviour worth calling out explicitly: a
link **inside** a table cell now renders as markdown, where before it
rendered as plain text. That previous behaviour was not by design — it
only held when no link preceded the table. With a link before the table,
`main` stamps the stale href onto every cell:
```
main: '[pre](http://STALE.com)\n\t[cellA](http://STALE.com)\t[cellB](http://STALE.com)'
branch: '[pre](http://STALE.com)\n\tcellA\tcellB'
```
Those cells are not links. Both symptoms are the same sticky-state bug,
so they are fixed together rather than left half-done.
### Testing
`test/unit_test/data_source/test_html_utils.py` is new —
`format_document_soup` had no test coverage. 11 tests: 8 fail on `main`
and pass on this branch, 3 are controls that pass on both (the table
itself still separates rows and cells, anchor text is still linkified,
the default `strip` strategy still strips).
Representative failures on `main`:
```
assert '\nAfter' in 'Before\n\tA\tB After'
assert '\n- item1' in 'Before\n\tA\tB item1 item2'
assert 'see [link](http://x.com) [ after](http://x.com)' == 'see [link](http://x.com) after'
assert '[next paragraph]' not in '[link](http://x.com)\n[next paragraph](http://x.com)'
```
Reverting each clause of the fix independently keeps the anchors honest:
reverting only the table clause fails exactly the 4 table tests and
leaves the link tests green; reverting only the link clause fails
exactly the 3 link tests and leaves the table tests green.
(`test_link_inside_a_table_cell_is_linkified` needs both clauses broken
to fail, so it appears in neither single-clause revert — it is covered
by the 8-fail run against `main`.)
Full `test/unit_test/data_source/` suite: **3 failed, 199 passed**, and
the failure set is byte-identical to clean `main` (**3 failed, 188
passed**) — the 3 are `TestSSRFValidation::*`, which resolve
`api.example.com` against real DNS and are unrelated to this change.
`ruff check` and `ruff format --check` are clean on both touched files.
---
This PR was drafted with AI assistance (Claude). I reviewed the change,
independently reproduced both symptoms against `main`, and take
responsibility for it.
### Summary
- Propagate the dataset language through Go DOCX, Markdown, PDF
figure-enhancement, and standalone-image vision paths.
- Explicitly render the shared figure prompt's `{{ language }}`
placeholder in Go.
- Use English when the dataset language is empty.
- Make the default standalone-image prompt request the dataset language
while preserving visible text in its original language.
- Add focused tests for caller propagation, language fallback, prompt
rendering, and prompt-cache isolation.
