### What problem does this PR solve?
Fixes#17074. When a keyword-extraction LLM returns a malformed,
extremely long response, the resulting `important_kwd` term can exceed
Elasticsearch's keyword term-byte limit (32766 bytes). The chunk insert
then fails with `document_parsing_exception`.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
Signed-off-by: Rahil <viperboom0786110@gmail.com>
Co-authored-by: Jin Hai <haijin.chn@gmail.com>
### Summary
- keep literal whitespace outside streamed message references
- retain single, double, and spaced outer-brace reference forms
- apply the same boundary rule to iteration aliases
- add regression coverage for adjacent and consecutive references
Fixes#18665
---------
Co-authored-by: Jin Hai <haijin.chn@gmail.com>
Fixes#18621
Legacy .doc files are OLE/CFB compound documents; python-docx only
parses OOXML .docx ZIP packages. The Manual parser's suffix regex
`\.docx?$` (introduced in PR #4325) matches both extensions, so .doc
uploads were routed to `Docx()` and crashed with raw `BadZipFile` / OPC
relationship errors instead of a clear message.
### Summary
Fix OpenSearch retrieval returning `vector_similarity = 0.000` for every
chunk when hybrid search is enabled.
On the OpenSearch backend, retrieval uses a second KNN-only search
(`Dealer._knn_scores()`) to recover per-chunk cosine scores for
reranking. That pass intentionally sends `MatchDenseExpr(...,
{"similarity": 0.0})` to mean “no minimum similarity cutoff.”
However, `OSConnection.search()` was incorrectly mapping `similarity` to
the KNN clause `boost` field:
```python
knn_query[vector_column_name]["boost"] = similarity
```
With `similarity=0.0`, this produced `boost=0.0`, which zeroed out KNN
`_score` values. `get_scores()` then returned `0.0` for every hit, so
`vector_similarity` was always zero and hybrid ranking ignored the
vector component — with no exception raised.
This is separate from the `get_scores()` `AttributeError` crash
addressed in #14970 / #15390; here retrieval succeeds but vector scores
are silently lost.
## Summary
- Broaden `update_metadata_to` to keep `bool` / `int` / `float` / `None`
and structured values (e.g. PDF `outline` list[dict]) while retaining
string / list[str] merge+dedupe for LLM metadata.
- Fixes document system fields (`_isCurrent`, `_version`,
`_processStatus`) being dropped when parse persists PDF outline via full
`meta_fields` replace.
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
## Summary
- route REST retrieval metadata conditions through the existing
metadata-index push-down service
- keep full metadata loading lazy and preserve the Python in-memory
filter as an exact fallback
- add focused coverage for successful and empty push-down results and
multi-valued fallback semantics
### Summary
The fix is one line per helper — assign the `order_by(...)` result back
so the
ordering is actually applied to the query.
Co-authored-by: rsnetworkinginc <rsnetworkinginc@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
### Summary
Complete the Go Agent OpenAI-compatible response path for `POST
/api/v1/agents/chat/completions` when `openai-compatible` is true. This
endpoint is already documented and routed, but the Go implementation
returned a placeholder response instead of executing the Agent.
### Summary
The fix turns a negative timeout into a finite 60 second wait inside
`MysqlDatabaseLock`, which both servers accept. Callers keep expressing
"block until available", and the equivalent PostgreSQL and GaussDB work
on the same `-1` semantics (#16346, #18506) is untouched. Non-negative
timeouts keep their current behaviour.
---------
Co-authored-by: Krim <git@krim.dev>
### Summary
OIDC discovery builds its URL as
`f"{issuer}/.well-known/openid-configuration"`. Providers whose issuer
carries a trailing slash therefore get asked for a URL with a double
slash in it, which 404s. authentik is one of them: its issuer is
`https://auth.example.com/application/o/<app>/`, so the request goes to
`.../o/<app>//.well-known/openid-configuration` and login fails right at
the start with `Failed to fetch OIDC metadata`.
Stripping trailing slashes off the issuer before joining the well-known
path is enough. The `issuer` used later for ID token validation still
comes from the discovery document itself, so nothing else about the flow
changes.
Co-authored-by: Krim <git@krim.dev>
### Summary
Multi-type memory prompts were assembled from sets, so their instruction
and output sections could change order across Python processes with
different hash seeds. Because the generated default prompt is persisted
and later compared as a string, a restart could make an untouched
default look custom and prevent it from being regenerated when memory
types change.
### Summary
Adds You.com as a built-in Web Search provider for RAGFlow Chat,
alongside Tavily and Querit, using the provider-neutral dispatch #17813
put in place. No changes to existing Tavily or Querit behaviour.
You.com runs its own web index and returns several extracted passages
per result rather than a single meta description, so retrieved chunks
arrive with usable context.
---------
Co-authored-by: Brian Sparker <brainsparker@users.noreply.github.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
## Summary
Fixes#18025. Both merge paths deduplicated IDs by scanning a plain list
(`item not in list`) inside a loop while appending — O(n²) per merge.
Replaced with a set-backed `seen` check alongside the existing ordered
list: same order, same dedup result, O(n).
---------
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
An image whose OCR text is shorter than the CV LLM threshold produces zero chunks when the tenant has no image2text model configured. The extracted text is discarded.
## Summary
- Pass `vector_similarity_weight` from Python and Go retrieval requests
into Infinity's weighted fusion expression.
- Keep fusion weights ordered as text first and vector second, with the
existing default vector weight of `0.3`.
---------
Co-authored-by: chenglinpeng <1042527908@qq.com>
### Summary
This PR adds [Serply](https://serply.io) as a third web search provider
for chat assistants, alongside the existing Tavily and Querit options.
### Summary
Closes#18414.
`rag/res/term.freq` is not shipped, and both term-weight implementations
therefore assigned the same `300` fallback frequency to every lowercase
Latin token. With no tokenizer frequency, NER, or POS signal, function
words and content words received identical lexical boosts.
This PR adds the same bounded out-of-vocabulary prior to Python and Go:
- Use it only when the explicit DF dictionary or tokenizer has no
frequency.
- Count Latin, Greek, and Cyrillic letters, including uppercase and
accented forms.
- Keep the existing frequency of `300` for words up to three letters,
halve it every two additional letters, and clamp it at `10`.
- Reject digits, underscores, and logographic terms so Chinese and other
existing fine-grained-tokenizer paths are unchanged.
- Treat an absent optional `term.freq` as the supported fallback path
without a startup warning, while still logging inaccessible or malformed
dictionaries.
A corpus-derived table was intentionally not added: that would require
provenance/licensing decisions, language detection, and handling
cross-language homographs. The bounded prior is deterministic,
dependency-free, and fixes the equal-weight degradation for
whitespace-delimited alphabetic languages without claiming
corpus-specific precision.
Python and Go consume one shared fixture covering ASCII, uppercase,
accented Latin, Greek, Cyrillic, separators, invalid mixed tokens, and a
CJK non-match. Both sides also verify the issue's ordering (`was <
largest < supplier < equipment`) and that an explicit dictionary entry
still takes precedence.
Co-authored-by: Loong <184861530+yzl0ng@users.noreply.github.com>
## Summary
- require a `valid_at` timestamp in the semantic-memory output schema
- tell the extraction model to use conversation time when a fact has no
date of its own
- add a regression test for the assembled semantic prompt
This addresses the deterministic prompt inconsistency reported in
#18415. The invalid-timestamp fallback is intentionally left unchanged
because selecting its replacement policy requires a separate design
decision.
## Summary
- allow ISO 8601 normalization callers to provide an explicit fallback
while preserving the existing default behavior
- use the extraction conversation time when `valid_at` is missing or
invalid
- clear an invalid optional `invalid_at` instead of writing an
unparseable value
- include the rejected timestamp value in the error log
This addresses the timestamp write-through portion of #18415. It is
intentionally separate from #18462, which fixes the semantic output
prompt.
Co-authored-by: Loong <184861530+yzl0ng@users.noreply.github.com>
### Summary
- Forward each memory's stored system_prompt and user_prompt to
extract_by_llm.
- Cover both immediate save and queued extraction paths with focused
regression tests.
- Preserve the existing default-prompt fallback when stored prompts are
empty.
Fixes#18413.
### Summary
RAGFlow's "Create empty document" flow accepts names without a file
extension. The `POST /datasets/<dataset_id>/documents?type=empty` route
calls `_upload_empty_document()`, where `Path(name).suffix.lstrip(".")`
returns `""`.
In GaussDB's A/ORA compatibility mode, that empty string is persisted as
SQL `NULL`. Because `document.suffix` was defined as `NOT NULL`, the
insert failed with a constraint violation.
### Summary
Providers whose static catalogue is empty discover their models by
calling the base URL the user typed. `verify_api_key` wrapped that call
in a bare `except Exception: pass` and then returned a flat `No models
found for provider 'X'`, so an unreachable host, a closed port, a wrong
scheme and a bad TLS setup all produced the same sentence, with the
actual error discarded and not even logged.
### What problem does this PR solve?
- Update version tags in README files (including translations) from
v0.26.4 to v0.27.0
- Modify Docker image references and documentation to reflect new
version
- Update version badges and image descriptions
- Maintain consistency across all language variants of README files
### Type of change
- [x] Documentation Update
## Summary
- treat empty Markdown binaries as in-memory content instead of local
file paths
- add a regression test ensuring empty content does not access the
filesystem
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
## Summary
- Disable DeepSeek V4 thinking (chain-of-thought) by default.
- litellm 1.82.x drops `thinking: disabled`; carry the toggle through
`extra_body.thinking.type` and strip `reasoning_effort` to avoid the
400.
- Use local timezone for agent `sys.date` instead of UTC.
Reference: https://api-docs.deepseek.com/guides/thinking_mode
---------
Co-authored-by: Claude <claude@example.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Claude <noreply@anthropic.com>
This PR fixes two issues with MinerU PDF parser where table and image chunks were not properly classified or associated:
1. **Table chunks** were incorrectly classified as `text` instead of `table`
2. **Image chunks** were missing image resource association and classified as `text`
### Summary
GaussDB DocEngine could return no chunks for conversational queries even
when relevant content was available. `Dealer.search()` supplies
`minimum_should_match` (30%, then 10% on retry), but the GaussDB adapter
discarded it and built a single `plainto_tsquery` from every token. This
effectively required all conversational filler terms to match.
Follow-up to #17526 ("Refactor: merge dataset scope graph"), which introduced two code paths that touch Infinity columns the deployed schema does not declare. This PR makes the runtime robust against the old schema while also adding the new column to the new schema so freshly created tables are correct.
