### Summary
Closes#18414.
`rag/res/term.freq` is not shipped, and both term-weight implementations
therefore assigned the same `300` fallback frequency to every lowercase
Latin token. With no tokenizer frequency, NER, or POS signal, function
words and content words received identical lexical boosts.
This PR adds the same bounded out-of-vocabulary prior to Python and Go:
- Use it only when the explicit DF dictionary or tokenizer has no
frequency.
- Count Latin, Greek, and Cyrillic letters, including uppercase and
accented forms.
- Keep the existing frequency of `300` for words up to three letters,
halve it every two additional letters, and clamp it at `10`.
- Reject digits, underscores, and logographic terms so Chinese and other
existing fine-grained-tokenizer paths are unchanged.
- Treat an absent optional `term.freq` as the supported fallback path
without a startup warning, while still logging inaccessible or malformed
dictionaries.
A corpus-derived table was intentionally not added: that would require
provenance/licensing decisions, language detection, and handling
cross-language homographs. The bounded prior is deterministic,
dependency-free, and fixes the equal-weight degradation for
whitespace-delimited alphabetic languages without claiming
corpus-specific precision.
Python and Go consume one shared fixture covering ASCII, uppercase,
accented Latin, Greek, Cyrillic, separators, invalid mixed tokens, and a
CJK non-match. Both sides also verify the issue's ordering (`was <
largest < supplier < equipment`) and that an explicit dictionary entry
still takes precedence.
Co-authored-by: Loong <184861530+yzl0ng@users.noreply.github.com>
## Summary
- require a `valid_at` timestamp in the semantic-memory output schema
- tell the extraction model to use conversation time when a fact has no
date of its own
- add a regression test for the assembled semantic prompt
This addresses the deterministic prompt inconsistency reported in
#18415. The invalid-timestamp fallback is intentionally left unchanged
because selecting its replacement policy requires a separate design
decision.
## Summary
- allow ISO 8601 normalization callers to provide an explicit fallback
while preserving the existing default behavior
- use the extraction conversation time when `valid_at` is missing or
invalid
- clear an invalid optional `invalid_at` instead of writing an
unparseable value
- include the rejected timestamp value in the error log
This addresses the timestamp write-through portion of #18415. It is
intentionally separate from #18462, which fixes the semantic output
prompt.
Co-authored-by: Loong <184861530+yzl0ng@users.noreply.github.com>
### Summary
- Forward each memory's stored system_prompt and user_prompt to
extract_by_llm.
- Cover both immediate save and queued extraction paths with focused
regression tests.
- Preserve the existing default-prompt fallback when stored prompts are
empty.
Fixes#18413.
### Summary
RAGFlow's "Create empty document" flow accepts names without a file
extension. The `POST /datasets/<dataset_id>/documents?type=empty` route
calls `_upload_empty_document()`, where `Path(name).suffix.lstrip(".")`
returns `""`.
In GaussDB's A/ORA compatibility mode, that empty string is persisted as
SQL `NULL`. Because `document.suffix` was defined as `NOT NULL`, the
insert failed with a constraint violation.
### Summary
Providers whose static catalogue is empty discover their models by
calling the base URL the user typed. `verify_api_key` wrapped that call
in a bare `except Exception: pass` and then returned a flat `No models
found for provider 'X'`, so an unreachable host, a closed port, a wrong
scheme and a bad TLS setup all produced the same sentence, with the
actual error discarded and not even logged.
### What problem does this PR solve?
- Update version tags in README files (including translations) from
v0.26.4 to v0.27.0
- Modify Docker image references and documentation to reflect new
version
- Update version badges and image descriptions
- Maintain consistency across all language variants of README files
### Type of change
- [x] Documentation Update
## Summary
- treat empty Markdown binaries as in-memory content instead of local
file paths
- add a regression test ensuring empty content does not access the
filesystem
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
### Summary
`AsyncExecutor.Execute` acquires a worker-pool slot, while
`ExecuteWithRetry` duplicated the task lifecycle without acquiring one.
Pregel supplies a retry configuration for every node, so those
executions
bypassed `WithMaxConcurrency`.
This change:
- routes retry execution through the shared `Execute` path;
- waits on the task-owned context and rechecks cancellation after slot
acquisition;
- adds deterministic regression coverage for worker-pool occupancy and
queued-task cancellation.
## Summary
- Frontend `Images` already includes `bmp`
(`web/src/constants/common.ts`), and picture parsing already accepts
`.bmp` (`internal/parser/parser/picture_parser.go`, `rag/app/picture.py`
via Pillow).
- This PR adds `bmp` to both whitelist sites and a small Go unit test.
Co-authored-by: zhangjiangshan1 <zhangjiangshan1@kingsoft.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Refactor the Extractor component into a pure, unified **5-in-1 modular extraction engine** across both Dataset (`knowledgebase.parser_config`) and Pipeline (Canvas DSL).
## Summary
- Disable DeepSeek V4 thinking (chain-of-thought) by default.
- litellm 1.82.x drops `thinking: disabled`; carry the toggle through
`extra_body.thinking.type` and strip `reasoning_effort` to avoid the
400.
- Use local timezone for agent `sys.date` instead of UTC.
Reference: https://api-docs.deepseek.com/guides/thinking_mode
---------
Co-authored-by: Claude <claude@example.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Claude <noreply@anthropic.com>
This PR fixes two issues with MinerU PDF parser where table and image chunks were not properly classified or associated:
1. **Table chunks** were incorrectly classified as `text` instead of `table`
2. **Image chunks** were missing image resource association and classified as `text`
### Summary
GaussDB DocEngine could return no chunks for conversational queries even
when relevant content was available. `Dealer.search()` supplies
`minimum_should_match` (30%, then 10% on retry), but the GaussDB adapter
discarded it and built a single `plainto_tsquery` from every token. This
effectively required all conversational filler terms to match.
## Summary
The TitleChunker (both `hierarchy` and `group` methods) had no
token-size ceiling, so a long section without sub-headings became one
giant chunk. This adds a configurable `chunk_token_cap` that guarantees
every text chunk stays within a token budget.
### Summary
1. It changes the fallback semantics of the locate phase. When no chunks
are found, the system stays in locate. If the same claim has two
consecutive locate rounds with neither evidence chunks nor newly routed
document scope, web_search is admitted to the candidate tool set on the
next locate round as an external fallback.
2. It makes locate_empty_streak claim-scoped instead of shared in the
global context. This prevents one claim’s empty locate rounds from
affecting sibling claims running in parallel.
3. On the config side, it only raises max_parallel_agents for high /
ultra to 4, without changing max_agent_cycles. This increases parallel
claim execution without deepening per-claim search.
