## Summary
Remediates nine litellm CVEs (3 CRITICAL, 6 HIGH) by upgrading the exact
pin from `==1.82.5` to `==1.84.0` in `pyproject.toml`.
| CVE | Severity | Fixed in |
|---|---|---|
| CVE-2026-35030 | CRITICAL | 1.83.0 |
| CVE-2026-42208 | CRITICAL | 1.83.7 |
| CVE-2026-49468 | CRITICAL | 1.84.0 |
| CVE-2026-35029 | HIGH | 1.83.0 |
| CVE-2026-40217 | HIGH | 1.83.10 |
| CVE-2026-42203 | HIGH | 1.83.7 |
| CVE-2026-42271 | HIGH | 1.83.7 |
| CVE-2026-47101 | HIGH | 1.83.14 |
| CVE-2026-47102 | HIGH | 1.83.10 |
`litellm` was pinned to `==1.82.5` because versions 1.82.6 - 1.82.8 had
a broken `litellm.caching.caching` import chain, and 1.88.0 had a broken
`litellm.integrations.custom_logger` import.
Both issues are resolved in 1.84.0.
The exact pin is retained (rather than a range) due to litellm's history
of introducing regressions on minor bumps.
## Problem
`TestMergeByTokenSizeFromJSON_ClampsOverlappedPct` reused a single
`items` fixture across four calls to `mergeByTokenSizeFromJSON`.
`mergeByTokenSizeFromJSON` mutates its `perItem` argument in place and
returns the same backing array (token.go: `perItem[idx] = merged`). As a
result:
- The 2nd and later calls merged already-merged chunks instead of the
original input.
- Because the returned slice aliases the input, later calls silently
overwrote the earlier results (`at100`, `at0`, ...).
- `reflect.DeepEqual(at100, at150)` was therefore vacuously true — the
test was a **false positive** that never actually exercised the clamp.
It would still pass even if the clamp were broken.
This is exactly the defect flagged in the review comment on PR #17396.
## Fix
Add a `clampOverlapFixture()` factory and build a fresh fixture for
every call, so that 150 / 1e300 / -5 / -1e300 are applied to the
original input.
## Verification
- The test passes after the fix.
- When the clamp logic was temporarily disabled, the test **failed**
(panic at token.go:768 — the negative index produced by an out-of-range
pct), proving the fixed test is no longer a false positive and can catch
a clamp regression.
## Scope
Test file only. No production code change (verified `token.go` is
identical to `upstream/main`).
Refs: review comment on PR #17396.
## Summary
Remediates five nltk CVEs by adding `nltk>=3.10.0` to
`constraint-dependencies` in `pyproject.toml`.
| CVE | Severity | Installed | Fixed in |
|---|---|---|---|
| CVE-2025-14009 | CRITICAL | 3.9.2 | 3.9.3 |
| CVE-2026-0846 | HIGH | 3.9.2 | 3.9.3 |
| CVE-2026-0847 | HIGH | 3.9.2 | — |
| CVE-2026-33231 | HIGH | 3.9.2 | 3.9.4 |
| CVE-2026-33236 | HIGH | 3.9.2 | — |
| CVE-2026-54293 | HIGH | 3.9.2 | 3.10.0 |
| CVE-2026-33230 | MEDIUM | 3.9.2 | 3.9.4 |
CVE-2026-0847 and CVE-2026-33236 have no confirmed fixed version.
`nltk` is a transitive dependency (pulled in by `crawl4ai` and
`infinity-sdk`). No package in the graph caps it below 3.10.0. It was
stuck at 3.9.2 simply because the lockfile had never been re-resolved.
## Summary
- Parse chat, embedding, and rerank usage from provider responses
- Record usage with the correct model type even when no usage sink is
provided
- Cover SiliconFlow, Aliyun, Huawei Cloud, Qiniu, and VolcEngine
response formats
## Summary
Closes two Chunker migration gaps documented in
`docs/migration_python_go_diff.md`
(diffs **2.6** and **1.7**), improving parity with the Python ingestion
pipeline.
This is the code portion of commit `261e1fd0b` on branch
`fix/batch-3-4`; the
migration document itself is tracked separately (untracked in this
commit).
### Diff 2.6 — `overlapped_percent` missing Python normalization
Python's `normalize_overlapped_percent` (`common/float_utils.py:50-58`)
accepts a
`[0,1)` fraction (the flow-canvas UI validates `[0,1)`), multiplies it
by 100,
`int()`-truncates, and clamps to `[0,90]`. Go previously only accepted a
raw
`[0,90]` percentage and **rejected** out-of-range input, so a Python
config
passing `0.1` (meaning 10%) silently produced ~0% overlap.
Added `normalizeOverlappedPercent`
(`internal/ingestion/component/chunker/common.go`) mirroring the Python
helper:
- parses numbers and numeric strings (mirrors Python `float()`; bad /
`NaN` / `Inf` → `0`),
- `0 < v < 1 → v *= 100`,
- `int()` truncation,
- clamps to `[0,90]`.
Wired into `tokenChunkerParam.Update` (`token.go`); the merge math
(`token.go:705`, `(100-x)/100`) already matched Python.
`TokenChunkerParam.Validate`
(`schema/chunker.go`) now only guards direct struct construction.
Tests: `TestNormalizeOverlappedPercent`, extended
`TestTokenChunker_NewAcceptsPythonOverlappedRange` (adds fraction/clamp
inputs),
new `TestTokenChunker_NormalizesOverlappedPercent`. The existing
reject-test cases
for `<0` / `>90` were removed because they are now normalized/clamped
(Python parity).
### Diff 1.7 — missing `BULLET_PATTERN` title-level fallback
`resolveTitleLevels` (`title.go`) now applies a 4th-level fallback: when
outline +
regex + layout all yield body level, `bulletsCategory` selects the
best-matching
bullet-pattern group (Chinese legal / numbering / Chinese numbering /
English legal
— mirroring `rag/nlp/__init__.py:258-320`) and assigns structural
levels. Guarded by
`allBodyLevel` so it never overrides an existing outline/regex level.
Tests: `TestResolveTitleLevels_BulletFallback` (4 subtests).
### Incidental test adjustments included in the commit
- `token_batch1_test.go`: overlap input changed `0.3` → `30.0` to
reflect the
post-normalization 30% semantics.
- `real_consumer_test.go`: updated `LoadFromIngestionTask(task)` →
`LoadFromIngestionTask(ctx, task)` for the new context-first signature.
## Verification
`bash build.sh --test ./internal/ingestion/component/...` — chunker +
schema suites
pass, no regression (CGO build).
## Migration doc reference
`docs/migration_python_go_diff.md` §Chunker 1.7 and 2.6 are marked
**Fixed** for
these changes.
## Summary
Improve the UX of the **"Delimiter for text"** field on the dataset
configuration page. The field is a single string with a backtick-based
mini-syntax, but both the tooltip and the surrounding UI failed to
surface what delimiters the backend would actually derive from a given
value — leaving users to discover by trial-and-error that the same
string produces different splits depending on file type (see #7436,
#4704, #9680).
### Summary
The FunASR provider added in #17171 defaults to a local
`http://localhost:8000/v1` server, but it still inherited the global
API-key requirement and unconditionally built an Authorization header.
This prevented the default unauthenticated self-hosted deployment from
working. The transcription path also dereferenced a missing model name
while building its multipart request.
This change:
- allows an empty API key for FunASR, matching other local providers
- omits the Authorization header when no key is configured while
preserving trimmed Bearer authentication when one is provided
- validates and trims the ASR model name before building the multipart
request, returning an error instead of panicking
- adds HTTP-level regression coverage for unauthenticated
transcription/model listing and optional authentication
Fix broken Table of Contents anchor link (📌 vs 🔥 emoji mismatch),
navigation bar link text ("Document" → "Documentation"), Title Case
alignment between ToC entries and section headings, and inconsistent
file format capitalization.
### Summary
This PR adds **aimlapi.com** as a model provider, so a RAGFlow user can
enter one API key in the model settings and use AIMLAPI's models across
the app. AIMLAPI ([aimlapi.com](https://aimlapi.com)) is an
OpenAI-compatible aggregator that serves 700+ models (LLM, embedding,
vision, TTS, ASR) from many providers behind a single API.
The change mirrors the repo's existing "add provider" pattern (e.g.
FuturMix / OpenRouter): provider logic lives in the same files those
providers use, and shared / UI files get only registration entries.
**Backend**
- `conf/llm_factories.json` — the `aimlapi.com` factory entry.
- `rag/llm/__init__.py`, `rag/llm/{chat,embedding,cv}_model.py` —
LiteLLM adapters (chat, embedding, image2text) with a production base
URL, overridable via `AIMLAPI_API_URL`.
- `rag/llm/model_meta.py` — an `AIMLAPI` model-meta so the provider
lists its full `/v1/models` catalog dynamically (classified by the
endpoint `type`), the same way OpenRouter does.
- `api/apps/restful_apis/aimlapi_api.py` — an optional "Get API key"
flow using AIMLAPI's agent-authorization (OAuth 2.0 Device Authorization
Grant, RFC 8628). The device code is kept server-side (Redis); only the
issued key reaches the browser.
**Frontend (`web/`)**
- Provider registration (constant, icon allowlist, brand logo), the
model picker (`LIST_MODEL_PROVIDERS` + a `buildLocalConfig` entry), and
the "Get API key" button in the provider dialog. Locales added to `en`
and `zh`.
**Configuration** — production defaults are compiled in; endpoints and
the partner id are overridable through `AIMLAPI_*` environment
variables, so the same build works across environments.
**Testing** — the `web` build passes; chat, embedding and dynamic model
listing were smoke-tested against the live API.
## Description
This PR fixes a typo in comment text and a missing verb in error output
in `show_env.sh`.
## Details
1. Fixed typo in comment (`lsd_release` $\to$ `lsb_release`).
2. Added missing verb in fallback output string (`It NOT a Git repo`
$\to$ `It is NOT a Git repo`).
---------
Co-authored-by: ferkans-amir <amir.rezaei@tu-berlin.de>
### What problem does this PR solve?
Adds first-class support for **Mistral OCR** (`POST /v1/ocr`) as a
document parser, and fixes the long-standing bug where selecting
`mistral-ocr-latest` fails with `Can't find model for
<tenant>/image2text/mistral-ocr-latest`.
`mistral-ocr-latest` is Mistral's dedicated document-OCR endpoint, not a
vision-chat (`image2text`) model, but the catalog tagged it `image2text`
— so it resolved to the `CvModel` registry, which has no `Mistral`
entry, and there was no `OcrModel` entry either. This PR registers it
correctly and wires it end to end.
Closes#17056Closes#5782Closes#7075
**What it does**
1. **`MistralParser` + `MistralOcrModel`**
(`deepdoc/parser/mistral_parser.py`, `rag/llm/ocr_model.py`) — a proper
`OcrModel` factory `Mistral OCR`, mirroring the SoMark cloud-OCR
template. Tables stay inline as HTML; the page range maps to Mistral's
native `pages` selector (absolute page indices, billed per selected
page, so multi-task documents do not re-OCR the whole file); documents
over the inline limit go through the `/v1/files` signed-URL flow with
cleanup.
2. **Removes the `image2text` mis-tag** for `mistral-ocr-latest` from
the `Mistral` factory in `conf/llm_factories.json` (it now lives only in
the `Mistral OCR` factory, typed `ocr`). This is what closes the `Can't
find model` path.
3. **`MistralCV`** (`rag/llm/cv_model.py`) — a thin `GptV4` subclass
over Mistral's OpenAI-compatible endpoint, registering a `Mistral` entry
in the `CvModel` registry so Mistral vision models (`pixtral-*`) become
usable as `image2text` at all.
4. **Figure description** — Mistral-OCR-extracted figures are captioned
using the tenant's configured `image2text` model (any provider),
matching MinerU/deepdoc behaviour.
5. **Wires the parser into every chunking method** (`naive`, `paper`,
`book`, `laws`, `manual`, `one`, `presentation`) and the `rag/flow` DAG
path. This also fixes a related latent gap where those chunkers
forwarded only `mineru_llm_name`, so any model-based OCR provider
selected on a non-`naive` method silently fell through.
**Notes on the API contract** (verified against the live Mistral API):
`pages` is a selector (returns absolute `index`, bills only the
requested pages); `include_blocks: true` returns per-block bounding
boxes usable for chunk highlighting and figure cropping; large files use
`POST /v1/files` → signed URL → OCR → `DELETE`.
**Testing**: new unit tests cover the response→sections contract (both
the 2-tuple `naive` path and the typed 3-tuple DAG path), the
position-tag rescale, the HTTP client incl. upload failure/cleanup
paths, `parse_pdf` page-range threading, registry registration, env
config, the suffix normalization, the factory catalog entry, `MistralCV`
registration, and figure-description injection. Verified end to end
against the live Mistral API on real PDFs (table extraction,
page-selector cost avoidance, figure captioning).
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
- [x] New Feature (non-breaking change which adds functionality)
