## Summary
- Parse chat, embedding, and rerank usage from provider responses
- Record usage with the correct model type even when no usage sink is
provided
- Cover SiliconFlow, Aliyun, Huawei Cloud, Qiniu, and VolcEngine
response formats
## Summary
Closes two Chunker migration gaps documented in
`docs/migration_python_go_diff.md`
(diffs **2.6** and **1.7**), improving parity with the Python ingestion
pipeline.
This is the code portion of commit `261e1fd0b` on branch
`fix/batch-3-4`; the
migration document itself is tracked separately (untracked in this
commit).
### Diff 2.6 — `overlapped_percent` missing Python normalization
Python's `normalize_overlapped_percent` (`common/float_utils.py:50-58`)
accepts a
`[0,1)` fraction (the flow-canvas UI validates `[0,1)`), multiplies it
by 100,
`int()`-truncates, and clamps to `[0,90]`. Go previously only accepted a
raw
`[0,90]` percentage and **rejected** out-of-range input, so a Python
config
passing `0.1` (meaning 10%) silently produced ~0% overlap.
Added `normalizeOverlappedPercent`
(`internal/ingestion/component/chunker/common.go`) mirroring the Python
helper:
- parses numbers and numeric strings (mirrors Python `float()`; bad /
`NaN` / `Inf` → `0`),
- `0 < v < 1 → v *= 100`,
- `int()` truncation,
- clamps to `[0,90]`.
Wired into `tokenChunkerParam.Update` (`token.go`); the merge math
(`token.go:705`, `(100-x)/100`) already matched Python.
`TokenChunkerParam.Validate`
(`schema/chunker.go`) now only guards direct struct construction.
Tests: `TestNormalizeOverlappedPercent`, extended
`TestTokenChunker_NewAcceptsPythonOverlappedRange` (adds fraction/clamp
inputs),
new `TestTokenChunker_NormalizesOverlappedPercent`. The existing
reject-test cases
for `<0` / `>90` were removed because they are now normalized/clamped
(Python parity).
### Diff 1.7 — missing `BULLET_PATTERN` title-level fallback
`resolveTitleLevels` (`title.go`) now applies a 4th-level fallback: when
outline +
regex + layout all yield body level, `bulletsCategory` selects the
best-matching
bullet-pattern group (Chinese legal / numbering / Chinese numbering /
English legal
— mirroring `rag/nlp/__init__.py:258-320`) and assigns structural
levels. Guarded by
`allBodyLevel` so it never overrides an existing outline/regex level.
Tests: `TestResolveTitleLevels_BulletFallback` (4 subtests).
### Incidental test adjustments included in the commit
- `token_batch1_test.go`: overlap input changed `0.3` → `30.0` to
reflect the
post-normalization 30% semantics.
- `real_consumer_test.go`: updated `LoadFromIngestionTask(task)` →
`LoadFromIngestionTask(ctx, task)` for the new context-first signature.
## Verification
`bash build.sh --test ./internal/ingestion/component/...` — chunker +
schema suites
pass, no regression (CGO build).
## Migration doc reference
`docs/migration_python_go_diff.md` §Chunker 1.7 and 2.6 are marked
**Fixed** for
these changes.
## Summary
Improve the UX of the **"Delimiter for text"** field on the dataset
configuration page. The field is a single string with a backtick-based
mini-syntax, but both the tooltip and the surrounding UI failed to
surface what delimiters the backend would actually derive from a given
value — leaving users to discover by trial-and-error that the same
string produces different splits depending on file type (see #7436,
#4704, #9680).
### Summary
The FunASR provider added in #17171 defaults to a local
`http://localhost:8000/v1` server, but it still inherited the global
API-key requirement and unconditionally built an Authorization header.
This prevented the default unauthenticated self-hosted deployment from
working. The transcription path also dereferenced a missing model name
while building its multipart request.
This change:
- allows an empty API key for FunASR, matching other local providers
- omits the Authorization header when no key is configured while
preserving trimmed Bearer authentication when one is provided
- validates and trims the ASR model name before building the multipart
request, returning an error instead of panicking
- adds HTTP-level regression coverage for unauthenticated
transcription/model listing and optional authentication
Fix broken Table of Contents anchor link (📌 vs 🔥 emoji mismatch),
navigation bar link text ("Document" → "Documentation"), Title Case
alignment between ToC entries and section headings, and inconsistent
file format capitalization.
### Summary
This PR adds **aimlapi.com** as a model provider, so a RAGFlow user can
enter one API key in the model settings and use AIMLAPI's models across
the app. AIMLAPI ([aimlapi.com](https://aimlapi.com)) is an
OpenAI-compatible aggregator that serves 700+ models (LLM, embedding,
vision, TTS, ASR) from many providers behind a single API.
The change mirrors the repo's existing "add provider" pattern (e.g.
FuturMix / OpenRouter): provider logic lives in the same files those
providers use, and shared / UI files get only registration entries.
**Backend**
- `conf/llm_factories.json` — the `aimlapi.com` factory entry.
- `rag/llm/__init__.py`, `rag/llm/{chat,embedding,cv}_model.py` —
LiteLLM adapters (chat, embedding, image2text) with a production base
URL, overridable via `AIMLAPI_API_URL`.
- `rag/llm/model_meta.py` — an `AIMLAPI` model-meta so the provider
lists its full `/v1/models` catalog dynamically (classified by the
endpoint `type`), the same way OpenRouter does.
- `api/apps/restful_apis/aimlapi_api.py` — an optional "Get API key"
flow using AIMLAPI's agent-authorization (OAuth 2.0 Device Authorization
Grant, RFC 8628). The device code is kept server-side (Redis); only the
issued key reaches the browser.
**Frontend (`web/`)**
- Provider registration (constant, icon allowlist, brand logo), the
model picker (`LIST_MODEL_PROVIDERS` + a `buildLocalConfig` entry), and
the "Get API key" button in the provider dialog. Locales added to `en`
and `zh`.
**Configuration** — production defaults are compiled in; endpoints and
the partner id are overridable through `AIMLAPI_*` environment
variables, so the same build works across environments.
**Testing** — the `web` build passes; chat, embedding and dynamic model
listing were smoke-tested against the live API.
## Description
This PR fixes a typo in comment text and a missing verb in error output
in `show_env.sh`.
## Details
1. Fixed typo in comment (`lsd_release` $\to$ `lsb_release`).
2. Added missing verb in fallback output string (`It NOT a Git repo`
$\to$ `It is NOT a Git repo`).
---------
Co-authored-by: ferkans-amir <amir.rezaei@tu-berlin.de>
### What problem does this PR solve?
Adds first-class support for **Mistral OCR** (`POST /v1/ocr`) as a
document parser, and fixes the long-standing bug where selecting
`mistral-ocr-latest` fails with `Can't find model for
<tenant>/image2text/mistral-ocr-latest`.
`mistral-ocr-latest` is Mistral's dedicated document-OCR endpoint, not a
vision-chat (`image2text`) model, but the catalog tagged it `image2text`
— so it resolved to the `CvModel` registry, which has no `Mistral`
entry, and there was no `OcrModel` entry either. This PR registers it
correctly and wires it end to end.
Closes#17056Closes#5782Closes#7075
**What it does**
1. **`MistralParser` + `MistralOcrModel`**
(`deepdoc/parser/mistral_parser.py`, `rag/llm/ocr_model.py`) — a proper
`OcrModel` factory `Mistral OCR`, mirroring the SoMark cloud-OCR
template. Tables stay inline as HTML; the page range maps to Mistral's
native `pages` selector (absolute page indices, billed per selected
page, so multi-task documents do not re-OCR the whole file); documents
over the inline limit go through the `/v1/files` signed-URL flow with
cleanup.
2. **Removes the `image2text` mis-tag** for `mistral-ocr-latest` from
the `Mistral` factory in `conf/llm_factories.json` (it now lives only in
the `Mistral OCR` factory, typed `ocr`). This is what closes the `Can't
find model` path.
3. **`MistralCV`** (`rag/llm/cv_model.py`) — a thin `GptV4` subclass
over Mistral's OpenAI-compatible endpoint, registering a `Mistral` entry
in the `CvModel` registry so Mistral vision models (`pixtral-*`) become
usable as `image2text` at all.
4. **Figure description** — Mistral-OCR-extracted figures are captioned
using the tenant's configured `image2text` model (any provider),
matching MinerU/deepdoc behaviour.
5. **Wires the parser into every chunking method** (`naive`, `paper`,
`book`, `laws`, `manual`, `one`, `presentation`) and the `rag/flow` DAG
path. This also fixes a related latent gap where those chunkers
forwarded only `mineru_llm_name`, so any model-based OCR provider
selected on a non-`naive` method silently fell through.
**Notes on the API contract** (verified against the live Mistral API):
`pages` is a selector (returns absolute `index`, bills only the
requested pages); `include_blocks: true` returns per-block bounding
boxes usable for chunk highlighting and figure cropping; large files use
`POST /v1/files` → signed URL → OCR → `DELETE`.
**Testing**: new unit tests cover the response→sections contract (both
the 2-tuple `naive` path and the typed 3-tuple DAG path), the
position-tag rescale, the HTTP client incl. upload failure/cleanup
paths, `parse_pdf` page-range threading, registry registration, env
config, the suffix normalization, the factory catalog entry, `MistralCV`
registration, and figure-description injection. Verified end to end
against the live Mistral API on real PDFs (table extraction,
page-selector cost avoidance, figure captioning).
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
- [x] New Feature (non-breaking change which adds functionality)
## Summary
Aligns the Go ingestion pipeline with the Python implementation, closing
several behavioral gaps found during the Python→Go migration (tracked in
`docs/migration_python_go_diff.md`). Covers parser/media dispatch
alignment, the PDF coordinate-chain (preview images, outline→title,
chunk coordinate finalization), and the Chunker Token/QA batches below.
Commits are grouped as follows.
### 1. Fix parser params (c524f450e)
Fixes parser/media wiring and several dispatch gaps:
- **docx/pdf vision dispatch**: correct parameter handling and VLM
invocation.
- **markdown vision (diff 2.5)**: also enhance items whose
`doc_type_kwd` is `table`, not only `image` (parser/utils.py:181).
- **media audio (diff 2.11)**: when `output_format` is `json`, carry the
ASR transcription as a JSON item instead of only the `Text` field (the
Invoke switch had no `json` branch and dropped it).
- **email (diff 2.2)**: default `output_format` is `json`
(parser.py:212), not `text`.
- **tokenizer**: handle empty/whitespace-only names; trim before
embedding.
- **extractor**: tag-matching parameter wiring.
- **split**: keyword-split regex now covers CJK/English separators.
- **parser.go**: parser-param plumbing.
### 2. fix parser gap (373537da1)
Image dispatch now mirrors `rag/app/picture.py:chunk()`:
- Always OCR the image (PaddleOCR or local ONNX).
- When OCR text is short, also call VLM (`describe`) and combine `OCR +
VLM` text.
- Emits a **structured JSON item** carrying the image data-URI and
`doc_type_kwd:"image"`, instead of a bare `Text` string. This fixes the
payload being rejected downstream by OneChunker/TokenChunker (JSON=nil).
### 3. PDF coordinate-chain fixes (55367a820, 727f8167c)
Closes three items from the migration tracker in the
chunker/tokenizer/task layer:
- **(Chunker-1.3) `restore_pdf_text_previews`** — `needsCrop` now also
returns true for `text` chunks that carry PDF positions
(`pdfcrop_cgo.go`), so text blocks get a rendered preview image uploaded
to storage via `imageUploadDecorator`/`ChunkImageUploader`, matching
Python `restore_pdf_text_previews` + `image2id`.
- **(Chunker-1.5) PDF outline → title levels** — `title.go` adds
`outlineSimilarity` (rune-bigram Jaccard, mirroring
`common.py:_outline_similarity`), `resolveOutlineLevels` (matches text
lines to outline entries at similarity > 0.8, with a sparse guard
`len(outline)/len(records) <= 0.03`), and `outlineFromInputs` (reads
`file.outline`). Wired into `newLevelContext` in both `group.go` and
`hierarchy.go`; falls back to the title-shape heuristic when no outline
is present.
- **(Tokenizer-(T)1) `finalize_pdf_chunk`** — the coordinate →
`position_int`/`page_num_int`/`top_int` conversion is owned by the task
layer (`processChunkPositions`→`AddPositions`), which runs *after* the
tokenizer and consumes the tokenizer-owned fields. The tokenizer only
preserves the raw `positions`/`_pdf_positions` (no duplicate
conversion), pinned by `TestChunkDocsToMaps_PreservesPDFPositions`.
### 4. Integration test made environment-free
(`internal/ingestion/task/pipeline_real_integration_test.go`)
- Removed the `//go:build integration` tag so the contract tests run
under the default `build.sh --test` (which does not pass `-tags
integration`).
- External dependencies replaced with in-memory substitutes so no
MySQL/MinIO/ES is required:
- MySQL → on-disk sqlite (`glebarez/sqlite`) with the needed tables
auto-migrated.
- MinIO → `storage.NewMemoryStorage()`.
- Elasticsearch → chunks captured via `WithInsertFunc` instead of
`engine.InsertChunks`/`Search`.
- `requireTokenizerPool` still skips gracefully when the native
tokenizer pool is unavailable; `WithLogCreateFunc(noop)` avoids
depending on the operation-log table.
- Added `taskChunkFieldEqualsStr` to tolerate `kb_id` being a
`[]string`/`[]any` in the raw chunk payload (the search engine flattens
it to a string on read).
### 5. TokenChunker alignment — Batch 1
(`internal/ingestion/component/chunker/token.go`)
Closes four Chunker items from the migration tracker:
- **(Chunker-2.1) sentence delimiter** — the boundary regex now also
breaks on ASCII `!`/`?`. Extracted to a package-level `var
sentenceDelimiter` and used in `mergeByTokenSize`, matching Python's
full delimiter set.
- **(Chunker-2.2) overlap tag leakage** — when a new chunk starts, its
overlap prefix is taken from the previous chunk *after* `removeTag`, in
both the text path (`mergeByTokenSize`) and the JSON path
(`mergeByTokenSizeFromJSON`). Parser tags (`@@…##`) no longer leak into
the overlap region (mirrors `nlp/__init__.py:1181`).
- **(Chunker-2.11) empty-text merge** — merging a non-empty chunk into
an empty previous chunk now assigns the text directly instead of being
skipped (`mergeByTokenSizeFromJSON`), mirroring
`token_chunker.py:236-239`.
- **(Chunker-2.4) overlap token counting** —
`takeFromEnd`/`takeFromStart` now count tokens exactly via `tokenizeStr`
instead of the 4-bytes/token heuristic, fixing over-counting for CJK
text.
### 6. QA Chunker alignment — Batch 2
(`internal/ingestion/component/chunker/qa.go` + `schema`)
Closes three Chunker items from the migration tracker:
- **(Chunker-2.13) default language** — an empty `lang` now defaults to
Chinese prefixes (`问题：`/`回答：`) instead of English, matching `qa.py:299`.
- **(Chunker-2.12) `rmQAPrefix` regex** — the separator is changed to
`[\t:： ]+` (one-or-more), matching `qa.py:241`, so multiple separators
(e.g. `Q:: answer`) are fully stripped.
- **(Chunker-1.8 QA) missing chunk fields** — QA chunks now preserve:
- `top_int` — the source row/record index, threaded through the
tab/csv/markdown extractors (mirrors `qa.py` `beAdoc(..., row_num=i)`);
- `image` + `doc_type_kwd:"image"`;
- `_pdf_positions` / `positions` carried from the upstream JSON item.
`schema.ChunkDoc` gains a `TopInt []int` field (serialized as `top_int`,
registered in `UnmarshalJSON`). Note: the Tag/Table/Presentation/One
chunker field gaps under 1.8 remain pending.
## Test plan
- Added/updated unit tests: `pdfcrop_cgo_test.go` (`TestNeedsCrop`,
`TestRestorePDFTextPreview`), `title_test.go`
(`TestResolveOutlineLevels`, `TestResolveOutlineLevels_SparseGuard`,
`TestNewLevelContext_OutlineBranch`, `TestOutlineFromInputs`),
`tokenizer_unit_test.go` (`TestChunkDocsToMaps_PreservesPDFPositions`),
`token_pdfpos_test.go`.
- **Batch 1** — `token_batch1_test.go`:
`TestSentenceDelimiterMatchesBangAndQuestion`,
`TestMergeByTokenSizeFromJSON_OverlapStripsTags`,
`TestMergeByTokenSizeFromJSON_EmptyPrevKeepsChunk`,
`TestTakeFromEndRespectsTokenCount`,
`TestTakeFromStartRespectsTokenCount`.
- **Batch 2** — `qa_batch2_test.go`:
`TestQAChunker_DefaultLangIsChinese`,
`TestRmQAPrefixStripsMultipleSeparators`, `TestQAChunker_SetsTopInt`,
`TestQAChunker_CarriesImageAndPositions`. Existing `qa_test.go`
expectations were updated to the corrected language default / separator
behavior.
- `pipeline_real_integration_test.go`
(`TestPipelineExecutor_Run_RealCanvasDSL_UsesGeneralPipeline`,
`TestPipelineExecutor_Run_RealPDF_ProducesIndexedChunks`,
`TestRunPipeline_RealPipelineOutput_ProducesIndexFields`) now runs
without any external service.
- `bash build.sh --test ./internal/ingestion/...` passes.
- No files deleted.
## Summary
Remediates four CVEs in `lxml` and `lxml_html_clean` by upgrading
transitive dependency constraints and bumping `crawl4ai` to unblock the
resolution.
| CVE | Severity | Package | Installed | Fixed in |
|---|---|---|---|---|
| CVE-2026-41066 | HIGH | lxml | 5.4.0 | 6.1.0 |
| CVE-2026-49825 | HIGH | lxml_html_clean | 0.4.3 | 0.4.5 |
| CVE-2026-28348 | MEDIUM | lxml_html_clean | 0.4.3 | 0.4.5 |
| CVE-2026-28350 | MEDIUM | lxml_html_clean | 0.4.3 | 0.4.5 |
- Bumped `crawl4ai` from `>=0.9.0,<0.9.1` to `>=0.9.2,<0.9.3`, version
0.9.2 relaxes lxml constraint to `>=5.3,<7`. The strict `<0.9.3` upper
bound on crawl4ai is maintained per project convention
- Added `inscriptis>=2.7.3` to `constraint-dependencies`, version 2.7.3
relaxes lxml constraint to `>=5.4.0,<6.2.0`
- Added `lxml>=6.1.0` to `constraint-dependencies`
- Added `lxml_html_clean>=0.4.5` to `constraint-dependencies`
`exclude-dependencies` entry for `unclecode-litellm` (still declared by
crawl4ai 0.9.2) continues to prevent the namespace collision that
previously crashed RAGFlow.
## Verification
- `lxml` is used widely across the codebase for HTML/XML parsing. The
5.x → 6.x upgrade maintains API compatibility for standard usage
(`lxml.html`, `lxml.etree`).
- `lxml_html_clean` API is unchanged between 0.4.3 and 0.4.5.
- `crawl4ai` 0.9.2 is a patch bump from 0.9.0 with no API changes
affecting RAGFlow's usage.
- `inscriptis` 2.7.3 is a patch bump from 2.7.0.
## Testing
- Full unit test suite passes
- `uv sync` resolves cleanly
- Verified `crawl4ai` imports without litellm namespace collision
## Summary
- Root cause: Pipeline.Run keys the eino checkpoint by taskID only.
Resuming a failed run after the user edits the pipeline DSL recompiles a
graph with different topology, but the old checkpoint (bound to the
previous graph's node ids / wiring) is restored, causing eino to error.
- Fix: fingerprint the DSL file (full canvas DSL) and the runtime
override_params, persisted next to the eino checkpoint. On resume, if
either fingerprint differs, discard the stale checkpoint + interrupt
marker and re-run from scratch. The warning log distinguishes a DSL-file
edit from a runtime-override edit.
## Test plan
- TestPipelineRunResumableDSLChanged: editing the DSL between runs
discards the checkpoint and re-runs from scratch.
- TestPipelineRunResumableOverrideChanged: editing only the runtime
override does the same.
- TestClassifyDSLChange: unit-tests the mismatch-reason classifier.
- bash build.sh --test ./internal/ingestion/pipeline/... passes.
## Notes
- Component-code changes (e.g. a component's output contract) are not
covered by the DSL fingerprint; that is a separate, smaller-blast-radius
gap noted in code comments.
---------
Signed-off-by: xugangqiang <xugangqiang@hotmail.com>
Co-authored-by: CodeBuddy <noreply@tencent.com>
## Description
This PR fixes Title Case capitalization matching for Table of Contents
items in `README.md`.
## Details
Updated TOC entries (`Build a Docker image` -> `Build a Docker Image`,
`Launch service from source` -> `Launch Service from Source`) to match
section headers.
Co-authored-by: ferkans-amir <amir.rezaei@tu-berlin.de>
