mirror of
https://github.com/infiniflow/ragflow.git
synced 2026-07-25 18:03:29 +08:00
feat: add Mistral OCR (/v1/ocr) as a document parser; fix "Can't find model" mis-tag (#5782, #7075) (#17057)
### What problem does this PR solve? Adds first-class support for **Mistral OCR** (`POST /v1/ocr`) as a document parser, and fixes the long-standing bug where selecting `mistral-ocr-latest` fails with `Can't find model for <tenant>/image2text/mistral-ocr-latest`. `mistral-ocr-latest` is Mistral's dedicated document-OCR endpoint, not a vision-chat (`image2text`) model, but the catalog tagged it `image2text` — so it resolved to the `CvModel` registry, which has no `Mistral` entry, and there was no `OcrModel` entry either. This PR registers it correctly and wires it end to end. Closes #17056 Closes #5782 Closes #7075 **What it does** 1. **`MistralParser` + `MistralOcrModel`** (`deepdoc/parser/mistral_parser.py`, `rag/llm/ocr_model.py`) — a proper `OcrModel` factory `Mistral OCR`, mirroring the SoMark cloud-OCR template. Tables stay inline as HTML; the page range maps to Mistral's native `pages` selector (absolute page indices, billed per selected page, so multi-task documents do not re-OCR the whole file); documents over the inline limit go through the `/v1/files` signed-URL flow with cleanup. 2. **Removes the `image2text` mis-tag** for `mistral-ocr-latest` from the `Mistral` factory in `conf/llm_factories.json` (it now lives only in the `Mistral OCR` factory, typed `ocr`). This is what closes the `Can't find model` path. 3. **`MistralCV`** (`rag/llm/cv_model.py`) — a thin `GptV4` subclass over Mistral's OpenAI-compatible endpoint, registering a `Mistral` entry in the `CvModel` registry so Mistral vision models (`pixtral-*`) become usable as `image2text` at all. 4. **Figure description** — Mistral-OCR-extracted figures are captioned using the tenant's configured `image2text` model (any provider), matching MinerU/deepdoc behaviour. 5. **Wires the parser into every chunking method** (`naive`, `paper`, `book`, `laws`, `manual`, `one`, `presentation`) and the `rag/flow` DAG path. This also fixes a related latent gap where those chunkers forwarded only `mineru_llm_name`, so any model-based OCR provider selected on a non-`naive` method silently fell through. **Notes on the API contract** (verified against the live Mistral API): `pages` is a selector (returns absolute `index`, bills only the requested pages); `include_blocks: true` returns per-block bounding boxes usable for chunk highlighting and figure cropping; large files use `POST /v1/files` → signed URL → OCR → `DELETE`. **Testing**: new unit tests cover the response→sections contract (both the 2-tuple `naive` path and the typed 3-tuple DAG path), the position-tag rescale, the HTTP client incl. upload failure/cleanup paths, `parse_pdf` page-range threading, registry registration, env config, the suffix normalization, the factory catalog entry, `MistralCV` registration, and figure-description injection. Verified end to end against the live Mistral API on real PDFs (table extraction, page-selector cost avoidance, figure captioning). ### Type of change - [x] Bug Fix (non-breaking change which fixes an issue) - [x] New Feature (non-breaking change which adds functionality)
This commit is contained in:
@@ -257,7 +257,7 @@ class ParserParam(ProcessParamBase):
|
||||
pdf_parse_method = pdf_config.get("parse_method", "")
|
||||
self.check_empty(pdf_parse_method, "Parse method abnormal.")
|
||||
|
||||
if pdf_parse_method.lower() not in ["deepdoc", "plain_text", "mineru", "docling", "opendataloader", "tcadp parser", "paddleocr", "somark"]:
|
||||
if pdf_parse_method.lower() not in ["deepdoc", "plain_text", "mineru", "docling", "opendataloader", "tcadp parser", "paddleocr", "somark", "mistral ocr"]:
|
||||
self.check_empty(pdf_config.get("lang", ""), "PDF VLM language")
|
||||
|
||||
pdf_output_format = pdf_config.get("output_format", "")
|
||||
@@ -369,6 +369,9 @@ class Parser(ProcessBase):
|
||||
# downstream requires all three segments.
|
||||
parser_model_name = raw_parse_method
|
||||
parse_method = "SoMark"
|
||||
elif lowered.endswith("@mistral ocr"):
|
||||
parser_model_name = raw_parse_method
|
||||
parse_method = "Mistral OCR"
|
||||
|
||||
# DeepDOC returns structured page boxes directly.
|
||||
if parse_method.lower() == "deepdoc":
|
||||
@@ -565,6 +568,52 @@ class Parser(ProcessBase):
|
||||
box["image"] = image
|
||||
bboxes.append(box)
|
||||
|
||||
elif parse_method.lower() == "mistral ocr":
|
||||
|
||||
def resolve_mistral_ocr_llm_name():
|
||||
configured = parser_model_name or conf.get("mistral_ocr_llm_name")
|
||||
if configured:
|
||||
return configured
|
||||
tenant_id = self._canvas._tenant_id
|
||||
if not tenant_id:
|
||||
return None
|
||||
from api.db.joint_services.tenant_model_service import ensure_mistral_ocr_from_env
|
||||
|
||||
return ensure_mistral_ocr_from_env(tenant_id)
|
||||
|
||||
parser_model_name = resolve_mistral_ocr_llm_name()
|
||||
if not parser_model_name:
|
||||
raise RuntimeError("Mistral OCR model not configured. Please add Mistral OCR in Model Providers or set MISTRAL_OCR_* env.")
|
||||
|
||||
tenant_id = self._canvas._tenant_id
|
||||
ocr_model_config = resolve_model_config(tenant_id, LLMType.OCR, parser_model_name)
|
||||
ocr_model = LLMBundle(tenant_id, ocr_model_config)
|
||||
pdf_parser = ocr_model.mdl
|
||||
|
||||
lines, _ = pdf_parser.parse_pdf(
|
||||
filepath=name,
|
||||
binary=blob,
|
||||
callback=self.callback,
|
||||
parse_method="pipeline",
|
||||
)
|
||||
bboxes = []
|
||||
for item in lines or []:
|
||||
if not isinstance(item, tuple) or len(item) < 3:
|
||||
continue
|
||||
text, layout_type, poss = item[0], item[1], item[2]
|
||||
box = {
|
||||
"text": text,
|
||||
"layout_type": layout_type or "text",
|
||||
}
|
||||
if isinstance(poss, str) and poss:
|
||||
positions = [[pos[0][-1] + 1, *pos[1:]] for pos in pdf_parser.extract_positions(poss)]
|
||||
if positions:
|
||||
box["positions"] = positions
|
||||
image = pdf_parser.crop(poss, 1)
|
||||
if image is not None:
|
||||
box["image"] = image
|
||||
bboxes.append(box)
|
||||
|
||||
elif parse_method.lower() == "tcadp parser":
|
||||
# ADP is a document parsing tool using Tencent Cloud API
|
||||
table_result_type = conf.get("table_result_type", "1")
|
||||
|
||||
Reference in New Issue
Block a user