Refa: implement unified lazy image loading for Docx parsers (qa/manual) (#13329)

## Summary
This PR is the direct successor to the previous `docx` lazy-loading
implementation. It addresses the technical debt intentionally left out
in the last PR by fully migrating the `qa` and `manual` parsing
strategies to the new lazy-loading model.

Additionally, this PR comprehensively refactors the underlying `docx`
parsing pipeline to eliminate significant code redundancy and introduces
robust fallback mechanisms to handle completely corrupted image streams
safely.


## What's Changed

* **Centralized Abstraction (`docx_parser.py`)**: Moved the
`get_picture` extraction logic up to the `RAGFlowDocxParser` base class.
Previously, `naive`, `qa`, and `manual` parsers maintained separate,
redundant copies of this method. All downstream strategies now natively
gather raw blobs and return `LazyDocxImage` objects automatically.
* **Robust Corrupted Image Fallback (`docx_parser.py`)**: Handled edge
cases where `python-docx` encounters critically malformed magic headers.
Implemented an explicit `try-except` structure that safely intercepts
`UnrecognizedImageError` (and similar exceptions) and seamlessly falls
back to retrieving the raw binary via `getattr(related_part, "blob",
None)`, preventing parser crashes on damaged documents.

* **Legacy Code & Redundancy Purge**:
* Removed the duplicate `get_picture` methods from `naive.py`, `qa.py`,
and `manual.py`.
* Removed the standalone, immediate-decoding `concat_img` method in
`manual.py`. It has been completely replaced by the globally unified,
lazy-loading-compatible `rag.nlp.concat_img`.
* Cleaned up unused legacy imports (e.g., `PIL.Image`, docx exception
packages) across all updated strategy files.

## Scope
To keep this PR focused, I have restricted these changes strictly to the
unification of `docx` extraction logic and the lazy-load migration of
`qa` and `manual`.

## Validation & Testing
I've tested this to ensure no regressions and validated the fallback
logic:

* **Output Consistency**: Compared identical `.docx` inputs using `qa`
and `manual` strategies before and after this branch: chunk counts,
extracted text, table HTML, and attached images match perfectly.
* **Memory Footprint Drop**: Confirmed a noticeable drop in peak memory
usage when processing image-dense documents through the `qa` and
`manual` pipelines, bringing them up to parity with the `naive`
strategy's performance gains.

## Breaking Changes
* None.
This commit is contained in:
eviaaaaa
2026-03-11 10:00:07 +08:00
committed by GitHub
parent d36e3c97d1
commit d0ca388bec
6 changed files with 74 additions and 93 deletions

View File

@@ -20,9 +20,54 @@ import pandas as pd
from collections import Counter
from rag.nlp import rag_tokenizer
from io import BytesIO
import logging
from docx.image.exceptions import (
InvalidImageStreamError,
UnexpectedEndOfFileError,
UnrecognizedImageError,
)
from rag.utils.lazy_image import LazyDocxImage
class RAGFlowDocxParser:
def get_picture(self, document, paragraph):
imgs = paragraph._element.xpath(".//pic:pic")
if not imgs:
return None
image_blobs = []
for img in imgs:
embed = img.xpath(".//a:blip/@r:embed")
if not embed:
continue
embed = embed[0]
image_blob = None
try:
related_part = document.part.related_parts[embed]
except Exception as e:
logging.warning(f"Skipping image due to unexpected error getting related_part: {e}")
continue
try:
image = related_part.image
if image is not None:
image_blob = image.blob
except (
UnrecognizedImageError,
UnexpectedEndOfFileError,
InvalidImageStreamError,
UnicodeDecodeError,
) as e:
logging.info(f"Damaged image encountered, attempting blob fallback: {e}")
except Exception as e:
logging.warning(f"Unexpected error getting image, attempting blob fallback: {e}")
if image_blob is None:
image_blob = getattr(related_part, "blob", None)
if image_blob:
image_blobs.append(image_blob)
if not image_blobs:
return None
return LazyDocxImage(image_blobs)
def __extract_table_content(self, tb):
df = []