Fixes#7316.
## Problem
`deepdoc/vision/operators.py` defines the image-standardize
preprocessing op as `class StandardizeImag` (missing the final `e`), but
every caller — including
`deepdoc/vision/recognizer.py::Recognizer.preprocess` — looks the class
up by the canonical string `"StandardizeImage"` via:
```python
op_type = new_op_info.pop("type") # "StandardizeImage"
preprocess_ops.append(getattr(operators, op_type)(**new_op_info))
```
So `getattr(operators, "StandardizeImage")` raised `AttributeError`, and
the "StandardizeImage" preprocessing step silently never ran for any
image pipeline that used the dynamic dispatch (LayoutLMv3 and friends).
The user-visible symptom is that the standardize step is missing
entirely from the preprocessing chain, so the model gets un-normalized
images.
## Production fix
```diff
-class StandardizeImag:
+class StandardizeImage:
"""normalize image
Args:
mean (list): im - mean
std (list): im / std
is_scale (bool): whether need im / 255
norm_type (str): type in ['mean_std', 'none']
"""
```
That's the entire production change — a one-character class rename. The
misnamed `StandardizeImag` had no other references in the codebase
(verified via `git grep`), so removing it is safe; every caller uses the
canonical `"StandardizeImage"` string and will now resolve correctly.
## Tests
New `test/unit_test/deepdoc/vision/test_operators_standardize_image.py`
with six regression tests, all green locally:
```
test_standardize_image_class_resolves_by_canonical_name PASSED
test_standardize_image_callable_matches_legacy_alias_name PASSED
test_standardize_image_normalizes_input_with_mean_std_and_is_scale PASSED
test_standardize_image_skips_scaling_when_is_scale_false PASSED
test_standardize_image_norm_type_none_passes_image_through PASSED
test_standardize_image_via_module_getattr_dispatch_path PASSED
6 passed in 0.18s
```
The tests:
1. **Pin the dispatch contract** (`hasattr(operators,
"StandardizeImage")`) — this is the exact check the recognizer's
`getattr` would do, so any future regression fails the same way the
runtime would.
2. **Pin that the misspelled name is gone** — if a downstream caller
ever relied on it, this fails loudly.
3–5. **Behavioural coverage** of the three documented code paths:
`is_scale=True, norm_type="mean_std"`, `is_scale=False,
norm_type="mean_std"`, and `norm_type="none"`.
6. **End-to-end via the same `getattr(operators, "StandardizeImage")`
call** the recognizer uses, with a real numpy image, so any rename or
removal surfaces as `AttributeError` instead of silently skipping the
step.
Verified both ways:
- Without the fix → **all 6 tests fail** (Python even suggests
`'StandardizeImag' → 'StandardizeImage'`)
- With the fix → all 6 pass in 0.15s
The test file follows the project's existing pattern
(`test/unit_test/deepdoc/parser/test_html_parser.py`): load the target
module via `importlib.util.spec_from_file_location`, stub the only
project-internal import (`rag.utils.lazy_image`), and assert against the
loaded module — no full RAGFlow runtime required.
## Risk
Very low. The class is renamed; no public Python API was using the
misnamed class. The only reference path is the `"StandardizeImage"`
string in `recognizer.py:270`, which now resolves correctly.
## Out of scope
- No other ops in `operators.py` are affected; checked all the others
(DecodeImage, NormalizeImage, Permute, etc.) and they all use correct
names.
- The dynamic-dispatch lookups in `recognizer.py` for `LinearResize`,
`StandardizeImage`, `Permute`, `PadStride` all use the same dispatch
path; only the `StandardizeImage` key was broken. No other keys need
fixing.
Made with [Cursor](https://cursor.com)
---------
Co-authored-by: Taranum01 <Taranum01@users.noreply.github.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Zhichang Yu <yuzhichang@gmail.com>
**Summary**
This PR tackles a significant memory bottleneck when processing
image-heavy Word documents. Previously, our pipeline eagerly decoded
DOCX images into `PIL.Image` objects, which caused high peak memory
usage. To solve this, I've introduced a **lazy-loading approach**:
images are now stored as raw blobs and only decoded exactly when and
where they are consumed.
This successfully reduces the memory footprint while keeping the parsing
output completely identical to before.
**What's Changed**
Instead of a dry file-by-file list, here is the logical breakdown of the
updates:
* **The Core Abstraction (`lazy_image.py`)**: Introduced `LazyDocxImage`
along with helper APIs to handle lazy decoding, image-type checks, and
NumPy compatibility. It also supports `.close()` and detached PIL access
to ensure safe lifecycle management and prevent memory leaks.
* **Pipeline Integration (`naive.py`, `figure_parser.py`, etc.)**:
Updated the general DOCX picture extraction to return these new lazy
images. Downstream consumers (like the figure/VLM flow and base64
encoding paths) now decode images right at the use site using detached
PIL instances, avoiding shared-instance side effects.
* **Compatibility Hooks (`operators.py`, `book.py`, etc.)**: Added
necessary compatibility conversions so these lazy images flow smoothly
through existing merging, filtering, and presentation steps without
breaking.
**Scope & What is Intentionally Left Out**
To keep this PR focused, I have restricted these changes strictly to the
**general Word pipeline** and its downstream consumers.
The `QA` and `manual` Word parsing pipelines are explicitly **not
modified** in this PR. They can be safely migrated to this new lazy-load
model in a subsequent, standalone PR.
**Design Considerations**
I briefly considered adding image compression during processing, but
decided against it to avoid any potential quality degradation in the
derived outputs. I also held off on a massive pipeline re-architecture
to avoid overly invasive changes right now.
**Validation & Testing**
I've tested this to ensure no regressions:
* Compared identical DOCX inputs before and after this branch: chunk
counts, extracted text, table HTML, and image descriptions match
perfectly.
* **Confirmed a noticeable drop in peak memory usage when processing
image-dense documents.** For a 30MB Word document containing 243 1080p
screenshots, memory consumption is reduced by approximately 1.5GB.
**Breaking Changes**
None.
Addresses a potential RCE vulnerability in NormalizeImage by using
ast.literal_eval for safer string parsing.
---------
Co-authored-by: Kevin Hu <kevinhu.sh@gmail.com>
### What problem does this PR solve?
This patch drop useless fastext which is seems useless in the code base
and its very kind of hard install
should close#4498
### Type of change
- [x] Refactoring
Signed-off-by: yihong0618 <zouzou0208@gmail.com>
Use `np.float32()` instead.
### What problem does this PR solve?
Using `eval()` can lead to code injections.
I think `eval()` is only used to parse a floating point number here.
This change preserves the correct behavior if the string `"None"` is
supplied. But if that behavior isn't intended then this part could be
just deleted instead, since `np.float32()` is parsing strings anyway:
```Python
if isinstance(scale, str):
scale = eval(scale)
```
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
### What problem does this PR solve?
Use consistent log file names, introduced initLogger
### Type of change
- [ ] Bug Fix (non-breaking change which fixes an issue)
- [ ] New Feature (non-breaking change which adds functionality)
- [ ] Documentation Update
- [x] Refactoring
- [ ] Performance Improvement
- [ ] Other (please describe):
### What problem does this PR solve?
substract -> subtract
### Type of change
- [ ] Bug Fix (non-breaking change which fixes an issue)
- [ ] New Feature (non-breaking change which adds functionality)
- [ ] Documentation Update
- [x] Refactoring
- [ ] Performance Improvement
- [ ] Other (please describe):
### What problem does this PR solve?
Related source file is in Windows/DOS format, they are format to Unix
format.
### Type of change
- [x] Refactoring
Signed-off-by: Jin Hai <haijin.chn@gmail.com>
