mirror of
https://github.com/infiniflow/ragflow.git
synced 2026-07-21 23:21:04 +08:00
b7a3a2760f
### What problem does this PR solve?
The Elasticsearch and Infinity metadata push-down translators are meant
to be
interchangeable pre-filters that both mirror the in-memory `meta_filter`
fallback — the shared test section is even titled
*"is_pushdown_supported
pre-check (same logic for both backends)"*. But the two
`is_pushdown_supported`
implementations diverge:
- `common/metadata_es_filter.py` defines
`MULTIVALUE_UNSAFE_NEGATIVE_OPS = frozenset({"≠", "not in"})` and
refuses
push-down for those operators.
- `common/metadata_infinity_filter.py` has **no such guard** and pushes
them
down.
**Why the guard exists:** `meta_fields.<key>` can hold a JSON array, and
the
in-memory `meta_filter` matches a document when **any** of its values
satisfies
the predicate (per-value-bucket semantics). A document whose `tag` is
`[a, b]`
therefore still matches `tag ≠ a` — bucket `b` satisfies it. The
Infinity
push-down emits `NOT JSON_CONTAINS(meta_fields, '$.tag', '"a"')`, which
means
*"the array contains no `a` at all"*, so it **silently drops** that
document.
Same divergence for `not in`. The result: `tag ≠ a` / `tag not in (...)`
under-counts results for any document that has the excluded value
alongside
others, but only on the Infinity backend.
This is on a live production path:
`DocMetadataService._filter_doc_ids_by_metadata_infinity`
(`api/db/services/doc_metadata_service.py:948`) calls this exact
`is_pushdown_supported` as the sole gate before building the Infinity
SQL,
mirroring the ES branch at line 880 which uses the guarded ES version.
Reproduction (both real modules, no services needed):
```python
>>> from common import metadata_es_filter as es, metadata_infinity_filter as inf
>>> f = [{"op": "≠", "key": "tag", "value": "a"}]
>>> es.is_pushdown_supported(f), inf.is_pushdown_supported(f)
(False, True) # ES falls back to in-memory (correct); Infinity pushes down (wrong)
```
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
### Fix
Add the same `MULTIVALUE_UNSAFE_NEGATIVE_OPS` set to
`metadata_infinity_filter` and reject those operators in
`is_pushdown_supported`, so a single such filter forces the whole
request to
the in-memory path — the only place the per-bucket semantics are
reproduced.
`not contains` is intentionally still allowed, matching the ES backend
(`all(not contains)` == `not any(contains)`, which the push-down
expresses
correctly on multi-valued fields). The `≠` / `not in` translators
themselves
are unchanged — they remain correct for the in-memory-fallback path;
only the
push-down eligibility gate is fixed.
### Testing
- Confirmed `is_pushdown_supported([{op}])` now returns `False` for `≠`
and
`not in` on **both** backends (previously ES=False, Infinity=True).
- Added `test_pushdown_check_rejects_multivalue_unsafe_negative_ops`,
which
asserts both backends reject these ops. Confirmed red→green: it fails
against
the pre-fix Infinity module, passes after.
- `ruff check` / `ruff format --check` clean.
### Note on overlap
Open PR #16833 also edits
`test/unit_test/common/test_metadata_filter.py`, but
only **appends** at the end of the file (line 640+) and changes
`metadata_es_filter.py` / `metadata_utils.py`, not the Infinity module —
no
overlap with this change.
### Disclosure
AI-assisted (Claude Code): the divergence was surfaced by an AI-assisted
review
pass, but I independently reproduced it against the real modules,
confirmed the
production call path, and verified the fix and test before submitting.
Signed-off-by: Andrew Chen <48723787+chuenchen309@users.noreply.github.com>
Signed-off-by: chuenchen309 <48723787+chuenchen309@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
(1). Deploy RAGFlow services and images
https://ragflow.io/docs/build_docker_image
(2). Configure the required environment for testing
Install Python dependencies (including test dependencies):
uv sync --python 3.13 --only-group test --no-default-groups --frozen
Activate the environment:
source .venv/bin/activate
Install SDK:
uv pip install sdk/python
Modify the .env file: Add the following code:
COMPOSE_PROFILES=${COMPOSE_PROFILES},tei-cpu
TEI_MODEL=BAAI/bge-small-en-v1.5
RAGFLOW_IMAGE=infiniflow/ragflow:v0.26.4 #Replace with the image you are using
Start the container（wait two minutes）:
docker compose -f docker/docker-compose.yml up -d
(3). Test Elasticsearch
a) Run sdk tests against Elasticsearch:
export HTTP_API_TEST_LEVEL=p2
export HOST_ADDRESS=http://127.0.0.1:9380 # Ensure that this port is the API port mapped to your localhost
pytest -s --tb=short --level=${HTTP_API_TEST_LEVEL} test/testcases/test_sdk_api
b) Run http api tests against Elasticsearch:
pytest -s --tb=short --level=${HTTP_API_TEST_LEVEL} test/testcases/test_http_api
(4). Test Infinity
Modify the .env file:
DOC_ENGINE=${DOC_ENGINE:-infinity}
Start the container:
docker compose -f docker/docker-compose.yml down -v
docker compose -f docker/docker-compose.yml up -d
a) Run sdk tests against Infinity:
DOC_ENGINE=infinity pytest -s --tb=short --level=${HTTP_API_TEST_LEVEL} test/testcases/test_sdk_api
b) Run http api tests against Infinity:
DOC_ENGINE=infinity pytest -s --tb=short --level=${HTTP_API_TEST_LEVEL} test/testcases/test_http_api