mirror of
https://github.com/infiniflow/ragflow.git
synced 2026-07-17 05:07:23 +08:00
bda703b588
### Summary #16524 reports that a manual metadata filter matching more documents than the ES push-down cap (`filter_doc_ids_by_meta_pushdown`'s default `limit=10000`) drops documents once the request falls back to the in-memory path — e.g. a `canon Not in ["0"]` filter over a 39,573-document KB where ~38,500 matching documents never come back. I traced through the current code path for this exact scenario: - `_filter_doc_ids_by_metadata_es` correctly detects when the match total exceeds the push-down cap and bails to the in-memory fallback instead of returning a truncated slice. - `get_flatted_meta_by_kbs` (fixed by #16095) now fully paginates through every document in the KB rather than stopping after the first page. - `es_conn.py`'s `search()` already switches to `search_after`-based pagination once `offset + limit` would exceed ES's `max_result_window` (10,000), so the outer pagination loop doesn't get cut off by that ceiling either. - `meta_filter()` then aggregates over the complete flattened metadata with no additional cap. I couldn't reproduce the drop against current `main` following that path. This PR adds a test that simulates the exact reported scenario (12,000 synthetic documents, `canon Not in ["0"]` matching all but 30 of them) against a fake, paginated `docStoreConn` standing in for Elasticsearch — both assertions pass on current `main`. To make sure this is a meaningful regression test and not a false positive, I temporarily reverted `get_flatted_meta_by_kbs` to stop after the first page (the pre-#16095 behavior) and confirmed the test correctly fails (970 of the expected 11,970 documents), then restored the original code before committing. Given all of that, it looks like #16524 may already be fixed by the combination of #16095 and the existing `search_after` handling in `es_conn.py`, but I could be missing something about the reporter's specific deployment or a scenario I haven't considered (e.g. a downstream cap once matched doc_ids feed into the content-chunk retrieval query). I've left a comment on the issue with this same analysis so a maintainer familiar with the history here can confirm or point me at what I'm missing. Either way, this test is a useful regression guard for the pagination behavior going forward.
(1). Deploy RAGFlow services and images
https://ragflow.io/docs/build_docker_image
(2). Configure the required environment for testing
Install Python dependencies (including test dependencies):
uv sync --python 3.13 --only-group test --no-default-groups --frozen
Activate the environment:
source .venv/bin/activate
Install SDK:
uv pip install sdk/python
Modify the .env file: Add the following code:
COMPOSE_PROFILES=${COMPOSE_PROFILES},tei-cpu
TEI_MODEL=BAAI/bge-small-en-v1.5
RAGFLOW_IMAGE=infiniflow/ragflow:v0.26.4 #Replace with the image you are using
Start the container（wait two minutes）:
docker compose -f docker/docker-compose.yml up -d
(3). Test Elasticsearch
a) Run sdk tests against Elasticsearch:
export HTTP_API_TEST_LEVEL=p2
export HOST_ADDRESS=http://127.0.0.1:9380 # Ensure that this port is the API port mapped to your localhost
pytest -s --tb=short --level=${HTTP_API_TEST_LEVEL} test/testcases/test_sdk_api
b) Run http api tests against Elasticsearch:
pytest -s --tb=short --level=${HTTP_API_TEST_LEVEL} test/testcases/test_http_api
(4). Test Infinity
Modify the .env file:
DOC_ENGINE=${DOC_ENGINE:-infinity}
Start the container:
docker compose -f docker/docker-compose.yml down -v
docker compose -f docker/docker-compose.yml up -d
a) Run sdk tests against Infinity:
DOC_ENGINE=infinity pytest -s --tb=short --level=${HTTP_API_TEST_LEVEL} test/testcases/test_sdk_api
b) Run http api tests against Infinity:
DOC_ENGINE=infinity pytest -s --tb=short --level=${HTTP_API_TEST_LEVEL} test/testcases/test_http_api