mirror of
https://github.com/infiniflow/ragflow.git
synced 2026-07-07 03:48:44 +08:00
91983106f2
### What problem does this PR solve? Closes #15433 Reranked retrieval drops results and returns short pages once pagination crosses the first candidate block, for the common page sizes 10 and 30. In `rag/nlp/search.py`, the candidate window (`RERANK_LIMIT`) is rounded up to a multiple of `page_size` to keep block based pagination aligned, and then clamped back to 64: ```python RERANK_LIMIT = math.ceil(64 / page_size) * page_size if page_size > 1 else 1 # e.g. 70 for page_size=10 RERANK_LIMIT = max(30, RERANK_LIMIT) if rerank_mdl and top > 0: RERANK_LIMIT = min(RERANK_LIMIT, top, 64) # clamps back to 64, breaking the multiple ``` `RERANK_LIMIT` is used both as the backend block size (`page = global_offset // RERANK_LIMIT`) and as the modulus that slices a page out of a reranked block (`begin = global_offset % RERANK_LIMIT`). When it stops being a multiple of `page_size`, the block that gets fetched and the slice taken from it no longer agree. With `page_size=10` and `top=1024`, page 7 returns only 4 of 10 results and the head of the next block is never shown on any page. This happens whenever the result set spans more than one block, which is the default. **Fix** The window math is moved into a small reusable helper, `Dealer._rerank_window`, which: - targets a pool of about 64 candidates, - bounds it by `top` when a reranker is active, and - always rounds to a whole number of pages, so the window stays an exact multiple of `page_size`. The call site becomes a single line, and the alignment invariant now lives in one documented place. Behavior is unchanged on every path that was already aligned (the non reranked path and any `top` that already produced a page multiple). **Verification** A simulation of the full retrieval path (per block rerank, similarity threshold filter, and the exact `page // window` and `offset % window` math) confirms the fix loses nothing where the old code lost real results: ``` ps=10 top=1024: new window=70 dropped_valid=0 | old window=64 dropped_valid=16 ps=30 top=1024: new window=90 dropped_valid=0 | old window=64 dropped_valid=66 ``` New unit tests in `test/unit_test/rag/test_search_pagination.py` cover the alignment invariant, cross block pagination (every candidate surfaced once, in order, no gaps, no short interior pages), the reported regression, and parity with the old window on the previously correct paths. All 114 cases pass and `ruff check` is clean. Fixes the reranked deep pagination data loss described above. ### Type of change - [x] Bug Fix (non-breaking change which fixes an issue)
(1). Deploy RAGFlow services and images
https://ragflow.io/docs/build_docker_image
(2). Configure the required environment for testing
Install Python dependencies (including test dependencies):
uv sync --python 3.13 --only-group test --no-default-groups --frozen
Activate the environment:
source .venv/bin/activate
Install SDK:
uv pip install sdk/python
Modify the .env file: Add the following code:
COMPOSE_PROFILES=${COMPOSE_PROFILES},tei-cpu
TEI_MODEL=BAAI/bge-small-en-v1.5
RAGFLOW_IMAGE=infiniflow/ragflow:v0.25.6 #Replace with the image you are using
Start the container（wait two minutes）:
docker compose -f docker/docker-compose.yml up -d
(3). Test Elasticsearch
a) Run sdk tests against Elasticsearch:
export HTTP_API_TEST_LEVEL=p2
export HOST_ADDRESS=http://127.0.0.1:9380 # Ensure that this port is the API port mapped to your localhost
pytest -s --tb=short --level=${HTTP_API_TEST_LEVEL} test/testcases/test_sdk_api
b) Run http api tests against Elasticsearch:
pytest -s --tb=short --level=${HTTP_API_TEST_LEVEL} test/testcases/test_http_api
(4). Test Infinity
Modify the .env file:
DOC_ENGINE=${DOC_ENGINE:-infinity}
Start the container:
docker compose -f docker/docker-compose.yml down -v
docker compose -f docker/docker-compose.yml up -d
a) Run sdk tests against Infinity:
DOC_ENGINE=infinity pytest -s --tb=short --level=${HTTP_API_TEST_LEVEL} test/testcases/test_sdk_api
b) Run http api tests against Infinity:
DOC_ENGINE=infinity pytest -s --tb=short --level=${HTTP_API_TEST_LEVEL} test/testcases/test_http_api