mirror of
https://github.com/infiniflow/ragflow.git
synced 2026-07-06 03:18:36 +08:00
91983106f2
### What problem does this PR solve? Closes #15433 Reranked retrieval drops results and returns short pages once pagination crosses the first candidate block, for the common page sizes 10 and 30. In `rag/nlp/search.py`, the candidate window (`RERANK_LIMIT`) is rounded up to a multiple of `page_size` to keep block based pagination aligned, and then clamped back to 64: ```python RERANK_LIMIT = math.ceil(64 / page_size) * page_size if page_size > 1 else 1 # e.g. 70 for page_size=10 RERANK_LIMIT = max(30, RERANK_LIMIT) if rerank_mdl and top > 0: RERANK_LIMIT = min(RERANK_LIMIT, top, 64) # clamps back to 64, breaking the multiple ``` `RERANK_LIMIT` is used both as the backend block size (`page = global_offset // RERANK_LIMIT`) and as the modulus that slices a page out of a reranked block (`begin = global_offset % RERANK_LIMIT`). When it stops being a multiple of `page_size`, the block that gets fetched and the slice taken from it no longer agree. With `page_size=10` and `top=1024`, page 7 returns only 4 of 10 results and the head of the next block is never shown on any page. This happens whenever the result set spans more than one block, which is the default. **Fix** The window math is moved into a small reusable helper, `Dealer._rerank_window`, which: - targets a pool of about 64 candidates, - bounds it by `top` when a reranker is active, and - always rounds to a whole number of pages, so the window stays an exact multiple of `page_size`. The call site becomes a single line, and the alignment invariant now lives in one documented place. Behavior is unchanged on every path that was already aligned (the non reranked path and any `top` that already produced a page multiple). **Verification** A simulation of the full retrieval path (per block rerank, similarity threshold filter, and the exact `page // window` and `offset % window` math) confirms the fix loses nothing where the old code lost real results: ``` ps=10 top=1024: new window=70 dropped_valid=0 | old window=64 dropped_valid=16 ps=30 top=1024: new window=90 dropped_valid=0 | old window=64 dropped_valid=66 ``` New unit tests in `test/unit_test/rag/test_search_pagination.py` cover the alignment invariant, cross block pagination (every candidate surfaced once, in order, no gaps, no short interior pages), the reported regression, and parity with the old window on the previously correct paths. All 114 cases pass and `ruff check` is clean. Fixes the reranked deep pagination data loss described above. ### Type of change - [x] Bug Fix (non-breaking change which fixes an issue)