mirror of
https://github.com/infiniflow/ragflow.git
synced 2026-07-17 05:07:23 +08:00
99a77a574e
### What problem does this PR solve?
When a table dataset's `field_map` is missing or stale,
`aggregate_table_doc_metadata` falls back to probing chunk dictionaries
for each column's Elasticsearch field key. It currently performs that
probe only once, against the first dictionary chunk, and caches `(None,
"none")` if the field is absent there.
Sparse table rows commonly omit empty columns. If the first row has no
`notes` field but a later row contains `notes_raw`, the cached miss
causes every later row to be skipped and the document-level `notes`
metadata is silently lost. The result depends only on row order:
```python
chunks = [{}, {"notes_raw": "Handle with care"}]
aggregate_table_doc_metadata(chunks, task) # before: {}
aggregate_table_doc_metadata(list(reversed(chunks)), task)
# before: {"notes": ["Handle with care"]}
```
This was also identified in CodeRabbit's review of the merged
table-metadata implementation in #15780, but remained unfixed after that
PR merged:
https://github.com/infiniflow/ragflow/pull/15780#pullrequestreview-4448490676
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
### Fix
When the initial lookup found no key for a column, retry the existing
`_resolve_es_chunk_field_key` against the current chunk. Cache the first
successful resolution so subsequent rows retain the existing fast path.
Field-map-backed columns and columns found in the first chunk are
unchanged.
### Testing
- Added `test_aggregate_auto_mode_probes_later_sparse_chunks` with an
empty first row and a populated second row.
- Confirmed red→green: before the fix the assertion received `{}`; after
the fix it receives `{"notes": ["Handle with care"]}`.
- Full existing `test_table_metadata_aggregation.py`: **15 passed**.
- `ruff check` and `ruff format --check`: clean.
- `compileall` for both changed files: clean.
The local test environment did not contain the repository's full service
dependency set and had a corrupt pre-existing NLTK `wordnet.zip`. The
test module does not use those services or corpora, so the run stubbed
only `common.settings` engine flags, `json_repair`, and the global
conftest's NLTK resource lookup; the production module and aggregation
tests themselves ran unchanged.
### Duplicate-work check
Checked all currently open PRs (including changed file paths) and found
none touching `rag/utils/table_es_metadata.py` or its aggregation test.
The earlier #15780 review is historical context, not active competing
work.
### Disclosure
AI-assisted (Codex): the candidate came from an AI-assisted review
queue. I independently reproduced the order-dependent data loss against
the real module, checked the historical review and all open PR file
paths, and ran the regression plus full existing test file before
submitting.
Signed-off-by: chuenchen309 <48723787+chuenchen309@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>