From 99a77a574e4fd192ee7abfb07a82cbfcbd125013 Mon Sep 17 00:00:00 2001
From: Andrew Chen <48723787+chuenchen309@users.noreply.github.com>
Date: Thu, 16 Jul 2026 13:41:46 +0800
Subject: [PATCH] fix(metadata): discover ES fields after sparse table chunks
 (#16949)
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

### What problem does this PR solve?

When a table dataset's `field_map` is missing or stale,
`aggregate_table_doc_metadata` falls back to probing chunk dictionaries
for each column's Elasticsearch field key. It currently performs that
probe only once, against the first dictionary chunk, and caches `(None,
"none")` if the field is absent there.

Sparse table rows commonly omit empty columns. If the first row has no
`notes` field but a later row contains `notes_raw`, the cached miss
causes every later row to be skipped and the document-level `notes`
metadata is silently lost. The result depends only on row order:

```python
chunks = [{}, {"notes_raw": "Handle with care"}]
aggregate_table_doc_metadata(chunks, task)           # before: {}
aggregate_table_doc_metadata(list(reversed(chunks)), task)
# before: {"notes": ["Handle with care"]}
```

This was also identified in CodeRabbit's review of the merged
table-metadata implementation in #15780, but remained unfixed after that
PR merged:
https://github.com/infiniflow/ragflow/pull/15780#pullrequestreview-4448490676

### Type of change

- [x] Bug Fix (non-breaking change which fixes an issue)

### Fix

When the initial lookup found no key for a column, retry the existing
`_resolve_es_chunk_field_key` against the current chunk. Cache the first
successful resolution so subsequent rows retain the existing fast path.
Field-map-backed columns and columns found in the first chunk are
unchanged.

### Testing

- Added `test_aggregate_auto_mode_probes_later_sparse_chunks` with an
empty first row and a populated second row.
- Confirmed red→green: before the fix the assertion received `{}`; after
the fix it receives `{"notes": ["Handle with care"]}`.
- Full existing `test_table_metadata_aggregation.py`: **15 passed**.
- `ruff check` and `ruff format --check`: clean.
- `compileall` for both changed files: clean.

The local test environment did not contain the repository's full service
dependency set and had a corrupt pre-existing NLTK `wordnet.zip`. The
test module does not use those services or corpora, so the run stubbed
only `common.settings` engine flags, `json_repair`, and the global
conftest's NLTK resource lookup; the production module and aggregation
tests themselves ran unchanged.

### Duplicate-work check

Checked all currently open PRs (including changed file paths) and found
none touching `rag/utils/table_es_metadata.py` or its aggregation test.
The earlier #15780 review is historical context, not active competing
work.

### Disclosure

AI-assisted (Codex): the candidate came from an AI-assisted review
queue. I independently reproduced the order-dependent data loss against
the real module, checked the historical review and all open PR file
paths, and ran the regression plus full existing test file before
submitting.

Signed-off-by: chuenchen309 <48723787+chuenchen309@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
---
 rag/utils/table_es_metadata.py                     | 11 ++++++++---
 .../rag/svr/test_table_metadata_aggregation.py     | 14 ++++++++++++++
 2 files changed, 22 insertions(+), 3 deletions(-)

diff --git a/rag/utils/table_es_metadata.py b/rag/utils/table_es_metadata.py
index 82baddf5be..7f24e7775c 100644
--- a/rag/utils/table_es_metadata.py
+++ b/rag/utils/table_es_metadata.py
@@ -244,9 +244,14 @@ def aggregate_table_doc_metadata(chunks: list, task: dict) -> dict:
             for col in meta_cols:
                 tk, _src = es_col_keys.get(col, (None, "none"))
                 if not tk:
-                    if i == 0:
-                        logging.debug(f"[TABLE_META_DEBUG] no resolved ES key for column '{col}'")
-                    continue
+                    tk, src = _resolve_es_chunk_field_key(col, fm, ck)
+                    if tk:
+                        es_col_keys[col] = (tk, src)
+                        logging.debug(f"[TABLE_META_DEBUG] column '{col}' -> ES key {tk!r} (source={src}, chunk={i})")
+                    else:
+                        if i == 0:
+                            logging.debug(f"[TABLE_META_DEBUG] no resolved ES key for column '{col}'")
+                        continue
                 raw_k = _es_raw_field_key_from_typed(tk)
                 val = None
                 from_tks = False
diff --git a/test/unit_test/rag/svr/test_table_metadata_aggregation.py b/test/unit_test/rag/svr/test_table_metadata_aggregation.py
index 9406d1fe40..1924688d5a 100644
--- a/test/unit_test/rag/svr/test_table_metadata_aggregation.py
+++ b/test/unit_test/rag/svr/test_table_metadata_aggregation.py
@@ -91,6 +91,20 @@ class TestAggregateTableManualDocMetadata:
         out = aggregate_table_doc_metadata(chunks, task)
         assert out == {"country": ["Brazil"]}
 
+    def test_aggregate_auto_mode_probes_later_sparse_chunks(self, es_engine):
+        task = {
+            "parser_id": "table",
+            "parser_config": {},
+            "kb_parser_config": {
+                "table_column_mode": "auto",
+                "table_column_names": ["notes"],
+                "table_column_roles": {"notes": "both"},
+            },
+        }
+        chunks = [{}, {"notes_raw": "Handle with care"}]
+
+        assert aggregate_table_doc_metadata(chunks, task) == {"notes": ["Handle with care"]}
+
     def test_aggregate_auto_mode_all_columns_both(self, es_engine):
         task = {
             "parser_id": "table",