### What problem does this PR solve?
`common/misc_utils.once` set `executed=True` before invoking the wrapped
function, so an exception on the first call permanently disabled future
calls and returned the cached `None`.
This change marks `executed=True` only after a successful call, allowing
retries after transient failures while preserving once-only behavior
after success. It also adds regression tests for retry-after-exception
and thread-safe single execution.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
---------
Co-authored-by: Harsh Kashyap <harshkashyap@Harshs-MacBook-Pro.local>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Jin Hai <haijin.chn@gmail.com>
### Summary
Aligns Go and Python error codes/messages so both backends honor the
same RESTful API contract, removing implementation-specific error leaks
(MySQL errors, `ValueError`, `AttributeError`, Gin validator format) in
favor of clean business error codes.
**Chat list** — invalid `orderby` now returns code 101 (was: raw Python
`AttributeError` code 100); invalid `page`/`page_size` values fall back
to defaults (was: raw `ValueError`/`ProgrammingError` code 100).
**Dataset create/update/delete** — adds UUID validation (101),
extra-field rejection (101), duplicate-id detection (101), content-type
/ JSON-syntax / object-shape checks (101), and "lacks permission" for
nonexistent datasets (IDOR). Create auto-deduplicates dataset names.
Pagerank updates tolerate a missing ES index. List response includes
`parser_config` and `pagerank`.
**Session list/update** — adds filtering, sorting, and pagination
support. Empty payloads are valid no-ops. Authorization errors map to
code 109.
**Chunk list** — doc object uses Python key names (`chunk_count`,
`dataset_id`, `chunk_method`, run text status). Add validates list
element types.
**Document update** — adds `chunk_method` alias, pydantic-style Field
error messages, metadata index auto-create with refresh, and "These
documents do not belong to dataset" messages. List validates
`metadata_condition` and reports ownership errors for unmatched name/id
filters.
**Search completion** — `kb_ids` ownership failure returns code 102
instead of 109.
Released 31 contract tests from `GO_ONLY_SKIPS` (all verified passing on
both Go and Python backends with real LLM keys).
## Summary
When `STORAGE_IMPL=AWS_S3`, the Admin Service status page keeps showing
MinIO. Three things conspire:
1. `admin/server/config.py::load_configurations` only knows the
`minio` and `minio_0` config keys. An `s3` block lands on the
`case _:` branch and logs `Unknown configuration key: s3`
(issue #17294).
2. `admin/server/services.py::ServiceMgr.get_all_services` filters
retrieval services by `DOC_ENGINE` but has no equivalent filter
for `file_store` services by `STORAGE_IMPL`. A stale MinIO block
is returned regardless of the active backend.
3. The wired health check is hardcoded to `check_minio_alive`, which
calls `settings.MINIO['host']`. With `STORAGE_IMPL=AWS_S3` that
block is uninitialized, so the check always times out.
Fixes#17294
### Summary
The **aimlapi.com** provider added in #17311 does not identify itself on
any of its outgoing requests, so its traffic cannot be attributed to the
integration. This PR adds the two headers AIMLAPI expects —
`X-AIMLAPI-Source` and `X-AIMLAPI-Partner-ID` — to every request the
provider makes.
### Summary
Adds a `tenki` sandbox provider that runs each agent code execution in a
disposable Tenki (https://tenki.cloud) microVM (create → exec → destroy,
no volumes or snapshots).
Registration mirrors PR #15039, configure `api_key` and `project_id` in
Admin > Sandbox Settings.
Both runtimes are covered:
- Python: `agent/sandbox/providers/tenki.py` (structured results +
artifact collection).
- Go: `internal/agent/sandbox/tenki.go`, mirroring the e2b provider and
wired into the provider manager.
`tenki-sandbox` is an optional dependency (it requires `protobuf>=6.31`,
which differs from RAGFlow's pinned gRPC stack), lazily imported with a
clear error when missing; installation is documented in the sandbox
quickstart.
Unit tests cover execution, structured results, artifacts
(symlink/size/extension limits), non-zero exit, timeout, error mapping,
and idempotent destroy.
---------
Co-authored-by: yiming.wang <yiming.wang@luxor.com>
## Summary
GreenPT is a European AI provider with an OpenAI-compatible API,
optimized infrastructure, and datacenters powered by 100% renewable
energy.
This adds native GreenPT support across RAGFlow’s Go-first provider
system and its Python compatibility layer:
- discovers the current catalog from `GET /v1/models`
- features `glm-5.2` and `kimi-k2.7-code` for chat and coding
- supports `green-embedding` through `/v1/embeddings`
- supports `green-rerank` through `/v1/rerank`
- supports `green-s` and `green-s-pro` speech-to-text through
`/v1/listen`
- adds provider configuration, UI icon, and supported-provider
documentation
### What problem does this PR solve?
Adds first-class support for **Mistral OCR** (`POST /v1/ocr`) as a
document parser, and fixes the long-standing bug where selecting
`mistral-ocr-latest` fails with `Can't find model for
<tenant>/image2text/mistral-ocr-latest`.
`mistral-ocr-latest` is Mistral's dedicated document-OCR endpoint, not a
vision-chat (`image2text`) model, but the catalog tagged it `image2text`
— so it resolved to the `CvModel` registry, which has no `Mistral`
entry, and there was no `OcrModel` entry either. This PR registers it
correctly and wires it end to end.
Closes#17056Closes#5782Closes#7075
**What it does**
1. **`MistralParser` + `MistralOcrModel`**
(`deepdoc/parser/mistral_parser.py`, `rag/llm/ocr_model.py`) — a proper
`OcrModel` factory `Mistral OCR`, mirroring the SoMark cloud-OCR
template. Tables stay inline as HTML; the page range maps to Mistral's
native `pages` selector (absolute page indices, billed per selected
page, so multi-task documents do not re-OCR the whole file); documents
over the inline limit go through the `/v1/files` signed-URL flow with
cleanup.
2. **Removes the `image2text` mis-tag** for `mistral-ocr-latest` from
the `Mistral` factory in `conf/llm_factories.json` (it now lives only in
the `Mistral OCR` factory, typed `ocr`). This is what closes the `Can't
find model` path.
3. **`MistralCV`** (`rag/llm/cv_model.py`) — a thin `GptV4` subclass
over Mistral's OpenAI-compatible endpoint, registering a `Mistral` entry
in the `CvModel` registry so Mistral vision models (`pixtral-*`) become
usable as `image2text` at all.
4. **Figure description** — Mistral-OCR-extracted figures are captioned
using the tenant's configured `image2text` model (any provider),
matching MinerU/deepdoc behaviour.
5. **Wires the parser into every chunking method** (`naive`, `paper`,
`book`, `laws`, `manual`, `one`, `presentation`) and the `rag/flow` DAG
path. This also fixes a related latent gap where those chunkers
forwarded only `mineru_llm_name`, so any model-based OCR provider
selected on a non-`naive` method silently fell through.
**Notes on the API contract** (verified against the live Mistral API):
`pages` is a selector (returns absolute `index`, bills only the
requested pages); `include_blocks: true` returns per-block bounding
boxes usable for chunk highlighting and figure cropping; large files use
`POST /v1/files` → signed URL → OCR → `DELETE`.
**Testing**: new unit tests cover the response→sections contract (both
the 2-tuple `naive` path and the typed 3-tuple DAG path), the
position-tag rescale, the HTTP client incl. upload failure/cleanup
paths, `parse_pdf` page-range threading, registry registration, env
config, the suffix normalization, the factory catalog entry, `MistralCV`
registration, and figure-description injection. Verified end to end
against the live Mistral API on real PDFs (table extraction,
page-selector cost avoidance, figure captioning).
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
- [x] New Feature (non-breaking change which adds functionality)
## Summary
Adds page-range parsing support to the Go-native pipeline path and
introduces strict `parse_type` validation for both dataset and document
update endpoints.
## What changed
### Pages range parsing
- **`internal/utility/pdf_pages.go`** — `NormalizePDFPages`: normalizes
raw page ranges (list of `[from,to]` 1-indexed inclusive ranges) into
sorted, merged, deduplicated `[][]int`. Invalid ranges are dropped.
- **`internal/ingestion/pipeline/pdf_pages.go`** —
`NormalizeParserConfigPages`: walks any parser_config map and normalizes
`"pages"` values under every component → filetype setup, so the
persisted config always carries clean, merged ranges.
- **`internal/deepdoc/parser/pdf/parser.go`** — integrates
`resolvePagesToProcess` to filter parsed PDF pages by the configured
ranges.
- Pipeline integration (parser pages):
`internal/parser/parser/pdf_parser_common.go`, `chunk_process.go`, plus
associated e2e and unit tests.
### Parse type validation (shared logic)
- **`internal/service/parser_mode.go`** (new) — `ValidateParseTypeMode`:
shared function that validates `parse_type` (1=BuiltIn/parser_id,
2=Pipeline/pipeline_id) and ensures the corresponding field is present.
Used by both dataset and document update endpoints.
- **`internal/service/dataset/crud.go`** / `update.go` — replaces inline
`isPipelineMode`/`isBuiltinMode` computation with the shared
`service.ValidateParseTypeMode`.
- **`internal/service/document/document_dataset_update.go`** — adds
strict `parse_type` validation in `validateDatasetDocumentUpdate`,
simplifies the reparse logic to a two-way switch (isBuiltin/isPipeline)
now that parse_type is always valid.
- **`internal/service/document/document.go`** — adds `ParseType` field
to `UpdateDatasetDocumentRequest`.
- **`internal/service/document/document_dataset_update.go`** —
`updateDocumentParserConfig` fallback path when DSL loading fails.
- **`internal/service/parser_mode_test.go`** (new) — test coverage for
nil, invalid, and missing-field scenarios.
### Frontend
- **`web/src/interfaces/request/document.ts`** — adds `parseType` to
`IChangeParserRequestBody`.
- **`web/src/hooks/use-document-request.ts`** —
`useSetDocumentPipelineParser` sends `parse_type` in the PATCH payload.
- **`web/src/pages/dataset/dataset/use-change-document-parser.ts`** —
Go/Python branching for the document parser config dialog.
-
**`web/src/components/document-pipeline-dialog/use-document-pipeline-form.ts`**
— `buildSubmitData` returns `parseType` (bugfix: was dropped from the
return value).
### Test changes
- **Removed**: 2 tests that verified the old "mutually exclusive" error
(replaced by `ValidateParseTypeMode` coverage).
- **Modified**: 6 tests across document and dataset packages to include
`ParseType` in request structs.
- **Added**: new e2e tests for pages parsing (`pages_e2e_test.go`,
`pdf_parser_pages_e2e_test.go`) and unit tests for `NormalizePDFPages`,
`NormalizeParserConfigPages`, `resolvePagesToProcess`.
## Backward compatibility
- The `parse_type` field is **required** when `parser_id` or
`pipeline_id` is sent. This changes the contract for both dataset and
document PATCH endpoints, but aligns the Go backend with the existing
frontend behavior (the frontend already sends `parse_type`). Callers
that omit `parse_type` when updating parser/pipeline selections will
receive a clear error message.
- Existing callers that only update fields like `name`, `enabled`, or
`meta_fields` are unaffected.
- Test updates ensure all known call sites are compliant.
### Summary
`agent/tools/google.py` indexes `search["organic_results"]` directly
after `GoogleSearch(params).get_dict()`. serpapi returns `{"error":
...}` **without** an `organic_results` key on realistic conditions — an
invalid API key, an exhausted plan/quota, or a query that matched
nothing. that raised `KeyError('organic_results')`, which the tool's
retry loop then surfaced to the model as the opaque `"Google error:
'organic_results'"` instead of the real reason.
this guards the missing key and raises serpapi's actual `error` message
(with a clear fallback) into the existing retry/`_ERROR` path, so the
model sees e.g. `"Google error: Invalid API key."`. valid responses are
unchanged.
adds `test/unit_test/agent/tools/test_google_unit.py` covering the
error-response path (asserts the real message is surfaced, no
`KeyError`) and the normal result path, mirroring the existing
`test_googlescholar.py` pattern (skips when the `serpapi` SDK is
absent).
---------
Co-authored-by: Yaroslav98214 <diakovichyaroslav30@gmail.com>
### Summary
`agent/tools/github.py` indexes `response["items"]` directly after
`requests.get(...).json()`. the github search api returns `{"message":
...}` **without** an `items` key on realistic conditions — a rate limit
(403/429; this tool sends no auth token, so the unauthenticated ~10
req/min search limit is easy to hit) or an invalid query (422). that
raised `KeyError('items')`, which the tool's retry loop then surfaced to
the model as the opaque `"GitHub error: 'items'"` instead of the real
reason.
this guards the missing key and raises the api's actual `message` (with
a clear fallback) into the existing retry/`_ERROR` path, so the model
sees e.g. `"GitHub error: API rate limit exceeded ..."`. valid responses
are unchanged.
adds `test/unit_test/agent/tools/test_github_unit.py` covering the
rate-limit response (asserts the real message is surfaced, no
`KeyError`) and the normal result path.
Co-authored-by: Yaroslav98214 <diakovichyaroslav30@gmail.com>
Co-authored-by: Haruko386 <tryeverypossible@163.com>
Fixes#16855.
## Problem
`ListDatasetReq` declares `include_parsing_status: bool = False` but
`dataset_api_service.list_datasets` ignored the flag, so callers passing
`include_parsing_status=true` got `null` for every parsing-status field.
The helper that produces the counts
(`DocumentService.get_parsing_status_by_kb_ids`) already existed and
returned the documented shape; it just wasn't being called.
## Fix
Read the flag in `list_datasets`, call the helper only when truthy, and
attach per-kb counts to each record before serialisation. The flag
follows the same string-coercion idiom already used for `desc` in this
function.
When the flag is false or absent, no new field is added to the response,
so the change is non-breaking and additive. Existing callers see
byte-identical responses.
## Tests
New file
`test/unit_test/api/apps/services/test_dataset_api_service_list_datasets.py`
(7 tests, all passing):
-
`test_list_datasets_without_include_parsing_status_does_not_call_helper`
- `test_list_datasets_with_include_parsing_status_true_attaches_counts`
- `test_list_datasets_with_include_parsing_status_string_true`
- `test_list_datasets_with_include_parsing_status_false_skips_helper`
-
`test_list_datasets_with_include_parsing_status_string_false_skips_helper`
-
`test_list_datasets_with_empty_kb_list_skips_helper_even_when_flag_true`
-
`test_list_datasets_with_include_parsing_status_missing_kb_gets_empty_dict`
```
$ uv run pytest test/unit_test/api/apps/services/test_dataset_api_service_list_datasets.py -v
============================== 7 passed in 0.19s ==============================
```
## Files changed
- `api/apps/services/dataset_api_service.py` (+15)
-
`test/unit_test/api/apps/services/test_dataset_api_service_list_datasets.py`
(new)
## Follow-up (out of scope)
Register the `parsing_status` field in the OpenAPI response schema at
`api/apps/restful_apis/dataset_api.py` so Swagger / generated clients
document it.
Co-authored-by: Harsh23Kashyap <harsh@example.com>
### What problem does this PR solve?
The Elasticsearch and Infinity metadata push-down translators are meant
to be
interchangeable pre-filters that both mirror the in-memory `meta_filter`
fallback — the shared test section is even titled
*"is_pushdown_supported
pre-check (same logic for both backends)"*. But the two
`is_pushdown_supported`
implementations diverge:
- `common/metadata_es_filter.py` defines
`MULTIVALUE_UNSAFE_NEGATIVE_OPS = frozenset({"≠", "not in"})` and
refuses
push-down for those operators.
- `common/metadata_infinity_filter.py` has **no such guard** and pushes
them
down.
**Why the guard exists:** `meta_fields.<key>` can hold a JSON array, and
the
in-memory `meta_filter` matches a document when **any** of its values
satisfies
the predicate (per-value-bucket semantics). A document whose `tag` is
`[a, b]`
therefore still matches `tag ≠ a` — bucket `b` satisfies it. The
Infinity
push-down emits `NOT JSON_CONTAINS(meta_fields, '$.tag', '"a"')`, which
means
*"the array contains no `a` at all"*, so it **silently drops** that
document.
Same divergence for `not in`. The result: `tag ≠ a` / `tag not in (...)`
under-counts results for any document that has the excluded value
alongside
others, but only on the Infinity backend.
This is on a live production path:
`DocMetadataService._filter_doc_ids_by_metadata_infinity`
(`api/db/services/doc_metadata_service.py:948`) calls this exact
`is_pushdown_supported` as the sole gate before building the Infinity
SQL,
mirroring the ES branch at line 880 which uses the guarded ES version.
Reproduction (both real modules, no services needed):
```python
>>> from common import metadata_es_filter as es, metadata_infinity_filter as inf
>>> f = [{"op": "≠", "key": "tag", "value": "a"}]
>>> es.is_pushdown_supported(f), inf.is_pushdown_supported(f)
(False, True) # ES falls back to in-memory (correct); Infinity pushes down (wrong)
```
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
### Fix
Add the same `MULTIVALUE_UNSAFE_NEGATIVE_OPS` set to
`metadata_infinity_filter` and reject those operators in
`is_pushdown_supported`, so a single such filter forces the whole
request to
the in-memory path — the only place the per-bucket semantics are
reproduced.
`not contains` is intentionally still allowed, matching the ES backend
(`all(not contains)` == `not any(contains)`, which the push-down
expresses
correctly on multi-valued fields). The `≠` / `not in` translators
themselves
are unchanged — they remain correct for the in-memory-fallback path;
only the
push-down eligibility gate is fixed.
### Testing
- Confirmed `is_pushdown_supported([{op}])` now returns `False` for `≠`
and
`not in` on **both** backends (previously ES=False, Infinity=True).
- Added `test_pushdown_check_rejects_multivalue_unsafe_negative_ops`,
which
asserts both backends reject these ops. Confirmed red→green: it fails
against
the pre-fix Infinity module, passes after.
- `ruff check` / `ruff format --check` clean.
### Note on overlap
Open PR #16833 also edits
`test/unit_test/common/test_metadata_filter.py`, but
only **appends** at the end of the file (line 640+) and changes
`metadata_es_filter.py` / `metadata_utils.py`, not the Infinity module —
no
overlap with this change.
### Disclosure
AI-assisted (Claude Code): the divergence was surfaced by an AI-assisted
review
pass, but I independently reproduced it against the real modules,
confirmed the
production call path, and verified the fix and test before submitting.
Signed-off-by: Andrew Chen <48723787+chuenchen309@users.noreply.github.com>
Signed-off-by: chuenchen309 <48723787+chuenchen309@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
### Problem
Parsing a Q&A `.csv` can splice unrelated text into the wrong answers
(reported in #16791).
### Root cause
The `.csv` branch of `rag/app/qa.py`'s `chunk()` builds records with
`csv.reader(lines, delimiter=delimiter)` (default `quotechar='"'`), but
then indexes `lines[i]` by the reader's *record* index in `answer +=
"\n" + lines[i]`. When a line's field opens with a `"`, `csv.reader`
treats it as an unclosed quoted field and merges several physical lines
into one record. From there the record index permanently desyncs from
the physical line numbers, so `lines[i]` returns the wrong line and
unrelated Q&A content gets appended to the wrong answer.
### Reproduction (stdlib only)
```python
import csv
lines = 'Q1,A1\n"quoted answer start\ncontinues here,extra\nQ2,A2\n'.split("\n")
list(csv.reader(lines, delimiter=","))
# record 1 swallows 3 physical lines: ['quoted answer startcontinues here,extraQ2,A2']
# -> the reader index no longer matches lines[i]
list(csv.reader(lines, delimiter=",", quoting=csv.QUOTE_NONE))
# one physical line per record; index stays aligned
```
### Fix
Pass `quoting=csv.QUOTE_NONE` so one physical line maps to one record,
keeping the reader index aligned with `lines[i]` (the surrounding code
already relies on that 1:1 mapping).
Fixes#16791.
---------
Signed-off-by: Yash Raj Pandey <yashpn62@gmail.com>
Co-authored-by: Yingfeng <yingfeng.zhang@gmail.com>
### What problem does this PR solve?
Table structure recognition rows, columns, headers, and spans are
produced in cropped table image coordinates, while OCR boxes are matched
later in page-cumulative coordinates. Comparing those boxes without
normalization can skip or misassign table row and column metadata.
Closes#16992.
### What is changed?
- Map TSR components from cropped or rotated table-image coordinates
back into page-cumulative coordinates before matching OCR boxes.
- Reuse one inverse rotation transform for rotated OCR boxes and TSR
components.
- Keep TSR layout ids in the same `table-N` form used by table OCR
boxes.
- Sort columns by mapped page x-coordinate after coordinate
normalization.
- Add focused unit coverage for page offsets, zoom scaling, and
90/180/270 degree rotated tables.
### Type of change
- [x] Bug fix
- [x] Test coverage
### How has this been tested?
- `uv run --group test pytest
test/unit_test/deepdoc/parser/test_pdf_parser_table_coordinates.py -q`
- `uv run --no-sync --group test pytest
--confcutdir=test/unit_test/deepdoc/parser
test/unit_test/deepdoc/parser/test_pdf_parser_table_coordinates.py -q`
- `uv run ruff check deepdoc/parser/pdf_parser.py
test/unit_test/deepdoc/parser/test_pdf_parser_table_coordinates.py`
- `uv run --no-sync python -m py_compile deepdoc/parser/pdf_parser.py
test/unit_test/deepdoc/parser/test_pdf_parser_table_coordinates.py`
- `git diff --check`
A later dependency-sync attempt was blocked while resolving the
`en-core-web-sm` wheel from GitHub, and the repository-level unit-test
conftest can try to download missing NLTK `wordnet` data when it is not
already present locally. The focused parser test above does not require
that data fixture.
---------
Co-authored-by: zq <zhouquan1511@163.com>
### Summary
1. list and get by id API for builtin DSL
2. add DSL default component param values validation
3. remove all hard code keys for parser config
## What
`meta_filter()`'s in-memory `filter_out()` helper has two related bugs
in how it coerces `input`/`value` for comparison operators (`=`, `≠`,
`>`, `<`, `≥`, `≤`):
**1. Asymmetric commit on partial `literal_eval` failure.** The original
code:
```python
input = ast.literal_eval(input)
value = ast.literal_eval(value)
```
runs as two separate statements inside one `try`. If the first succeeds
and the second raises, the first assignment already committed — `input`
and `value` end up as different types, and the subsequent `.lower()`
case-folding silently no-ops for whichever side didn't get lowered as a
string. Concretely: metadata cell `"None"` is a valid Python literal
(`ast.literal_eval("None")` → `None`), but a query value `"none"`
(lowercase) is not — so `status = "none"` never matches a cell whose
value is `"None"`, even though the intended semantics are
case-insensitive.
**2. `value` mutated in place, reused across dict entries.**
`filter_out(v2docs, operator, value)` loops over every `(input, docids)`
pair in `v2docs` and coerces `value` inside the loop body without
resetting it — so once one entry's `literal_eval(value)` succeeds and
rebinds `value` to a non-string, every later entry in the same call
compares against that already-coerced leftover instead of the original
filter value.
## Fix
- Commit both `literal_eval` results together via tuple assignment
(`input, value = ast.literal_eval(input), ast.literal_eval(value)`), so
a failure on either side leaves both operands in their pre-coercion form
instead of a mismatched mix.
- Save the original `value` before the loop and reset it at the top of
each iteration, so per-entry coercion never leaks into the next entry.
## Testing
Added 3 tests to
`test/unit_test/common/test_metadata_filter_operators.py` covering both
symptoms (case-insensitive match against a metadata cell that's a Python
keyword literal, both `=` and `≠`; a numeric `>` comparison unaffected
by an earlier dict entry having coerced the query value). Confirmed red
on `common/metadata_utils.py` at HEAD (`git stash` the fix, tests fail
with the exact symptom described above), green after. Full existing
`test_metadata_filter_operators.py` suite (22/22, including the 3 new
tests) passes. `ruff check` and `ruff format --check` clean on both
touched files.
Sandbox note: this environment has no network access to install
`pytest`/`pytest-asyncio`, so tests were run by importing the test
module and invoking each `test_*` function directly (same approach as
prior PRs from this account against this repo, e.g. #16949).
`test_apply_semi_auto_meta_data_filter.py` (the other file exercising
`meta_filter` indirectly through `apply_meta_data_filter`) needs
`pytest-asyncio` + heavier mocking and wasn't run, but it exercises
`apply_meta_data_filter`'s async/LLM-filter-generation path, not
`filter_out`'s coercion logic touched here.
Cross-referenced open PR #16833 (also touches
`common/metadata_utils.py`) — confirmed via `gh pr diff` it only touches
`convert_conditions`/operator-alias normalization and
`apply_meta_data_filter`'s `None`-vs-`["-999"]` sentinel logic, not
`filter_out`'s comparison-coercion code path. No overlap.
---
This PR was drafted with AI assistance (Claude); I reviewed the change,
independently reproduced both symptoms, and take responsibility for it.
Signed-off-by: chuenchen309 <48723787+chuenchen309@users.noreply.github.com>
### What problem does this PR solve?
When a table dataset's `field_map` is missing or stale,
`aggregate_table_doc_metadata` falls back to probing chunk dictionaries
for each column's Elasticsearch field key. It currently performs that
probe only once, against the first dictionary chunk, and caches `(None,
"none")` if the field is absent there.
Sparse table rows commonly omit empty columns. If the first row has no
`notes` field but a later row contains `notes_raw`, the cached miss
causes every later row to be skipped and the document-level `notes`
metadata is silently lost. The result depends only on row order:
```python
chunks = [{}, {"notes_raw": "Handle with care"}]
aggregate_table_doc_metadata(chunks, task) # before: {}
aggregate_table_doc_metadata(list(reversed(chunks)), task)
# before: {"notes": ["Handle with care"]}
```
This was also identified in CodeRabbit's review of the merged
table-metadata implementation in #15780, but remained unfixed after that
PR merged:
https://github.com/infiniflow/ragflow/pull/15780#pullrequestreview-4448490676
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
### Fix
When the initial lookup found no key for a column, retry the existing
`_resolve_es_chunk_field_key` against the current chunk. Cache the first
successful resolution so subsequent rows retain the existing fast path.
Field-map-backed columns and columns found in the first chunk are
unchanged.
### Testing
- Added `test_aggregate_auto_mode_probes_later_sparse_chunks` with an
empty first row and a populated second row.
- Confirmed red→green: before the fix the assertion received `{}`; after
the fix it receives `{"notes": ["Handle with care"]}`.
- Full existing `test_table_metadata_aggregation.py`: **15 passed**.
- `ruff check` and `ruff format --check`: clean.
- `compileall` for both changed files: clean.
The local test environment did not contain the repository's full service
dependency set and had a corrupt pre-existing NLTK `wordnet.zip`. The
test module does not use those services or corpora, so the run stubbed
only `common.settings` engine flags, `json_repair`, and the global
conftest's NLTK resource lookup; the production module and aggregation
tests themselves ran unchanged.
### Duplicate-work check
Checked all currently open PRs (including changed file paths) and found
none touching `rag/utils/table_es_metadata.py` or its aggregation test.
The earlier #15780 review is historical context, not active competing
work.
### Disclosure
AI-assisted (Codex): the candidate came from an AI-assisted review
queue. I independently reproduced the order-dependent data loss against
the real module, checked the historical review and all open PR file
paths, and ran the regression plus full existing test file before
submitting.
Signed-off-by: chuenchen309 <48723787+chuenchen309@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
