### Summary
1. list and get by id API for builtin DSL
2. add DSL default component param values validation
3. remove all hard code keys for parser config
### Summary
```
RAGFlow(api/default)> CHAT WITH 'glm-4-flash@new_test@zhipu-ai' MESSAGE '30 words describes LLM';
Answer: Hello! I'm ChatGLM, an AI assistant. Feel free to ask me any questions or request help with any tasks.
Input tokens: 5
Output tokens: 28
Time: 12.748241
```
Signed-off-by: Jin Hai <haijin.chn@gmail.com>
## Summary
- Preserve request-scoped system variables such as files and user IDs
during Canvas execution.
- Persist conversation history, turn counts, and tool memory in the
session DSL across turns.
- Parse agent uploads into `sys.files` and align system variable
rendering with Python.
## Testing
- `bash build.sh --test ./internal/agent/...`
- `bash build.sh --test ./internal/service/...`
<img width="1896" height="1232" alt="image"
src="https://github.com/user-attachments/assets/b420cd97-53c3-470f-a3e1-d39cea26a213"
/>
### What problem does this PR solve?
Await Response incorrectly consumed the initial `sys.query` as its
input, so the first user interaction was skipped.
This change makes Await Response wait for actual user input while
preserving the existing initial-query behavior for the Begin node.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
## Summary
- Add `GetInputForm()` for `ListOperationsComponent` to expose `Query`
input field in debug UI
- Add `GetInputForm()` for `VariableAggregatorComponent` to expose
`Variables` input field in debug UI
## Test
- Verify input form fields render correctly for both components in the
debug UI
Both backends serve GET /api/v1/language. Frontend calls it once and
caches. By this way, front end can know the backend is go or python and
thus can determine which part of logic to load.
---------
Co-authored-by: Claude <noreply@anthropic.com>
## What
`meta_filter()`'s in-memory `filter_out()` helper has two related bugs
in how it coerces `input`/`value` for comparison operators (`=`, `≠`,
`>`, `<`, `≥`, `≤`):
**1. Asymmetric commit on partial `literal_eval` failure.** The original
code:
```python
input = ast.literal_eval(input)
value = ast.literal_eval(value)
```
runs as two separate statements inside one `try`. If the first succeeds
and the second raises, the first assignment already committed — `input`
and `value` end up as different types, and the subsequent `.lower()`
case-folding silently no-ops for whichever side didn't get lowered as a
string. Concretely: metadata cell `"None"` is a valid Python literal
(`ast.literal_eval("None")` → `None`), but a query value `"none"`
(lowercase) is not — so `status = "none"` never matches a cell whose
value is `"None"`, even though the intended semantics are
case-insensitive.
**2. `value` mutated in place, reused across dict entries.**
`filter_out(v2docs, operator, value)` loops over every `(input, docids)`
pair in `v2docs` and coerces `value` inside the loop body without
resetting it — so once one entry's `literal_eval(value)` succeeds and
rebinds `value` to a non-string, every later entry in the same call
compares against that already-coerced leftover instead of the original
filter value.
## Fix
- Commit both `literal_eval` results together via tuple assignment
(`input, value = ast.literal_eval(input), ast.literal_eval(value)`), so
a failure on either side leaves both operands in their pre-coercion form
instead of a mismatched mix.
- Save the original `value` before the loop and reset it at the top of
each iteration, so per-entry coercion never leaks into the next entry.
## Testing
Added 3 tests to
`test/unit_test/common/test_metadata_filter_operators.py` covering both
symptoms (case-insensitive match against a metadata cell that's a Python
keyword literal, both `=` and `≠`; a numeric `>` comparison unaffected
by an earlier dict entry having coerced the query value). Confirmed red
on `common/metadata_utils.py` at HEAD (`git stash` the fix, tests fail
with the exact symptom described above), green after. Full existing
`test_metadata_filter_operators.py` suite (22/22, including the 3 new
tests) passes. `ruff check` and `ruff format --check` clean on both
touched files.
Sandbox note: this environment has no network access to install
`pytest`/`pytest-asyncio`, so tests were run by importing the test
module and invoking each `test_*` function directly (same approach as
prior PRs from this account against this repo, e.g. #16949).
`test_apply_semi_auto_meta_data_filter.py` (the other file exercising
`meta_filter` indirectly through `apply_meta_data_filter`) needs
`pytest-asyncio` + heavier mocking and wasn't run, but it exercises
`apply_meta_data_filter`'s async/LLM-filter-generation path, not
`filter_out`'s coercion logic touched here.
Cross-referenced open PR #16833 (also touches
`common/metadata_utils.py`) — confirmed via `gh pr diff` it only touches
`convert_conditions`/operator-alias normalization and
`apply_meta_data_filter`'s `None`-vs-`["-999"]` sentinel logic, not
`filter_out`'s comparison-coercion code path. No overlap.
---
This PR was drafted with AI assistance (Claude); I reviewed the change,
independently reproduced both symptoms, and take responsibility for it.
Signed-off-by: chuenchen309 <48723787+chuenchen309@users.noreply.github.com>
### What problem does this PR solve?
When a table dataset's `field_map` is missing or stale,
`aggregate_table_doc_metadata` falls back to probing chunk dictionaries
for each column's Elasticsearch field key. It currently performs that
probe only once, against the first dictionary chunk, and caches `(None,
"none")` if the field is absent there.
Sparse table rows commonly omit empty columns. If the first row has no
`notes` field but a later row contains `notes_raw`, the cached miss
causes every later row to be skipped and the document-level `notes`
metadata is silently lost. The result depends only on row order:
```python
chunks = [{}, {"notes_raw": "Handle with care"}]
aggregate_table_doc_metadata(chunks, task) # before: {}
aggregate_table_doc_metadata(list(reversed(chunks)), task)
# before: {"notes": ["Handle with care"]}
```
This was also identified in CodeRabbit's review of the merged
table-metadata implementation in #15780, but remained unfixed after that
PR merged:
https://github.com/infiniflow/ragflow/pull/15780#pullrequestreview-4448490676
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
### Fix
When the initial lookup found no key for a column, retry the existing
`_resolve_es_chunk_field_key` against the current chunk. Cache the first
successful resolution so subsequent rows retain the existing fast path.
Field-map-backed columns and columns found in the first chunk are
unchanged.
### Testing
- Added `test_aggregate_auto_mode_probes_later_sparse_chunks` with an
empty first row and a populated second row.
- Confirmed red→green: before the fix the assertion received `{}`; after
the fix it receives `{"notes": ["Handle with care"]}`.
- Full existing `test_table_metadata_aggregation.py`: **15 passed**.
- `ruff check` and `ruff format --check`: clean.
- `compileall` for both changed files: clean.
The local test environment did not contain the repository's full service
dependency set and had a corrupt pre-existing NLTK `wordnet.zip`. The
test module does not use those services or corpora, so the run stubbed
only `common.settings` engine flags, `json_repair`, and the global
conftest's NLTK resource lookup; the production module and aggregation
tests themselves ran unchanged.
### Duplicate-work check
Checked all currently open PRs (including changed file paths) and found
none touching `rag/utils/table_es_metadata.py` or its aggregation test.
The earlier #15780 review is historical context, not active competing
work.
### Disclosure
AI-assisted (Codex): the candidate came from an AI-assisted review
queue. I independently reproduced the order-dependent data loss against
the real module, checked the historical review and all open PR file
paths, and ran the regression plus full existing test file before
submitting.
Signed-off-by: chuenchen309 <48723787+chuenchen309@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
