## Summary
`Pipeline.Run` already accepts an `override_params` map (keyed by
`cpnID`, override-wins) that is merged into each component's params at
compile time via `canvas.WithOverrideParams`. `PatchDSL` performed a
byte-for-byte equivalent merge by baking `ParserConfig` into the DSL
text before compilation, so `runPipelineWithDSL` applied the same
`Doc.ParserConfig` twice with no effect.
## Change
- Delete the now-dead `PatchDSL` function
(`internal/ingestion/pipeline/pipeline.go`); it had a single call site.
- In `runPipelineWithDSL`, compile the DSL unchanged and pass the
extracted component params to `Run` as `override_params`.
## Behavioral note (important)
Previously `PatchDSL` was also the channel that delivered the tenant LLM
id into Extractor components: it baked `parserConfig` — after
`common.InjectExtractorLLMID(parserConfig, Tenant.LLMID)` had injected
`llm_id` — into the DSL. The old `Run` override_params re-cast
`Doc.ParserConfig` (which intentionally omits `llm_id`). So removing
`PatchDSL` while keeping that re-cast would have silently dropped LLM id
injection. This change passes the local `parserConfig` (the
LLMID-injected copy) to `Run`, preserving the prior behavior.
The DSL string returned by `runPipelineWithDSL` is only consumed by
`recordPipelineLog` (structure storage) and `ExtractPayload` (which
reads the terminal structure, not params), so returning the original DSL
is equivalent.
## Test plan
- `bash build.sh --test ./internal/ingestion/task/...
./internal/ingestion/pipeline/...` — both packages pass.
- `gofmt` clean; pre-commit hooks pass.
Co-authored-by: Claude <noreply@anthropic.com>
## What
The Japanese locale (`web/src/locales/ja.ts`) was missing translations
for **1703 of 2686 keys (~63%)**. Because i18next's `fallbackLng`
silently falls back to English for any missing key, most of the UI
stayed in English even after selecting 日本語 — e.g. the Settings > Model
Providers page, the Agent/Flow builder, dataset configuration panels,
and more.
This PR fills in all 1703 missing keys with Japanese translations.
Follow-up to #16468.
PR #16468 fixed the Python
(`api/db/joint_services/tenant_model_service.py
split_model_name`) and Go (`internal/service/model_service.go
parseModelName`)
parsers to right-anchor the '@'-split on the composite
"model_name@instance@provider" key, but the matching helper in the
front-end
(`web/src/utils/llm-util.ts parseModelValue`) was missed.
parseModelValue used `split('@')` (anchored on the first '@'), which
silently
mangles composite keys whose model_name itself contains '@' (e.g. LM
Studio
embedding IDs like `text-embedding-nomic-embed-text-v1.5@q8_0`):
text-embedding-nomic-embed-text-v1.5@q8_0@lmstudio@LM-Studio
became
model_name: text-embedding-nomic-embed-text-v1.5
model_instance: q8_0@lmstudio
model_provider: LM-Studio
`PATCH /api/v1/models/default` then sent those to the server, which
returned
HTTP 200 with body `{"code": 102, "message": "Instance 'q8_0@lmstudio'
not
found for provider 'LM-Studio'"}`. The UI swallowed the body-level error
code, so the failure was silent: no toast, the Embedding field stayed
empty.
This change makes parseModelValue do a right-anchored split that mirrors
the
Python `split_model_name` `rsplit('@', 2)` exactly:
- 3-part form: model_name@instance@provider -> same three fields.
- 2-part form: model_name@provider -> model_instance defaults to
"default"
(matching Python).
- 4+-part form (embedded '@' in model name): last two fields are
anchored
as instance and provider, everything to the left is the bare model name.
`parseModelUuid` is updated to the same right-anchored split so the
factoryId portion of `model_name@factory_id[#instance]` keeps the last
'@'
as the separator even when the model name itself contains '@'.
Adds jest cases under `web/src/utils/tests/llm-util.test.ts` (12 total)
covering the plain 3-part, 2-part, 4-part, multi-'@' name,
buildModelValue
round-trip and parseModelUuid variants.
Note for reviewers: the front-end is bundled into the published docker
image, so a release containing this fix will need `docker compose build
--no-cache ragflow-cpu` (or equivalent) for it to be visible to end
users.
Refs #16468, #16467
Fixes#16855.
## Problem
`ListDatasetReq` declares `include_parsing_status: bool = False` but
`dataset_api_service.list_datasets` ignored the flag, so callers passing
`include_parsing_status=true` got `null` for every parsing-status field.
The helper that produces the counts
(`DocumentService.get_parsing_status_by_kb_ids`) already existed and
returned the documented shape; it just wasn't being called.
## Fix
Read the flag in `list_datasets`, call the helper only when truthy, and
attach per-kb counts to each record before serialisation. The flag
follows the same string-coercion idiom already used for `desc` in this
function.
When the flag is false or absent, no new field is added to the response,
so the change is non-breaking and additive. Existing callers see
byte-identical responses.
## Tests
New file
`test/unit_test/api/apps/services/test_dataset_api_service_list_datasets.py`
(7 tests, all passing):
-
`test_list_datasets_without_include_parsing_status_does_not_call_helper`
- `test_list_datasets_with_include_parsing_status_true_attaches_counts`
- `test_list_datasets_with_include_parsing_status_string_true`
- `test_list_datasets_with_include_parsing_status_false_skips_helper`
-
`test_list_datasets_with_include_parsing_status_string_false_skips_helper`
-
`test_list_datasets_with_empty_kb_list_skips_helper_even_when_flag_true`
-
`test_list_datasets_with_include_parsing_status_missing_kb_gets_empty_dict`
```
$ uv run pytest test/unit_test/api/apps/services/test_dataset_api_service_list_datasets.py -v
============================== 7 passed in 0.19s ==============================
```
## Files changed
- `api/apps/services/dataset_api_service.py` (+15)
-
`test/unit_test/api/apps/services/test_dataset_api_service_list_datasets.py`
(new)
## Follow-up (out of scope)
Register the `parsing_status` field in the OpenAPI response schema at
`api/apps/restful_apis/dataset_api.py` so Swagger / generated clients
document it.
Co-authored-by: Harsh23Kashyap <harsh@example.com>
### What problem does this PR solve?
The Elasticsearch and Infinity metadata push-down translators are meant
to be
interchangeable pre-filters that both mirror the in-memory `meta_filter`
fallback — the shared test section is even titled
*"is_pushdown_supported
pre-check (same logic for both backends)"*. But the two
`is_pushdown_supported`
implementations diverge:
- `common/metadata_es_filter.py` defines
`MULTIVALUE_UNSAFE_NEGATIVE_OPS = frozenset({"≠", "not in"})` and
refuses
push-down for those operators.
- `common/metadata_infinity_filter.py` has **no such guard** and pushes
them
down.
**Why the guard exists:** `meta_fields.<key>` can hold a JSON array, and
the
in-memory `meta_filter` matches a document when **any** of its values
satisfies
the predicate (per-value-bucket semantics). A document whose `tag` is
`[a, b]`
therefore still matches `tag ≠ a` — bucket `b` satisfies it. The
Infinity
push-down emits `NOT JSON_CONTAINS(meta_fields, '$.tag', '"a"')`, which
means
*"the array contains no `a` at all"*, so it **silently drops** that
document.
Same divergence for `not in`. The result: `tag ≠ a` / `tag not in (...)`
under-counts results for any document that has the excluded value
alongside
others, but only on the Infinity backend.
This is on a live production path:
`DocMetadataService._filter_doc_ids_by_metadata_infinity`
(`api/db/services/doc_metadata_service.py:948`) calls this exact
`is_pushdown_supported` as the sole gate before building the Infinity
SQL,
mirroring the ES branch at line 880 which uses the guarded ES version.
Reproduction (both real modules, no services needed):
```python
>>> from common import metadata_es_filter as es, metadata_infinity_filter as inf
>>> f = [{"op": "≠", "key": "tag", "value": "a"}]
>>> es.is_pushdown_supported(f), inf.is_pushdown_supported(f)
(False, True) # ES falls back to in-memory (correct); Infinity pushes down (wrong)
```
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
### Fix
Add the same `MULTIVALUE_UNSAFE_NEGATIVE_OPS` set to
`metadata_infinity_filter` and reject those operators in
`is_pushdown_supported`, so a single such filter forces the whole
request to
the in-memory path — the only place the per-bucket semantics are
reproduced.
`not contains` is intentionally still allowed, matching the ES backend
(`all(not contains)` == `not any(contains)`, which the push-down
expresses
correctly on multi-valued fields). The `≠` / `not in` translators
themselves
are unchanged — they remain correct for the in-memory-fallback path;
only the
push-down eligibility gate is fixed.
### Testing
- Confirmed `is_pushdown_supported([{op}])` now returns `False` for `≠`
and
`not in` on **both** backends (previously ES=False, Infinity=True).
- Added `test_pushdown_check_rejects_multivalue_unsafe_negative_ops`,
which
asserts both backends reject these ops. Confirmed red→green: it fails
against
the pre-fix Infinity module, passes after.
- `ruff check` / `ruff format --check` clean.
### Note on overlap
Open PR #16833 also edits
`test/unit_test/common/test_metadata_filter.py`, but
only **appends** at the end of the file (line 640+) and changes
`metadata_es_filter.py` / `metadata_utils.py`, not the Infinity module —
no
overlap with this change.
### Disclosure
AI-assisted (Claude Code): the divergence was surfaced by an AI-assisted
review
pass, but I independently reproduced it against the real modules,
confirmed the
production call path, and verified the fix and test before submitting.
Signed-off-by: Andrew Chen <48723787+chuenchen309@users.noreply.github.com>
Signed-off-by: chuenchen309 <48723787+chuenchen309@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
### Problem
Parsing a Q&A `.csv` can splice unrelated text into the wrong answers
(reported in #16791).
### Root cause
The `.csv` branch of `rag/app/qa.py`'s `chunk()` builds records with
`csv.reader(lines, delimiter=delimiter)` (default `quotechar='"'`), but
then indexes `lines[i]` by the reader's *record* index in `answer +=
"\n" + lines[i]`. When a line's field opens with a `"`, `csv.reader`
treats it as an unclosed quoted field and merges several physical lines
into one record. From there the record index permanently desyncs from
the physical line numbers, so `lines[i]` returns the wrong line and
unrelated Q&A content gets appended to the wrong answer.
### Reproduction (stdlib only)
```python
import csv
lines = 'Q1,A1\n"quoted answer start\ncontinues here,extra\nQ2,A2\n'.split("\n")
list(csv.reader(lines, delimiter=","))
# record 1 swallows 3 physical lines: ['quoted answer startcontinues here,extraQ2,A2']
# -> the reader index no longer matches lines[i]
list(csv.reader(lines, delimiter=",", quoting=csv.QUOTE_NONE))
# one physical line per record; index stays aligned
```
### Fix
Pass `quoting=csv.QUOTE_NONE` so one physical line maps to one record,
keeping the reader index aligned with `lines[i]` (the surrounding code
already relies on that 1:1 mapping).
Fixes#16791.
---------
Signed-off-by: Yash Raj Pandey <yashpn62@gmail.com>
Co-authored-by: Yingfeng <yingfeng.zhang@gmail.com>
### What problem does this PR solve?
Table structure recognition rows, columns, headers, and spans are
produced in cropped table image coordinates, while OCR boxes are matched
later in page-cumulative coordinates. Comparing those boxes without
normalization can skip or misassign table row and column metadata.
Closes#16992.
### What is changed?
- Map TSR components from cropped or rotated table-image coordinates
back into page-cumulative coordinates before matching OCR boxes.
- Reuse one inverse rotation transform for rotated OCR boxes and TSR
components.
- Keep TSR layout ids in the same `table-N` form used by table OCR
boxes.
- Sort columns by mapped page x-coordinate after coordinate
normalization.
- Add focused unit coverage for page offsets, zoom scaling, and
90/180/270 degree rotated tables.
### Type of change
- [x] Bug fix
- [x] Test coverage
### How has this been tested?
- `uv run --group test pytest
test/unit_test/deepdoc/parser/test_pdf_parser_table_coordinates.py -q`
- `uv run --no-sync --group test pytest
--confcutdir=test/unit_test/deepdoc/parser
test/unit_test/deepdoc/parser/test_pdf_parser_table_coordinates.py -q`
- `uv run ruff check deepdoc/parser/pdf_parser.py
test/unit_test/deepdoc/parser/test_pdf_parser_table_coordinates.py`
- `uv run --no-sync python -m py_compile deepdoc/parser/pdf_parser.py
test/unit_test/deepdoc/parser/test_pdf_parser_table_coordinates.py`
- `git diff --check`
A later dependency-sync attempt was blocked while resolving the
`en-core-web-sm` wheel from GitHub, and the repository-level unit-test
conftest can try to download missing NLTK `wordnet` data when it is not
already present locally. The focused parser test above does not require
that data fixture.
---------
Co-authored-by: zq <zhouquan1511@163.com>
### Summary
1. list and get by id API for builtin DSL
2. add DSL default component param values validation
3. remove all hard code keys for parser config
### Summary
```
RAGFlow(api/default)> CHAT WITH 'glm-4-flash@new_test@zhipu-ai' MESSAGE '30 words describes LLM';
Answer: Hello! I'm ChatGLM, an AI assistant. Feel free to ask me any questions or request help with any tasks.
Input tokens: 5
Output tokens: 28
Time: 12.748241
```
Signed-off-by: Jin Hai <haijin.chn@gmail.com>
## Summary
- Preserve request-scoped system variables such as files and user IDs
during Canvas execution.
- Persist conversation history, turn counts, and tool memory in the
session DSL across turns.
- Parse agent uploads into `sys.files` and align system variable
rendering with Python.
## Testing
- `bash build.sh --test ./internal/agent/...`
- `bash build.sh --test ./internal/service/...`
<img width="1896" height="1232" alt="image"
src="https://github.com/user-attachments/assets/b420cd97-53c3-470f-a3e1-d39cea26a213"
/>
