Fixes#16855.
## Problem
`ListDatasetReq` declares `include_parsing_status: bool = False` but
`dataset_api_service.list_datasets` ignored the flag, so callers passing
`include_parsing_status=true` got `null` for every parsing-status field.
The helper that produces the counts
(`DocumentService.get_parsing_status_by_kb_ids`) already existed and
returned the documented shape; it just wasn't being called.
## Fix
Read the flag in `list_datasets`, call the helper only when truthy, and
attach per-kb counts to each record before serialisation. The flag
follows the same string-coercion idiom already used for `desc` in this
function.
When the flag is false or absent, no new field is added to the response,
so the change is non-breaking and additive. Existing callers see
byte-identical responses.
## Tests
New file
`test/unit_test/api/apps/services/test_dataset_api_service_list_datasets.py`
(7 tests, all passing):
-
`test_list_datasets_without_include_parsing_status_does_not_call_helper`
- `test_list_datasets_with_include_parsing_status_true_attaches_counts`
- `test_list_datasets_with_include_parsing_status_string_true`
- `test_list_datasets_with_include_parsing_status_false_skips_helper`
-
`test_list_datasets_with_include_parsing_status_string_false_skips_helper`
-
`test_list_datasets_with_empty_kb_list_skips_helper_even_when_flag_true`
-
`test_list_datasets_with_include_parsing_status_missing_kb_gets_empty_dict`
```
$ uv run pytest test/unit_test/api/apps/services/test_dataset_api_service_list_datasets.py -v
============================== 7 passed in 0.19s ==============================
```
## Files changed
- `api/apps/services/dataset_api_service.py` (+15)
-
`test/unit_test/api/apps/services/test_dataset_api_service_list_datasets.py`
(new)
## Follow-up (out of scope)
Register the `parsing_status` field in the OpenAPI response schema at
`api/apps/restful_apis/dataset_api.py` so Swagger / generated clients
document it.
Co-authored-by: Harsh23Kashyap <harsh@example.com>
### What problem does this PR solve?
- Clear stale pipeline IDs and generated data when updating documents
without `pipeline_id`.
- Support tree compilation results in pipeline workflows.
- Update compilation templates in place while preserving existing
template IDs.
- Improve duplicate-template validation messages.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
- [x] New Feature (non-breaking change which adds functionality)
Co-authored-by: Jin Hai <haijin.chn@gmail.com>
### Summary
Handle searching dataset without embedding model
In this PR, Searching datasets with different embedding models or
searching dataset with/without embedding models are not allowed. We will
improve the behavior later.
### What problem does this PR solve?
This PR adds Google BigQuery as a first-class data source connector in
RAGFlow.
It enables users to ingest and sync BigQuery data using the same
row-to-document model used by relational database connectors: selected
content columns become document text, metadata columns become document
metadata, an optional ID column provides stable document IDs, and an
optional timestamp column enables cursor-based incremental sync.
The connector supports service-account JSON credentials, table mode,
custom query mode, GoogleSQL queries, cursor-based incremental sync,
deleted-row pruning support, configurable query limits such as
`maximum_bytes_billed`, dry-run validation, batch loading, stable
document IDs, and BigQuery-aware value serialization.
### What problem does this PR solve?
The `get_ingestion_log` endpoint (both Python
`dataset_api_service.get_ingestion_log` and Go
`DatasetService.GetIngestionLog`) was returning only the
**dataset-level** field set, which omits critical fields such as `dsl`,
`document_id`, `parser_id`, `document_name`, `pipeline_id`, etc.
This caused the front-end **dataflow-result page** to be unable to
render the pipeline timeline and chunks when viewing a single ingestion
log, regardless of whether the log was a dataset-level operation
(graph/raptor/mindmap) or a per-file parse.
### Background
`PipelineOperationLogService` provides two field sets:
| Method | Fields |
|---|---|
| `get_dataset_logs_fields` | Minimal set (progress, status, timestamps,
etc.) |
| `get_file_logs_fields` | Superset — includes `document_id`, `dsl`,
`parser_id`, `document_name`, `pipeline_id`, … |
When listing logs, the API correctly distinguishes dataset-level vs
file-level logs and uses the appropriate converter. However, when
**fetching a single log by ID**, both the Python and Go implementations
were hardcoded to the dataset-level set, dropping the extra fields that
the front-end needs.
### What problem does this PR solve?
Feat:
- Allow upsert model_type for instance model
Fix:
- Allow create instance with duplicate api_key
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
- [x] New Feature (non-breaking change which adds functionality)
### What problem does this PR solve?
Fix: Remove the pagination from the search and retrieval pages.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
## Summary
Fixes#15532 — `delete_datasets()` crashes with `IndexError` when a
document has no `File2Document` row.
`delete_datasets()` in `dataset_api_service.py` called
`File2DocumentService.get_by_document_id()` and immediately accessed
`f2d[0].file_id` without checking whether the lookup returned any rows.
Documents created via API ingestion or connector sync may exist without
a linked file record, causing dataset deletion to abort with HTTP 500.
This PR mirrors the existing guard already used in `file_service.py` and
`document_api_service.py`.
### What problem does this PR solve?
FIx replicate model provider failing with valid api key
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
---------
Co-authored-by: Wang Qi <wangq8@outlook.com>
### What problem does this PR solve?
Fix: The dataset retrieval test returned an incorrect total number.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
---------
Co-authored-by: balibabu <assassin_cike@163.com>
### What problem does this PR solve?
Support factory models with multiple model types, so visual chat models
can be exposed as both image2text and chat while preserving the database
model-type-per-record design.
This also updates the SILICONFLOW model list and adds a helper script to
refresh SiliconFlow models from the provider API.
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
