feat(api): add unified index API and dataset management endpoints (#14222)

### What problem does this PR solve?

## Summary

Refactor the dataset API layer into a clean service/REST separation
pattern, add a unified `/index` API for graph/raptor/mindmap operations,
and introduce several new dataset management endpoints with full test
coverage.

## Changes

### Service Layer (`dataset_api_service.py`)

- Added `trace_index(dataset_id, tenant_id, index_type)` — unified trace
function for all index types
- Added `run_index`, `delete_index` service functions
- Added `get_dataset`, `get_ingestion_summary`, `list_ingestion_logs`,
`get_ingestion_log`
- Added `run_embedding`, `list_tags`, `aggregate_tags`, `delete_tags`,
`rename_tag`
- Added `get_flattened_metadata`, `get_auto_metadata`,
`update_auto_metadata`

### REST API Layer (`dataset_api.py`)

**New unified routes:**

| Method | Route | Description |
|--------|-------|-------------|
| POST | `/datasets/<id>/index?type=graph\|raptor\|mindmap` | Run index
task |
| GET | `/datasets/<id>/index?type=graph\|raptor\|mindmap` | Trace index
task |
| DELETE | `/datasets/<id>/<index_type>` | Delete index |
| GET | `/datasets/<id>` | Get dataset details |
| GET | `/datasets/<id>/ingestions/summary` | Ingestion summary |
| GET | `/datasets/<id>/ingestions` | List ingestion logs |
| GET | `/datasets/<id>/ingestions/<log_id>` | Get single ingestion log
|
| POST | `/datasets/<id>/embedding` | Run embedding |
| GET | `/datasets/<id>/tags` | List tags |
| GET | `/datasets/tags/aggregation` | Aggregate tags across datasets |
| DELETE | `/datasets/<id>/tags` | Delete tags |
| PUT | `/datasets/<id>/tags` | Rename tag |
| GET | `/datasets/metadata/flattened` | Get flattened metadata |
| GET/PUT | `/datasets/<id>/metadata/config` | New metadata config path
|

**Removed routes (replaced by unified `/index`):**

- `POST /datasets/<id>/mindmap`
- `GET /datasets/<id>/mindmap`

**Preserved legacy routes (backward compatibility):**

- `/run_graphrag`, `/trace_graphrag`, `/run_raptor`, `/trace_raptor`
- `/auto_metadata` GET/PUT

### Test Suite

- Updated `common.py` helpers: added `trace_index`, removed
`run_mindmap`/`trace_mindmap`
- Added 7 new test files with 39 test cases total:

| Test File | Cases |
|-----------|-------|
| `test_get_dataset.py` | 4 |
| `test_ingestion_summary.py` | 2 |
| `test_ingestion_logs.py` | 5 |
| `test_index_api.py` | 14 |
| `test_embedding.py` | 2 |
| `test_tags.py` | 8 |
| `test_flattened_metadata.py` | 4 |

- Deleted `test_mindmap_tasks.py` (covered by unified index tests)

## Design Decisions

1. **Unified `/index?type=...`** — single endpoint replaces 3 separate
route pairs for graph/raptor/mindmap
2. **Backward compatibility** — old routes (`/run_graphrag`,
`/run_raptor`, `/auto_metadata`) preserved alongside new paths
3. **`_VALID_INDEX_TYPES = {"graph", "raptor", "mindmap"}`** — input
validation via constant set
4. **`_INDEX_TYPE_TO_TASK_ID_FIELD`** — maps index type to KB model task
ID field for clean dispatch

## Files Changed

- `api/apps/restful_apis/dataset_api.py`
- `api/apps/services/dataset_api_service.py`
- `sdk/python/ragflow_sdk/modules/dataset.py`
- `test/testcases/test_http_api/common.py`
- `test/testcases/test_http_api/test_dataset_management/` (7 new files)
### Type of change

- [x] New Feature (non-breaking change which adds functionality)
- [x] Refactoring

---------

Signed-off-by: noob <yixiao121314@outlook.com>

test/testcases/test_http_api/common.py

@@ -23,7 +23,8 @@ from utils.file_utils import create_txt_file
 HEADERS = {"Content-Type": "application/json"}
 DATASETS_API_URL = f"/api/{VERSION}/datasets"
 FILE_API_URL = f"/api/{VERSION}/datasets/{{dataset_id}}/documents"
-FILE_CHUNK_API_URL = f"/api/{VERSION}/datasets/{{dataset_id}}/chunks"
+FILE_PARSE_API_URL = f"/api/{VERSION}/datasets/{{dataset_id}}/documents/parse"
+FILE_STOP_PARSE_API_URL = f"/api/{VERSION}/datasets/{{dataset_id}}/documents/stop"
 CHUNK_API_URL = f"/api/{VERSION}/datasets/{{dataset_id}}/documents/{{document_id}}/chunks"
 CHAT_ASSISTANT_API_URL = f"/api/{VERSION}/chats"
 SESSION_WITH_CHAT_ASSISTANT_API_URL = f"/api/{VERSION}/chats/{{chat_id}}/sessions"
@@ -136,15 +137,15 @@ def delete_all_documents(auth, dataset_id, *, page_size=1000):
     return delete_documents(auth, dataset_id, {"ids": None, "delete_all": True})
 
 
-def parse_documents(auth, dataset_id, payload=None):
-    url = f"{HOST_ADDRESS}{FILE_CHUNK_API_URL}".format(dataset_id=dataset_id)
-    res = requests.post(url=url, headers=HEADERS, auth=auth, json=payload)
+def parse_documents(auth, dataset_id, payload=None, *, headers=HEADERS):
+    url = f"{HOST_ADDRESS}{FILE_PARSE_API_URL}".format(dataset_id=dataset_id)
+    res = requests.post(url=url, headers=headers, auth=auth, json=payload)
     return res.json()
 
 
 def stop_parse_documents(auth, dataset_id, payload=None):
-    url = f"{HOST_ADDRESS}{FILE_CHUNK_API_URL}".format(dataset_id=dataset_id)
-    res = requests.delete(url=url, headers=HEADERS, auth=auth, json=payload)
+    url = f"{HOST_ADDRESS}{FILE_STOP_PARSE_API_URL}".format(dataset_id=dataset_id)
+    res = requests.post(url=url, headers=HEADERS, auth=auth, json=payload)
     return res.json()
 
 
@@ -161,9 +162,9 @@ def bulk_upload_documents(auth, dataset_id, num, tmp_path):
 
 
 # CHUNK MANAGEMENT WITHIN DATASET
-def add_chunk(auth, dataset_id, document_id, payload=None):
+def add_chunk(auth, dataset_id, document_id, payload=None, *, headers=HEADERS):
     url = f"{HOST_ADDRESS}{CHUNK_API_URL}".format(dataset_id=dataset_id, document_id=document_id)
-    res = requests.post(url=url, headers=HEADERS, auth=auth, json=payload)
+    res = requests.post(url=url, headers=headers, auth=auth, json=payload)
     return res.json()
 
 
@@ -195,9 +196,9 @@ def delete_all_chunks(auth, dataset_id, document_id, *, page_size=1000):
     return delete_chunks(auth, dataset_id, document_id, {"chunk_ids": None, "delete_all": True})
 
 
-def retrieval_chunks(auth, payload=None):
+def retrieval_chunks(auth, payload=None, *, headers=HEADERS):
     url = f"{HOST_ADDRESS}{RETRIEVAL_API_URL}"
-    res = requests.post(url=url, headers=HEADERS, auth=auth, json=payload)
+    res = requests.post(url=url, headers=headers, auth=auth, json=payload)
     return res.json()
 
 
@@ -210,9 +211,9 @@ def batch_add_chunks(auth, dataset_id, document_id, num):
 
 
 # CHAT ASSISTANT MANAGEMENT
-def create_chat_assistant(auth, payload=None):
+def create_chat_assistant(auth, payload=None, *, headers=HEADERS):
     url = f"{HOST_ADDRESS}{CHAT_ASSISTANT_API_URL}"
-    res = requests.post(url=url, headers=HEADERS, auth=auth, json=payload)
+    res = requests.post(url=url, headers=headers, auth=auth, json=payload)
     return res.json()
 
 
@@ -259,9 +260,9 @@ def batch_create_chat_assistants(auth, num):
 
 
 # SESSION MANAGEMENT
-def create_session_with_chat_assistant(auth, chat_assistant_id, payload=None):
+def create_session_with_chat_assistant(auth, chat_assistant_id, payload=None, *, headers=HEADERS):
     url = f"{HOST_ADDRESS}{SESSION_WITH_CHAT_ASSISTANT_API_URL}".format(chat_id=chat_assistant_id)
-    res = requests.post(url=url, headers=HEADERS, auth=auth, json=payload)
+    res = requests.post(url=url, headers=headers, auth=auth, json=payload)
     return res.json()
 
 
@@ -297,13 +298,13 @@ def batch_add_sessions_with_chat_assistant(auth, chat_assistant_id, num):
 
 # DATASET GRAPH AND TASKS
 def knowledge_graph(auth, dataset_id, params=None):
-    url = f"{HOST_ADDRESS}{DATASETS_API_URL}/{dataset_id}/knowledge_graph"
+    url = f"{HOST_ADDRESS}{DATASETS_API_URL}/{dataset_id}/graph/search"
     res = requests.get(url=url, headers=HEADERS, auth=auth, params=params)
     return res.json()
 
 
 def delete_knowledge_graph(auth, dataset_id, payload=None):
-    url = f"{HOST_ADDRESS}{DATASETS_API_URL}/{dataset_id}/knowledge_graph"
+    url = f"{HOST_ADDRESS}{DATASETS_API_URL}/{dataset_id}/graph"
     if payload is None:
         res = requests.delete(url=url, headers=HEADERS, auth=auth)
     else:
@@ -311,39 +312,15 @@ def delete_knowledge_graph(auth, dataset_id, payload=None):
     return res.json()
 
 
-def run_graphrag(auth, dataset_id, payload=None):
-    url = f"{HOST_ADDRESS}{DATASETS_API_URL}/{dataset_id}/run_graphrag"
-    res = requests.post(url=url, headers=HEADERS, auth=auth, json=payload)
-    return res.json()
-
-
-def trace_graphrag(auth, dataset_id, params=None):
-    url = f"{HOST_ADDRESS}{DATASETS_API_URL}/{dataset_id}/trace_graphrag"
-    res = requests.get(url=url, headers=HEADERS, auth=auth, params=params)
-    return res.json()
-
-
-def run_raptor(auth, dataset_id, payload=None):
-    url = f"{HOST_ADDRESS}{DATASETS_API_URL}/{dataset_id}/run_raptor"
-    res = requests.post(url=url, headers=HEADERS, auth=auth, json=payload)
-    return res.json()
-
-
-def trace_raptor(auth, dataset_id, params=None):
-    url = f"{HOST_ADDRESS}{DATASETS_API_URL}/{dataset_id}/trace_raptor"
-    res = requests.get(url=url, headers=HEADERS, auth=auth, params=params)
-    return res.json()
-
-
 def metadata_summary(auth, dataset_id, params=None):
     url = f"{HOST_ADDRESS}{DATASETS_API_URL}/{dataset_id}/metadata/summary"
     res = requests.get(url=url, headers=HEADERS, auth=auth, params=params)
     return res.json()
 
 
-def metadata_batch_update(auth, dataset_id, payload=None):
+def metadata_batch_update(auth, dataset_id, payload=None, *, headers=HEADERS):
     url = f"{HOST_ADDRESS}{DATASETS_API_URL}/{dataset_id}/metadata/update"
-    res = requests.post(url=url, headers=HEADERS, auth=auth, json=payload)
+    res = requests.post(url=url, headers=headers, auth=auth, json=payload)
     return res.json()
 
 
@@ -358,16 +335,16 @@ def update_documents_metadata(auth, dataset_id, payload=None):
 
 
 # CHAT COMPLETIONS AND RELATED QUESTIONS
-def related_questions(auth, payload=None):
+def related_questions(auth, payload=None, *, headers=HEADERS):
     url = f"{HOST_ADDRESS}/api/{VERSION}/sessions/related_questions"
-    res = requests.post(url=url, headers=HEADERS, auth=auth, json=payload)
+    res = requests.post(url=url, headers=headers, auth=auth, json=payload)
     return res.json()
 
 
 # AGENT MANAGEMENT AND SESSIONS
-def create_agent(auth, payload=None):
+def create_agent(auth, payload=None, *, headers=HEADERS):
     url = f"{HOST_ADDRESS}{AGENT_API_URL}"
-    res = requests.post(url=url, headers=HEADERS, auth=auth, json=payload)
+    res = requests.post(url=url, headers=headers, auth=auth, json=payload)
     return res.json()
 
 
@@ -439,7 +416,7 @@ def chat_completions(auth, chat_id=None, payload=None):
     return res.json()
 
 
-def chat_completions_openai(auth, chat_id, payload=None):
+def chat_completions_openai(auth, chat_id, payload=None, *, headers=HEADERS):
     """
     Send a request to the OpenAI-compatible chat completions endpoint.
 
@@ -454,5 +431,88 @@ def chat_completions_openai(auth, chat_id, payload=None):
         Response JSON in OpenAI chat completions format with usage information
     """
     url = f"{HOST_ADDRESS}/api/{VERSION}/chats_openai/{chat_id}/chat/completions"
-    res = requests.post(url=url, headers=HEADERS, auth=auth, json=payload)
+    res = requests.post(url=url, headers=headers, auth=auth, json=payload)
+    return res.json()
+
+
+# NEW DATASET ENDPOINTS
+def get_dataset(auth, dataset_id, *, headers=HEADERS):
+    url = f"{HOST_ADDRESS}{DATASETS_API_URL}/{dataset_id}"
+    res = requests.get(url=url, headers=headers, auth=auth)
+    return res.json()
+
+
+def get_ingestion_summary(auth, dataset_id, *, headers=HEADERS):
+    url = f"{HOST_ADDRESS}{DATASETS_API_URL}/{dataset_id}/ingestions/summary"
+    res = requests.get(url=url, headers=headers, auth=auth)
+    return res.json()
+
+
+def list_ingestion_logs(auth, dataset_id, params=None, *, headers=HEADERS):
+    url = f"{HOST_ADDRESS}{DATASETS_API_URL}/{dataset_id}/ingestions"
+    res = requests.get(url=url, headers=headers, auth=auth, params=params)
+    return res.json()
+
+
+def get_ingestion_log(auth, dataset_id, log_id, *, headers=HEADERS):
+    url = f"{HOST_ADDRESS}{DATASETS_API_URL}/{dataset_id}/ingestions/{log_id}"
+    res = requests.get(url=url, headers=headers, auth=auth)
+    return res.json()
+
+
+def run_index(auth, dataset_id, index_type, payload=None, *, headers=HEADERS):
+    url = f"{HOST_ADDRESS}{DATASETS_API_URL}/{dataset_id}/index"
+    params = {"type": index_type}
+    res = requests.post(url=url, headers=headers, auth=auth, json=payload, params=params)
+    return res.json()
+
+
+def trace_index(auth, dataset_id, index_type, params=None, *, headers=HEADERS):
+    url = f"{HOST_ADDRESS}{DATASETS_API_URL}/{dataset_id}/index"
+    all_params = {"type": index_type}
+    if params:
+        all_params.update(params)
+    res = requests.get(url=url, headers=headers, auth=auth, params=all_params)
+    return res.json()
+
+
+def delete_index(auth, dataset_id, index_type, *, headers=HEADERS):
+    url = f"{HOST_ADDRESS}{DATASETS_API_URL}/{dataset_id}/{index_type}"
+    res = requests.delete(url=url, headers=headers, auth=auth)
+    return res.json()
+
+
+def run_embedding(auth, dataset_id, payload=None, *, headers=HEADERS):
+    url = f"{HOST_ADDRESS}{DATASETS_API_URL}/{dataset_id}/embedding"
+    res = requests.post(url=url, headers=headers, auth=auth, json=payload)
+    return res.json()
+
+
+def list_tags(auth, dataset_id, *, headers=HEADERS):
+    url = f"{HOST_ADDRESS}{DATASETS_API_URL}/{dataset_id}/tags"
+    res = requests.get(url=url, headers=headers, auth=auth)
+    return res.json()
+
+
+def aggregate_tags(auth, dataset_ids, *, headers=HEADERS):
+    url = f"{HOST_ADDRESS}{DATASETS_API_URL}/tags/aggregation"
+    res = requests.get(url=url, headers=headers, auth=auth, params={"dataset_ids": ",".join(dataset_ids)})
+    return res.json()
+
+
+def delete_tags(auth, dataset_id, tags, *, headers=HEADERS):
+    url = f"{HOST_ADDRESS}{DATASETS_API_URL}/{dataset_id}/tags"
+    res = requests.delete(url=url, headers=headers, auth=auth, json={"tags": tags})
+    return res.json()
+
+
+def rename_tag(auth, dataset_id, from_tag, to_tag, *, headers=HEADERS):
+    url = f"{HOST_ADDRESS}{DATASETS_API_URL}/{dataset_id}/tags"
+    res = requests.put(url=url, headers=headers, auth=auth, json={"from_tag": from_tag, "to_tag": to_tag})
+    return res.json()
+
+
+def get_flattened_metadata(auth, dataset_ids, *, headers=HEADERS):
+    url = f"{HOST_ADDRESS}{DATASETS_API_URL}/metadata/flattened"
+    res = requests.get(url=url, headers=headers, auth=auth, params={"dataset_ids": ",".join(dataset_ids)})
     return res.json()