zlei6/ragflow
1
0
Fork 0
mirror of https://github.com/infiniflow/ragflow.git synced 2026-06-29 15:31:05 +08:00
Code Issues Packages Projects Releases Wiki Activity

feat(api): add unified index API and dataset management endpoints (#14222)

Browse Source 
### What problem does this PR solve?

## Summary

Refactor the dataset API layer into a clean service/REST separation
pattern, add a unified `/index` API for graph/raptor/mindmap operations,
and introduce several new dataset management endpoints with full test
coverage.

## Changes

### Service Layer (`dataset_api_service.py`)

- Added `trace_index(dataset_id, tenant_id, index_type)` — unified trace
function for all index types
- Added `run_index`, `delete_index` service functions
- Added `get_dataset`, `get_ingestion_summary`, `list_ingestion_logs`,
`get_ingestion_log`
- Added `run_embedding`, `list_tags`, `aggregate_tags`, `delete_tags`,
`rename_tag`
- Added `get_flattened_metadata`, `get_auto_metadata`,
`update_auto_metadata`

### REST API Layer (`dataset_api.py`)

**New unified routes:**

| Method | Route | Description |
|--------|-------|-------------|
| POST | `/datasets/<id>/index?type=graph\|raptor\|mindmap` | Run index
task |
| GET | `/datasets/<id>/index?type=graph\|raptor\|mindmap` | Trace index
task |
| DELETE | `/datasets/<id>/<index_type>` | Delete index |
| GET | `/datasets/<id>` | Get dataset details |
| GET | `/datasets/<id>/ingestions/summary` | Ingestion summary |
| GET | `/datasets/<id>/ingestions` | List ingestion logs |
| GET | `/datasets/<id>/ingestions/<log_id>` | Get single ingestion log
|
| POST | `/datasets/<id>/embedding` | Run embedding |
| GET | `/datasets/<id>/tags` | List tags |
| GET | `/datasets/tags/aggregation` | Aggregate tags across datasets |
| DELETE | `/datasets/<id>/tags` | Delete tags |
| PUT | `/datasets/<id>/tags` | Rename tag |
| GET | `/datasets/metadata/flattened` | Get flattened metadata |
| GET/PUT | `/datasets/<id>/metadata/config` | New metadata config path
|

**Removed routes (replaced by unified `/index`):**

- `POST /datasets/<id>/mindmap`
- `GET /datasets/<id>/mindmap`

**Preserved legacy routes (backward compatibility):**

- `/run_graphrag`, `/trace_graphrag`, `/run_raptor`, `/trace_raptor`
- `/auto_metadata` GET/PUT

### Test Suite

- Updated `common.py` helpers: added `trace_index`, removed
`run_mindmap`/`trace_mindmap`
- Added 7 new test files with 39 test cases total:

| Test File | Cases |
|-----------|-------|
| `test_get_dataset.py` | 4 |
| `test_ingestion_summary.py` | 2 |
| `test_ingestion_logs.py` | 5 |
| `test_index_api.py` | 14 |
| `test_embedding.py` | 2 |
| `test_tags.py` | 8 |
| `test_flattened_metadata.py` | 4 |

- Deleted `test_mindmap_tasks.py` (covered by unified index tests)

## Design Decisions

1. **Unified `/index?type=...`** — single endpoint replaces 3 separate
route pairs for graph/raptor/mindmap
2. **Backward compatibility** — old routes (`/run_graphrag`,
`/run_raptor`, `/auto_metadata`) preserved alongside new paths
3. **`_VALID_INDEX_TYPES = {"graph", "raptor", "mindmap"}`** — input
validation via constant set
4. **`_INDEX_TYPE_TO_TASK_ID_FIELD`** — maps index type to KB model task
ID field for clean dispatch

## Files Changed

- `api/apps/restful_apis/dataset_api.py`
- `api/apps/services/dataset_api_service.py`
- `sdk/python/ragflow_sdk/modules/dataset.py`
- `test/testcases/test_http_api/common.py`
- `test/testcases/test_http_api/test_dataset_management/` (7 new files)
### Type of change

- [x] New Feature (non-breaking change which adds functionality)
- [x] Refactoring

---------

Signed-off-by: noob <yixiao121314@outlook.com>
This commit is contained in:
euvre
2026-04-27 01:38:01 +00:00
committed by GitHub
parent fb95136f39
commit 4dcc42e0e1
51 changed files with 1765 additions and 4381 deletions
Download Patch File Download Diff File Expand all files Collapse all files

4
sdk/python/ragflow_sdk/modules/dataset.py
View File

@@ -165,7 +165,7 @@ class DataSet(Base):
"""
Retrieve auto-metadata configuration for a dataset via SDK.
"""
res = self.get(f"/datasets/{self.id}/auto_metadata")
res = self.get(f"/datasets/{self.id}/metadata/config")
res = res.json()
if res.get("code") == 0:
return res["data"]
@@ -175,7 +175,7 @@ class DataSet(Base):
"""
Update auto-metadata configuration for a dataset via SDK.
"""
res = self.put(f"/datasets/{self.id}/auto_metadata", config)
res = self.put(f"/datasets/{self.id}/metadata/config", config)
res = res.json()
if res.get("code") == 0:
return res["data"]

33
sdk/python/test/test_frontend_api/common.py
View File

@@ -19,38 +19,33 @@ import os
import requests
HOST_ADDRESS = os.getenv("HOST_ADDRESS", "http://127.0.0.1:9380")
API_VERSION = "v1"
DATASETS_API_URL = f"/api/{API_VERSION}/datasets"
DATASET_NAME_LIMIT = 128
def create_dataset(auth, dataset_name):
authorization = {"Authorization": auth}
url = f"{HOST_ADDRESS}/v1/kb/create"
json = {"name": dataset_name}
res = requests.post(url=url, headers=authorization, json=json)
def create_dataset(auth, payload=None):
url = f"{HOST_ADDRESS}{DATASETS_API_URL}"
res = requests.post(url=url, headers={"Content-Type": "application/json"}, auth=auth, json=payload)
return res.json()
def list_dataset(auth, page_number, page_size=30):
authorization = {"Authorization": auth}
url = f"{HOST_ADDRESS}/v1/kb/list?page={page_number}&page_size={page_size}"
json = {}
res = requests.post(url=url, headers=authorization, json=json)
def list_dataset(auth, params=None):
url = f"{HOST_ADDRESS}{DATASETS_API_URL}"
res = requests.get(url=url, headers={"Content-Type": "application/json"}, auth=auth, params=params)
return res.json()
def rm_dataset(auth, dataset_id):
authorization = {"Authorization": auth}
url = f"{HOST_ADDRESS}/v1/kb/rm"
json = {"kb_id": dataset_id}
res = requests.post(url=url, headers=authorization, json=json)
def rm_dataset(auth, dataset_ids):
url = f"{HOST_ADDRESS}{DATASETS_API_URL}"
res = requests.delete(url=url, headers={"Content-Type": "application/json"}, auth=auth, json={"ids": dataset_ids})
return res.json()
def update_dataset(auth, json_req):
authorization = {"Authorization": auth}
url = f"{HOST_ADDRESS}/v1/kb/update"
res = requests.post(url=url, headers=authorization, json=json_req)
def update_dataset(auth, dataset_id, payload=None):
url = f"{HOST_ADDRESS}{DATASETS_API_URL}/{dataset_id}"
res = requests.put(url=url, headers={"Content-Type": "application/json"}, auth=auth, json=payload)
return res.json()

12
sdk/python/test/test_frontend_api/test_chunk.py
View File

@@ -21,7 +21,7 @@ from timeit import default_timer as timer
def test_parse_txt_document(get_auth):
# create dataset
res = create_dataset(get_auth, "test_parse_txt_document")
res = create_dataset(get_auth, {"name": "test_parse_txt_document"})
assert res.get("code") == 0, f"{res.get('message')}"
# list dataset
@@ -29,8 +29,10 @@ def test_parse_txt_document(get_auth):
dataset_list = []
dataset_id = None
while True:
res = list_dataset(get_auth, page_number)
data = res.get("data").get("kbs")
res = list_dataset(get_auth, {"page": page_number, "page_size": 150})
data = res.get("data")
if isinstance(data, dict):
data = data.get("kbs", [])
for item in data:
dataset_id = item.get("id")
dataset_list.append(dataset_id)
@@ -66,7 +68,7 @@ def test_parse_txt_document(get_auth):
print('time cost {:.1f}s'.format(timer() - start_ts))
# delete dataset
for dataset_id in dataset_list:
res = rm_dataset(get_auth, dataset_id)
if dataset_list:
res = rm_dataset(get_auth, dataset_list)
assert res.get("code") == 0, f"{res.get('message')}"
print(f"{len(dataset_list)} datasets are deleted")

81
sdk/python/test/test_frontend_api/test_dataset.py
View File

@@ -22,15 +22,17 @@ import string
def test_dataset(get_auth):
# create dataset
res = create_dataset(get_auth, "test_create_dataset")
res = create_dataset(get_auth, {"name": "test_create_dataset"})
assert res.get("code") == 0, f"{res.get('message')}"
# list dataset
page_number = 1
dataset_list = []
while True:
res = list_dataset(get_auth, page_number)
data = res.get("data").get("kbs")
res = list_dataset(get_auth, {"page": page_number, "page_size": 150})
data = res.get("data")
if isinstance(data, dict):
data = data.get("kbs", [])
for item in data:
dataset_id = item.get("id")
dataset_list.append(dataset_id)
@@ -40,8 +42,8 @@ def test_dataset(get_auth):
print(f"found {len(dataset_list)} datasets")
# delete dataset
for dataset_id in dataset_list:
res = rm_dataset(get_auth, dataset_id)
if dataset_list:
res = rm_dataset(get_auth, dataset_list)
assert res.get("code") == 0, f"{res.get('message')}"
print(f"{len(dataset_list)} datasets are deleted")
@@ -49,15 +51,17 @@ def test_dataset(get_auth):
def test_dataset_1k_dataset(get_auth):
# create dataset
for i in range(1000):
res = create_dataset(get_auth, f"test_create_dataset_{i}")
res = create_dataset(get_auth, {"name": f"test_create_dataset_{i}"})
assert res.get("code") == 0, f"{res.get('message')}"
# list dataset
page_number = 1
dataset_list = []
while True:
res = list_dataset(get_auth, page_number)
data = res.get("data").get("kbs")
res = list_dataset(get_auth, {"page": page_number, "page_size": 150})
data = res.get("data")
if isinstance(data, dict):
data = data.get("kbs", [])
for item in data:
dataset_id = item.get("id")
dataset_list.append(dataset_id)
@@ -67,8 +71,8 @@ def test_dataset_1k_dataset(get_auth):
print(f"found {len(dataset_list)} datasets")
# delete dataset
for dataset_id in dataset_list:
res = rm_dataset(get_auth, dataset_id)
if dataset_list:
res = rm_dataset(get_auth, dataset_list)
assert res.get("code") == 0, f"{res.get('message')}"
print(f"{len(dataset_list)} datasets are deleted")
@@ -76,12 +80,14 @@ def test_dataset_1k_dataset(get_auth):
def test_duplicated_name_dataset(get_auth):
# create dataset
for i in range(20):
res = create_dataset(get_auth, "test_create_dataset")
res = create_dataset(get_auth, {"name": "test_create_dataset"})
assert res.get("code") == 0, f"{res.get('message')}"
# list dataset
res = list_dataset(get_auth, 1)
data = res.get("data").get("kbs")
res = list_dataset(get_auth, {"page": 1})
data = res.get("data")
if isinstance(data, dict):
data = data.get("kbs", [])
dataset_list = []
pattern = r'^test_create_dataset.*'
for item in data:
@@ -91,19 +97,18 @@ def test_duplicated_name_dataset(get_auth):
match = re.match(pattern, dataset_name)
assert match is not None
for dataset_id in dataset_list:
res = rm_dataset(get_auth, dataset_id)
if dataset_list:
res = rm_dataset(get_auth, dataset_list)
assert res.get("code") == 0, f"{res.get('message')}"
print(f"{len(dataset_list)} datasets are deleted")
def test_invalid_name_dataset(get_auth):
# create dataset
# with pytest.raises(Exception) as e:
res = create_dataset(get_auth, 0)
res = create_dataset(get_auth, {"name": 0})
assert res['code'] != 0
res = create_dataset(get_auth, "")
res = create_dataset(get_auth, {"name": ""})
assert res['code'] != 0
long_string = ""
@@ -111,22 +116,24 @@ def test_invalid_name_dataset(get_auth):
while len(long_string.encode("utf-8")) <= DATASET_NAME_LIMIT:
long_string += random.choice(string.ascii_letters + string.digits)
res = create_dataset(get_auth, long_string)
res = create_dataset(get_auth, {"name": long_string})
assert res['code'] != 0
print(res)
def test_update_different_params_dataset_success(get_auth):
# create dataset
res = create_dataset(get_auth, "test_create_dataset")
res = create_dataset(get_auth, {"name": "test_create_dataset"})
assert res.get("code") == 0, f"{res.get('message')}"
# list dataset
page_number = 1
dataset_list = []
while True:
res = list_dataset(get_auth, page_number)
data = res.get("data").get("kbs")
res = list_dataset(get_auth, {"page": page_number, "page_size": 150})
data = res.get("data")
if isinstance(data, dict):
data = data.get("kbs", [])
for item in data:
dataset_id = item.get("id")
dataset_list.append(dataset_id)
@@ -137,15 +144,18 @@ def test_update_different_params_dataset_success(get_auth):
print(f"found {len(dataset_list)} datasets")
dataset_id = dataset_list[0]
json_req = {"kb_id": dataset_id, "name": "test_update_dataset", "description": "test", "permission": "me",
"parser_id": "presentation",
"language": "spanish"}
res = update_dataset(get_auth, json_req)
res = update_dataset(get_auth, dataset_id, {
"name": "test_update_dataset",
"description": "test",
"permission": "me",
"chunk_method": "presentation",
"language": "spanish",
})
assert res.get("code") == 0, f"{res.get('message')}"
# delete dataset
for dataset_id in dataset_list:
res = rm_dataset(get_auth, dataset_id)
if dataset_list:
res = rm_dataset(get_auth, dataset_list)
assert res.get("code") == 0, f"{res.get('message')}"
print(f"{len(dataset_list)} datasets are deleted")
@@ -153,15 +163,17 @@ def test_update_different_params_dataset_success(get_auth):
# update dataset with different parameters
def test_update_different_params_dataset_fail(get_auth):
# create dataset
res = create_dataset(get_auth, "test_create_dataset")
res = create_dataset(get_auth, {"name": "test_create_dataset"})
assert res.get("code") == 0, f"{res.get('message')}"
# list dataset
page_number = 1
dataset_list = []
while True:
res = list_dataset(get_auth, page_number)
data = res.get("data").get("kbs")
res = list_dataset(get_auth, {"page": page_number, "page_size": 150})
data = res.get("data")
if isinstance(data, dict):
data = data.get("kbs", [])
for item in data:
dataset_id = item.get("id")
dataset_list.append(dataset_id)
@@ -172,12 +184,11 @@ def test_update_different_params_dataset_fail(get_auth):
print(f"found {len(dataset_list)} datasets")
dataset_id = dataset_list[0]
json_req = {"kb_id": dataset_id, "id": "xxx"}
res = update_dataset(get_auth, json_req)
res = update_dataset(get_auth, dataset_id, {"id": "xxx"})
assert res.get("code") == 101
# delete dataset
for dataset_id in dataset_list:
res = rm_dataset(get_auth, dataset_id)
if dataset_list:
res = rm_dataset(get_auth, dataset_list)
assert res.get("code") == 0, f"{res.get('message')}"
print(f"{len(dataset_list)} datasets are deleted")