Files
ragflow/web/src/services/data-source-service.ts

68 lines
2.2 KiB
TypeScript
Raw Normal View History

import api from '@/utils/api';
import registerServer from '@/utils/register-server';
import request from '@/utils/request';
const { dataSourceSet, dataSourceList } = api;
const methods = {
dataSourceSet: {
url: dataSourceSet,
method: 'post',
},
dataSourceList: {
url: dataSourceList,
method: 'get',
},
} as const;
const dataSourceService = registerServer<keyof typeof methods>(
methods,
request,
);
export const deleteDataSource = (id: string) =>
request.delete(api.dataSourceDel(id));
export const dataSourceRebuild = (id: string, data: { kb_id: string }) => {
return request.post(api.dataSourceRebuild(id), { data });
};
Feat: full optimization on connector dashboard (#14979) ### What problem does this PR solve? This PR improves the connector dashboard task management experience and adds better visibility into connector execution logs. ### Overview: #### Before <img width="700" alt="image" src="https://github.com/user-attachments/assets/e4a8ed6f-2e18-4f0f-8528-41a514550052" /> #### Now: <img width="700" alt="Screenshot from 2026-05-18 16-31-30" src="https://github.com/user-attachments/assets/d4ca193b-847a-49ae-9e4f-5fbca60ea627" /> ### 1. Add a new logging page to the connector dashboard A new logging page has been added so users can view connector task execution logs directly from the connector dashboard. ### 2. Merge the Resume button into Confirm The separate **Resume** button has been removed. The **Confirm** button now represents different actions depending on the current task state: - **Save**: Save form changes and reschedule tasks. - **Stop**: Cancel currently scheduled or running tasks. - **Resume**: Create new scheduled tasks after the previous tasks have been stopped. - **Start**: Start tasks when no task has been started yet. ### 3. Separate syncing and pruning tasks Connector tasks are now separated into **syncing** and **pruning**. Pruning is controlled by the **Sync deleted files** option: - When **Sync deleted files** is disabled, only syncing tasks are shown. - When **Sync deleted files** is enabled, both syncing and pruning tasks are shown. **Now: Sync deleted files disabled** <img width="700" alt="Sync deleted files disabled" src="https://github.com/user-attachments/assets/dbd9232e-614a-407f-a0b1-c109e5fa567d" /> **Now: Sync deleted files enabled** <img width="700" alt="Sync deleted files enabled" src="https://github.com/user-attachments/assets/1f527f48-ccb3-4ee8-97ca-086891489296" /> ### 4. Update logs in backend <img width="700" alt="image" src="https://github.com/user-attachments/assets/10a95a3f-98c1-4e67-8afa-ddf6cda5b0b2" /> ### 5. Remove connector resume API - Removed: `POST /v1/connectors/<connector_id>/resume` - Replaced by: `PATCH /v1/connectors/<connector_id>` ### Type of change - [x] New Feature (non-breaking change which adds functionality)
2026-05-19 10:07:11 +08:00
export const dataSourceUpdate = (id: string, data: Record<string, any>) => {
return request.patch(api.dataSourceUpdate(id), { data });
};
export const getDataSourceLogs = (id: string, params?: any) =>
request.get(api.dataSourceLogs(id), { params });
export const featchDataSourceDetail = (id: string) =>
request.get(api.dataSourceDetail(id));
Feature/generic api connector (#13545) # feat: Add Generic REST API Connector ## What problem does this PR solve? RAGFlow supports many specific data source connectors (MySQL, Slack, Google Drive, etc.), but there was no way to connect an arbitrary REST API as a data source. Users with custom or third-party APIs had to write a new connector class for each one. This PR adds a **generic, configuration-driven REST API connector** that lets users connect any REST API as a data source entirely through the UI — no code changes needed per API. --- ## Features ### Core Connector (`common/data_source/rest_api_connector.py`) - Implements `LoadConnector` and `PollConnector` interfaces for full and incremental sync - **Configurable authentication:** None, API Key (custom header), Bearer Token, Basic Auth - **Pluggable pagination:** Page-based, Offset-based, Cursor-based, or None - Smart page-size inference from user's query parameters to avoid duplicate/conflicting params - Configurable request delay between pages to prevent API rate limiting - Auto-detection of the items array in JSON responses (`items`, `results`, `data`, `records`, or first list found) - **Advanced field mapping** with dot-notation (`country.name`), array wildcards (`newsType[*].name`), type hints, and default values - Optional content template rendering (`"Title: {title}\nBody: {body}"`) - HTML stripping for content fields - Stable document IDs via `hash128` from a configurable ID field or auto-generated from item content - Pydantic configuration schema with automatic coercion of UI string inputs to dicts/lists ### Backend Registration (`rag/svr/sync_data_source.py`, `common/constants.py`, `common/data_source/config.py`) - `REST_API` sync class wired into RAGFlow's `func_factory` - Full sync (`load_from_state`) and incremental polling (`poll_source`) support - Credentials and config passed from task to connector following existing patterns (MySQL, SeaFile, etc.) ### Test Connection Endpoint (`api/apps/connector_app.py`) - `POST /v1/connector/<id>/test` validates config schema, authentication, and API connectivity without triggering a sync - Clear error messages for auth failures vs. config issues ### Frontend UI (`web/src/pages/user-setting/data-source/constant/`) - **Postman-style configuration:** Base URL, Query Parameters (key=value per line), Auth, Content Fields, Metadata Fields, Pagination Type - Auth-type-aware form: fields for API key header/value, Bearer token, or Basic username/password appear only when relevant - **Advanced Settings** toggle for: Custom Headers, Max Pages, Request Delay, Poll Timestamp Field, Request Body (POST) - Connector icon (SVG) and i18n strings (English) - **"Test Connection"** button to validate before syncing --- ## Controls & Safety - Configurable max pages safety cap (default: 1000, adjustable in UI) - Configurable request delay between pages (default: 0.5s, adjustable in UI) - Auth errors (401/403) fail immediately without retries; transient errors retry with exponential backoff - Diagnostic logging: auth setup confirmation, request details on failure, content field extraction status --- ## Type of change - [x] New Feature (non-breaking change which adds functionality) ##Visual Screenshots of Features <img width="482" height="510" alt="Screenshot 2026-03-11 at 5 19 52 PM" src="https://github.com/user-attachments/assets/dcb7ab4a-1622-44f3-bb02-d6f0527314c4" /> (Connector can be configured within the external data sources tab) Configuration Parameters: <img width="661" height="682" alt="Screenshot 2026-03-11 at 5 20 46 PM" src="https://github.com/user-attachments/assets/5e154e71-4ab5-4872-bfb2-04f02b73c18a" /> <img width="661" height="682" alt="Screenshot 2026-03-11 at 5 20 54 PM" src="https://github.com/user-attachments/assets/00cb14b7-0bcf-4b94-9d71-34e93369ecb2" /> Connection can be tested before attaching to dataset: <img width="981" height="681" alt="Screenshot 2026-03-11 at 5 21 40 PM" src="https://github.com/user-attachments/assets/aaa6eeeb-89a7-4349-bc34-2423bf8be9ee" /> Ingestion tested with API connector (works perfectly fine): <img width="1062" height="705" alt="Screenshot 2026-03-11 at 5 22 30 PM" src="https://github.com/user-attachments/assets/afcd0d58-cadd-4152-badc-d2f14d96fbec" /> Search & Retrieval works as well with metadata flow: <img width="1062" height="705" alt="Screenshot 2026-03-11 at 5 23 05 PM" src="https://github.com/user-attachments/assets/d41ee935-dcf7-4456-b317-22a76ca032c0" /> --------- Co-authored-by: Ahmad Intisar <ahmadintisar@Ahmads-MacBook-M4-Pro.local> Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
2026-05-13 17:35:01 +05:00
export const testDataSource = (id: string) =>
request.post(api.dataSourceTest(id));
export const startGoogleDriveWebAuth = (payload: {
credentials: string;
redirect_uri?: string;
}) => request.post(api.googleWebAuthStart('google-drive'), { data: payload });
export const pollGoogleDriveWebAuthResult = (payload: { flow_id: string }) =>
request.post(api.googleWebAuthResult('google-drive'), { data: payload });
// Gmail web auth follows the same pattern as Google Drive, but uses
// Gmail-specific endpoints and is consumed by the GmailTokenField UI.
export const startGmailWebAuth = (payload: {
credentials: string;
redirect_uri?: string;
}) => request.post(api.googleWebAuthStart('gmail'), { data: payload });
export const pollGmailWebAuthResult = (payload: { flow_id: string }) =>
request.post(api.googleWebAuthResult('gmail'), { data: payload });
export const startBoxWebAuth = (payload: {
client_id: string;
client_secret: string;
redirect_uri?: string;
}) => request.post(api.boxWebAuthStart(), { data: payload });
export const pollBoxWebAuthResult = (payload: { flow_id: string }) =>
request.post(api.boxWebAuthResult(), { data: payload });
export default dataSourceService;