mirror of
https://github.com/infiniflow/ragflow.git
synced 2026-07-31 13:03:49 +08:00
### What problem does this PR solve? Closes #15330. RAGFlow had no connector for OneDrive / OneDrive for Business. Users who store working documents in OneDrive could not index them into a knowledge base without manually downloading and re-uploading files. This PR adds a net-new OneDrive data source that: - Authenticates against Microsoft Graph with the same MSAL client-credentials flow already used by the SharePoint and Teams connectors (no new auth primitives). - Enumerates every drive visible to the service principal and pages through `/drives/{id}/root/delta`, persisting `@odata.deltaLink` values per drive so subsequent syncs only fetch changed items. - Optionally narrows ingestion to a sub-folder (`folder_path`) without needing a separate code path. - Surfaces typed errors on the validation probe (`GET /drives?$top=1`): 401 → `ConnectorMissingCredentialError`, 403 → `InsufficientPermissionsError` (with a `Files.Read.All` hint), 5xx → `UnexpectedValidationError`. - Filters folders, soft-deleted items, and unsupported extensions (`.pdf .docx .doc .xlsx .xls .pptx .ppt .txt .md .csv`). #### Files | File | Change | |------|--------| | `common/data_source/onedrive_connector.py` | **New** — `OneDriveConnector` + `OneDriveCheckpoint`. | | `common/data_source/config.py` | `DocumentSource.ONEDRIVE = "onedrive"`. | | `common/constants.py` | `FileSource.ONEDRIVE = "onedrive"`. | | `common/data_source/__init__.py` | Export `OneDriveConnector`. | | `rag/svr/sync_data_source.py` | `OneDrive(SyncBase)` with `batch_size` normalisation; registered in `func_factory`. | | `web/src/pages/user-setting/data-source/constant/index.tsx` | `DataSourceKey.ONEDRIVE`, visibility map (`syncDeletedFiles: true`), info entry, form fields (tenant_id, client_id, client_secret, folder_path, batch_size), default values. | | `web/src/locales/en.ts`, `web/src/locales/zh.ts` | `onedriveDescription` + 4 tooltip keys (EN + ZH). | | `test/unit_test/data_source/test_onedrive_connector_unit.py` | **New** — 13 unit tests (`p1`/`p2`) covering auth, validation, checkpoint helpers, and document filtering. | #### Required Azure AD permission `Files.Read.All` (Application, admin-granted). #### Out of scope - Interactive end-user OAuth (delegated permissions) — the connector uses app-only credentials, consistent with the SharePoint / Teams precedent. - Binary download of file contents — the sync layer emits `Document`s carrying `webUrl` + metadata; bytes are hydrated downstream by the parse pipeline. ### Type of change - [x] New Feature (non-breaking change which adds functionality)
This commit is contained in:
6
web/src/assets/svg/data-source/onedrive.svg
Normal file
6
web/src/assets/svg/data-source/onedrive.svg
Normal file
@@ -0,0 +1,6 @@
|
||||
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 48 48" width="48" height="48">
|
||||
<path d="M28.5 22.5a7.5 7.5 0 0 0-14 0 6 6 0 0 0-5 4.4 6 6 0 0 0-3.5 5.6A6.5 6.5 0 0 0 12 39h26a7 7 0 0 0 1.4-13.8 7 7 0 0 0-10.9-2.7z" fill="#0364B8"/>
|
||||
<path d="M16.5 25.5a6 6 0 0 1 12-2 7 7 0 0 1 8.4 1 7 7 0 0 1 2.5 4.7 6 6 0 0 0-2.6-3.4 7 7 0 0 0-8.6-1A6 6 0 0 0 14 26a6 6 0 0 0-5.5 5.5 6.5 6.5 0 0 1 0-1c0-3 2.2-5.4 5.1-6 0-.2 0-.5.2-.7l2.7 1.7z" fill="#0078D4"/>
|
||||
<path d="M14.5 26a6 6 0 0 1 11.7-2 7 7 0 0 1 10.3 1.8A6 6 0 0 1 36.5 38H12a6 6 0 0 1-1-12 7 7 0 0 1 3.5 0z" fill="#1490DF"/>
|
||||
<path d="M12 28a4.5 4.5 0 0 0 0 9h25.5a5.5 5.5 0 0 0 .8-11 7 7 0 0 0-9.4-2.6A6 6 0 0 0 17 24a5 5 0 0 0-5 4z" fill="#28A8EA"/>
|
||||
</svg>
|
||||
|
After Width: | Height: | Size: 718 B |
@@ -1385,6 +1385,16 @@ Example: Virtual Hosted Style`,
|
||||
'Datetime/timestamp column for incremental sync. Only rows modified after the last sync will be fetched.',
|
||||
rest_apiDescription:
|
||||
'Connect any REST API endpoint as a data source using a flexible, configuration-driven connector.',
|
||||
onedriveDescription:
|
||||
'Connect OneDrive or OneDrive for Business to index files and folders via Microsoft Graph delta queries.',
|
||||
onedriveTenantIdTip:
|
||||
'Azure Active Directory tenant ID (Directory ID) of the Microsoft 365 organisation.',
|
||||
onedriveClientIdTip:
|
||||
'Application (client) ID of the Azure AD app registration with Files.Read.All permission.',
|
||||
onedriveClientSecretTip:
|
||||
'Client secret value generated in the Azure AD app registration.',
|
||||
onedriveFolderPathTip:
|
||||
'Optional sub-folder path to limit indexing (e.g. /Documents/Reports). Leave blank to index the entire drive.',
|
||||
restApiQueryParamsTip:
|
||||
'Key=value pairs (one per line) sent as URL query parameters. Use this instead of embedding params in the URL.',
|
||||
restApiHeadersTip:
|
||||
|
||||
@@ -1099,8 +1099,12 @@ NER:使用 spaCy NER 和基于规则的关键词提取来抽取实体和关系
|
||||
gmailTokenTip:
|
||||
'请上传由 Google Console 生成的 OAuth JSON。如果仅包含 client credentials,请通过浏览器授权一次以获取长期有效的刷新 Token。',
|
||||
dropboxDescription: '连接 Dropbox,同步指定账号下的文件与文件夹。',
|
||||
teamsDescription:
|
||||
'通过 Microsoft Graph 连接 Microsoft Teams,同步频道帖子与回复。',
|
||||
onedriveDescription: '连接 OneDrive 或 OneDrive for Business,通过 Microsoft Graph delta 查询索引文件和文件夹。',
|
||||
onedriveTenantIdTip: 'Microsoft 365 组织的 Azure Active Directory 租户 ID(目录 ID)。',
|
||||
onedriveClientIdTip: '拥有 Files.Read.All 权限的 Azure AD 应用注册的应用程序(客户端)ID。',
|
||||
onedriveClientSecretTip: '在 Azure AD 应用注册中生成的客户端密钥值。',
|
||||
onedriveFolderPathTip: '可选的子文件夹路径,用于限制索引范围(例如 /Documents/Reports)。留空则索引整个云盘。',
|
||||
teamsDescription: '通过 Microsoft Graph 连接 Microsoft Teams,同步频道帖子与回复。',
|
||||
teamsTenantIdTip:
|
||||
'Azure AD 租户 ID。需要具备 Team.ReadBasic.All 与 ChannelMessage.Read.All 应用权限(管理员同意)的应用。',
|
||||
slackDescription: '连接你的 Slack 工作区,同步频道消息与讨论串。',
|
||||
|
||||
@@ -43,6 +43,7 @@ export enum DataSourceKey {
|
||||
POSTGRESQL = 'postgresql',
|
||||
REST_API = 'rest_api',
|
||||
RSS = 'rss',
|
||||
ONEDRIVE = 'onedrive',
|
||||
TEAMS = 'teams',
|
||||
SLACK = 'slack',
|
||||
SHAREPOINT = 'sharepoint',
|
||||
@@ -129,6 +130,9 @@ export const DataSourceFeatureVisibilityMap: Partial<
|
||||
[DataSourceKey.MOODLE]: {
|
||||
syncDeletedFiles: true,
|
||||
},
|
||||
[DataSourceKey.ONEDRIVE]: {
|
||||
syncDeletedFiles: true,
|
||||
},
|
||||
[DataSourceKey.TEAMS]: {
|
||||
syncDeletedFiles: true,
|
||||
},
|
||||
@@ -317,6 +321,11 @@ export const generateDataSourceInfo = (t: TFunction) => {
|
||||
description: t(`setting.${DataSourceKey.POSTGRESQL}Description`),
|
||||
icon: <SvgIcon name={'data-source/postgresql'} width={38} />,
|
||||
},
|
||||
[DataSourceKey.ONEDRIVE]: {
|
||||
name: 'OneDrive',
|
||||
description: t(`setting.${DataSourceKey.ONEDRIVE}Description`),
|
||||
icon: <SvgIcon name={'data-source/onedrive'} width={38} />,
|
||||
},
|
||||
};
|
||||
};
|
||||
|
||||
@@ -403,6 +412,49 @@ export const getCommonExtraDefaultValues = () => ({
|
||||
});
|
||||
|
||||
export const DataSourceFormFields = {
|
||||
[DataSourceKey.ONEDRIVE]: [
|
||||
{
|
||||
label: 'Tenant ID',
|
||||
name: 'config.credentials.tenant_id',
|
||||
type: FormFieldType.Text,
|
||||
required: true,
|
||||
placeholder: 'xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx',
|
||||
tooltip: t('setting.onedriveTenantIdTip'),
|
||||
},
|
||||
{
|
||||
label: 'Client ID',
|
||||
name: 'config.credentials.client_id',
|
||||
type: FormFieldType.Text,
|
||||
required: true,
|
||||
placeholder: 'xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx',
|
||||
tooltip: t('setting.onedriveClientIdTip'),
|
||||
},
|
||||
{
|
||||
label: 'Client Secret',
|
||||
name: 'config.credentials.client_secret',
|
||||
type: FormFieldType.Password,
|
||||
required: true,
|
||||
tooltip: t('setting.onedriveClientSecretTip'),
|
||||
},
|
||||
{
|
||||
label: 'Folder Path (optional)',
|
||||
name: 'config.folder_path',
|
||||
type: FormFieldType.Text,
|
||||
required: false,
|
||||
placeholder: '/Documents/Reports',
|
||||
tooltip: t('setting.onedriveFolderPathTip'),
|
||||
},
|
||||
{
|
||||
label: 'Batch Size',
|
||||
name: 'config.batch_size',
|
||||
type: FormFieldType.Number,
|
||||
required: false,
|
||||
validation: {
|
||||
min: 1,
|
||||
message: 'Batch Size must be at least 1',
|
||||
},
|
||||
},
|
||||
],
|
||||
[DataSourceKey.RSS]: [
|
||||
{
|
||||
label: 'Feed URL',
|
||||
@@ -1843,6 +1895,19 @@ export const DataSourceFormDefaultValues = {
|
||||
},
|
||||
},
|
||||
},
|
||||
[DataSourceKey.ONEDRIVE]: {
|
||||
name: '',
|
||||
source: DataSourceKey.ONEDRIVE,
|
||||
config: {
|
||||
folder_path: '',
|
||||
batch_size: 2,
|
||||
credentials: {
|
||||
tenant_id: '',
|
||||
client_id: '',
|
||||
client_secret: '',
|
||||
},
|
||||
},
|
||||
},
|
||||
[DataSourceKey.REST_API]: {
|
||||
name: '',
|
||||
source: DataSourceKey.REST_API,
|
||||
|
||||
Reference in New Issue
Block a user