diff --git a/docs/guides/data_source/_category_.json b/docs/guides/data_source/_category_.json new file mode 100644 index 0000000000..7bc0a83335 --- /dev/null +++ b/docs/guides/data_source/_category_.json @@ -0,0 +1,11 @@ +{ + "label": "Data Sources", + "position": 0.5, + "link": { + "type": "generated-index", + "description": "Guides on adding and configuring data sources." + }, + "customProps": { + "categoryIcon": "LucideServer" + } +} diff --git a/docs/guides/data_source/add_to_knowledge_base_and_sync.md b/docs/guides/data_source/add_to_knowledge_base_and_sync.md new file mode 100644 index 0000000000..d597fe53b4 --- /dev/null +++ b/docs/guides/data_source/add_to_knowledge_base_and_sync.md @@ -0,0 +1,37 @@ +--- +sidebar_position: 2 +title: Add a Data Source to a Knowledge Base and Synchronization and Updates +sidebar_label: Add and Sync +slug: /add_data_source/add_to_knowledge_base_and_sync +sidebar_custom_props: { + categoryIcon: LucideServer +} +--- + +# Add a Data Source to a Knowledge Base and Synchronization and Updates + +## Add a Data Source to a Knowledge Base + +Creating a data source connection only saves the access configuration for the external system. To bring external content into a knowledge base, you also need to add that data source in the target knowledge base and start synchronization. + +1. Enter the knowledge base page. +2. Open the target knowledge base that needs to synchronize external content. +3. In the data import or data source settings entry of the knowledge base, select the created data source connection. +4. Follow the prompts on the knowledge base page to select the synchronization scope, parsing method, or chunking parameters. +5. Save and start synchronization, then wait for the synchronization task to complete. + +## Synchronization and Updates + +After synchronization starts, RAGFlow reads content from the external system and writes parseable content to the knowledge base index. + +After synchronization is complete, users can retrieve and ask questions about this content in the knowledge base. Synchronization results can be understood according to the following rules: + +- The first synchronization imports the existing content in the current synchronization scope. +- Content newly added or modified in the external system is usually synchronized to the knowledge base when the next refresh interval arrives. +- If sync deleted files is enabled, content deleted from the external system is removed from the knowledge base index during subsequent synchronization or cleanup tasks. + +If changes to external content do not appear in the knowledge base immediately, first check the refresh interval, synchronization status, and data source logs. + +Some data sources support the test connection feature. After saving or modifying a connection, you can test the connection first to confirm that account permissions, connection parameters, and data access are normal, and then start synchronization in the knowledge base. + +![Synchronization and Updates](https://raw.githubusercontent.com/infiniflow/ragflow-docs/2ee87008723d56cb6ebf0e9c92f6ef2ad1a45254/images/Synchronization_and_Updates.jpg) diff --git a/docs/guides/data_source/data_source_categories_and_selection.md b/docs/guides/data_source/data_source_categories_and_selection.md new file mode 100644 index 0000000000..dbdeba79a0 --- /dev/null +++ b/docs/guides/data_source/data_source_categories_and_selection.md @@ -0,0 +1,29 @@ +--- +sidebar_position: 3 +title: Data Source Categories and Selection +sidebar_label: Categories and Selection +slug: /add_data_source/data_source_categories_and_selection +sidebar_custom_props: { + categoryIcon: LucideServer +} +--- + +# Data Source Categories and Selection + +RAGFlow supports many data sources. It is recommended to first select a major category according to the content source, and then go to the corresponding data source configuration. The following categories help users quickly determine which connection method to choose. + +**Documents and collaboration platforms**: Used to synchronize team documents, wikis, cloud drive files, and collaborative documents. This category is suitable for building enterprise knowledge bases or project repositories. It includes Confluence, Notion, Google Drive, OneDrive, SharePoint, Box, Dropbox, WebDAV, and SeaFile. + +**Object storage**: Used to synchronize large numbers of files in buckets or containers. This category is suitable for historical data archiving, batch file import, and centralized management of cloud files. It includes S3, Google Cloud Storage, Oracle Storage, R2, and Azure Blob Storage. + +**Databases and data warehouses**: Used to synchronize structured records or query results. This category is suitable for business data Q&A, metric knowledge retention, and retrieval of data analysis results. It includes MySQL, PostgreSQL, and BigQuery. + +**Code repositories and project management**: Used to synchronize code collaboration, issues, pull requests, tasks, and project records. This category is suitable for technical knowledge retention and project history queries. It includes GitHub, GitLab, Bitbucket, Jira, and Asana. + +**Email, messages, and collaborative communication**: Used to synchronize emails, channel messages, and team discussion content. This category is suitable for customer service communication, business correspondence, and collaborative record queries. It includes Gmail, Outlook, IMAP, Microsoft Teams, Slack, and Discord. + +**Business systems and tables**: Used to synchronize CRM, tickets, online tables, courses, or operation records. This category is suitable for building business ledgers and service knowledge bases. It includes Salesforce, Airtable, Dingtalk AI Table, Zendesk, and Moodle. + +**General APIs and subscriptions**: Used to connect custom APIs or public subscription feeds. This category is suitable for business systems without standard connectors and for collecting public content. It includes REST API and RSS. + +Account version note: In this manual, **Personal Edition** and **Enterprise Edition** refer to the external system account or service version, not the RAGFlow license. Whether synchronization is available ultimately depends on whether the external system allows the current account to access the target content and API. diff --git a/docs/guides/data_source/data_source_configuration.md b/docs/guides/data_source/data_source_configuration.md new file mode 100644 index 0000000000..e1d57b5e8d --- /dev/null +++ b/docs/guides/data_source/data_source_configuration.md @@ -0,0 +1,654 @@ +--- +sidebar_position: 4 +title: Data Source Configuration +sidebar_label: Configuration +slug: /add_data_source/data_source_configuration +sidebar_custom_props: { + categoryIcon: LucideServer +} +--- + +# Data Source Configuration +## Confluence + +The Confluence data source is used to synchronize spaces, pages, and subpages maintained by a team in Confluence to a RAGFlow knowledge base. After configuration, users can query product documents, project descriptions, R&D standards, and other content in the knowledge base. + +**Permission requirements**: Read permissions for Confluence spaces or pages. If attachments or subpages need to be synchronized, read permissions for the corresponding content are also required. + +**Account version requirements**: Personal spaces or free team spaces in Confluence Cloud can be used. If enterprise spaces, Server, or Data Center content is synchronized, access permissions for the corresponding organization space are required. + +**Configuration parameters**: + +- **Name**: Customize the name in RAGFlow to identify this Confluence connection. +- **Confluence username**: Fill in the account used to access Confluence. +- **Confluence access token**: Fill in the access token used to access Confluence. +- **Wiki base URL**: Fill in the base address of the Confluence site. +- **Is cloud version**: Select according to the Confluence deployment type. +- **Index mode**: Select the synchronization scope. The options are **All**, **Space**, and **Page**. +- **Page ID**: Fill this in when the index mode is **Page**. +- **Space key**: Fill this in when the index mode is **Space**. +- **Recursive indexing**: Available when the index mode is **Page**. After it is enabled, the page and its subpages are synchronized. +- **Sync deleted files**: After this is enabled, content deleted from the external system is removed from the knowledge base index. + +![Confluence](https://raw.githubusercontent.com/infiniflow/ragflow-docs/2ee87008723d56cb6ebf0e9c92f6ef2ad1a45254/images/Confluence.jpg) + +## Notion + +The Notion data source is used to synchronize Notion pages and database content to a RAGFlow knowledge base. After configuration, documents, project records, and knowledge cards organized by the team in Notion can be used for retrieval and Q&A. + +**Permission requirements**: The Notion integration must be granted access to the target page or database. + +**Account version requirements**: Both Notion personal pages and team spaces can be used. As long as the target page or database has been shared with the Notion integration, it can be synchronized to RAGFlow. + +**Configuration parameters**: + +- **Name**: Customize the name in RAGFlow to identify this Notion connection. +- **Notion integration token**: Fill in the integration token used to access Notion. +- **Root page ID**: Fill in the root page ID that needs to be synchronized. +- **Sync deleted files**: After this is enabled, content deleted from the external system is removed from the knowledge base index. + +![Notion](https://raw.githubusercontent.com/infiniflow/ragflow-docs/2ee87008723d56cb6ebf0e9c92f6ef2ad1a45254/images/Notion.jpg) + +## Google Drive + +The Google Drive data source is used to synchronize files or folders in Google Drive to a RAGFlow knowledge base. After configuration, team cloud documents can be used directly for knowledge base retrieval and Q&A. + +**Permission requirements**: The Google account or OAuth authorization must have read permissions for the target files, shared folders, or cloud drives. + +**Account version requirements**: Folders that a user is authorized to access in a personal Google Drive can be used. To synchronize organization shared folders or multi-user cloud drives, a Google Workspace administrator or an account with the corresponding authorization is usually required. + +**Configuration parameters**: + +- **Name**: Customize the name in RAGFlow to identify this Google Drive connection. +- **Primary administrator email**: Fill in the administrator email that has access to the target Drive content. +- **OAuth token JSON**: Fill in or upload the OAuth token JSON used to authorize Google Drive. +- **My Drive email**: Fill in the email address of the user whose **My Drive** needs to be indexed. +- **Shared folder URL**: Fill in the Google Drive shared folder URL that needs to be synchronized. + +![Google Drive](https://raw.githubusercontent.com/infiniflow/ragflow-docs/2ee87008723d56cb6ebf0e9c92f6ef2ad1a45254/images/Google_Drive.jpg) + +## OneDrive + +The OneDrive data source is used to synchronize files in OneDrive or OneDrive for Business to a RAGFlow knowledge base. After configuration, personal or department cloud files can be queried in a unified way. + +**Permission requirements**: The Microsoft account or application authorization must have read permissions for the target folders and files. + +**Account version requirements**: This is usually configured for Microsoft 365 or Entra ID organization accounts. Personal OneDrive is generally not suitable for the current connection method based on tenant, client ID, and client secret. + +**Configuration parameters**: + +- **Name**: Customize the name in RAGFlow to identify this OneDrive connection. +- **Tenant ID**: Fill in the Microsoft Entra ID tenant ID. +- **Client ID**: Fill in the client ID from the app registration. +- **Client secret**: Fill in the client secret from the app registration. +- **Folder path (optional)**: Fill in the OneDrive subfolder path. +- **Batch size**: Set the number of items processed in each batch. +- **Sync deleted files**: After this is enabled, content deleted from the external system is removed from the knowledge base index. + +![OneDrive](https://raw.githubusercontent.com/infiniflow/ragflow-docs/2ee87008723d56cb6ebf0e9c92f6ef2ad1a45254/images/OneDrive.jpg) + +## SharePoint + +The SharePoint data source is used to synchronize SharePoint site document libraries to a RAGFlow knowledge base. After configuration, enterprise policies, department materials, and project files can be managed and queried in a unified way. + +**Permission requirements**: The Microsoft account or application authorization must have read permissions for the target site, document library, and files. + +**Account version requirements**: A Microsoft 365 or SharePoint organization site and Entra ID application authorization are required. Personal accounts are not applicable. + +**Configuration parameters**: + +- **Name**: Customize the name in RAGFlow to identify this SharePoint connection. +- **Site URL**: Fill in the complete URL of the SharePoint site. +- **Tenant ID**: Fill in the Microsoft Entra ID tenant ID. +- **Client ID**: Fill in the client ID from the app registration. +- **Client secret**: Fill in the client secret from the app registration. +- **Sync deleted files**: After this is enabled, content deleted from the external system is removed from the knowledge base index. + +![SharePoint](https://raw.githubusercontent.com/infiniflow/ragflow-docs/2ee87008723d56cb6ebf0e9c92f6ef2ad1a45254/images/SharePoint.jpg) + +## Box + +The Box data source is used to synchronize files and folders in Box cloud drive to a RAGFlow knowledge base. After configuration, enterprise cloud drive materials and externally collaborated files can be queried in a unified way. + +**Permission requirements**: The Box application or account must have read permissions for the target folders and files. + +**Account version requirements**: Both personal Box folders and enterprise Box spaces can be used. Enterprise spaces require the administrator or application authorization to allow access to the target folder. + +**Configuration parameters**: + +- **Name**: Customize the name in RAGFlow to identify this Box connection. +- **Box OAuth configuration**: Fill in the Box OAuth configuration information. +- **Folder ID**: Fill in the Box folder ID that needs to be synchronized. +- **Sync deleted files**: After this is enabled, content deleted from the external system is removed from the knowledge base index. + +![Box](https://raw.githubusercontent.com/infiniflow/ragflow-docs/2ee87008723d56cb6ebf0e9c92f6ef2ad1a45254/images/Box.jpg) + +## Dropbox + +The Dropbox data source is used to synchronize files and folders in Dropbox to a RAGFlow knowledge base. After configuration, cloud collaboration files can be used for knowledge base retrieval and Q&A. + +**Permission requirements**: The Dropbox application or account must have read permissions for the target folders and files. + +**Account version requirements**: Both personal Dropbox and team Dropbox can be used. Team spaces require application authorization with read permissions for the target files or folders. + +**Configuration parameters**: + +- **Name**: Customize the name in RAGFlow to identify this Dropbox connection. +- **Access token**: Fill in the Dropbox access token. +- **Batch size**: Set the number of files synchronized in each batch. +- **Sync deleted files**: After this is enabled, content deleted from the external system is removed from the knowledge base index. + +![Dropbox](https://raw.githubusercontent.com/infiniflow/ragflow-docs/2ee87008723d56cb6ebf0e9c92f6ef2ad1a45254/images/Dropbox.jpg) + +## SeaFile + +The SeaFile data source is used to synchronize account, library, or directory content in Seafile to a RAGFlow knowledge base. After configuration, private cloud drive materials and department files can be retrieved in a unified way. + +**Permission requirements**: The SeaFile account must have read permissions for the target libraries, directories, and files. + +**Account version requirements**: Both personal libraries and enterprise libraries can be used. When synchronizing shared libraries or specified directories, the corresponding library token or account permissions are required. + +**Configuration parameters**: + +- **Name**: Customize the name in RAGFlow to identify this SeaFile connection. +- **SeaFile server URL**: Fill in the SeaFile service address. +- **Synchronization scope**: Select the entire account, a single library, or a specified directory. +- **Account API token**: Fill in the account API token according to the synchronization scope. +- **Include shared libraries**: When synchronizing the entire account, select whether to include shared libraries. +- **Library token**: Fill this in when synchronizing a single library or specified directory. +- **Library ID**: Fill this in when synchronizing a single library or specified directory. +- **Directory path**: Fill this in when the synchronization scope is **Specified directory**. +- **Batch size**: Set the number of items processed in each batch. +- **Sync deleted files**: After this is enabled, content deleted from the external system is removed from the knowledge base index. + +![SeaFile](https://raw.githubusercontent.com/infiniflow/ragflow-docs/2ee87008723d56cb6ebf0e9c92f6ef2ad1a45254/images/SeaFile.jpg) + +## S3 + +The S3 data source is used to synchronize files in an object storage bucket to a RAGFlow knowledge base. After configuration, documents stored in the cloud or in private object storage can be uniformly included in knowledge base management. + +**Permission requirements**: The access key, IAM role, or compatible storage account must have bucket listing and object reading permissions. + +**Account version requirements**: An AWS S3 or S3-compatible object storage account is required. This type of data source is determined by cloud account and bucket permissions, and does not distinguish between personal and enterprise editions. + +**Configuration parameters**: + +- **Name**: Customize the name in RAGFlow to identify this S3 connection. +- **Bucket name**: Fill in the bucket name that needs to be synchronized. +- **Region**: Fill in or select the region where the bucket is located. +- **Prefix**: Used to limit the synchronization path. Leave it empty to synchronize the entire bucket. +- **Mode**: Select **S3** or **S3 Compatible**. +- **Authentication method**: In S3 mode, select access key, IAM role, or AssumeRole. +- **AWS Access Key ID**: Fill this in when selecting access key or S3-compatible mode. +- **AWS Secret Access Key**: Fill this in when selecting access key or S3-compatible mode. +- **Role ARN**: Fill this in when using an IAM role or AssumeRole. +- **Addressing style**: Select virtual-hosted style or path style in S3-compatible mode. +- **Endpoint URL**: Fill in the endpoint address in S3-compatible mode. +- **Sync deleted files**: After this is enabled, content deleted from the external system is removed from the knowledge base index. + +## Google Cloud Storage + +The Google Cloud Storage data source is used to synchronize files in a GCS bucket to a RAGFlow knowledge base. After configuration, project materials, reports, and documents on Google Cloud can enter the knowledge base for team retrieval. + +**Permission requirements**: The GCS credentials must have listing and object reading permissions for the target bucket. + +**Account version requirements**: Google Cloud project and bucket permissions are required. They are usually configured according to cloud project permissions, and do not distinguish between personal and enterprise editions. + +**Configuration parameters**: + +- **Name**: Customize the name in RAGFlow to identify this Google Cloud Storage connection. +- **GCS access key ID**: Fill in the GCS access key ID. +- **GCS access key**: Fill in the GCS access key. +- **Bucket name**: Fill in the bucket name that needs to be synchronized. +- **Sync deleted files**: After this is enabled, content deleted from the external system is removed from the knowledge base index. + +![Google Cloud Storage](https://raw.githubusercontent.com/infiniflow/ragflow-docs/2ee87008723d56cb6ebf0e9c92f6ef2ad1a45254/images/Google_Cloud_Storage.jpg) + +## Oracle Storage + +The Oracle Storage data source is used to synchronize files in Oracle Cloud Object Storage to a RAGFlow knowledge base. After configuration, documents and archive materials stored on OCI can be retrieved in a unified way. + +**Permission requirements**: The OCI credentials must have listing and object reading permissions for the target bucket. + +**Account version requirements**: Oracle Cloud tenancy and object storage permissions are required. They are usually configured according to cloud account permissions. + +**Configuration parameters**: + +- **Name**: Customize the name in RAGFlow to identify this Oracle Storage connection. +- **Namespace**: Fill in the OCI object storage namespace. +- **OCI region**: Fill in the OCI region. +- **OCI access key ID**: Fill in the OCI access key ID. +- **OCI access key**: Fill in the OCI access key. +- **Bucket name**: Fill in the bucket name that needs to be synchronized. +- **Sync deleted files**: After this is enabled, content deleted from the external system is removed from the knowledge base index. + +![Oracle Storage](https://raw.githubusercontent.com/infiniflow/ragflow-docs/2ee87008723d56cb6ebf0e9c92f6ef2ad1a45254/images/Oracle_Storage.jpg) + +## R2 + +The R2 data source is used to synchronize files in a Cloudflare R2 bucket to a RAGFlow knowledge base. After configuration, public materials, business files, or historical documents stored in R2 can be queried in a unified way. + +**Permission requirements**: The R2 access key must have listing and object reading permissions for the target bucket. + +**Account version requirements**: A Cloudflare account and R2 bucket permissions are required. Both personal and enterprise Cloudflare accounts can be used, depending on whether they have access to the target bucket. + +**Configuration parameters**: + +- **Name**: Customize the name in RAGFlow to identify this R2 connection. +- **R2 account ID**: Fill in the Cloudflare account ID. +- **R2 access key ID**: Fill in the R2 access key ID. +- **R2 access key**: Fill in the R2 access key. +- **Bucket name**: Fill in the bucket name that needs to be synchronized. +- **Sync deleted files**: After this is enabled, content deleted from the external system is removed from the knowledge base index. + +![R2](https://raw.githubusercontent.com/infiniflow/ragflow-docs/2ee87008723d56cb6ebf0e9c92f6ef2ad1a45254/images/R2.jpg) + +## Azure Blob Storage + +The Azure Blob Storage data source is used to synchronize files in an Azure Blob container to a RAGFlow knowledge base. After configuration, materials, reports, and archive files in an Azure cloud environment can be queried in a unified way. + +**Permission requirements**: The Azure credentials must have read permissions for the target container and blobs. + +**Account version requirements**: An Azure storage account, connection string, or SAS authorization is required. They are usually configured according to Azure subscription and storage container permissions. + +**Configuration parameters**: + +- **Name**: Customize the name in RAGFlow to identify this Azure Blob Storage connection. +- **Authentication mode**: Select account key, connection string, or SAS URL. +- **Account name**: Fill this in when using account key mode. +- **Account key**: Fill this in when using account key mode. +- **Connection string**: Fill this in when using connection string mode. +- **Container URL**: Fill this in when using SAS URL mode. +- **SAS token**: Fill this in when using SAS URL mode. +- **Container name**: Fill in the container name that needs to be synchronized. +- **Prefix (optional)**: Used to limit the synchronization path. +- **Batch size**: Set the number of files processed in each batch. +- **Sync deleted files**: After this is enabled, content deleted from the external system is removed from the knowledge base index. + +![Azure Blob Storage](https://raw.githubusercontent.com/infiniflow/ragflow-docs/2ee87008723d56cb6ebf0e9c92f6ef2ad1a45254/images/Azure_Blob_Storage.jpg) + +## MySQL + +The MySQL data source is used to synchronize table records or query results in MySQL to a RAGFlow knowledge base. After configuration, structured business data can be used for retrieval and Q&A. + +**Permission requirements**: The database account must have SELECT permission on the target database tables. A read-only account is recommended. + +**Account version requirements**: Both personal self-built databases and enterprise databases can be used. It is recommended to use a read-only account to connect to the target database tables. + +**Configuration parameters**: + +- **Name**: Customize the name in RAGFlow to identify this MySQL connection. +- **Host**: Fill in the database host address. +- **Port**: Fill in the database port. +- **Database**: Fill in the database name that needs to be synchronized. +- **Username**: Fill in the database username. +- **Password**: Fill in the database password. +- **SQL query**: Fill in the SQL statement used to read data. +- **Content column**: Select the field to be written as document content. +- **Metadata column**: Select the field to be written as metadata. +- **ID column**: Select the unique ID field. +- **Timestamp column**: Select the field used to determine incremental updates. +- **Sync deleted files**: After this is enabled, deleted records can be removed from the knowledge base index according to the field mapping or cleanup task. + +![MySQL](https://raw.githubusercontent.com/infiniflow/ragflow-docs/2ee87008723d56cb6ebf0e9c92f6ef2ad1a45254/images/MySQL.jpg) + +## PostgreSQL + +The PostgreSQL data source is used to synchronize table records or query results in PostgreSQL to a RAGFlow knowledge base. After configuration, structured business data can be used for retrieval and Q&A. + +**Permission requirements**: The database account must have SELECT permission on the target database tables. A read-only account is recommended. + +**Account version requirements**: Both personal self-built databases and enterprise databases can be used. It is recommended to use a read-only account to connect to the target database tables. + +**Configuration parameters**: The configuration fields are the same as those for MySQL, including name, host, port, database, username, password, SQL query, content column, metadata column, ID column, timestamp column, and sync deleted files. + +![PostgreSQL](https://raw.githubusercontent.com/infiniflow/ragflow-docs/2ee87008723d56cb6ebf0e9c92f6ef2ad1a45254/images/PostgreSQL.jpg) + +## BigQuery + +The BigQuery data source is used to synchronize table records or query results in BigQuery to a RAGFlow knowledge base. After configuration, structured data and analysis results in Google Cloud can be retrieved and used for Q&A. + +**Permission requirements**: The Google Cloud account or service account must have query and read permissions for the target BigQuery project, dataset, and table. + +**Account version requirements**: Google Cloud project and BigQuery permissions are required. They are configured according to cloud project permissions. + +**Configuration parameters**: + +- **Name**: Customize the name in RAGFlow to identify this BigQuery connection. +- **Service account JSON**: Fill in or upload the service account JSON used to access BigQuery. +- **Project ID**: Fill in the Google Cloud project ID. +- **Dataset**: Fill in the dataset name. +- **Table**: Fill in the table name. +- **SQL query**: Fill in the SQL statement used to read data. +- **Content column**: Select the field to be written as document content. +- **Metadata column**: Select the field to be written as metadata. +- **ID column**: Select the unique ID field. +- **Timestamp column**: Select the field used to determine incremental updates. +- **Sync deleted files**: After this is enabled, deleted records can be removed from the knowledge base index according to the field mapping or cleanup task. + +![BigQuery](https://raw.githubusercontent.com/infiniflow/ragflow-docs/2ee87008723d56cb6ebf0e9c92f6ef2ad1a45254/images/BigQuery.jpg) + +## GitHub + +The GitHub data source is used to synchronize issues and pull requests in GitHub repositories to a RAGFlow knowledge base. After configuration, open-source project materials, development discussions, and issue handling records can be queried. + +**Permission requirements**: The GitHub token must have read permissions for the target repositories, issues, and pull requests. + +**Account version requirements**: Both personal repositories and organization repositories can be used. Organization private repositories require the token to have read permissions for the corresponding repositories, issues, and pull requests. + +**Configuration parameters**: + +- **Name**: Customize the name in RAGFlow to identify this GitHub connection. +- **Repository owner**: Fill in the GitHub repository owner. +- **Repository name**: Fill in the GitHub repository name. +- **GitHub access token**: Fill in the GitHub access token. +- **Include Pull Requests**: Select whether to synchronize pull requests. +- **Include Issues**: Select whether to synchronize issues. +- **Sync deleted files**: After this is enabled, content deleted from the external system is removed from the knowledge base index. + +![GitHub](https://raw.githubusercontent.com/infiniflow/ragflow-docs/2ee87008723d56cb6ebf0e9c92f6ef2ad1a45254/images/GitHub.jpg) + +## GitLab + +The GitLab data source is used to synchronize GitLab repositories, issues, merge requests, and code files to a RAGFlow knowledge base. After configuration, project development records and code-related knowledge can be queried in a unified way. + +**Permission requirements**: The GitLab access token must have read permissions for the target project, issues, merge requests, and repository files. + +**Account version requirements**: Both personal projects and group projects can be used. Private group projects require the token to have the corresponding read permissions. + +**Configuration parameters**: + +- **Name**: Customize the name in RAGFlow to identify this GitLab connection. +- **GitLab instance URL**: Fill in the GitLab service address. +- **Project ID or path**: Fill in the GitLab project ID or project path. +- **Access token**: Fill in the GitLab access token. +- **Include merge requests**: Select whether to synchronize merge requests. +- **Include issues**: Select whether to synchronize issues. +- **Include repository files**: Select whether to synchronize repository files. +- **Sync deleted files**: After this is enabled, content deleted from the external system is removed from the knowledge base index. + +![GitLab](https://raw.githubusercontent.com/infiniflow/ragflow-docs/2ee87008723d56cb6ebf0e9c92f6ef2ad1a45254/images/GitLab.jpg) + +## Bitbucket + +The Bitbucket data source is used to synchronize Bitbucket repository content, issues, and pull requests to a RAGFlow knowledge base. After configuration, team code collaboration records can be queried. + +**Permission requirements**: The Bitbucket account, app password, or token must have read permissions for the target workspace and repository. + +**Account version requirements**: Both personal workspaces and team workspaces can be used. Private repositories require authorization with read permissions. + +**Configuration parameters**: + +- **Name**: Customize the name in RAGFlow to identify this Bitbucket connection. +- **Workspace**: Fill in the Bitbucket workspace name. +- **Repository slug**: Fill in the Bitbucket repository identifier. +- **Username**: Fill in the Bitbucket username. +- **App password or access token**: Fill in the credential used to access Bitbucket. +- **Include pull requests**: Select whether to synchronize pull requests. +- **Include issues**: Select whether to synchronize issues. +- **Sync deleted files**: After this is enabled, content deleted from the external system is removed from the knowledge base index. + +![Bitbucket](https://raw.githubusercontent.com/infiniflow/ragflow-docs/2ee87008723d56cb6ebf0e9c92f6ef2ad1a45254/images/Bitbucket.jpg) + +## Jira + +The Jira data source is used to synchronize issues, comments, and project records in Jira to a RAGFlow knowledge base. After configuration, project tasks, requirements, bugs, and handling records can be queried. + +**Permission requirements**: The Jira account or API token must have read permissions for the target project and issues. + +**Account version requirements**: Jira Cloud and Jira Server or Data Center can be used. Enterprise projects require the account to have access permissions for the corresponding project. + +**Configuration parameters**: + +- **Name**: Customize the name in RAGFlow to identify this Jira connection. +- **Jira base URL**: Fill in the Jira site address. +- **Email or username**: Fill in the Jira account. +- **API token or password**: Fill in the Jira API token or password. +- **Project key**: Fill in the Jira project key that needs to be synchronized. +- **JQL**: Fill in the JQL condition used to limit the synchronization scope. +- **Sync comments**: Select whether to synchronize issue comments. +- **Sync attachments**: Select whether to synchronize attachments. +- **Sync deleted files**: After this is enabled, content deleted from the external system is removed from the knowledge base index. + +![Jira](https://raw.githubusercontent.com/infiniflow/ragflow-docs/2ee87008723d56cb6ebf0e9c92f6ef2ad1a45254/images/Jira.jpg) + +## Asana + +The Asana data source is used to synchronize tasks, projects, and comments in Asana to a RAGFlow knowledge base. After configuration, team task records and project progress can be queried. + +**Permission requirements**: The Asana account or access token must have read permissions for the target workspace, project, and tasks. + +**Account version requirements**: Both personal projects and organization workspaces can be used. Organization workspaces require the account to have access permissions for the target project. + +**Configuration parameters**: + +- **Name**: Customize the name in RAGFlow to identify this Asana connection. +- **Access token**: Fill in the Asana access token. +- **Workspace ID**: Fill in the Asana workspace ID. +- **Project ID**: Fill in the Asana project ID that needs to be synchronized. +- **Sync comments**: Select whether to synchronize task comments. +- **Sync attachments**: Select whether to synchronize attachments. +- **Sync deleted files**: After this is enabled, content deleted from the external system is removed from the knowledge base index. + +![Asana](https://raw.githubusercontent.com/infiniflow/ragflow-docs/2ee87008723d56cb6ebf0e9c92f6ef2ad1a45254/images/Asana.jpg) + +## Gmail + +The Gmail data source is used to synchronize Gmail email content to a RAGFlow knowledge base. After configuration, customer service emails, business correspondence, and historical communication records can be queried and reused. + +**Permission requirements**: Google OAuth authorization must have read permissions for the target mailbox emails. + +**Account version requirements**: The current interface includes a primary administrator email and OAuth JSON, which is more suitable for Google Workspace administrators or organization mailbox synchronization. Ordinary personal Gmail is recommended only after the actual authorization flow supports it. + +**Configuration parameters**: + +- **Name**: Customize the name in RAGFlow to identify this Gmail connection. +- **Primary administrator email**: Fill in the Google Workspace administrator email. +- **OAuth JSON**: Fill in or upload the OAuth JSON used for Gmail authorization. +- **User email**: Fill in the mailbox address that needs to be synchronized. +- **Label or folder**: Fill in the Gmail label or folder scope that needs to be synchronized. +- **Batch size**: Set the number of emails processed in each batch. +- **Sync deleted files**: After this is enabled, emails deleted from the external system are removed from the knowledge base index. + +![Gmail](https://raw.githubusercontent.com/infiniflow/ragflow-docs/2ee87008723d56cb6ebf0e9c92f6ef2ad1a45254/images/Gmail.jpg) + +## Outlook + +The Outlook data source is used to synchronize Outlook mailbox emails to a RAGFlow knowledge base. After configuration, business emails and communication records in Microsoft 365 can be queried. + +**Permission requirements**: The Microsoft account or application authorization must have read permissions for the target mailbox emails. + +**Account version requirements**: Microsoft 365 or Outlook organization accounts are recommended. Personal Outlook depends on whether the authorization flow and API permissions are available. + +**Configuration parameters**: + +- **Name**: Customize the name in RAGFlow to identify this Outlook connection. +- **Tenant ID**: Fill in the Microsoft Entra ID tenant ID. +- **Client ID**: Fill in the client ID from the app registration. +- **Client secret**: Fill in the client secret from the app registration. +- **Mailbox address**: Fill in the mailbox address that needs to be synchronized. +- **Folder**: Fill in the mail folder that needs to be synchronized. +- **Batch size**: Set the number of emails processed in each batch. +- **Sync deleted files**: After this is enabled, emails deleted from the external system are removed from the knowledge base index. + +![Outlook](https://raw.githubusercontent.com/infiniflow/ragflow-docs/2ee87008723d56cb6ebf0e9c92f6ef2ad1a45254/images/Outlook.jpg) + +## IMAP + +The IMAP data source is used to synchronize email content in mail services that support the IMAP protocol to a RAGFlow knowledge base. After configuration, mailboxes that are not covered by standard email connectors can also be included in knowledge base retrieval. + +**Permission requirements**: The mailbox account must enable IMAP and have read permissions for the target folder. + +**Account version requirements**: Personal mailboxes and enterprise mailboxes can both be used, as long as the mail service supports IMAP login and the account has the required permissions. + +**Configuration parameters**: + +- **Name**: Customize the name in RAGFlow to identify this IMAP connection. +- **IMAP server**: Fill in the IMAP server address. +- **Port**: Fill in the IMAP port. +- **Username**: Fill in the mailbox username. +- **Password or authorization code**: Fill in the mailbox password or application authorization code. +- **Mailbox folder**: Fill in the folder that needs to be synchronized. +- **SSL/TLS**: Select whether to enable secure connection. +- **Batch size**: Set the number of emails processed in each batch. +- **Sync deleted files**: After this is enabled, emails deleted from the external system are removed from the knowledge base index. + +![IMAP](https://raw.githubusercontent.com/infiniflow/ragflow-docs/2ee87008723d56cb6ebf0e9c92f6ef2ad1a45254/images/IMAP.jpg) + +## Microsoft Teams + +The Microsoft Teams data source is used to synchronize Teams channel messages and related collaboration records to a RAGFlow knowledge base. After configuration, team discussion content can be retrieved and reused. + +**Permission requirements**: The Microsoft account or application authorization must have read permissions for the target team, channel, and messages. + +**Account version requirements**: Microsoft 365 organization accounts and Teams permissions are required. Personal accounts are generally not applicable. + +**Configuration parameters**: + +- **Name**: Customize the name in RAGFlow to identify this Microsoft Teams connection. +- **Tenant ID**: Fill in the Microsoft Entra ID tenant ID. +- **Client ID**: Fill in the client ID from the app registration. +- **Client secret**: Fill in the client secret from the app registration. +- **Team ID**: Fill in the Teams team ID that needs to be synchronized. +- **Channel ID**: Fill in the channel ID that needs to be synchronized. +- **Batch size**: Set the number of messages processed in each batch. +- **Sync deleted files**: After this is enabled, messages deleted from the external system are removed from the knowledge base index. + +![Microsoft Teams](https://raw.githubusercontent.com/infiniflow/ragflow-docs/2ee87008723d56cb6ebf0e9c92f6ef2ad1a45254/images/Microsoft_Teams.jpg) + +## Slack + +The Slack data source is used to synchronize Slack channel messages and collaboration content to a RAGFlow knowledge base. After configuration, team communication records can be retrieved and queried. + +**Permission requirements**: The Slack application or token must have read permissions for the target workspace and channels. + +**Account version requirements**: Both personal workspaces and enterprise workspaces can be used. Private channels require the application to be invited or granted the corresponding permissions. + +**Configuration parameters**: + +- **Name**: Customize the name in RAGFlow to identify this Slack connection. +- **Slack bot token**: Fill in the Slack bot token. +- **Workspace ID**: Fill in the Slack workspace ID. +- **Channel IDs**: Fill in the channels that need to be synchronized. +- **Batch size**: Set the number of messages processed in each batch. +- **Sync deleted files**: After this is enabled, messages deleted from the external system are removed from the knowledge base index. + +![Slack](https://raw.githubusercontent.com/infiniflow/ragflow-docs/2ee87008723d56cb6ebf0e9c92f6ef2ad1a45254/images/Slack.jpg) + +## Discord + +The Discord data source is used to synchronize Discord server and channel messages to a RAGFlow knowledge base. After configuration, community communication records can be queried and analyzed. + +**Permission requirements**: The Discord bot must have permission to read the target servers and channels. + +**Account version requirements**: Personal servers and community servers can both be used, as long as the bot has been added to the target server and granted message reading permissions. + +**Configuration parameters**: + +- **Name**: Customize the name in RAGFlow to identify this Discord connection. +- **Discord bot token**: Fill in the Discord bot access token. +- **Server IDs**: Fill in the list of Discord server IDs that need to be synchronized. +- **Channels**: Fill in the Discord channel IDs or channel names that need to be synchronized. +- **Batch size**: Set the number of messages processed in each batch. +- **Sync deleted files**: After this is enabled, messages deleted from the external system are removed from the knowledge base index. + +![Discord](https://raw.githubusercontent.com/infiniflow/ragflow-docs/2ee87008723d56cb6ebf0e9c92f6ef2ad1a45254/images/Discord.jpg) + +## Dingtalk AI Table + +The Dingtalk AI Table data source is used to synchronize records in Dingtalk AI Table to a RAGFlow knowledge base. After configuration, online table records, business ledgers, and collaborative data can be queried. + +**Permission requirements**: The Dingtalk account or application authorization must have read permissions for the target AI Table. + +**Account version requirements**: Dingtalk organization accounts and AI Table permissions are required. Whether it can be synchronized depends on the organization permissions and application authorization. + +**Configuration parameters**: + +- **Name**: Customize the name in RAGFlow to identify this Dingtalk AI Table connection. +- **Application key**: Fill in the Dingtalk application key. +- **Application secret**: Fill in the Dingtalk application secret. +- **Table ID**: Fill in the AI Table ID that needs to be synchronized. +- **View ID**: Fill in the view ID that needs to be synchronized. +- **Content field**: Select the field to be written as document content. +- **Metadata field**: Select the field to be written as metadata. +- **Sync deleted files**: After this is enabled, deleted records are removed from the knowledge base index. + +![Dingtalk AI Table](https://raw.githubusercontent.com/infiniflow/ragflow-docs/2ee87008723d56cb6ebf0e9c92f6ef2ad1a45254/images/Dingtalk_AI_Table.jpg) + +## Zendesk + +The Zendesk data source is used to synchronize Zendesk tickets, comments, and knowledge base content to a RAGFlow knowledge base. After configuration, customer service records and support knowledge can be queried in a unified way. + +**Permission requirements**: The Zendesk account, API token, or application authorization must have read permissions for the target tickets and knowledge base content. + +**Account version requirements**: Zendesk team or enterprise accounts can be used. The account must have access permissions for the target tickets or help center content. + +**Configuration parameters**: + +- **Name**: Customize the name in RAGFlow to identify this Zendesk connection. +- **Subdomain**: Fill in the Zendesk subdomain. +- **Email**: Fill in the Zendesk account email. +- **API token**: Fill in the Zendesk API token. +- **Synchronization scope**: Select tickets, help center articles, or both. +- **Batch size**: Set the number of records processed in each batch. +- **Sync deleted files**: After this is enabled, content deleted from the external system is removed from the knowledge base index. + +## Moodle + +The Moodle data source is used to synchronize course content, forums, resources, and learning records in Moodle to a RAGFlow knowledge base. After configuration, course materials and teaching records can be queried. + +**Permission requirements**: The Moodle account or token must have read permissions for the target courses, activities, and resources. + +**Account version requirements**: Self-built Moodle and organization Moodle platforms can be used. The account must have access permissions for the target courses. + +**Configuration parameters**: + +- **Name**: Customize the name in RAGFlow to identify this Moodle connection. +- **Moodle site URL**: Fill in the Moodle service address. +- **Access token**: Fill in the Moodle access token. +- **Course ID**: Fill in the course ID that needs to be synchronized. +- **Synchronization scope**: Select course resources, forums, assignments, or other content. +- **Batch size**: Set the number of records processed in each batch. +- **Sync deleted files**: After this is enabled, content deleted from the external system is removed from the knowledge base index. + +![Moodle](https://raw.githubusercontent.com/infiniflow/ragflow-docs/2ee87008723d56cb6ebf0e9c92f6ef2ad1a45254/images/Moodle.jpg) + +## REST API + +The REST API data source is used to synchronize data returned by custom business APIs to a RAGFlow knowledge base. After configuration, records in internal systems or third-party systems can enter the knowledge base according to the API structure. + +**Permission requirements**: The API credentials must have permission to call the target API, and the data returned by the API must be readable. + +**Account version requirements**: Personal or enterprise custom systems can both be used, provided that the target API is accessible and provides a stable data structure and authentication method. + +**Configuration parameters**: + +- **Name**: Customize the name in RAGFlow to identify this REST API connection. +- **Request URL**: Fill in the API request address. +- **Request method**: Select GET, POST, or another request method. +- **Request headers**: Fill in the header information required for API authentication or requests. +- **Request body**: Fill in the request body when POST or other methods require it. +- **Pagination method**: Configure the pagination method to ensure that the next page of data can be continuously read. +- **Content field**: Specify the content field returned by the API. +- **Metadata field**: Specify the metadata field returned by the API. +- **Unique ID field**: Specify the unique ID field of each record. +- **Sync deleted files**: After this is enabled, deleted records are removed from the knowledge base index according to the synchronization or cleanup task. + +![REST API](https://raw.githubusercontent.com/infiniflow/ragflow-docs/2ee87008723d56cb6ebf0e9c92f6ef2ad1a45254/images/REST_API.jpg) + +## RSS + +The RSS data source is used to subscribe public websites, blogs, announcements, or product updates to a RAGFlow knowledge base. After configuration, the knowledge base can continuously obtain new content from the subscription source, making it convenient for users to view and ask questions in a unified way. + +**Permission requirements**: The subscription source must be accessible. If the subscription source is protected, make sure the access address or credentials have read permissions. + +**Account version requirements**: Public subscription sources can usually be used directly. If the RSS address is protected, provide an account or access address that can access the subscription source. + +**Configuration parameters**: + +- **Name**: Customize the name in RAGFlow to identify this RSS connection. +- **Feed URL**: Fill in the RSS or Atom subscription address on the page that needs to be synchronized. +- **Batch size**: Set the number of subscription entries processed in each batch. +- **Sync deleted files**: After this is enabled, content deleted from the external system is removed from the knowledge base index. + +![RSS](https://raw.githubusercontent.com/infiniflow/ragflow-docs/2ee87008723d56cb6ebf0e9c92f6ef2ad1a45254/images/RSS.jpg) diff --git a/docs/guides/data_source/overview_and_page_management.md b/docs/guides/data_source/overview_and_page_management.md new file mode 100644 index 0000000000..d3e78600b0 --- /dev/null +++ b/docs/guides/data_source/overview_and_page_management.md @@ -0,0 +1,80 @@ +--- +sidebar_position: 1 +title: Data Source Overview and Page Management +sidebar_label: Overview and Page Management +slug: /add_data_source/overview_and_page_management +sidebar_custom_props: { + categoryIcon: LucideServer +} +--- + +# Data Source Overview and Page Management + +## Data Source Overview + +Data sources are used to connect external systems and synchronize their documents, records, emails, or messages to RAGFlow knowledge bases. After configuring a data source, users can reduce the workload of manually uploading files, and can retrieve and ask questions about content from external systems in the knowledge base. + +## Data Source Page Management + +### Enter the Data Source Page + +Enter the **Data Source** page from **User Settings**. The upper part of the page displays connected data sources, and the lower part displays data sources that can be added. + +![Enter the Data Source Page](https://raw.githubusercontent.com/infiniflow/ragflow-docs/2ee87008723d56cb6ebf0e9c92f6ef2ad1a45254/images/Enter_Data_Source_Page.jpg) + +### Create a Data Source Connection + +Select the system you need to connect from the available data sources. The system opens the creation window for the corresponding data source. When creating a data source, you usually need to complete the following operations: + +1. Fill in the connection name, which is used to identify the connection on the data source page and in the knowledge base. +2. Fill in connection information such as the account, token, address, path, or storage scope. +3. Set synchronization parameters as needed, such as sync deletion, refresh interval, cleanup interval, and timeout. +4. Click **Save** to create the data source connection. + +Different data source creation windows display different fields. Users only need to fill in the fields in the current window. Credentials, addresses, spaces, buckets, database tables, and other information should be based on the actual configuration in the corresponding external system. + +![Create a Data Source Connection](https://raw.githubusercontent.com/infiniflow/ragflow-docs/2ee87008723d56cb6ebf0e9c92f6ef2ad1a45254/images/Create_Data_Source_Connection.jpg) + +### Manage Connected Data Sources + +Successfully created data sources are displayed in the connected data source area. Each connection displays the data source type, connection name, and operation buttons that can be executed. + +**Settings**: Click the settings button to enter the settings page of the data source connection and view or modify connection parameters. + +**Delete**: Click the delete button to delete the data source connection. Before deleting it, confirm whether the connection is still used by a knowledge base to avoid affecting subsequent synchronization. + +Area description: the connected data source area is used to manage the connection itself; the available data source area is used to create new data source connections. + +### Set a Connected Data Source + +Click the settings button on the right side of a connected data source to enter the settings page of that connection. The settings page is used to view and maintain the configuration of the current connection. It is suitable for use when a token expires, the synchronization scope changes, the refresh frequency needs to be adjusted, or the synchronization result is abnormal. + +Common fields on the settings page include: + +- **Name**: The name displayed for the current data source connection in RAGFlow. +- **Authentication information**: The account, token, key, or authorization file used to access the external system. +- **Synchronization scope**: The pages, folders, buckets, database tables, channels, or project scope that need to be read. +- **Sync deleted files**: After this is enabled, content deleted from the external system is removed from the knowledge base index during subsequent synchronization or cleanup tasks. +- **Refresh interval**: The interval at which RAGFlow checks the external system for newly added or updated content. +- **Cleanup interval**: The interval at which RAGFlow checks the external system for deleted content. It takes effect only when sync deleted files is enabled. +- **Timeout**: The maximum time that a single connection or synchronization task is allowed to wait. + +After completing the modification, click **Save**. The new configuration is used for subsequent synchronization tasks. Content that has already been synchronized to the knowledge base is gradually updated according to the next synchronization or cleanup task. + +![Set a Connected Data Source](https://raw.githubusercontent.com/infiniflow/ragflow-docs/2ee87008723d56cb6ebf0e9c92f6ef2ad1a45254/images/Set_a_Connected_Data_Source.jpg) + +### View Connected Data Source Logs + +A log area is provided at the bottom of the data source settings page to view synchronization task records related to the connection. Logs can help determine whether synchronization has been executed, whether it failed, and the reason for failure. + +Log fields are described as follows: + +- **Start time**: The time when the synchronization task started. +- **Status**: The execution result of the synchronization task, such as successful, failed, or in progress. +- **Knowledge base**: The knowledge base that the synchronization task writes to. +- **Task type**: For example, initial synchronization, incremental synchronization, or cleanup task. +- **Summary**: The synchronization summary or error description, used to troubleshoot permissions, fields, networks, rate limits, and other issues. + +If the log is empty, it usually means that the data source has not been used by any knowledge base, or that no synchronization task has been triggered. + +![View Connected Data Source Logs](https://raw.githubusercontent.com/infiniflow/ragflow-docs/2ee87008723d56cb6ebf0e9c92f6ef2ad1a45254/images/View_Connected_Data_Source_Logs.jpg) diff --git a/docs/guides/dataset/add_data_source/_category_.json b/docs/guides/dataset/add_data_source/_category_.json deleted file mode 100644 index 5decae389f..0000000000 --- a/docs/guides/dataset/add_data_source/_category_.json +++ /dev/null @@ -1,11 +0,0 @@ -{ - "label": "Add Data Sources", - "position": 18, - "link": { - "type": "generated-index", - "description": "Add various data sources" - }, - "customProps": { - "categoryIcon": "LucideServer" - } -} diff --git a/docs/guides/dataset/add_data_source/add_bitbucket.md b/docs/guides/dataset/add_data_source/add_bitbucket.md deleted file mode 100644 index 0acd81e6cd..0000000000 --- a/docs/guides/dataset/add_data_source/add_bitbucket.md +++ /dev/null @@ -1,53 +0,0 @@ ---- -sidebar_position: 16 -title: Add Bitbucket -sidebar_label: Add Bitbucket -slug: /add_confluence -sidebar_custom_props: { - categoryIcon: SiGoogledrive -} ---- - -# Add Bitbucket - -Integrate Bitbucket as a data source. - ---- - -This guide outlines the integration of Bitbucket as a data source for RAGFlow. - -## Prerequisites - -Before starting, ensure you have the following: - -- **Bitbucket API token:** A Personal Access Token (PAT) with the appropriate scopes or permissions. -- **Repository URL:** The full URL of the repository you wish to index. -- **Workspace ID:** The unique identifier for your Bitbucket workspace. - -## Configuration Steps - -### Define Bitbucket as an External Data Source - -Navigate to the **Connectors** or **External Data Source** section in the RAGFlow Admin Panel and select **Bitbucket**. Fill in the connector details in the popup window: - -- **Name**: A descriptive name for this connector. -- **Bitbucket Account Email**: The email address for your Bitbucket account. -- **Bitbucket API Token**: The API token with proper permissions created in the previous step. -- **Workspace** The `WORKSPACE_NAME` from your Bitbucket URL, e.g., `https://bitbucket.org/{WORKSPACE_NAME}/...` -- **Index Mode** - - **Workspace**: (Default) Indexes all repositories in the workspace. - - **Repositories**: Indexes specified repositories in the workspace. - - **Repository Slugs**: A comma-separated list of repository slugs, e.g., `repo2,repo2`. - - **Projects**: Indexes specified projects in the workspace. - - **Projects**: A comma-separated list of project keys, e.g., `PROJ1,PROJ2`. - -*RAGFlow validates the connection immediately and indexes all pull requests from the specified repos or projects.* - -### Link to a Dataset - -Credentials alone do not trigger indexing. You must link the data source to a specific dataset: - -1. Navigate to the **Dataset** tab. -2. Select or create the target Dataset. -3. Navigate to the Dataset's **Configuration** page and select **Link data source**. -4. Choose the previously created Bitbucket connector in the popup window. diff --git a/docs/guides/dataset/add_data_source/add_confluence.md b/docs/guides/dataset/add_data_source/add_confluence.md deleted file mode 100644 index b41bf01355..0000000000 --- a/docs/guides/dataset/add_data_source/add_confluence.md +++ /dev/null @@ -1,60 +0,0 @@ ---- -sidebar_position: 4 -title: Add Confluence -sidebar_label: Add Confluence -slug: /add_confluence -sidebar_custom_props: { - categoryIcon: SiGoogledrive -} ---- - -# Add Confluence - -Integrate Confluence as a data source. - ---- - -This guide outlines the integration of Confluence as a data source for RAGFlow. - -## Prerequisites - -Before configuring the connector, obtain the following credentials from your Atlassian account: - -- **Confluence user Email**: The email address of the account performing the indexing. -- **Atlassian API Token**: Generated via [Atlassian Account Settings](https://id.atlassian.com/manage-profile/security/api-tokens). -- **Confluence base URL**: The instance URL (e.g., `https://.atlassian.net/wiki`). - -## Configuration Steps - -### Define Confluence as an External Data Source - -Navigate to the **Connectors** or **External Data Source** section in the RAGFlow Admin Panel and select **Confluence**. Enter the following in the popup window: - -- **Is Cloud** - A toggle indicating whether this is a Confluence Cloud instance. - - `Yes` (default): Confluence Cloud. - - `False`: Confluence Server/Data Center. -- **Name**: *Required* A unique identifier for your Confluence connector (e.g., `Engineering-Wiki`). -- **Confluence Username**: *Required* - - For Confluence Cloud: The full email address you use to log into Confluence. - - For Confluence Server/Data Center: Your login ID, often a shorthand name. -- **Confluence Access Token**: *Required* The authentication key that allows RAGFlow to act on your behalf to read and index your wiki pages. - - For Confluence Cloud: An Atlassian API Token, a secure string generated from your global Atlassian account. Create one from id.atlassian.com/manage-profile/security/api-tokens. - - For Confluence Server/Data Center: Your Personal Access Token (PAT). You are required to log in to your company’s Confluence, click on your Profile Picture in the top right, select Settings, then, in the left-hand sidebar, look for Personal Access Tokens. -- **Wiki Base URL**： The base URL of your confluence instance, e.g., https://your-domain.atlassian.net/wiki. -- **Index Mode** - - `Everything`: (Default) Indexes all pages the provided credentials have access to. - - `Space`： RAGFlow restricts indexing only to the Space Keys you explicitly list in the configuration. - - **Space Keys:** Specify keys (e.g., `ENG, HR`) separated by commas to restrict indexing. Leave blank to index all accessible spaces. - -Once configuration is complete, click **Confirm** to save your changes. - -*RAGFlow validates the connection immediately.* - -### Link to a Dataset - -Credentials alone do not trigger indexing. You must link the data source to a specific dataset: - -1. Navigate to the **Dataset** tab. -2. Select or create the target Dataset. -3. Navigate to the Dataset's **Configuration** page and select **Link data source**. -4. Choose the previously created Confluence connector in the popup window. diff --git a/docs/guides/dataset/add_data_source/add_discord.md b/docs/guides/dataset/add_data_source/add_discord.md deleted file mode 100644 index d3ea674d75..0000000000 --- a/docs/guides/dataset/add_data_source/add_discord.md +++ /dev/null @@ -1,60 +0,0 @@ ---- -sidebar_position: 7 -title: Add Discord -sidebar_label: Add Discord -slug: /add_discord -sidebar_custom_props: { - categoryIcon: SiGoogledrive -} ---- - -# Add Discord - -Integrate Discord as a data source. - ---- - -This guide outlines how to ingest messages from your Discord servers into RAGFlow by setting up a dedicated bot. - -## Prerequisites - -- Administrative privileges for the target Discord server. -- Permissions to add data sources within your RAGFlow environment. - -## Setting Up a Discord Bot - -You need a bot application to access and read messages from your server securely. - -- Go to the Discord Developer Portal. -- Select "New Application" and assign it a descriptive name. -- Navigate to the "Bot" section in the left menu and add a new bot. -- Scroll down to the "Privileged Gateway Intents" section and toggle on "Message Content Intent" so the application can extract message text. -- Click "Reset Token" to generate your bot token. Copy this token immediately and store it safely. - -## Invite the Bot to Your Server - -The bot must be authorized to view the specific channels you intend to sync. - -- In the Developer Portal, open the "OAuth2" menu and select "URL Generator". -- Check the `bot` scope. -- In the permission list, select "View Channels" and "Read Message History". -- Copy the resulting URL generated at the bottom of the screen. -- Open this URL in your browser, select your desired server from the dropdown, and approve the authorization prompt. - -## Configure the Connection in RAGFlow - -With the bot active in your server, you can finalize the integration inside RAGFlow. - -- Open RAGFlow and access the data sources configuration module. -- Choose "Discord" from the list of supported external platforms. -- Paste your saved bot token into the designated input field. -- Configure any specific channels or indexing preferences as required by the interface. -- Save your settings to establish the connection. -- Attach this newly created Discord data source to your target dataset to begin syncing your conversations. - -### Link to a Dataset - -1. Navigate to the **Dataset** tab. -2. Select or create the target Dataset. -3. Navigate to the Dataset's **Configuration** page and select **Link data source**. -4. Choose the previously created Discord connector in the popup window. diff --git a/docs/guides/dataset/add_data_source/add_github_repo.md b/docs/guides/dataset/add_data_source/add_github_repo.md deleted file mode 100644 index beb6d3a178..0000000000 --- a/docs/guides/dataset/add_data_source/add_github_repo.md +++ /dev/null @@ -1,71 +0,0 @@ ---- -sidebar_position: 15 -title: Add GitHub Repo -sidebar_label: Add GitHub Repo -slug: /add_github_repo -sidebar_custom_props: { - categoryIcon: SiGoogledrive -} ---- - -# Add GitHub Repo - -Link your GitHub repo for pull request or issue synchronization. - ---- - -This document explains how to link your GitHub repository to RAGFlow to synchronize pull requests and issues. - -## 1. GitHub Configuration - -Before configuring RAGFlow, you must prepare your GitHub account and generate the necessary credentials. - -### Step A: Public Email Configuration - -To ensure smooth identity matching and permission synchronization between your organization and the RAG engine, it is a best practice to make your email visible. - -1. Go to your GitHub **Settings** > **Emails**. -2. Uncheck "Keep my email addresses private". -3. Go to **Public profile** and ensure your primary email is selected in the **Public email** dropdown. - -### Step B: Generate a Personal Access Token (PAT) - -1. Navigate to **Settings** > **Developer settings** > **Personal access tokens** > **Tokens (classic)**. -2. Click **Generate new token (classic)**. -3. **Required scopes:** - **`repo` (Full control):** Essential for accessing private repositories, PRs, and issues. - - **`read:org` (Optional):** If you are syncing repositories across an entire organization. - - **`workflow` (Optional):** Recommended if you intend to index GitHub Action logs or CI/CD metadata. -4. **Copy the token:** Save this immediately; it will not be displayed again. - -## 2. RAGFlow Connector Setup - -Once your GitHub token is ready, register the external data source within your RAGFlow instance. - -1. **Access data sources:** Click on your profile icon in RAGFlow and select **Data source**. -2. **Add GitHub connector:** Click **+ Add** and select the **GitHub** icon. -3. **Input configuration:** - - **Source name:** Name it based on the repository (e.g., `ragflow-repo`). - - **Repo owner:** The username or organization (e.g., `infiniflow`). - - **Repo name:** The repository identifier (e.g., `ragflow`). - - **Access token:** Paste the PAT generated in section 1. - - **Include Pull Request** Whether to include pull requests in the selected repo. - - **Include Issues** Whether to include issues in the selected repo. -4. Click **Save** to confirm your changes. - *RAGFlow validates the connection immediately.* - -![GitHub](https://raw.githubusercontent.com/infiniflow/ragflow-docs/main/images/github.jpg) - -:::tip NOTE -Currently deleted or modified files are not synchronized automatically. This feature is coming soon. Thanks to Gisselle-Gonzalez for requesting [this feature](https://github.com/infiniflow/ragflow/issues/13708). -::: - -## 3. Dataset Binding & Ingestion - -Finally, link the connector to a specific knowledge base to begin the RAG process. - -1. **Create/select dataset:** Go to the **Dataset** tab and enter your target dataset. -2. **Link external source:** Click **+ Add file** and select **External data source**. -3. **Select GitHub source:** Pick the connector you just created. -4. **Trigger initial sync:** - The files from the repo will appear in your file list. - - Select the files and click **Run/parsing**. - - **Parser selection:** For codebases, use the **"Naive"** parser for general text extraction or a specific code-aware template if available in your current version. diff --git a/docs/guides/dataset/add_data_source/add_google_drive.md b/docs/guides/dataset/add_data_source/add_google_drive.md deleted file mode 100644 index 7df04b80cc..0000000000 --- a/docs/guides/dataset/add_data_source/add_google_drive.md +++ /dev/null @@ -1,115 +0,0 @@ ---- -sidebar_position: 3 -title: Add Google Drive -sidebar_label: Add Google Drive -slug: /add_google_drive -sidebar_custom_props: { - categoryIcon: SiGoogledrive -} ---- - -# Add Google Drive - -Add Google Drive as one of the data sources in RAGFlow. - ---- - -This document provides step-by-step instructions for integrating Google Drive as a data source in RAGFlow. - -## 1. Create a Google Cloud Project - -You can either create a dedicated project for RAGFlow or use an existing Google Cloud external project. In this case, we create a Google Cloud project from scratch: - -1. Open the project creation page `https://console.cloud.google.com/projectcreate`: -![Google Drive image 1](https://raw.githubusercontent.com/infiniflow/ragflow-docs/040e4acd4c1eac6dc73dc44e934a6518de78d097/images/google_drive/image1.jpeg) -2. Under **App Information**, provide an App name and your Gmail account as user support email: -![Google Drive image 2](https://raw.githubusercontent.com/infiniflow/ragflow-docs/040e4acd4c1eac6dc73dc44e934a6518de78d097/images/google_drive/image2.png) -3. Select **External**: - _Your app will start in testing mode and will only be available to a selected list of users._ -![Google Drive image 3](https://raw.githubusercontent.com/infiniflow/ragflow-docs/040e4acd4c1eac6dc73dc44e934a6518de78d097/images/google_drive/image3.jpeg) -4: Click **Create** to confirm creation. - -## 2. Configure OAuth Consent Screen - -You need to configure the OAuth Consent Screen because it is the step where you define how your app asks for permission and what specific data it wants to access on behalf of a user. It's a mandatory part of setting up OAuth 2.0 authentication with Google. Think of it as creating a standardized permission slip for your app. Without it, Google will not allow your app to request access to user data. - -1. Go to **APIs & Services** → **OAuth consent screen**. -2. Ensure **User Type** is set to **External**: -![Google Drive image 4](https://raw.githubusercontent.com/infiniflow/ragflow-docs/040e4acd4c1eac6dc73dc44e934a6518de78d097/images/google_drive/image4.jpeg) -3. Under **Test Users**, click **+ Add users** to add test users: -![Google Drive image 5](https://raw.githubusercontent.com/infiniflow/ragflow-docs/040e4acd4c1eac6dc73dc44e934a6518de78d097/images/google_drive/image5.jpeg) -![Google Drive image 6](https://raw.githubusercontent.com/infiniflow/ragflow-docs/040e4acd4c1eac6dc73dc44e934a6518de78d097/images/google_drive/image6.jpeg) - -## 3. Create OAuth Client Credentials - -1. Navigate to `https://console.cloud.google.com/auth/clients`. -2. Select **Web Application** as **Application type** for the created project: -![Google Drive image 7](https://raw.githubusercontent.com/infiniflow/ragflow-docs/040e4acd4c1eac6dc73dc44e934a6518de78d097/images/google_drive/image7.png) -3. Enter a client name. -4. Add `http://localhost:9380/api/v1/connectors/google-drive/oauth/web/callback` as **Authorized redirect URIs**: -5. Add **Authorized JavaScript origins**: - - If deploying RAGFlow from Docker, use `http://localhost:80`: - ![Google Drive image 8](https://raw.githubusercontent.com/infiniflow/ragflow-docs/040e4acd4c1eac6dc73dc44e934a6518de78d097/images/google_drive/image8.png) - - If building RAGFlow from source, use `http://localhost:9222` - ![Google Drive image 9](https://raw.githubusercontent.com/infiniflow/ragflow-docs/040e4acd4c1eac6dc73dc44e934a6518de78d097/images/google_drive/image9.png) - -6. After saving, click **Download JSON** in the popup window; this credential file will later be uploaded into RAGFlow. - -![Google Drive image 10](https://raw.githubusercontent.com/infiniflow/ragflow-docs/040e4acd4c1eac6dc73dc44e934a6518de78d097/images/google_drive/image10.png) - -## 4. Add Scopes - -You need to add scopes to explicitly define the specific level of access your application requires from a user's Google Drive, such as read-only access to files. These scopes are presented to the user on the consent screen, ensuring transparency by showing exactly what permissions they are granted. To do so: - -1. Click **Data Access** → **Add or remove scopes**, and add the following entries and click **Update**: - -``` -https://www.googleapis.com/auth/drive.readonly -https://www.googleapis.com/auth/drive.metadata.readonly -https://www.googleapis.com/auth/admin.directory.group.readonly -https://www.googleapis.com/auth/admin.directory.user.readonly -``` - -![Google Drive image 11](https://raw.githubusercontent.com/infiniflow/ragflow-docs/040e4acd4c1eac6dc73dc44e934a6518de78d097/images/google_drive/image11.jpeg) -![Google Drive image 12](https://raw.githubusercontent.com/infiniflow/ragflow-docs/040e4acd4c1eac6dc73dc44e934a6518de78d097/images/google_drive/image12.jpeg) - -2. Click **Save** to save your data access changes: - -![Google Drive image 13](https://raw.githubusercontent.com/infiniflow/ragflow-docs/040e4acd4c1eac6dc73dc44e934a6518de78d097/images/google_drive/image13.jpeg) - -## 5. Enable Required APIs - -You need to enable the required APIs (such as the Google Drive API) to formally grant your Google Cloud project permission to communicate with Google's services on behalf of your application. These APIs act as a gateway; even if you have valid OAuth credentials, Google will block requests to a disabled API. Enabling them ensures that when RAGFlow attempts to list or retrieve files, Google's servers recognize and authorize the request. - -1. Navigate to the Google API Library `https://console.cloud.google.com/apis/library`: -![Google Drive image 14](https://raw.githubusercontent.com/infiniflow/ragflow-docs/040e4acd4c1eac6dc73dc44e934a6518de78d097/images/google_drive/image14.png) - -2. Enable the following APIs: - - Google Drive API - - Admin SDK API - - Google Sheets API - - Google Docs API - -![Google Drive image 15](https://raw.githubusercontent.com/infiniflow/ragflow-docs/040e4acd4c1eac6dc73dc44e934a6518de78d097/images/google_drive/image15.png) -![Google Drive image 16](https://raw.githubusercontent.com/infiniflow/ragflow-docs/040e4acd4c1eac6dc73dc44e934a6518de78d097/images/google_drive/image16.png) -![Google Drive image 17](https://raw.githubusercontent.com/infiniflow/ragflow-docs/040e4acd4c1eac6dc73dc44e934a6518de78d097/images/google_drive/image17.png) -![Google Drive image 18](https://raw.githubusercontent.com/infiniflow/ragflow-docs/040e4acd4c1eac6dc73dc44e934a6518de78d097/images/google_drive/image18.png) -![Google Drive image 19](https://raw.githubusercontent.com/infiniflow/ragflow-docs/040e4acd4c1eac6dc73dc44e934a6518de78d097/images/google_drive/image19.png) -![Google Drive image 21](https://raw.githubusercontent.com/infiniflow/ragflow-docs/040e4acd4c1eac6dc73dc44e934a6518de78d097/images/google_drive/image21.png) - -## 6. Add Google Drive as a Data Source in RAGFlow - -1. Go to **Data Sources** inside RAGFlow and select **Google Drive**. -2. Under **OAuth Token JSON**, upload the previously downloaded JSON credentials you saved in [Section 3](#3-create-oauth-client-credentials): -![Google Drive image 22](https://raw.githubusercontent.com/infiniflow/ragflow-docs/040e4acd4c1eac6dc73dc44e934a6518de78d097/images/google_drive/image22.jpeg) -3. Enter the url of the shared Google Drive folder link: -![Google Drive image 23](https://raw.githubusercontent.com/infiniflow/ragflow-docs/040e4acd4c1eac6dc73dc44e934a6518de78d097/images/google_drive/image23.png) -4. Click **Authorize with Google** - _A browser window appears showing that Google hasn't verified this app._ -![Google Drive image 25](https://raw.githubusercontent.com/infiniflow/ragflow-docs/040e4acd4c1eac6dc73dc44e934a6518de78d097/images/google_drive/image25.jpeg) -5. Click **Continue** → **Select All** → **Continue**. -6. When the authorization succeeds, select **OK** to add the data source. -![Google Drive image 26](https://raw.githubusercontent.com/infiniflow/ragflow-docs/040e4acd4c1eac6dc73dc44e934a6518de78d097/images/google_drive/image26.jpeg) -![Google Drive image 27](https://raw.githubusercontent.com/infiniflow/ragflow-docs/040e4acd4c1eac6dc73dc44e934a6518de78d097/images/google_drive/image27.jpeg) -![Google Drive image 28](https://raw.githubusercontent.com/infiniflow/ragflow-docs/040e4acd4c1eac6dc73dc44e934a6518de78d097/images/google_drive/image28.png) -![Google Drive image 29](https://raw.githubusercontent.com/infiniflow/ragflow-docs/040e4acd4c1eac6dc73dc44e934a6518de78d097/images/google_drive/image29.png) diff --git a/docs/guides/dataset/add_data_source/add_notion.md b/docs/guides/dataset/add_data_source/add_notion.md deleted file mode 100644 index 82ad818ebf..0000000000 --- a/docs/guides/dataset/add_data_source/add_notion.md +++ /dev/null @@ -1,85 +0,0 @@ ---- -sidebar_position: 5 -title: Add Notion -sidebar_label: Add Notion -slug: /add_notion -sidebar_custom_props: { - categoryIcon: SiGoogledrive -} ---- - -# Add Notion - -Connecting your Notion workspace to RAGFlow allows you to ingest and sync your notes, databases, and documents directly into your dataset. Once configured, RAGFlow fetches data from the specified Notion pages to provide context for your RAG applications. - -## Prerequisites - -Before you begin, ensure you have: -* A Notion account with **Workspace Owner** permissions (required to create integrations). -* The specific pages or databases you intend to sync. - ---- - -## Create an Internal Integration - -To allow RAGFlow to access your Notion data, you must first create an internal integration in the Notion developer portal to generate a secret token. - -1. Navigate to the [Notion My Integrations](https://www.notion.com/my-integrations) page. -2. Click **+ New integration**. -3. In the **Name** field, enter a name (e.g., "RAGFlow Connector"). -4. Select the **Associated workspace** where your data resides. -5. Under **Capabilities**, ensure **Read content** is selected. RAGFlow does not require write or user-related permissions. -6. Click **Submit**. -7. Under the **Secrets** tab, click **Show** and then **Copy** to save your **Internal Integration Token**. - ---- - -## Grant Access to Your Pages - -By default, an integration has no access to any pages in your workspace. You must explicitly share the pages you want RAGFlow to index. - -1. Open the Notion page or database you wish to use as the root of your data source. -2. Click the **...** (three dots) menu in the top-right corner. -3. Scroll down to **Connect to** (or **Add connections**). -4. Search for the integration you created (e.g., "RAGFlow Connector") and select it. -5. Confirm the connection when prompted. - -:::tip NOTE -If you share a parent page, all its nested child pages and databases will automatically be accessible to the integration. -::: - ---- - -## Identify the Root Page ID - -The **Root Page Id** tells RAGFlow where to start indexing. You can find this in the URL of your Notion page. - -1. Open your target root page in a web browser. -2. Look at the URL in the address bar. The page ID is the 32-character alphanumeric string at the end of the URL. - * **Format:** `https://www.notion.so/workspace-name/Page-Title-`**`11a047149aef80578303e705001bb90e`** -3. Copy only the 32-character string (exclude any parameters following a `?`). - ---- - -## Configure the Notion Connector in RAGFlow - -Once you have your token and ID, add the connector within the RAGFlow interface. - -| Field | Description | Required | -| :--- | :--- | :--- | -| **Name** | A unique label for this data source (e.g., `Engineering Wiki`). | Yes | -| **Notion Integration Token** | The "Internal Integration Secret" copied from your Notion developer portal. | Yes | -| **Root Page Id** | The 32-character ID of the top-level page you want to sync. | No | - -Once configuration is complete, click **Confirm** to save your changes. - -*RAGFlow validates the connection immediately.* - -### Link to a Dataset - -Credentials alone do not trigger indexing. You must link the data source to a specific dataset: - -1. Navigate to the **Dataset** tab. -2. Select or create the target Dataset. -3. Navigate to the Dataset's **Configuration** page and select **Link data source**. -4. Choose the previously created Notion connector in the popup window. diff --git a/docs/guides/dataset/add_data_source/add_rss.md b/docs/guides/dataset/add_data_source/add_rss.md deleted file mode 100644 index 657a9dbebc..0000000000 --- a/docs/guides/dataset/add_data_source/add_rss.md +++ /dev/null @@ -1,57 +0,0 @@ ---- -sidebar_position: 9 -title: Add RSS -sidebar_label: Add RSS -slug: /add_rss -sidebar_custom_props: { - categoryIcon: SiGoogledrive -} ---- - -# Add RSS - -Integrate an RSS feed as a data source. - ---- - -This guide explains how to add an RSS feed as a data source to your dataset in RAGFlow. - -RSS (Really Simple Syndication) is a standardized web feed format used to publish frequently updated content—such as blog entries, news headlines, and podcasts. By connecting an RSS feed to RAGFlow, you can automatically ingest new content from a website as soon as it is published. - -## Benefits - -Integrating an RSS data source provides the following advantages: - -- **Automated ingestion**: Automatically fetch and process the latest articles, news, and updates from any website or blog that supports RSS or Atom feeds. -- **Dynamic dataset**: Keeps your Retrieval-Augmented Generation (RAG) system up to date with continuous, hands-free content delivery. -- **Deleted-file synchronization**: RAGFlow tracks the state of the RSS feed in the background. If an item is removed from the upstream feed, the system automatically synchronizes this change and deletes the corresponding parsed file from your dataset. This prevents stale or outdated information from polluting your RAG context. - -## Prerequisites - -- A valid RSS feed URL. -- An existing dataset in RAGFlow. - -## Find an RSS Feed URL - -Before adding the data source, you need the direct URL of the RSS feed you want to monitor. You can typically find this in a few different ways: - -- **Look for the RSS icon**: Many blogs and news sites display the standard orange RSS icon, often located in the site's header or footer. For example, tech sites like the **AWS News Blog** or **Smashing Magazine** display this icon prominently. Clicking it usually takes you directly to the feed URL. -- **Try common URL paths**: Often, you can find the feed by appending common RSS paths to the website's main URL (e.g., `https://example.com/rss`, `https://example.com/feed`, or `https://example.com/atom.xml`). -- **Check the page source**: Right-click on the webpage, select **View Page Source**, and press `Ctrl+F` (or `Cmd+F`) to search for `rss` or `application/rss+xml`. The URL will be listed in the `href` attribute of that tag. - -## Add an RSS Data Source - -To add an RSS feed to your dataset, follow these steps: - -1. Log in to RAGFlow. -2. Navigate to the **Datasets** page and select the dataset you want to populate. -3. Go to the **Dataset** tab and click **+ Add data source**. -4. Select **RSS** from the list of available integrations. -5. In the configuration dialog, configure the following settings: - - **Name**: Enter a descriptive name to identify this RSS feed. - - **Feed URL**: Enter the complete URL of the RSS feed (e.g., `https://news.ycombinator.com/rss`). - - **Batch size**: Specify the maximum number of new articles or items RAGFlow should fetch and process during a single background sync cycle. The default is 2. This setting helps manage the ingestion rate and prevents system overload, especially when connecting to highly active feeds or performing the initial fetch. - - **Sync deleted files**: Toggle this option on to automatically remove parsed files from your dataset if the corresponding items are deleted from the upstream RSS feed. If disabled, RAGFlow retains all historically ingested content, even if it is no longer available in the source feed. -6. Click **OK** to save the configuration. - -*Once configured, RAGFlow's background task executors will automatically poll the RSS feed. The system continuously downloads new entries for parsing and chunking, while concurrently running the deleted-file sync to remove files that are no longer present in the source feed, requiring no manual scheduling on your part.*