mkaaad
|
3ce1b9be3d
|
Add MySQL and PostgreSQL connectors to Go syncer (#18162)
As title
---------
Co-authored-by: Jin Hai <haijin.chn@gmail.com>
|
2026-08-13 20:16:44 +08:00 |
|
mkaaad
|
0dbfaa8b67
|
feat(cli): add sync logs listing commands for the user CLI (#18195)
### Summary
```
RAGFlow(api/default)> LIST SYNC LOGS WITH PAGE_SIZE 2 PAGE 2;
Total: 764
+----------------------------------+-------------------------+-------------+-----------+----------------------------------+----------------------------------+---------+------------------+------------+--------------+--------+-----------+---------------------+--------------------+---------------------+
| connector_id | docs_removed_from_index | error_count | error_msg | id | kb_id | kb_name | new_docs_indexed | prune_freq | refresh_freq | status | task_type | time_started | total_docs_indexed | update_date |
+----------------------------------+-------------------------+-------------+-----------+----------------------------------+----------------------------------+---------+------------------+------------+--------------+--------+-----------+---------------------+--------------------+---------------------+
| 02b5b14183884445bd55ad6b9d322204 | 0 | 0 | | fb55b5f1fb754475834cd9eac8b52054 | 0e6e28e65b7044e58070e0e4252319a6 | test | 0 | 5 | 0 | 3 | sync | 2026-08-13 09:49:45 | 16 | 2026-08-13 09:49:46 |
| 02b5b14183884445bd55ad6b9d322204 | 0 | 0 | | 3cad779d7d2e4efb8d8d2dbf05d45653 | 0e6e28e65b7044e58070e0e4252319a6 | test | 0 | 5 | 0 | 3 | sync | 2026-08-13 09:49:42 | 16 | 2026-08-13 09:49:43 |
+----------------------------------+-------------------------+-------------+-----------+----------------------------------+----------------------------------+---------+------------------+------------+--------------+--------+-----------+---------------------+--------------------+---------------------+
```
```
RAGFlow(api/default)> LIST SYNC LOGS;
Total: 764
+----------------------------------+-------------------------+-------------+--------------------------------------------------------------------------------------------+----------------------------------+----------------------------------+---------+------------------+------------+--------------+--------+-----------+---------------------+--------------------+---------------------+
| connector_id | docs_removed_from_index | error_count | error_msg | id | kb_id | kb_name | new_docs_indexed | prune_freq | refresh_freq | status | task_type | time_started | total_docs_indexed | update_date |
+----------------------------------+-------------------------+-------------+--------------------------------------------------------------------------------------------+----------------------------------+----------------------------------+---------+------------------+------------+--------------+--------+-----------+---------------------+--------------------+---------------------+
| 02b5b14183884445bd55ad6b9d322204 | 0 | 1 | sync task failed: failed to fetch URL: Get "https://xxxxxxxxxxxx/rss.xml": EOF | 87592e5623fc4284bf69b9290f322718 | 0e6e28e65b7044e58070e0e4252319a6 | test | 0 | 5 | 0 | 4 | sync | 2026-08-13 09:49:48 | 16 | 2026-08-13 09:51:01 |
| 570138201cf44e1180711f1ebdf8492f | 0 | 1 | sync task failed: failed to fetch URL: Get "https://www.ruanyifeng.com/blog/atom.xml": EOF | 456adbbf21a147af99adfce4405abd99 | 0e6e28e65b7044e58070e0e4252319a6 | test | 0 | 5 | 1 | 4 | sync | 2026-08-13 09:50:00 | 3 | 2026-08-13 09:50:18 |
| 02b5b14183884445bd55ad6b9d322204 | 0 | 0 | | fb55b5f1fb754475834cd9eac8b52054 | 0e6e28e65b7044e58070e0e4252319a6 | test | 0 | 5 | 0 | 3 | sync | 2026-08-13 09:49:45 | 16 | 2026-08-13 09:49:46 |
| 02b5b14183884445bd55ad6b9d322204 | 0 | 0 | | 3cad779d7d2e4efb8d8d2dbf05d45653 | 0e6e28e65b7044e58070e0e4252319a6 | test | 0 | 5 | 0 | 3 | sync | 2026-08-13 09:49:42 | 16 | 2026-08-13 09:49:43 |
| 02b5b14183884445bd55ad6b9d322204 | 0 | 0 | | dd09320aa5b34d6cbcfa7b03ede68c4a | 0e6e28e65b7044e58070e0e4252319a6 | test | 0 | 5 | 0 | 3 | sync | 2026-08-13 09:49:36 | 16 | 2026-08-13 09:49:38 |
| 02b5b14183884445bd55ad6b9d322204 | 0 | 0 | | afbcee73240f420ca1997e6506f387fb | 0e6e28e65b7044e58070e0e4252319a6 | test | 0 | 5 | 0 | 3 | sync | 2026-08-13 09:49:33 | 16 | 2026-08-13 09:49:34 |
| 02b5b14183884445bd55ad6b9d322204 | 0 | 0 | | 7d018146a0f14a25b7263d5980fdc30f | 0e6e28e65b7044e58070e0e4252319a6 | test | 0 | 5 | 0 | 3 | sync | 2026-08-13 09:49:30 | 16 | 2026-08-13 09:49:32 |
| 02b5b14183884445bd55ad6b9d322204 | 0 | 0 | | 520e0d6f421e40628ab541fc4fee1a69 | 0e6e28e65b7044e58070e0e4252319a6 | test | 0 | 5 | 0 | 3 | sync | 2026-08-13 09:49:27 | 16 | 2026-08-13 09:49:28 |
| 02b5b14183884445bd55ad6b9d322204 | 0 | 0 | | 3d3ce49825de4ef99642e7994ae954e3 | 0e6e28e65b7044e58070e0e4252319a6 | test | 0 | 5 | 0 | 3 | sync | 2026-08-13 09:49:21 | 16 | 2026-08-13 09:49:22 |
| 02b5b14183884445bd55ad6b9d322204 | 0 | 0 | | 1c511f9b4a7c425fb30a956589ca56fa | 0e6e28e65b7044e58070e0e4252319a6 | test | 0 | 5 | 0 | 3 | sync | 2026-08-13 09:49:18 | 16 | 2026-08-13 09:49:19 |
| 02b5b14183884445bd55ad6b9d322204 | 0 | 0 | | b9ab73c3f48c4f47a84fd55b542b32b4 | 0e6e28e65b7044e58070e0e4252319a6 | test | 0 | 5 | 0 | 3 | sync | 2026-08-13 09:49:15 | 16 | 2026-08-13 09:49:16 |
| 02b5b14183884445bd55ad6b9d322204 | 0 | 0 | | db0cad90a7744d758b24d2e3d3828963 | 0e6e28e65b7044e58070e0e4252319a6 | test | 0 | 5 | 0 | 3 | sync | 2026-08-13 09:49:12 | 16 | 2026-08-13 09:49:13 |
| 02b5b14183884445bd55ad6b9d322204 | 0 | 0 | | a501eadb2cce452187f3c86431352c70 | 0e6e28e65b7044e58070e0e4252319a6 | test | 0 | 5 | 0 | 3 | sync | 2026-08-13 09:49:08 | 16 | 2026-08-13 09:49:10 |
| 02b5b14183884445bd55ad6b9d322204 | 0 | 0 | | b4265a3b7baf4466a525b91a9fe21b71 | 0e6e28e65b7044e58070e0e4252319a6 | test | 0 | 5 | 0 | 3 | sync | 2026-08-13 09:49:03 | 16 | 2026-08-13 09:49:05 |
| 02b5b14183884445bd55ad6b9d322204 | 0 | 0 | | a6ad590931c046dda5fc3222b4010c06 | 0e6e28e65b7044e58070e0e4252319a6 | test | 0 | 5 | 0 | 3 | sync | 2026-08-13 09:49:00 | 16 | 2026-08-13 09:49:02 |
......
+----------------------------------+-------------------------+-------------+--------------------------------------------------------------------------------------------+----------------------------------+----------------------------------+---------+------------------+------------+--------------+--------+-----------+---------------------+--------------------+---------------------+
```
list all logs
```
RAGFlow(api/default)> LIST DATASET 'test' SYNC LOGS;
Total: 764
+----------------------------------+-------------------------+-------------+--------------------------------------------------------------------------------------------+----------------------------------+----------------------------------+---------+------------------+------------+--------------+--------+-----------+---------------------+--------------------+---------------------+
| connector_id | docs_removed_from_index | error_count | error_msg | id | kb_id | kb_name | new_docs_indexed | prune_freq | refresh_freq | status | task_type | time_started | total_docs_indexed | update_date |
+----------------------------------+-------------------------+-------------+--------------------------------------------------------------------------------------------+----------------------------------+----------------------------------+---------+------------------+------------+--------------+--------+-----------+---------------------+--------------------+---------------------+
| 02b5b14183884445bd55ad6b9d322204 | 0 | 1 | sync task failed: failed to fetch URL: Get "https://xxxxxxxxxxxx/rss.xml": EOF | 87592e5623fc4284bf69b9290f322718 | 0e6e28e65b7044e58070e0e4252319a6 | test | 0 | 5 | 0 | 4 | sync | 2026-08-13 09:49:48 | 16 | 2026-08-13 09:51:01 |
| 570138201cf44e1180711f1ebdf8492f | 0 | 1 | sync task failed: failed to fetch URL: Get "https://www.ruanyifeng.com/blog/atom.xml": EOF | 456adbbf21a147af99adfce4405abd99 | 0e6e28e65b7044e58070e0e4252319a6 | test | 0 | 5 | 1 | 4 | sync | 2026-08-13 09:50:00 | 3 | 2026-08-13 09:50:18 |
| 02b5b14183884445bd55ad6b9d322204 | 0 | 0 | | fb55b5f1fb754475834cd9eac8b52054 | 0e6e28e65b7044e58070e0e4252319a6 | test | 0 | 5 | 0 | 3 | sync | 2026-08-13 09:49:45 | 16 | 2026-08-13 09:49:46 |
| 02b5b14183884445bd55ad6b9d322204 | 0 | 0 | | 3cad779d7d2e4efb8d8d2dbf05d45653 | 0e6e28e65b7044e58070e0e4252319a6 | test | 0 | 5 | 0 | 3 | sync | 2026-08-13 09:49:42 | 16 | 2026-08-13 09:49:43 |
| 02b5b14183884445bd55ad6b9d322204 | 0 | 0 | | dd09320aa5b34d6cbcfa7b03ede68c4a | 0e6e28e65b7044e58070e0e4252319a6 | test | 0 | 5 | 0 | 3 | sync | 2026-08-13 09:49:36 | 16 | 2026-08-13 09:49:38 |
| 02b5b14183884445bd55ad6b9d322204 | 0 | 0 | | afbcee73240f420ca1997e6506f387fb | 0e6e28e65b7044e58070e0e4252319a6 | test | 0 | 5 | 0 | 3 | sync | 2026-08-13 09:49:33 | 16 | 2026-08-13 09:49:34 |
| 02b5b14183884445bd55ad6b9d322204 | 0 | 0 | | 7d018146a0f14a25b7263d5980fdc30f | 0e6e28e65b7044e58070e0e4252319a6 | test | 0 | 5 | 0 | 3 | sync | 2026-08-13 09:49:30 | 16 | 2026-08-13 09:49:32 |
| 02b5b14183884445bd55ad6b9d322204 | 0 | 0 | | 520e0d6f421e40628ab541fc4fee1a69 | 0e6e28e65b7044e58070e0e4252319a6 | test | 0 | 5 | 0 | 3 | sync | 2026-08-13 09:49:27 | 16 | 2026-08-13 09:49:28 |
| 02b5b14183884445bd55ad6b9d322204 | 0 | 0 | | 3d3ce49825de4ef99642e7994ae954e3 | 0e6e28e65b7044e58070e0e4252319a6 | test | 0 | 5 | 0 | 3 | sync | 2026-08-13 09:49:21 | 16 | 2026-08-13 09:49:22 |
| 02b5b14183884445bd55ad6b9d322204 | 0 | 0 | | 1c511f9b4a7c425fb30a956589ca56fa | 0e6e28e65b7044e58070e0e4252319a6 | test | 0 | 5 | 0 | 3 | sync | 2026-08-13 09:49:18 | 16 | 2026-08-13 09:49:19 |
| 02b5b14183884445bd55ad6b9d322204 | 0 | 0 | | b9ab73c3f48c4f47a84fd55b542b32b4 | 0e6e28e65b7044e58070e0e4252319a6 | test | 0 | 5 | 0 | 3 | sync | 2026-08-13 09:49:15 | 16 | 2026-08-13 09:49:16 |
| 02b5b14183884445bd55ad6b9d322204 | 0 | 0 | | db0cad90a7744d758b24d2e3d3828963 | 0e6e28e65b7044e58070e0e4252319a6 | test | 0 | 5 | 0 | 3 | sync | 2026-08-13 09:49:12 | 16 | 2026-08-13 09:49:13 |
| 02b5b14183884445bd55ad6b9d322204 | 0 | 0 | | a501eadb2cce452187f3c86431352c70 | 0e6e28e65b7044e58070e0e4252319a6 | test | 0 | 5 | 0 | 3 | sync | 2026-08-13 09:49:08 | 16 | 2026-08-13 09:49:10 |
| 02b5b14183884445bd55ad6b9d322204 | 0 | 0 | | b4265a3b7baf4466a525b91a9fe21b71 | 0e6e28e65b7044e58070e0e4252319a6 | test | 0 | 5 | 0 | 3 | sync | 2026-08-13 09:49:03 | 16 | 2026-08-13 09:49:05 |
| 02b5b14183884445bd55ad6b9d322204 | 0 | 0 | | a6ad590931c046dda5fc3222b4010c06 | 0e6e28e65b7044e58070e0e4252319a6 | test | 0 | 5 | 0 | 3 | sync | 2026-08-13 09:49:00 | 16 | 2026-08-13 09:49:02 |
......
+----------------------------------+-------------------------+-------------+--------------------------------------------------------------------------------------------+----------------------------------+----------------------------------+---------+------------------+------------+--------------+--------+-----------+---------------------+--------------------+---------------------+
```
|
2026-08-13 18:39:29 +08:00 |
|
Haruko386
|
96f7c39753
|
feat[Go]: resuming transmission from the point of interruption during data source synchronisation (#18176)
### Summary
As title
|
2026-08-13 18:38:37 +08:00 |
|
Jin Hai
|
d7661b676d
|
Go: fix context and warnings (#18097)
Signed-off-by: Jin Hai <haijin.chn@gmail.com>
|
2026-08-11 16:18:49 +08:00 |
|
Haruko386
|
dd52dee600
|
feat[Go]: monitoring NATs and refactoring concurrency logic (#18049)
### Summary
As title
|
2026-08-11 14:36:11 +08:00 |
|
Haruko386
|
615c7d8bcb
|
feat[Go]: implement data source gmail and test prune (#17969)
### Summary
As title
|
2026-08-07 13:35:57 +08:00 |
|
Haruko386
|
f532f27f1f
|
feat[Go]: complete the base for data Syncer (#17890)
### Summary
As title
---------
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
|
2026-08-06 20:09:46 +08:00 |
|
Jin Hai
|
7df7384b21
|
Go: refactor UUID functions (#16695)
As title
---------
Signed-off-by: Jin Hai <haijin.chn@gmail.com>
|
2026-07-07 17:22:08 +08:00 |
|
Jin Hai
|
1aa8abe373
|
Go: file syncer service framework (#16579)
### Summary
./ragflow_main --syncer to start file syncer
config yaml file has following config
```
file_syncer:
max_concurrent_syncs: 4 # concurrent file sync threads
sync_interval: 3 # check interval
```
---------
Signed-off-by: Jin Hai <haijin.chn@gmail.com>
|
2026-07-03 11:14:02 +08:00 |
|