Jin Hai
e96bc37d06
Go: use NATS as the message queue ( #15327 )
...
### What problem does this PR solve?
```
RAGFlow(admin)> mq publish 'msg2';
SUCCESS
RAGFlow(admin)> mq publish 'msg3';
SUCCESS
RAGFlow(admin)> mq list;
+---------+---------------+
| message | subject |
+---------+---------------+
| msg1 | tasks.RAGFLOW |
| msg2 | tasks.RAGFLOW |
| msg3 | tasks.RAGFLOW |
+---------+---------------+
RAGFlow(admin)> mq pull 2;
+---------+---------------+
| message | subject |
+---------+---------------+
| msg1 | tasks.RAGFLOW |
| msg2 | tasks.RAGFLOW |
+---------+---------------+
RAGFlow(admin)> mq pull noack;
+---------+---------------+
| message | subject |
+---------+---------------+
| abc | tasks.RAGFLOW |
+---------+---------------+
RAGFlow(admin)> mq show
+-------------------+----------------+--------+---------------+---------------+-------------------+---------------+
| ack_pending_count | consumer_count | memory | message_count | pending_count | redelivered_count | waiting_count |
+-------------------+----------------+--------+---------------+---------------+-------------------+---------------+
| 2 | 1 | 0 | 2 | 0 | 1 | 0 |
+-------------------+----------------+--------+---------------+---------------+-------------------+---------------+
RAGFlow(admin)> list ingestors;
+--------------+-------------------------------------------+--------+
| host | name | status |
+--------------+-------------------------------------------+--------+
| 192.168.1.38 | ingestor-8f0e4bd5650a4ac58b0151969fbf6935 | alive |
+--------------+-------------------------------------------+--------+
RAGFlow(admin)> list ingestion tasks;
+----------------------------------+----------------------------------+-----------+------+-------------+----------------------------------+
| document_id | id | status | step | user | user_id |
+----------------------------------+----------------------------------+-----------+------+-------------+----------------------------------+
| ffe64fae423411f1a2d938a74640adcc | 90d3d0f6528941c1ac8eb0360effccc4 | COMPLETED | 5 | aaa@aaa.com | 2ba4881420fa11f19e9c38a74640adcc |
+----------------------------------+----------------------------------+-----------+------+-------------+----------------------------------+
RAGFlow(admin)> remove ingestion tasks '90d3d0f6528941c1ac8eb0360effccc4';
+---------+----------------------------------+
| delete | task_id |
+---------+----------------------------------+
| success | 90d3d0f6528941c1ac8eb0360effccc4 |
+---------+----------------------------------+
RAGFlow(admin)> stop ingestion tasks 'e89e20d9a25848a1b79bd9345ddbfe1d';
+----------+----------------------------------+
| status | task_id |
+----------+----------------------------------+
| STOPPING | e89e20d9a25848a1b79bd9345ddbfe1d |
+----------+----------------------------------+
# Publish a message
RAGFlow(admin)> mq publish 'cdd';
SUCCESS
# List current tasks in the message queue
RAGFlow(admin)> mq list
+----------------------------------+---------------+
| message | subject |
+----------------------------------+---------------+
| 7ce392a3c1624cd2be4b5276e8825059 | tasks.RAGFLOW |
+----------------------------------+---------------+
# Consume a task from the message queue
RAGFlow(admin)> mq pull
+------+-----+----------------+
| ack | id | type |
+------+-----+----------------+
| true | cdd | ingestion_test |
+------+-----+----------------+
# User mode
# List ingestion tasks, followed by dataset id
RAGFlow(user)> list ingestion tasks from '0abe79f9423311f1ad8d38a74640adcc';
+---------------------------+---------------+----------------------------------+----------------------------------+----------------------------------+--------+-----------+---------------------------+---------------+----------------------------------+
| create_date | create_time | dataset_id | document_id | id | schema | status | update_date | update_time | user_id |
+---------------------------+---------------+----------------------------------+----------------------------------+----------------------------------+--------+-----------+---------------------------+---------------+----------------------------------+
| 2026-05-30T20:21:06+08:00 | 1780143666289 | 0abe79f9423311f1ad8d38a74640adcc | ffe64fae423411f1a2d938a74640adcc | 8d758cd14a8b4ba8ab505003fb52017d | | COMPLETED | 2026-05-30T20:21:26+08:00 | 1780143686431 | 2ba4881420fa11f19e9c38a74640adcc |
+---------------------------+---------------+----------------------------------+----------------------------------+----------------------------------+--------+-----------+---------------------------+---------------+----------------------------------+
RAGFlow(user)> list ingestion tasks;
+---------------------------+---------------+----------------------------------+----------------------------------+----------------------------------+--------+-----------+---------------------------+---------------+----------------------------------+
| create_date | create_time | dataset_id | document_id | id | schema | status | update_date | update_time | user_id |
+---------------------------+---------------+----------------------------------+----------------------------------+----------------------------------+--------+-----------+---------------------------+---------------+----------------------------------+
| 2026-06-02T19:02:31+08:00 | 1780398151417 | 0abe79f9423311f1ad8d38a74640adcc | ffe64fae423411f1a2d938a74640adcc | e89e20d9a25848a1b79bd9345ddbfe1d | | COMPLETED | 2026-06-02T19:02:52+08:00 | 1780398172208 | 2ba4881420fa11f19e9c38a74640adcc |
+---------------------------+---------------+----------------------------------+----------------------------------+----------------------------------+--------+-----------+---------------------------+---------------+----------------------------------+
# Create an ingestion task
# First argument is document id, second argument is dataset id
RAGFlow(user)> start ingestion 'ffe64fae423411f1a2d938a74640adcc' from '0abe79f9423311f1ad8d38a74640adcc';
+----------------------------------+-------------------------------------------+
| document_id | result |
+----------------------------------+-------------------------------------------+
| ffe64fae423411f1a2d938a74640adcc | task_id: 8d758cd14a8b4ba8ab505003fb52017d |
+----------------------------------+-------------------------------------------+
# Pause an ingestion task, first argument is ingestion id
RAGFlow(user)> stop ingestion '8d758cd14a8b4ba8ab505003fb52017d';
+---------------------------+---------------+----------------------------------+----------------------------------+----------------------------------+--------+-----------+---------------------------+---------------+----------------------------------+
| create_date | create_time | dataset_id | document_id | id | schema | status | update_date | update_time | user_id |
+---------------------------+---------------+----------------------------------+----------------------------------+----------------------------------+--------+-----------+---------------------------+---------------+----------------------------------+
| 2026-05-30T20:21:06+08:00 | 1780143666289 | 0abe79f9423311f1ad8d38a74640adcc | ffe64fae423411f1a2d938a74640adcc | 8d758cd14a8b4ba8ab505003fb52017d | | COMPLETED | 2026-05-30T20:21:26+08:00 | 1780143686431 | 2ba4881420fa11f19e9c38a74640adcc |
+---------------------------+---------------+----------------------------------+----------------------------------+----------------------------------+--------+-----------+---------------------------+---------------+----------------------------------+
# Delete an ingestion task
RAGFlow(api/default)> remove ingestion tasks 'f366450a27d54677aec1c7090add30f0';
+---------+----------------------------------+
| remove | task_id |
+---------+----------------------------------+
| success | f366450a27d54677aec1c7090add30f0 |
+---------+----------------------------------+
```
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
---------
Signed-off-by: Jin Hai <haijin.chn@gmail.com >
2026-06-12 14:56:44 +08:00
Jin Hai
f8c626bbc8
Go: add ingestion server ( #15094 )
...
### What problem does this PR solve?
1. Go ingestion server will connected with admin server with gRPC stream
2. Go ingestion server will be responsible for ingestion tasks
```
RAGFlow(admin)> list ingestors;
+-----------------+-----------+----------------------------------+---------------------------+----------+------------+--------------+--------+------------+---------------+
| address | cpu_usage | id | last_heartbeat | name | process_id | rss_usage | status | task_count | vms_usage |
+-----------------+-----------+----------------------------------+---------------------------+----------+------------+--------------+--------+------------+---------------+
| 127.0.0.1:58564 | 0 | bdd1870eea2646e0aacb8a2cd3307aa2 | 2026-05-24T18:16:17+08:00 | ingestor | 680152 | 212.72265625 | active | 0 | 2589.12109375 |
+-----------------+-----------+----------------------------------+---------------------------+----------+------------+--------------+--------+------------+---------------+
RAGFlow(admin)> start ingestion 'abc';
+----------------------------------+
| task_id |
+----------------------------------+
| e714777639ca4760ab427b5f211e81ad |
+----------------------------------+
RAGFlow(admin)> stop ingestion 'f7bd39d0a724457eb5fdce6d81699776';
+----------------------------------+
| task_id |
+----------------------------------+
| f7bd39d0a724457eb5fdce6d81699776 |
+----------------------------------+
RAGFlow(admin)> list tasks;
+-----+----------------------------------+-------+------+----------------------------------+---------------------------+------------+------------+
| ETA | assign_to | error | from | id | last_update | start_time | status |
+-----+----------------------------------+-------+------+----------------------------------+---------------------------+------------+------------+
| 0 | 17937da188b84f23a5c10bb87588944b | | CLI | eae6431da72a40e796cff3a03008091b | 2026-05-24T19:46:03+08:00 | | COMPLETED |
| 0 | 17937da188b84f23a5c10bb87588944b | | CLI | 6cccdd174bd049ecb05a774bbb47593f | 2026-05-24T19:46:03+08:00 | | COMPLETED |
| 0 | 17937da188b84f23a5c10bb87588944b | | CLI | ef360d777e57485799adb96b30f2b4b8 | 2026-05-24T19:46:03+08:00 | | CANCELED |
| 0 | 17937da188b84f23a5c10bb87588944b | | CLI | bcc5c5448cb64de48b6b6171c36fb790 | 2026-05-24T19:46:03+08:00 | | CANCELED |
| 0 | 17937da188b84f23a5c10bb87588944b | | CLI | bfc25384c43a443294fe2da979a38ac2 | 2026-05-24T19:46:03+08:00 | | DISPATCHED |
| 0 | 17937da188b84f23a5c10bb87588944b | | CLI | 84960537b85d413b8990a9efd5952d67 | 2026-05-24T19:46:04+08:00 | | DISPATCHED |
| 0 | 17937da188b84f23a5c10bb87588944b | | CLI | 3d223c1b51e24b36861a3bfb2f1d58d4 | 2026-05-24T19:46:03+08:00 | | CANCELED |
| 0 | 17937da188b84f23a5c10bb87588944b | | CLI | e433b0e356b846c89c301621a3c54494 | 2026-05-24T19:46:03+08:00 | | COMPLETED |
| 0 | 17937da188b84f23a5c10bb87588944b | | CLI | 7c93a3880f074ebd8eca14e6b51bb7ef | 2026-05-24T19:46:03+08:00 | | COMPLETED |
| 0 | 17937da188b84f23a5c10bb87588944b | | CLI | df2e4ef51aaf4390bff9a23f2692486e | 2026-05-24T19:46:04+08:00 | | DISPATCHED |
| 0 | 17937da188b84f23a5c10bb87588944b | | CLI | 7377c53010194ef7a83aa206698d66ff | 2026-05-24T19:46:05+08:00 | | DISPATCHED |
| 0 | 17937da188b84f23a5c10bb87588944b | | CLI | df64d1a1f9d348e3a2f174c4d7d69e73 | 2026-05-24T19:46:05+08:00 | | DISPATCHED |
| 0 | 17937da188b84f23a5c10bb87588944b | | CLI | b59834512e2847e1bdf13ace04b8a456 | 2026-05-24T19:46:06+08:00 | | DISPATCHED |
| 0 | 17937da188b84f23a5c10bb87588944b | | CLI | 0064bb0ab69344028d1ecfda053826f4 | 2026-05-24T19:46:03+08:00 | | QUEUED |
+-----+----------------------------------+-------+------+----------------------------------+---------------------------+------------+------------+
```
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
---------
Signed-off-by: Jin Hai <haijin.chn@gmail.com >
2026-05-25 14:00:08 +08:00
Jake Armstrong
93d3deb5e4
Fix admin CLI system variable commands ( #14956 )
...
## What
Fixes #12409 .
Implements admin CLI support for:
- `list vars;`
- `show var <name-or-prefix>;`
- `set var <name> <value>;`
## Changes
- Wire Go CLI variable commands to the admin API.
- Support integer and quoted string values in `SET VAR`.
- Return variable rows as `data_type`, `name`, `setting_type`, and
`value`.
- Add exact-name lookup with prefix fallback for `SHOW VAR`.
- Validate values by stored data type: `string`, `integer`, `bool`, and
`json`.
- Keep the legacy Python admin CLI/server behavior aligned.
- Update admin CLI docs and add focused tests.
## Verification
- `go test -count=1 ./internal/cli`
- `python3.12 -m py_compile admin/server/services.py
admin/server/routes.py api/db/services/system_settings_service.py
admin/client/parser.py admin/client/ragflow_client.py`
- Python admin CLI parser smoke test for `SET VAR`, quoted values, `SHOW
VAR`, and `LIST VARS`.
- Attempted `./run_go_tests.sh`; local environment is missing native
tokenizer/linker artifacts:
- `internal/cpp/cmake-build-release/librag_tokenizer_c_api.a`
- `-lstdc++`
Co-authored-by: Jin Hai <haijin.chn@gmail.com >
2026-05-18 19:08:45 +08:00
buua436
f0122179dd
GO: align time units with Python and centralize timestamp injection in BaseModel ( #14875 )
...
### What problem does this PR solve?
align time units with Python and centralize timestamp injection in
BaseModel
### Type of change
- [x] Refactoring
2026-05-14 13:46:46 +08:00
Jin Hai
b6abce50b1
Go: Admin list ingestion tasks ( #14695 )
...
### What problem does this PR solve?
```
RAGFlow(admin)> list tasks;
+-------------+------------------+----------------------------------+-------------+-----------+----------------------------------+----------+----------------------+-------------+-----------+---------+
| chunk_count | digest | document_id | duration | from_page | id | priority | progress | retry_count | task_type | to_page |
+-------------+------------------+----------------------------------+-------------+-----------+----------------------------------+----------+----------------------+-------------+-----------+---------+
| 16 | 8a0016a0dc3cbdbb | f6aa38bb4ad111f1ba6338a74640adcc | 1511.156966 | 0 | f91e4f104ad111f1aaaf38a74640adcc | 0 | 1 | 1 | | 12 |
+-------------+------------------+----------------------------------+-------------+-----------+----------------------------------+----------+----------------------+-------------+-----------+---------+
```
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
---------
Signed-off-by: Jin Hai <haijin.chn@gmail.com >
2026-05-09 10:03:23 +08:00
Jin Hai
aa57b5bd8b
Go: move logger to common module ( #14545 )
...
### What problem does this PR solve?
As title
### Type of change
- [x] Refactoring
Signed-off-by: Jin Hai <haijin.chn@gmail.com >
2026-05-06 10:41:58 +08:00
Jin Hai
f32a832f92
Add rename model directory to entity to avoid name misunderstanding ( #13829 )
...
### What problem does this PR solve?
Model-> entity
### Type of change
- [x] Refactoring
Signed-off-by: Jin Hai <haijin.chn@gmail.com >
2026-03-27 19:25:18 +08:00
Jin Hai
1fff48b656
Add minio go test ( #13800 )
...
### What problem does this PR solve?
1. Add go test
2. Update CI process
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
---------
Signed-off-by: Jin Hai <haijin.chn@gmail.com >
2026-03-27 18:12:56 +08:00
Jin Hai
61cc5ffef2
Add api tokens commands of go admin cli ( #13765 )
...
### What problem does this PR solve?
- GENERATE TOKENS OF USER 'xxx@xxx.com '
- DROP KEY 'ragflow-yyyyy' OF 'xxx@xxx.com '
- LIST KEYS OF 'xxx@xxx.com '
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
---------
Signed-off-by: Jin Hai <haijin.chn@gmail.com >
2026-03-25 21:39:14 +08:00
Jin Hai
b308cd3a02
Update go cli ( #13717 )
...
### What problem does this PR solve?
Go cli
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
---------
Signed-off-by: Jin Hai <haijin.chn@gmail.com >
2026-03-24 20:08:36 +08:00
chanx
e1dbfb8a9c
fix(dao): Remove unnecessary status filter conditions in user queries ( #13698 )
...
### What problem does this PR solve?
Fix: Enhanced the user deletion function to return detailed deletion
information.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
2026-03-19 21:05:15 +08:00
Jin Hai
cc7e94ffb6
Use different API route according the ENV ( #13597 )
...
### What problem does this PR solve?
1. Fix go server date precision
2. Use API_SCHEME_PROXY to control the web API route
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
---------
Signed-off-by: Jin Hai <haijin.chn@gmail.com >
2026-03-13 19:05:30 +08:00
chanx
a3e6c2e84a
Fix: Enhanced user management functionality and cascading data deletion. ( #13594 )
...
### What problem does this PR solve?
Fix: Enhanced user management functionality and cascading data deletion.
Added tenant and related data initialization functionality during user
creation, including tenants, user-tenant relationships, LLM
configuration, and root folder.
Added cascading deletion logic for user deletion, ensuring that all
associated data is cleaned up simultaneously when a user is deleted.
Implemented a Werkzeug-compatible password hash algorithm (scrypt) and
verification functionality.
Added multiple DAO methods to support batch data operations and
cascading deletion.
Improved user login processing and added token signing functionality.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
2026-03-13 16:53:54 +08:00
Jin Hai
5c955a31cc
Update go server ( #13589 )
...
### What problem does this PR solve?
1. Add more CLI command
2. Add some license hooks
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
- [x] Refactoring
Signed-off-by: Jin Hai <haijin.chn@gmail.com >
2026-03-13 14:41:02 +08:00
chanx
1df804a14a
Feature (System Settings): Implemented system settings management functionality ( #13556 )
...
### What problem does this PR solve?
Feature (System Settings): Implemented system settings management
functionality
- Added a new SystemSettings model, including creation and update time
fields.
- Implemented SystemSettingsDAO, providing CRUD operations and
transaction support.
- Implemented management interfaces for variables, configurations, and
environment variables in the admin service.
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
Co-authored-by: Yingfeng <yingfeng.zhang@gmail.com >
2026-03-12 19:06:20 +08:00
chanx
9ca2bac984
Feat: Implement user creation, deletion, and permission management functionality. ( #13519 )
...
### What problem does this PR solve?
Feat: Implement user creation, deletion, and permission management
functionality.
- Added the `ListByEmail` method to `user.go` to query users by email
address.
- Updated the user activation status handling logic in `handler.go`,
adding input validation.
- Added RSA password decryption functionality to `password.go`.
- Implemented complete user management functionality in `service.go`,
including user creation, deletion, password modification, activation
status, and permission management.
- Added input validation and error handling logic.
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
2026-03-11 14:04:00 +08:00
Jin Hai
2028e895fd
Add license and time record DAO ( #13522 )
...
### What problem does this PR solve?
1. Change go server default port to 9382
2. Compatible with EE data model.
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
---------
Signed-off-by: Jin Hai <haijin.chn@gmail.com >
2026-03-11 14:02:24 +08:00
chanx
f14b53c764
feat(admin): Implemented default administrator initialization and login functionality. ( #13504 )
...
### What problem does this PR solve?
feat(admin): Implemented default administrator initialization and login
functionality.
Added support for default administrator configuration, including super
user nickname, email, and password.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
2026-03-10 17:30:21 +08:00
Jin Hai
7f6a9e8ee9
Update ext field type of heartbeat message ( #13490 )
...
### What problem does this PR solve?
As title
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
Signed-off-by: Jin Hai <haijin.chn@gmail.com >
2026-03-10 10:49:39 +08:00
Jin Hai
4fe706876c
Service list and minio status ( #13480 )
...
### What problem does this PR solve?
1. Resolve standard user can access admin service
2. Get RAGFlow service status
3. Fix minio status fetching
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
- [x] New Feature (non-breaking change which adds functionality)
---------
Signed-off-by: Jin Hai <haijin.chn@gmail.com >
2026-03-10 09:56:43 +08:00
Jin Hai
52bcd98d29
Add scheduled tasks ( #13470 )
...
### What problem does this PR solve?
1. RAGFlow server will send heartbeat periodically.
2. This PR will including:
- Scheduled task
- API server message sending
- Admin server API to receive the message.
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
---------
Signed-off-by: Jin Hai <haijin.chn@gmail.com >
2026-03-09 17:48:29 +08:00
Jin Hai
610c1b507d
Add more API of admin server of go ( #13403 )
...
### What problem does this PR solve?
Add APIs to admin server.
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
---------
Signed-off-by: Jin Hai <haijin.chn@gmail.com >
2026-03-09 10:44:53 +08:00
Jin Hai
01a100bb29
Fix data models ( #13444 )
...
### What problem does this PR solve?
Since database model is updated in python version, go server also need
to update
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
---------
Signed-off-by: Jin Hai <haijin.chn@gmail.com >
2026-03-06 20:05:10 +08:00
Jin Hai
3e3b665b89
RAGFlow admin server go version ( #13394 )
...
### What problem does this PR solve?
1. init go admin server
2. refactor api server router
3. add benchmark CI to 450s time limit
4. remove docker builder container after building
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
- [x] New Feature (non-breaking change which adds functionality)
---------
Signed-off-by: Jin Hai <haijin.chn@gmail.com >
2026-03-05 15:18:40 +08:00
