Files
ragflow/conf/service_conf.yaml

158 lines
4.1 KiB
YAML
Raw Normal View History

ragflow:
host: 0.0.0.0
http_port: 9380
Feat: add admin CLI and admin service (#10186) ### What problem does this PR solve? Introduce new feature: RAGFlow system admin service and CLI ### Introduction Admin Service is a dedicated management component designed to monitor, maintain, and administrate the RAGFlow system. It provides comprehensive tools for ensuring system stability, performing operational tasks, and managing users and permissions efficiently. The service offers monitoring of critical components, including the RAGFlow server, Task Executor processes, and dependent services such as MySQL, Infinity / Elasticsearch, Redis, and MinIO. It automatically checks their health status, resource usage, and uptime, and performs restarts in case of failures to minimize downtime. For user and system management, it supports listing, creating, modifying, and deleting users and their associated resources like knowledge bases and Agents. Built with scalability and reliability in mind, the Admin Service ensures smooth system operation and simplifies maintenance workflows. It consists of a server-side Service and a command-line client (CLI), both implemented in Python. User commands are parsed using the Lark parsing toolkit. - **Admin Service**: A backend service that interfaces with the RAGFlow system to execute administrative operations and monitor its status. - **Admin CLI**: A command-line interface that allows users to connect to the Admin Service and issue commands for system management. ### Starting the Admin Service 1. Before start Admin Service, please make sure RAGFlow system is already started. 2. Run the service script: ```bash python admin/admin_server.py ``` The service will start and listen for incoming connections from the CLI on the configured port. ### Using the Admin CLI 1. Ensure the Admin Service is running. 2. Launch the CLI client: ```bash python admin/admin_client.py -h 0.0.0.0 -p 9381 ## Supported Commands Commands are case-insensitive and must be terminated with a semicolon (`;`). ### Service Management Commands - [x] `LIST SERVICES;` - Lists all available services within the RAGFlow system. - [ ] `SHOW SERVICE <id>;` - Shows detailed status information for the service identified by `<id>`. - [ ] `STARTUP SERVICE <id>;` - Attempts to start the service identified by `<id>`. - [ ] `SHUTDOWN SERVICE <id>;` - Attempts to gracefully shut down the service identified by `<id>`. - [ ] `RESTART SERVICE <id>;` - Attempts to restart the service identified by `<id>`. ### User Management Commands - [x] `LIST USERS;` - Lists all users known to the system. - [ ] `SHOW USER '<username>';` - Shows details and permissions for the specified user. The username must be enclosed in single or double quotes. - [ ] `DROP USER '<username>';` - Removes the specified user from the system. Use with caution. - [ ] `ALTER USER PASSWORD '<username>' '<new_password>';` - Changes the password for the specified user. ### Data and Agent Commands - [ ] `LIST DATASETS OF '<username>';` - Lists the datasets associated with the specified user. - [ ] `LIST AGENTS OF '<username>';` - Lists the agents associated with the specified user. ### Meta-Commands Meta-commands are prefixed with a backslash (`\`). - `\?` or `\help` - Shows help information for the available commands. - `\q` or `\quit` - Exits the CLI application. ## Examples ```commandline admin> list users; +-------------------------------+------------------------+-----------+-------------+ | create_date | email | is_active | nickname | +-------------------------------+------------------------+-----------+-------------+ | Fri, 22 Nov 2024 16:03:41 GMT | jeffery@infiniflow.org | 1 | Jeffery | | Fri, 22 Nov 2024 16:10:55 GMT | aya@infiniflow.org | 1 | Waterdancer | +-------------------------------+------------------------+-----------+-------------+ admin> list services; +-------------------------------------------------------------------------------------------+-----------+----+---------------+-------+----------------+ | extra | host | id | name | port | service_type | +-------------------------------------------------------------------------------------------+-----------+----+---------------+-------+----------------+ | {} | 0.0.0.0 | 0 | ragflow_0 | 9380 | ragflow_server | | {'meta_type': 'mysql', 'password': 'infini_rag_flow', 'username': 'root'} | localhost | 1 | mysql | 5455 | meta_data | | {'password': 'infini_rag_flow', 'store_type': 'minio', 'user': 'rag_flow'} | localhost | 2 | minio | 9000 | file_store | | {'password': 'infini_rag_flow', 'retrieval_type': 'elasticsearch', 'username': 'elastic'} | localhost | 3 | elasticsearch | 1200 | retrieval | | {'db_name': 'default_db', 'retrieval_type': 'infinity'} | localhost | 4 | infinity | 23817 | retrieval | | {'database': 1, 'mq_type': 'redis', 'password': 'infini_rag_flow'} | localhost | 5 | redis | 6379 | message_queue | +-------------------------------------------------------------------------------------------+-----------+----+---------------+-------+----------------+ ``` ### Type of change - [x] New Feature (non-breaking change which adds functionality) Signed-off-by: jinhai <haijin.chn@gmail.com>
2025-09-22 10:37:49 +08:00
admin:
host: 0.0.0.0
http_port: 9381
mysql:
name: 'rag_flow'
user: 'root'
password: 'infini_rag_flow'
host: 'localhost'
port: 5455
max_connections: 900
stale_timeout: 300
max_allowed_packet: 1073741824
minio:
user: 'rag_flow'
password: 'infini_rag_flow'
host: 'localhost:9000'
feat: Add Single Bucket Mode for MinIO/S3 (#11416) ## Overview This PR adds support for **Single Bucket Mode** in RAGFlow, allowing users to configure MinIO/S3 to use a single bucket with a directory structure instead of creating multiple buckets per Knowledge Base and user folder. ## Problem Statement The current implementation creates one bucket per Knowledge Base and one bucket per user folder, which can be problematic when: - Cloud providers charge per bucket - IAM policies restrict bucket creation - Organizations want centralized data management in a single bucket ## Solution Added a `prefix_path` configuration option to the MinIO connector that enables: - Using a single bucket with directory-based organization - Backward compatibility with existing multi-bucket deployments - Support for MinIO, AWS S3, and other S3-compatible storage backends ## Changes - **`rag/utils/minio_conn.py`**: Enhanced MinIO connector to support single bucket mode with prefix paths - **`conf/service_conf.yaml`**: Added new configuration options (`bucket` and `prefix_path`) - **`docker/service_conf.yaml.template`**: Updated template with single bucket configuration examples - **`docker/.env.single-bucket-example`**: Added example environment variables for single bucket setup - **`docs/single-bucket-mode.md`**: Comprehensive documentation covering usage, migration, and troubleshooting ## Configuration Example ```yaml minio: user: "access-key" password: "secret-key" host: "minio.example.com:443" bucket: "ragflow-bucket" # Single bucket name prefix_path: "ragflow" # Optional prefix path ``` ## Backward Compatibility ✅ Fully backward compatible - existing deployments continue to work without any changes - If `bucket` is not configured, uses default multi-bucket behavior - If `bucket` is configured without `prefix_path`, uses bucket root - If both are configured, uses `bucket/prefix_path/` structure ## Testing - Tested with MinIO (local and cloud) - Verified backward compatibility with existing multi-bucket mode - Validated IAM policy restrictions work correctly ## Documentation Included comprehensive documentation in `docs/single-bucket-mode.md` covering: - Configuration examples - Migration guide from multi-bucket to single-bucket mode - IAM policy examples - Troubleshooting guide --- **Related Issue**: Addresses use cases where bucket creation is restricted or costly
2025-12-11 12:22:47 +01:00
bucket: ''
prefix_path: ''
es:
hosts: 'http://localhost:1200'
username: 'elastic'
password: 'infini_rag_flow'
Feat: Adds OpenSearch2.19.1 as the vector_database support (#7140) ### What problem does this PR solve? This PR adds the support for latest OpenSearch2.19.1 as the store engine & search engine option for RAGFlow. ### Main Benefit 1. OpenSearch2.19.1 is licensed under the [Apache v2.0 License] which is much better than Elasticsearch 2. For search, OpenSearch2.19.1 supports full-text search、vector_search、hybrid_search those are similar with Elasticsearch on schema 3. For store, OpenSearch2.19.1 stores text、vector those are quite simliar with Elasticsearch on schema ### Changes - Support opensearch_python_connetor. I make a lot of adaptions since the schema and api/method between ES and Opensearch differs in many ways(especially the knn_search has a significant gap) : rag/utils/opensearch_coon.py - Support static config adaptions by changing: conf/service_conf.yaml、api/settings.py、rag/settings.py - Supprt some store&search schema changes between OpenSearch and ES: conf/os_mapping.json - Support OpenSearch python sdk : pyproject.toml - Support docker config for OpenSearch2.19.1 : docker/.env、docker/docker-compose-base.yml、docker/service_conf.yaml.template ### How to use - I didn't change the priority that ES as the default doc/search engine. Only if in docker/.env , we set DOC_ENGINE=${DOC_ENGINE:-opensearch}, it will work. ### Others Our team tested a lot of docs in our environment by using OpenSearch as the vector database ,it works very well. All the conifg for OpenSearch is necessary. ### Type of change - [x] New Feature (non-breaking change which adds functionality) --------- Co-authored-by: Yongteng Lei <yongtengrey@outlook.com> Co-authored-by: writinwaters <93570324+writinwaters@users.noreply.github.com> Co-authored-by: Yingfeng <yingfeng.zhang@gmail.com>
2025-04-24 16:03:31 +08:00
os:
hosts: 'http://localhost:1201'
username: 'admin'
password: 'infini_rag_flow_OS_01'
infinity:
uri: 'localhost:23817'
postgres_port: 5432
db_name: 'default_db'
oceanbase:
scheme: 'oceanbase' # set 'mysql' to create connection using mysql config
config:
db_name: 'test'
user: 'root@ragflow'
password: 'infini_rag_flow'
host: 'localhost'
port: 2881
redis:
db: 1
username: ''
password: 'infini_rag_flow'
host: 'localhost:6379'
task_executor:
message_queue_type: 'redis'
user_default_llm:
default_models:
embedding_model:
api_key: 'xxx'
base_url: 'http://localhost:6380'
# postgres:
# name: 'rag_flow'
# user: 'rag_flow'
# password: 'infini_rag_flow'
# host: 'postgres'
# port: 5432
# max_connections: 100
# stale_timeout: 30
# s3:
# access_key: 'access_key'
# secret_key: 'secret_key'
# region: 'region'
#gcs:
# bucket: 'bridgtl-edm-d-bucket-ragflow'
# oss:
# access_key: 'access_key'
# secret_key: 'secret_key'
# endpoint_url: 'https://s3.oss-cn-hangzhou.aliyuncs.com'
# region: 'cn-hangzhou'
# bucket: 'bucket_name'
# signature_version: 's3'
# addressing_style: 'virtual'
# azure:
# auth_type: 'sas'
# container_url: 'container_url'
# sas_token: 'sas_token'
# azure:
# auth_type: 'spn'
# account_url: 'account_url'
# client_id: 'client_id'
# secret: 'secret'
# tenant_id: 'tenant_id'
# container_name: 'container_name'
# The OSS object storage uses the MySQL configuration above by default. If you need to switch to another object storage service, please uncomment and configure the following parameters.
# opendal:
# scheme: 'mysql' # Storage type, such as s3, oss, azure, etc.
# config:
# oss_table: 'opendal_storage'
# user_default_llm:
# factory: 'BAAI'
# api_key: 'backup'
# base_url: 'backup_base_url'
# default_models:
# chat_model:
# name: 'qwen2.5-7b-instruct'
# factory: 'xxxx'
# api_key: 'xxxx'
# base_url: 'https://api.xx.com'
# embedding_model:
# api_key: 'xxx'
# base_url: 'http://localhost:6380'
# rerank_model: 'bge-reranker-v2'
# asr_model:
# model: 'whisper-large-v3' # alias of name
# image2text_model: ''
# oauth:
# oauth2:
# display_name: "OAuth2"
# client_id: "your_client_id"
# client_secret: "your_client_secret"
# authorization_url: "https://your-oauth-provider.com/oauth/authorize"
# token_url: "https://your-oauth-provider.com/oauth/token"
# userinfo_url: "https://your-oauth-provider.com/oauth/userinfo"
# redirect_uri: "https://your-app.com/v1/user/oauth/callback/oauth2"
# oidc:
# display_name: "OIDC"
# client_id: "your_client_id"
# client_secret: "your_client_secret"
# issuer: "https://your-oauth-provider.com/oidc"
# scope: "openid email profile"
# redirect_uri: "https://your-app.com/v1/user/oauth/callback/oidc"
# github:
# type: "github"
# icon: "github"
# display_name: "Github"
# client_id: "your_client_id"
# client_secret: "your_client_secret"
# redirect_uri: "https://your-app.com/v1/user/oauth/callback/github"
# authentication:
# client:
# switch: false
# http_app_key:
# http_secret_key:
# site:
# switch: false
# permission:
# switch: false
# component: false
# dataset: false
# smtp:
# mail_server: ""
# mail_port: 465
# mail_use_ssl: true
# mail_use_tls: false
# mail_username: ""
# mail_password: ""
# mail_default_sender:
# - "RAGFlow" # display name
# - "" # sender email address
# mail_frontend_url: "https://your-frontend.example.com"
# tcadp_config:
# secret_id: 'tencent_secret_id'
# secret_key: 'tencent_secret_key'
# region: 'tencent_region'