mirror of
https://github.com/infiniflow/ragflow.git
synced 2026-08-04 23:00:30 +08:00
197b142cef
## What Adds [**SereneDB**](https://serenedb.com) as a selectable doc-store engine on **both** RAGFlow paths: - the **Go** `DocEngine` (`internal/engine/serenedb`), alongside Elasticsearch and Infinity; - the **Python** `DocStoreConnection` (`rag/utils/serenedb_conn.py`) + `DOC_ENGINE=serenedb` registration. SereneDB is a PostgreSQL-wire engine (DuckDB execution) whose single inverted index carries **both** a scored text column (`@@`, BM25) and an IVF vector column (`<#>`, inner product), so hybrid search is one SQL statement. The Go engine connects with `database/sql` + `lib/pq` (already a dependency, no new module); the Python connector uses psycopg2 (already a dependency). ## Storage model One table per tenant with `kb_id` as a filter column - the **Elasticsearch / OceanBase** model, not Infinity's per-dataset tables. This keeps BM25 statistics (IDF, avgdl) computed over the whole tenant corpus (global IDF). Both connectors use this identical layout, so they are storage- and retrieval-compatible: `hybrid` proxy routing and Python↔Go switching are safe. On the Python side the connector is wired as OceanBase's plain-SQL sibling (chunk_data JSON metadata, inline chunk vectors, verbatim ES field names); the ES tokenizer path is unchanged. Metadata stays one table per tenant (`ragflow_doc_meta_<tenant>`). The query shapes mirror the Python connector, including the five empirically-found landmines: the scored dictionary needs `frequency + norm` (else `BM25()` silently returns 0.0), the `@@` query is the tokenized query, the scored lexical branch matches one column, vectors use an L2-normalized shadow column with `ip`/`sq8`, and the similarity threshold goes directly in the ANN scan's `WHERE`. **Minimum engine version: SereneDB 26.07.4.** --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
(1). Deploy RAGFlow services and images
https://ragflow.io/docs/build_docker_image
(2). Configure the required environment for testing
Install Python dependencies (including test dependencies):
uv sync --python 3.13 --only-group test --no-default-groups --frozen
Activate the environment:
source .venv/bin/activate
Install SDK:
uv pip install sdk/python
Modify the .env file: Add the following code:
COMPOSE_PROFILES=${COMPOSE_PROFILES},tei-cpu
TEI_MODEL=BAAI/bge-small-en-v1.5
RAGFLOW_IMAGE=infiniflow/ragflow:v0.26.4 #Replace with the image you are using
Start the container（wait two minutes）:
docker compose -f docker/docker-compose.yml up -d
(3). Test Elasticsearch
a) Run sdk tests against Elasticsearch:
export HTTP_API_TEST_LEVEL=p2
export HOST_ADDRESS=http://127.0.0.1:9380 # Ensure that this port is the API port mapped to your localhost
pytest -s --tb=short --level=${HTTP_API_TEST_LEVEL} test/testcases/test_sdk_api
b) Run http api tests against Elasticsearch:
pytest -s --tb=short --level=${HTTP_API_TEST_LEVEL} test/testcases/test_http_api
(4). Test Infinity
Modify the .env file:
DOC_ENGINE=${DOC_ENGINE:-infinity}
Start the container:
docker compose -f docker/docker-compose.yml down -v
docker compose -f docker/docker-compose.yml up -d
a) Run sdk tests against Infinity:
DOC_ENGINE=infinity pytest -s --tb=short --level=${HTTP_API_TEST_LEVEL} test/testcases/test_sdk_api
b) Run http api tests against Infinity:
DOC_ENGINE=infinity pytest -s --tb=short --level=${HTTP_API_TEST_LEVEL} test/testcases/test_http_api