Commit Graph

385 Commits

Author SHA1 Message Date
Liu An
ec9c08d809 Docs: Update version references to v0.27.0 in READMEs and docs (#18500)
### What problem does this PR solve?

- Update version tags in README files (including translations) from
v0.26.4 to v0.27.0
- Modify Docker image references and documentation to reflect new
version
- Update version badges and image descriptions
- Maintain consistency across all language variants of README files

### Type of change

- [x] Documentation Update
2026-08-19 17:06:49 +08:00
zws202606
d02723ca76 feat: Gaussdb adaptation (#17703) 2026-08-18 09:53:11 +08:00
Mpider-San
c7aca4c97e Replace pgsty/minio image with pgsty/silo (#18231)
### Summary

Closes #18019.

Pigsty renamed its community MinIO fork from `pgsty/minio` to
`pgsty/silo` on 2026-08-06 ([announcement in the repo
README](https://github.com/pgsty/silo#readme)). `pgsty/minio` is frozen
at `RELEASE.2026-08-04T00-00-00Z` — all further releases, including
security fixes, ship as `pgsty/silo`. RAGFlow still pins the old name,
so it is now parked on an image that no longer gets updates.
2026-08-13 18:45:43 +08:00
Haruko386
96f7c39753 feat[Go]: resuming transmission from the point of interruption during data source synchronisation (#18176)
### Summary

As title
2026-08-13 18:38:37 +08:00
qinling0210
0c225b0c39 Bump to infinity v0.7.3 and update go sdk version (#18137)
### Summary

Bump to infinity v0.7.3 and update go sdk version
2026-08-12 11:30:34 +08:00
Haruko386
dd52dee600 feat[Go]: monitoring NATs and refactoring concurrency logic (#18049)
### Summary

As title
2026-08-11 14:36:11 +08:00
Yee
27c8b1a061 fix(nginx): use 127.0.0.1 in python conf to avoid ~1s IPv6 fallback latency (#17909)
## Problem

In the default deployment (`API_PROXY_SCHEME=python`), the nginx config
generated from `docker/nginx/ragflow.conf.python` uses:

```nginx
proxy_pass http://localhost:9381;  # admin
proxy_pass http://localhost:9380;  # api
```

Inside the RAGFlow container, `/etc/hosts` maps `localhost` to both
`127.0.0.1` and `::1`:

```
127.0.0.1   localhost
::1         localhost ip6-localhost ip6-loopback
```

But the backend services (`ragflow_server.py` on 9380, admin server on
9381) only listen on IPv4 (`127.0.0.1`). When nginx resolves `localhost`
to `::1`, the upstream connection fails and nginx falls back to the IPv4
address - each fallback adds ~1 second of latency.

Co-authored-by: Jin Hai <haijin.chn@gmail.com>
2026-08-07 11:32:43 +08:00
Zhichang Yu
2e37997ab9 Go knowledge compiler with scheduler-driven dataset compilation (#17913)
Ports dataset knowledge compilation (wiki/graph/tree/mindmap) to the Go
scheduler with a status contract, aligns wiki storage/retrieval with
Python, sizes prompts by content_length, and resolves embedding batch
size from provider capability.
2026-08-06 15:54:00 +08:00
Jin Hai
cf13082a1a Revert "feat: Go knowledge compiler with scheduler-driven dataset compilation" (#17897)
Reverts infiniflow/ragflow#17881
2026-08-05 21:50:28 +08:00
Zhichang Yu
14b943a04a feat: Go knowledge compiler with scheduler-driven dataset compilation (#17881)
Ports the dataset knowledge compilation (wiki/graph/tree/mindmap) to the
Go scheduler with a status contract, aligns wiki storage/retrieval with
Python, and sizes prompts by content_length.
2026-08-05 20:00:42 +08:00
deadtrickster
197b142cef feat(serenedb): add SereneDB doc-store engine (Go + Python connectors) (#17375)
## What

Adds [**SereneDB**](https://serenedb.com) as a selectable doc-store
engine on **both** RAGFlow paths:
- the **Go** `DocEngine` (`internal/engine/serenedb`), alongside
Elasticsearch and Infinity;
- the **Python** `DocStoreConnection` (`rag/utils/serenedb_conn.py`) +
`DOC_ENGINE=serenedb` registration.

SereneDB is a PostgreSQL-wire engine (DuckDB execution) whose single
inverted index carries **both** a scored text column (`@@`, BM25) and an
IVF vector column (`<#>`, inner product), so hybrid search is one SQL
statement. The Go engine connects with `database/sql` + `lib/pq`
(already a dependency, no new module); the Python connector uses
psycopg2 (already a dependency).

## Storage model

One table per tenant with `kb_id` as a filter column - the
**Elasticsearch / OceanBase** model, not Infinity's per-dataset tables.
This keeps BM25 statistics (IDF, avgdl) computed over the whole tenant
corpus (global IDF). Both connectors use this identical layout, so they
are storage- and retrieval-compatible: `hybrid` proxy routing and
Python↔Go switching are safe. On the Python side the connector is wired
as OceanBase's plain-SQL sibling (chunk_data JSON metadata, inline chunk
vectors, verbatim ES field names); the ES tokenizer path is unchanged.
Metadata stays one table per tenant (`ragflow_doc_meta_<tenant>`).

The query shapes mirror the Python connector, including the five
empirically-found landmines: the scored dictionary needs `frequency +
norm` (else `BM25()` silently returns 0.0), the `@@` query is the
tokenized query, the scored lexical branch matches one column, vectors
use an L2-normalized shadow column with `ip`/`sq8`, and the similarity
threshold goes directly in the ANN scan's `WHERE`. **Minimum engine
version: SereneDB 26.07.4.**

---------

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-08-04 14:16:39 +08:00
Wang Qi
bb1879ba35 REDIS: allkeys-lru -> volatile-lru to avoid got evict if lack of memory (#17720)
Follow on PR: #17707
2026-08-03 15:44:59 +08:00
maoyifeng
3874ef8765 Dockerfile base part separate to base image (#17701)
Unify open-source and closed-source Dockerfile expect nginx
1. per install tools to base image
2. fix copy nginx config
3. fix copy entrypoint*.sh
4. fix go ci error
5. add comment
2026-08-03 14:29:48 +08:00
Jin Hai
b2521ebf51 Go: refactor log and config (#17684)
Signed-off-by: Jin Hai <haijin.chn@gmail.com>
2026-08-02 20:55:19 +08:00
Jin Hai
36ae39bc60 Go: refactor config (#17544)
Signed-off-by: Jin Hai <haijin.chn@gmail.com>
2026-07-31 17:18:45 +08:00
Jin Hai
0eb8e0b393 Go: refactor ragflow_server.go (#17494)
Signed-off-by: Jin Hai <haijin.chn@gmail.com>
2026-07-29 11:38:27 +08:00
Wang Qi
b4170d0dfd Dev/enhance dev script (#17487) 2026-07-28 17:47:45 +08:00
euvre
f45f03a016 Add Go service/handler tests for API contract parity (#16905) 2026-07-20 20:02:41 +08:00
Jin Hai
75a8228d33 Fix missing info (#17107)
### Summary

1. Fix docker/service_conf.yaml.template
2. Remove unused config

---------

Signed-off-by: Jin Hai <haijin.chn@gmail.com>
2026-07-20 17:32:32 +08:00
qinling0210
6c002ced86 Bump Infinity to v0.7.2 (#17075)
### Summary

Bump to infinity v0.7.2

Infinity image: infiniflow/infinity:v0.7.2-x64-v3
2026-07-20 14:30:24 +08:00
Wang Qi
c420de0dc5 Security fix: CVE-2025-21521 (#17046) 2026-07-17 19:35:20 +08:00
Jin Hai
f8474a67f5 Go: tracing framework (#17004)
### Summary

Tracing framework

---------

Signed-off-by: Jin Hai <haijin.chn@gmail.com>
2026-07-16 19:35:43 +08:00
Jin Hai
c7a623ff81 Go: introduce clickhouse (#16996)
### Summary

Introduce Clickhouse for data statistics.

Signed-off-by: Jin Hai <haijin.chn@gmail.com>
2026-07-16 18:36:56 +08:00
Jin Hai
b506d8ec5f Go: add more frameworks (#16915)
### Summary

1. Add hooks for server init
2. Add hooks for router init
3. Add jaeger and otel related config

Signed-off-by: Jin Hai <haijin.chn@gmail.com>
2026-07-14 22:20:26 +08:00
Jin Hai
bdef878821 Go: fix nats and go command (#16828)
### Summary
1. update docker compose file to start NATS healthy
2. Add two commands
```
RAGFlow(admin)> live;
SUCCESS
RAGFlow(admin)> health;
+---------------+-------+
| field         | value |
+---------------+-------+
| storage       | ok    |
| message_queue | ok    |
| status        | ok    |
| db            | ok    |
| redis         | ok    |
| doc_engine    | ok    |
+---------------+-------+
```

---------

Signed-off-by: Jin Hai <haijin.chn@gmail.com>
2026-07-11 19:37:57 +08:00
Jin Hai
21266286cb Go: add more commands and GCS supports (#16741)
### Summary

1. GCS supports
2. More commands
```
RAGFlow(admin)> ping store;
SUCCESS
RAGFlow(admin)> ping engine;
SUCCESS
RAGFlow(admin)> ping cache;
SUCCESS
```

---------

Signed-off-by: Jin Hai <haijin.chn@gmail.com>
2026-07-08 17:49:02 +08:00
Lynn
0ae5961e1c Feat: v0.27.0 model provider (#16604) 2026-07-08 09:47:29 +08:00
Liu An
4da9429451 Docs: Update version references to v0.26.4 in READMEs and docs (#16716) 2026-07-07 19:36:58 +08:00
Wang Qi
06aa169df7 Update development script (#16623) 2026-07-03 20:34:30 +08:00
Jin Hai
1880e65e99 Go: refactor (#16602)
### Summary

1. update doc
2. refactor route code

---------

Signed-off-by: Jin Hai <haijin.chn@gmail.com>
2026-07-03 17:00:43 +08:00
Jin Hai
1aa8abe373 Go: file syncer service framework (#16579)
### Summary

./ragflow_main --syncer to start file syncer


config yaml file has following config
```
file_syncer:
  max_concurrent_syncs: 4 # concurrent file sync threads
  sync_interval: 3 # check interval

```

---------

Signed-off-by: Jin Hai <haijin.chn@gmail.com>
2026-07-03 11:14:02 +08:00
Jin Hai
7d64a78f83 Go: unify three services into one binary (#16462)
### Summary

Plan to start api_server, admin_server and ingestor in one binary:
- ./ragflow_main --admin
- ./ragflow_main --api
- ./ragflow_main --ingestor

---------

Signed-off-by: Jin Hai <haijin.chn@gmail.com>
2026-07-02 21:21:10 +08:00
Liu An
32c5cb16e9 Docs: Update version references to v0.26.3 in READMEs and docs (#16574) 2026-07-02 20:55:15 +08:00
qinling0210
7862f69f39 Implement chat completions in go (#16491)
### Summary

POST   /api/v1/chat/completions
2026-07-01 15:52:52 +08:00
Zhichang Yu
4c54cefd29 Port 14 upstream agent security / correctness fixes to Go canvas (#16455)
Mirrors 14 merged upstream PRs into the Go agent port.

PRs ported:
  - #15609 ExeSQL SSRF guard + DNS pin
  - #15436 HTTP timeout on external API tools
  - #16363 be_output restore + DeepL error path
  - #15644 switch no longer matches empty condition
  - #15374 session_id bind to path agent_id (DAO idor guard)
  - #16169 sandbox artifact ownership gate
  - #15457 tenant ownership on agentbots
  - #15145 rerun agent document access check
- #15446 thinking switch (component portion; provider policy lives in
internal/llm)
  - #15426 Invoke URL/proxy SSRF + DNS pin + no-redirects
  - #15238 agentbot thinking-logs beta endpoint
  - #14589 UserFillUp SSE event propagation
  - #14890 anonymous webhook opt-in
- #15068 PipelineChunker new component (text/file_ref/parser_id
dispatch; file-format extraction is a follow-up)

40 files, +2355 / -58 lines. 33 new tests, all targeted package suites
pass (1721 + 4 skipped); 1 pre-existing flaky test unrelated.
2026-06-30 16:28:48 +08:00
Liu An
f86a0e7386 Docs: Update version references to v0.26.2 in READMEs and docs (#16387) 2026-06-29 09:45:16 +08:00
Jack
70250ec88c Fix: remove deepdoc dep (#16372) 2026-06-26 11:32:16 +08:00
Jack
304d9e02bb Refactor: migrate pdf_parser.py to golang (#16323)
### What problem does this PR solve?

Http API based on onnx model.
pdf_parser.py to golang

### Type of change

- [x] Refactoring
2026-06-25 20:16:16 +08:00
Wang Qi
97c519662a Add env ALLOW_ANY_HOST to skip host check (#16351) 2026-06-25 17:17:02 +08:00
Zhichang Yu
2362210caf refactor(log): unify Go logging to zap with rotation, strip per-package levels (#16261)
Refactor the Go agent port's logging so every log line — gin access,
agent canvas events, harness warnings, fatal boot errors — flows through
a single common.Logger (zap) backed by a rotated file, with structured
fields, level filtering, and configurable rotation.

---------

Co-authored-by: Claude <noreply@anthropic.com>
2026-06-23 16:21:46 +08:00
Wang Qi
27a05be643 Fix the launch script (#16159)
Fix the launch script
2026-06-17 20:20:37 +08:00
Liu An
4379269374 Docs: Update version references to v0.26.1 in READMEs and docs (#16158)
### What problem does this PR solve?

- Update version tags in README files (including translations) from
v0.26.0 to v0.26.1
- Modify Docker image references and documentation to reflect new
version
- Update version badges and image descriptions
- Maintain consistency across all language variants of README files

### Type of change

- [x] Documentation Update
2026-06-17 19:35:32 +08:00
Wang Qi
fcb4f78d97 Dev: add go starter (#16138)
Dev: add go starter
2026-06-17 16:09:53 +08:00
Zhichang Yu
e45659868a feat(agent): ship the Go agent canvas port — eino interrupt/resume + Redis check-pointing (#16035)
Replaces the Python agent canvas runtime with a Go implementation that
runs inside `cmd/server_main`.

The canvas compiles into an eino Workflow that pauses on wait-for-user
via native Interrupt/Resume (no sentinel flag) and resumes from a
Redis-backed CheckPointStore.

All 21 Python agent components and ~35 tools are ported with functional
parity.

Sandbox providers now read their JSON config from the admin-panel
system_settings table with env fallback.

234 files / +35,413 / -6,111. All Go files are gofmt-clean (CI gate
added); drops the v2 DSL E2E step and the gap-analysis plan (both
redundant after the port ships).

## Type of change

- [x] Refactoring
- [x] New feature
- [x] Bug fix

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude <noreply@anthropic.com>
2026-06-17 13:24:03 +08:00
Jin Hai
fad82fd1c0 Go: fix register user (#16058)
### What problem does this PR solve?

Fix register user

### Type of change

- [x] Bug Fix (non-breaking change which fixes an issue)

---------

Signed-off-by: Jin Hai <haijin.chn@gmail.com>
2026-06-16 14:03:53 +08:00
Wang Qi
a6f71e0e12 Dev: consoldiate dev script (#16066)
Dev: consoldiate dev script
2026-06-16 11:53:13 +08:00
Wang Qi
f6a2075ad0 Fix one data source can be synced to multiple dataset (#16023)
Fix one data source can be synced to multiple dataset
Test add/delete - worked.
2026-06-15 16:54:25 +08:00
Jin Hai
e96bc37d06 Go: use NATS as the message queue (#15327)
### What problem does this PR solve?

```
RAGFlow(admin)> mq publish 'msg2';
SUCCESS
RAGFlow(admin)> mq publish 'msg3';
SUCCESS
RAGFlow(admin)> mq list;
+---------+---------------+
| message | subject       |
+---------+---------------+
| msg1    | tasks.RAGFLOW |
| msg2    | tasks.RAGFLOW |
| msg3    | tasks.RAGFLOW |
+---------+---------------+
RAGFlow(admin)> mq pull 2;
+---------+---------------+
| message | subject       |
+---------+---------------+
| msg1    | tasks.RAGFLOW |
| msg2    | tasks.RAGFLOW |
+---------+---------------+
RAGFlow(admin)> mq pull noack;
+---------+---------------+
| message | subject       |
+---------+---------------+
| abc     | tasks.RAGFLOW |
+---------+---------------+
RAGFlow(admin)> mq show
+-------------------+----------------+--------+---------------+---------------+-------------------+---------------+
| ack_pending_count | consumer_count | memory | message_count | pending_count | redelivered_count | waiting_count |
+-------------------+----------------+--------+---------------+---------------+-------------------+---------------+
| 2                 | 1              | 0      | 2             | 0             | 1                 | 0             |
+-------------------+----------------+--------+---------------+---------------+-------------------+---------------+

RAGFlow(admin)> list ingestors;
+--------------+-------------------------------------------+--------+
| host         | name                                      | status |
+--------------+-------------------------------------------+--------+
| 192.168.1.38 | ingestor-8f0e4bd5650a4ac58b0151969fbf6935 | alive  |
+--------------+-------------------------------------------+--------+

RAGFlow(admin)> list ingestion tasks;
+----------------------------------+----------------------------------+-----------+------+-------------+----------------------------------+
| document_id                      | id                               | status    | step | user        | user_id                          |
+----------------------------------+----------------------------------+-----------+------+-------------+----------------------------------+
| ffe64fae423411f1a2d938a74640adcc | 90d3d0f6528941c1ac8eb0360effccc4 | COMPLETED | 5    | aaa@aaa.com | 2ba4881420fa11f19e9c38a74640adcc |
+----------------------------------+----------------------------------+-----------+------+-------------+----------------------------------+

RAGFlow(admin)> remove ingestion tasks '90d3d0f6528941c1ac8eb0360effccc4';
+---------+----------------------------------+
| delete  | task_id                          |
+---------+----------------------------------+
| success | 90d3d0f6528941c1ac8eb0360effccc4 |
+---------+----------------------------------+

RAGFlow(admin)> stop ingestion tasks 'e89e20d9a25848a1b79bd9345ddbfe1d';
+----------+----------------------------------+
| status   | task_id                          |
+----------+----------------------------------+
| STOPPING | e89e20d9a25848a1b79bd9345ddbfe1d |
+----------+----------------------------------+

# Publish a message
RAGFlow(admin)> mq publish 'cdd';
SUCCESS

# List current tasks in the message queue
RAGFlow(admin)> mq list
+----------------------------------+---------------+
| message                          | subject       |
+----------------------------------+---------------+
| 7ce392a3c1624cd2be4b5276e8825059 | tasks.RAGFLOW |
+----------------------------------+---------------+

# Consume a task from the message queue
RAGFlow(admin)> mq pull
+------+-----+----------------+
| ack  | id  | type           |
+------+-----+----------------+
| true | cdd | ingestion_test |
+------+-----+----------------+

# User mode
# List ingestion tasks, followed by dataset id
RAGFlow(user)> list ingestion tasks from '0abe79f9423311f1ad8d38a74640adcc';
+---------------------------+---------------+----------------------------------+----------------------------------+----------------------------------+--------+-----------+---------------------------+---------------+----------------------------------+
| create_date               | create_time   | dataset_id                       | document_id                      | id                               | schema | status    | update_date               | update_time   | user_id                          |
+---------------------------+---------------+----------------------------------+----------------------------------+----------------------------------+--------+-----------+---------------------------+---------------+----------------------------------+
| 2026-05-30T20:21:06+08:00 | 1780143666289 | 0abe79f9423311f1ad8d38a74640adcc | ffe64fae423411f1a2d938a74640adcc | 8d758cd14a8b4ba8ab505003fb52017d |        | COMPLETED | 2026-05-30T20:21:26+08:00 | 1780143686431 | 2ba4881420fa11f19e9c38a74640adcc |
+---------------------------+---------------+----------------------------------+----------------------------------+----------------------------------+--------+-----------+---------------------------+---------------+----------------------------------+

RAGFlow(user)> list ingestion tasks;
+---------------------------+---------------+----------------------------------+----------------------------------+----------------------------------+--------+-----------+---------------------------+---------------+----------------------------------+
| create_date               | create_time   | dataset_id                       | document_id                      | id                               | schema | status    | update_date               | update_time   | user_id                          |
+---------------------------+---------------+----------------------------------+----------------------------------+----------------------------------+--------+-----------+---------------------------+---------------+----------------------------------+
| 2026-06-02T19:02:31+08:00 | 1780398151417 | 0abe79f9423311f1ad8d38a74640adcc | ffe64fae423411f1a2d938a74640adcc | e89e20d9a25848a1b79bd9345ddbfe1d |        | COMPLETED | 2026-06-02T19:02:52+08:00 | 1780398172208 | 2ba4881420fa11f19e9c38a74640adcc |
+---------------------------+---------------+----------------------------------+----------------------------------+----------------------------------+--------+-----------+---------------------------+---------------+----------------------------------+

# Create an ingestion task
# First argument is document id, second argument is dataset id
RAGFlow(user)> start ingestion 'ffe64fae423411f1a2d938a74640adcc' from '0abe79f9423311f1ad8d38a74640adcc';
+----------------------------------+-------------------------------------------+
| document_id                      | result                                    |
+----------------------------------+-------------------------------------------+
| ffe64fae423411f1a2d938a74640adcc | task_id: 8d758cd14a8b4ba8ab505003fb52017d |
+----------------------------------+-------------------------------------------+

# Pause an ingestion task, first argument is ingestion id
RAGFlow(user)> stop ingestion '8d758cd14a8b4ba8ab505003fb52017d';
+---------------------------+---------------+----------------------------------+----------------------------------+----------------------------------+--------+-----------+---------------------------+---------------+----------------------------------+
| create_date               | create_time   | dataset_id                       | document_id                      | id                               | schema | status    | update_date               | update_time   | user_id                          |
+---------------------------+---------------+----------------------------------+----------------------------------+----------------------------------+--------+-----------+---------------------------+---------------+----------------------------------+
| 2026-05-30T20:21:06+08:00 | 1780143666289 | 0abe79f9423311f1ad8d38a74640adcc | ffe64fae423411f1a2d938a74640adcc | 8d758cd14a8b4ba8ab505003fb52017d |        | COMPLETED | 2026-05-30T20:21:26+08:00 | 1780143686431 | 2ba4881420fa11f19e9c38a74640adcc |
+---------------------------+---------------+----------------------------------+----------------------------------+----------------------------------+--------+-----------+---------------------------+---------------+----------------------------------+

# Delete an ingestion task
RAGFlow(api/default)> remove ingestion tasks 'f366450a27d54677aec1c7090add30f0';
+---------+----------------------------------+
| remove  | task_id                          |
+---------+----------------------------------+
| success | f366450a27d54677aec1c7090add30f0 |
+---------+----------------------------------+

```

### Type of change

- [x] New Feature (non-breaking change which adds functionality)

---------

Signed-off-by: Jin Hai <haijin.chn@gmail.com>
2026-06-12 14:56:44 +08:00
Liu An
92c4b7688b Docs: Update version references to v0.26.0 in READMEs and docs (#15941)
### What problem does this PR solve?

- Update version tags in README files (including translations) from
v0.25.6 to v0.26.0
- Modify Docker image references and documentation to reflect new
version
- Update version badges and image descriptions
- Maintain consistency across all language variants of README files

### Type of change

- [x] Documentation Update
2026-06-11 18:34:26 +08:00
Idriss Sbaaoui
faf78d3069 Fix: OceanBase tenant startup drift and docs (#15829)
### What problem does this PR solve?

OceanBase could start without the `ragflow` tenant, so RAGFlow failed to
connect with `root@ragflow`. This PR adds a safe startup reconcile step
and documents the required host limits before using OceanBase.

### Type of change

- [x] Bug Fix (non-breaking change which fixes an issue)
- [x] Documentation Update
2026-06-09 13:04:05 +08:00