zlei6/ragflow
1
0
Fork 0
mirror of https://github.com/infiniflow/ragflow.git synced 2026-07-23 17:06:42 +08:00
Code Issues Packages Projects Releases Wiki Activity
7,614 Commits 2 Branches 81 Tags
64be8585aa64449d37a72fe5818ec8a531cb0706
Commit Graph

93 Commits

Author SHA1 Message Date
Jin Hai
5478f955ab Go: add context to DAO (#17269) 
### Summary

DAO layer doesn't use context, this PR is to fix it.

---------

Signed-off-by: Jin Hai <haijin.chn@gmail.com>
 2026-07-23 12:15:58 +08:00
euvre
7b29fc10ca fix(agent): wire memory persistence in Message component (Go runtime) (#17256) 2026-07-23 11:37:59 +08:00
Jin Hai
1fdf167f79 Go: refactor system stats (#17089) 
### Summary

Move stats to a specific service from system

---------

Signed-off-by: Jin Hai <haijin.chn@gmail.com>
 2026-07-20 14:13:05 +08:00
Jack
965590ccbe Refactor: dataset/document/file service (#17071) 
### Summary

Refactor dataset.go document.do file.go file2document.go in
internal/service.
 2026-07-20 09:48:24 +08:00
Jin Hai
20f866e703 Go: refactor (#17072) 
Refactor stats

---------

Signed-off-by: Jin Hai <haijin.chn@gmail.com>
 2026-07-19 09:26:44 +08:00
Jack
10c00a9614 Feat: add built in DSL file API (#17003) 
### Summary

1. list and get by id API for builtin DSL
2. add DSL default component param values validation
3. remove all hard code keys for parser config
 2026-07-18 11:00:08 +08:00
Jin Hai
cf71c7193a Go: update server config (#17027) 
### Summary

```
RAGFlow(admin)> list services;
+--------+--------------+----+---------------------+------+---------------+-----------+
| enable | host         | id | name                | port | service_type  | status    |
+--------+--------------+----+---------------------+------+---------------+-----------+
|        | localhost    | 0  | redis               | 6379 | cache         | alive     |
|        | localhost    | 1  | minio               | 9000 | file_store    | alive     |
|        | localhost    | 2  | elasticsearch       | 1200 | retrieval     | alive     |
|        | localhost    | 3  | mysql               | 3306 | meta_data     | alive     |
| false  | localhost    | 4  | jaeger              | 4318 | tracing       | unknown   |
|        | localhost    | 5  | clickhouse          | 9900 | olap          | unknown   |
|        | localhost    | 6  | nats                | 4222 | message_queue | CONNECTED |
|        | 192.168.1.68 | 7  | ragflow-server-9384 | 9384 | api_server    | alive     |
+--------+--------------+----+---------------------+------+---------------+-----------+

```

Signed-off-by: Jin Hai <haijin.chn@gmail.com>
 2026-07-17 13:54:19 +08:00
Jin Hai
f8474a67f5 Go: tracing framework (#17004) 
### Summary

Tracing framework

---------

Signed-off-by: Jin Hai <haijin.chn@gmail.com>
 2026-07-16 19:35:43 +08:00
Jin Hai
c7a623ff81 Go: introduce clickhouse (#16996) 
### Summary

Introduce Clickhouse for data statistics.

Signed-off-by: Jin Hai <haijin.chn@gmail.com>
 2026-07-16 18:36:56 +08:00
Jin Hai
2a6e210020 Go: align db schema to python (#16935) 
As title.

---------

Signed-off-by: Jin Hai <haijin.chn@gmail.com>
 2026-07-15 23:05:06 +08:00
Jack
e543ff02c4 Feat: nats message processing refactor and support built in DSL (#16873) 2026-07-15 14:53:16 +08:00
Jin Hai
b506d8ec5f Go: add more frameworks (#16915) 
### Summary

1. Add hooks for server init
2. Add hooks for router init
3. Add jaeger and otel related config

Signed-off-by: Jin Hai <haijin.chn@gmail.com>
 2026-07-14 22:20:26 +08:00
Jin Hai
8bc18154d2 Go: refactor and add version type (#16863) 
### Summary

```
RAGFlow(admin)> show version;
+--------------+-----------------------+
| field        | value                 |
+--------------+-----------------------+
| version      | v0.26.4-84-g547bc8614 |
| version_type | open source           |
+--------------+-----------------------+
```

---------

Signed-off-by: Jin Hai <haijin.chn@gmail.com>
 2026-07-13 18:00:35 +08:00
Jack
cfacaccad7 Refactor: message processing (#16852) 
### Summary

1. refactor message processing
2. delete un-used componentIndexMap
3. unfold (delete) internal/ingestion/task/task_handler.go
 2026-07-13 16:32:34 +08:00
Jack
3c0f8fc192 Refactor: refactor dataflow_service.go and guard nats message re-delivery (#16826) 
### Summary

1. refactor dataflow_service.go 
2. guard nats message re-delivery
3. support document parse cancelling & re-run
 2026-07-13 11:08:04 +08:00
Zhichang Yu
fb42e5531d Refactor: drop dead canvas runtime selector and tokenizer embedding wiring (#16809) 
Two refactors on the Go port (agent-go-port):

- Remove the dead per-tenant canvas runtime selector (write-only Redis
scaffolding with no runtime callers) and its dependent metrics/admin
code.
- Move the tokenizer embedding-model id from the shared ingestion
globals into a Tokenizer-scoped setup, and wire the production embedder
resolver in the ingestion task package.

32 files changed, 861 insertions, 1228 deletions.
 2026-07-10 15:46:45 +08:00
Jin Hai
0b01171a86 Go: Update response message (#16803) 
### Summary

As title.

---------

Signed-off-by: Jin Hai <haijin.chn@gmail.com>
 2026-07-10 14:26:54 +08:00
Jin Hai
15d1ee7654 Go: set gin mode in admin server (#16801) 
### Summary

As title.

---------

Signed-off-by: Jin Hai <haijin.chn@gmail.com>
 2026-07-10 12:19:10 +08:00
Jin Hai
add7b9486f Go: merge duplicate codes (#16783) 
### Summary

1. merge heartbeat function.
2. introduce all environments

---------

Signed-off-by: Jin Hai <haijin.chn@gmail.com>
 2026-07-10 11:58:32 +08:00
Jack
0dd0ac06f8 Feature: task executor migration to go (#16549) 
### Summary

Feature: Integrate parser
 2026-07-08 19:08:11 +08:00
Jin Hai
21266286cb Go: add more commands and GCS supports (#16741) 
### Summary

1. GCS supports
2. More commands
```
RAGFlow(admin)> ping store;
SUCCESS
RAGFlow(admin)> ping engine;
SUCCESS
RAGFlow(admin)> ping cache;
SUCCESS
```

---------

Signed-off-by: Jin Hai <haijin.chn@gmail.com>
 2026-07-08 17:49:02 +08:00
Jin Hai
568bac673b Go: add migrate database flag (#16719) 
### Summary

--migrate will trigger database migration
```
./ragflow_main --api --migrate
```

Signed-off-by: Jin Hai <haijin.chn@gmail.com>
 2026-07-07 20:11:40 +08:00
Jin Hai
7df7384b21 Go: refactor UUID functions (#16695) 
As title

---------

Signed-off-by: Jin Hai <haijin.chn@gmail.com>
 2026-07-07 17:22:08 +08:00
Zhichang Yu
014c3f634f Align Go ingestion boundaries with Python (#16647) 
Moves doc_id blob resolution into Parser, tightens chunker/tokenizer to
Python output_format semantics, updates extractor list handling, and
fixes real-template integration tests.
 2026-07-05 20:43:52 +08:00
Jin Hai
1880e65e99 Go: refactor (#16602) 
### Summary

1. update doc
2. refactor route code

---------

Signed-off-by: Jin Hai <haijin.chn@gmail.com>
 2026-07-03 17:00:43 +08:00
Jin Hai
1aa8abe373 Go: file syncer service framework (#16579) 
### Summary

./ragflow_main --syncer to start file syncer


config yaml file has following config
```
file_syncer:
  max_concurrent_syncs: 4 # concurrent file sync threads
  sync_interval: 3 # check interval

```

---------

Signed-off-by: Jin Hai <haijin.chn@gmail.com>
 2026-07-03 11:14:02 +08:00
Jin Hai
7d64a78f83 Go: unify three services into one binary (#16462) 
### Summary

Plan to start api_server, admin_server and ingestor in one binary:
- ./ragflow_main --admin
- ./ragflow_main --api
- ./ragflow_main --ingestor

---------

Signed-off-by: Jin Hai <haijin.chn@gmail.com>
 2026-07-02 21:21:10 +08:00
qinling0210
133b1e15fd Port agent PRs to GO (#16529) 
### Summary

Port the following PRs to GO in this PR

https://github.com/infiniflow/ragflow/pull/14210
https://github.com/infiniflow/ragflow/pull/14641
https://github.com/infiniflow/ragflow/pull/15220
https://github.com/infiniflow/ragflow/pull/15228
https://github.com/infiniflow/ragflow/pull/15384
https://github.com/infiniflow/ragflow/pull/15754
https://github.com/infiniflow/ragflow/pull/16413
https://github.com/infiniflow/ragflow/pull/16483
https://github.com/infiniflow/ragflow/pull/16419
https://github.com/infiniflow/ragflow/pull/16361   
https://github.com/infiniflow/ragflow/pull/16050
 2026-07-02 09:45:01 +08:00
Wang Qi
8f24b30652 [Go] Add API /api/v1/chat/recommendation and consolidate with /api/v1/searchbots/related_questions (#16500) 2026-07-01 13:17:16 +08:00
Wang Qi
3bb976b383 [Go] Add /api/v1/searchbots/mindmap and /api/v1/chat/mindmap (#16443) 2026-06-30 16:35:33 +08:00
Zhichang Yu
4c54cefd29 Port 14 upstream agent security / correctness fixes to Go canvas (#16455) 
Mirrors 14 merged upstream PRs into the Go agent port.

PRs ported:
  - #15609 ExeSQL SSRF guard + DNS pin
  - #15436 HTTP timeout on external API tools
  - #16363 be_output restore + DeepL error path
  - #15644 switch no longer matches empty condition
  - #15374 session_id bind to path agent_id (DAO idor guard)
  - #16169 sandbox artifact ownership gate
  - #15457 tenant ownership on agentbots
  - #15145 rerun agent document access check
- #15446 thinking switch (component portion; provider policy lives in
internal/llm)
  - #15426 Invoke URL/proxy SSRF + DNS pin + no-redirects
  - #15238 agentbot thinking-logs beta endpoint
  - #14589 UserFillUp SSE event propagation
  - #14890 anonymous webhook opt-in
- #15068 PipelineChunker new component (text/file_ref/parser_id
dispatch; file-format extraction is a follow-up)

40 files, +2355 / -58 lines. 33 new tests, all targeted package suites
pass (1721 + 4 skipped); 1 pre-existing flaky test unrelated.
 2026-06-30 16:28:48 +08:00
Haruko386
1c0cdd84ce feat[Go]: implement searches/<search_id>/completions POST (#16440) 
### Summary

As title

### Type of change

- [x] New Feature (non-breaking change which adds functionality)
- [x] Bug Fix (non-breaking change which fixes an issue)
 2026-06-29 20:07:12 +08:00
Zhichang Yu
dfe2dc346d feat[Go]: port agent attachment download, chatbot + agentbot completion/info endpoints from Python (#16405) 
## Summary

Ports five Python agent APIs to Go under the v1 Gin router:

- `GET  /api/v1/agents/attachments/<attachment_id>/download`
- `POST /api/v1/chatbots/<dialog_id>/completions`  (SSE)
- `GET  /api/v1/chatbots/<dialog_id>/info`
- `POST /api/v1/agentbots/<agent_id>/completions` (SSE)
- `GET  /api/v1/agentbots/<agent_id>/inputs`

Mirrors the existing Python wire shape (`{code, message,
data:{answer,reference,...}}` per Python `canvas_service.completion`) so
the iframe SDK and existing JS widgets keep working.

## Behavioural parity with Python

| # | Concern | How it's met |
|---|---------|--------------|
| R0 | Bot routes must not require regular user session | Routes mount
on `apiNoAuth` (router.go:198-202), with `BetaAuthMiddleware` only |
| R3 | Two SSE formats in Go drift | F2: `AgentChatCompletions` and
`AgentbotCompletion` share `service.WriteChatbotRunEvent` |
| R7 | `GetBySessionID` returns `(nil, nil)` on miss | Defensive
nil-check before `session.UserID != tenantID` |
| R8 | Begin component name vs ID | `FindBeginComponentID` resolves name
→ ID first, then `ExtractComponentInputForm(dsl, beginID)` |
| R9 | Defensive PromptConfig parsing | `stringFromMap` helper used for
`prologue` and `tavily_api_key` |
| R10 | `BetaAuthMiddleware` Bearer-prefix pre-filter | Removed —
`GetUserByToken` is called unconditionally, falls back to
`GetUserByBetaAPIToken` |
| F8 | Multi-turn chatbot history | `ChatbotCompletion` reads prior
turns from `session.Message`, appends user turn, calls LLM, persists new
pair via new `API4ConversationDAO.Update` |
| F9 | UUID gate stricter than plan | Removed — only `filepath.Base` +
CR/LF/quote header sanitization remains |
| H2 | Defence-in-depth IDOR | `AgentbotCompletion` calls `loadCanvas`
before delegating to `RunAgent` |
| M2 | SSE error leakage | `WriteChatbotFrame` emits generic `"an
internal error occurred"`; real error logged via `common.Error` |

## Verification

```bash
$ go vet ./...                                     # clean (only pre-existing issues)
$ go build ./...                                   # success
$ go test ./internal/handler/ ./internal/service/ ./internal/agent/dsl/ ./internal/common/ ./internal/dao/
ok  ragflow/internal/handler     0.617s
ok  ragflow/internal/service     1.729s
ok  ragflow/internal/agent/dsl   0.008s
ok  ragflow/internal/common      0.087s
ok  ragflow/internal/dao         0.083s
```

1199 tests pass across 5 packages.

## Known follow-ups (out of scope for this PR)

- **F1**: token-level streaming in `ChatbotCompletion` (currently emits
one frame per turn)
- **F3**: per-route `auth_types` attribute in Go (currently applied via
route group middleware)

---------

Co-authored-by: Claude <noreply@anthropic.com>
 2026-06-29 09:45:16 +08:00
Zhichang Yu
f58fae5fb7 feat(go-agent): Ported retrieval node, added Keenable web search tool (#16396) 
Ported retrieval node, added Keenable web search tool
- [x] New Feature (non-breaking change which adds functionality)
 2026-06-29 09:45:16 +08:00
Hz_
e290a0d23e feat(go-api): Langfuse API key migration behavior (#16356) 
## Summary

- Align Langfuse API key set/get/delete behavior with the Python
implementation.
- Improve DAO handling for Langfuse credential save/delete flows.
- Add tests for Langfuse service error handling and API key lifecycle
behavior.
 2026-06-25 19:25:55 +08:00
Jin Hai
919f596066 Fix release (#16278) 
### What problem does this PR solve?

Fix release

### Type of change

- [x] Bug Fix (non-breaking change which fixes an issue)

---------

Signed-off-by: Jin Hai <haijin.chn@gmail.com>
 2026-06-23 22:04:34 +08:00
Zhichang Yu
2362210caf refactor(log): unify Go logging to zap with rotation, strip per-package levels (#16261) 
Refactor the Go agent port's logging so every log line — gin access,
agent canvas events, harness warnings, fatal boot errors — flows through
a single common.Logger (zap) backed by a rotated file, with structured
fields, level filtering, and configurable rotation.

---------

Co-authored-by: Claude <noreply@anthropic.com>
 2026-06-23 16:21:46 +08:00
Hz_
4e0db3053d feat(go-api): complete chat channel API migration with tests (#16139) 
close #16132

## Summary

This PR completes the Go-side merge and cleanup for chat channel APIs,
including handler/service wiring, route registration, and test coverage.

Implemented and aligned 5 chat channel APIs:

```
- POST `/api/v1/chat-channels`
- GET `/api/v1/chat-channels`
- GET `/api/v1/chat-channels/:channel_id`
- PATCH `/api/v1/chat-channels/:channel_id`
- DELETE `/api/v1/chat-channels/:channel_id`
```


Co-authored-by: Haruko386 <tryeverypossible@163.com>
 2026-06-22 18:16:15 +08:00
Zhichang Yu
3f805a64f1 feat(agent): align Go agent behavior with Python (except retrieval component) (#16225) 
## Summary

Aligns the **Go agent runtime/canvas/components/tools** behavior with
the **Python `agent/` implementation** so the same stored canvas DSL
produces the same execution result on either side. Every component,
tool, and runtime primitive in `internal/agent/` is now driven by the
same semantics as its Python counterpart — variable resolution, template
substitution, control flow, error reporting, retry/cancel, and stream
event shapes.

The **retrieval component is the one explicit exception** in this PR. It
is being reworked in a separate change and is excluded from this
alignment pass; the wrapper slot (`universe_a_wrappers.go →
newRetrievalComponent`) is preserved.

## Scope of alignment

### Components (all aligned with `agent/component/`)
`Begin` · `Message` · `LLM` (incl. ChatTemplateKwargs,
MessageHistoryWindowSize, VisualFiles, Cite, OutputStructure,
JSONOutput, TopP, MaxRetries, DelayAfterError, credentials) · `Agent`
(react + tool artifact capture + `Reset()` interface-assert) · `Switch`
(12/12 operators, Python-equivalent semantics) · `Categorize` · `Invoke`
· `Iteration` · `Loop` (macro-expansion through `workflowx.AddLoopNode`)
· `UserFillUp` (Python-equivalent interrupt/resume via eino
`compose.Interrupt`/`ResumeWithData`) · `FillUp` · `DataOperations` ·
`ListOperations` · `StringTransform` · `VariableAggregator` ·
`VariableAssigner` · `Browser` (full stagehand runtime parity) ·
`DocsGenerator` · `ExcelProcessor`.

### Tools (all aligned with `agent/tools/`)
`Retrieval` (wrapper slot only — logic out of scope) · `MCPToolAdapter`
(streamable-HTTP) · `CodeExec` (sandbox bridge with
`code_exec_contract.go` matching Python contract) · `AkShare` · `ArXiv`
· `Crawler` · `DeepL` · `DuckDuckGo` · `Email` · `ExeSQL` · `GitHub` ·
`Google` · `GoogleScholar` · `Jin10` · `PubMed` · `QWeather` · `SearXNG`
· `Tavily` · `Tushare` · `Wencai` · `Wikipedia` · `YahooFinance` —
uniform `eino tool.InvokableTool` interface, SSRF protection, shared
HTTP client.

### Canvas execution engine (`internal/agent/canvas/`)
Aligned with Python's `agent/canvas.py`:
- **Scheduler** (`scheduler.go`): state pre/post handlers, node lambdas,
per-component timeout resolver (4-level: per-class env → per-class table
→ uniform env → 600s fallback), `legacyNoOpNames`.
- **Loop subgraph** (`loop_subgraph.go`): Python-equivalent
`AddLoopNode` macro expansion + condition translation.
- **Multibranch** (`multibranch.go`): `Switch` / `Categorize` routing
via `compose.NewGraphMultiBranch` — same branch selection semantics as
Python.
- **Parallel subgraph** (`parallel_subgraph.go`): matches Python's
parallel fan-out contract.
- **Interrupt/Resume** (`interrupt_resume.go`): `UserFillUpNodeBody` /
`IsInterruptError` / `ExtractInterruptContexts` — replaces the
deprecated Python sentinel chain with eino's native interrupt API,
preserving the same external behavior.
- **Checkpoint** (`checkpoint_store.go`): `RedisCheckPointStore`
Get/Set/Delete, with business metadata (status / canvas_id /
parent_run_id) on a parallel Redis Hash.
- **RunTracker** (`run_tracker.go`): Start / MarkSucceeded / MarkFailed
/ MarkCancelled / AttachCheckpoint — same lifecycle as the Python run
record.
- **Cancel** (`cancel.go`): Redis pub/sub watch.
- **Stream** (`stream.go`): SSE channel with `messages` / `waiting` /
`errors` / `done` events, same shape as Python's `agent.canvas.RunEvent`
payload.

### DSL bridge (`internal/agent/dsl/`)
- `normalize.go`: v1↔v2 collapsed into a single wire format — Python and
Go consume the same stored JSON.
- `reset.go`: per-run state reset matches Python's `Canvas.reset()`
semantics.
- Testdata mirrors Python's `agent_msg.json` / `all.json` / etc.

### Runtime (`internal/agent/runtime/`)
- `CanvasState` / `NewCanvasState` / `GetVar` / `SetVar` / `ReadVars`:
same `{{cpn_id@param}}` resolution model.
- `ResolveTemplate` (regex fast path + gonja fallback) — Python
Jinja-style semantics.
- `selector.go`, `metrics.go`, `component.go`: shared runtime contracts.

## Out of scope (intentionally)

- **`Retrieval` component logic** — wrapped only; full parity lands in a
follow-up PR.
- **Frontend** — only minor dsl-bridge / canvas UX fixes ride along.
- **CLI / admin / model registry** — orthogonal to agent behavior.

## How alignment is verified

`internal/service/agent_run_e2e_test.go` exercises the **full production
chain** against real Python-shaped DSL fixtures:
```
loadCanvasForUser → versionDAO.GetLatest → decodeCanvasFromDSL →
canvas.Compile → cc.Workflow.Invoke → answer extraction
```
using in-memory SQLite + miniredis (no Docker). Covers:
- `TestRunAgent_RealCanvas_BeginMessage` — happy path, `{{sys.query}}`
resolution
- `TestRunAgent_RealCanvas_WaitForUserResume` — two-run resume cycle
(Python-equivalent)
- `TestRunAgent_RealCanvas_CompileFails` — unknown component name →
sanitized error (Python-equivalent)
- `TestRunAgent_RealCanvas_InvokeFails` — unresolvable template ref
(Python-equivalent)
- `TestRunAgent_RunTracker_AttachCheckpoint_CallSequence` —
Start→AttachCheckpoint→MarkSucceeded lifecycle

`internal/handler/agent_test.go` — SSE streaming parity (`Content-Type:
text/event-stream`, `data: {…}\n\n`, trailing `data: [DONE]\n\n`,
OpenAI-compatible non-stream `choices`).

`internal/agent/canvas/fixture_compile_test.go` + per-component tests
pin the Python-equivalent outputs.

```
go test -count=1 -v -run 'TestRunAgent_RealCanvas|TestRunAgent_RunTracker' ./internal/service/
```

## Design reference

`docs/develop/agent-go-port-design.md` (1329 lines, last cross-checked
2026-06-17) — module layout, per-component / per-tool inventory,
corner-case catalogue, and the actionable backlog (Section 14, including
the retrieval alignment follow-up).

---------

Co-authored-by: Claude <noreply@anthropic.com>
 2026-06-22 11:58:29 +08:00
qinling0210
563d855780 Implement OpenAI chat completions in GO (#16177) 
### What problem does this PR solve?

Implement OpenAI chat completions in GO

POST /api/v1/openai/<chat_id>/chat/completions

OpenAI chat cli: internal/development.md

### Type of change

- [x] Refactoring
 2026-06-18 18:07:27 +08:00
Zhichang Yu
e45659868a feat(agent): ship the Go agent canvas port — eino interrupt/resume + Redis check-pointing (#16035) 
Replaces the Python agent canvas runtime with a Go implementation that
runs inside `cmd/server_main`.

The canvas compiles into an eino Workflow that pauses on wait-for-user
via native Interrupt/Resume (no sentinel flag) and resumes from a
Redis-backed CheckPointStore.

All 21 Python agent components and ~35 tools are ported with functional
parity.

Sandbox providers now read their JSON config from the admin-panel
system_settings table with env fallback.

234 files / +35,413 / -6,111. All Go files are gofmt-clean (CI gate
added); drops the v2 DSL E2E step and the gap-analysis plan (both
redundant after the port ships).

## Type of change

- [x] Refactoring
- [x] New feature
- [x] Bug fix

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude <noreply@anthropic.com>
 2026-06-17 13:24:03 +08:00
Jin Hai
6865039a22 Go: add more start server parameters (#16093) 
### What problem does this PR solve?

```
$ ./bin/ragflow_server --version 
RAGFlow version: v0.26.0-65-g549f6109c

$ ./bin/ragflow_server --debug # start server with debug log level

$ ./bin/admin_server --version 
RAGFlow version: v0.26.0-65-g549f6109c

$ ./bin/admin_server --debug # start server with debug log level

$ ./bin/admin_server --init-superuser # init default superuser

$ ./bin/ingestor --version
RAGFlow version: v0.26.0-68-g6f6c39706

$ ./bin/ingestor --debug
```


### Type of change

- [x] New Feature (non-breaking change which adds functionality)

---------

Signed-off-by: Jin Hai <haijin.chn@gmail.com>
 2026-06-16 20:27:37 +08:00
Jin Hai
417f805bd9 Go: add API mode check in file system command (#16022) 
### What problem does this PR solve?

As title.

### Type of change

- [x] Bug Fix (non-breaking change which fixes an issue)

Signed-off-by: Jin Hai <haijin.chn@gmail.com>
 2026-06-15 16:37:47 +08:00
Jin Hai
fcebcebe1e Move REDIS to engine dir (#16006) 
### What problem does this PR solve?

as title.

### Type of change

- [x] Refactoring

Signed-off-by: Jin Hai <haijin.chn@gmail.com>
 2026-06-15 14:44:16 +08:00
Yingfeng
b5bea72e4b Add git-like file commit API (#15978) 
### What problem does this PR solve?

| # | Method | Endpoint | Description | Git Equivalent |
|---|--------|----------|-------------|----------------|
| 1 | `POST` | `/api/v1/{prefix}/{folder_id}/commits` | Create a
snapshot commit with file changes (add/modify/delete/rename) | `git add`
+ `git commit` |
| 2 | `GET` | `/api/v1/{prefix}/{folder_id}/commits` | List commit
history (paginated) | `git log` |
| 3 | `GET` | `/api/v1/{prefix}/{folder_id}/commits/{commit_id}` | Get
commit detail with file changes | `git show` |
| 4 | `GET` | `/api/v1/{prefix}/{folder_id}/commits/{commit_id}/files` |
List file changes in a commit | `git show --name-status` |
| 5 | `GET` |
`/api/v1/{prefix}/{folder_id}/commits/diff?from=...&to=...` | Compare
two commits and return differences | `git diff` |
| 6 | `GET` | `/api/v1/{prefix}/{folder_id}/changes` | Get uncommitted
changes (add/modify/delete) | `git status` |
| 7 | `GET` | `/api/v1/{prefix}/{folder_id}/commits/{commit_id}/tree` |
Get the folder tree snapshot at commit time | `git ls-tree` |
| 8 | `GET` |
`/api/v1/{prefix}/{folder_id}/commits/{commit_id}/files/{file_id}/content`
| Get a file's content as it existed in a specific commit | `git show
HEAD:file` |
| 9 | `GET` | `/api/v1/{prefix}/{file_id}/versions` | Get version
history for a specific file across all commits | `git log -- file` |

Where `{prefix}/{id}` can be:
- `folders/{folder_id}` — direct folder access
- `workspaces/{workspace_id}` — alias of `folders/{folder_id}`
- `datasets/{dataset_id}` — resolves to the dataset's folder
- `memories/{memory_id}` — resolves to the memory's folder
- `skills/{skill_id}` — resolves to the skill's folder

### Type of change

- [x] New Feature (non-breaking change which adds functionality)
- [x] Documentation Update
 2026-06-15 11:19:56 +08:00
Zhichang Yu
3fa15c0e2f feat(agent): Go port — canvas engine, 22 components, DSL v2, 13 endpoints (#15952) 
Ports the agent canvas subsystem from Python to Go.

## What's included

### Canvas Engine (Phase 0/1)
- State engine, scheduler, variable resolver, Redis checkpoint store,
cancel protocol
- **209 tests** across canvas / component / io packages

### 22 Components (P0–P4)
| Tier | Components |
|---|---|
| P0 T1+T2+T3 | LLM, Agent, ExitLoop, Switch, Categorize, Begin,
Message, Invoke |
| P1 T3 | VariableAggregator, VariableAssigner, StringTransform,
ListOperations, DataOperations |
| P2 T3 | Iteration, IterationItem, Loop, LoopItem |
| P3 T3 | UserFillUp, Fillup |
| P4 T5 | Browser, ExcelProcessor, DocsGenerator |

### DSL v2 Schema (Phase 2.5)
- Typed v2 in-memory model with v1-to-v2 auto-detect converter
- v1 legacy field stripping per plan §2.11.7

### HTTP Endpoints & Bug Fixes (Plans PR1–PR3)
- **DELETE SQL bug fix**: gorm v2 `Where("id = ?", id).Delete(...)`
pattern
- **CreateAgent validation**: title/DSL required, duplicate check, 103
envelope
- **13 new endpoints**: templates, prompts, tags, sessions CRUD,
chat/completions (SSE + non-stream stubs), rerun, test_db_connection,
logs, webhook/logs
- **756 Go unit tests** (745 → 756, +18)
- **17 → 0 Python integration test failures** (test_agents.py +
test_session_management/)

### Tools
21 eino tools: HTTPHelper, search tools, financial/data tools, mandatory
stubs

### Infrastructure
OTel observability, NATS message queue, DeepDoc gRPC client, SSRF
guards, IDOR mitigation
 2026-06-12 22:58:28 +08:00
Jin Hai
e96bc37d06 Go: use NATS as the message queue (#15327) 
### What problem does this PR solve?

```
RAGFlow(admin)> mq publish 'msg2';
SUCCESS
RAGFlow(admin)> mq publish 'msg3';
SUCCESS
RAGFlow(admin)> mq list;
+---------+---------------+
| message | subject       |
+---------+---------------+
| msg1    | tasks.RAGFLOW |
| msg2    | tasks.RAGFLOW |
| msg3    | tasks.RAGFLOW |
+---------+---------------+
RAGFlow(admin)> mq pull 2;
+---------+---------------+
| message | subject       |
+---------+---------------+
| msg1    | tasks.RAGFLOW |
| msg2    | tasks.RAGFLOW |
+---------+---------------+
RAGFlow(admin)> mq pull noack;
+---------+---------------+
| message | subject       |
+---------+---------------+
| abc     | tasks.RAGFLOW |
+---------+---------------+
RAGFlow(admin)> mq show
+-------------------+----------------+--------+---------------+---------------+-------------------+---------------+
| ack_pending_count | consumer_count | memory | message_count | pending_count | redelivered_count | waiting_count |
+-------------------+----------------+--------+---------------+---------------+-------------------+---------------+
| 2                 | 1              | 0      | 2             | 0             | 1                 | 0             |
+-------------------+----------------+--------+---------------+---------------+-------------------+---------------+

RAGFlow(admin)> list ingestors;
+--------------+-------------------------------------------+--------+
| host         | name                                      | status |
+--------------+-------------------------------------------+--------+
| 192.168.1.38 | ingestor-8f0e4bd5650a4ac58b0151969fbf6935 | alive  |
+--------------+-------------------------------------------+--------+

RAGFlow(admin)> list ingestion tasks;
+----------------------------------+----------------------------------+-----------+------+-------------+----------------------------------+
| document_id                      | id                               | status    | step | user        | user_id                          |
+----------------------------------+----------------------------------+-----------+------+-------------+----------------------------------+
| ffe64fae423411f1a2d938a74640adcc | 90d3d0f6528941c1ac8eb0360effccc4 | COMPLETED | 5    | aaa@aaa.com | 2ba4881420fa11f19e9c38a74640adcc |
+----------------------------------+----------------------------------+-----------+------+-------------+----------------------------------+

RAGFlow(admin)> remove ingestion tasks '90d3d0f6528941c1ac8eb0360effccc4';
+---------+----------------------------------+
| delete  | task_id                          |
+---------+----------------------------------+
| success | 90d3d0f6528941c1ac8eb0360effccc4 |
+---------+----------------------------------+

RAGFlow(admin)> stop ingestion tasks 'e89e20d9a25848a1b79bd9345ddbfe1d';
+----------+----------------------------------+
| status   | task_id                          |
+----------+----------------------------------+
| STOPPING | e89e20d9a25848a1b79bd9345ddbfe1d |
+----------+----------------------------------+

# Publish a message
RAGFlow(admin)> mq publish 'cdd';
SUCCESS

# List current tasks in the message queue
RAGFlow(admin)> mq list
+----------------------------------+---------------+
| message                          | subject       |
+----------------------------------+---------------+
| 7ce392a3c1624cd2be4b5276e8825059 | tasks.RAGFLOW |
+----------------------------------+---------------+

# Consume a task from the message queue
RAGFlow(admin)> mq pull
+------+-----+----------------+
| ack  | id  | type           |
+------+-----+----------------+
| true | cdd | ingestion_test |
+------+-----+----------------+

# User mode
# List ingestion tasks, followed by dataset id
RAGFlow(user)> list ingestion tasks from '0abe79f9423311f1ad8d38a74640adcc';
+---------------------------+---------------+----------------------------------+----------------------------------+----------------------------------+--------+-----------+---------------------------+---------------+----------------------------------+
| create_date               | create_time   | dataset_id                       | document_id                      | id                               | schema | status    | update_date               | update_time   | user_id                          |
+---------------------------+---------------+----------------------------------+----------------------------------+----------------------------------+--------+-----------+---------------------------+---------------+----------------------------------+
| 2026-05-30T20:21:06+08:00 | 1780143666289 | 0abe79f9423311f1ad8d38a74640adcc | ffe64fae423411f1a2d938a74640adcc | 8d758cd14a8b4ba8ab505003fb52017d |        | COMPLETED | 2026-05-30T20:21:26+08:00 | 1780143686431 | 2ba4881420fa11f19e9c38a74640adcc |
+---------------------------+---------------+----------------------------------+----------------------------------+----------------------------------+--------+-----------+---------------------------+---------------+----------------------------------+

RAGFlow(user)> list ingestion tasks;
+---------------------------+---------------+----------------------------------+----------------------------------+----------------------------------+--------+-----------+---------------------------+---------------+----------------------------------+
| create_date               | create_time   | dataset_id                       | document_id                      | id                               | schema | status    | update_date               | update_time   | user_id                          |
+---------------------------+---------------+----------------------------------+----------------------------------+----------------------------------+--------+-----------+---------------------------+---------------+----------------------------------+
| 2026-06-02T19:02:31+08:00 | 1780398151417 | 0abe79f9423311f1ad8d38a74640adcc | ffe64fae423411f1a2d938a74640adcc | e89e20d9a25848a1b79bd9345ddbfe1d |        | COMPLETED | 2026-06-02T19:02:52+08:00 | 1780398172208 | 2ba4881420fa11f19e9c38a74640adcc |
+---------------------------+---------------+----------------------------------+----------------------------------+----------------------------------+--------+-----------+---------------------------+---------------+----------------------------------+

# Create an ingestion task
# First argument is document id, second argument is dataset id
RAGFlow(user)> start ingestion 'ffe64fae423411f1a2d938a74640adcc' from '0abe79f9423311f1ad8d38a74640adcc';
+----------------------------------+-------------------------------------------+
| document_id                      | result                                    |
+----------------------------------+-------------------------------------------+
| ffe64fae423411f1a2d938a74640adcc | task_id: 8d758cd14a8b4ba8ab505003fb52017d |
+----------------------------------+-------------------------------------------+

# Pause an ingestion task, first argument is ingestion id
RAGFlow(user)> stop ingestion '8d758cd14a8b4ba8ab505003fb52017d';
+---------------------------+---------------+----------------------------------+----------------------------------+----------------------------------+--------+-----------+---------------------------+---------------+----------------------------------+
| create_date               | create_time   | dataset_id                       | document_id                      | id                               | schema | status    | update_date               | update_time   | user_id                          |
+---------------------------+---------------+----------------------------------+----------------------------------+----------------------------------+--------+-----------+---------------------------+---------------+----------------------------------+
| 2026-05-30T20:21:06+08:00 | 1780143666289 | 0abe79f9423311f1ad8d38a74640adcc | ffe64fae423411f1a2d938a74640adcc | 8d758cd14a8b4ba8ab505003fb52017d |        | COMPLETED | 2026-05-30T20:21:26+08:00 | 1780143686431 | 2ba4881420fa11f19e9c38a74640adcc |
+---------------------------+---------------+----------------------------------+----------------------------------+----------------------------------+--------+-----------+---------------------------+---------------+----------------------------------+

# Delete an ingestion task
RAGFlow(api/default)> remove ingestion tasks 'f366450a27d54677aec1c7090add30f0';
+---------+----------------------------------+
| remove  | task_id                          |
+---------+----------------------------------+
| success | f366450a27d54677aec1c7090add30f0 |
+---------+----------------------------------+

```

### Type of change

- [x] New Feature (non-breaking change which adds functionality)

---------

Signed-off-by: Jin Hai <haijin.chn@gmail.com>
 2026-06-12 14:56:44 +08:00
Jin Hai
139f4515e8 Go: refactor CLI (#15898) 
### What problem does this PR solve?

1. remove unused code
2. fix login issue

### Type of change

- [x] Bug Fix (non-breaking change which fixes an issue)
- [x] Refactoring

Signed-off-by: Jin Hai <haijin.chn@gmail.com>
 2026-06-10 16:06:30 +08:00
Jack
87b8062df4 feat: implement POST /api/v1/searchbots/ask — streaming RAG with citations and think-tag processing (#15825) 
Implements POST /api/v1/searchbots/ask in Go with streaming SSE,
citations, and think-tag processing. 23 files, 90+ unit tests.

---------

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
 2026-06-09 22:48:50 +08:00
Jin Hai
d02eb6b596 Go: refactor CLI (#15728) 
### What problem does this PR solve?

```
RAGFlow(user)> add api server 'ccc' host '127.0.0.1:9980';
SUCCESS
RAGFlow(user)> list api server;
+------------+---------------+-----------------+---------+-------------+---------------+
| api_server | api_server_ip | api_server_port | auth    | user_name   | user_password |
+------------+---------------+-----------------+---------+-------------+---------------+
| ccc        | 127.0.0.1     | 9980            | no auth |             |               |
| default    | 127.0.0.1     | 9384            | login   | aaa@aaa.com | ***           |
+------------+---------------+-----------------+---------+-------------+---------------+
RAGFlow(user)> delete api server 'ccc';
SUCCESS
RAGFlow(user)> list api server;
+------------+---------------+-----------------+---------+
| api_server | api_server_ip | api_server_port | auth    |
+------------+---------------+-----------------+---------+
| default    | 127.0.0.1     | 9384            | no auth |
+------------+---------------+-----------------+---------+

RAGFlow(user)> show admin server;
+--------------+-------+
| field        | value |
+--------------+-------+
| admin_server | N/A   |
+--------------+-------+
RAGFlow(user)> add admin server host '127.0.0.1:9880';
SUCCESS
RAGFlow(user)> show admin server;
+-------------------+-----------+
| field             | value     |
+-------------------+-----------+
| admin_server_ip   | 127.0.0.1 |
| admin_server_port | 9880      |
| auth              | no auth   |
+-------------------+-----------+
RAGFlow(user)> delete admin server;
SUCCESS
RAGFlow(user)> show admin server;
+--------------+-------+
| field        | value |
+--------------+-------+
| admin_server | N/A   |
+--------------+-------+

RAGFlow(user)> show current
+-----------------+-------------+
| field           | value       |
+-----------------+-------------+
| api_server_port | 9384        |
| user_name       | aaa@aaa.com |
| user_password   | ***         |
| mode            | api         |
| verbose         | false       |
| api_server      | default     |
| api_server_ip   | 127.0.0.1   |
| auth            | login       |
| output          | table       |
| interactive     | true        |
+-----------------+-------------+
```
### Type of change

- [x] Refactoring

---------

Signed-off-by: Jin Hai <haijin.chn@gmail.com>
 2026-06-09 15:22:50 +08:00