Commit Graph

549 Commits

Author SHA1 Message Date
Jin Hai
7d64a78f83 Go: unify three services into one binary (#16462)
### Summary

Plan to start api_server, admin_server and ingestor in one binary:
- ./ragflow_main --admin
- ./ragflow_main --api
- ./ragflow_main --ingestor

---------

Signed-off-by: Jin Hai <haijin.chn@gmail.com>
2026-07-02 21:21:10 +08:00
qinling0210
dcbd0d260c Port agent PRs to GO - 2 (#16565)
### Summary

Port the following PRs to GO in this PR

https://github.com/infiniflow/ragflow/pull/16420
https://github.com/infiniflow/ragflow/pull/13295
2026-07-02 20:20:11 +08:00
qinling0210
24118ac0d1 Fix chat thinking & Figure issue in GO (#16558)
### Summary

Fix chat thinking & Figure issue
2026-07-02 20:19:50 +08:00
Hz_
42aba36c1b fix(go): chunk stats after chunk deletion (#16553)
## Summary
- Decrement document and knowledgebase chunk counts after chunks are
deleted
- Keep token counts unchanged because deleted chunk token totals are not
available
- Add tests for stats update, zero-delete behavior, error handling, and
transaction rollback
2026-07-02 19:54:42 +08:00
Hz_
dfd95c9c5c fix(go): Add tenant filter to file queries (#16526)
## Summary

- Add `tenant_id` filtering to `FileDAO.Query`.
- Pass tenant IDs through existing file query call sites.
- Prevent cross-tenant filename and folder duplicate checks.
2026-07-02 19:54:22 +08:00
Jin Hai
11dfea489d Fix Go: fix minio port issue (#16552)
### Summary

1. env 'MINIO_PORT' is used for MINIO external access, which shouldn't
be used in Go config.
2. After RAGFlow 1.0 release, MINIO_PORT will be used for docker compose
internal usage. new ENV MINIO_EXTERNAL_PORT will be used for external
access.

Signed-off-by: Jin Hai <haijin.chn@gmail.com>
2026-07-02 19:15:58 +08:00
euvre
fc9116578c Fix: PDF page count detection for compressed PDFs (#16487) 2026-07-02 19:08:49 +08:00
Renzo
7d422ba67d feat(go): implement chatbots/<dialog_id>/info and searchbots/detail (#15420)
### What problem does this PR solve?

Part of #15240 (rewriting the RAGFlow API server in Go).

Implements the two public bot endpoints from
`api/apps/restful_apis/bot_api.py`:

- **`GET /api/v1/chatbots/<dialog_id>/info`** (`chatbots_inputs`) —
returns `{title, avatar, prologue, has_tavily_key}` for a dialog the
authenticated tenant owns (tenant match + `status == VALID`), otherwise
`"Authentication error: no access to this chatbot!"`.
- **`GET /api/v1/searchbots/detail`** (`detail_share_embedded`) —
returns search-app detail for a `search_id` the tenant can access.
Permission is checked across the tenant's joined tenants; denial returns
`"Has no permission for this operation."` (operating error, `data:
false`) and a missing app returns `"Can't find this Search App!"`.

Both endpoints authenticate with an SDK **beta token** (`Authorization:
Bearer <beta>`) rather than a session — the token is resolved to a
tenant via `APIToken.query(beta=token)`, backed by a new
`APITokenDAO.GetByBeta`. Because they perform their own token-based
auth, the routes are registered on the unauthenticated route group
(mirroring the Python blueprint, which has no `@login_required`).

Both live in a new `internal/handler/bot.go` + `internal/service/bot.go`
since they share the same source module. Handler unit tests cover the
auth, success, and error-mapping paths.

### Type of change

- [x] New Feature (non-breaking change which adds functionality)

---------

Co-authored-by: Claude Code <claude@anthropic.com>
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-authored-by: Ling Qin <qinling0210@163.com>
2026-07-02 18:46:00 +08:00
Jack
7ae18a45ee Fix: correct download_deps.py path in error messages and add native libs doc (#16557)
## Summary

Fix error messages in `build.sh` and add documentation in
`internal/development.md` for downloading native static libraries
(pdfium, pdf_oxide, office_oxide).

## Changes

- `build.sh`: change error hint from `uv run download_deps.py` to `uv
run ragflow_deps/download_deps.py` (correct path from project root)
- `internal/development.md`: add section 2.1 documenting how to download
native libs and install lld
2026-07-02 18:41:39 +08:00
Hz_
d31640a7a2 fix(go): shared chatbot session id length (#16559)
## Summary
- use the project-standard 32-character ID generator when creating
shared chatbot sessions
- fix MySQL insert failures caused by writing 36-character UUID strings
into `api_4_conversation.id`
2026-07-02 17:42:33 +08:00
Jack
c8cf0c967d Feat: add DOCX parser (#16521)
### Summary

Add DOCX parser - go.
2026-07-02 16:31:09 +08:00
Haruko386
9c8d8c7b83 fix: unable to load pic in chunk result (#16485)
### Summary

As title:
2026-07-02 16:05:49 +08:00
Haruko386
3a5bc1371a fix: unable to build go backend (#16542) 2026-07-02 15:57:51 +08:00
Haruko386
92e8eb5fe7 fix: add search keywords and filter for datasets-search (#16550) 2026-07-02 15:57:07 +08:00
Hz_
cbb24944e8 fix(go): clear task cancel signals and chunk counters on rerunWithDelete (#16544) 2026-07-02 15:46:11 +08:00
Hz_
fa1b52ca74 fix(go): prevent moving folders into themselves (#16522) 2026-07-02 15:45:30 +08:00
Jin Hai
0b9ab12c58 Go: fix lint (#16533)
### Summary

as title.

---------

Signed-off-by: Jin Hai <haijin.chn@gmail.com>
2026-07-02 13:44:05 +08:00
grandpig
17e3e34e78 refactor: use WaitGroup.Go to simplify code (#16539)
### Summary

Adopt sync.WaitGroup.Go (Go 1.25) to simplify tracked goroutine
spawning. This replaces the error-prone trio of wg.Add(1), go func(),
and defer wg.Done() with a single, self-contained call.

More info: https://github.com/golang/go/issues/63796

Signed-off-by: grandpig <grandpig@outlook.com>
2026-07-02 13:41:53 +08:00
Hz_
d0d0339428 fix(go): agent settings update clearing DSL (#16495)
### Summary

This PR fixes a Go backend bug where updating agent settings, such as
description, could clear the agent DSL.

Root cause:
PUT /api/v1/agents/:canvas_id only bound the dsl field in Go. When the
frontend submitted settings without dsl, the service still updated the
canvas with an empty DSL value.

Changes:

- Treat agent updates as partial patches.
- Preserve existing DSL when dsl is not present in the request.
- Update only specified user_canvas fields instead of saving the full
row.
- Add a regression test for settings updates preserving DSL.

Test:

`go test ./internal/service ./internal/handler`

Co-authored-by: Zhichang Yu <yuzhichang@gmail.com>
2026-07-02 13:41:24 +08:00
Hz_
a67026f714 fix(go): agent explore thumbnail loading for multiple doc_ids (#16514)
## Summary
- align the Go `/api/v1/thumbnails` endpoint with the frontend request
format for repeated `doc_ids`
- return thumbnail mappings for multiple documents instead of failing on
a single missing document
- preserve Python-compatible thumbnail formatting, including base64
thumbnail passthrough
2026-07-02 12:35:10 +08:00
Hz_
cb8012e30b fix(go): accept disabled chunk filter in list chunks handler (#16532)
### Summary

Fixes a bug in the Go chunk list handler where the available` query
parser rejected `false` and `0` even though they were documented as
supported values.`

This caused requests from the "Disabled" chunk filter to return HTTP 400
and broke the chunk list page when filtering disabled chunks.
2026-07-02 12:07:19 +08:00
Haruko386
b4825166a7 fix: JSONMap scan in dataset index chunking config (#16489)
### Summary

As title

This PR fixes dataset index task creation failing with unsupported data
type: entity.JSONMap when loading document chunking config.

#### issues:
```
2026/06/30 15:19:40 /home/infiniflow/Documents/development/ragflow/internal/dao/document.go:162 
[error] unsupported data type: ragflow/internal/entity.JSONMap
```

#### Changes:
+ Adds the missing GORM type:longtext tag to ParserConfig in
DocumentDAO.GetChunkingConfig.
+ Adds a DAO regression test covering GetChunkingConfig joins across
document, knowledgebase, and tenant while scanning parser_config.
2026-07-02 12:06:53 +08:00
Haruko386
d6b1c5937b fix: get duplicate datasetID when get-Chat (#16498)
### Summary

As title

```go
// Resolve kb_ids to kb_names
	kbNames, datasetIDs := s.getDatasetNamesAndIDs(chat.KBIDs)

        // duplicated add datasetID(removed)
	for _, kbID := range chat.KBIDs {
		datasetID, ok := kbID.(string)
		if !ok {
			continue
		}
		datasetIDs = append(datasetIDs, datasetID)
	}
```
2026-07-02 12:06:29 +08:00
Haruko386
ee45c97b0b fix: unadble to add metadata for file in kb (#16523)
### Summary

As title

Before, it return `update success` but never insert or update any
metadata

fixed:

```go
	_, err = s.docEngine.InsertMetadata(nil, []map[string]interface{}{
		{
			"id":          docID,
			"kb_id":       doc.KbID,
			"meta_fields": meta,
		},
	}, tenantID)
```
2026-07-02 12:06:05 +08:00
Zhichang Yu
ba552f64b9 Stabilize timeout tests with semantic assertions (#16537)
Replace fragile wall-clock timeout assertions with semantic checks for
deadline errors, retry suppression, and event ordering. Keep only
lower-bound timing checks where they prove backoff behavior. This
reduces CPU-load flakes without weakening regression coverage.
2026-07-02 10:56:38 +08:00
Wang Qi
7abc69434f [Go] Fix to allow duplicate key for provider (#16543) 2026-07-02 10:34:36 +08:00
Hz_
9b83d0f154 fix(go): document count in kb (#16490)
### Summary
This PR fixes incorrect dataset document counters in the Go service.

Several document creation paths inserted document records directly
through documentDAO.Create, bypassing the shared InsertDocument logic
that increments knowledgebase.doc_num. As a result, datasets could
contain documents while doc_num remained 0.
2026-07-02 10:34:14 +08:00
Hz_
0de69e5bba feat(go-api) sessions message update (#16517)
### Summary
```
/api/v1/chats/<chat_id>/sessions/<session_id>/messages/<msg_id> DELETE
/api/v1/chats/<chat_id>/sessions/<session_id>/messages/<msg_id>/feedback PUT
```
Migrates the chat session message delete and feedback APIs to the Go
server, matching the Python behavior for authorization, session
ownership checks, message/reference updates, and feedback validation.
2026-07-02 10:33:27 +08:00
Jack
5bc4753d1e Feat/oss parser no post (#16464)
### Summary

Remove dead code
2026-07-02 09:46:33 +08:00
qinling0210
133b1e15fd Port agent PRs to GO (#16529)
### Summary

Port the following PRs to GO in this PR

https://github.com/infiniflow/ragflow/pull/14210
https://github.com/infiniflow/ragflow/pull/14641
https://github.com/infiniflow/ragflow/pull/15220
https://github.com/infiniflow/ragflow/pull/15228
https://github.com/infiniflow/ragflow/pull/15384
https://github.com/infiniflow/ragflow/pull/15754
https://github.com/infiniflow/ragflow/pull/16413
https://github.com/infiniflow/ragflow/pull/16483
https://github.com/infiniflow/ragflow/pull/16419
https://github.com/infiniflow/ragflow/pull/16361   
https://github.com/infiniflow/ragflow/pull/16050
2026-07-02 09:45:01 +08:00
Jack
42a0faad18 Fix: use .a to replace .so for pdfium/pdf_oxide/office_oxide (#16496)
### Summary

Use .a to replace .so for pdfium/pdf_oxide/office_oxide

---------

Co-authored-by: Claude <noreply@anthropic.com>
2026-07-01 21:21:31 +08:00
Haruko386
4a72c973e8 fix: return call failed when LLM not available (#16518)
### Summary

As title
2026-07-01 20:10:42 +08:00
euvre
fb0376561f fix: normalize Q&A parser ID key to lowercase 'qa' (#16530) 2026-07-01 19:33:18 +08:00
Jin Hai
f4f9e4466b Go CLI: fix list provider models (#16493)
### Summary

As title.

Signed-off-by: Jin Hai <haijin.chn@gmail.com>
2026-07-01 19:08:17 +08:00
Haruko386
9eacbb418f fix: unable to open filter in agent page(no agent tags...) (#16531)
### Summary

As title
2026-07-01 19:07:15 +08:00
blackflytech
fcf2ca869b refactor: replace context.WithCancel with t.Context (#16509)
### Summary

The addition of the Context method to Go's testing.T provides
significant improvements for writing concurrent tests. It allows better
management of goroutines, ensuring they properly exit and preventing
issues like deadlocks and unfinished processes.

By using Context, errors and cancellations can be handled more
effectively, making tests more robust and easier to reason about. This
change also enables tighter integration between tests and the
application code, especially for systems that span multiple concurrent
components. Overall, it simplifies test code and enhances test stability
and maintainability.

More info: [golang/go#18368](https://github.com/golang/go/issues/18368)

Signed-off-by: blackflytech <blackflytech@outlook.com>
2026-07-01 18:37:11 +08:00
qinling0210
5ba25a5267 Fix GetProjectRoot in GO (#16520)
### Summary

Fix GetProjectRoot in GO
2026-07-01 18:17:53 +08:00
ZF
97a4c64cc8 fix(harness): truncate text on rune boundary to keep UTF-8 valid (#16511)
### Summary

`truncateText` in the `reduction` and `summarization` middlewares
truncates with `s[:maxLen]`, which slices by byte. When `maxLen` lands
inside a multi-byte character (common with CJK or other non-ASCII
content flowing through the agent), the string is cut mid-rune and the
tail byte(s) become invalid UTF-8. That broken text then goes into the
reduced context / summary prompt.

`TruncateToolResult` in the same `reduction` package already avoids this
by slicing on a rune boundary and even notes it in a comment. This PR
makes the two `truncateText` helpers do the same, so they stay
consistent with the existing helper.

Both functions keep their existing output shape (summarization still
appends `...`). Added a small unit test in each package covering ASCII
truncation and a CJK string, asserting the result stays valid UTF-8.
2026-07-01 16:45:26 +08:00
qinling0210
7862f69f39 Implement chat completions in go (#16491)
### Summary

POST   /api/v1/chat/completions
2026-07-01 15:52:52 +08:00
Wang Qi
8f24b30652 [Go] Add API /api/v1/chat/recommendation and consolidate with /api/v1/searchbots/related_questions (#16500) 2026-07-01 13:17:16 +08:00
Yingfeng
6648fe4151 Fix g++ 11 incompatibility issue (#16512) 2026-07-01 10:16:47 +08:00
Jack
8e1dc4f308 revert: roll back tests.yml CI changes from PR #16391 (#16505)
## Summary

Two changes to make Go build \& run independent of native libraries
(office_oxide, pdfium, pdf_oxide).

## 1. Make native libraries optional (build.sh + Go source)

## 2. Roll back tests.yml CI changes from PR #16391
2026-06-30 21:50:37 +08:00
Yingfeng
5af361ed68 Add spacy based ner and relationship extractor for both python and Go version with equivalent outputs (#16456)
As title
2026-06-30 21:40:24 +08:00
Hz_
3633d08495 feat(go-api): Migrate Box web OAuth connector APIs to Go (#16480)
This PR migrates the Box web OAuth flow from Python to Go for:

  - POST /api/v1/connectors/box/oauth/web/start
  - GET /api/v1/connectors/box/oauth/web/callback
  - POST /api/v1/connectors/box/oauth/web/result
2026-06-30 18:10:36 +08:00
Yingfeng
63bdf5c5b1 Fix harness streaming emit (#16486) 2026-06-30 18:06:03 +08:00
Haruko386
b3af9fc068 fix: remove dup-prefix in bot_routes (#16492) 2026-06-30 17:02:58 +08:00
Wang Qi
3bb976b383 [Go] Add /api/v1/searchbots/mindmap and /api/v1/chat/mindmap (#16443) 2026-06-30 16:35:33 +08:00
Zhichang Yu
4c54cefd29 Port 14 upstream agent security / correctness fixes to Go canvas (#16455)
Mirrors 14 merged upstream PRs into the Go agent port.

PRs ported:
  - #15609 ExeSQL SSRF guard + DNS pin
  - #15436 HTTP timeout on external API tools
  - #16363 be_output restore + DeepL error path
  - #15644 switch no longer matches empty condition
  - #15374 session_id bind to path agent_id (DAO idor guard)
  - #16169 sandbox artifact ownership gate
  - #15457 tenant ownership on agentbots
  - #15145 rerun agent document access check
- #15446 thinking switch (component portion; provider policy lives in
internal/llm)
  - #15426 Invoke URL/proxy SSRF + DNS pin + no-redirects
  - #15238 agentbot thinking-logs beta endpoint
  - #14589 UserFillUp SSE event propagation
  - #14890 anonymous webhook opt-in
- #15068 PipelineChunker new component (text/file_ref/parser_id
dispatch; file-format extraction is a follow-up)

40 files, +2355 / -58 lines. 33 new tests, all targeted package suites
pass (1721 + 4 skipped); 1 pre-existing flaky test unrelated.
2026-06-30 16:28:48 +08:00
Jin Hai
bd56a1473f Go CLI: merge function (#16458)
### Summary

1. remove unused code.

---------

Signed-off-by: Jin Hai <haijin.chn@gmail.com>
2026-06-30 15:47:26 +08:00
Wang Qi
2018eec0dc Fix: allow any host for url for development (#16459) 2026-06-30 10:19:04 +08:00