Commit Graph

682 Commits

Author SHA1 Message Date
maybehokori
f3a2fb7a10 fix: update somark.tech to somark.cn/somark.ai with purchase URLs (#16892)
## Summary

Update all `somark.tech` references to `somark.cn` (default for China) /
`somark.ai` (for overseas including Taiwan, China; Hong Kong, China;
Macau, China). Users fill in `base_url` manually — default is
`somark.cn`.

### Changes

| File | Change |
|------|--------|
| `constants.py` | Default → `somark.cn` |
| `somark_parser.py` | SAAS_BASE_URL + fallback → `somark.cn` |
| `ocr_model.py` | Default → `somark.cn` |
| `pdf_parser_common.go` | Go default → `somark.cn` |
| `llm.ts` | API key URL → `somark.cn` |
| `en.ts` | Base URL descriptions + purchase URLs |
| `zh.ts` | Base URL descriptions + purchase URLs |

### Purchase URLs
- China: `https://somark.cn/workbench/purchase`
- Overseas: `https://somark.ai/studio/purchase`

---------

Co-authored-by: justinychuang <huangyicheng@soulcode.cn>
Co-authored-by: Jin Hai <haijin.chn@gmail.com>
2026-07-16 19:18:05 +08:00
Haruko386
101155bdc7 fix: support message content (#16980)
### Summary

As Title
<img width="3241" height="1934" alt="image"
src="https://github.com/user-attachments/assets/cb22031c-f200-462a-b871-b4a739e7a66a"
/>
2026-07-16 19:08:36 +08:00
Haruko386
5307ecd520 Go: add tools for moonshot, baidu and minimax (#17000)
### Summary

As title
related to #16990
2026-07-16 19:03:59 +08:00
Jin Hai
c7a623ff81 Go: introduce clickhouse (#16996)
### Summary

Introduce Clickhouse for data statistics.

Signed-off-by: Jin Hai <haijin.chn@gmail.com>
2026-07-16 18:36:56 +08:00
euvre
feba9e1158 fix: custom empty_response not shown in streaming chat (Go mode) (#16989) 2026-07-16 17:02:40 +08:00
Hz_
9003584e90 fix(go-agent): add input form of ListOperationsComponent and VariableAggregatorComponent (#16987)
## Summary
- Add `GetInputForm()` for `ListOperationsComponent` to expose `Query`
input field in debug UI
- Add `GetInputForm()` for `VariableAggregatorComponent` to expose
`Variables` input field in debug UI

## Test
- Verify input form fields render correctly for both components in the
debug UI
2026-07-16 16:58:38 +08:00
Jin Hai
8bb2cd0fb7 Go: fix db alignment (#16964)
### Summary

Align go db schema with python

---------

Signed-off-by: Jin Hai <haijin.chn@gmail.com>
2026-07-16 16:00:56 +08:00
Yingfeng
5ab424c10c Revert "fix: remove MySQL document.meta_fields mapping to match Python schema" (#16981)
Reverts infiniflow/ragflow#16929
2026-07-16 15:17:49 +08:00
Jack
fd5487e064 feat: add /api/v1/language endpoint for Go/Python runtime detection (#16952)
Both backends serve GET /api/v1/language. Frontend calls it once and
caches. By this way, front end can know the backend is go or python and
thus can determine which part of logic to load.

---------

Co-authored-by: Claude <noreply@anthropic.com>
2026-07-16 15:13:43 +08:00
Wang Qi
46e14fe52b Fix doc.MetaFields undefined (#16976)
Follow on #16929
2026-07-16 13:27:08 +08:00
Zhichang Yu
d2f4304ff4 Fix Pipeline.Run callers and rename SetupOverrides (#16973)
Adapt Pipeline.Run call sites to the new signature and rename the
SetupOverrides family to OverrideParams for consistency.
2026-07-16 13:12:25 +08:00
Haruko386
f9a1fbfab9 fix: support categorize component in agent (#16966)
<img width="3716" height="2039" alt="image"
src="https://github.com/user-attachments/assets/62709d24-60ba-4c92-81f1-48b6d23e3be1"
/>
2026-07-16 13:10:29 +08:00
euvre
30de65991b fix: remove MySQL document.meta_fields mapping to match Python schema (#16929) 2026-07-16 13:07:49 +08:00
Jack
047c714ffb Feat: ingestion cleanup (#16953)
### Summary

1. Remove dead code (replaced by builtin ingestion pipeline)
2. Refactor (move document parsing progress from http api into ingestion
executor)
2026-07-16 11:05:26 +08:00
Jin Hai
2a6e210020 Go: align db schema to python (#16935)
As title.

---------

Signed-off-by: Jin Hai <haijin.chn@gmail.com>
2026-07-15 23:05:06 +08:00
Haruko386
ff3d566a4f fix[go]: support retrieval kb in agent chat (#16944)
### Summary

As title
- [x] fix tool & component `Retrieval KB`
- [x] fix agent cannot use  `Retrieval KB` in agent chat

#### Main cause

Model provider do not set up:
```Go
if chatModelConfig.Tools != nil {
    reqBody["tools"] = chatModelConfig.Tools
}
```

#### Working Now
<img width="3774" height="2128" alt="image"
src="https://github.com/user-attachments/assets/400a349d-0211-43e5-a7ec-7a014acf77a6"
/>

```
 ____________________________________
< This PR takes me all day to do it. >
 ------------------------------------
  \
   \   (\__/)
       (•ㅅ•)
       /   づ                                                                                                                                                                                                    
```
2026-07-15 21:44:28 +08:00
Hz_
a7da78d0d7 refactor(go-agent): unify tool-backed canvas components (#16912)
## Summary

This PR consolidates Eino-backed Agent tools behind the shared
ToolBackedComponent implementation and aligns their Canvas
configuration,
runtime inputs, output conversion, validation, and registration.

## What changed

- Migrated these Canvas components to the unified tool-backed path:
    - Tavily Search and Extract
    - Execute SQL
    - Google
    - Yahoo Finance
    - Email
    - DuckDuckGo
    - Wikipedia
    - Google Scholar
    - ArXiv
    - PubMed
    - BGPT
    - GitHub
    - WenCai
    - SearXNG
    - Keenable Search

- Removed superseded component wrappers and their duplicate tests.
- Added dedicated registry builders for node-level configuration and
validation.
- Kept model-emitted runtime inputs separate from Canvas node
configuration.
- Moved Email defaults, template resolution, recipient parsing, SMTP
execution, and output conversion into the owning tool.
- Added complete ToolComponent specifications for Canvas inputs,
outputs, and input forms.
- Preserved raw upstream fields where downstream workflows may depend on
them.
- Added workflow registration coverage for all migrated component names.
- Kept HTTP Request, Docs Generator, and Browser as standalone
components because they are not Eino-backed tools.

## Testing

Passed:

bash build.sh --test ./internal/agent/tool/...
bash build.sh --test ./internal/agent/component/...
bash build.sh --test ./internal/agent/runtime/...

<img width="2057" height="1111" alt="image"
src="https://github.com/user-attachments/assets/c728d7a3-9d15-4c5c-b0eb-6b77ad0e41ac"
/>
<img width="2057" height="1111" alt="image"
src="https://github.com/user-attachments/assets/ef13119a-ef92-4b43-9061-eee9511bf492"
/>
<img width="2057" height="1111" alt="image"
src="https://github.com/user-attachments/assets/f50f2c30-eab0-463c-b0b3-0a02523219f1"
/>
2026-07-15 21:42:08 +08:00
Jakob
d55de09b7d Updated RAGcon model provider (go) (#16950)
### Summary
We have updated our model driver to work with go.
It is based on OpenAI-API-Compatible model provider.

Draft
#15519

Our old model provider
#13425
2026-07-15 20:59:23 +08:00
zhifu gao
794c6b63e4 fix(agent): return DataOperations input form (#16930)
## Summary

- return the input form collected by
`DataOperationsComponent.GetInputForm`
- add a regression test covering system variables, component outputs,
and duplicate references
2026-07-15 20:32:59 +08:00
euvre
70a49c947d Remove unused /datasets/<id>/embedding endpoint (#16936) 2026-07-15 17:42:41 +08:00
Lynn
63c010cd04 Fix: code-rabbit review comment (#16940) 2026-07-15 15:44:08 +08:00
Jack
e543ff02c4 Feat: nats message processing refactor and support built in DSL (#16873) 2026-07-15 14:53:16 +08:00
Haruko386
14d361aa51 refactor: remove unused getInputElementsFromText function (#16927)
### Summary

Removed unused function getInputElementsFromText and its associated
comments.
2026-07-15 11:38:52 +08:00
Haruko386
a19af34d31 fix: unable to use agent in agent (#16910)
### Summary

As title

#### Agent:
<img width="3707" height="2030" alt="image"
src="https://github.com/user-attachments/assets/68480004-9c79-4302-ae22-eb3375b7fac9"
/>

#### Data_operations
<img width="3716" height="2044" alt="image"
src="https://github.com/user-attachments/assets/068cc1a6-1ba5-4746-a447-a706d2149ef5"
/>
2026-07-15 10:58:59 +08:00
Haruko386
215d291cfa fix: support email send for go backend (#16895)
### Summary

As title

<img width="3761" height="2036" alt="image"
src="https://github.com/user-attachments/assets/775f8c2a-784e-4713-bcf0-c26bfc368dac"
/>


<img width="1256" height="2760" alt="d2aeeb50dd11cc8dc76bc4743425680e"
src="https://github.com/user-attachments/assets/80f9c572-549d-427f-8e48-35dc04cfa98a"
/>
2026-07-15 10:56:36 +08:00
Haruko386
654279a745 fix: support code_execute for go backend (#16903)
### Summary

As title

<img width="3764" height="2033" alt="image"
src="https://github.com/user-attachments/assets/3f5084c2-0a04-4a44-bb31-a39a59c2a91f"
/>
2026-07-15 10:55:39 +08:00
Haruko386
3339040987 fix: add GetInputForm for string_transform (#16900)
### Summary

As title:

#### merge:
<img width="2972" height="1473" alt="image"
src="https://github.com/user-attachments/assets/2171e31f-13f4-4799-a6d0-1b9326210658"
/>

---
#### split
<img width="858" height="416" alt="image"
src="https://github.com/user-attachments/assets/f89d39e3-da85-4349-a53c-e27a17cc9dfa"
/>
2026-07-15 10:54:18 +08:00
Hz_
276941f7b4 fix(go-agent): add VariableAssigner input form (#16902)
## Summary

- Add the runtime input form for VariableAssigner components
- Allow the component input-form endpoint to resolve VariableAssigner
nodes

## Testing

- `bash build.sh --test ./internal/agent/component/...`
2026-07-15 10:47:49 +08:00
Jin Hai
b506d8ec5f Go: add more frameworks (#16915)
### Summary

1. Add hooks for server init
2. Add hooks for router init
3. Add jaeger and otel related config

Signed-off-by: Jin Hai <haijin.chn@gmail.com>
2026-07-14 22:20:26 +08:00
Jin Hai
c242669a9d Go: fix bugs (#16911)
### Summary

1. Fix 'list users' in admin CLI
2. Change 3 slow test cases priority.

Signed-off-by: Jin Hai <haijin.chn@gmail.com>
2026-07-14 21:58:33 +08:00
Lynn
bd7a3bb878 Fix: align go provider apis with python apis (#16867)
### Summary

As title.

---------

Co-authored-by: Jin Hai <haijin.chn@gmail.com>
Co-authored-by: Wang Qi <wangq8@outlook.com>
2026-07-14 21:06:11 +08:00
Jin Hai
26161ce844 Go: refactor (#16906)
### Summary

remove and merge code

Signed-off-by: Jin Hai <haijin.chn@gmail.com>
2026-07-14 19:29:34 +08:00
euvre
22dd1ad401 fix: resolve provider name case mismatch and 301 redirect in Go mode (#16889) 2026-07-14 17:05:21 +08:00
euvre
315e5725c2 Fix: Search mind map not displaying (#16866) 2026-07-14 17:04:47 +08:00
Hz_
ea18ab3ba0 fix(go-agent): align ExeSQL component inputs with Python (#16896)
## Summary

- Resolve configured ExeSQL SQL references before tool invocation.
- Align ExeSQL parameters, defaults, JSON tags, and tool schema with
Python.
- Preserve SQL string literals and restore Canvas output fields.

## Testing

- `bash build.sh --test ./internal/agent/tool/...`
- `bash build.sh --test ./internal/agent/component/...`
- `bash build.sh --test ./internal/agent/runtime/...`

<img width="2039" height="1041" alt="image"
src="https://github.com/user-attachments/assets/9f4beca7-ca28-4641-adda-0570415fcaa1"
/>
2026-07-14 15:02:28 +08:00
Haruko386
9289a183a2 fix: support keenable component (#16884)
### Summary

As title


<img width="3763" height="2042" alt="image"
src="https://github.com/user-attachments/assets/0bcf2d41-9475-4d22-b8f0-64a527e22aba"
/>
2026-07-14 14:57:22 +08:00
Haruko386
3fe9d291fb fix: support doc generator component (#16858)
### Summary

As title

<img width="3165" height="2011" alt="image"
src="https://github.com/user-attachments/assets/6947aa4f-5dfd-4d8c-b6af-82015b1f1537"
/>

---

<img width="3735" height="1721" alt="image"
src="https://github.com/user-attachments/assets/3a7ec8e8-d5b9-4195-bba9-c435218f6142"
/>
2026-07-14 13:17:30 +08:00
Zhichang Yu
d279aee1ff Go ports of workflow and chunker fixes (#16878)
Ports two Python fixes to Go: the variable_ref_patt underscore/colon fix
(#16792) and the TokenChunker upstream-chunks fix (#16825). Keeps Go
behavior aligned with upstream Python.
2026-07-13 23:32:07 +08:00
Jin Hai
844e3ea64d Go and Python: fix IDOR issue of search completions (#16864)
### Summary

In Go and python implementation, the dataset / KB id isn't validated if
it is accessible by this user.

---------

Signed-off-by: Jin Hai <haijin.chn@gmail.com>
2026-07-13 21:00:19 +08:00
Jin Hai
8bc18154d2 Go: refactor and add version type (#16863)
### Summary

```
RAGFlow(admin)> show version;
+--------------+-----------------------+
| field        | value                 |
+--------------+-----------------------+
| version      | v0.26.4-84-g547bc8614 |
| version_type | open source           |
+--------------+-----------------------+
```

---------

Signed-off-by: Jin Hai <haijin.chn@gmail.com>
2026-07-13 18:00:35 +08:00
Haruko386
466f33e6b4 fix: support component brwoser (#16848)
### Summary

As title
Unable to test it since I don't have apiKey for `openai` , `Anthropic`
and `Gemini`

---

<img width="1130" height="557" alt="image"
src="https://github.com/user-attachments/assets/11570c75-68f3-490d-8186-4ecbcd8b8f40"
/>
2026-07-13 16:57:56 +08:00
Hz_
d03b09b6a3 fix(go-agent): Align Go WenCai and SearXNG agent components with python (#16854)
## Summary

- Align Go WenCai and SearXNG behavior, schemas, and node parameters
with Python.
- Add the `WenCai` and `SearXNG` Canvas components and register their
tool factories.
- Match Python's current WenCai behavior by returning an empty report
while its upstream request is disabled.
- Add SearXNG request validation, SSRF-safe DNS pinning, raw result
preservation, and reference rendering.
- Support context cancellation, error envelopes, and lock-safe retrieval
references.

  ## Tests

  Passed:

  - `bash build.sh --test ./internal/agent/tool/...`
  - `bash build.sh --test ./internal/agent/component/...`
  - `bash build.sh --test ./internal/agent/runtime/...`
  - `bash build.sh --test ./internal/agent/...`
  - `cd web && npm run type-check`
  
  
<img width="1900" height="1102" alt="image"
src="https://github.com/user-attachments/assets/ec77d217-d9fd-455a-96ec-9aabf6841109"
/>
  
<img width="1900" height="1102" alt="image"
src="https://github.com/user-attachments/assets/52ac129f-cb65-453d-ae48-cc518803ac23"
/>
2026-07-13 16:50:17 +08:00
euvre
3bfad1f00e fix: correct model type mappings and improve system setting persistence (#16501) 2026-07-13 16:42:55 +08:00
Sbaaoui Idriss
d35e957252 fix: test drift on go specific proxy scheme (#16796)
### Summary

certain tests fail because of test drift and were fixed, other because
of go issues

---------

Co-authored-by: Wang Qi <wangq8@outlook.com>
2026-07-13 16:41:38 +08:00
Jack
cfacaccad7 Refactor: message processing (#16852)
### Summary

1. refactor message processing
2. delete un-used componentIndexMap
3. unfold (delete) internal/ingestion/task/task_handler.go
2026-07-13 16:32:34 +08:00
Hz_
2a482f3ae7 fix(go-agent): align GitHub and Invoke Canvas components (#16849)
## Summary

- Add the GitHub Canvas component with tool registration and reference
propagation.
- Align the Invoke component with the Python contract for node config,
input form, response output, and timing fields.
- GitHub search and HTTP Invoke now work correctly in the Go Canvas
runtime.

## Tests

- `bash build.sh --test ./internal/agent/tool/...`
- `bash build.sh --test ./internal/agent/component/...`

Note: the untracked go_ragflow_cli file is not part of the PR changes.

<img width="1813" height="1102" alt="image"
src="https://github.com/user-attachments/assets/f69cef32-59a0-4287-a06b-6843d85198cf"
/>


<img width="1813" height="1102" alt="image"
src="https://github.com/user-attachments/assets/b37dfc31-bc9b-4937-a38e-d2184bb157fe"
/>
2026-07-13 15:48:07 +08:00
Jin Hai
9f403ac3ca Go: more APIs (#16850)
### Summary

as title.

---------

Signed-off-by: Jin Hai <haijin.chn@gmail.com>
2026-07-13 15:24:17 +08:00
qinling0210
d549194562 Implement builtin chunk method as ingestion pipeline in GO (#16822)
### Summary

Implement builtin chunk mehtod as ingestion pipeline in GO
2026-07-13 13:51:40 +08:00
Lynn
b0cac0ac9d Fix: Align part of Go provider APIs with Python APIs (#16823) 2026-07-13 11:20:02 +08:00
Jack
3c0f8fc192 Refactor: refactor dataflow_service.go and guard nats message re-delivery (#16826)
### Summary

1. refactor dataflow_service.go 
2. guard nats message re-delivery
3. support document parse cancelling & re-run
2026-07-13 11:08:04 +08:00