Commit Graph

7297 Commits

Author SHA1 Message Date
Jin Hai
106c2d0a41 Go: add show users plan quota (#16818)
### Summary

```
RAGFlow(admin)> show users plan quota 100;
+---------+------------------------------------------+
| field   | value                                    |
+---------+------------------------------------------+
| quota   | 100                                      |
| command | show_users_plan_quota                    |
| error   | 'Show users plan quota' is not supported |
+---------+------------------------------------------+
```

Signed-off-by: Jin Hai <haijin.chn@gmail.com>
2026-07-10 20:27:33 +08:00
maoyifeng
8ced3d1f47 fix ci (#16824)
### Summary

fix ci
2026-07-10 20:20:29 +08:00
Öndery
e53235ed3d fix(docker): [URGENT] Docker build fails for all PRs - missing web/scripts/prepare.js during npm install (#16821)
## 🚨 Urgent: this breaks the Docker image build for **every** open PR

The `ragflow_tests_elasticsearch` and `ragflow_tests_infinity` jobs —
which build the image before running tests — currently **fail on all
pull requests**, not just this one. CI is effectively red repo-wide
until this lands. **Please review and merge ASAP.**

Example failing runs (from an unrelated PR):
-
https://github.com/infiniflow/ragflow/actions/runs/29086020168/job/86341032173
-
https://github.com/infiniflow/ragflow/actions/runs/29086020168/job/86341032196

## Symptom

```
Error: Cannot find module '/ragflow/web/scripts/prepare.js'
npm error command sh -c node scripts/prepare.js
ERROR: process "/bin/bash -c cd web && NODE_OPTIONS=..." did not complete successfully: exit code: 1
```

## Root cause

`web/package.json` defines:

```json
"prepare": "node scripts/prepare.js"
```

npm runs the `prepare` lifecycle script during `npm install`. But the
frontend-deps layer in the `Dockerfile` only copies the package
manifests before installing:

```dockerfile
COPY web/package.json web/package-lock.json web/.npmrc ./web/
RUN ... cd web && npm install     # runs `node scripts/prepare.js` → file not present yet
```

Since `web/scripts/prepare.js` is not in the image at that point, `npm
install` aborts and the whole build fails.

## Fix

Copy `web/scripts` before `npm install` so the `prepare` script is
present:

```dockerfile
COPY web/package.json web/package-lock.json web/.npmrc ./web/
COPY web/scripts ./web/scripts
RUN ... cd web && npm install
```

- Minimal and safe: `scripts/prepare.js` only installs lefthook git
hooks and already no-ops (wrapped in try/catch) inside the image where
there is no Git repo.
- Preferred over `--ignore-scripts`, which would also disable
dependencies' legitimate install scripts.
- `web/scripts` changes rarely, so build-cache impact on this layer is
negligible.

## Verification

Build reaches the frontend `npm install` step without the
`MODULE_NOT_FOUND` error. No application code is touched — this is a
build-infrastructure fix only.
2026-07-10 19:15:57 +08:00
Jin Hai
2a83ad6cb2 Go: Fix error code (#16807)
### Summary

As title.

---------

Signed-off-by: Jin Hai <haijin.chn@gmail.com>
2026-07-10 17:02:25 +08:00
Hz_
07a3523b09 fix(go-agent): unknown component "Wikipedia" canvas error. (#16784)
## Summary

- Register Wikipedia component + tool alias
`wikipedia`/`wikipedia_search`
- Use `generator=search` to get title/summary/url in one request (was
N+1)
- Node params `top_n`/`language` with validation
- Return `formalized_content` for downstream
- tests pass

<img width="1817" height="972" alt="image"
src="https://github.com/user-attachments/assets/f6d79599-6d1f-4ea6-84f7-ac06d0de13b0"
/>
2026-07-10 16:12:09 +08:00
Zhichang Yu
fb42e5531d Refactor: drop dead canvas runtime selector and tokenizer embedding wiring (#16809)
Two refactors on the Go port (agent-go-port):

- Remove the dead per-tenant canvas runtime selector (write-only Redis
scaffolding with no runtime callers) and its dependent metrics/admin
code.
- Move the tokenizer embedding-model id from the shared ingestion
globals into a Tokenizer-scoped setup, and wire the production embedder
resolver in the ingestion task package.

32 files changed, 861 insertions, 1228 deletions.
2026-07-10 15:46:45 +08:00
balibabu
d317742975 Feat: Rewrite wiki template with reui (#16797) 2026-07-10 15:44:04 +08:00
Jack
7db39822db Feature: user select pipeline support (#16788)
### Summary

Feature: user select pipeline support
2026-07-10 14:30:28 +08:00
Jin Hai
0b01171a86 Go: Update response message (#16803)
### Summary

As title.

---------

Signed-off-by: Jin Hai <haijin.chn@gmail.com>
2026-07-10 14:26:54 +08:00
maoyifeng
c33a5a9d49 GO:Cli fix show user storage display truncation (#16805)
### Summary

GO: Cli  fix show user storage display truncation
2026-07-10 13:59:07 +08:00
Hz_
5797f81fea feat(go-agent): merge Google Scholar node params with runtime inputs (#16802)
## Summary

- align the Go Google Scholar component with the Python-side config
pattern
- merge node-level params with runtime inputs so canvas defaults are
preserved and per-run inputs can override them
- add tests covering node param fallback and runtime override behavior

## Verification

- `bash build.sh --test ./internal/agent/component/... -run
TestGoogleScholar`

<img width="1873" height="1165" alt="image"
src="https://github.com/user-attachments/assets/67198c6f-6a0e-43bf-a500-8e88d82b8751"
/>
2026-07-10 13:30:21 +08:00
buua436
28340f6218 GO: improve model info parsing and add model_id/tenant context to list response (#16804) 2026-07-10 13:29:01 +08:00
chanx
868e524f29 fix: pass ownerTenantId to LLMLabel and related components for improved model fetching (#16800) 2026-07-10 13:27:14 +08:00
dependabot[bot]
eca8e87f44 build(deps): bump mistune from 3.2.0 to 3.3.0 (#16799)
Bumps [mistune](https://github.com/lepture/mistune) from 3.2.0 to 3.3.0.

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-07-10 12:24:56 +08:00
dependabot[bot]
a1a2ba8f6c build(deps): bump soupsieve from 2.8.1 to 2.8.4 (#16798)
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-07-10 12:24:21 +08:00
Jin Hai
15d1ee7654 Go: set gin mode in admin server (#16801)
### Summary

As title.

---------

Signed-off-by: Jin Hai <haijin.chn@gmail.com>
2026-07-10 12:19:10 +08:00
Jin Hai
add7b9486f Go: merge duplicate codes (#16783)
### Summary

1. merge heartbeat function.
2. introduce all environments

---------

Signed-off-by: Jin Hai <haijin.chn@gmail.com>
2026-07-10 11:58:32 +08:00
Harsh Kashyap
289ca28ce2 Fix OpenAI agent stream chunk shape (#16402)
### What problem does this PR solve?

Closes #8175.

The Agent OpenAI-compatible streaming path uses `get_data_openai(...,
stream=True)`, but that helper currently returns a minimal chunk shape.
The main OpenAI-compatible chat endpoint already includes chunk metadata
such as `created`, `system_fingerprint`, `usage`, `logprobs`, and
assistant role/tool placeholders.

This PR aligns the Agent stream helper with that existing
OpenAI-compatible chunk shape while keeping the current `delta.content`
behavior and existing reference injection path intact.

### Type of change

- [x] Bug Fix (non-breaking change which fixes an issue)
- [ ] New Feature (non-breaking change which adds functionality)
- [ ] Documentation Update
- [ ] Refactoring
- [ ] Performance Improvement
- [ ] Other (please describe):

### Verification

- `./.venv/bin/python -m pytest
test/unit_test/api/utils/test_api_utils.py -q`
- `python3 -m py_compile api/utils/api_utils.py
test/unit_test/api/utils/test_api_utils.py`
- `uvx ruff check api/utils/api_utils.py
test/unit_test/api/utils/test_api_utils.py`

---------

Co-authored-by: Harsh Kashyap <harshkashyap@Harshs-MacBook-Pro.local>
2026-07-10 10:59:32 +08:00
Zhichang Yu
12787996d1 feat(agent): Go ingestion pipeline progress mirroring and DeepDOC parser hardening (#16795)
feat(ingestion): mirror Go pipeline progress into the document table;
harden resume guards
- pipeline: bind the owning document via WithDocumentID; after each
TrackProgress event aggregate ingestion_task_log progress and mirror
progress/run/progress_msg back into the document table, so GET
/api/v1/datasets/{dataset_id}/documents reflects live Go pipeline
progress without a bespoke endpoint.
- canvas: extend the S3 resume guard to reject legacy no-op nodes (e.g.
ExitLoop) so component_total equals the count of progress-reporting
components and the aggregate percent can reach 100%.
- runtime/canvas: route progress through TrackProgress; add interrupt
test coverage (r3_interrupt_test.go).
- dao/entity: add IngestionTask.DocumentID column and AggregateProgress
support used by the mirror; IngestionTaskLog keeps a Checkpoint column
alongside the progress fields.

feat(deepdoc): cache DocAnalyzer inference results in Redis (1h TTL)
- Redis-backed DocAnalyzerCache decorator over inference.Client; cache
key = "ddoc:cache:<method>:" + sha256 of the JPEG-encoded image bytes
(deterministic).
- TTL = 1h; hits skip the inner HTTP call and return cached JSON; inner
errors are not cached.

refactor(deepdoc): align figure cropping with Python cropout + bounded
page caches
- CropSectionByDLA mirrors Python cropout: best-overlap DLA
figure/equation region, fallback to section bbox per page, vertical
concat on gray background.
- sliding-window page-image cache bounds peak memory to the recent
window instead of the whole PDF.
- rename DLADebug -> DLARegions across parser/chunker/tests.

refactor(parser): drop lib_type selector; align NewXxxParser with
NewPDFParser
- remove config["lib_type"] lookup and the libType param/field/switch
from all nine constructors; surface the CGO-required error at
ParseWithResult time instead of construction time; drop resolveLibType,
its test, and the four lib_type constants.

feat(utility): add a reusable workerpool for bounded concurrent
execution
- internal/utility/workerpool.go (+ tests).

refactor: translate Chinese prose comments to English in non-harness Go
files.

chore: upgrade github.com/cloudwego/eino from v0.9.9 to v0.9.12.
2026-07-10 10:36:10 +08:00
buua436
74bbbba3e0 fix: align model default handling (#16782) 2026-07-10 10:34:19 +08:00
Hz_
d48a5622df fix(go-agent): align Go DuckDuckGo component with canvas input form (#16775)
## Summary

- register the Go `DuckDuckGo` canvas component and restore its dynamic
input form metadata
- align the Go component input/output surface with the current canvas
usage for `query`, `channel`, and `top_n`
- fix DuckDuckGo news search in Go by fetching the required `vqd` token
before calling `news.js`, and add targeted regression tests

## Testing

Passed:
- `bash build.sh --test ./internal/agent/tool/... -run 'DuckDuckGo'`
- `bash build.sh --test ./internal/agent/component/... -run
'DuckDuckGo|TestVerifyRegistration_P1'`
- `bash build.sh --test ./internal/agent/component/... -run
'DuckDuckGo'`

Not run:
- frontend tests
- frontend build
- full Go test suite

<img width="1776" height="1092" alt="image"
src="https://github.com/user-attachments/assets/9f3f8e4b-f6b4-4915-b96c-3c5b8c7b8b30"
/>
2026-07-10 10:11:13 +08:00
chanx
8236f2cabb Fix: update LayoutRecognizeFormField to accept ownerTenantId and refactor model handling in LLM requests (#16781) dev-20260710 2026-07-10 09:36:25 +08:00
chanx
0095fa048f fix: Show full text on hover when text overflows the cards on the list page. (#16787) 2026-07-10 09:25:05 +08:00
Hz_
5c8b51cbbf fix(go-agent): add Google search wrapper component and tool registry (#16768)
### Summary

- Implemented googleComponent wrapper to bridge the canvas component
contract with Eino's SerpApi-backed GoogleTool.
- Added parameter alias mapping (query to q, max_results to num) and
content formatting logic to match Python search result representation.
- Registered the "Google" component and the "google" tool factory in the
Go agent runtime to support web search nodes.

<img width="1776" height="1092" alt="image"
src="https://github.com/user-attachments/assets/e295ab88-e48c-4fe2-bcb7-47ca5b977c9b"
/>
2026-07-09 18:58:55 +08:00
balibabu
0083ad0deb Feat: Add a data compilation layer. (#16777)
### Summary

Feat: Add a data compilation layer.
2026-07-09 17:49:16 +08:00
Lynn
5de823eab9 Fix: delete unused tenant_llm testcase (#16786) 2026-07-09 17:48:02 +08:00
Jack
8bd62573eb Fix: delete tasklet (#16778)
### Summary

1. Fix: delete tasklet
2. Remove panic in text processing
2026-07-09 17:31:28 +08:00
Hz_
b29a4a61eb fix(ao-agent): add Tavily input_form, wire real search component, and port TavilyExtract to Go (#16693)
### Summary

- TavilySearch now stores api_key from component params and injects it
into tool calls when runtime inputs omit it.
- TavilyExtract and BGPT now follow the same stored api_key behavior.
- Canvas decoding now recovers api_key from graph.nodes[].data.form when
components[].obj.params.api_key is empty, matching frontend payload
behavior without changing frontend data.
- Added regression tests for graph form key recovery and stored key
injection / caller key precedence.

Tests: build.sh --test ./internal/agent/component ./internal/service —
all pass.

<img width="1476" height="850" alt="image"
src="https://github.com/user-attachments/assets/0be31587-c1ba-4f3e-b43a-4fe0fca5a44c"
/>

<img width="1476" height="850" alt="image"
src="https://github.com/user-attachments/assets/e3edd92c-c62e-4db4-abe2-772bdf4fe1b2"
/>
2026-07-09 16:38:42 +08:00
Jack
8ac80284c4 Feat: add embedding support (#16769)
### Summary

Feat: add embedding support
2026-07-09 16:24:39 +08:00
Lynn
0ba1d37a10 Feat: optional url v1 (#16774) 2026-07-09 15:53:06 +08:00
chanx
a9420a7832 Fix: resolve shared embedding/LLM model selection errors (#16773) 2026-07-09 15:17:49 +08:00
Lynn
cc94639555 Fix: get_by_id (#16765) 2026-07-09 14:52:41 +08:00
Lynn
7fa2a0b607 Fix: rm role_id in user.go (#16771) 2026-07-09 14:44:08 +08:00
buua436
6a77523bf0 refa: resolve tenant model refs consistently (#16744) 2026-07-09 14:02:08 +08:00
Yingfeng
794fcc2517 Fix docs (#16770) 2026-07-09 13:38:29 +08:00
Yingfeng
b9ed0773ec Fix naive chunking for windows (#16767)
`\r` is ignored for splitting boundaries
2026-07-09 12:02:19 +08:00
qinling0210
ae96e636e9 Handle searching dataset without embedding model (#16742)
### Summary

Handle searching dataset without embedding model

In this PR, Searching datasets with different embedding models or
searching dataset with/without embedding models are not allowed. We will
improve the behavior later.
2026-07-09 11:38:55 +08:00
Lynn
1430d0e431 Fix: provider name (#16733) dev-20260709 2026-07-09 10:19:10 +08:00
balibabu
575984877f Fix: Rapid clicking results in multiple message requests being sent. (#16739) 2026-07-09 09:57:54 +08:00
euvre
3ec9187cd2 fix(web): prevent 'last saved at' label from vertical stacking in agent home card (#16756) 2026-07-09 09:46:03 +08:00
chanx
080dd84fed Feat: apply prose typography styling to markdown preview (#16752) 2026-07-09 09:45:52 +08:00
chanx
36f053a248 fix: Fixed the empty state styling on the home page. (#16755) 2026-07-09 09:45:43 +08:00
euvre
70019810a1 fix(web): show memory owner name in shared memory card (#16751) 2026-07-08 20:13:07 +08:00
euvre
74d3508a37 Fix: prevent auto-incrementing memory name suffix when only permissions change (#16750) 2026-07-08 20:12:45 +08:00
euvre
a41fef49d0 fix(web): hide folder tab in agent JSON import uploader (#16754) 2026-07-08 20:11:55 +08:00
Kevin Hu
2c59d07bdb Feat: add wiki folder (#16749)
### Summary

Add wiki folders.
2026-07-08 20:08:14 +08:00
qinling0210
8e3bbad4da Port agent PRs to GO - 5 (#16667)
### Summary

Port 

https://github.com/infiniflow/ragflow/pull/15376
https://github.com/infiniflow/ragflow/pull/16401
https://github.com/infiniflow/ragflow/pull/15484
https://github.com/infiniflow/ragflow/pull/16685
2026-07-08 19:54:29 +08:00
Virus
0f08dc070d fix: update nginx version. fix CVE-2026-9256 (#16732)
### Summary
Updates the NGINX package used in the RAGFlow Docker image from
`1.31.0-1~noble` to `1.31.2-1~noble`.

NGINX 1.31.0 is affected by CVE-2026-9256. NGINX 1.31.2 includes
the corresponding security fix and is available from the official
NGINX mainline repository for Ubuntu Noble.

### References
- nginx security advisories:
https://nginx.org/en/security_advisories.html
- Vendor advisory: https://my.f5.com/manage/s/article/K000161377

### Fix
Single-line change in `Dockerfile:62`:

```diff
-ARG NGINX_VERSION=1.31.0-1~noble
+ARG NGINX_VERSION=1.31.2-1~noble

Co-authored-by: duanyuan <duanyaun@uyuyue.com>
2026-07-08 19:19:06 +08:00
Jack
0dd0ac06f8 Feature: task executor migration to go (#16549)
### Summary

Feature: Integrate parser
2026-07-08 19:08:11 +08:00
Wang Qi
c8d1b21ae3 Fix Build-in metadata not working (#788) (#16748) 2026-07-08 19:06:54 +08:00