## Summary
Align the Go implementations of these APIs with the Python behavior:
- `POST /api/v1/datasets/:dataset_id/metadata/update`
- `PATCH /api/v1/datasets/:dataset_id/documents/metadatas`
- `POST /api/v1/documents/upload`
## What changed
- Added the Go routes and handlers for the 3 APIs.
- Aligned batch document metadata updates with Python semantics:
- support `match` in update items
- support list append / replace behavior
- support deleting specific list values
- remove metadata entirely when it becomes empty
- create metadata for documents that previously had none when updates
apply
- count `updated` only when a document actually changes
- Aligned `documents/upload` file uploads with Python-style
`upload_info` behavior:
- store upload-info blobs in the per-user downloads bucket
- return lightweight upload descriptors instead of normal
file-management responses
- Improved URL upload behavior:
- SSRF-guarded fetch with redirect validation
- redirect limit aligned to Python behavior
- normalize filename and MIME type
- add `.pdf` when the fetched content is PDF
- normalize HTML content into readable text instead of storing raw HTML
shells
## Validation
### Unit tests
Passed:
- `go test ./internal/service`
- `go test ./internal/handler`
Also verified targeted cases for:
- batch metadata update semantics
- upload_info URL handling
- upload_info download bucket behavior
### curl checks
Verified the new Go endpoints with `curl` and compared the response
shape and behavior with Python for:
- `POST /api/v1/datasets/{dataset_id}/metadata/update`
- `PATCH /api/v1/datasets/{dataset_id}/documents/metadatas`
- `POST /api/v1/documents/upload`
The Go responses were checked against Python for:
- argument validation
- success response shape
- metadata update results
- upload_info result structure
- file vs URL input handling
### Description
Migrates the datasets tags aggregation API `GET
/api/v1/datasets/tags/aggregation` from Python to Go.
### Changes
- Registered the `GET /api/v1/datasets/tags/aggregation` route.
- Implemented `AggregateTags` in datasets `handler` and `service`.
- Added handler and service `unit tests`.
### Test Verification
- Verified by comparing results between Python (9380) and Go (9384)
services.
- Tested scenarios: single dataset, multiple datasets, empty parameters,
and unauthorized/invalid IDs.
- All tests and Go `unit tests` passed.
### What problem does this PR solve?
```
RAGFlow(api/default)> add admin host '127.0.0.1:9383';
SUCCESS
RAGFlow(api/default)> use admin;
SUCCESS
RAGFlow(admin)> delete api 'default';
SUCCESS
RAGFlow(admin)> delete api 'default';
CLI error: api server: default not found
RAGFlow(admin)> add api 'default' host '127.0.0.1:9384';
SUCCESS
RAGFlow(admin)> use api 'default';
SUCCESS
RAGFlow(api/default)> delete admin
SUCCESS
RAGFlow(api/default)> delete admin;
CLI error: admin server not exists
RAGFlow(api/default)> list api server;
+------------+---------------+-----------------+---------+
| api_server | api_server_ip | api_server_port | auth |
+------------+---------------+-----------------+---------+
| default | 127.0.0.1 | 9384 | no auth |
+------------+---------------+-----------------+---------+
RAGFlow(api/default)> add admin host '127.0.0.1:9383';
SUCCESS
RAGFlow(api/default)> show admin server;
+-------------------+-----------+
| field | value |
+-------------------+-----------+
| admin_server_ip | 127.0.0.1 |
| admin_server_port | 9383 |
| auth | no auth |
+-------------------+-----------+
```
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
Signed-off-by: Jin Hai <haijin.chn@gmail.com>
### What problem does this PR solve?
Fix release
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
---------
Signed-off-by: Jin Hai <haijin.chn@gmail.com>
## Summary
This PR fixes two issues discovered during testing of the PaddleOCR
async API refactoring:
### 1. PP-OCRv6 returns `ocrResults` instead of `layoutParsingResults`
Models like PP-OCRv6 are pure text recognition models that return
results in `ocrResults.prunedResult.rec_texts` format rather than the
`layoutParsingResults.prunedResult.parsing_res_list` format used by
layout-aware models (PaddleOCR-VL series).
**Changes:**
- `deepdoc/parser/paddleocr_parser.py`: Extract `ocrResults` alongside
`layoutParsingResults` in `_send_request()`, add fallback logic in
`_transfer_to_sections()` and `parse_image()`
- `internal/entity/models/paddleocr.go`: Add `ocrResults` struct and
fallback extraction in Go OCR handler
### 2. Image parsing not integrated into picture chunker
The `parse_image()` method existed in PaddleOCRParser but was never
called from `rag/app/picture.py` (the module that handles image file
uploads). Users configuring PaddleOCR as their layout recognizer would
still get local deepdoc OCR for images.
**Changes:**
- `rag/app/picture.py`: When `layout_recognize` is set to PaddleOCR, use
`PaddleOCROcrModel.parse_image()` instead of local OCR. Falls back
gracefully to local OCR on failure.
## Testing
Verified end-to-end in Docker:
- PaddleOCR-VL-1.6 PDF parsing: ✅ (10 text blocks with bbox)
- PaddleOCR-VL-1.6 image parsing: ✅ (219 chars)
- PP-OCRv6 PDF parsing with ocrResults fallback: ✅ (10 text blocks)
- PP-OCRv6 image parsing with ocrResults fallback: ✅ (136 chars)
## Related PRs
- #15967 (merged) - PaddleOCR async Job API refactoring + new models
- #16086 (merged) - PaddleOCR image parsing support
### What problem does this PR solve?
- Tools management
- Pregel engine wrapper for better usage
- UT race
- Coding style
### Type of change
- [x] Refactoring
### What problem does this PR solve?
```
RAGFlow(admin)> show model 'abc';
+------------+----------------------------------------------------------------+
| field | value |
+------------+----------------------------------------------------------------+
| command | get_model_by_model_name |
| error | 'get model by model name' is implemented in enterprise edition |
| model_name | abc |
+------------+----------------------------------------------------------------+
RAGFlow(admin)> list models;
+-----------------+--------------------------------------------------------+
| command | error |
+-----------------+--------------------------------------------------------+
| list_all_models | 'list all models' is implemented in enterprise edition |
+-----------------+--------------------------------------------------------+
```
### Type of change
- [x] Refactoring
Signed-off-by: Jin Hai <haijin.chn@gmail.com>
### What problem does this PR solve?
Implement:
1. `/api/v1/datasets/<dataset_id>/documents/<document_id>/chunks GET`
2.
`/api/v1/datasets/<dataset_id>/documents/<document_id>/chunks/<chunk_id>
PATCH`
3. `/api/v1/datasets/<dataset_id>/documents/<document_id>/chunks PATCH`
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
### What problem does this PR solve?
Provider default public key for CLI
### Type of change
- [x] Refactoring
Signed-off-by: Jin Hai <haijin.chn@gmail.com>
Refactor the Go agent port's logging so every log line — gin access,
agent canvas events, harness warnings, fatal boot errors — flows through
a single common.Logger (zap) backed by a rotated file, with structured
fields, level filtering, and configurable rotation.
---------
Co-authored-by: Claude <noreply@anthropic.com>
## Summary
Stabilizes the Go unit-test surface so the test suite can run reliably
in CI and locally via \`bash build.sh --test\`.
## Verification
\`\`\`bash
bash build.sh --test -- -count=10 -run TestWithCancel_SequentialAgent
./internal/harness/core/
bash build.sh --test -- -count=5 -run TestSiliconflowChatExtracts
./internal/entity/models/
bash build.sh --test # full suite
\`\`\`
All previously failing packages (\`admin\`, \`cli\`, \`handler\`,
\`parser\`,
\`router\`, \`service\`, \`service/chunk\`) now build and test
successfully.
\`TestWithCancel_SequentialAgent\` passes 10/10 (was flaky). SiliconFlow
reasoning test passes after switching the assertion to the SiliconFlow
wire
format.
---------
Co-authored-by: Claude <noreply@anthropic.com>
### What problem does this PR solve?
```
RAGFlow(admin)> show role 'user' default models;
+--------------------------+-----------------------------------------------------------------+-----------+
| command | error | role_name |
+--------------------------+-----------------------------------------------------------------+-----------+
| show_role_default_models | 'show role default models' is implemented in enterprise edition | user |
+--------------------------+-----------------------------------------------------------------+-----------+
RAGFlow(admin)> set role 'user' default chat 'glm4.5@test@zhipu-ai';
+------------+---------------------------------------------------------------+
| field | value |
+------------+---------------------------------------------------------------+
| model_id | |
| model_type | chat |
| role_name | user |
| command | set_role_default_model |
| error | 'set role default model' is implemented in enterprise edition |
+------------+---------------------------------------------------------------+
RAGFlow(admin)> reset role 'user' default chat;
+------------+-----------------------------------------------------------------+
| field | value |
+------------+-----------------------------------------------------------------+
| command | reset_role_default_model |
| error | 'reset role default model' is implemented in enterprise edition |
| model_type | chat |
| role_name | user |
+------------+-----------------------------------------------------------------+
```
---------
Signed-off-by: Jin Hai <haijin.chn@gmail.com>
## Summary
- add public Go route for `/api/v1/searchbots/detail`
- implement beta-token auth flow for shared search access
- add tenant-based access check for shared search apps
- add joined search detail query for the share response
- align Go response shape with the current Python runtime behavior
- add DAO / service / handler tests for the new endpoint
close#16132
## Summary
This PR completes the Go-side merge and cleanup for chat channel APIs,
including handler/service wiring, route registration, and test coverage.
Implemented and aligned 5 chat channel APIs:
```
- POST `/api/v1/chat-channels`
- GET `/api/v1/chat-channels`
- GET `/api/v1/chat-channels/:channel_id`
- PATCH `/api/v1/chat-channels/:channel_id`
- DELETE `/api/v1/chat-channels/:channel_id`
```
Co-authored-by: Haruko386 <tryeverypossible@163.com>
### What problem does this PR solve?
```
fixed:
RAGFlow(api/default)> use model 'minimax-m2.5@test@minimax'
SUCCESS
RAGFlow(api/default)> chat message 'who r u'
Answer: Hey! I'm MiniMax-M2.5, an AI assistant here to help you with questions, tasks, or whatever you need. What can I do for you?
Time: 1.727263
```
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
### What problem does this PR solve?
As title.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
---------
Signed-off-by: Jin Hai <haijin.chn@gmail.com>
### What problem does this PR solve?
- list configs;
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
---------
Signed-off-by: Jin Hai <haijin.chn@gmail.com>
### What problem does this PR solve?
- list resources
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
- [x] Refactoring
---------
Signed-off-by: Jin Hai <haijin.chn@gmail.com>
### What problem does this PR solve?
1. Remove unused file
2. Remove duplicate models
3. Resort the function order
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
- [x] Refactoring
Signed-off-by: Jin Hai <haijin.chn@gmail.com>
### What problem does this PR solve?
Implement OpenAI chat completions in GO
POST /api/v1/openai/<chat_id>/chat/completions
OpenAI chat cli: internal/development.md
### Type of change
- [x] Refactoring
### What problem does this PR solve?
1. add modelID for delete_model and update_status
2. fix the bug when update-status delete model
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
- [x] New Feature (non-breaking change which adds functionality)
### What problem does this PR solve?
RAGFlow(admin)> show tasks summary;
+---------+-----------------------------------------------------------------+
| field | value |
+---------+-----------------------------------------------------------------+
| command | show_users_quota_summary |
| error | 'Show users quota summary' is implemented in enterprise
edition |
+---------+-----------------------------------------------------------------+
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
Signed-off-by: Jin Hai <haijin.chn@gmail.com>
### What problem does this PR solve?
Prepare for enterprise command
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
---------
Signed-off-by: Jin Hai <haijin.chn@gmail.com>
### What problem does this PR solve?
- Migrated MCP server detail and export (download) API from Python to
Go.
- Registered route: `GET /api/v1/mcp/servers/:mcp_id` (supporting
`?mode=download` query parameter).
### What problem does this PR solve?
This PR implements the Go backend counterpart for the document partial
update API:
`PATCH /api/v1/datasets/:dataset_id/documents/:document_id`
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
- [x] Refactoring
### What problem does this PR solve?
This PR improves code readability in the CLI parser by renaming the loop
index `i` to `modelIndex`. It also renames the loop label `A` to
`optionsLoop` to align with standard Go naming conventions.
### Type of change
- [x] Refactoring
### What problem does this PR solve?
Fixes two bugs in the OpenRouter streaming chat request builder
(`internal/entity/models/openrouter.go`, `ChatStreamlyWithSender`):
1. **qwen/glm models streamed to a broken URL.** The code routed any
`qwen`/`glm` model to
`URLSuffix.AsyncChat`, but `conf/models/openrouter.json` defines no
`async_chat` suffix
(empty), so the request was POSTed to `<base>/` instead of
`<base>/chat/completions` —
breaking streaming for every qwen/glm model. The non-stream path has no
such branch.
Fix: all models use the standard `Chat` suffix, consistent with the
non-stream path.
2. **Streaming reasoning was never enabled.** The request set reasoning
via a non-standard
`thinking` key, which OpenRouter ignores. OpenRouter's API — and this
provider's own
non-stream request (line ~110) and its streamed `delta.reasoning` parser
(line ~311) —
use the `reasoning` object. Fix: send `reasoning: {"enabled":
<thinking>}` (and
`{"effort": ...}` when set, taking precedence as in the non-stream
path).
Closes#16110
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
### What problem does this PR solve?
Part of #15853 (provider model-list refactor).
Refactors **Ollama** `ListModels` onto the shared `ParseListModel`
pattern and fixes two correctness issues:
- **Endpoint:** switch the models suffix from `api/ps` (only
currently-running models) to `api/tags` (all installed models) — the
latter is what a model picker should show.
- **Parsing:** Ollama returns `{"models":[{"name","model"}]}`, a
non-OpenAI shape. Decode it into a typed struct, map the names into
`ModelList`, then enrich through `ParseListModel`. This removes the
previous unchecked type assertions (`result["models"].([]interface{})` /
`.(map[string]interface{})` / `.(string)`) that **panicked** when the
body was missing the `models` array or any field, and adds a fallback to
the `model` field when `name` is blank.
- Drops the no-op GET request body and a dead base-URL reassignment.
#### Drive-by fix
Shared gitee_test.go `DSModelList` -> `ModelList` compile fix (renamed
in #15900) so the models test package builds; auto-resolves against the
sibling #15853 PRs.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
- [x] Refactoring
Replaces the Python agent canvas runtime with a Go implementation that
runs inside `cmd/server_main`.
The canvas compiles into an eino Workflow that pauses on wait-for-user
via native Interrupt/Resume (no sentinel flag) and resumes from a
Redis-backed CheckPointStore.
All 21 Python agent components and ~35 tools are ported with functional
parity.
Sandbox providers now read their JSON config from the admin-panel
system_settings table with env fallback.
234 files / +35,413 / -6,111. All Go files are gofmt-clean (CI gate
added); drops the v2 DSL E2E step and the gap-analysis plan (both
redundant after the port ships).
## Type of change
- [x] Refactoring
- [x] New feature
- [x] Bug fix
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude <noreply@anthropic.com>
### What problem does this PR solve?
The `get_ingestion_log` endpoint (both Python
`dataset_api_service.get_ingestion_log` and Go
`DatasetService.GetIngestionLog`) was returning only the
**dataset-level** field set, which omits critical fields such as `dsl`,
`document_id`, `parser_id`, `document_name`, `pipeline_id`, etc.
This caused the front-end **dataflow-result page** to be unable to
render the pipeline timeline and chunks when viewing a single ingestion
log, regardless of whether the log was a dataset-level operation
(graph/raptor/mindmap) or a per-file parse.
### Background
`PipelineOperationLogService` provides two field sets:
| Method | Fields |
|---|---|
| `get_dataset_logs_fields` | Minimal set (progress, status, timestamps,
etc.) |
| `get_file_logs_fields` | Superset — includes `document_id`, `dsl`,
`parser_id`, `document_name`, `pipeline_id`, … |
When listing logs, the API correctly distinguishes dataset-level vs
file-level logs and uses the appropriate converter. However, when
**fetching a single log by ID**, both the Python and Go implementations
were hardcoded to the dataset-level set, dropping the extra fields that
the front-end needs.
### What problem does this PR solve?
Part of #15853 (provider model-list refactor). Final two providers.
- **voyage:** Voyage AI exposes no live model-list endpoint — its public
API only has `/v1/embeddings` and `/v1/rerank` — so the previous
`ListModels` was a `no such method` stub. Replace it with a
static-catalog listing sourced from the loaded provider definition,
carrying each model's `max_tokens`, `model_types`, and embedding
`dimensions`. `list models from voyage` now returns the 13-model catalog
instead of erroring.
- **fishaudio:** route the existing `/model` voice listing through the
shared `ParseListModel` helper for consistency; keep the human-readable
`title` as the model name and fall back to `_id` when a title is blank.
#### Drive-by fix
Shared gitee_test.go `DSModelList` -> `ModelList` compile fix (renamed
in #15900); auto-resolves against the sibling #15853 PRs.
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
- [x] Refactoring
Co-authored-by: Haruko386 <tryeverypossible@163.com>
## Changes
1. **Entity (`internal/entity/chat_channel.go`)**:
- Implemented `ChatChannel` struct mapping the `chat_channel` database
table.
- Declared `ChatChannelListResponse` as a DTO to filter out sensitive
credentials (`config` field) and fetch the associated `dialog_name` via
left join.
2. **GORM Migration (`internal/dao/database.go`)**:
- Registered `&entity.ChatChannel{}` in the `dataModels` array inside
`InitDB()` to enable safe GORM schema synchronization.
3. **DAO (`internal/dao/chat_channel.go`)**:
- Implemented `ChatChannelDAO` wrapping GORM CRUD methods (`Create`,
`GetByID`, `UpdateByID`, `DeleteByID`).
- Implemented `ListByTenantID` performing a `LEFT JOIN` on the `dialog`
table to retrieve `dialog_name` while excluding `config` values to avoid
credential leaks.
4. **Test (`internal/dao/chat_channel_test.go`)**:
- Added integration unit tests testing the full CRUD lifecycle and GORM
left-join mapping list querying.
## Summary
Migrate PaddleOCR integration from the deprecated synchronous HTTP API
to the new asynchronous Job API (`submit → poll → fetch`), aligning with
PaddleOCR 3.6.0+ architecture.
## Changes
### Python (`deepdoc/parser/paddleocr_parser.py`)
- Replace synchronous `requests.post()` with async Job API flow (submit
→ poll → fetch)
- Authentication: `token {token}` → `Bearer {token}`
- File transfer: base64 JSON body → multipart file upload
- Polling: exponential backoff (initial 3s, ×1.5, max 15s, timeout
controlled by `request_timeout`)
- Result: fetch full JSONL from result URL, preserving `prunedResult`
with bbox info for crop functionality
- Rename `api_url` → `base_url` (backward compatible: `api_url` still
accepted as fallback)
### Python (`rag/llm/ocr_model.py`)
- Prefer `paddleocr_base_url` / `PADDLEOCR_BASE_URL`, fallback to
`paddleocr_api_url` / `PADDLEOCR_API_URL`
### Go (`internal/entity/models/paddleocr.go`)
- Add `Client-Platform: ragflow` header to submit and poll requests
- Change polling from fixed 3s to exponential backoff (initial 3s, ×1.5,
max 15s)
### Python (`common/constants.py`)
- Add `PADDLEOCR_BASE_URL` to env keys and default config
## Backward Compatibility
- Old env var `PADDLEOCR_API_URL` still works (used as fallback)
- Frontend field `paddleocr_api_url` still works (backend reads it as
fallback)
- No user-facing configuration changes required for existing setups
## Why not use the `paddleocr` SDK package directly?
RAGFlow's `_transfer_to_sections()` relies on `prunedResult` (containing
`block_bbox`, `block_label`, `parsing_res_list`) from the raw API
response for PDF crop functionality. The SDK's public `parse_document()`
API only returns `DocParsingResult` with `markdown_text`, discarding the
bbox data. Therefore we implement the async Job API flow directly via
HTTP, following the same logic as the SDK internally.
