### What problem does this PR solve?
Two commands are used for ingestion file testing
```
RAGFlow(api/default)> chunk 'file' with 'dsl';
Chunk file: file, DSL: dsl
SUCCESS
RAGFlow(api/default)> parse file 'filename' chat 'xxx';
Success to parse local file "filename", vision: , chat: xxx, asr: , ocr: , embedding: , doc_parse:
SUCCESS
```
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
Signed-off-by: Jin Hai <haijin.chn@gmail.com>
### What problem does this PR solve?
`ChatStreamlyWithSender` in two Go model drivers could panic on nil
pointer dereferences when a caller passes a nil model config or omits
the reasoning `Effort`:
- **deepseek.go** - `switch *chatModelConfig.Effort` dereferenced
`Effort` without a nil check. It now defaults to `"high"` when nil.
- **volcengine.go** - the `modelConfig` pointer itself was dereferenced
(`Stream`, `MaxTokens`, `Temperature`, .) with no guard, and `Effort`
was dereferenced unchecked. `modelConfig` now defaults to an empty
`&ChatConfig{}` when nil so the optional-field accesses are safe, and
`Effort` defaults to `"medium"` when nil.
Addresses the CodeRabbit review on `volcengine.go`
`ChatStreamlyWithSender`. Per maintainer feedback ("one PR do one
thing"), the unrelated `handler/auth.go` and
`service/heartbeat_sender.go` changes were removed so this PR is scoped
to the model-provider fixes.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
## Summary
Implements **chunk 1** of #15282 — the four `/api/v1/auth/password/...`
endpoints from the login-page Go port. **Chunk 2 (OAuth/OIDC) is
deferred** to its own subtask, matching the issue author's own
confidence-low recommendation ("multi-provider, stateful redirect flow
with external dependencies; recommend its own subtask").
New endpoints, all registered under `apiNoAuth` (forgot-password users
are unauthenticated by definition):
| Method | Path | Status |
|--------|------|--------|
| `POST` | `/api/v1/auth/password/forgot/captcha` | new |
| `POST` | `/api/v1/auth/password/forgot/otp` | new |
| `POST` | `/api/v1/auth/password/forgot/otp/verify` | new |
| `POST` | `/api/v1/auth/password/reset` | new |
## Wire compatibility with the Python backend
The two backends share state through Redis, so the Go port had to use
identical keys, encodings, and constants. Either backend can now
validate a code the other minted.
- **Redis keys**: `captcha:<email>`, `otp:<email>`,
`otp_attempts:<email>`, `otp_last_sent:<email>`, `otp_lock:<email>`,
`otp:verified:<email>` — same as `api/utils/web_utils.py`.
- **Stored OTP value**: `"<hex_hash>:<hex_salt>"` — same as Python.
- **Hash**: HMAC-SHA256 with a `crypto/rand` 16-byte salt — same as
`hash_code()`.
- **Constants**: `OTP_LENGTH=4`, `OTP_TTL=5min`, `ATTEMPT_LIMIT=5`,
`ATTEMPT_LOCK_SECONDS=30min`, `RESEND_COOLDOWN_SECONDS=60s` — all match
`api/utils/web_utils.py`.
- **Email body**: matches `RESET_CODE_EMAIL_TMPL` byte-for-byte.
## Files
### New
| File | Purpose |
|---|---|
| `internal/utility/otp.go` | OTP/captcha constants, Redis key builders
(`CaptchaRedisKey`, `OTPRedisKeys`, `OTPVerifiedRedisKey`),
`HashOTPCode`, `GenerateOTPCode` / `GenerateCaptchaCode` /
`GenerateOTPSalt` via `crypto/rand`, and `EncodeOTPStorageValue` /
`DecodeOTPStorageValue` matching Python's storage shape. |
| `internal/utility/smtp.go` | Minimal stdlib `net/smtp` sender.
`SendResetCodeEmail(to, otp, ttlMin)` builds an RFC 5322 plain-text
message and dispatches via implicit TLS / STARTTLS / plain — same
selectors as Python `aiosmtplib`. Returns `SMTPNotConfiguredError` if
the config block is empty. |
### Modified
| File | Change |
|---|---|
| `internal/server/config.go` | New `SMTPConfig` struct + `Config.SMTP`
field. Field names mirror the `smtp:` keys in `common/settings.py`
(`mail_server`, `mail_port`, `mail_use_ssl`, `mail_use_tls`,
`mail_username`, `mail_password`, `mail_from_name`, `mail_from_address`,
`mail_frontend_url`) so a single `conf/service_conf.yaml` powers both
backends. |
| `internal/service/user.go` | Four methods — `ForgotIssueCaptcha`,
`ForgotSendOTP`, `ForgotVerifyOTP`, `ForgotResetPassword`. Reuses the
existing `decryptPassword`, `HashPassword`, `userDAO.Update`, and
`utility.GenerateToken` so the reset+auto-login path is identical to
`LoginByEmail`. |
| `internal/handler/user.go` | Four handlers in the same `c.JSON` shape
as `LoginByEmail`. The reset handler rotates the access token and emits
an `Authorization` header for auto-login (matches Python
`construct_response(auth=user.get_id())`). |
| `internal/router/router.go` | Routes registered under `apiNoAuth`,
with an explanatory comment on why they sit outside the auth middleware.
|
## Known divergence — captcha rendering
The Python endpoint returns a rendered `image/JPEG` from the
`python-captcha` library. The Go side has **no image-captcha dependency
vendored** in `go.mod`, and hand-rolling a raster generator was out of
scope for this PR.
This commit returns JSON `{captcha: "<text>"}` instead. Implications:
- **Backend gate is identical** — the OTP step still verifies the
user-submitted captcha string against the Redis value, so the security
model is unchanged.
- **Frontend impact**: the password-reset page rendering needs a small
tweak (text display instead of `<img>`) until a Go captcha library is
wired in.
- The handler comments call this out explicitly so the next PR knows
what to swap.
Possible follow-ups (any one closes the gap):
1. Add `github.com/mojocn/base64Captcha` or `github.com/dchest/captcha`
to `go.mod` and replace the JSON response with an `image/JPEG`.
2. Hand-roll a 5x7 bitmap font + `image/png` writer using only the
stdlib.
3. Render a server-side SVG (cheap, but trivially OCR-able — only useful
as a UI shim).
## Test plan
- [ ] **Captcha**: `POST
/api/v1/auth/password/forgot/captcha?email=<existing>` returns `{code:
0, data: {captcha: "ABCD"}}`. Redis shows `captcha:<email>` with that
value and ~60s TTL. Unknown email returns `code: CodeDataError`.
- [ ] **OTP send**: `POST /api/v1/auth/password/forgot/otp` with the
right captcha mints an OTP, stores `<hash>:<salt>` under `otp:<email>`
for 5 min, sends an email, returns success. With a wrong captcha returns
`CodeAuthenticationError`. Hitting it again within 60s returns "you
still have to wait …" with `CodeNotEffective`.
- [ ] **OTP verify**: correct OTP → `code: 0`, OTP keys cleared,
`otp:verified:<email>` = `"1"`. Wrong OTP → `code:
CodeAuthenticationError`, attempt counter bumped; after 5 wrong tries
`otp_lock:<email>` is set and further attempts hit `CodeNotEffective`.
- [ ] **Reset**: with the verified flag set, supply a new password
(RSA-encrypted+base64, same as `LoginByEmail`). Returns `code: 0`,
`Authorization` header set, verified flag deleted. Without the verified
flag returns `CodeAuthenticationError`.
- [ ] **Wire-compat smoke**: mint an OTP from the Python backend, verify
it via the Go endpoint, and vice versa. Should both succeed.
- [ ] **SMTP misconfigured**: drop `smtp.mail_server` from
`conf/service_conf.yaml`. The OTP-send endpoint should now return
"failed to send email" without panicking; check the log for the
`SMTPNotConfiguredError` warning.
- [ ] **End-to-end FE**: hit the password-reset flow from
`web/src/pages/login-next/`. Confirm the text-captcha shim works after
the FE tweak.
- [ ] `go build ./...` and `go vet ./...` — I could not run these in the
sandbox; please confirm a clean build before merging.
- [ ] `uv run pytest` to confirm no Python regressions (shared Redis
schema).
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
## Summary
- Implemented the Go API endpoint for Memory message forgetting:
- `DELETE /api/v1/messages/{memory_id}:{message_id}`
- Added route registration for the Memory message DELETE endpoint only.
- Added request path validation for `memory_id:message_id`.
- Added service logic to mark a message as forgotten by setting
`forget_at`.
- Preserved Python-compatible response behavior:
- Success returns `code: 0`, `message: true`, `data: null`.
- Added focused unit tests for message path parsing and invalid message
ID handling.
- Fixed Linux cgo linker config to use the installed shared PCRE2
library so Go tests/builds can run in this environment.
## Related Issue
Closes: #15240
## Change Type
- [x] Feature
- [x] Test
- [x] Build / CI compatibility
## Implemented API
- `DELETE /api/v1/messages/{memory_id}:{message_id}`
## Real Behavior Proof
Validated with targeted Go tests:
```bash
/tmp/go1.25.0/bin/go test ./internal/handler ./internal/router
```
Result:
```text
ok ragflow/internal/handler
? ragflow/internal/router [no test files]
```
Validated server entrypoint build:
```bash
/tmp/go1.25.0/bin/go build -o /tmp/ragflow-server-main ./cmd/server_main.go
```
Result:
```text
build succeeded
```
Validated patch formatting:
```bash
git diff --check
```
Result:
```text
no whitespace errors
```
## Checklist
- [x] Implemented only `DELETE
/api/v1/messages/{memory_id}:{message_id}`.
- [x] Did not implement unrelated Memory message APIs.
- [x] Added route registration.
- [x] Added handler validation.
- [x] Added service-level memory access check.
- [x] Added tests.
- [x] Ran targeted Go tests.
- [x] Ran server build validation.
- [x] Ran `git diff --check`.
### What problem does this PR solve?
1. Fix go test, some cases still failed.
2. Remove unused code.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
---------
Signed-off-by: Jin Hai <haijin.chn@gmail.com>
## Summary
- Normalize model alias index keys to lowercase
- Detect lowercase alias collisions during provider manager
initialization
- Fix ListModels metadata mapping for mixed-case provider aliases
## What does this PR do?
This PR migrates the Agent Temporary File Download endpoint (`GET
/api/v1/agents/download`) from the Python backend to the Go backend,
optimizing the data retrieval flow and maintaining strict functional
parity. It also fixes a persistent parsing error in the Sandbox code
execution node.
## Checklist
- [x] Code logic matches Python implementation
- [x] All local unit tests passed
- [x] No breaking changes to existing router interfaces
Co-authored-by: Yingfeng <yingfeng.zhang@gmail.com>
### What problem does this PR solve?
1. remove unused code
2. fix login issue
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
- [x] Refactoring
Signed-off-by: Jin Hai <haijin.chn@gmail.com>
This PR expands conf/all_models.json with DeepSeek model entries and
provider aliases.
Changes:
- Added DeepSeek model entries across `V4`, `V3.2`, `V3.1`, `V3`, `R1`,
`Coder`, `Math`, `VL`, `OCR`, `Prover`, `MoE`, and `LLM` series.
- Normalized model name values to lowercase canonical IDs.
- Added alias values for official DeepSeek/Hugging Face names and
provider-specific names from OpenRouter, VolcEngine, SiliconFlow,
HuaweiCloud, and QiniuCloud.
- Preserved model metadata such as max_tokens, model_types, and thinking
where applicable.
- Added Gitee ListModels tests to verify DeepSeek aliases map back to
model metadata from all_models.json.
- Added an optional Gitee integration test gated by
GITEE_LIST_MODELS_INTEGRATION=1.
Test:
/usr/local/go/bin/go clean -cache
/usr/local/go/bin/go test ./internal/entity/models -run
'TestGiteeListModels(MapsAllDeepSeekAliasesToModelMetadata|KeepsOwnedBySuffixAfterAliasMetadataLookup|
Integration)'
## Summary
- Add knowledge-base retrieval support to Go chat completions.
## What changed
- Routes KB-backed chat sessions through the Go retrieval service
instead of falling back to solo chat.
- Resolves embedding and rerank models, validates accessible knowledge
bases, and preserves tenant-aware retrieval.
- Rejects mixed embedding models across selected knowledge bases before
retrieval to avoid incompatible vector dimensions.
- Threads the HTTP request context into streaming retrieval so cancelled
requests can stop downstream retrieval work.
- Applies metadata filters and message-level `doc_ids` before retrieval.
- Expands parent/child chunks before building references and prompt
context.
- Injects retrieved knowledge through a copied dialog prompt config so
the caller's original dialog is not mutated.
- Honors configured empty responses when no chunks are found.
- Names the metadata no-match sentinel and reuses it across
retrieval/handler paths.
- Adds a defensive content cast while appending streamed answers.
- Adds focused unit coverage for retrieval, metadata filtering,
authorization, multimodal messages, references, empty-response behavior,
prompt immutability, and mixed embedding models.
---------
Co-authored-by: Yingfeng <yingfeng.zhang@gmail.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
### What problem does this PR solve?
```
RAGFlow(api/default)> use admin
SUCCESS
RAGFlow(api/default)> use api 'abc';
SUCCESS
```
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
---------
Signed-off-by: Jin Hai <haijin.chn@gmail.com>
### What problem does this PR solve?
As title
```
/api/v1/datasets/<dataset_id>/documents/<document_id>/metadata/config PUT
```
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
Co-authored-by: Yingfeng <yingfeng.zhang@gmail.com>
### What problem does this PR solve?
Add two API in go
```
/api/v1/agents/test_db_connection POST
/api/v1/agents/<agent_id>/sessions DELETE
```
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
---------
Co-authored-by: Yingfeng <yingfeng.zhang@gmail.com>
Implements POST /api/v1/searchbots/ask in Go with streaming SSE,
citations, and think-tag processing. 23 files, 90+ unit tests.
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
## Summary
This PR re-enables the Go test steps in CI that were previously
commented out, and fixes all compilation errors that have accumulated in
`internal/entity/models/` since the `ListModels` return type was changed
from `[]string` to `[]ListModelResponse`.
## Changes
### CI (`.github/workflows/tests.yml`)
- Re-enable **Prepare test resources** step (clones resource repo with
WordNet data)
- Re-enable **Test Go packages** step (runs `go test ./internal/...`)
- Fix resource path race condition by using
`/tmp/resource-${GITHUB_RUN_ID}` instead of `/tmp/resource`
- Exclude `/cli` package from Go tests (contains `main` redeclarations)
### Test fixes (16 model provider test files)
All errors were caused by the upstream change from `[]string` to
`[]ListModelResponse` in the `ListModels` interface:
- Add `joinModelNames` test helper to extract `.Name` from
`[]ListModelResponse` slices
- `strings.Join(models, ",")` → `joinModelNames(models, ",")` (11 files)
- `ids[i] != "..."` → `ids[i].Name != "..."` (cometapi, mistral)
- `got[i] != want[i]` → `got[i].Name != want[i]` (bedrock)
- `[]string` return types → `[]ListModelResponse` (google)
### Pre-existing bugs in model_test.go
Bugs introduced by the upstream `entity/` → `entity/models/` directory
rename:
- Add missing `pm := GetProviderManager()` calls in 3 test functions
- Fix `InitProviderManager` signature (`_, err :=` → `err :=`)
- Fix `MaxTokens` `*int` dereference (6 comparisons)
- Fix `readProviderConfig` relative path (3 levels up instead of 2)
### model.go
- Add `findRepoRoot()` to make `conf/all_models.json` resolution work
from any CWD, fixing `TestSiliconFlowProviderConfigLoadsLatestProModels`
### Test validation
```bash
go build ./internal/... # ✅
go test ./internal/entity/models/... -count=1 # ✅ all pass
```
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
## Summary
- Complete the Go Elasticsearch result functions that remained stubbed
after #15160.
- Add focused unit coverage for field mapping, aggregation, IDs, and
highlighting behavior.
- Update a stale query-builder test type import discovered during
validation.
## What changed
- Keep the Elasticsearch Go implementation merged in #15160 and fill in
`GetFields`, `GetAggregation`, `GetHighlight`, and `GetDocIDs` in
`internal/engine/elasticsearch/chunk.go`.
- Add regression and invariant coverage in
`internal/engine/elasticsearch/chunk_helpers_test.go`.
- Update `internal/service/nlp/query_builder_test.go` to use the current
`types.MatchTextExpr` type.
## Why
- #15160 implemented the main Go Elasticsearch surface, but
retrieval/tag flows still call result functions that returned stubs.
- Completing these functions keeps Elasticsearch result processing
aligned with the expected document-engine behavior for field extraction,
tag aggregation, doc ID extraction, and snippet highlighting.
## Validation
- `go test ./internal/engine/elasticsearch`
- `GOARCH=arm64 CGO_ENABLED=1 go test ./internal/service/nlp -run
TestQueryBuilder`
- `git diff --check`
- CodeRabbit review reported 0 issues after follow-up fixes.
- Codex Security diff scan found no reportable issues.
## Notes
- This PR is now a follow-up to #15160 rather than a competing
implementation.
- A full local `go test ./internal/service/nlp` run is blocked by local
WordNet resource prerequisites; the query-builder tests touched by this
PR pass with the arm64 CGO path.
### What problem does this PR solve?
Fixes four Go paths that dereference a pointer with no prior nil check,
each
causing a **runtime panic**. Closes#15814.
| # | File | Bug | Fix |
|---|------|-----|-----|
| 1 | `internal/entity/models/deepseek.go` | streaming path runs `switch
*chatModelConfig.Effort` inside `if *Thinking`; panics when
`Thinking=true` and `Effort==nil` | nil-check with default `"high"`,
matching the non-streaming path in the same file |
| 2 | `internal/entity/models/volcengine.go` | identical oversight:
`switch *modelConfig.Effort` with no guard | nil-check with default
`"medium"`, matching its non-streaming path |
| 3 | `internal/handler/auth.go` | `AuthMiddleware` does `if
*user.IsSuperuser`; panics on every authenticated request when the DB
column is `NULL` | guard with `user.IsSuperuser != nil &&`, matching
every other call site (`admin/handler.go`, `admin/service.go`,
`user.go`) |
| 4 | `internal/service/heartbeat_sender.go` |
`responseBody["code"].(float64)` panics on any non-200 response lacking
a numeric `code`; the upstream `recover()` calls `Fatal()` →
`os.Exit(1)`, taking down the whole server | comma-ok assertion (`code,
ok := ...`); return an error instead of panicking |
- [x] Bug Fix (non-breaking change which fixes an issue)
### What problem does this PR solve?
Fixes four panic / spurious-error paths in the Go model layer. Closes
#15818.
| # | File | Bug | Fix |
|---|------|-----|-----|
| 1 | | Thinking-mode streaming path: accessed unconditionally; Gemini
emits usage-only chunks with an empty slice, causing a runtime panic |
Guard each step: , , before indexing |
| 2 | | is a plain for ordinary requests; the cast to silently returns ,
then panics immediately | Switch on concrete type; handle both and |
| 3 | | Identical panic on the streaming path | Same switch-on-type fix
|
| 4 | | The field is optional (absent for non-thinking models) but the
code returned an error when it was missing, breaking every ordinary
Ollama completion | Change to a silent comma-ok assertion; is empty
string when the field is absent |
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
### Description
This PR ports the `GET /api/v1/agents/prompts` endpoint from the Python
backend to the Go backend.
### Changes Made
- **Handler**: Added `GetPrompts` method to `internal/handler/agent.go`.
- **Router**: Registered the `agents.GET("/prompts")` route in
`internal/router/router.go`.
- **Logic**: Leveraged the existing `service.LoadPrompt` utility to read
`analyze_task_system`, `analyze_task_user`, `next_step`, `reflect`, and
`citation_prompt` templates directly from the `rag/prompts` directory.
- **Unit Test**: Added `TestGetPrompts_Success` to
`internal/handler/agent_test.go` to mock the HTTP context and validate
the JSON response structure.
### Motivation
This is part of the ongoing effort to port the Agent API surface to Go.
Since this specific endpoint only serves static prompt templates and
does not require the complex DAG/Canvas execution engine, it can be
seamlessly and safely handled by the Go backend.
### Testing
- [x] Added automated unit test `TestGetPrompts_Success` (verified
passing).
- [x] Tested locally via `curl` against the Go server (port 9380) and
Python server (port 9384).
- [x] Verified that the Go JSON response structure and loaded prompt
text are logically 100% identical to the Python implementation.
## Problem
`TestXiaomiNewModelWithCustomDefaultTransport` panics on Go 1.25:
```
panic: interface conversion: http.RoundTripper is models.roundTripperFunc, not *http.Transport
```
In Go 1.25, `http.DefaultTransport` is no longer `*http.Transport`, so
the unchecked type assertion in `NewXiaomiModel` panics when the test
replaces it with a `roundTripperFunc`.
## Fix
Use a safe type assertion with fallback to a new `http.Transport`,
matching the pattern already used in `modelscope.go`.
## Verification
```bash
go test -run TestXiaomiNewModelWithCustomDefaultTransport ./internal/entity/models/...
# PASS
```
Internal contributors only.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
## What problem does this PR solve?
Go test files are never compiled in CI — only production binaries via
`go build`. This allowed a missing `"sort"` import in
`metadata_filter_test.go` to be merged without detection.
## Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
- [x] New Feature (non-breaking change which adds functionality)
## Changes
- Add `go test -count=1 ./internal/...` step after Go build in CI
workflow
- Fix missing `"sort"` import in `metadata_filter_test.go` (pre-existing
compile error)
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
## Summary
Normalizes Qwen model-family names before reasoning extraction so
provider-prefixed Qwen models use the existing `<think>...</think>`
fallback.
## Summary
- keep MiniMax chat calls in non-streaming mode and streaming calls in
SSE mode
- make MiniMax model listing and connection checks use a bodyless GET
/v1/models
- add focused MiniMax request/response regression tests
### What problem does this PR solve?
**Verified from CLI**
```
RAGFlow(user)> chat with 'mimo-v2.5@test@xiaomi' message 'who r u'
Answer: Hello! I'm MiMo-v2.5, a large language model developed by Xiaomi's LLM Core Team. You can think of me as a friendly AI assistant ready to help you answer questions, have conversations, or work on creative tasks. My context window can handle up to 1 million tokens, so we can dive into pretty long discussions or documents if you'd like. What can I help you with today?
Time: 3.831830
RAGFlow(user)> stream chat with 'mimo-v2.5@test@xiaomi' message 'who r u'
Answer: there! I'm MiMo-v2.5, an AI assistant created by the Xiaomi LLM Core Team. I'm here to chat, help out, answer questions, or just have a friendly conversation. Think of me as a helpful buddy with a pretty big memory (1 million tokens worth!). What can I do for you today?😊
Time: 2.421630
RAGFlow(user)> think chat with 'mimo-v2.5@test@xiaomi' message 'who r u'
Thinking: The user is asking a simple question about who I am. According to my system prompt, I should:
- Identify myself as **MiMo-v2.5**
- State that I was developed by the **Xiaomi LLM Core Team**
- Answer in first person and be warm and conversational
Answer: Hey there! 👋
I'm **MiMo**, an AI assistant created by the **Xiaomi LLM Core Team**. Think of me as a friendly chat buddy who's here to help you with all sorts of questions and tasks!
I love having conversations, answering questions, brainstorming ideas, and helping people figure things out. Whether you want to chat, need help with something specific, or just want to explore ideas together — I'm here for it! 😊
What can I help you with today?
Time: 6.651589
RAGFlow(user)> tts with 'mimo-v2.5-tts@test@xiaomi' text 'hello? show yourself' play format 'wav' param '{"voice": "Chloe"}'
SUCCESS
RAGFlow(user)> asr with 'mimo-v2.5-asr@test@xiaomi' audio './internal/test.wav' param '{"language": "zh"}'
+------------------------------------------------------------------------------------------------------------------------+
| text |
+------------------------------------------------------------------------------------------------------------------------+
| 1 The examination and testimony of the experts enabled the commission to conclude that five shots may have been fired. |
+------------------------------------------------------------------------------------------------------------------------+
```
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
- [x] Refactoring
## Summary
- keep Moonshot chat calls in non-streaming mode and streaming calls in
SSE mode
- make Moonshot model listing and balance checks use bodyless GET
requests
- add focused Moonshot request/response regression tests
## What problem does this PR solve?
Implements `FetchChunkVectors` — the infrastructure needed to hydrate
chunk embedding vectors on demand. This is a prerequisite for
`insert_citations` (citation insertion in the `searchbots/ask`
endpoint), matching the Python `Dealer.fetch_chunk_vectors` pattern.
Without this, citation insertion cannot compute answer-vs-chunk vector
similarity.
## Type of change
- [x] New Feature (non-breaking change which adds functionality)
## Changes
### New Function
- `FetchChunkVectors(engine, chunkIDs, tenantIDs, kbIDs, dim)` — fetches
embedding vectors for a set of chunk IDs
- Consumer-side `vectorFetcher` interface with only `Search` + `GetType`
methods
- Both `*elasticsearchEngine` and `*infinityEngine` implicitly satisfy
the interface
### Engine Behavior
- **ES**: queries by chunk ID list via `Search` with filter `{"id":
chunkIDs}`, parses tab-separated `q_N_vec` string format
- **Infinity / OceanBase**: skips the round-trip (vectors already
shipped with chunks)
- **Degrades gracefully**: engine errors return zero vectors — citation
insertion will use placeholders instead of failing
### Vector Parsing
- Handles ES tab-separated string format (`"0.1\t0.2\t0.3"`)
- Handles `[]float64` and `[]interface{}` formats
- Returns zero vector for wrong-dimension or unparseable input
### Bug Fix
- `metadata_filter_test.go`: add missing `"sort"` import (pre-existing
build break)
### Tests
- 12 unit tests: empty input, Infinity/OceanBase skip, ES string vector,
ES float slice, ES interface slice, search error degradation, missing
chunk → zero, wrong dimension → zero, parse edge cases
## Files Changed
| File | Change |
|------|--------|
| `internal/service/chunk_vector.go` | New — FetchChunkVectors + parse
helpers |
| `internal/service/chunk_vector_test.go` | New — 12 tests |
| `internal/service/metadata_filter_test.go` | Fix missing `"sort"`
import |
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
### What problem does this PR solve?
As Title
Codes were tested by Postman
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
## What problem does this PR solve?
Implements `POST /api/v1/searchbots/retrieval_test` in the Go API
server, aligning with the Python `bot_api.py` counterpart. Also applies
security hardening and consistency fixes discovered during CTO-level
code review:
- **Missing endpoint**: `retrieval_test` was not available in Go,
requiring Python fallback
- **Security**: Both `chunkHandler` and `searchBotHandler` leaked
`err.Error()` to API consumers
- **Python alignment**: Default values, empty question handling, and
`top_k <= 0` validation differed from Python behavior
- **Test gaps**: `chunkHandler.RetrievalTest` had zero unit tests;
several edge cases uncovered
## Type of change
- [x] New Feature (non-breaking change which adds functionality)
- [x] Bug Fix (non-breaking change which fixes an issue)
- [x] Refactoring
## Summary
### New Endpoint
- `POST /api/v1/searchbots/retrieval_test` — retrieval test with full
field support (page, size, top_k, use_kg, cross_languages, keyword,
similarity_threshold, vector_similarity_weight)
### New Type
- `common.StringSlice` — JSON type that accepts both `"kb1"` and
`["kb1", "kb2"]`, matching Python API flexibility
### Security
- Both `searchBotHandler` and `chunkHandler` now use `common.Warn()` +
generic error messages instead of leaking `err.Error()` to API consumers
- All error responses include consistent `"data": nil` shape
- `chunkHandler.RetrievalTest` uses interface-based DI (`chunkService`)
to enable testability
### Python Alignment
- Handler-level defaults align with Python `bot_api.py` (page=1,
size=30, top_k=1024, similarity_threshold=0.0,
vector_similarity_weight=0.3)
- `top_k <= 0` validation matching Python behavior
- Empty/whitespace question returns 200 + empty result (matches
`chunk_api.py`)
- `chunkHandler` `Datasets` field uses `common.StringSlice` for
string-or-array flexibility
### Refactoring
- `ChunkServiceIface` → `ChunkRetriever`, `chunkSvcIface` →
`chunkService` (Go-conventional naming)
- Extracted `applyRetrievalDefaults`, `toRetrievalServiceRequest` from
handler body
- Regex moved to package-level var in `parseRelatedQuestions`
- `service.RetrievalTestRequest.Datasets` type changed to
`common.StringSlice`
- `chunkHandler` now uses consumer-side interface for DI
### Tests
- 37 unit tests across both handlers: auth, validation, defaults,
StringSlice edge cases, empty/whitespace KbID, service errors, JSON
format, `top_k <= 0`, field mapping verification
## Files Changed
| File | Change |
|------|--------|
| `cmd/server_main.go` | Wire new handler + chunkService +
difyRetrievalHandler |
| `internal/common/json_types.go` | New StringSlice type |
| `internal/common/json_types_test.go` | StringSlice tests |
| `internal/handler/chunk.go` | Interface-based DI, security, Python
alignment, defaults |
| `internal/handler/chunk_test.go` | New — 9 comprehensive tests |
| `internal/handler/searchbot.go` | New endpoint + refactoring + `top_k
<= 0` validation |
| `internal/handler/searchbot_test.go` | 18 tests covering all edge
cases |
| `internal/router/router.go` | Register new route +
difyRetrievalHandler |
| `internal/service/chunk.go` | Datasets type → StringSlice, Question
binding relaxed |
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
### What problem does this PR solve?
Closes#15611.
RAGFlow's fallback reasoning parser only recognized the exact model
family `qwen3`. For provider-prefixed Qwen model names such as
SiliconFlow's `qwen/qwen3-8b`, the derived model class can be
`qwen/qwen3`, so inline `<think>...</think>` content was not split from
the visible answer when `reasoning_content` was absent.
This PR normalizes model-family detection before fallback reasoning
extraction, keeps the parser nil-safe, and adds focused tests for Qwen3
variants plus Gitee and SiliconFlow chat responses.
It also makes SiliconFlow propagate `ChatConfig.Thinking` into the chat
request body, matching the existing Gitee behavior, so Qwen thinking
mode is actually enabled when requested.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
- [x] Refactoring
### Validation
- `/root/go/bin/gofmt -l internal/entity/models/common.go
internal/entity/models/common_test.go
internal/entity/models/reasoning_family_provider_test.go
internal/entity/models/siliconflow.go`
- `git diff --check`
- `/root/go/bin/go test ./internal/entity/models -run
'Test(NormalizeModelFamily|GetThinkingAndAnswer|GiteeChatExtractsQwenThinkingFromInlineContent|SiliconflowChatExtractsProviderPrefixedQwenThinkingFromInlineContent)'
-vet=off -count=1`
Note: the full package command `/root/go/bin/go test
./internal/entity/models -vet=off -count=1` now runs locally, but it
currently fails on an unrelated existing
`TestAstraflowEmbedReturnsNoSuchMethod` panic in
`internal/entity/models/astraflow.go:482`.
## Summary
Port the Python `GET /v1/plugin/tools` endpoint to the Go API server.
Listed in the Go-API port checklist of #15240.
Returns the metadata of every embedded LLM tool plugin in the same JSON
shape the Python endpoint emits (camelCase keys preserved), so existing
frontends bind to the Go server without changes.
### Description
This PR syncs the `documentServiceIface` interface and introduces
handler methods for document preview, artifact fetching, and downloading
in the Go API. It also ensures that strict dataset alignment and access
checks are enforced when retrieving or downloading documents.
Furthermore, this PR introduces comprehensive unit tests for both the
newly added Handler and Service methods to ensure robustness and prevent
future regressions.
### Key Changes
* **Router & Handler Integration**:
* Added and wired new API endpoints in `internal/router/router.go`.
* Synchronized the `documentServiceIface` with `GetDocumentArtifact`,
`GetDocumentPreview`, and `DownloadDocument`.
* Implemented handlers for these endpoints in
`internal/handler/document.go`.
* **Access & Validation Enforcement**:
* Refactored `internal/service/document.go` to strictly check if a
document belongs to the requested dataset before allowing downloads or
previews.
* Added robust artifact file sanitization (`sanitizeArtifactFilename`)
and attachment handling (`shouldForceArtifactAttachment`).
* **Comprehensive Unit Testing**:
* **Handler Layer (`internal/handler/document_test.go`)**: Added mock
service implementations and Gin router tests covering success,
not-found, and internal error states for all 3 new endpoints.
* **Service Layer (`internal/service/document_test.go`)**: Added
extensive business logic tests including dataset mismatch checks,
non-existent document checks, and artifact file validation.
## Summary
This PR improves the Go CLI in two areas:
1. It adds batch model management support, allowing multiple models to
be added or removed in a single command.
2. It makes the `dimension` argument optional for the `embed text`
command.
These changes keep the existing single-model and explicit-dimension
behaviors compatible while making the CLI more convenient for common
workflows.
## What Changed
### 1. Batch model add/remove support
The CLI now supports operating on multiple model names provided in a
single quoted string.
Supported commands include:
```
add model 'x1 x2 x3' to provider 'vllm' instance 'test' with tokens 1024 chat think vision, token 2048 chat, token 1024 think vision;
drop model 'x1 x2 x3' from 'vllm' 'test';
remove model 'x1 x2 x3' from 'vllm' 'test';
```
For add model, each config segment after with is matched to the
corresponding model name by position.
Example mapping:
- x1 -> tokens 1024, chat + vision, thinking=true
- x2 -> tokens 2048, chat
- x3 -> tokens 1024, vision, thinking=true
The existing single-model syntax remains supported.
### 2. Optional embedding dimension
Previously, the Go CLI required dimension to be explicitly provided for
embed text.
Before:
embed text 'what is rag' 'who are you' with 'model@test@provider'
dimension 8192;
Now both forms are supported:
embed text 'what is rag' 'who are you' with 'model@test@provider'
dimension 8192;
embed text 'what is rag' 'who are you' with 'model@test@provider';
When omitted, the CLI leaves dimension unset and relies on
provider/backend behavior.
## Tests
Added parser tests covering:
- Multiple models with multiple config segments
- Model type deduplication
- Model/config count mismatch
- Drop/remove multiple models
- Optional embedding dimension parsing
### What problem does this PR solve?
Fixes#15542.
AWS Bedrock support for the Go model provider layer was added in #15166,
but embedding support was intentionally left out of scope and
`BedrockModel.Embed(...)` still returned the `no such method` sentinel.
This PR implements Bedrock text embeddings under the umbrella provider
tracker #14736.
### What this PR includes
- `internal/entity/models/bedrock.go`: implement
`BedrockModel.Embed(...)` through Bedrock Runtime `InvokeModel` with
existing SigV4 auth, region resolution, and runtime URL helpers.
- Titan embeddings: supports `amazon.titan-embed-text-v1` and
`amazon.titan-embed-text-v2:0`; v2 forwards `EmbeddingConfig.Dimension`
as `dimensions` when provided, while v1 keeps the payload minimal.
- Cohere embeddings: supports `cohere.embed-english-v3`,
`cohere.embed-multilingual-v3`, and `cohere.embed-v4:0`; batches input
texts and maps returned vectors to RAGFlow `EmbeddingData` in input
order.
- `conf/models/bedrock.json`: adds the `embedding` URL suffix (`invoke`)
and Bedrock embedding model entries.
- `internal/entity/models/bedrock_test.go`: adds unit tests for Titan,
Cohere, typed Cohere responses, validation, empty input, unsupported
models, and HTTP error propagation.
Reference docs:
- Bedrock InvokeModel API:
https://docs.aws.amazon.com/bedrock/latest/APIReference/API_runtime_InvokeModel.html
- Titan Text Embeddings:
https://docs.aws.amazon.com/bedrock/latest/userguide/titan-embedding-models.html
- Cohere Embed models on Bedrock:
https://docs.aws.amazon.com/bedrock/latest/userguide/model-parameters-embed.html
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
### How was this tested?
- [x] `jq empty conf/models/bedrock.json`
- [x] `git diff --check`
- [x] `go test ./internal/entity/models/... -run Bedrock -count=1`
- [x] `go test ./internal/entity/models/... -run '^$' -count=1`
- [x] `go test ./internal/entity/models/... -run Bedrock -race -count=1`
Note: `go test ./internal/entity/models/... -count=1` currently fails in
unrelated existing Astraflow coverage
(`TestAstraflowEmbedReturnsNoSuchMethod` panics in
`internal/entity/models/astraflow.go`). The Bedrock-specific tests and
compile-only package check pass.
## Summary
Ports the MCP (Model Context Protocol) server management endpoints that
power `web/src/pages/user-setting/mcp/` from Python
(`api/apps/restful_apis/mcp_api.py`) to Go. There were no MCP routes in
the Go server before this change.
Closes#15275 (subtask of #15240).
## Endpoints implemented (base path `/api/v1`)
| Method | Path | Description |
|--------|------|-------------|
| GET | `/mcp/servers` | List tenant servers (keyword / order /
pagination) |
| POST | `/mcp/servers` | Create a server |
| GET | `/mcp/servers/{mcp_id}` | Get one (`?mode=download` exports
config) |
| PUT | `/mcp/servers/{mcp_id}` | Update a server |
| DELETE | `/mcp/servers/{mcp_id}` | Delete a server |
| POST | `/mcp/import` | Bulk import from JSON config |
| POST | `/mcp/servers/{mcp_id}/test` | Connect + list tools (see notes)
|
## Implementation
Follows the existing `handler → service → dao` layering (per PR #14790):
- **entity** (`internal/entity/mcp.go`): added `MCPServerType` constants
and `IsValidMCPServerType` over the existing `MCPServer` model.
- **dao** (`internal/dao/mcp.go`): new `MCPServerDAO` with tenant-scoped
CRUD, a keyword filter, and a **whitelisted order-column map** (guards
against SQL injection via the caller-supplied `orderby`).
- **service** (`internal/service/mcp.go`): new `MCPService` —
list/get/export/create/update/delete/import/test — mirroring
`MCPServerService` and the `mcp_api` request validation, with sentinel
errors for clean code mapping.
- **handler** (`internal/handler/mcp.go`): new `MCPHandler` with the
seven handlers and Python-compatible response codes.
- **router / server_main**: registered the `/mcp` group and wired the
handler.
## Deviations from Python (documented in code)
1. **Bulk import is at `POST /mcp/import`, not `/mcp/servers/import`.**
gin (v1.9.1) cannot register a static segment and a path param at the
same tree node, so `/mcp/servers/import` would collide with
`/mcp/servers/:mcp_id` and panic at startup. The frontend should call
`/mcp/import`.
2. **No live tool discovery on create/update/import.** The Python path
runs `get_mcp_tools` over SSE / streamable-HTTP and stores
`variables.tools`. The Go server has no MCP client yet, so these persist
`variables`/`headers` but leave `variables.tools` unpopulated.
3. **`/test` returns a data error (`ErrMCPTestUnsupported`)** until a Go
MCP client lands. Per the issue, the live-connection path is scoped as a
follow-up; the handler still validates `url` + `server_type`.
## Testing
- Added `internal/service/mcp_test.go` covering `IsValidMCPServerType`
and the `TestServer` validation/short-circuit paths (no DB required).
- No Go toolchain was available in the dev environment, so `go build
./...` / `go vet ./...` verification is left to CI.
## Follow-ups
- Go MCP client (SSE / streamable-HTTP) to enable live tool discovery
and the real `/test` behavior.
- Reconcile the `/mcp/import` vs `/mcp/servers/import` path with the
frontend.
---------
