## Summary
- Align Go agent logs access with the Python behavior by checking tenant
access to the target agent instead of requiring the token to be bound
to that agent.
- Marshal canvas lifecycle event payloads safely so non-serializable
inputs do not break Thinking expansion in the UI.
## Testing
- `CGO_ENABLED=0 go test -count=1 -v -run
"Handler|Attachment|Logs|Chatbot|Agentbot|BuildWorkflow|NodeLifecycle|VarRef|Resolve|Extract"./internal/handler/
./internal/agent/canvas/...` PASS
### Summary
`TestSearchHandlerUpdateRejectsInvalidSearchID` fails on `main`:
```
--- FAIL: TestSearchHandlerUpdateRejectsInvalidSearchID (0.00s)
search_handler_test.go:98: expected 'no authorization' in message, got No authorization.
```
`UpdateSearch` answers an unauthorized `search_id` with `"No
authorization."` (`internal/handler/search.go:337`), which mirrors the
Python API's message verbatim, but the assertion searched for the
lowercase `"no authorization"`.
The fix lowercases the response before the substring check rather than
changing the handler's message: the capitalized text is the contract the
Python API exposes, so the test — not the handler — was wrong.
`bash build.sh --test -run TestSearchHandler ./internal/handler/`
passes.
## Summary
Adds a side-effect-free DataFlow canvas **debug (dry-run) mode** plus a
**debug run log with a "View result" panel**, so a canvas can be
executed synchronously and inspected end-to-end (per-component progress
and parsed chunks) without persisting anything.
### Dry-run execution (inline parsed chunks)
- `task/debug.go`: `NewDebugTaskContext` builds an in-memory
`TaskContext` with **`KB.ID == ""` — the single debug signal used across
the ingestion pipeline**. A canvas debug run has no knowledgebase, and
production ingestion always supplies one, so `kb_id == ""` occurs ONLY
in debug mode. Components gate their own side effects on this signal
without any dedicated debug vocabulary (the former `CANVAS_DEBUG_DOC_ID`
marker constant is removed).
- `task/pipeline_executor.go`: `validateTaskContext` no longer requires
a KB when `KB.ID == ""` (debug); debug runs return `collectDebugOutput`
(chunks) instead of a no-op; uploaded bytes are delivered as
`inputs['binary']` for doc-less runs; `injectDebugPageCap` caps the
parser to the first pages for a fast preview via the production
`override_params` channel (Parser cpnID + family).
- `component/tokenizer.go`: `shouldHaveEmbedding` skips embedding when
`kb_id == ""` — the embedder is configured on the knowledgebase, so a
debug run has nothing to resolve against and stays side-effect free.
- `chunker/register.go`: chunk images are uploaded to MinIO only when a
KB is present (persist run). **In debug mode the raw image bytes are
intentionally dropped (`delete(ck, "image")`)** — the debug preview does
not render chunk images, and dropping the bytes keeps them out of memory
and out of the Redis-stored debug log. This is a deliberate trade-off,
not an oversight.
- `component/file.go`: pass through in-memory binary bytes, skipping
`doc_id` -> storage resolution.
- `handler/agent.go` + `agent_webhook.go`: detect `dataflow_canvas` and
run a sync debug returning chunks inline on the existing
chat/completions endpoint; reject DataFlow canvases from webhooks (fixes
the previously dead `== "DataFlow"` check; mirrors Python
`agent_api.py`).
- `parser_dispatch.go`: export `ParserFileFamily` for the executor's
page-cap injection.
### Debug run log + "View result"
Mirrors Python's debug-log contract so the front-end can replay each
component's progress and parsed output:
- `task/debug_log_sink.go`: a `DebugLogSink` records every component's
lifecycle into a `[{component_id, trace}]` array (each trace entry
carries `message`, `progress`, `timestamp`, `elapsed_time`). `Flush`
appends a terminal `END` marker whose first trace message is non-empty
so the front-end detects completion. On failure the END marker is
prefixed `[ERROR]` yet still carries the run, so the failure timeline
renders instead of being stuck empty. Timestamps and `elapsed_time` are
in seconds (matching the rest of the app).
- `task/debug_result_dsl.go`: `BuildDebugResultDSL` builds the `dsl` the
END marker carries — the Go analogue of Python's `Graph.__str__` +
END-marker `dsl` in `rag/flow/pipeline.py`. It combines the static DSL
structure (component_name / downstream / params / graph.nodes) with the
run output map (`output["state"][<id>]`) to emit, per component,
`obj.params.outputs[<format>].value` (chunks / text / json / html /
markdown) — the exact keys the front-end `dataflow-result` page reads to
render each step's parsed chunks. Raw embedding vectors (including the
dimension-scoped `q_<dim>_vec` keys) are stripped so the stored log
stays Python-scale.
- `task/pipeline_executor.go`: after the run, attach the built `dsl` to
the END marker via the `ResultSink` capability.
- `handler/agent.go`: `runCanvasPipelineDebug` generates a stable
`message_id` up-front and always flushes the log (success or failure);
`respondWithDebugResult` returns `message_id` in **both** the success
and the error envelope so the front-end can poll the log. The debug-log
endpoint `GET /agents/:id/logs/:message_id` serves the array.
- `web/src/pages/agent/hooks/use-run-dataflow.ts`: on a run failure,
also surface `message_id` via `setMessageId` so the log sheet renders
the failure timeline (the `[ERROR]` END marker is already written).
Guarded by `if (msgId)`, so it is a safe no-op when the back-end does
not return an id.
## Behavioral notes
- Debug parses only the first pages (`debugPageCapPages`) for a fast
preview; an explicit `pages` cap already present in the ParserConfig is
respected.
- Debug mode does not keep chunk images (see above) and does not compute
embeddings — it exercises parse + chunk only.
## Test plan
- Go: `debug_test.go`, `debug_log_sink_test.go` (trace pairing, END
marker, `[ERROR]` prefix, fractional-second timestamp/elapsed_time, size
caps, and a real-pipeline test asserting the END-marker `dsl` carries
non-empty per-component `params.outputs` with chunks),
`debug_result_dsl_test.go` (flat and real nested `output["state"]`
shapes, vector stripping, format priority),
`debug_pages_integration_test.go`, `pipeline_executor_persist_test.go`,
`handler/agent_pipeline_debug_test.go`, `handler/agent_logs_test.go`
(incl. `TestRunCanvasPipelineDebug_ErrorStillExposesMessageID` /
`TestRespondWithDebugResult_ErrorCarriesMessageID` locking `message_id`
on failure), plus updates to `agent_test.go` / `agent_webhook_test.go` /
`chunker/image_upload_test.go` / `tokenizer*.go`.
- `go build ./...` and `./build.sh --test` for affected packages.
🤖 Generated with [CodeBuddy Code](https://cnb.cool/codebuddy)
---------
Co-authored-by: CodeBuddy Code <noreply@tencent.com>
## Summary
- The `eng` flag passed to `removeTOCWord`/`removeContentsTable` was
hardcoded `false`, so English documents used the CJK 3-char TOC prefix
instead of Python's 2-word English prefix. This caused English tables of
contents to be under-deleted (left in indexed text) or over-deleted
(body paragraphs sharing the 3-char prefix dropped).
- `eng` is now derived from the parsed item content via `isEnglishItems`
(a port of Python `is_english`/`_is_english`), mirroring Python's
content heuristic.
## Changes
- `internal/parser/parser/text_toc.go`: add
`isEnglishItems`/`isEnglishTexts` (ASCII-ratio >80% ⇒ English,
fullmatch-anchored regex).
- `internal/parser/parser/docx_parser.go`: pass
`isEnglishItems(sections)` / `isEnglishItems(lineItems)` to
`removeTOCWord` (json + markdown paths).
- `internal/parser/parser/html_parser.go`: pass `isEnglishItems(items)`
to `removeContentsTable`.
- `docs/migration_python_go_diff.md`: close the 1.9/1.10 `eng` residual
and resolve the contradictory Parser 2.11 "Partially fixed" note.
## Notes
- `remove_toc` is a Parser-stage, DSL-configured feature (`remove_toc:
true/false` in the parser family setup). This change only fixes the
internal English/CJK prefix decision; it does not alter the pipeline or
the DSL contract. `remove_header_footer` (precise match) is unaffected.
- `eng` is auto-derived from content rather than exposed as a new config
knob, to stay faithful to Python behavior.
## Test plan
- `CGO_ENABLED=0 go test ./internal/parser/parser/ -run
'TestRemoveContentsTable|TestIsEnglishTexts|TestRemoveTOCWordEnglishDetection'`
- Added `TestIsEnglishTexts` (ASCII-ratio cases) and
`TestRemoveTOCWordEnglishDetection` (English TOC no longer over-deletes
"Chapter 2 Method").
---------
Co-authored-by: CodeBuddy <noreply@codebuddy.ai>
## Summary
- Fixes a compile error in `internal/channels/qqbot.go`: the local
`stringValue(value any)` helper collided with the existing
`stringValue(value *string)` in `feishu.go` within the same package
(`stringValue redeclared in this block`).
- Rename the qqbot variant to `anyToString` (definition + 13 call
sites). The two helpers do different jobs (any->string via `fmt.Sprint`
vs. `*string` deref), so they are not merged.
## Summary
Classify the Go test suite by dependency level using build tags so the
default
`go test ./...` run stays self-contained, and add local convenience
commands
plus a documented convention.
- Add build tags to 5 real-service tests that were previously un-tagged
and only
soft-isolated via `t.Skip`: `kg_test.go` (integration), `minio_test.go`
(integration), `template_integration_test.go` (integration),
`stagehand_runtime_integration_test.go` (integration),
`pipeline_e2e_test.go` (e2e).
The default unit run no longer compiles/attempts these.
- Reclassify the full-pipeline `real_consumer` tests from `integration`
to `e2e`.
- Add `build.sh` shortcuts: `--test-integration`, `--test-e2e`,
`--test-manual`,
`--test-all` (integration + e2e; `manual` is excluded and is local
opt-in only,
never run in CI).
- Document the tier scheme (unit / integration / e2e / manual +
orthogonal cgo) in
`AGENTS.md`.
## Tier definitions
| Tier | Build tag | Runs by default? |
|---|---|---|
| Unit | (none) | Yes — in-memory SQLite / miniredis / httptest stubs |
| Integration | `integration` | No (`-tags integration`) — single real
service |
| E2E | `e2e` | No (`-tags e2e`) — full ingest→index→retrieve pipeline |
| Manual | `manual` | No (`-tags manual`) — very slow; never in CI |
## Verification
- `gofmt -l` clean on all changed files; `bash -n build.sh` OK.
- `go list` confirms the default set excludes the tagged files, and
`-tags integration` / `-tags e2e` include them.
- Full regression: unit / integration / e2e each **97 ok, 0 FAIL**.
- Fixed a regression where `pipeline_knowledge_compiler_test.go` relied
on a
transitive import side-effect from `template_integration_test.go` to
register
the `File`/`Parser`/`TokenChunker` components; it now blank-imports the
component packages directly.
## Test plan
- [ ] `./build.sh --test` (unit) passes
- [ ] `./build.sh --test-integration` passes (needs real services; skips
otherwise)
- [ ] `./build.sh --test-e2e` passes (needs real services; skips
otherwise)
---------
Co-authored-by: CodeBuddy <noreply@cnb.cool>
### Summary
Add token usage reporting for the StepFun model provider. Related
to #17284.
Co-authored-by: Haruko386 <tryeverypossible@163.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
## Summary
- LongCat's token usage did not appear in the server terminal, while
DeepSeek's did — even though both drivers parse and record usage through
the same shared helpers. The gap was a missing debug log in LongCat's
SSE loop, so aggregate usage events were silently processed instead of
printed.
- The official LongCat API docs and live responses from
`api.longcat.chat` return a nested
`usage.completion_tokens_details.reasoning_tokens` breakdown for
thinking mode. The previous flat `LongCatChatResponse.Usage` struct
dropped this field on unmarshal.
- Add test coverage for the nested `reasoning_tokens` field on both the
non-streaming and streaming paths.
Closes#17284
### Summary
Add token usage reporting for the Groq model provider. Related to
#17284.
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Two regressions from #17536: (1) MySQL 8.0 AutoMigrate crash on
knowledge_compile_docs (Error 1101) from a TEXT default; (2) ingestor
Start() sync.Once re-entrant deadlock. Both fixed with regression tests.
### Summary
Aligns Go and Python error codes/messages so both backends honor the
same RESTful API contract, removing implementation-specific error leaks
(MySQL errors, `ValueError`, `AttributeError`, Gin validator format) in
favor of clean business error codes.
**Chat list** — invalid `orderby` now returns code 101 (was: raw Python
`AttributeError` code 100); invalid `page`/`page_size` values fall back
to defaults (was: raw `ValueError`/`ProgrammingError` code 100).
**Dataset create/update/delete** — adds UUID validation (101),
extra-field rejection (101), duplicate-id detection (101), content-type
/ JSON-syntax / object-shape checks (101), and "lacks permission" for
nonexistent datasets (IDOR). Create auto-deduplicates dataset names.
Pagerank updates tolerate a missing ES index. List response includes
`parser_config` and `pagerank`.
**Session list/update** — adds filtering, sorting, and pagination
support. Empty payloads are valid no-ops. Authorization errors map to
code 109.
**Chunk list** — doc object uses Python key names (`chunk_count`,
`dataset_id`, `chunk_method`, run text status). Add validates list
element types.
**Document update** — adds `chunk_method` alias, pydantic-style Field
error messages, metadata index auto-create with refresh, and "These
documents do not belong to dataset" messages. List validates
`metadata_condition` and reports ownership errors for unmatched name/id
filters.
**Search completion** — `kb_ids` ownership failure returns code 102
instead of 109.
Released 31 contract tests from `GO_ONLY_SKIPS` (all verified passing on
both Go and Python backends with real LLM keys).
## Summary
Fixes the 6 pre-existing failures in `internal/handler`, aligns one
response
field with the Python API contract, and re-enables packages in CI that
were
previously excluded because of (or unrelated to) those failures.
### Test/handler fixes
- **plugin.go**: return `"success"` (lowercase) instead of `"SUCCESS"`
so the
response envelope matches the Python backend, keeping backend-swap
transparent.
- **search_handler_test.go**: expect lowercase `"no authorization"` to
match the
service error message, which mirrors Python.
- **agent_test.go**: seed versions with `CreateTime` instead of
`UpdateTime` so
`ListVersions` ordering (`create_time DESC`) is exercised correctly.
- **agent_wait_for_user_test.go**: a clean run may emit only the
`[DONE]` frame;
relax the SSE assertion to require a non-empty stream ending in
`[DONE]`.
- **bot_test.go**: align attachment-download assertions with actual
behavior
(`Content-Disposition: attachment; filename="file"`, default
`application/octet-stream`), matching Python
`resolve_attachment_content_type`.
### CI
- **tests.yml / sep-tests.yml**:
- Remove the `grep -v '/internal/handler$'` filter — the 6 failures that
motivated it are now fixed.
- Remove the `grep -v '/internal/storage$'` filter — `internal/storage`
tests self-skip via `t.Skipf` when MinIO is unavailable, so the package
is
safe to run in CI.
- Remove the `grep -v '/internal/agent$'` filter (only present in the
tests.yml infinity job) — the exclusion was undocumented and
inconsistent
with the other jobs that already run `internal/agent`.
- Keep the `internal/tokenizer` exclusion: it is a genuine environmental
dependency (dict files at `/usr/share/infinity/resource`, absent in the
Go
test environment).
## Test plan
- `build.sh --test ./internal/handler/` is green (previously 6 FAIL).
- `build.sh --test ./internal/storage/ ./internal/agent/` should pass;
the
MinIO-backed `internal/storage` tests skip gracefully without a MinIO
server.
- After this PR, `internal/handler`, `internal/storage`, and
`internal/agent`
run again in CI unit-test jobs; `internal/tokenizer` stays excluded.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
## Summary
Fixes embedding inputs being emptied and documents chunker
heading-detection parity.
- `component/tokenizer.go`: `truncateForEmbedding` now mirrors Python
`common/token_utils.py:183-185` `truncate(string, max_len)` (keep the
first `max_len` tokens). For an unconfigured embedder reporting
`maxTokens <= 0`, instead of mirroring Python's `truncate` (which
returns `""` for `max_len <= 0` and would make the embeddings API reject
the batch with "inputs cannot be empty"), Go **clamps the limit to a
safe default** (`defaultEmbeddingTokenLimit = 8192`) so truncation stays
active and never produces empty inputs. The previous code returned `""`
for `maxTokens <= 10`, which emptied non-empty chunks into the batch and
triggered the API error.
- A **10-token safety margin** is reserved (mirroring Python's embedding
path `rag/svr/task_executor.py` which uses `mdl.max_length - 10`) so Go
does not over-send tokens to hard-capped embedding APIs; it is only
applied when the limit `> 10` so small limits remain non-empty.
- The clamp is applied **centrally inside `truncateForEmbedding`**,
covering both Builtin and generic callers — no separate
`model_service.go` change is required.
- `component/chunker/title.go`: comment-only — documents the
already-ported PDF-outline detection (omission 1.5 / Gap C) and the
hardcoded `BULLET_PATTERN` fallback (omission 1.7 / Gap C), porting
`common.py:_outline_similarity` and `rag/nlp` `BULLET_PATTERN`.
- `tokenizer_unit_test.go`: update truncation expectations to the new
positive-keeps-tokens / non-positive-clamps-to-default behaviour.
## Test plan
- `tokenizer_unit_test.go` updated for the new truncation behaviour
(`TestTruncateForEmbedding_SmallMaxTokens`,
`TestTruncateForEmbedding_UnconfiguredClampsToDefault`).
- `./build.sh --test ./internal/ingestion/component/` passes.
🤖 Generated with [CodeBuddy Code](https://cnb.cool/codebuddy)
Ports doc-level auto-metadata extraction to Go and adds the
knowledge_compiler component with scheduler/routing. Fixes Extractor
metadata injection type assertion and enable_metadata default-on.
## Summary
- Add provider-local Chat, Embedding, and Rerank response structures
with usage mapping.
- Align streaming usage and tool-call state handling across Gitee,
OpenRouter, Jiekou.AI, and Hunyuan.
- Preserve Jina's non-streaming behavior and explicit unsupported
streaming response, while fixing Jiekou.AI thinking=false handling.
### Summary
The Go Anthropic driver's `ChatStreamlyWithSender`
(internal/entity/models/anthropic.go) was a stub that always returned
`"no such method"`, so any caller requesting a streamed response from a
Claude model via the Go path failed outright — diverging from the Python
`AnthropicCV` driver, which already supports streaming.
This implements the method by opening the Messages API with
`stream=true` and parsing the SSE response via the shared
`ParseSSEStream` helper, forwarding `text_delta`/`thinking_delta`
content through the `sender` callback and treating `message_stop` as the
terminal event — consistent with the other Go drivers in this package
(e.g. Cohere).
Fixes#17333
---------
Co-authored-by: Abhay Yadav <abhayyadav@Abhays-MacBook-Air.local>
### Summary
Add token usage reporting for the LongCat (Meituan) model provider.
Related
to #17284.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
### Summary
Adds a `tenki` sandbox provider that runs each agent code execution in a
disposable Tenki (https://tenki.cloud) microVM (create → exec → destroy,
no volumes or snapshots).
Registration mirrors PR #15039, configure `api_key` and `project_id` in
Admin > Sandbox Settings.
Both runtimes are covered:
- Python: `agent/sandbox/providers/tenki.py` (structured results +
artifact collection).
- Go: `internal/agent/sandbox/tenki.go`, mirroring the e2b provider and
wired into the provider manager.
`tenki-sandbox` is an optional dependency (it requires `protobuf>=6.31`,
which differs from RAGFlow's pinned gRPC stack), lazily imported with a
clear error when missing; installation is documented in the sandbox
quickstart.
Unit tests cover execution, structured results, artifacts
(symlink/size/extension limits), non-zero exit, timeout, error mapping,
and idempotent destroy.
---------
Co-authored-by: yiming.wang <yiming.wang@luxor.com>
## Summary
This PR hardens the Go ingestion **Extractor** and **LLM retry** paths
and closes several Python->Go parity gaps in the keyword/question/tag
extraction flow.
- **Generic retry utility** (`internal/common/retry.go`):
`RetryWithBackoff` with exponential backoff (default 3 retries, 2s
initial delay, capped at 1m), context-aware sleep, and a `maxRetries<=0`
fast path. Covered by `internal/common/retry_test.go`.
- **LLM retry reuse**: `agent/component/llm_retry.go` now delegates to
`common.RetryWithBackoff` instead of an inline loop (behavior preserved:
ctx cancellation short-circuits the backoff).
- **Extractor LLM calls** (`extractor.go`):
- `call()` now retries transient LLM failures via `RetryWithBackoff`
(retry exhaustion fails the chunk instead of silently skipping).
- Sets `temperature = 0.2`, matching Python `generator.py:230,245`.
- Runs keyword and question extraction **concurrently** per chunk when
both are enabled (`task_executor.py:444-448`), with mutex-guarded map
writes to avoid data races.
- Substitutes `{field_name}` placeholders (including `{chunks}` -> chunk
text) in `prompt`/`system_prompt` before the call, mirroring Python
`string_format` (`extractor.py:102-103`); unmatched placeholders are
left as-is.
- Falls back to the **tenant default chat model** when `llm_id` is empty
(`task_executor.py:573-574`).
- Strips `` **greedily** (`strings.LastIndex`) in
`cleanExtractionResult`.
- **Auto-tagging** (`extractor_tag.go`): drops the `in.llmID != ""`
guards so an empty `llm_id` no longer skips tagging (uses the tenant
default model), and strips `` greedily in `parseTaggerResponse`.
- **Docs**: fixes a misleading `PresentationChunker` docstring that
claimed per-slide `image`/`position` output (the PPTX path emits none —
unlike PDF), and removes a stale `docs/migration_python_go_diff.md`
reference in `media_dispatch.go`.
## Test plan
- `bash build.sh --test ./internal/common/...` — passes (new retry
utility + tests).
- `bash build.sh --test ./internal/ingestion/component/...` — passes
(extractor/chunker/schema).
- `gofmt` and lefthook pre-commit checks pass.
Note: the personal `docs/migration_python_go_diff.md` working notebook
in the tree is intentionally **not** part of this PR.
---------
Co-authored-by: CodeBuddy <noreply@codebuddy.ai>
## Summary
GreenPT is a European AI provider with an OpenAI-compatible API,
optimized infrastructure, and datacenters powered by 100% renewable
energy.
This adds native GreenPT support across RAGFlow’s Go-first provider
system and its Python compatibility layer:
- discovers the current catalog from `GET /v1/models`
- features `glm-5.2` and `kimi-k2.7-code` for chat and coding
- supports `green-embedding` through `/v1/embeddings`
- supports `green-rerank` through `/v1/rerank`
- supports `green-s` and `green-s-pro` speech-to-text through
`/v1/listen`
- adds provider configuration, UI icon, and supported-provider
documentation
## Summary
- Return the configured `empty_response` when retrieval has no query or
no chunks.
- Preserve `formalized_content` for downstream Message nodes.
## Testing
- `bash build.sh --go`
- `ok ragflow/internal/agent/tool`
- `ok ragflow/internal/agent/component`
## Problem
`TestMergeByTokenSizeFromJSON_ClampsOverlappedPct` reused a single
`items` fixture across four calls to `mergeByTokenSizeFromJSON`.
`mergeByTokenSizeFromJSON` mutates its `perItem` argument in place and
returns the same backing array (token.go: `perItem[idx] = merged`). As a
result:
- The 2nd and later calls merged already-merged chunks instead of the
original input.
- Because the returned slice aliases the input, later calls silently
overwrote the earlier results (`at100`, `at0`, ...).
- `reflect.DeepEqual(at100, at150)` was therefore vacuously true — the
test was a **false positive** that never actually exercised the clamp.
It would still pass even if the clamp were broken.
This is exactly the defect flagged in the review comment on PR #17396.
## Fix
Add a `clampOverlapFixture()` factory and build a fresh fixture for
every call, so that 150 / 1e300 / -5 / -1e300 are applied to the
original input.
## Verification
- The test passes after the fix.
- When the clamp logic was temporarily disabled, the test **failed**
(panic at token.go:768 — the negative index produced by an out-of-range
pct), proving the fixed test is no longer a false positive and can catch
a clamp regression.
## Scope
Test file only. No production code change (verified `token.go` is
identical to `upstream/main`).
Refs: review comment on PR #17396.
