Ports remaining Go parser wiring and PDF backends, adds tenant-aware VLM
dispatch, aligns post-processing with Python, and adds end-to-end
pipeline coverage with a generated six-page PDF.
### Summary
Keep `data` as the uploaded document array when dataset document upload
partially succeeds.
This matches the Python API behavior and allows parse-on-creation to run
for successfully uploaded files when other files in the same folder are
unsupported.
## Summary
- Add Go dynamic input form support for ExeSQL and Browser components.
- Align their input form metadata with the Python implementation.
- Add regression tests for `/components/:component_id/input-form`.
## Summary
Debugging YahooFinance component in agent canvas returns "unknown
component" and "no input_form".
YahooFinance was only registered as an eino tool, not as a runtime
component. The component factory only searches the runtime registry.
- `universe_a_wrappers.go`: add `yahooFinanceComponent` wrapper
delegating to `agenttool.YahooFinanceTool` with `GetInputForm()`
- `fixture_stubs.go`: register `"YahooFinance"` component
## TEST
`go build` and `go test ./internal/agent/component/...` all pass.
## What this fixes
Closes#16400.
`get_data_openai()` currently returns `created: null` when callers do
not pass a timestamp, and it replaces explicit timestamp values with the
current time. This makes non-streaming OpenAI-compatible responses
inconsistent with the expected integer `created` timestamp field.
## Change
- Preserve explicit `created` values when provided.
- Default non-streaming responses to `int(time.time())` when `created`
is not provided.
- Add focused unit coverage for default timestamps, explicit timestamps,
and unchanged streaming chunk shape.
## Verification
- `./.venv/bin/python -m pytest
test/unit_test/api/utils/test_api_utils.py -q`
- `python3 -m py_compile api/utils/api_utils.py
test/unit_test/api/utils/test_api_utils.py`
- `uvx ruff check api/utils/api_utils.py
test/unit_test/api/utils/test_api_utils.py`
---------
Co-authored-by: Harsh Kashyap <harshkashyap@Harshs-MacBook-Pro.local>
### What problem does this PR solve?
The Go chunk pipeline's `PostprocessOperator` `filter` stage
(`internal/ingestion/chunk/postprocess.go`) only filtered by length
(`min_length`/`max_length`). It could not drop empty/whitespace-only
chunks or duplicate chunks — both standard RAG post-processing steps
(blank chunks shouldn't be indexed; identical chunks waste embedding
compute and add redundant retrieval results).
This adds two optional, default-off booleans to the `filter` config:
- `drop_empty` — drop chunks whose content is empty or whitespace-only.
- `drop_duplicates` — drop chunks whose exact content already appeared
(order-preserving; the first occurrence is kept).
They compose with the existing length bounds and are reflected in
`String()` for plan explainability. Also adds the first unit tests for
the postprocess filter (length bounds, drop_empty, drop_duplicates,
combined, exact-content matching, and config parsing).
Validation: `gofmt` clean, `go vet ./internal/ingestion/chunk/` clean,
`go build` ok, `go test ./internal/ingestion/chunk/` — all tests pass.
Closes#16048
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
Co-authored-by: Ling Qin <qinling0210@163.com>
### What problem does this PR solve?
The Go ingestion chunk pipeline's `SplitOperator`
(`internal/ingestion/chunk/split.go`) supported only `sentence`, `char`,
and `paragraph` strategies, but not **fixed-size (length) chunking with
overlap** — the canonical RAG strategy for bounding chunk length while
preserving cross-boundary context.
This adds a `length` strategy alongside the existing ones, configurable
via DSL `params`:
- `chunk_size` — target window size in **runes** (rune-aware:
multi-byte/CJK text is windowed by character, never split mid-rune).
- `overlap` — runes carried from the end of each window into the next.
The window advances by `chunk_size - overlap`. `chunk_size` falls back
to a default (256) when unset/non-positive, and `overlap` is clamped to
`[0, chunk_size-1]` so the window always advances and the operation
terminates. Implementation follows the existing
`splitByChar`/`splitByParagraph` pattern and reuses `DetectLanguage` for
chunk metadata.
It also adds `split_test.go` — the first unit tests for the `chunk`
package — covering basic windowing, overlap, overlap
clamping/termination, rune-awareness (CJK), default sizing, no-overlap
reconstruction, empty input, and DSL param parsing.
Validation: `gofmt` clean, `go vet ./internal/ingestion/chunk/` clean,
`go build` ok, `go test ./internal/ingestion/chunk/` — all tests pass.
Closes#16046
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
Co-authored-by: bittoby <218712309+bittoby@users.noreply.github.com>
## Summary
- derive Go Agent debug input forms from prompt variable references
instead of Agent meta fields
- seed `sys.*` debug params into `CanvasState.Sys` so single-component
debug resolves prompt variables like Python
- restore Agent test-run parity for form rendering and debug execution
## Tests
- `go test ./internal/agent/component -run
'TestAgent_(GetInputForm_UsesPromptReferences|GetInputForm_DeduplicatesPromptReferences|Meta_DefaultsToEmpty|Reset_NoTools)$'`
- `go test ./internal/handler -run
'Test(DebugComponent_SeedsSysInputsIntoCanvasState|DebugComponent_HappyPath_Begin|GetComponentInputForm_HappyPath)$'`
AFTER:
<img width="669" height="456" alt="image"
src="https://github.com/user-attachments/assets/4fd86559-aafc-4027-91ae-6e666137ee1b"
/>
## Related issues
Closes#15375
### What problem does this PR solve?
`GET /api/v1/messages` and `GET /api/v1/messages/search` accepted
unbounded `limit` / `top_n` query parameters while other REST list
endpoints enforce `REST_API_MAX_PAGE_SIZE` (100) via
`validate_rest_api_page_size()`. Oversized values can trigger expensive
memory index queries and large result sets (DoS risk).
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
### Changes
| File | Change |
|------|--------|
| `api/apps/restful_apis/memory_api.py` | Cap `limit` and `top_n` with
`validate_rest_api_page_size`; return argument error when exceeded |
|
`test/testcases/test_web_api/test_message_app/test_message_routes_unit.py`
| Regression tests for oversized `limit` / `top_n` |
### Test plan
- [x] Unit tests added
- [ ] `pytest
test/testcases/test_web_api/test_message_app/test_message_routes_unit.py`
Co-authored-by: Cursor <cursoragent@cursor.com>
Follow-up from #13896Fixes#13840
### What problem does this PR solve?
In #13896, only the docker-compose-base.yml was adjusted. However, in
the Helm chart, the unmaintained minio/minio image is still referenced.
This PR syncs the Helm chart with the docker compose setup again.
I also added a line to AGENTS.md, so agents should know to do this
automatically in the future.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
- [ ] New Feature (non-breaking change which adds functionality)
- [x] Documentation Update
- [ ] Refactoring
- [ ] Performance Improvement
- [ ] Other (please describe):
Co-authored-by: Zhichang Yu <yuzhichang@gmail.com>
## Summary
Add **per-node thinking mode control** for LLM components in RAGFlow
Agent canvas, supporting Qwen3/Qwen3.6/B200M thinking-capable models.
Users can now independently configure thinking mode
(thinking/non-thinking) for each LLM node via the existing UI dropdown.
## Motivation
When Qwen3.6-27B (and other thinking-capable models like Qwen3-32B,
B200M) are used in RAGFlow Agent nodes, different nodes need different
thinking behavior:
- **Reasoning nodes** (complex analysis, math, coding): thinking mode ON
- **Simple nodes** (direct Q&A, intent classification): thinking mode
OFF
The web UI already has a `thinking` dropdown (default/enabled/disabled)
in LLM settings, and the LLM backend `_apply_model_family_policies()`
already supports `enable_thinking`. **The missing link was `gen_conf()`
not forwarding the parameter.**
This 3-line fix completes the chain.
## Changes
**`agent/component/llm.py`** — `LLMParam.gen_conf()`:
```python
if hasattr(self, "thinking") and self.thinking and self.thinking != "default":
conf["thinking"] = self.thinking
```
## End-to-end flow
```
UI dropdown: default / enabled / disabled
→ DSL: {"thinking": "enabled"}
→ LLMParam.thinking = "enabled"
→ gen_conf() returns {"thinking": "enabled"}
→ _apply_model_family_policies()
→ extra_body {enable_thinking: true}
→ Model API call with thinking ON
```
## Backward compatibility
- **Fully backward compatible** — only 3 lines added, nothing changed
- When `thinking` is "default" or not set, existing behavior is
preserved
- Qwen3 models default to `enable_thinking: false` (non-thinking),
unchanged
## Related issues
- Closes#16321 (thinking content leaks in non-streaming agent API
responses)
- Closes#13957 (how to view model reasoning process in agent API)
## Testing
- Verified in
`test/unit_test/rag/llm/test_chat_model_thinking_policy.py` that
thinking policy already tested for Qwen3 models
- The 3-line change passes through existing tested code path
(`_apply_model_family_policies`)
Co-authored-by: Hermes Agent <hermes-agent@agent.local>
## Summary
Adds ownership/access checks before updating or deleting documents,
setting document metadata, and reading file contents from storage. Also
adds tests for authorized and unauthorized access paths.
## Summary
Fix Elasticsearch-backed skill search by mapping skill search fields to
their indexed token fields.
`name`, `tags`, `description`, and `content` are stored for display but
are not searchable in the skill ES mapping. Search queries now target
`name_tks`, `tags_tks`, `description_tks`, and `content_tks`.
## Testing
- Ran Go unit tests:
```bash
/usr/local/go/bin/go test -count=1 ./internal/engine/elasticsearch
```
- Frontend verification:
1. Open /files/skills.
2. Enter a skill space.
3. Reindex the skill space if existing skills were created before this
fix.
4. Search by skill name or description keyword.
5. Confirm matching skills are returned.
### Summary
This PR fixes two issues that prevented the Agent component's
single-component debug/test run from working under the Go backend:
1. **Dynamic input_form generation**: Some components (e.g. `Agent`) do
not store a static `input_form` in the DSL. The Go handler now falls
back to the runtime component's `GetInputForm()` method, matching
Python's `Canvas.get_component_input_form` behavior. This resolves the
frontend 102 error: `component has no input_form`.
2. **Tenant ID injection for debug**: Single-component debug runs use a
fresh `CanvasState` that previously lacked `tenant_id`.
`AgentComponent.Invoke` resolves LLM credentials via the tenant tables,
so the debug run failed with `api key is required`. The handler now
seeds `state.Sys["tenant_id"]` with the authenticated user's ID,
mirroring Python's `@add_tenant_id_to_kwargs` decorator.
### Changes
- `internal/handler/agent_component.go`:
- Added `componentInputForm` helper that first reads the static
`input_form` and, if missing, instantiates the component and calls
`GetInputForm()`.
- In `DebugComponent`, set `debugState.Sys["tenant_id"] = user.ID`
before invoking the component.
### Summary
The Go backend Agent component was not returning artifacts produced by
the CodeExec tool. While the Python agent collects the "`_ARTIFACTS`"
envelope from tool responses and appends artifact markdown to the final
content, the Go agent only returned the assistant text, so generated
images were missing from the chat output.
### Changes
- Wire `react.WithMessageFuture()` in `runEinoReActAgent` and store the
resulting `MessageFuture` in the invocation context.
- After the ReAct loop finishes, drain the future and extract
``_ARTIFACTS`` entries from every tool response message.
- Support reading the tool payload from both `msg.Content` and
`msg.UserInputMultiContent` to match eino's tool contract.
- De-duplicate artifacts by URL and render images as `!` and other files
as download links.
- Add `agent_artifact_test.go` with a regression test that simulates a
CodeExec-style tool response carrying an image artifact and verifies it
is collected and formatted.
### Verification
- `go test ./internal/agent/component/... -run
TestAgent_ReActAgent_CollectsArtifactsFromCodeExecTool` passes.
- `go test ./internal/agent/component/... -count=1` compiles; the only
failure is an unrelated DNS-pinning timeout test
(`TestInvoke_ProxyDNSPin`).
- `gofmt` clean for modified files.
### Related
Fixes the behavior shown in the screenshot where the Go agent ignored
the CodeExec-generated PNG artifact.
## Problem
`raptor.py` computes `n_neighbors = int((len(embeddings) - 1) ** 0.8)`
and
passes it to `umap.UMAP(...)`. In a dataset-scope RAPTOR build the first
layer's `embeddings` is the entire KB's chunk set, so this is
effectively
unbounded: ~93k chunks → n_neighbors ≈ 9,446.
UMAP's k-NN graph is `N × n_neighbors`; at these values the raw neighbor
arrays alone are ~14 GB (93k × 9446 × 16 B), and the symmetrized fuzzy
simplicial set + spectral init push peak well past 30 GB. The task
executor is OOM-killed inside `fit_transform` before any clustering runs
—
the log shows "Task has been received" with no "Cluster one layer" line
—
after which the unacked task re-queues and OOMs again in a loop.
The line above already flags this: `# Degrade too much ??`.
## Fix
Cap `n_neighbors` at 100. UMAP's neighborhood size has strongly
diminishing returns well below this (default 15; a few dozen already
captures global structure), so the ceiling preserves — likely improves —
cluster quality while bounding memory to O(N). Mirrors the existing
`n_components=min(12, len(embeddings) - 2)` clamp two lines down.
```diff
- n_neighbors = int((len(embeddings) - 1) ** 0.8)
+ n_neighbors = min(int((len(embeddings) - 1) ** 0.8), 100)
```
## Repro
Dataset-scope RAPTOR over a KB with ~90k+ chunks on a box with <~64 GB
available: executor OOM-killed in the first-layer UMAP `fit_transform`.
With the cap, first-layer UMAP peaks in low single-digit GB and the
build
proceeds to completion.
## Scope
Only affects large dataset-scope builds; file-scope RAPTOR already had
n_neighbors well under 100. No behavior change beyond the ceiling.
### Summary
1. Move common functions to format.go
2. modify show name spaces to _
3. move _order _columns column sort group;
4. add dao empty enterprise file
## Summary
While fixing #16467 (IterationItem crash on `@` in user-defined output
keys), an audit of `agent/**/*.py` revealed **three additional sites**
with the same vulnerability. This PR hardens all of them with
`maxsplit=1` and adds regression tests.
This is **defense-in-depth hardening**, not a behavior change. The
current `variable_ref_patt` regex constrains `var_nm` to
`[A-Za-z0-9_.-]+`, so single-`@` templates resolve exactly as before.
The `maxsplit=1` only kicks in if the trailing side itself contains `@`
— currently unreachable from the public DSL surface, but trivially
exploitable the moment a user-defined output key happens to contain `@`
(e.g. `user@email`) or the regex is ever relaxed.
> **Note on issue scope**: The primary fix for #16467 (the
`list_tenant_added_models` `ValueError` crash on `@` in model names) is
in PR #16468. This PR is a **follow-up hardening sweep** of the same
vulnerability class found in `agent/` during that audit; it does not
duplicate or replace #16468.
## Sites hardened
| File | Line | Method |
|------|------|--------|
| `agent/canvas.py` | 206 | `Graph.get_variable_value` |
| `agent/canvas.py` | 256 | `Graph.set_variable_value` |
| `agent/component/base.py` | 533 |
`ComponentBase.get_input_elements_from_text` |
| `agent/component/iterationitem.py` | 88 |
`IterationItem.output_collation` |
All now use `split("@", 1)` with an inline comment explaining the
rationale. The trailing side keeps any embedded `@`.
## Sites already safe (audited but left alone)
| File | Reason safe |
|------|------------|
| `agent/canvas.py:708` (`is_reff`) | Pre-checks `len(arr) != 2` |
| `agent/component/categorize.py` | Uses `rsplit` |
| `agent/component/iteration.py` | Pre-validates via regex |
| Other call sites | `rsplit` or regex pre-validation |
## Regression tests
9 new tests across 2 files, all `pytest.mark.p2`:
| File | Tests |
|------|-------|
| `test/unit_test/agent/test_canvas_at_split.py` | 6 —
`get_variable_value`, `set_variable_value`, round-trip, single-`@`,
missing-component |
| `test/unit_test/agent/component/test_iterationitem_at_split.py` | 3 —
`output_collation` with `@` in var, single-`@`, non-matching cid |
Each test was **verified to fail with `ValueError: too many values to
unpack (expected 2)`** when the corresponding fix is temporarily
reverted, confirming the tests actually catch the bug rather than just
exercising the happy path.
## Test results
```
9 passed in 0.04s
```
Full agent unit suite also clean (38 passed, 3 skipped; 6 unrelated
pre-existing collection errors from missing `peewee`/`requests` in local
venv — not caused by this PR).
## Related
- Issue: #16467
- Primary fix PR: #16468 (closes the issue)
- This PR: defense-in-depth follow-up, intentionally non-blocking on
#16467
---------
Co-authored-by: skbs-eng <skbs-eng@users.noreply.github.com>
### What problem does this PR solve?
Closes#16414.
The **Crawler** agent tool (`agent/tools/crawler.py`) was never ported
to the modern `ToolBase`/`_invoke` interface during the agent module
redesign, so it was broken in three independent ways:
1. **Crashed on construction.** `CrawlerParam` extends `ToolParamBase`,
whose `__init__` reads `self.meta["parameters"]`, but `CrawlerParam`
defined no `meta`. Constructing it raised `AttributeError:
'CrawlerParam' object has no attribute 'meta'`. Because
`agent/canvas.py` instantiates `component_class(component_name +
"Param")()` while loading a canvas, **any agent containing a Crawler
node failed to load.**
2. **`_invoke` missing.** It extends `ToolBase` (whose `invoke()`
dispatches to `self._invoke`) but only implemented the legacy `_run`, so
`_invoke` resolved to `ComponentBase._invoke` → `NotImplementedError`.
3. **`be_output` removed.** `_run` called `Crawler.be_output(...)`,
which no longer exists on the base classes.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
### Changes
- Add a `ToolMeta` to `CrawlerParam` (defined before
`super().__init__()`, matching every other ported tool such as
`ArXivParam`/`TavilyExtractParam`) advertising a required `query`
parameter — the URL to crawl, default `{sys.query}`, consistent with the
`{sys.query}` convention shared by the other tools.
- Replace the legacy `_run`/`be_output` with `_invoke`/`set_output`,
writing the extracted page content to `formalized_content` (errors
surfaced via `_ERROR`), consistent with the other tools.
- Preserve the existing SSRF guard (`assert_url_is_safe` +
`pin_dns_global`).
- Add regression tests
(`test/unit_test/agent/component/test_crawler.py`) covering param
construction, validation, and the tool descriptor.
Same class of defect as #16329 (DeepL). Backend-only; no frontend
changes.
---------
Co-authored-by: Zhichang Yu <yuzhichang@gmail.com>
