Ports two Python fixes to Go: the variable_ref_patt underscore/colon fix
(#16792) and the TokenChunker upstream-chunks fix (#16825). Keeps Go
behavior aligned with upstream Python.
Fixes#16812
### Problem
In the `rag/flow` ingestion pipeline, when `TitleChunker` feeds
`TokenChunker`, the chapter-aware chunks are silently discarded and the
parser's raw flat json is re-chunked instead.
`TitleChunker` emits `output_format="chunks"` and writes its
chapter-aware output to the `chunks` field
(`rag/flow/chunker/title_chunker/common.py`,
`set_output("output_format", "chunks")`). But `TokenChunker._invoke`
only handles `output_format` in `["markdown", "text", "html"]`, then
falls through to the `# json` path which reads
`from_upstream.json_result`. There is no branch for `"chunks"`, so
`from_upstream.chunks` is never read.
Downstream effects reported in #16812: PageIndex/TOC extraction receives
flat line-level text instead of structured chapter blocks
(incorrect/duplicate/missing chapters), and retrieval quality degrades
because chunks are no longer aligned to document structure.
### Fix
Select the source list based on `output_format`, mirroring the exact
pattern already used in `title_chunker/common.py`:
```python
json_result = (from_upstream.chunks if from_upstream.output_format == "chunks" else from_upstream.json_result) or []
```
`chunks` items share the same dict shape as `json_result` items (both
consumed via `.get("text")`, `.get("doc_type_kwd")`, etc.), so they flow
through the existing token-sizing path unchanged. One-line change, no
behavior change for the `json`/`markdown`/`text`/`html` paths.
### Test
Adds `rag/flow/tests/test_token_chunker.py`, an isolated unit test that
runs the real `TokenChunker._invoke` (heavy deps stubbed; real pydantic
schema used when available) and asserts that with
`output_format="chunks"` the upstream `chunks` are consumed rather than
the raw parser `json`.
Verified RED -> GREEN: the test fails against the current code (reads
the raw json) and passes with the fix.
Signed-off-by: Yash Raj Pandey <yashpn62@gmail.com>
## Summary
`ComponentBase.variable_ref_patt` (and its duplicate in
`agent.canvas.Graph.get_value_with_variable`) is the regex the canvas
runtime uses to find `cpn_id@var_nm` template refs in component prompts.
The `cpn_id` half was constrained to `[a-zA-Z:0-9]+`, which silently
dropped underscores. Component ids emitted by the frontend all contain
underscores (`userfillup_abc`, `retrieval_xyz`, `llm_0`, `message_0`,
…), so any template ref like `{userfillup_abc@line}` failed to match.
The placeholder then leaked through to the LLM verbatim, and the Agent
answered only its system-prompt directive.
This is exactly the "unconsidered await response" symptom in #16758:
```
Begin(Task) -> Await response -> Agent -> Message
```
Widen `cpn_id` from `[a-zA-Z:0-9]+` to `[a-zA-Z0-9_]+`. Bare `{line}`
(no cpn_id) remains unrecognised so it stays literal until the user
wires it up — matching the existing `VARIABLE_REF_PATTERN` shape used by
`agent.dsl_migration` for the same purpose.
## Changes
- `agent/component/base.py` — fix `variable_ref_patt` class attribute.
- `agent/canvas.py` — same fix applied to the inline regex inside
`Graph.get_value_with_variable` (kept as the literal regex to avoid
coupling the two unrelated sites).
-
`test/testcases/test_web_api/test_canvas_app/test_variable_ref_pattern_unit.py`
— new regression test pinning both the regex shape and end-to-end
resolution.
## Regression coverage
```
test_variable_ref_patt_matches_underscored_component_ids PASSED
test_variable_ref_patt_still_matches_legacy_ids PASSED
test_get_input_elements_from_text_resolves_underscored_id PASSED
test_string_format_substitutes_underscored_ref PASSED
test_variable_ref_patt_does_not_match_bare_var_name PASSED
```
All five regression tests fail against the pre-fix regex (verified via
`git stash` round trip — drop fix, tests fail, restore fix, tests pass).
The two targeted existing tests in the same directory
(`test_fillup_unit.py`, `test_iterationitem_unit.py`) continue to pass.
## Repro before the fix
```python
import re
patt = r"\{* *\{([a-zA-Z:0-9]+@[A-Za-z0-9_.-]+|sys\.[A-Za-z0-9_.]+|env\.[A-Za-z0-9_.]+)\} *\}*"
list(re.finditer(patt, "{userfillup_abc@line}"))
# => [] # <-- bug
```
## Repro after the fix
```python
import re
patt = r"\{* *\{([a-zA-Z0-9_]+@[A-Za-z0-9_.-]+|sys\.[A-Za-z0-9_.]+|env\.[A-Za-z0-9_.]+)\} *\}*"
list(re.finditer(patt, "{userfillup_abc@line}"))
# => [<re.Match object; span=(0, 24), match='{userfillup_abc@line}'>]
```
Fixes#16758
## Test plan
- [x] New unit tests pass
- [x] Reverse-apply the fix and confirm the regression tests fail (they
do)
- [x] `test_fillup_unit.py` (existing sibling suite) still passes
- [x] `test_iterationitem_unit.py` (existing sibling suite) still passes
- [ ] Project CI green
---------
Co-authored-by: Taranum01 <taranum01@users.noreply.github.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
### Summary
In Go and python implementation, the dataset / KB id isn't validated if
it is accessible by this user.
---------
Signed-off-by: Jin Hai <haijin.chn@gmail.com>
### Summary
```
RAGFlow(admin)> show version;
+--------------+-----------------------+
| field | value |
+--------------+-----------------------+
| version | v0.26.4-84-g547bc8614 |
| version_type | open source |
+--------------+-----------------------+
```
---------
Signed-off-by: Jin Hai <haijin.chn@gmail.com>
## Summary
- Align Go WenCai and SearXNG behavior, schemas, and node parameters
with Python.
- Add the `WenCai` and `SearXNG` Canvas components and register their
tool factories.
- Match Python's current WenCai behavior by returning an empty report
while its upstream request is disabled.
- Add SearXNG request validation, SSRF-safe DNS pinning, raw result
preservation, and reference rendering.
- Support context cancellation, error envelopes, and lock-safe retrieval
references.
## Tests
Passed:
- `bash build.sh --test ./internal/agent/tool/...`
- `bash build.sh --test ./internal/agent/component/...`
- `bash build.sh --test ./internal/agent/runtime/...`
- `bash build.sh --test ./internal/agent/...`
- `cd web && npm run type-check`
<img width="1900" height="1102" alt="image"
src="https://github.com/user-attachments/assets/ec77d217-d9fd-455a-96ec-9aabf6841109"
/>
<img width="1900" height="1102" alt="image"
src="https://github.com/user-attachments/assets/52ac129f-cb65-453d-ae48-cc518803ac23"
/>
### Summary
Port the **QWeather** agent tool to the modern `ToolBase` / `_invoke`
interface. It was still written against the removed legacy
`ComponentBase` / `_run` / `be_output` API, so it was non-functional as
an Agent tool — adding it to an Agent raised `AttributeError` because it
had no `get_meta()`. This is the same defect that was fixed for the
AkShare tool in #16417.
**Changes**
- `QWeatherParam` now extends `ToolParamBase` with a `meta` exposing a
`query` (location) parameter, and adds `get_input_form()`. Existing
config (`web_apikey`, `lang`, `type`, `user_type`, `time_period`) is
preserved.
- `QWeather` now extends `ToolBase` and implements `_invoke(**kwargs)`
with the standard retry loop, cancellation checks,
`set_output("formalized_content", ...)`, and `thoughts()`. The weather /
indices / air-quality branches and the API error-code messages are kept.
- Added `test/unit_test/agent/component/test_qweather.py` covering the
restored `meta`, param validation, the weather-now and multi-day and
indices branches, the empty-query short-circuit, and the location-lookup
error message.
**Testing**
- `ruff check agent/tools/qweather.py
test/unit_test/agent/component/test_qweather.py` — clean
- `ruff format --check` — clean
- `pytest test/unit_test/agent/component/test_qweather.py`
## Summary
- Add the GitHub Canvas component with tool registration and reference
propagation.
- Align the Invoke component with the Python contract for node config,
input form, response output, and timing fields.
- GitHub search and HTTP Invoke now work correctly in the Go Canvas
runtime.
## Tests
- `bash build.sh --test ./internal/agent/tool/...`
- `bash build.sh --test ./internal/agent/component/...`
Note: the untracked go_ragflow_cli file is not part of the PR changes.
<img width="1813" height="1102" alt="image"
src="https://github.com/user-attachments/assets/f69cef32-59a0-4287-a06b-6843d85198cf"
/>
<img width="1813" height="1102" alt="image"
src="https://github.com/user-attachments/assets/b37dfc31-bc9b-4937-a38e-d2184bb157fe"
/>
## Summary
- register the Go `ArXiv` canvas component and add its input form
- align the Go ArXiv request/schema with Python by keeping only `query`
in runtime args and moving `top_n`/`sort_by` to node params
- keep ArXiv results consistent for canvas output and tool response
handling
## Test
- `bash build.sh --test ./internal/agent/tool
./internal/agent/component`
<img width="1817" height="972" alt="image"
src="https://github.com/user-attachments/assets/7f726dfa-a996-4561-b481-cb0b44bec81c"
/>
### Summary
1. update docker compose file to start NATS healthy
2. Add two commands
```
RAGFlow(admin)> live;
SUCCESS
RAGFlow(admin)> health;
+---------------+-------+
| field | value |
+---------------+-------+
| storage | ok |
| message_queue | ok |
| status | ok |
| db | ok |
| redis | ok |
| doc_engine | ok |
+---------------+-------+
```
---------
Signed-off-by: Jin Hai <haijin.chn@gmail.com>
### What problem does this PR solve?
Issue [#16758](https://github.com/infiniflow/ragflow/issues/16758) —
clicking a chunk whose data references a single-line variable from an
Await-Response (UserFillUp) component, the Agent's `user_prompt` is
being resolved against the **previous** canvas run's captured value
instead of the current run's value. The system-prompt path works only
because the system prompt is computed upstream and re-reads the value on
the new run.
### Root cause
`Canvas._run_impl` reset every path component with `only_output=True`,
so `_param.inputs` was never cleared between runs.
`ComponentBase.get_input()` calls `set_input_value(var, resolved)` at
line 482, which writes the resolved variable into
`self._param.inputs[var]["value"]`. On the next canvas run, that input
was never cleared, so the previous run's resolved value stuck around.
The Agent's `kwargs.get("user_prompt")` then read the stale string and
forwarded it to the LLM, which produced the "Understood. Please provide
the text..." fallback because the prompt looked empty.
### What changed?
- `agent/canvas.py` — differentiate `begin` (still `only_output=True`,
since it has no inputs and the webhook payload branch below populates
`request` explicitly) from non-begin path components (reset with
`only_output=False`, which clears both `inputs` and `outputs`).
- `test/unit_test/agent/test_canvas_input_reset.py` — new pytest module.
Pinned the contract: non-begin path components receive
`only_output=False`. The fix is small enough to verify with a stub
canvas rather than a full canvas-runtime test (the existing agent
conftest hits an unrelated `scholarly` import on Python 3.13, so a real
canvas import would require fixing that first).
### Backward compatibility
- `Begin` behaviour unchanged.
- All non-begin path components: previously persisted inputs across runs
(the bug); now reset between runs. Components that were relying on stale
inputs (none found in the existing test suite) would lose that as a side
effect, but that is the entire point of the fix.
- No API surface change. No backend change.
### Testing
```
$ uv run pytest test/unit_test/agent/test_canvas_input_reset.py -v
collected 4 items
test/unit_test/agent/test_canvas_input_reset.py::test_begin_is_reset_with_only_output_true PASSED
test/unit_test/agent/test_canvas_input_reset.py::test_non_begin_path_components_are_reset_with_only_output_false PASSED
test/unit_test/agent/test_canvas_input_reset.py::test_only_path_components_are_reset PASSED
test/unit_test/agent/test_canvas_input_reset.py::test_inputs_reset_flag_is_passed_to_non_begin_components PASSED
4 passed in 0.14s
```
`python3 -m py_compile agent/canvas.py` clean. Existing agent test files
(`test_switch.py`, `test_llm_prompt.py`) hit a pre-existing `scholarly`
import error on Python 3.13 (unrelated to this PR), so I couldn't run
the full agent suite. Recommend fixing the `scholarly` import
separately.
### Files changed
- `agent/canvas.py` (+9 / −1)
- `test/unit_test/agent/test_canvas_input_reset.py` (new, +104)
Fixes#16758
---------
Co-authored-by: Harsh Kashyap <harshkashyap@Harshs-MacBook-Pro.local>
Co-authored-by: Zhichang Yu <yuzhichang@gmail.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Fixes#7316.
## Problem
`deepdoc/vision/operators.py` defines the image-standardize
preprocessing op as `class StandardizeImag` (missing the final `e`), but
every caller — including
`deepdoc/vision/recognizer.py::Recognizer.preprocess` — looks the class
up by the canonical string `"StandardizeImage"` via:
```python
op_type = new_op_info.pop("type") # "StandardizeImage"
preprocess_ops.append(getattr(operators, op_type)(**new_op_info))
```
So `getattr(operators, "StandardizeImage")` raised `AttributeError`, and
the "StandardizeImage" preprocessing step silently never ran for any
image pipeline that used the dynamic dispatch (LayoutLMv3 and friends).
The user-visible symptom is that the standardize step is missing
entirely from the preprocessing chain, so the model gets un-normalized
images.
## Production fix
```diff
-class StandardizeImag:
+class StandardizeImage:
"""normalize image
Args:
mean (list): im - mean
std (list): im / std
is_scale (bool): whether need im / 255
norm_type (str): type in ['mean_std', 'none']
"""
```
That's the entire production change — a one-character class rename. The
misnamed `StandardizeImag` had no other references in the codebase
(verified via `git grep`), so removing it is safe; every caller uses the
canonical `"StandardizeImage"` string and will now resolve correctly.
## Tests
New `test/unit_test/deepdoc/vision/test_operators_standardize_image.py`
with six regression tests, all green locally:
```
test_standardize_image_class_resolves_by_canonical_name PASSED
test_standardize_image_callable_matches_legacy_alias_name PASSED
test_standardize_image_normalizes_input_with_mean_std_and_is_scale PASSED
test_standardize_image_skips_scaling_when_is_scale_false PASSED
test_standardize_image_norm_type_none_passes_image_through PASSED
test_standardize_image_via_module_getattr_dispatch_path PASSED
6 passed in 0.18s
```
The tests:
1. **Pin the dispatch contract** (`hasattr(operators,
"StandardizeImage")`) — this is the exact check the recognizer's
`getattr` would do, so any future regression fails the same way the
runtime would.
2. **Pin that the misspelled name is gone** — if a downstream caller
ever relied on it, this fails loudly.
3–5. **Behavioural coverage** of the three documented code paths:
`is_scale=True, norm_type="mean_std"`, `is_scale=False,
norm_type="mean_std"`, and `norm_type="none"`.
6. **End-to-end via the same `getattr(operators, "StandardizeImage")`
call** the recognizer uses, with a real numpy image, so any rename or
removal surfaces as `AttributeError` instead of silently skipping the
step.
Verified both ways:
- Without the fix → **all 6 tests fail** (Python even suggests
`'StandardizeImag' → 'StandardizeImage'`)
- With the fix → all 6 pass in 0.15s
The test file follows the project's existing pattern
(`test/unit_test/deepdoc/parser/test_html_parser.py`): load the target
module via `importlib.util.spec_from_file_location`, stub the only
project-internal import (`rag.utils.lazy_image`), and assert against the
loaded module — no full RAGFlow runtime required.
## Risk
Very low. The class is renamed; no public Python API was using the
misnamed class. The only reference path is the `"StandardizeImage"`
string in `recognizer.py:270`, which now resolves correctly.
## Out of scope
- No other ops in `operators.py` are affected; checked all the others
(DecodeImage, NormalizeImage, Permute, etc.) and they all use correct
names.
- The dynamic-dispatch lookups in `recognizer.py` for `LinearResize`,
`StandardizeImage`, `Permute`, `PadStride` all use the same dispatch
path; only the `StandardizeImage` key was broken. No other keys need
fixing.
Made with [Cursor](https://cursor.com)
---------
Co-authored-by: Taranum01 <Taranum01@users.noreply.github.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Zhichang Yu <yuzhichang@gmail.com>
## What
An **Await Response** (`UserFillUp`) node placed inside a **Loop** now
pauses and waits for a fresh user response on **every** iteration,
instead of only on the first one.
## Problem
When a `UserFillUp` node lives inside a `Loop`, it only paused for input
on the first iteration. On subsequent iterations the loop ran straight
through, silently reusing the answer the user gave the first time.
Root cause is in `UserFillUp._invoke` / the canvas wait-check
(`agent/canvas.py`). The wait-check decides whether to pause by calling
`Canvas._is_input_field_satisfied` on the node's form fields — a field
counts as satisfied as soon as its `value` is not `None`:
```python
@staticmethod
def _is_input_field_satisfied(field):
...
if value is None:
return False
return True
```
The same component object is reused across loop iterations, and
`UserFillUp._invoke` writes the answer into
`self._param.inputs[...]["value"]` via `set_input_value`. Nothing
cleared those values when the node was re-entered for the next
iteration, so:
| Iteration | Entry (no answer yet) | Field value | Satisfied? | Result
|
|---|---|---|---|---|
| 1 | fresh | `None` | no | pauses ✅ |
| 1 | resume w/ answer | `answer` | yes | continues ✅ |
| 2 | fresh | `answer` (**stale**) | yes | continues ❌ (should pause) |
## Fix
When a `UserFillUp` is entered without a fresh user answer
(`merged_inputs` is empty), clear the retained form values so the
wait-check treats the form as unsatisfied and pauses again:
```python
merged_inputs = self._merge_runtime_inputs(kwargs.get("inputs", {}))
if not merged_inputs:
self._clear_form_values()
```
- Fresh entry / new loop iteration → no answer supplied → values cleared
→ node pauses and waits.
- Resume with an answer → `merged_inputs` is non-empty → values applied
normally, nothing cleared.
- Non-loop behavior is unchanged: the first entry already had `None`
values, so clearing is a no-op there.
`Begin` overrides `_invoke` and is unaffected.
## Tests
Added to
`test/testcases/test_web_api/test_canvas_app/test_fillup_unit.py`:
- `test_user_fillup_clears_stale_values_on_reentry_without_answer` — a
retained value is cleared on a fresh entry with no answer (loop
re-entry).
- `test_user_fillup_keeps_values_when_answer_supplied` — a supplied
answer is applied and not cleared.
All unit tests pass and `ruff check` is clean.
## Scope
This targets the Python agent runtime (`agent/`). It is independent of
any other in-flight Await Response change.
### Summary
- Make `BLOB_STORAGE_SIZE_THRESHOLD` configurable through an environment
variable.
- Preserve the existing 20 MiB default.
- Add tests for the default and configured values.
### Why
Blob storage, Seafile, and WebDAV connectors currently use a hardcoded
20 MiB limit. Self-hosted users
cannot raise this limit without modifying the source code inside the
container.
### Testing
- `test/unit_test/data_source/test_config.py`: 2 passed
- `ruff check common/data_source/config.py
test/unit_test/data_source/test_config.py`
Fixes#16634
### What problem does this PR solve?
- Clear stale pipeline IDs and generated data when updating documents
without `pipeline_id`.
- Support tree compilation results in pipeline workflows.
- Update compilation templates in place while preserving existing
template IDs.
- Improve duplicate-template validation messages.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
- [x] New Feature (non-breaking change which adds functionality)
Co-authored-by: Jin Hai <haijin.chn@gmail.com>
