Compare commits

...

208 Commits

Author SHA1 Message Date
balibabu
999eb533a9 Feat: Adjust the color and size of the graph based on the data. (#16868) 2026-07-14 10:20:31 +08:00
Zhichang Yu
d279aee1ff Go ports of workflow and chunker fixes (#16878)
Ports two Python fixes to Go: the variable_ref_patt underscore/colon fix
(#16792) and the TokenChunker upstream-chunks fix (#16825). Keeps Go
behavior aligned with upstream Python.
2026-07-13 23:32:07 +08:00
Yash Raj Pandey
e54e7ec7ef Fix: TokenChunker discards TitleChunker chunks when output_format is 'chunks' (#16825)
Fixes #16812

### Problem
In the `rag/flow` ingestion pipeline, when `TitleChunker` feeds
`TokenChunker`, the chapter-aware chunks are silently discarded and the
parser's raw flat json is re-chunked instead.

`TitleChunker` emits `output_format="chunks"` and writes its
chapter-aware output to the `chunks` field
(`rag/flow/chunker/title_chunker/common.py`,
`set_output("output_format", "chunks")`). But `TokenChunker._invoke`
only handles `output_format` in `["markdown", "text", "html"]`, then
falls through to the `# json` path which reads
`from_upstream.json_result`. There is no branch for `"chunks"`, so
`from_upstream.chunks` is never read.

Downstream effects reported in #16812: PageIndex/TOC extraction receives
flat line-level text instead of structured chapter blocks
(incorrect/duplicate/missing chapters), and retrieval quality degrades
because chunks are no longer aligned to document structure.

### Fix
Select the source list based on `output_format`, mirroring the exact
pattern already used in `title_chunker/common.py`:

```python
json_result = (from_upstream.chunks if from_upstream.output_format == "chunks" else from_upstream.json_result) or []
```

`chunks` items share the same dict shape as `json_result` items (both
consumed via `.get("text")`, `.get("doc_type_kwd")`, etc.), so they flow
through the existing token-sizing path unchanged. One-line change, no
behavior change for the `json`/`markdown`/`text`/`html` paths.

### Test
Adds `rag/flow/tests/test_token_chunker.py`, an isolated unit test that
runs the real `TokenChunker._invoke` (heavy deps stubbed; real pydantic
schema used when available) and asserts that with
`output_format="chunks"` the upstream `chunks` are consumed rather than
the raw parser `json`.

Verified RED -> GREEN: the test fails against the current code (reads
the raw json) and passes with the fix.

Signed-off-by: Yash Raj Pandey <yashpn62@gmail.com>
2026-07-13 22:04:56 +08:00
Taranum Wasu
d48fd37ff1 fix(agent): allow underscores in variable_ref_patt component_id (#16758) (#16792)
## Summary

`ComponentBase.variable_ref_patt` (and its duplicate in
`agent.canvas.Graph.get_value_with_variable`) is the regex the canvas
runtime uses to find `cpn_id@var_nm` template refs in component prompts.

The `cpn_id` half was constrained to `[a-zA-Z:0-9]+`, which silently
dropped underscores. Component ids emitted by the frontend all contain
underscores (`userfillup_abc`, `retrieval_xyz`, `llm_0`, `message_0`,
…), so any template ref like `{userfillup_abc@line}` failed to match.
The placeholder then leaked through to the LLM verbatim, and the Agent
answered only its system-prompt directive.

This is exactly the "unconsidered await response" symptom in #16758:

```
Begin(Task) -> Await response -> Agent -> Message
```

Widen `cpn_id` from `[a-zA-Z:0-9]+` to `[a-zA-Z0-9_]+`. Bare `{line}`
(no cpn_id) remains unrecognised so it stays literal until the user
wires it up — matching the existing `VARIABLE_REF_PATTERN` shape used by
`agent.dsl_migration` for the same purpose.

## Changes

- `agent/component/base.py` — fix `variable_ref_patt` class attribute.
- `agent/canvas.py` — same fix applied to the inline regex inside
`Graph.get_value_with_variable` (kept as the literal regex to avoid
coupling the two unrelated sites).
-
`test/testcases/test_web_api/test_canvas_app/test_variable_ref_pattern_unit.py`
— new regression test pinning both the regex shape and end-to-end
resolution.

## Regression coverage

```
test_variable_ref_patt_matches_underscored_component_ids     PASSED
test_variable_ref_patt_still_matches_legacy_ids              PASSED
test_get_input_elements_from_text_resolves_underscored_id    PASSED
test_string_format_substitutes_underscored_ref                PASSED
test_variable_ref_patt_does_not_match_bare_var_name          PASSED
```

All five regression tests fail against the pre-fix regex (verified via
`git stash` round trip — drop fix, tests fail, restore fix, tests pass).

The two targeted existing tests in the same directory
(`test_fillup_unit.py`, `test_iterationitem_unit.py`) continue to pass.

## Repro before the fix

```python
import re
patt = r"\{* *\{([a-zA-Z:0-9]+@[A-Za-z0-9_.-]+|sys\.[A-Za-z0-9_.]+|env\.[A-Za-z0-9_.]+)\} *\}*"
list(re.finditer(patt, "{userfillup_abc@line}"))
# => []   # <-- bug
```

## Repro after the fix

```python
import re
patt = r"\{* *\{([a-zA-Z0-9_]+@[A-Za-z0-9_.-]+|sys\.[A-Za-z0-9_.]+|env\.[A-Za-z0-9_.]+)\} *\}*"
list(re.finditer(patt, "{userfillup_abc@line}"))
# => [<re.Match object; span=(0, 24), match='{userfillup_abc@line}'>]
```

Fixes #16758

## Test plan

- [x] New unit tests pass
- [x] Reverse-apply the fix and confirm the regression tests fail (they
do)
- [x] `test_fillup_unit.py` (existing sibling suite) still passes
- [x] `test_iterationitem_unit.py` (existing sibling suite) still passes
- [ ] Project CI green

---------

Co-authored-by: Taranum01 <taranum01@users.noreply.github.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-07-13 22:03:48 +08:00
Jin Hai
844e3ea64d Go and Python: fix IDOR issue of search completions (#16864)
### Summary

In Go and python implementation, the dataset / KB id isn't validated if
it is accessible by this user.

---------

Signed-off-by: Jin Hai <haijin.chn@gmail.com>
2026-07-13 21:00:19 +08:00
balibabu
45862bf95d Feat: If the interval between two outputs exceeds 600ms, a loading state is displayed at the end. (#16861)
### Summary

Feat: If the interval between two outputs exceeds 600ms, a loading state
is displayed at the end.
2026-07-13 18:02:29 +08:00
Jin Hai
8bc18154d2 Go: refactor and add version type (#16863)
### Summary

```
RAGFlow(admin)> show version;
+--------------+-----------------------+
| field        | value                 |
+--------------+-----------------------+
| version      | v0.26.4-84-g547bc8614 |
| version_type | open source           |
+--------------+-----------------------+
```

---------

Signed-off-by: Jin Hai <haijin.chn@gmail.com>
2026-07-13 18:00:35 +08:00
buua436
09abe5f513 fix: ensure database model indexes (#16860) 2026-07-13 17:34:28 +08:00
Haruko386
466f33e6b4 fix: support component brwoser (#16848)
### Summary

As title
Unable to test it since I don't have apiKey for `openai` , `Anthropic`
and `Gemini`

---

<img width="1130" height="557" alt="image"
src="https://github.com/user-attachments/assets/11570c75-68f3-490d-8186-4ecbcd8b8f40"
/>
2026-07-13 16:57:56 +08:00
Hz_
d03b09b6a3 fix(go-agent): Align Go WenCai and SearXNG agent components with python (#16854)
## Summary

- Align Go WenCai and SearXNG behavior, schemas, and node parameters
with Python.
- Add the `WenCai` and `SearXNG` Canvas components and register their
tool factories.
- Match Python's current WenCai behavior by returning an empty report
while its upstream request is disabled.
- Add SearXNG request validation, SSRF-safe DNS pinning, raw result
preservation, and reference rendering.
- Support context cancellation, error envelopes, and lock-safe retrieval
references.

  ## Tests

  Passed:

  - `bash build.sh --test ./internal/agent/tool/...`
  - `bash build.sh --test ./internal/agent/component/...`
  - `bash build.sh --test ./internal/agent/runtime/...`
  - `bash build.sh --test ./internal/agent/...`
  - `cd web && npm run type-check`
  
  
<img width="1900" height="1102" alt="image"
src="https://github.com/user-attachments/assets/ec77d217-d9fd-455a-96ec-9aabf6841109"
/>
  
<img width="1900" height="1102" alt="image"
src="https://github.com/user-attachments/assets/52ac129f-cb65-453d-ae48-cc518803ac23"
/>
2026-07-13 16:50:17 +08:00
euvre
3bfad1f00e fix: correct model type mappings and improve system setting persistence (#16501) 2026-07-13 16:42:55 +08:00
Sbaaoui Idriss
d35e957252 fix: test drift on go specific proxy scheme (#16796)
### Summary

certain tests fail because of test drift and were fixed, other because
of go issues

---------

Co-authored-by: Wang Qi <wangq8@outlook.com>
2026-07-13 16:41:38 +08:00
Jack
cfacaccad7 Refactor: message processing (#16852)
### Summary

1. refactor message processing
2. delete un-used componentIndexMap
3. unfold (delete) internal/ingestion/task/task_handler.go
2026-07-13 16:32:34 +08:00
boskodev790
80a7a87427 fix(agent): port QWeather to ToolBase so it works as an Agent tool (#16692)
### Summary

Port the **QWeather** agent tool to the modern `ToolBase` / `_invoke`
interface. It was still written against the removed legacy
`ComponentBase` / `_run` / `be_output` API, so it was non-functional as
an Agent tool — adding it to an Agent raised `AttributeError` because it
had no `get_meta()`. This is the same defect that was fixed for the
AkShare tool in #16417.

**Changes**
- `QWeatherParam` now extends `ToolParamBase` with a `meta` exposing a
`query` (location) parameter, and adds `get_input_form()`. Existing
config (`web_apikey`, `lang`, `type`, `user_type`, `time_period`) is
preserved.
- `QWeather` now extends `ToolBase` and implements `_invoke(**kwargs)`
with the standard retry loop, cancellation checks,
`set_output("formalized_content", ...)`, and `thoughts()`. The weather /
indices / air-quality branches and the API error-code messages are kept.
- Added `test/unit_test/agent/component/test_qweather.py` covering the
restored `meta`, param validation, the weather-now and multi-day and
indices branches, the empty-query short-circuit, and the location-lookup
error message.

**Testing**
- `ruff check agent/tools/qweather.py
test/unit_test/agent/component/test_qweather.py` — clean
- `ruff format --check` — clean
- `pytest test/unit_test/agent/component/test_qweather.py`
2026-07-13 16:31:19 +08:00
Hz_
2a482f3ae7 fix(go-agent): align GitHub and Invoke Canvas components (#16849)
## Summary

- Add the GitHub Canvas component with tool registration and reference
propagation.
- Align the Invoke component with the Python contract for node config,
input form, response output, and timing fields.
- GitHub search and HTTP Invoke now work correctly in the Go Canvas
runtime.

## Tests

- `bash build.sh --test ./internal/agent/tool/...`
- `bash build.sh --test ./internal/agent/component/...`

Note: the untracked go_ragflow_cli file is not part of the PR changes.

<img width="1813" height="1102" alt="image"
src="https://github.com/user-attachments/assets/f69cef32-59a0-4287-a06b-6843d85198cf"
/>


<img width="1813" height="1102" alt="image"
src="https://github.com/user-attachments/assets/b37dfc31-bc9b-4937-a38e-d2184bb157fe"
/>
2026-07-13 15:48:07 +08:00
chanx
547bc86141 Fixed an issue where cited webpages could not be opened during online searches. (#16840) 2026-07-13 15:25:57 +08:00
Jin Hai
9f403ac3ca Go: more APIs (#16850)
### Summary

as title.

---------

Signed-off-by: Jin Hai <haijin.chn@gmail.com>
2026-07-13 15:24:17 +08:00
chanx
f21368057b fix(setting-model): fix instance display and api_key loss after auto-save (#16853) 2026-07-13 15:20:01 +08:00
balibabu
2b9569ff51 Feat: Added support for session graph and session essence templates. (#16851)
### Summary

Feat: Added support for session graph and session essence templates.
2026-07-13 14:48:02 +08:00
qinling0210
d549194562 Implement builtin chunk method as ingestion pipeline in GO (#16822)
### Summary

Implement builtin chunk mehtod as ingestion pipeline in GO
2026-07-13 13:51:40 +08:00
Zhichang Yu
ed27b5f9f8 uv tool install lefthook (#16842) 2026-07-13 13:01:33 +08:00
Lynn
b0cac0ac9d Fix: Align part of Go provider APIs with Python APIs (#16823) 2026-07-13 11:20:02 +08:00
Jack
3c0f8fc192 Refactor: refactor dataflow_service.go and guard nats message re-delivery (#16826)
### Summary

1. refactor dataflow_service.go 
2. guard nats message re-delivery
3. support document parse cancelling & re-run
2026-07-13 11:08:04 +08:00
Hz_
347a45967e fix(go-agent): ArXiv component registration and request parity (#16808)
## Summary
- register the Go `ArXiv` canvas component and add its input form
- align the Go ArXiv request/schema with Python by keeping only `query`
in runtime args and moving `top_n`/`sort_by` to node params
- keep ArXiv results consistent for canvas output and tool response
handling

## Test
- `bash build.sh --test ./internal/agent/tool
./internal/agent/component`

<img width="1817" height="972" alt="image"
src="https://github.com/user-attachments/assets/7f726dfa-a996-4561-b481-cb0b44bec81c"
/>
2026-07-13 10:19:50 +08:00
dependabot[bot]
a7ef7be056 build(deps): bump github.com/xuri/excelize/v2 from 2.10.1 to 2.11.0 (#16839) 2026-07-13 09:43:19 +08:00
Jin Hai
891261e108 Go: add dummy admin functions (#16830)
As title.

Signed-off-by: Jin Hai <haijin.chn@gmail.com>
2026-07-11 23:38:22 +08:00
Jin Hai
bdef878821 Go: fix nats and go command (#16828)
### Summary
1. update docker compose file to start NATS healthy
2. Add two commands
```
RAGFlow(admin)> live;
SUCCESS
RAGFlow(admin)> health;
+---------------+-------+
| field         | value |
+---------------+-------+
| storage       | ok    |
| message_queue | ok    |
| status        | ok    |
| db            | ok    |
| redis         | ok    |
| doc_engine    | ok    |
+---------------+-------+
```

---------

Signed-off-by: Jin Hai <haijin.chn@gmail.com>
2026-07-11 19:37:57 +08:00
Hz_
156a11c56b fix(go-agent): Added PubMed component support (#16817)
## Summary

- Merge upstream main and retain PubMed component support.
- Preserve newly registered tool components and update registry
verification.

## Tests

- `bash build.sh --test ./internal/agent/component/...`
- `bash build.sh --test ./internal/agent/tool/...`

<img width="1817" height="972" alt="image"
src="https://github.com/user-attachments/assets/9fcb9448-9e26-41b9-940c-a9bfde9835e9"
/>

---------

Co-authored-by: Jin Hai <haijin.chn@gmail.com>
2026-07-11 17:49:59 +08:00
Harsh Kashyap
8a3699fa87 fix(agent): clear component inputs on canvas re-run (#16790)
### What problem does this PR solve?

Issue [#16758](https://github.com/infiniflow/ragflow/issues/16758) —
clicking a chunk whose data references a single-line variable from an
Await-Response (UserFillUp) component, the Agent's `user_prompt` is
being resolved against the **previous** canvas run's captured value
instead of the current run's value. The system-prompt path works only
because the system prompt is computed upstream and re-reads the value on
the new run.

### Root cause

`Canvas._run_impl` reset every path component with `only_output=True`,
so `_param.inputs` was never cleared between runs.
`ComponentBase.get_input()` calls `set_input_value(var, resolved)` at
line 482, which writes the resolved variable into
`self._param.inputs[var]["value"]`. On the next canvas run, that input
was never cleared, so the previous run's resolved value stuck around.
The Agent's `kwargs.get("user_prompt")` then read the stale string and
forwarded it to the LLM, which produced the "Understood. Please provide
the text..." fallback because the prompt looked empty.

### What changed?

- `agent/canvas.py` — differentiate `begin` (still `only_output=True`,
since it has no inputs and the webhook payload branch below populates
`request` explicitly) from non-begin path components (reset with
`only_output=False`, which clears both `inputs` and `outputs`).
- `test/unit_test/agent/test_canvas_input_reset.py` — new pytest module.
Pinned the contract: non-begin path components receive
`only_output=False`. The fix is small enough to verify with a stub
canvas rather than a full canvas-runtime test (the existing agent
conftest hits an unrelated `scholarly` import on Python 3.13, so a real
canvas import would require fixing that first).

### Backward compatibility

- `Begin` behaviour unchanged.
- All non-begin path components: previously persisted inputs across runs
(the bug); now reset between runs. Components that were relying on stale
inputs (none found in the existing test suite) would lose that as a side
effect, but that is the entire point of the fix.
- No API surface change. No backend change.

### Testing

```
$ uv run pytest test/unit_test/agent/test_canvas_input_reset.py -v
collected 4 items
test/unit_test/agent/test_canvas_input_reset.py::test_begin_is_reset_with_only_output_true PASSED
test/unit_test/agent/test_canvas_input_reset.py::test_non_begin_path_components_are_reset_with_only_output_false PASSED
test/unit_test/agent/test_canvas_input_reset.py::test_only_path_components_are_reset PASSED
test/unit_test/agent/test_canvas_input_reset.py::test_inputs_reset_flag_is_passed_to_non_begin_components PASSED
4 passed in 0.14s
```

`python3 -m py_compile agent/canvas.py` clean. Existing agent test files
(`test_switch.py`, `test_llm_prompt.py`) hit a pre-existing `scholarly`
import error on Python 3.13 (unrelated to this PR), so I couldn't run
the full agent suite. Recommend fixing the `scholarly` import
separately.

### Files changed

- `agent/canvas.py` (+9 / −1)
- `test/unit_test/agent/test_canvas_input_reset.py` (new, +104)

Fixes #16758

---------

Co-authored-by: Harsh Kashyap <harshkashyap@Harshs-MacBook-Pro.local>
Co-authored-by: Zhichang Yu <yuzhichang@gmail.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
2026-07-11 16:33:37 +08:00
Taranum Wasu
0ee02fb6d8 [Fix] Rename StandardizeImag -> StandardizeImage to fix deepdoc OCR preprocessing (#7316) (#16785)
Fixes #7316.

## Problem

`deepdoc/vision/operators.py` defines the image-standardize
preprocessing op as `class StandardizeImag` (missing the final `e`), but
every caller — including
`deepdoc/vision/recognizer.py::Recognizer.preprocess` — looks the class
up by the canonical string `"StandardizeImage"` via:

```python
op_type = new_op_info.pop("type")  # "StandardizeImage"
preprocess_ops.append(getattr(operators, op_type)(**new_op_info))
```

So `getattr(operators, "StandardizeImage")` raised `AttributeError`, and
the "StandardizeImage" preprocessing step silently never ran for any
image pipeline that used the dynamic dispatch (LayoutLMv3 and friends).
The user-visible symptom is that the standardize step is missing
entirely from the preprocessing chain, so the model gets un-normalized
images.

## Production fix

```diff
-class StandardizeImag:
+class StandardizeImage:
     """normalize image
     Args:
         mean (list): im - mean
         std (list): im / std
         is_scale (bool): whether need im / 255
         norm_type (str): type in ['mean_std', 'none']
     """
```

That's the entire production change — a one-character class rename. The
misnamed `StandardizeImag` had no other references in the codebase
(verified via `git grep`), so removing it is safe; every caller uses the
canonical `"StandardizeImage"` string and will now resolve correctly.

## Tests

New `test/unit_test/deepdoc/vision/test_operators_standardize_image.py`
with six regression tests, all green locally:

```
test_standardize_image_class_resolves_by_canonical_name            PASSED
test_standardize_image_callable_matches_legacy_alias_name          PASSED
test_standardize_image_normalizes_input_with_mean_std_and_is_scale PASSED
test_standardize_image_skips_scaling_when_is_scale_false           PASSED
test_standardize_image_norm_type_none_passes_image_through         PASSED
test_standardize_image_via_module_getattr_dispatch_path            PASSED
6 passed in 0.18s
```

The tests:
1. **Pin the dispatch contract** (`hasattr(operators,
"StandardizeImage")`) — this is the exact check the recognizer's
`getattr` would do, so any future regression fails the same way the
runtime would.
2. **Pin that the misspelled name is gone** — if a downstream caller
ever relied on it, this fails loudly.
3–5. **Behavioural coverage** of the three documented code paths:
`is_scale=True, norm_type="mean_std"`, `is_scale=False,
norm_type="mean_std"`, and `norm_type="none"`.
6. **End-to-end via the same `getattr(operators, "StandardizeImage")`
call** the recognizer uses, with a real numpy image, so any rename or
removal surfaces as `AttributeError` instead of silently skipping the
step.

Verified both ways:
- Without the fix → **all 6 tests fail** (Python even suggests
`'StandardizeImag' → 'StandardizeImage'`)
- With the fix → all 6 pass in 0.15s

The test file follows the project's existing pattern
(`test/unit_test/deepdoc/parser/test_html_parser.py`): load the target
module via `importlib.util.spec_from_file_location`, stub the only
project-internal import (`rag.utils.lazy_image`), and assert against the
loaded module — no full RAGFlow runtime required.

## Risk

Very low. The class is renamed; no public Python API was using the
misnamed class. The only reference path is the `"StandardizeImage"`
string in `recognizer.py:270`, which now resolves correctly.

## Out of scope

- No other ops in `operators.py` are affected; checked all the others
(DecodeImage, NormalizeImage, Permute, etc.) and they all use correct
names.
- The dynamic-dispatch lookups in `recognizer.py` for `LinearResize`,
`StandardizeImage`, `Permute`, `PadStride` all use the same dispatch
path; only the `StandardizeImage` key was broken. No other keys need
fixing.

Made with [Cursor](https://cursor.com)

---------

Co-authored-by: Taranum01 <Taranum01@users.noreply.github.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Zhichang Yu <yuzhichang@gmail.com>
2026-07-11 16:32:03 +08:00
Jin Hai
e6e99b86a6 PY: fix admin error (#16827)
### Summary

Sync code from EE

Signed-off-by: Jin Hai <haijin.chn@gmail.com>
2026-07-11 14:57:26 +08:00
Öndery
4060cd1440 fix(agent): Await Response pauses on every Loop iteration (#16794)
## What

An **Await Response** (`UserFillUp`) node placed inside a **Loop** now
pauses and waits for a fresh user response on **every** iteration,
instead of only on the first one.

## Problem

When a `UserFillUp` node lives inside a `Loop`, it only paused for input
on the first iteration. On subsequent iterations the loop ran straight
through, silently reusing the answer the user gave the first time.

Root cause is in `UserFillUp._invoke` / the canvas wait-check
(`agent/canvas.py`). The wait-check decides whether to pause by calling
`Canvas._is_input_field_satisfied` on the node's form fields — a field
counts as satisfied as soon as its `value` is not `None`:

```python
@staticmethod
def _is_input_field_satisfied(field):
    ...
    if value is None:
        return False
    return True
```

The same component object is reused across loop iterations, and
`UserFillUp._invoke` writes the answer into
`self._param.inputs[...]["value"]` via `set_input_value`. Nothing
cleared those values when the node was re-entered for the next
iteration, so:

| Iteration | Entry (no answer yet) | Field value | Satisfied? | Result
|
|---|---|---|---|---|
| 1 | fresh | `None` | no | pauses  |
| 1 | resume w/ answer | `answer` | yes | continues  |
| 2 | fresh | `answer` (**stale**) | yes | continues  (should pause) |

## Fix

When a `UserFillUp` is entered without a fresh user answer
(`merged_inputs` is empty), clear the retained form values so the
wait-check treats the form as unsatisfied and pauses again:

```python
merged_inputs = self._merge_runtime_inputs(kwargs.get("inputs", {}))
if not merged_inputs:
    self._clear_form_values()
```

- Fresh entry / new loop iteration → no answer supplied → values cleared
→ node pauses and waits.
- Resume with an answer → `merged_inputs` is non-empty → values applied
normally, nothing cleared.
- Non-loop behavior is unchanged: the first entry already had `None`
values, so clearing is a no-op there.

`Begin` overrides `_invoke` and is unaffected.

## Tests

Added to
`test/testcases/test_web_api/test_canvas_app/test_fillup_unit.py`:

- `test_user_fillup_clears_stale_values_on_reentry_without_answer` — a
retained value is cleared on a fresh entry with no answer (loop
re-entry).
- `test_user_fillup_keeps_values_when_answer_supplied` — a supplied
answer is applied and not cleared.

All unit tests pass and `ruff check` is clean.

## Scope

This targets the Python agent runtime (`agent/`). It is independent of
any other in-flight Await Response change.
2026-07-10 23:24:04 +08:00
Jack
5e60fcec9f Refactor: Refine ingestion task state transitions (#16814)
### Summary

Refine ingestion task state transitions
2026-07-10 22:47:51 +08:00
hyotaek kim
9b60870fd6 feat: make blob storage size threshold configurable (#16806)
### Summary

- Make `BLOB_STORAGE_SIZE_THRESHOLD` configurable through an environment
variable.
  - Preserve the existing 20 MiB default.
  - Add tests for the default and configured values.

  ### Why

Blob storage, Seafile, and WebDAV connectors currently use a hardcoded
20 MiB limit. Self-hosted users
cannot raise this limit without modifying the source code inside the
container.

  ### Testing

  - `test/unit_test/data_source/test_config.py`: 2 passed
- `ruff check common/data_source/config.py
test/unit_test/data_source/test_config.py`

  Fixes #16634
2026-07-10 21:36:12 +08:00
buua436
d291e4641d fix: improve pipeline compilation, template updates, and document resets (#16815)
### What problem does this PR solve?

- Clear stale pipeline IDs and generated data when updating documents
without `pipeline_id`.
- Support tree compilation results in pipeline workflows.
- Update compilation templates in place while preserving existing
template IDs.
- Improve duplicate-template validation messages.

### Type of change

- [x] Bug Fix (non-breaking change which fixes an issue)
- [x] New Feature (non-breaking change which adds functionality)

Co-authored-by: Jin Hai <haijin.chn@gmail.com>
2026-07-10 21:17:19 +08:00
Yingfeng
30739b7f8e Fix compilation workflow (#16819)
Dataset_nav can not support large number of documents, introducing AHC
clustering as well as retrieval engine as the clustering database
2026-07-10 21:11:19 +08:00
chanx
ea9b3789ce fix(prompt-editor): prevent premature variable path merge during typing (#16811)
### Summary

fix(prompt-editor): prevent premature variable path merge during typing

---------

Co-authored-by: Jin Hai <haijin.chn@gmail.com>
2026-07-10 20:57:59 +08:00
Jin Hai
0aa2993d55 Go: format code (#16813)
### Summary

Aligned to EE

Signed-off-by: Jin Hai <haijin.chn@gmail.com>
2026-07-10 20:30:00 +08:00
Jin Hai
106c2d0a41 Go: add show users plan quota (#16818)
### Summary

```
RAGFlow(admin)> show users plan quota 100;
+---------+------------------------------------------+
| field   | value                                    |
+---------+------------------------------------------+
| quota   | 100                                      |
| command | show_users_plan_quota                    |
| error   | 'Show users plan quota' is not supported |
+---------+------------------------------------------+
```

Signed-off-by: Jin Hai <haijin.chn@gmail.com>
2026-07-10 20:27:33 +08:00
maoyifeng
8ced3d1f47 fix ci (#16824)
### Summary

fix ci
2026-07-10 20:20:29 +08:00
Öndery
e53235ed3d fix(docker): [URGENT] Docker build fails for all PRs - missing web/scripts/prepare.js during npm install (#16821)
## 🚨 Urgent: this breaks the Docker image build for **every** open PR

The `ragflow_tests_elasticsearch` and `ragflow_tests_infinity` jobs —
which build the image before running tests — currently **fail on all
pull requests**, not just this one. CI is effectively red repo-wide
until this lands. **Please review and merge ASAP.**

Example failing runs (from an unrelated PR):
-
https://github.com/infiniflow/ragflow/actions/runs/29086020168/job/86341032173
-
https://github.com/infiniflow/ragflow/actions/runs/29086020168/job/86341032196

## Symptom

```
Error: Cannot find module '/ragflow/web/scripts/prepare.js'
npm error command sh -c node scripts/prepare.js
ERROR: process "/bin/bash -c cd web && NODE_OPTIONS=..." did not complete successfully: exit code: 1
```

## Root cause

`web/package.json` defines:

```json
"prepare": "node scripts/prepare.js"
```

npm runs the `prepare` lifecycle script during `npm install`. But the
frontend-deps layer in the `Dockerfile` only copies the package
manifests before installing:

```dockerfile
COPY web/package.json web/package-lock.json web/.npmrc ./web/
RUN ... cd web && npm install     # runs `node scripts/prepare.js` → file not present yet
```

Since `web/scripts/prepare.js` is not in the image at that point, `npm
install` aborts and the whole build fails.

## Fix

Copy `web/scripts` before `npm install` so the `prepare` script is
present:

```dockerfile
COPY web/package.json web/package-lock.json web/.npmrc ./web/
COPY web/scripts ./web/scripts
RUN ... cd web && npm install
```

- Minimal and safe: `scripts/prepare.js` only installs lefthook git
hooks and already no-ops (wrapped in try/catch) inside the image where
there is no Git repo.
- Preferred over `--ignore-scripts`, which would also disable
dependencies' legitimate install scripts.
- `web/scripts` changes rarely, so build-cache impact on this layer is
negligible.

## Verification

Build reaches the frontend `npm install` step without the
`MODULE_NOT_FOUND` error. No application code is touched — this is a
build-infrastructure fix only.
2026-07-10 19:15:57 +08:00
Jin Hai
2a83ad6cb2 Go: Fix error code (#16807)
### Summary

As title.

---------

Signed-off-by: Jin Hai <haijin.chn@gmail.com>
2026-07-10 17:02:25 +08:00
Hz_
07a3523b09 fix(go-agent): unknown component "Wikipedia" canvas error. (#16784)
## Summary

- Register Wikipedia component + tool alias
`wikipedia`/`wikipedia_search`
- Use `generator=search` to get title/summary/url in one request (was
N+1)
- Node params `top_n`/`language` with validation
- Return `formalized_content` for downstream
- tests pass

<img width="1817" height="972" alt="image"
src="https://github.com/user-attachments/assets/f6d79599-6d1f-4ea6-84f7-ac06d0de13b0"
/>
2026-07-10 16:12:09 +08:00
Zhichang Yu
fb42e5531d Refactor: drop dead canvas runtime selector and tokenizer embedding wiring (#16809)
Two refactors on the Go port (agent-go-port):

- Remove the dead per-tenant canvas runtime selector (write-only Redis
scaffolding with no runtime callers) and its dependent metrics/admin
code.
- Move the tokenizer embedding-model id from the shared ingestion
globals into a Tokenizer-scoped setup, and wire the production embedder
resolver in the ingestion task package.

32 files changed, 861 insertions, 1228 deletions.
2026-07-10 15:46:45 +08:00
balibabu
d317742975 Feat: Rewrite wiki template with reui (#16797) 2026-07-10 15:44:04 +08:00
Jack
7db39822db Feature: user select pipeline support (#16788)
### Summary

Feature: user select pipeline support
2026-07-10 14:30:28 +08:00
Jin Hai
0b01171a86 Go: Update response message (#16803)
### Summary

As title.

---------

Signed-off-by: Jin Hai <haijin.chn@gmail.com>
2026-07-10 14:26:54 +08:00
maoyifeng
c33a5a9d49 GO:Cli fix show user storage display truncation (#16805)
### Summary

GO: Cli  fix show user storage display truncation
2026-07-10 13:59:07 +08:00
Hz_
5797f81fea feat(go-agent): merge Google Scholar node params with runtime inputs (#16802)
## Summary

- align the Go Google Scholar component with the Python-side config
pattern
- merge node-level params with runtime inputs so canvas defaults are
preserved and per-run inputs can override them
- add tests covering node param fallback and runtime override behavior

## Verification

- `bash build.sh --test ./internal/agent/component/... -run
TestGoogleScholar`

<img width="1873" height="1165" alt="image"
src="https://github.com/user-attachments/assets/67198c6f-6a0e-43bf-a500-8e88d82b8751"
/>
2026-07-10 13:30:21 +08:00
buua436
28340f6218 GO: improve model info parsing and add model_id/tenant context to list response (#16804) 2026-07-10 13:29:01 +08:00
chanx
868e524f29 fix: pass ownerTenantId to LLMLabel and related components for improved model fetching (#16800) 2026-07-10 13:27:14 +08:00
dependabot[bot]
eca8e87f44 build(deps): bump mistune from 3.2.0 to 3.3.0 (#16799)
Bumps [mistune](https://github.com/lepture/mistune) from 3.2.0 to 3.3.0.

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-07-10 12:24:56 +08:00
dependabot[bot]
a1a2ba8f6c build(deps): bump soupsieve from 2.8.1 to 2.8.4 (#16798)
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-07-10 12:24:21 +08:00
Jin Hai
15d1ee7654 Go: set gin mode in admin server (#16801)
### Summary

As title.

---------

Signed-off-by: Jin Hai <haijin.chn@gmail.com>
2026-07-10 12:19:10 +08:00
Jin Hai
add7b9486f Go: merge duplicate codes (#16783)
### Summary

1. merge heartbeat function.
2. introduce all environments

---------

Signed-off-by: Jin Hai <haijin.chn@gmail.com>
2026-07-10 11:58:32 +08:00
Harsh Kashyap
289ca28ce2 Fix OpenAI agent stream chunk shape (#16402)
### What problem does this PR solve?

Closes #8175.

The Agent OpenAI-compatible streaming path uses `get_data_openai(...,
stream=True)`, but that helper currently returns a minimal chunk shape.
The main OpenAI-compatible chat endpoint already includes chunk metadata
such as `created`, `system_fingerprint`, `usage`, `logprobs`, and
assistant role/tool placeholders.

This PR aligns the Agent stream helper with that existing
OpenAI-compatible chunk shape while keeping the current `delta.content`
behavior and existing reference injection path intact.

### Type of change

- [x] Bug Fix (non-breaking change which fixes an issue)
- [ ] New Feature (non-breaking change which adds functionality)
- [ ] Documentation Update
- [ ] Refactoring
- [ ] Performance Improvement
- [ ] Other (please describe):

### Verification

- `./.venv/bin/python -m pytest
test/unit_test/api/utils/test_api_utils.py -q`
- `python3 -m py_compile api/utils/api_utils.py
test/unit_test/api/utils/test_api_utils.py`
- `uvx ruff check api/utils/api_utils.py
test/unit_test/api/utils/test_api_utils.py`

---------

Co-authored-by: Harsh Kashyap <harshkashyap@Harshs-MacBook-Pro.local>
2026-07-10 10:59:32 +08:00
Zhichang Yu
12787996d1 feat(agent): Go ingestion pipeline progress mirroring and DeepDOC parser hardening (#16795)
feat(ingestion): mirror Go pipeline progress into the document table;
harden resume guards
- pipeline: bind the owning document via WithDocumentID; after each
TrackProgress event aggregate ingestion_task_log progress and mirror
progress/run/progress_msg back into the document table, so GET
/api/v1/datasets/{dataset_id}/documents reflects live Go pipeline
progress without a bespoke endpoint.
- canvas: extend the S3 resume guard to reject legacy no-op nodes (e.g.
ExitLoop) so component_total equals the count of progress-reporting
components and the aggregate percent can reach 100%.
- runtime/canvas: route progress through TrackProgress; add interrupt
test coverage (r3_interrupt_test.go).
- dao/entity: add IngestionTask.DocumentID column and AggregateProgress
support used by the mirror; IngestionTaskLog keeps a Checkpoint column
alongside the progress fields.

feat(deepdoc): cache DocAnalyzer inference results in Redis (1h TTL)
- Redis-backed DocAnalyzerCache decorator over inference.Client; cache
key = "ddoc:cache:<method>:" + sha256 of the JPEG-encoded image bytes
(deterministic).
- TTL = 1h; hits skip the inner HTTP call and return cached JSON; inner
errors are not cached.

refactor(deepdoc): align figure cropping with Python cropout + bounded
page caches
- CropSectionByDLA mirrors Python cropout: best-overlap DLA
figure/equation region, fallback to section bbox per page, vertical
concat on gray background.
- sliding-window page-image cache bounds peak memory to the recent
window instead of the whole PDF.
- rename DLADebug -> DLARegions across parser/chunker/tests.

refactor(parser): drop lib_type selector; align NewXxxParser with
NewPDFParser
- remove config["lib_type"] lookup and the libType param/field/switch
from all nine constructors; surface the CGO-required error at
ParseWithResult time instead of construction time; drop resolveLibType,
its test, and the four lib_type constants.

feat(utility): add a reusable workerpool for bounded concurrent
execution
- internal/utility/workerpool.go (+ tests).

refactor: translate Chinese prose comments to English in non-harness Go
files.

chore: upgrade github.com/cloudwego/eino from v0.9.9 to v0.9.12.
2026-07-10 10:36:10 +08:00
buua436
74bbbba3e0 fix: align model default handling (#16782) 2026-07-10 10:34:19 +08:00
Hz_
d48a5622df fix(go-agent): align Go DuckDuckGo component with canvas input form (#16775)
## Summary

- register the Go `DuckDuckGo` canvas component and restore its dynamic
input form metadata
- align the Go component input/output surface with the current canvas
usage for `query`, `channel`, and `top_n`
- fix DuckDuckGo news search in Go by fetching the required `vqd` token
before calling `news.js`, and add targeted regression tests

## Testing

Passed:
- `bash build.sh --test ./internal/agent/tool/... -run 'DuckDuckGo'`
- `bash build.sh --test ./internal/agent/component/... -run
'DuckDuckGo|TestVerifyRegistration_P1'`
- `bash build.sh --test ./internal/agent/component/... -run
'DuckDuckGo'`

Not run:
- frontend tests
- frontend build
- full Go test suite

<img width="1776" height="1092" alt="image"
src="https://github.com/user-attachments/assets/9f3f8e4b-f6b4-4915-b96c-3c5b8c7b8b30"
/>
2026-07-10 10:11:13 +08:00
chanx
8236f2cabb Fix: update LayoutRecognizeFormField to accept ownerTenantId and refactor model handling in LLM requests (#16781) 2026-07-10 09:36:25 +08:00
chanx
0095fa048f fix: Show full text on hover when text overflows the cards on the list page. (#16787) 2026-07-10 09:25:05 +08:00
Hz_
5c8b51cbbf fix(go-agent): add Google search wrapper component and tool registry (#16768)
### Summary

- Implemented googleComponent wrapper to bridge the canvas component
contract with Eino's SerpApi-backed GoogleTool.
- Added parameter alias mapping (query to q, max_results to num) and
content formatting logic to match Python search result representation.
- Registered the "Google" component and the "google" tool factory in the
Go agent runtime to support web search nodes.

<img width="1776" height="1092" alt="image"
src="https://github.com/user-attachments/assets/e295ab88-e48c-4fe2-bcb7-47ca5b977c9b"
/>
2026-07-09 18:58:55 +08:00
balibabu
0083ad0deb Feat: Add a data compilation layer. (#16777)
### Summary

Feat: Add a data compilation layer.
2026-07-09 17:49:16 +08:00
Lynn
5de823eab9 Fix: delete unused tenant_llm testcase (#16786) 2026-07-09 17:48:02 +08:00
Jack
8bd62573eb Fix: delete tasklet (#16778)
### Summary

1. Fix: delete tasklet
2. Remove panic in text processing
2026-07-09 17:31:28 +08:00
Hz_
b29a4a61eb fix(ao-agent): add Tavily input_form, wire real search component, and port TavilyExtract to Go (#16693)
### Summary

- TavilySearch now stores api_key from component params and injects it
into tool calls when runtime inputs omit it.
- TavilyExtract and BGPT now follow the same stored api_key behavior.
- Canvas decoding now recovers api_key from graph.nodes[].data.form when
components[].obj.params.api_key is empty, matching frontend payload
behavior without changing frontend data.
- Added regression tests for graph form key recovery and stored key
injection / caller key precedence.

Tests: build.sh --test ./internal/agent/component ./internal/service —
all pass.

<img width="1476" height="850" alt="image"
src="https://github.com/user-attachments/assets/0be31587-c1ba-4f3e-b43a-4fe0fca5a44c"
/>

<img width="1476" height="850" alt="image"
src="https://github.com/user-attachments/assets/e3edd92c-c62e-4db4-abe2-772bdf4fe1b2"
/>
2026-07-09 16:38:42 +08:00
Jack
8ac80284c4 Feat: add embedding support (#16769)
### Summary

Feat: add embedding support
2026-07-09 16:24:39 +08:00
Lynn
0ba1d37a10 Feat: optional url v1 (#16774) 2026-07-09 15:53:06 +08:00
chanx
a9420a7832 Fix: resolve shared embedding/LLM model selection errors (#16773) 2026-07-09 15:17:49 +08:00
Lynn
cc94639555 Fix: get_by_id (#16765) 2026-07-09 14:52:41 +08:00
Lynn
7fa2a0b607 Fix: rm role_id in user.go (#16771) 2026-07-09 14:44:08 +08:00
buua436
6a77523bf0 refa: resolve tenant model refs consistently (#16744) 2026-07-09 14:02:08 +08:00
Yingfeng
794fcc2517 Fix docs (#16770) 2026-07-09 13:38:29 +08:00
Yingfeng
b9ed0773ec Fix naive chunking for windows (#16767)
`\r` is ignored for splitting boundaries
2026-07-09 12:02:19 +08:00
qinling0210
ae96e636e9 Handle searching dataset without embedding model (#16742)
### Summary

Handle searching dataset without embedding model

In this PR, Searching datasets with different embedding models or
searching dataset with/without embedding models are not allowed. We will
improve the behavior later.
2026-07-09 11:38:55 +08:00
Lynn
1430d0e431 Fix: provider name (#16733) 2026-07-09 10:19:10 +08:00
balibabu
575984877f Fix: Rapid clicking results in multiple message requests being sent. (#16739) 2026-07-09 09:57:54 +08:00
euvre
3ec9187cd2 fix(web): prevent 'last saved at' label from vertical stacking in agent home card (#16756) 2026-07-09 09:46:03 +08:00
chanx
080dd84fed Feat: apply prose typography styling to markdown preview (#16752) 2026-07-09 09:45:52 +08:00
chanx
36f053a248 fix: Fixed the empty state styling on the home page. (#16755) 2026-07-09 09:45:43 +08:00
euvre
70019810a1 fix(web): show memory owner name in shared memory card (#16751) 2026-07-08 20:13:07 +08:00
euvre
74d3508a37 Fix: prevent auto-incrementing memory name suffix when only permissions change (#16750) 2026-07-08 20:12:45 +08:00
euvre
a41fef49d0 fix(web): hide folder tab in agent JSON import uploader (#16754) 2026-07-08 20:11:55 +08:00
Kevin Hu
2c59d07bdb Feat: add wiki folder (#16749)
### Summary

Add wiki folders.
2026-07-08 20:08:14 +08:00
qinling0210
8e3bbad4da Port agent PRs to GO - 5 (#16667)
### Summary

Port 

https://github.com/infiniflow/ragflow/pull/15376
https://github.com/infiniflow/ragflow/pull/16401
https://github.com/infiniflow/ragflow/pull/15484
https://github.com/infiniflow/ragflow/pull/16685
2026-07-08 19:54:29 +08:00
Virus
0f08dc070d fix: update nginx version. fix CVE-2026-9256 (#16732)
### Summary
Updates the NGINX package used in the RAGFlow Docker image from
`1.31.0-1~noble` to `1.31.2-1~noble`.

NGINX 1.31.0 is affected by CVE-2026-9256. NGINX 1.31.2 includes
the corresponding security fix and is available from the official
NGINX mainline repository for Ubuntu Noble.

### References
- nginx security advisories:
https://nginx.org/en/security_advisories.html
- Vendor advisory: https://my.f5.com/manage/s/article/K000161377

### Fix
Single-line change in `Dockerfile:62`:

```diff
-ARG NGINX_VERSION=1.31.0-1~noble
+ARG NGINX_VERSION=1.31.2-1~noble

Co-authored-by: duanyuan <duanyaun@uyuyue.com>
2026-07-08 19:19:06 +08:00
Jack
0dd0ac06f8 Feature: task executor migration to go (#16549)
### Summary

Feature: Integrate parser
2026-07-08 19:08:11 +08:00
Wang Qi
c8d1b21ae3 Fix Build-in metadata not working (#788) (#16748) 2026-07-08 19:06:54 +08:00
Hz_
e548108fff fixed(go-agent): agent-publish (#16713)
## Summary

This PR updates Go agent publish logic to persist the parent canvas
update and canvas-version save in the same transaction.

## Changes:

  - Reuse SaveOrReplaceLatest semantics for published versions
  - Add SaveOrReplaceLatestTx for transactional publish flow
  - Keep canvas release update and version persistence atomic
- Add a focused publish test covering canvas and released version state

## Tested:

```
  bash build.sh --test -run 'Test(PublishAgentUpdatesCanvasAndReleasedVersion|UpdateAgentDSLCreatesAndReplacesDraftVersion|
  UpdateAgentDSLDoesNotOverwriteLatestReleasedVersion)$' ./internal/service ./internal/dao
```

<img width="1476" height="850" alt="image"
src="https://github.com/user-attachments/assets/2c576581-1143-420b-8750-a77aa3c4292d"
/>
2026-07-08 18:47:01 +08:00
Lynn
330033d7c2 Fix(go): adapt to new db columns (#16745) 2026-07-08 18:02:11 +08:00
Jin Hai
21266286cb Go: add more commands and GCS supports (#16741)
### Summary

1. GCS supports
2. More commands
```
RAGFlow(admin)> ping store;
SUCCESS
RAGFlow(admin)> ping engine;
SUCCESS
RAGFlow(admin)> ping cache;
SUCCESS
```

---------

Signed-off-by: Jin Hai <haijin.chn@gmail.com>
2026-07-08 17:49:02 +08:00
Yingfeng
dc95b1d291 Fix skill search (#16743) 2026-07-08 17:17:50 +08:00
chanx
3d167204e7 fix: issue with memory error message display (#16738) 2026-07-08 16:47:14 +08:00
chanx
b9432bb43f Feat: add filter in chat and search page (#16707) 2026-07-08 16:47:01 +08:00
dependabot[bot]
5baf4089bf build(deps): bump golang.org/x/crypto from 0.51.0 to 0.52.0 (#16736)
Bumps [golang.org/x/crypto](https://github.com/golang/crypto) from
0.51.0 to 0.52.0.
2026-07-08 12:51:47 +08:00
Wang Qi
18ea093344 Dev: need set the stable version (#16730)
Resolve: #16729
2026-07-08 11:05:09 +08:00
Jin Hai
40cb581d16 Go: clean release.yml (#16725)
Signed-off-by: Jin Hai <haijin.chn@gmail.com>
2026-07-08 10:41:30 +08:00
euvre
699a25c19c fix(service): allow updating memory_type when memory is empty (#16668) 2026-07-08 10:03:58 +08:00
Lynn
0ae5961e1c Feat: v0.27.0 model provider (#16604) 2026-07-08 09:47:29 +08:00
Jin Hai
cb93883f3f Go: fix cgo build (#16724)
Signed-off-by: Jin Hai <haijin.chn@gmail.com>
2026-07-07 21:15:14 +08:00
Jin Hai
2b95fa2ba4 Go: remove cgo when build cli (#16723)
### Summary

Fix

Signed-off-by: Jin Hai <haijin.chn@gmail.com>
2026-07-07 21:05:07 +08:00
Jin Hai
fe835fd19c Go: fix release (#16722)
### Summary

Remove ragflow-cli building dependency

Signed-off-by: Jin Hai <haijin.chn@gmail.com>
2026-07-07 20:57:48 +08:00
Jin Hai
9e87fc6036 Go: fix ragflow-cli building (#16721) 2026-07-07 20:44:10 +08:00
writinwaters
49d9f6f98e Docs: Added v0.26.4 release notes. (#16720) 2026-07-07 20:14:16 +08:00
Jin Hai
568bac673b Go: add migrate database flag (#16719)
### Summary

--migrate will trigger database migration
```
./ragflow_main --api --migrate
```

Signed-off-by: Jin Hai <haijin.chn@gmail.com>
2026-07-07 20:11:40 +08:00
Liu An
4da9429451 Docs: Update version references to v0.26.4 in READMEs and docs (#16716) 2026-07-07 19:36:58 +08:00
euvre
41801ad2b8 fix: prevent memory name from auto-appending (1) on description update (#16714) 2026-07-07 19:34:51 +08:00
Öndery
28a41ed070 fix(task_executor): fix Langfuse flush/shutdown deadlock that freezes document parsing (#16502) 2026-07-07 19:06:30 +08:00
Yingfeng
6cd03d7a70 Fix broken logo for gitee mirrors (#16709) 2026-07-07 18:12:50 +08:00
Wang Qi
705754ea8b Fix PageIndex is not working (#16704)
Follow on PR #16515
2026-07-07 18:09:05 +08:00
Yingfeng
7db3eea37b Fix broken camo cache for french (#16706) 2026-07-07 17:50:06 +08:00
Yingfeng
ee7edddea3 Fix broken cache for release badge (#16705) 2026-07-07 17:43:29 +08:00
Yingfeng
f8f049e663 Fix docker pulls badge (#16702) 2026-07-07 17:26:47 +08:00
Jin Hai
7df7384b21 Go: refactor UUID functions (#16695)
As title

---------

Signed-off-by: Jin Hai <haijin.chn@gmail.com>
2026-07-07 17:22:08 +08:00
chanx
5236c8f659 fix: update similarity threshold fallback to use nullish coalescing (#16700) 2026-07-07 17:03:03 +08:00
Wang Qi
2de5940325 Fix cannot run raptor (#16694) 2026-07-07 17:02:04 +08:00
Wang Qi
b82169fba1 Fix: ValueError: Operation on closed image (#16697) 2026-07-07 17:00:40 +08:00
Hz_
ac5860c3e2 fix(go-pipline): list agents incorrectly filtering out ingestion pipelines (#16698) 2026-07-07 16:32:40 +08:00
Hz_
332d34c495 fix(agent): save draft version on agent update (#16691) 2026-07-07 16:32:11 +08:00
chanx
b7945f3a64 Fix: Referenced files not displaying. (#16696) 2026-07-07 16:29:17 +08:00
chanx
dd2f27d6a3 fix: Restrict the agent to using memory compatible with the embedding model. (#16699) 2026-07-07 16:28:58 +08:00
Jin Hai
9eba45249c Go: fix development guide (#16678)
### Summary

Go development guide

---------

Signed-off-by: Jin Hai <haijin.chn@gmail.com>
2026-07-07 15:09:39 +08:00
chanx
f082675e6f Fix: Prevent text overflow in confirm delete dialog (#16689) 2026-07-07 14:50:25 +08:00
chanx
5aa3e81a93 fix: remove duplicate error toast on memory update failure (#16690) 2026-07-07 14:50:09 +08:00
Wang Qi
3660b98ae9 fix: strip reasoning model thinking tags from document exports (#16687)
Co-authored-by: noob <yixiao121314@outlook.com>
2026-07-07 12:17:53 +08:00
Wang Qi
a0bda639e0 Fix Agent Chat not working (#16688)
Follow on this PR: #15484 it break the Agent chat
2026-07-07 12:10:52 +08:00
天海蒼灆
318045dda5 feat(agent): support JSON object input on begin node (#16685)
### Summary

Add object as a begin-node parameter type with JSON editor UI, webhook
schema support, and backend parsing in UserFillUp.

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-07-07 11:40:57 +08:00
euvre
b4540672e4 fix(go): seed built-in agent templates for Go backend (#16666)
### Summary

The Go backend never seeded the `canvas_template` table, so the "Create
agent from template" page was blank when the frontend proxies to the Go
API (`API_PROXY_SCHEME=go`). This PR adds `SeedCanvasTemplates()` in
`internal/dao`, invoked from `InitDB()` after migrations, which loads
`agent/templates/*.json` and mirrors Python's `add_graph_templates()`
behavior.

Changes:
- Add `internal/dao/canvas_template_seed.go` to parse and upsert
built-in templates.
- Call `SeedCanvasTemplates()` in `InitDB()`.
- Add `CanvasTypes` (`JSONSlice`) to `entity.CanvasTemplate` so the
frontend can filter/group by category.
- Skip seeding gracefully when the templates directory is absent.

This fixes the blank template catalogue in Go mode.
2026-07-07 11:25:53 +08:00
Hz_
d03a360fb1 fix(go-agent): add BGPT component and input form (#16684)
## Summary
Adds the missing input form metadata for the Go BGPT canvas component.

## Root Cause

The standalone BGPT component was registered in Go, but it did not
implement GetInputForm(). During component trial run, the backend asks
the component for its input_form. Since BGPT had none, the API returned:

component has no input_form: BGPT:<node_id>

Python BGPT already exposes the query input form, so the Go component
needed the same contract.

## Change

Added GetInputForm() to the Go BGPT component with a single query line
input.
Added test coverage to ensure BGPT exposes the input form.

## Validation

Backend:
bash build.sh --test -run TestBGPT ./internal/agent/component

<img width="1369" height="1184" alt="image"
src="https://github.com/user-attachments/assets/f99e4a81-2359-42e5-80bb-dcc4e6a63fea"
/>

<img width="1736" height="1152" alt="image"
src="https://github.com/user-attachments/assets/c11240a5-2c42-4d08-88e3-c6dfbf49eedb"
/>
2026-07-07 11:15:05 +08:00
weifanglab
7a4e1b6034 refactor: use slices.Contains to simplify code (#16680)
### Summary

There is a [new function](https://pkg.go.dev/slices@go1.21.0#Contains)
added in the go1.21 standard library, which can make the code more
concise and easy to read.

Signed-off-by: weifanglab <weifanglab@outlook.com>
2026-07-07 11:12:38 +08:00
Hz_
863b35db7f fix(go-agent-web): correct BGPT canvas form watcher usage (#16682)
## Summary

Fixes a page crash when opening the BGPT node configuration in the
canvas.

## Root Cause

BGPT was using the tool-form watcher call pattern in a normal canvas
component form.

Tool forms use:

useWatchFormChange(form)

Canvas component forms use:

useWatchFormChange(node?.id, form)

Tool is not equal to component. The BGPT canvas component imported the
component-level hook but called it like a tool-form hook, so the form
argument became undefined and React Hook Form tried to read control from
a null context.

## Change

Updated the BGPT canvas form to pass the node id and form instance
correctly.

## Validation

Ran ESLint for the changed file:
npx eslint src/pages/agent/form/bgpt-form/index.tsx

<img width="1369" height="1184" alt="image"
src="https://github.com/user-attachments/assets/a40c5202-7394-4f26-9da2-08329dcc7fbf"
/>
2026-07-07 11:08:50 +08:00
Shuo Liu
ab5958f518 fix: encapsulate terminal color output and add cross-platform color detection (#16672) 2026-07-07 09:41:26 +08:00
S
f477d3329d Fix: ValueError: too many values to unpack in list_tenant_added_models for model IDs containing '@' (#16467) (#16468) 2026-07-07 09:40:27 +08:00
Rodger Blom
d8cefcf052 feat: add native Dutch language support for BM25 tokenization (#14140)
## Summary
- Add language-aware Snowball stemmer to `RagTokenizer` supporting 16
languages (Dutch, German, French, Spanish, etc.)
- Thread the KB `language` parameter through the full tokenization
pipeline (14 parser modules + task executor)
- Add Dutch to the frontend language lists and cross-language form

## Problem
RAGFlow uses the English Porter stemmer + WordNet lemmatizer for **all**
BM25 tokenization, regardless of the knowledge base language setting.
This produces incorrect stems for non-English text. For example:

| Dutch word | Dutch stemmer | English Porter |
|---|---|---|
| documenten | document | documenten (unchanged!) |
| gebruikers | gebruiker | gebruik (over-stemmed) |
| instellingen | instell | instellingen (unchanged!) |

This degrades BM25 recall for any non-English knowledge base.

## Solution
NLTK already ships Snowball stemmers for 16 languages. This PR:

1. **`rag/nlp/rag_tokenizer.py`**: Overrides `tokenize()` with
`set_language()` and `_normalize_token()` that selects the correct NLTK
Snowball stemmer. Falls back to Porter for unmapped languages (Chinese,
Japanese, Korean, etc. — these use character-based tokenization anyway).
2. **`rag/nlp/__init__.py`** + **14 `rag/app/*.py` parsers** +
**`rag/svr/task_executor.py`**: Threads the `language` parameter through
`tokenize()`, `tokenize_chunks()`, `tokenize_table()`, and all callers.
3. **Frontend**: Adds Dutch (`Nederlands`) to `LanguageList`,
`LanguageMap`, `LanguageAbbreviationMap`, `LanguageTranslationMap`,
cross-language form field, and `en.ts` locale.

## Backward Compatibility
- Default language is `"English"`, preserving existing behavior for all
current users
- Languages without a Snowball stemmer mapping fall back to Porter (no
change)
- No new dependencies — NLTK Snowball is already bundled
2026-07-06 23:39:56 +08:00
Yingfeng
dd20561fca Feat: add event sourcing and replay to harness (#16326)
### Motivation

This PR evolves the harness from a pure execution runtime into an
**observable, replayable agent evaluation platform**. The current
`harness/graph` checkpoint mechanism is insufficient for true
event-sourced introspection—we need append-only event logs capturing
every tool call, state transition, memory write, and approval decision,
enabling deterministic replay, fork/diff, postmortem analysis, and
time-travel debugging.

### Key Design Goals

1. **Event-Sourced Execution Model**  
Replace coarse checkpoints with granular, append-only event logs. Every
operation becomes a durable event: tool invocation, state mutation,
memory update, human approval. This unlocks deterministic replay,
branching execution histories, and regression datasets derived directly
from production failures.

2. **First-Class Replay & Evaluation Loop**  
Replay is not an afterthought—it is a core primitive. A single live run
seeds an offline corpus that supports: repeated playback, model
substitution, tool result mocking, and strategy comparison. The harness
graduates from "executor" to "continuous evaluation platform" where
failed production traces convert directly into offline regression
suites.

3. **Operational Observability**  
   Beyond raw traces, expose metrics that prove stability over time:
   - Tool success / failure rates
   - Approval latency distributions
   - Retry frequencies
   - Checkpoint restore reliability
   - Memory retrieval quality
   - Cost per completed task
   - Fork replay pass rates

The underlying thesis: the bottleneck for most agent systems is not
execution capability, but the inability to **demonstrate continuous,
measurable improvement**.


### Type of change

- [x] New Feature (non-breaking change which adds functionality)
2026-07-06 23:31:54 +08:00
OrbisAI Security
82f3735770 fix: upgrade crawl4ai to 0.9.0 (GHSA-r253-r9jw-qg44) (#16426)
## Summary
Upgrade crawl4ai from 0.8.9 to 0.9.0 to fix GHSA-r253-r9jw-qg44.

## Vulnerability
| Field | Value |
|-------|-------|
| **ID** | GHSA-r253-r9jw-qg44 |
| **Severity** | CRITICAL |
| **Scanner** | trivy |
| **Rule** | `GHSA-r253-r9jw-qg44` |
| **File** | `uv.lock` |
| **Assessment** | Likely exploitable |

**Description**: Crawl4AI: Unauthenticated RCE via Chromium
launch-argument injection in browser_config.extra_args

## Evidence

**Scanner confirmation**: trivy rule `GHSA-r253-r9jw-qg44` flagged this
pattern.

**Production code**: This file is in the production codebase, not
test-only code.

## Threat Model Context

This is a web service - vulnerabilities in request handlers are directly
exploitable by remote attackers.

## Changes
- `pyproject.toml`
- `uv.lock`

## Verification
- [x] Build passes
- [x] Scanner re-scan confirms fix
- [x] LLM code review passed

---
*This change addresses a pattern flagged by static analysis. The code
path handles user-influenced input and the fix reduces the attack
surface against both manual and automated exploitation.*

---
*Automated security fix by [OrbisAI Security](https://orbisappsec.com)*

Co-authored-by: Ling Qin <qinling0210@163.com>
2026-07-06 21:28:19 +08:00
Hernandez Avelino
5a8660df23 [Bug]: Workflow agent completions default stream=True when stream is omitted (#15484)
## Summary

Closes #15483.

Default workflow/session agent completions to non-streaming when
`stream` is omitted.

## Changes

- `api/apps/restful_apis/agent_api.py`: `req.get("stream", False)` on
workflow paths.

## Test plan

- [ ] POST workflow completion without `stream`; assert JSON response.
2026-07-06 21:27:22 +08:00
Mattie Schraeder
8a19c6aa5a Make RAPTOR GMM robust on small reduced clusters (#16632) 2026-07-06 21:09:35 +08:00
Mei Zhihan
85b565244d fix(mcp): handle dict response in list_chats when /chats API returns paginated envelope (#16639) 2026-07-06 21:06:22 +08:00
Mattie Schraeder
0c2fb622e9 Collapse small RAPTOR layers in one step instead of one node per layer (#16633) 2026-07-06 21:06:04 +08:00
OSHA-B
779bf52549 fix: handle missing ES/OpenSearch index in check_embedding (HTTP 500 on empty dataset) (#16650) 2026-07-06 20:30:16 +08:00
Mattie Schraeder
8fb8e4197c fix(graphrag): filter negative-judgment and misattributed relationship edges (#16541) 2026-07-06 20:26:13 +08:00
Zhichang Yu
55af1d70f3 Align Go parser backends and PDF pipeline with Python (#16676)
Ports remaining Go parser wiring and PDF backends, adds tenant-aware VLM
dispatch, aligns post-processing with Python, and adds end-to-end
pipeline coverage with a generated six-page PDF.
2026-07-06 19:50:54 +08:00
euvre
3044283442 fix(go): add missing 'resume' chunk method for new tenants (#16660) 2026-07-06 19:21:14 +08:00
Hz_
8ee3f097b9 fix(go-document): keep upload partial success data as array (#16661)
### Summary

Keep `data` as the uploaded document array when dataset document upload
partially succeeds.

This matches the Python API behavior and allows parse-on-creation to run
for successfully uploaded files when other files in the same folder are
unsupported.
2026-07-06 19:15:29 +08:00
Hz_
7e0ccee7e2 fix(go-agent): missing input form for ExeSQL and Browser agent nodes (#16675)
## Summary
- Add Go dynamic input form support for ExeSQL and Browser components.
- Align their input form metadata with the Python implementation.
- Add regression tests for `/components/:component_id/input-form`.
2026-07-06 19:15:09 +08:00
Hz_
b2e82a42d6 fix(go-agent): Yahoofinance input and run (#16658)
## Summary

Debugging YahooFinance component in agent canvas returns "unknown
component" and "no input_form".
YahooFinance was only registered as an eino tool, not as a runtime
component. The component factory only searches the runtime registry.
- `universe_a_wrappers.go`: add `yahooFinanceComponent` wrapper
delegating to `agenttool.YahooFinanceTool` with `GetInputForm()`
- `fixture_stubs.go`: register `"YahooFinance"` component

## TEST
`go build` and `go test ./internal/agent/component/...` all pass.
2026-07-06 19:14:50 +08:00
Kevin Hu
52f985f43e Refactor: Remove redundant functions. (#16671)
### Summary

Remove redundant functions.
2026-07-06 19:02:25 +08:00
euvre
bfb9641128 fix(go): uploaded documents should be enabled by default (#16674) 2026-07-06 19:01:32 +08:00
Wang Qi
3a247dbb3c Fix filter to use Chinese (#16673) 2026-07-06 18:20:42 +08:00
Jin Hai
b3d536c48e Go: merge functions (#16622)
### Summary

Merge HTTP response functions into common/response.go

---------

Signed-off-by: Jin Hai <haijin.chn@gmail.com>
2026-07-06 18:14:05 +08:00
Wang Qi
ed52255868 Fix Agent session lost <think></think> tag (#16670) 2026-07-06 17:47:38 +08:00
Wang Qi
57625c919a Fix Tag weight should be greater than 0 (#16657) 2026-07-06 16:06:27 +08:00
balibabu
290ab557a5 Fix: Layout of the agent prompt dropdown menu is messed up. (#16653) 2026-07-06 14:48:42 +08:00
Harsh Kashyap
98189cd20a Fix OpenAI response created timestamp (#16401)
## What this fixes

Closes #16400.

`get_data_openai()` currently returns `created: null` when callers do
not pass a timestamp, and it replaces explicit timestamp values with the
current time. This makes non-streaming OpenAI-compatible responses
inconsistent with the expected integer `created` timestamp field.

## Change

- Preserve explicit `created` values when provided.
- Default non-streaming responses to `int(time.time())` when `created`
is not provided.
- Add focused unit coverage for default timestamps, explicit timestamps,
and unchanged streaming chunk shape.

## Verification

- `./.venv/bin/python -m pytest
test/unit_test/api/utils/test_api_utils.py -q`
- `python3 -m py_compile api/utils/api_utils.py
test/unit_test/api/utils/test_api_utils.py`
- `uvx ruff check api/utils/api_utils.py
test/unit_test/api/utils/test_api_utils.py`

---------

Co-authored-by: Harsh Kashyap <harshkashyap@Harshs-MacBook-Pro.local>
2026-07-06 14:16:16 +08:00
Crystora
b633ff0435 feat(go): drop empty and duplicate chunks in postprocess filter (#16049)
### What problem does this PR solve?

The Go chunk pipeline's `PostprocessOperator` `filter` stage
(`internal/ingestion/chunk/postprocess.go`) only filtered by length
(`min_length`/`max_length`). It could not drop empty/whitespace-only
chunks or duplicate chunks — both standard RAG post-processing steps
(blank chunks shouldn't be indexed; identical chunks waste embedding
compute and add redundant retrieval results).

This adds two optional, default-off booleans to the `filter` config:

- `drop_empty` — drop chunks whose content is empty or whitespace-only.
- `drop_duplicates` — drop chunks whose exact content already appeared
(order-preserving; the first occurrence is kept).

They compose with the existing length bounds and are reflected in
`String()` for plan explainability. Also adds the first unit tests for
the postprocess filter (length bounds, drop_empty, drop_duplicates,
combined, exact-content matching, and config parsing).

Validation: `gofmt` clean, `go vet ./internal/ingestion/chunk/` clean,
`go build` ok, `go test ./internal/ingestion/chunk/` — all tests pass.

Closes #16048

### Type of change

- [x] New Feature (non-breaking change which adds functionality)

Co-authored-by: Ling Qin <qinling0210@163.com>
2026-07-06 14:14:31 +08:00
Crystora
4effc242fd feat(go): add length split strategy with overlap to chunk pipeline (#16047)
### What problem does this PR solve?

The Go ingestion chunk pipeline's `SplitOperator`
(`internal/ingestion/chunk/split.go`) supported only `sentence`, `char`,
and `paragraph` strategies, but not **fixed-size (length) chunking with
overlap** — the canonical RAG strategy for bounding chunk length while
preserving cross-boundary context.

This adds a `length` strategy alongside the existing ones, configurable
via DSL `params`:

- `chunk_size` — target window size in **runes** (rune-aware:
multi-byte/CJK text is windowed by character, never split mid-rune).
- `overlap` — runes carried from the end of each window into the next.

The window advances by `chunk_size - overlap`. `chunk_size` falls back
to a default (256) when unset/non-positive, and `overlap` is clamped to
`[0, chunk_size-1]` so the window always advances and the operation
terminates. Implementation follows the existing
`splitByChar`/`splitByParagraph` pattern and reuses `DetectLanguage` for
chunk metadata.

It also adds `split_test.go` — the first unit tests for the `chunk`
package — covering basic windowing, overlap, overlap
clamping/termination, rune-awareness (CJK), default sizing, no-overlap
reconstruction, empty input, and DSL param parsing.

Validation: `gofmt` clean, `go vet ./internal/ingestion/chunk/` clean,
`go build` ok, `go test ./internal/ingestion/chunk/` — all tests pass.

Closes #16046

### Type of change

- [x] New Feature (non-breaking change which adds functionality)

Co-authored-by: bittoby <218712309+bittoby@users.noreply.github.com>
2026-07-06 14:14:21 +08:00
Hz_
c4166a91e0 fix(go-agent): align agent debug input form with Python (#16654)
## Summary

- derive Go Agent debug input forms from prompt variable references
instead of Agent meta fields
- seed `sys.*` debug params into `CanvasState.Sys` so single-component
debug resolves prompt variables like Python
- restore Agent test-run parity for form rendering and debug execution

## Tests

- `go test ./internal/agent/component -run
'TestAgent_(GetInputForm_UsesPromptReferences|GetInputForm_DeduplicatesPromptReferences|Meta_DefaultsToEmpty|Reset_NoTools)$'`
- `go test ./internal/handler -run
'Test(DebugComponent_SeedsSysInputsIntoCanvasState|DebugComponent_HappyPath_Begin|GetComponentInputForm_HappyPath)$'`

AFTER:
<img width="669" height="456" alt="image"
src="https://github.com/user-attachments/assets/4fd86559-aafc-4027-91ae-6e666137ee1b"
/>
2026-07-06 13:29:10 +08:00
chanx
c9f064d5fd fix(metadata): inline value edits not persisted to backend (#16655) 2026-07-06 13:04:04 +08:00
OSHA-B
d607b55c24 fix(nlp): prevent dotted-number cross-references from being classified as headings in Laws chunker (#16626) 2026-07-06 13:02:58 +08:00
Wang Qi
48ef1f4965 Dev: Fix nats host (#16656) 2026-07-06 11:50:37 +08:00
jony376
aaade3530e fix(api): cap memory message limit and top_n at REST_API_MAX_PAGE_SIZE (#15376)
## Related issues

Closes #15375

### What problem does this PR solve?

`GET /api/v1/messages` and `GET /api/v1/messages/search` accepted
unbounded `limit` / `top_n` query parameters while other REST list
endpoints enforce `REST_API_MAX_PAGE_SIZE` (100) via
`validate_rest_api_page_size()`. Oversized values can trigger expensive
memory index queries and large result sets (DoS risk).

### Type of change

- [x] Bug Fix (non-breaking change which fixes an issue)

### Changes

| File | Change |
|------|--------|
| `api/apps/restful_apis/memory_api.py` | Cap `limit` and `top_n` with
`validate_rest_api_page_size`; return argument error when exceeded |
|
`test/testcases/test_web_api/test_message_app/test_message_routes_unit.py`
| Regression tests for oversized `limit` / `top_n` |

### Test plan

- [x] Unit tests added
- [ ] `pytest
test/testcases/test_web_api/test_message_app/test_message_routes_unit.py`

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-07-06 11:20:13 +08:00
Sohaib Ahmed
534a3a7faa fix: Docling parser extracts mathematical formulas (#16645) 2026-07-06 11:17:46 +08:00
qinling0210
3d2f60c34f Port agent PRs to GO - 4 (#16652)
### Summary

Port

https://github.com/infiniflow/ragflow/pull/15399
https://github.com/infiniflow/ragflow/pull/16469
2026-07-06 10:58:40 +08:00
Carlo Beltrame
09eb9dbd21 Switch the default minio image in the helm chart as well (#16322)
Follow-up from #13896
Fixes #13840

### What problem does this PR solve?

In #13896, only the docker-compose-base.yml was adjusted. However, in
the Helm chart, the unmaintained minio/minio image is still referenced.
This PR syncs the Helm chart with the docker compose setup again.

I also added a line to AGENTS.md, so agents should know to do this
automatically in the future.

### Type of change

- [x] Bug Fix (non-breaking change which fixes an issue)
- [ ] New Feature (non-breaking change which adds functionality)
- [x] Documentation Update
- [ ] Refactoring
- [ ] Performance Improvement
- [ ] Other (please describe):

Co-authored-by: Zhichang Yu <yuzhichang@gmail.com>
2026-07-06 10:32:19 +08:00
AI-Mart
fe2f3b60a1 feat(agent): expose thinking mode control per LLM node in Agent canvas (#16640)
## Summary

Add **per-node thinking mode control** for LLM components in RAGFlow
Agent canvas, supporting Qwen3/Qwen3.6/B200M thinking-capable models.
Users can now independently configure thinking mode
(thinking/non-thinking) for each LLM node via the existing UI dropdown.

## Motivation

When Qwen3.6-27B (and other thinking-capable models like Qwen3-32B,
B200M) are used in RAGFlow Agent nodes, different nodes need different
thinking behavior:
- **Reasoning nodes** (complex analysis, math, coding): thinking mode ON
- **Simple nodes** (direct Q&A, intent classification): thinking mode
OFF

The web UI already has a `thinking` dropdown (default/enabled/disabled)
in LLM settings, and the LLM backend `_apply_model_family_policies()`
already supports `enable_thinking`. **The missing link was `gen_conf()`
not forwarding the parameter.**

This 3-line fix completes the chain.

## Changes

**`agent/component/llm.py`** — `LLMParam.gen_conf()`:

```python
if hasattr(self, "thinking") and self.thinking and self.thinking != "default":
    conf["thinking"] = self.thinking
```

## End-to-end flow

```
UI dropdown: default / enabled / disabled
  → DSL: {"thinking": "enabled"}
    → LLMParam.thinking = "enabled"
      → gen_conf() returns {"thinking": "enabled"}
        → _apply_model_family_policies()
          → extra_body {enable_thinking: true}
            → Model API call with thinking ON
```

## Backward compatibility

- **Fully backward compatible** — only 3 lines added, nothing changed
- When `thinking` is "default" or not set, existing behavior is
preserved
- Qwen3 models default to `enable_thinking: false` (non-thinking),
unchanged

## Related issues

- Closes #16321 (thinking content leaks in non-streaming agent API
responses)
- Closes #13957 (how to view model reasoning process in agent API)

## Testing

- Verified in
`test/unit_test/rag/llm/test_chat_model_thinking_policy.py` that
thinking policy already tested for Qwen3 models
- The 3-line change passes through existing tested code path
(`_apply_model_family_policies`)

Co-authored-by: Hermes Agent <hermes-agent@agent.local>
2026-07-06 10:19:27 +08:00
Hz_
358152f758 fix(go-document): add document and file access checks (#16592)
## Summary
Adds ownership/access checks before updating or deleting documents,
setting document metadata, and reading file contents from storage. Also
adds tests for authorized and unauthorized access paths.
2026-07-06 10:13:46 +08:00
Hz_
e5d217993b fix(go-skill): Elasticsearch skill search field mapping (#16611)
## Summary

Fix Elasticsearch-backed skill search by mapping skill search fields to
their indexed token fields.

`name`, `tags`, `description`, and `content` are stored for display but
are not searchable in the skill ES mapping. Search queries now target
`name_tks`, `tags_tks`, `description_tks`, and `content_tks`.

## Testing

- Ran Go unit tests:

```bash
/usr/local/go/bin/go test -count=1 ./internal/engine/elasticsearch
```

- Frontend verification:
    1. Open /files/skills.
    2. Enter a skill space.
3. Reindex the skill space if existing skills were created before this
fix.
    4. Search by skill name or description keyword.
    5. Confirm matching skills are returned.
2026-07-06 10:05:34 +08:00
euvre
0265ffbc53 fix(agent): enable single-component debug for Agent in Go backend (#16606)
### Summary

This PR fixes two issues that prevented the Agent component's
single-component debug/test run from working under the Go backend:

1. **Dynamic input_form generation**: Some components (e.g. `Agent`) do
not store a static `input_form` in the DSL. The Go handler now falls
back to the runtime component's `GetInputForm()` method, matching
Python's `Canvas.get_component_input_form` behavior. This resolves the
frontend 102 error: `component has no input_form`.

2. **Tenant ID injection for debug**: Single-component debug runs use a
fresh `CanvasState` that previously lacked `tenant_id`.
`AgentComponent.Invoke` resolves LLM credentials via the tenant tables,
so the debug run failed with `api key is required`. The handler now
seeds `state.Sys["tenant_id"]` with the authenticated user's ID,
mirroring Python's `@add_tenant_id_to_kwargs` decorator.

### Changes

- `internal/handler/agent_component.go`:
- Added `componentInputForm` helper that first reads the static
`input_form` and, if missing, instantiates the component and calls
`GetInputForm()`.
- In `DebugComponent`, set `debugState.Sys["tenant_id"] = user.ID`
before invoking the component.
2026-07-06 09:57:00 +08:00
euvre
8b065d3ddd fix(agent): collect CodeExec artifacts from ReAct tool responses (#16609)
### Summary

The Go backend Agent component was not returning artifacts produced by
the CodeExec tool. While the Python agent collects the "`_ARTIFACTS`"
envelope from tool responses and appends artifact markdown to the final
content, the Go agent only returned the assistant text, so generated
images were missing from the chat output.

### Changes

- Wire `react.WithMessageFuture()` in `runEinoReActAgent` and store the
resulting `MessageFuture` in the invocation context.
- After the ReAct loop finishes, drain the future and extract
``_ARTIFACTS`` entries from every tool response message.
- Support reading the tool payload from both `msg.Content` and
`msg.UserInputMultiContent` to match eino's tool contract.
- De-duplicate artifacts by URL and render images as `!` and other files
as download links.
- Add `agent_artifact_test.go` with a regression test that simulates a
CodeExec-style tool response carrying an image artifact and verifies it
is collected and formatted.

### Verification

- `go test ./internal/agent/component/... -run
TestAgent_ReActAgent_CollectsArtifactsFromCodeExecTool` passes.
- `go test ./internal/agent/component/... -count=1` compiles; the only
failure is an unrelated DNS-pinning timeout test
(`TestInvoke_ProxyDNSPin`).
- `gofmt` clean for modified files.

### Related

Fixes the behavior shown in the screenshot where the Go agent ignored
the CodeExec-generated PNG artifact.
2026-07-05 20:53:43 +08:00
Jack
1d3c100acb Refactor: pdf parser (#16625)
### Summary

PDF parser refactor
2026-07-05 20:45:35 +08:00
Zhichang Yu
014c3f634f Align Go ingestion boundaries with Python (#16647)
Moves doc_id blob resolution into Parser, tightens chunker/tokenizer to
Python output_format semantics, updates extractor list handling, and
fixes real-template integration tests.
2026-07-05 20:43:52 +08:00
Mattie Schraeder
0fcfb38365 Cap RAPTOR UMAP n_neighbors to prevent OOM on large datasets (#16627)
## Problem
`raptor.py` computes `n_neighbors = int((len(embeddings) - 1) ** 0.8)`
and
passes it to `umap.UMAP(...)`. In a dataset-scope RAPTOR build the first
layer's `embeddings` is the entire KB's chunk set, so this is
effectively
unbounded: ~93k chunks → n_neighbors ≈ 9,446.

UMAP's k-NN graph is `N × n_neighbors`; at these values the raw neighbor
arrays alone are ~14 GB (93k × 9446 × 16 B), and the symmetrized fuzzy
simplicial set + spectral init push peak well past 30 GB. The task
executor is OOM-killed inside `fit_transform` before any clustering runs
—
the log shows "Task has been received" with no "Cluster one layer" line
—
after which the unacked task re-queues and OOMs again in a loop.

The line above already flags this: `# Degrade too much ??`.

## Fix
Cap `n_neighbors` at 100. UMAP's neighborhood size has strongly
diminishing returns well below this (default 15; a few dozen already
captures global structure), so the ceiling preserves — likely improves —
cluster quality while bounding memory to O(N). Mirrors the existing
`n_components=min(12, len(embeddings) - 2)` clamp two lines down.

​```diff
-        n_neighbors = int((len(embeddings) - 1) ** 0.8)
+        n_neighbors = min(int((len(embeddings) - 1) ** 0.8), 100)
​```

## Repro
Dataset-scope RAPTOR over a KB with ~90k+ chunks on a box with <~64 GB
available: executor OOM-killed in the first-layer UMAP `fit_transform`.
With the cap, first-layer UMAP peaks in low single-digit GB and the
build
proceeds to completion.

## Scope
Only affects large dataset-scope builds; file-scope RAPTOR already had
n_neighbors well under 100. No behavior change beyond the ceiling.
2026-07-04 17:47:43 +08:00
Wang Qi
a0e65637eb Delete canvas_app.py and evaluation_service.py (#16614)
Follow on PR #13295
2026-07-03 21:03:54 +08:00
Kevin Hu
cf634b92b4 Feat: Put some wiki templates. (#16617)
### Summary

Add a few of wiki templates.
2026-07-03 20:52:27 +08:00
Wang Qi
06aa169df7 Update development script (#16623) 2026-07-03 20:34:30 +08:00
Liu An
63a4ed55d8 docs: update Docker build instructions for deps image (#16620)
### Summary

update Docker build instructions for deps image
2026-07-03 19:57:12 +08:00
monsterDavid
7da4f200e5 fix(agent): enable MCP file preview via doc_id (#15399)
## Summary
MCP-wrapped agents could only force-download files looked up by
`doc_id`. This adds an explicit preview path and inline response headers
for previewable file types.

- **New** `GET /api/v1/agents/attachments/{attachment_id}/preview` —
inline preview for PDFs, images, and other safe types (pass `ext` and/or
`mime_type`)
- **Improved** `GET /api/v1/documents/{doc_id}/preview` — sets inline
disposition using the document filename
- **Improved** attachment download routing — resolves `mime_type` /
`ext` query params (no default `markdown`), supports
`disposition=inline`
- **DocGenerator output** — includes URL-encoded `preview_url` for MCP
clients
- **Legacy `/document/download/...` aliases** — still use download
semantics; MCP clients should call `/preview` explicitly

Fixes #15398

## Test plan
- [x] `pytest test/unit_test/api/utils/test_file_response_headers.py`
(6/6)

---------

Co-authored-by: MkDev11 <mkdev11@users.noreply.github.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Ling Qin <qinling0210@163.com>
2026-07-03 19:56:01 +08:00
maoyifeng
0f4f2135f3 Go:cli move _order _columns sort group (#16615)
### Summary
1. Move common functions to format.go
2. modify show name spaces to _
3. move _order _columns column sort group;
4. add dao empty enterprise file
2026-07-03 19:37:53 +08:00
Jin Hai
6b571694df Go: Update error info (#16619)
As title.

Signed-off-by: Jin Hai <haijin.chn@gmail.com>
2026-07-03 19:37:25 +08:00
S
1861087787 fix(agent): defend against @ in var names at all template-split sites (#16469)
## Summary

While fixing #16467 (IterationItem crash on `@` in user-defined output
keys), an audit of `agent/**/*.py` revealed **three additional sites**
with the same vulnerability. This PR hardens all of them with
`maxsplit=1` and adds regression tests.

This is **defense-in-depth hardening**, not a behavior change. The
current `variable_ref_patt` regex constrains `var_nm` to
`[A-Za-z0-9_.-]+`, so single-`@` templates resolve exactly as before.
The `maxsplit=1` only kicks in if the trailing side itself contains `@`
— currently unreachable from the public DSL surface, but trivially
exploitable the moment a user-defined output key happens to contain `@`
(e.g. `user@email`) or the regex is ever relaxed.

> **Note on issue scope**: The primary fix for #16467 (the
`list_tenant_added_models` `ValueError` crash on `@` in model names) is
in PR #16468. This PR is a **follow-up hardening sweep** of the same
vulnerability class found in `agent/` during that audit; it does not
duplicate or replace #16468.

## Sites hardened

| File | Line | Method |
|------|------|--------|
| `agent/canvas.py` | 206 | `Graph.get_variable_value` |
| `agent/canvas.py` | 256 | `Graph.set_variable_value` |
| `agent/component/base.py` | 533 |
`ComponentBase.get_input_elements_from_text` |
| `agent/component/iterationitem.py` | 88 |
`IterationItem.output_collation` |

All now use `split("@", 1)` with an inline comment explaining the
rationale. The trailing side keeps any embedded `@`.

## Sites already safe (audited but left alone)

| File | Reason safe |
|------|------------|
| `agent/canvas.py:708` (`is_reff`) | Pre-checks `len(arr) != 2` |
| `agent/component/categorize.py` | Uses `rsplit` |
| `agent/component/iteration.py` | Pre-validates via regex |
| Other call sites | `rsplit` or regex pre-validation |

## Regression tests

9 new tests across 2 files, all `pytest.mark.p2`:

| File | Tests |
|------|-------|
| `test/unit_test/agent/test_canvas_at_split.py` | 6 —
`get_variable_value`, `set_variable_value`, round-trip, single-`@`,
missing-component |
| `test/unit_test/agent/component/test_iterationitem_at_split.py` | 3 —
`output_collation` with `@` in var, single-`@`, non-matching cid |

Each test was **verified to fail with `ValueError: too many values to
unpack (expected 2)`** when the corresponding fix is temporarily
reverted, confirming the tests actually catch the bug rather than just
exercising the happy path.

## Test results

```
9 passed in 0.04s
```

Full agent unit suite also clean (38 passed, 3 skipped; 6 unrelated
pre-existing collection errors from missing `peewee`/`requests` in local
venv — not caused by this PR).

## Related

- Issue: #16467
- Primary fix PR: #16468 (closes the issue)
- This PR: defense-in-depth follow-up, intentionally non-blocking on
#16467

---------

Co-authored-by: skbs-eng <skbs-eng@users.noreply.github.com>
2026-07-03 19:26:27 +08:00
Haruko386
fd7fb6669a fix: cannot get query in agent-log (#16610)
### Summary

As title

bug:


fixed:
<img width="1827" height="1286" alt="image"
src="https://github.com/user-attachments/assets/0cdc391c-43d7-4330-bc34-3aefe5d4f4ee"
/>
2026-07-03 18:56:32 +08:00
Jin Hai
83d09b16ce Fix Go: list providers order issue. (#16616)
### Summary

As title.

---------

Signed-off-by: Jin Hai <haijin.chn@gmail.com>
2026-07-03 18:27:32 +08:00
Haruko386
dde8b6d54c fix: get team's search in own search-list (#16599)
### Summary

As title:
2026-07-03 18:26:03 +08:00
Haruko386
226d0ff77c fix: get team merber's chat (#16597)
### Summary

As title
2026-07-03 18:25:31 +08:00
Haruko386
488574fd80 fix: get all memory in team with permission=me (#16593)
### Summary

As title:
2026-07-03 18:25:04 +08:00
Yingfeng
706fa4e87a Feat: add gbrain compile template for session/memory data (#16613) 2026-07-03 18:22:29 +08:00
qinling0210
ffc4d29a06 Port agent PRs to GO - 3 (#16596)
### Summary

Port
https://github.com/infiniflow/ragflow/pull/16415
https://github.com/infiniflow/ragflow/pull/16417
2026-07-03 18:03:23 +08:00
Yingfeng
8db68e3eec Refactor(harness): remove naive inline graph engine , unify graph execution under single pregel engine (#16608) 2026-07-03 17:50:30 +08:00
Muhammad Furqan
3cba34d67f fix(agent/tools): port Crawler to ToolBase so it can load and run (#16415)
### What problem does this PR solve?

Closes #16414.

The **Crawler** agent tool (`agent/tools/crawler.py`) was never ported
to the modern `ToolBase`/`_invoke` interface during the agent module
redesign, so it was broken in three independent ways:

1. **Crashed on construction.** `CrawlerParam` extends `ToolParamBase`,
whose `__init__` reads `self.meta["parameters"]`, but `CrawlerParam`
defined no `meta`. Constructing it raised `AttributeError:
'CrawlerParam' object has no attribute 'meta'`. Because
`agent/canvas.py` instantiates `component_class(component_name +
"Param")()` while loading a canvas, **any agent containing a Crawler
node failed to load.**
2. **`_invoke` missing.** It extends `ToolBase` (whose `invoke()`
dispatches to `self._invoke`) but only implemented the legacy `_run`, so
`_invoke` resolved to `ComponentBase._invoke` → `NotImplementedError`.
3. **`be_output` removed.** `_run` called `Crawler.be_output(...)`,
which no longer exists on the base classes.

### Type of change

- [x] Bug Fix (non-breaking change which fixes an issue)

### Changes

- Add a `ToolMeta` to `CrawlerParam` (defined before
`super().__init__()`, matching every other ported tool such as
`ArXivParam`/`TavilyExtractParam`) advertising a required `query`
parameter — the URL to crawl, default `{sys.query}`, consistent with the
`{sys.query}` convention shared by the other tools.
- Replace the legacy `_run`/`be_output` with `_invoke`/`set_output`,
writing the extracted page content to `formalized_content` (errors
surfaced via `_ERROR`), consistent with the other tools.
- Preserve the existing SSRF guard (`assert_url_is_safe` +
`pin_dns_global`).
- Add regression tests
(`test/unit_test/agent/component/test_crawler.py`) covering param
construction, validation, and the tool descriptor.

Same class of defect as #16329 (DeepL). Backend-only; no frontend
changes.

---------

Co-authored-by: Zhichang Yu <yuzhichang@gmail.com>
2026-07-03 17:15:48 +08:00
Jin Hai
1880e65e99 Go: refactor (#16602)
### Summary

1. update doc
2. refactor route code

---------

Signed-off-by: Jin Hai <haijin.chn@gmail.com>
2026-07-03 17:00:43 +08:00
chanx
79518973e5 Fix: optimize folder data handling in MoveDialog component (#16580) 2026-07-03 16:13:57 +08:00
euvre
4effd057f0 i18n: localize visual input file label in agent form (#16594) 2026-07-03 15:31:27 +08:00
Jin Hai
a4c370c5ba Go: fix 'list services' (#16598)
### Summary

```
RAGFlow(admin)> list services;
+-----------------------------------------------------------------------------+-----------+----+---------------+------+---------------+-----------+
| extra                                                                       | host      | id | name          | port | service_type  | status    |
+-----------------------------------------------------------------------------+-----------+----+---------------+------+---------------+-----------+
| map[database:1 mq_type:redis password:infini_rag_flow]                      | localhost | 0  | redis         | 6379 | message_queue | alive     |
| map[password:infini_rag_flow retrieval_type:elasticsearch username:elastic] | localhost | 1  | elasticsearch | 1200 | retrieval     | alive     |
|                                                                             | 0.0.0.0   | 2  | nats          | 4222 | message_queue | CONNECTED |
| map[meta_type:mysql password:infini_rag_flow username:root]                 | localhost | 3  | mysql         | 3306 | meta_data     | alive     |
| map[password:infini_rag_flow store_type:minio user:rag_flow]                | localhost | 4  | minio         | 9000 | file_store    | alive     |
+-----------------------------------------------------------------------------+-----------+----+---------------+------+---------------+-----------+

```

Signed-off-by: Jin Hai <haijin.chn@gmail.com>
2026-07-03 15:08:37 +08:00
euvre
994b603374 fix: prevent duplicate MCP server name when editing (#16588) 2026-07-03 14:30:43 +08:00
euvre
7b341539e7 fix: prevent exporting empty MCP server selection (#16589) 2026-07-03 14:22:17 +08:00
Hz_
ac5d0c4615 fix(go-file): KB counter drift when deleting files with linked documents (#16584)
## Summary

Use `DocumentService.RemoveDocumentKeepFile` when deleting files that
are linked to documents.

  ## Change

  - inject `DocumentService` into `FileService`
  - replace direct document deletion in `deleteSingleFile`
  - remove the obsolete file-local engine deletion helper

  ## Result

Deleting a file now cleans up linked documents through the same service
path used elsewhere, keeping KB counters and document engine cleanup
consistent.
2026-07-03 14:07:54 +08:00
Haruko386
ee942711c4 fix: unable to fetch tools for MCP (#16583) 2026-07-03 14:05:42 +08:00
Haruko386
b2e4740acd fix: unable to import mcp from local (#16590)
### Summary

As title
2026-07-03 14:05:07 +08:00
Haruko386
383d059969 fix: agent chat completions can not use (#16570)
### Summary

As title
<img width="2370" height="2039" alt="image"
src="https://github.com/user-attachments/assets/4cccf543-3908-49ee-8101-c5068fbf53ec"
/>
2026-07-03 13:25:14 +08:00
euvre
e65bac238e fix: preserve existing links when bulk linking files to knowledge bases (#16587) 2026-07-03 13:17:19 +08:00
Wang Qi
6a4b9be426 Refactor: reformat all code for lefthook using ruff and gofmt (#16585) 2026-07-03 12:53:39 +08:00
Yingfeng
19fcb4a981 Fix harness DAG slow-branch test cased by nil initialization of pregel engine (#16591) 2026-07-03 12:53:25 +08:00
euvre
918229613a fix: prevent duplicate 'skills' and '.knowledgebase' folders caused by race conditions (#16568) 2026-07-03 12:06:45 +08:00
Muhammad Furqan
83540185e1 fix(agent/tools): port AkShare to ToolBase so it works as an Agent tool (#16417)
### What problem does this PR solve?

Closes #16416.

The **AkShare** agent tool (`agent/tools/akshare.py`) was never ported
to the modern `ToolBase`/`_invoke` interface during the agent module
redesign and was still written against the removed legacy
`_run`/`be_output` API, so it was non-functional:

1. **Adding it to an Agent raised `AttributeError`.** `AkShare` extended
`ComponentBase` (not `ToolBase`) and `AkShareParam` defined no `meta`,
so it had no `get_meta()`. `agent/component/agent_with_tools.py` builds
each tool's function descriptor via `cpn.get_meta()`, so constructing an
Agent that includes the AkShare tool raised `AttributeError: 'AkShare'
object has no attribute 'get_meta'`.
2. **It could never run.** `invoke()` dispatches to `self._invoke`, but
`AkShare` only implemented the legacy `_run`, so `_invoke` fell through
to `ComponentBase._invoke` → `NotImplementedError`. `_run` also called
`be_output(...)`, which no longer exists on the base classes.

### Type of change

- [x] Bug Fix (non-breaking change which fixes an issue)

### Changes

- Port `AkShareParam` to `ToolParamBase` with a `ToolMeta` (defined
before `super().__init__()`, matching `ArXivParam`/`TavilyExtractParam`)
exposing a required `query` parameter — the stock symbol to look up,
default `{sys.query}`. `query` matches the `{sys.query}` convention
shared by the other tools.
- Rewrite the component with `_invoke`/`set_output("formalized_content",
...)` (errors surfaced via `_ERROR`), keeping `top_n` and importing
`akshare` lazily.
- Add regression tests
(`test/unit_test/agent/component/test_akshare.py`) covering param
construction, validation, and the tool descriptor.

Same class of defect as #16329 (DeepL) and #16414 (Crawler).
Backend-only; no frontend changes.

---------

Co-authored-by: Zhichang Yu <yuzhichang@gmail.com>
2026-07-03 11:39:26 +08:00
Jin Hai
1aa8abe373 Go: file syncer service framework (#16579)
### Summary

./ragflow_main --syncer to start file syncer


config yaml file has following config
```
file_syncer:
  max_concurrent_syncs: 4 # concurrent file sync threads
  sync_interval: 3 # check interval

```

---------

Signed-off-by: Jin Hai <haijin.chn@gmail.com>
2026-07-03 11:14:02 +08:00
Kevin Hu
62f94cd59b Feat: Add knowledge compilation workflows (#16515)
## Summary
- Add knowledge compilation template APIs, services, and builtin
template seed data
- Add advanced knowledge compile structure/artifact/RAPTOR workflow
support
- Update parsing, dataset/document APIs, and supporting services for
compilation workflows
2026-07-02 23:22:07 +08:00
Jin Hai
7d64a78f83 Go: unify three services into one binary (#16462)
### Summary

Plan to start api_server, admin_server and ingestor in one binary:
- ./ragflow_main --admin
- ./ragflow_main --api
- ./ragflow_main --ingestor

---------

Signed-off-by: Jin Hai <haijin.chn@gmail.com>
2026-07-02 21:21:10 +08:00
1978 changed files with 137747 additions and 52070 deletions

View File

@@ -27,9 +27,9 @@ docker/oceanbase/
docker/seekdb/
# ── Go and C++ build outputs ────────────────────────────────────────────────
internal/cpp/build/
internal/cpp/cmake-build-release/
internal/cpp/cmake-build-debug/
internal/binding/cpp/build/
internal/binding/cpp/cmake-build-release/
internal/binding/cpp/cmake-build-debug/
target/
# ── ragflow_deps build context (built as a separate image, mounted ──

View File

@@ -94,53 +94,28 @@ jobs:
- goos: linux
goarch: amd64
runner: ubuntu-24.04
native_asset: native-linux-x86_64.tar.gz
- goos: linux
goarch: arm64
runner: ubuntu-24.04-arm
native_asset: native-linux-aarch64.tar.gz
- goos: darwin
goarch: amd64
runner: macos-15-intel
native_asset: native-macos-x86_64.tar.gz
- goos: darwin
goarch: arm64
runner: macos-14
native_asset: native-macos-aarch64.tar.gz
- goos: windows
goarch: amd64
runner: windows-latest
native_asset: native-windows-x86_64.zip
msystem: CLANG64
rust_target: x86_64-pc-windows-gnullvm
msys2_packages: >-
mingw-w64-clang-x86_64-clang
mingw-w64-clang-x86_64-lld
mingw-w64-clang-x86_64-cmake
mingw-w64-clang-x86_64-ninja
mingw-w64-clang-x86_64-pcre2
mingw-w64-clang-x86_64-pkgconf
output_ext: .exe
- goos: windows
goarch: arm64
runner: windows-11-arm
native_asset: native-windows-aarch64.zip
msystem: CLANGARM64
rust_target: aarch64-pc-windows-gnullvm
msys2_packages: >-
mingw-w64-clang-aarch64-clang
mingw-w64-clang-aarch64-lld
mingw-w64-clang-aarch64-cmake
mingw-w64-clang-aarch64-ninja
mingw-w64-clang-aarch64-pcre2
mingw-w64-clang-aarch64-pkgconf
output_ext: .exe
runs-on: ${{ matrix.runner }}
env:
CLI_NAME: ragflow-cli
CLI_MAIN: ./cmd/ragflow-cli.go
DIST_DIR: dist/cli
OFFICE_OXIDE_VERSION: "0.1.2"
RELEASE_TAG: ${{ needs.prepare.outputs.release_tag }}
steps:
@@ -159,249 +134,7 @@ jobs:
go-version-file: go.mod
cache: true
- name: Install Unix native build dependencies
if: runner.os != 'Windows'
shell: bash
run: |
set -euo pipefail
if [[ "${{ matrix.goos }}" == "linux" ]]; then
sudo apt-get update
sudo apt-get install -y build-essential libpcre2-dev libsimde-dev pkg-config python3-pip
elif [[ "${{ matrix.goos }}" == "darwin" ]]; then
brew list pcre2 >/dev/null 2>&1 || brew install pcre2
brew list simde >/dev/null 2>&1 || brew install simde
fi
python3 -m pip install --user --upgrade 'cmake>=4.0,<5' || \
python3 -m pip install --user --break-system-packages --upgrade 'cmake>=4.0,<5'
python_user_base="$(python3 -m site --user-base)"
echo "${python_user_base}/bin" >> "${GITHUB_PATH}"
export PATH="${python_user_base}/bin:${PATH}"
cmake --version
- name: Download office_oxide native library
if: runner.os != 'Windows'
shell: bash
run: |
set -euo pipefail
OFFICE_OXIDE_PREFIX="${RUNNER_TEMP}/office_oxide"
OFFICE_OXIDE_URL="https://github.com/yfedoseev/office_oxide/releases/download/v${OFFICE_OXIDE_VERSION}/${{ matrix.native_asset }}"
mkdir -p "${OFFICE_OXIDE_PREFIX}"
curl -fsSL "${OFFICE_OXIDE_URL}" -o "${RUNNER_TEMP}/${{ matrix.native_asset }}"
tar xzf "${RUNNER_TEMP}/${{ matrix.native_asset }}" -C "${OFFICE_OXIDE_PREFIX}"
test -f "${OFFICE_OXIDE_PREFIX}/include/office_oxide_c/office_oxide.h"
test -f "${OFFICE_OXIDE_PREFIX}/lib/liboffice_oxide.a"
ln -sf "office_oxide_c/office_oxide.h" "${OFFICE_OXIDE_PREFIX}/include/office_oxide.h"
test -f "${OFFICE_OXIDE_PREFIX}/include/office_oxide.h"
echo "OFFICE_OXIDE_PREFIX=${OFFICE_OXIDE_PREFIX}" >> "${GITHUB_ENV}"
- name: Set up MSYS2
if: runner.os == 'Windows'
uses: msys2/setup-msys2@v2
with:
msystem: ${{ matrix.msystem }}
update: true
install: ${{ matrix.msys2_packages }}
path-type: inherit
- name: Install Windows SIMDe headers
if: runner.os == 'Windows'
shell: msys2 {0}
run: |
set -euo pipefail
simde_dir="$(cygpath -u "${RUNNER_TEMP}")/simde"
simde_archive="$(cygpath -u "${RUNNER_TEMP}")/simde-v0.8.2.tar.gz"
github_env="$(cygpath -u "${GITHUB_ENV}")"
rm -rf "${simde_dir}"
mkdir -p "${simde_dir}"
curl -fsSL "https://github.com/simd-everywhere/simde/archive/refs/tags/v0.8.2.tar.gz" -o "${simde_archive}"
tar xzf "${simde_archive}" -C "${simde_dir}" --strip-components=1
test -f "${simde_dir}/simde/x86/sse4.1.h"
# Install SIMDe headers into the MSYS2 toolchain include directory.
# CMake/Ninja invokes clang-scan-deps as a Windows-native executable;
# a POSIX-style include path like /c/a/_temp/simde may not be resolved
# there, so keep the headers under ${MINGW_PREFIX}/include instead.
rm -rf "${MINGW_PREFIX}/include/simde"
cp -R "${simde_dir}/simde" "${MINGW_PREFIX}/include/simde"
test -f "${MINGW_PREFIX}/include/simde/x86/sse4.1.h"
- name: Configure Windows C compiler
if: runner.os == 'Windows'
shell: msys2 {0}
run: |
set -euo pipefail
cc_path="$(command -v clang.exe 2>/dev/null || command -v clang)"
cxx_path="$(command -v clang++.exe 2>/dev/null || command -v clang++)"
if [[ ! -f "${cc_path}" && -f "${cc_path}.exe" ]]; then
cc_path="${cc_path}.exe"
fi
if [[ ! -f "${cxx_path}" && -f "${cxx_path}.exe" ]]; then
cxx_path="${cxx_path}.exe"
fi
test -f "${cc_path}"
test -f "${cxx_path}"
cc="$(cygpath -m "${cc_path}")"
cxx="$(cygpath -m "${cxx_path}")"
github_env="$(cygpath -u "${GITHUB_ENV}")"
pcre2_libdir="$(pkg-config --variable=libdir libpcre2-8)"
pcre2_includedir="$(pkg-config --variable=includedir libpcre2-8)"
echo "CC=${cc}" >> "${github_env}"
echo "CXX=${cxx}" >> "${github_env}"
echo "PCRE2_LIBDIR=$(cygpath -m "${pcre2_libdir}")" >> "${github_env}"
echo "PCRE2_INCLUDEDIR=$(cygpath -m "${pcre2_includedir}")" >> "${github_env}"
echo "Resolved MSYS2 clang: ${cc}"
"${cc_path}" --version
echo "Resolved MSYS2 clang++: ${cxx}"
"${cxx_path}" --version
- name: Set up Rust for Windows office_oxide staticlib
if: runner.os == 'Windows'
uses: dtolnay/rust-toolchain@stable
with:
targets: ${{ matrix.rust_target }}
- name: Build office_oxide native library for Windows GNU ABI
if: runner.os == 'Windows'
shell: msys2 {0}
run: |
set -euo pipefail
office_oxide_prefix="$(cygpath -u "${RUNNER_TEMP}")/office_oxide"
office_oxide_src="$(cygpath -u "${RUNNER_TEMP}")/office_oxide-src"
archive_path="$(cygpath -u "${RUNNER_TEMP}")/office_oxide-v${OFFICE_OXIDE_VERSION}.tar.gz"
github_env="$(cygpath -u "${GITHUB_ENV}")"
rm -rf "${office_oxide_prefix}" "${office_oxide_src}"
mkdir -p "${office_oxide_prefix}/include/office_oxide_c" "${office_oxide_prefix}/lib" "${office_oxide_src}"
curl -fsSL "https://github.com/yfedoseev/office_oxide/archive/refs/tags/v${OFFICE_OXIDE_VERSION}.tar.gz" -o "${archive_path}"
tar xzf "${archive_path}" -C "${office_oxide_src}" --strip-components=1
cd "${office_oxide_src}"
cc_path="$(command -v clang.exe 2>/dev/null || command -v clang)"
cxx_path="$(command -v clang++.exe 2>/dev/null || command -v clang++)"
if [[ ! -f "${cc_path}" && -f "${cc_path}.exe" ]]; then
cc_path="${cc_path}.exe"
fi
if [[ ! -f "${cxx_path}" && -f "${cxx_path}.exe" ]]; then
cxx_path="${cxx_path}.exe"
fi
test -f "${cc_path}"
test -f "${cxx_path}"
export CC="${cc_path}"
export CXX="${cxx_path}"
export AR="$(command -v llvm-ar || command -v ar)"
export CARGO_BUILD_TARGET="${{ matrix.rust_target }}"
export RUSTFLAGS="-C target-feature=+crt-static -C link-arg=-fuse-ld=lld"
case "${{ matrix.rust_target }}" in
x86_64-pc-windows-gnullvm)
export CARGO_TARGET_X86_64_PC_WINDOWS_GNULLVM_LINKER="${CC}"
export CARGO_TARGET_X86_64_PC_WINDOWS_GNULLVM_AR="${AR}"
;;
aarch64-pc-windows-gnullvm)
export CARGO_TARGET_AARCH64_PC_WINDOWS_GNULLVM_LINKER="${CC}"
export CARGO_TARGET_AARCH64_PC_WINDOWS_GNULLVM_AR="${AR}"
;;
*)
echo "Unsupported Rust target: ${{ matrix.rust_target }}"
exit 1
;;
esac
# The release workflow only needs the static archive for cgo.
# Building cdylib on Windows pulls in extra MinGW runtime libraries,
# which can fail under the CLANG64/CLANGARM64 environments.
perl -0pi -e 's/crate-type\s*=\s*\[[^\]]+\]/crate-type = ["staticlib"]/s' Cargo.toml
cargo build --release --lib --target "${CARGO_BUILD_TARGET}" --no-default-features
cp "include/office_oxide_c/office_oxide.h" "${office_oxide_prefix}/include/office_oxide_c/office_oxide.h"
cp "include/office_oxide_c/office_oxide.h" "${office_oxide_prefix}/include/office_oxide.h"
cp "target/${CARGO_BUILD_TARGET}/release/liboffice_oxide.a" "${office_oxide_prefix}/lib/liboffice_oxide.a"
test -f "${office_oxide_prefix}/include/office_oxide_c/office_oxide.h"
test -f "${office_oxide_prefix}/include/office_oxide.h"
test -f "${office_oxide_prefix}/lib/liboffice_oxide.a"
echo "OFFICE_OXIDE_PREFIX=$(cygpath -m "${office_oxide_prefix}")" >> "${github_env}"
- name: Build rag tokenizer native library
if: runner.os != 'Windows'
shell: bash
run: |
set -euo pipefail
cmake_args=(
-S internal/cpp
-B internal/cpp/cmake-build-release
-DCMAKE_BUILD_TYPE=Release
)
if [[ "${{ matrix.goos }}" == "darwin" ]]; then
simde_prefix="$(brew --prefix simde)"
pcre2_prefix="$(brew --prefix pcre2)"
cmake_args+=(
-DCMAKE_PREFIX_PATH="${pcre2_prefix};${simde_prefix}"
-DCMAKE_C_FLAGS="-I${simde_prefix}/include"
-DCMAKE_CXX_FLAGS="-I${simde_prefix}/include"
)
fi
cmake "${cmake_args[@]}"
cmake --build internal/cpp/cmake-build-release --target rag_tokenizer_c_api --parallel
test -f internal/cpp/cmake-build-release/librag_tokenizer_c_api.a
- name: Build rag tokenizer native library
if: runner.os == 'Windows'
shell: msys2 {0}
run: |
set -euo pipefail
cc_path="$(command -v clang.exe 2>/dev/null || command -v clang)"
cxx_path="$(command -v clang++.exe 2>/dev/null || command -v clang++)"
if [[ ! -f "${cc_path}" && -f "${cc_path}.exe" ]]; then
cc_path="${cc_path}.exe"
fi
if [[ ! -f "${cxx_path}" && -f "${cxx_path}.exe" ]]; then
cxx_path="${cxx_path}.exe"
fi
test -f "${cc_path}"
test -f "${cxx_path}"
test -f "${MINGW_PREFIX}/include/simde/x86/sse4.1.h"
cmake -S internal/cpp -B internal/cpp/cmake-build-release -G Ninja \
-DCMAKE_BUILD_TYPE=Release \
-DCMAKE_C_COMPILER="$(cygpath -m "${cc_path}")" \
-DCMAKE_CXX_COMPILER="$(cygpath -m "${cxx_path}")" \
-DCMAKE_C_FLAGS="-I${MINGW_PREFIX}/include" \
-DCMAKE_CXX_FLAGS="-I${MINGW_PREFIX}/include"
cmake --build internal/cpp/cmake-build-release --target rag_tokenizer_c_api --parallel
test -f internal/cpp/cmake-build-release/librag_tokenizer_c_api.a
- name: Build Go CLI release binaries
- name: Build Go CLI release binaries on non-Windows
if: runner.os != 'Windows'
shell: bash
run: |
@@ -424,27 +157,7 @@ jobs:
output="${DIST_DIR}/${CLI_NAME}-${RELEASE_TAG}-${{ matrix.goos }}-${{ matrix.goarch }}"
echo "Building ${{ matrix.goos }}/${{ matrix.goarch }} -> ${output}"
case "${{ matrix.goos }}" in
linux)
cgo_ldflags="${OFFICE_OXIDE_PREFIX}/lib/liboffice_oxide.a -lm -ldl -lpthread"
;;
darwin)
cgo_ldflags="${OFFICE_OXIDE_PREFIX}/lib/liboffice_oxide.a"
;;
*)
echo "::error::Unsupported Unix target: ${{ matrix.goos }}"
exit 1
;;
esac
cgo_cflags="-I${OFFICE_OXIDE_PREFIX}/include -I${OFFICE_OXIDE_PREFIX}/include/office_oxide_c"
echo "office_oxide prefix: ${OFFICE_OXIDE_PREFIX}"
echo "CGO_CFLAGS: ${cgo_cflags}"
echo "CGO_LDFLAGS: ${cgo_ldflags}"
CGO_ENABLED=1 \
CGO_CFLAGS="${cgo_cflags}" \
CGO_LDFLAGS="${cgo_ldflags}" \
CGO_ENABLED=0 \
GOOS="${{ matrix.goos }}" \
GOARCH="${{ matrix.goarch }}" \
go build \
@@ -455,16 +168,7 @@ jobs:
chmod +x "${output}"
if [[ "${{ matrix.goos }}" == "linux" ]] && command -v ldd >/dev/null 2>&1; then
if ldd "${output}" 2>&1 | grep -q "liboffice_oxide"; then
echo "::error::linux CLI unexpectedly links liboffice_oxide dynamically"
ldd "${output}" || true
exit 1
fi
echo "Verified linux CLI does not require liboffice_oxide.so at runtime"
fi
- name: Build Go CLI release binaries
- name: Build Go CLI release binaries on Windows
if: runner.os == 'Windows'
shell: pwsh
run: |
@@ -478,46 +182,10 @@ jobs:
$output = Join-Path $env:DIST_DIR "${env:CLI_NAME}-${env:RELEASE_TAG}-${{ matrix.goos }}-${{ matrix.goarch }}${{ matrix.output_ext }}"
Write-Host "Building ${{ matrix.goos }}/${{ matrix.goarch }} -> $output"
$officeOxidePrefix = $env:OFFICE_OXIDE_PREFIX -replace "\\", "/"
$cc = $env:CC
if ([string]::IsNullOrWhiteSpace($cc)) {
Write-Error "CC is not set"
exit 1
}
if (-not (Test-Path $cc)) {
Write-Error "C compiler does not exist: $cc"
exit 1
}
if (-not (Test-Path "${officeOxidePrefix}/lib/liboffice_oxide.a")) {
Write-Error "liboffice_oxide.a does not exist: ${officeOxidePrefix}/lib/liboffice_oxide.a"
exit 1
}
if (-not (Test-Path "internal/cpp/cmake-build-release/librag_tokenizer_c_api.a")) {
Write-Error "librag_tokenizer_c_api.a does not exist"
exit 1
}
if ([string]::IsNullOrWhiteSpace($env:PCRE2_LIBDIR) -or -not (Test-Path $env:PCRE2_LIBDIR)) {
Write-Error "PCRE2_LIBDIR is not set or does not exist: $env:PCRE2_LIBDIR"
exit 1
}
$ragTokenizerLib = (Resolve-Path "internal/cpp/cmake-build-release/librag_tokenizer_c_api.a").Path -replace '\\', '/'
$pcre2LibDir = $env:PCRE2_LIBDIR -replace '\\', '/'
$pcre2IncludeDir = $env:PCRE2_INCLUDEDIR -replace '\\', '/'
$env:CGO_ENABLED = "1"
$env:CC = $cc
$env:CGO_CFLAGS = "-I${officeOxidePrefix}/include -I${officeOxidePrefix}/include/office_oxide_c -I${pcre2IncludeDir}"
$env:CGO_LDFLAGS = "${officeOxidePrefix}/lib/liboffice_oxide.a ${ragTokenizerLib} -L${pcre2LibDir} -lpcre2-8 -lc++ -static -static-libgcc -static-libstdc++ -lws2_32 -lbcrypt -lntdll -luserenv -ladvapi32"
$env:CGO_ENABLED = "0"
$env:GOOS = "${{ matrix.goos }}"
$env:GOARCH = "${{ matrix.goarch }}"
Write-Host "CC: $env:CC"
& $env:CC --version
Write-Host "office_oxide prefix: $officeOxidePrefix"
Write-Host "CGO_CFLAGS: $env:CGO_CFLAGS"
Write-Host "CGO_LDFLAGS: $env:CGO_LDFLAGS"
go build `
-trimpath `
-ldflags="-s -w -X main.version=$env:RELEASE_TAG -X main.commit=$env:GITHUB_SHA" `

View File

@@ -40,6 +40,7 @@ jobs:
has_go_changes: ${{ steps.detect_changes.outputs.has_go_changes }}
has_python_changes: ${{ steps.detect_changes.outputs.has_python_changes }}
has_web_changes: ${{ steps.detect_changes.outputs.has_web_changes }}
api_proxy_schemes: ${{ steps.detect_changes.outputs.api_proxy_schemes }}
steps:
- name: Ensure workspace ownership
run: |
@@ -135,7 +136,9 @@ jobs:
trap 'rm -f "$changed_files"' EXIT
git diff --name-only ${{ github.event.pull_request.base.sha }}...${{ github.event.pull_request.head.sha }} \
| while read -r file; do
[[ -f "$file" ]] && printf '%s\0' "$file"
if [[ -f "$file" ]]; then
printf '%s\0' "$file"
fi
done > "$changed_files"
echo "Changed files to run lefthook on:"
if [[ -s "$changed_files" ]]; then
@@ -174,16 +177,24 @@ jobs:
if [[ "${GITHUB_EVENT_NAME}" == "pull_request" ]]; then
CHANGED_FILES=$(git diff --name-only ${{ github.event.pull_request.base.sha }}...${{ github.event.pull_request.head.sha }})
else
CHANGED_FILES=$(git diff --name-only HEAD~1 HEAD)
BASE_SHA="${{ github.event.before }}"
if [[ -n "${BASE_SHA}" && ! "${BASE_SHA}" =~ ^0+$ ]]; then
CHANGED_FILES=$(git diff --name-only "${BASE_SHA}" "${GITHUB_SHA}")
else
CHANGED_FILES=$(git diff --name-only HEAD~1 HEAD)
fi
fi
while IFS= read -r file; do
case "$file" in
*.go|go.mod|go.sum) has_go=true ;;
# Docker, test, SDK, and workflow changes can alter either server path.
Dockerfile|docker/**|.github/workflows/**|test/**|sdk/**)
has_go=true
has_python=true
;;
*.go|go.mod|go.sum|build.sh) has_go=true ;;
*.py|pyproject.toml|requirements*.txt) has_python=true ;;
web/*) has_web=true ;;
Dockerfile|docker-compose*.yml) has_go=true; has_python=true ;;
build.sh) has_go=true ;;
esac
done <<< "$CHANGED_FILES"
@@ -194,10 +205,19 @@ jobs:
fi
fi
if [[ "$has_go" == "true" && "$has_python" == "true" ]]; then
api_proxy_schemes='["go","python"]'
elif [[ "$has_go" == "true" ]]; then
api_proxy_schemes='["go"]'
else
api_proxy_schemes='["python"]'
fi
echo "has_go_changes=${has_go}" >> $GITHUB_OUTPUT
echo "has_python_changes=${has_python}" >> $GITHUB_OUTPUT
echo "has_web_changes=${has_web}" >> $GITHUB_OUTPUT
echo "Go: ${has_go}, Python: ${has_python}, Web: ${has_web}"
echo "api_proxy_schemes=${api_proxy_schemes}" >> $GITHUB_OUTPUT
echo "Go: ${has_go}, Python: ${has_python}, Web: ${has_web}, proxy schemes: ${api_proxy_schemes}"
- name: Prepare Python test environment
if: steps.detect_changes.outputs.has_python_changes == 'true'
@@ -216,13 +236,18 @@ jobs:
ragflow_tests_infinity:
name: ragflow_tests_infinity
name: ragflow_tests_infinity (${{ matrix.api_proxy_scheme }})
needs: ragflow_preflight
if: ${{ github.event_name != 'pull_request' || (github.event.pull_request.draft == false && contains(github.event.pull_request.labels.*.name, 'ci') && (github.event.action != 'labeled' || github.event.label.name == 'ci')) }}
runs-on: [ "self-hosted", "ragflow-test" ]
strategy:
fail-fast: false
matrix:
api_proxy_scheme: ${{ fromJSON(needs.ragflow_preflight.outputs.api_proxy_schemes) }}
env:
DOC_ENGINE: infinity
RAGFLOW_IMAGE: infiniflow/ragflow:${{ github.run_id }}-infinity
API_PROXY_SCHEME: ${{ matrix.api_proxy_scheme }}
RAGFLOW_IMAGE: infiniflow/ragflow:${{ github.run_id }}-infinity-${{ matrix.api_proxy_scheme }}
HTTP_API_TEST_LEVEL: ${{ needs.ragflow_preflight.outputs.http_api_test_level }}
HAS_GO: ${{ needs.ragflow_preflight.outputs.has_go_changes }}
HAS_PYTHON: ${{ needs.ragflow_preflight.outputs.has_python_changes }}
@@ -243,7 +268,7 @@ jobs:
fetch-tags: true
- name: Build ragflow go server
if: env.HAS_GO == 'true'
if: env.API_PROXY_SCHEME == 'go'
run: |
set -euo pipefail
BUILDER_CONTAINER=ragflow_build_${GITHUB_RUN_ID}_${DOC_ENGINE}_$(od -An -N4 -tx4 /dev/urandom | tr -d ' ')
@@ -259,13 +284,13 @@ jobs:
-e TZ="${TZ}" \
-e UV_INDEX=https://mirrors.aliyun.com/pypi/simple \
-v "${PWD}:/ragflow" \
-v "${PWD}/internal/cpp/resource:/usr/share/infinity/resource" \
-v "${PWD}/internal/binding/cpp/resource:/usr/share/infinity/resource" \
infiniflow/infinity_builder:ubuntu22_clang20
sudo docker exec "${BUILDER_CONTAINER}" bash -c 'git config --global safe.directory "*" && cd /ragflow && ./build.sh --cpp'
./build.sh --go
- name: Run Go unit tests
if: env.HAS_GO == 'true'
if: env.API_PROXY_SCHEME == 'go'
# Runs after `./build.sh --go`, which guarantees the C++ static
# library (librag_tokenizer_c_api.a) is present on disk. The Go
# test binaries link against it transitively through
@@ -295,30 +320,35 @@ jobs:
fi
- name: Build ragflow:nightly
if: env.HAS_GO == 'true' || env.HAS_PYTHON == 'true' || env.HAS_WEB == 'true'
run: |
set -euo pipefail
sudo docker pull ubuntu:24.04
sudo DOCKER_BUILDKIT=1 docker build --build-arg NEED_MIRROR=1 --build-arg HTTPS_PROXY=${HTTPS_PROXY} --build-arg HTTP_PROXY=${HTTP_PROXY} -f Dockerfile -t ${RAGFLOW_IMAGE} .
RUNNER_WORKSPACE_PREFIX=${RUNNER_WORKSPACE_PREFIX:-${HOME}}
BUILD_LOCK_FILE=${RUNNER_WORKSPACE_PREFIX}/artifacts/${GITHUB_REPOSITORY}/docker-build.lock
mkdir -p "$(dirname "${BUILD_LOCK_FILE}")"
(
flock -w 10800 9 || { echo "Timed out waiting for the shared Docker build slot" >&2; exit 1; }
echo "Acquired Docker build slot for ${DOC_ENGINE}/${API_PROXY_SCHEME}"
sudo docker pull ubuntu:24.04
sudo DOCKER_BUILDKIT=1 docker build --build-arg NEED_MIRROR=1 --build-arg HTTPS_PROXY=${HTTPS_PROXY} --build-arg HTTP_PROXY=${HTTP_PROXY} -f Dockerfile -t ${RAGFLOW_IMAGE} .
) 9>"${BUILD_LOCK_FILE}"
- name: Prepare Python test environment
if: env.HAS_PYTHON == 'true'
run: |
uv sync --python 3.13 --group test --frozen
uv pip install -e sdk/python
- name: Prepare function test environment
if: env.HAS_GO == 'true' || env.HAS_PYTHON == 'true' || env.HAS_WEB == 'true'
working-directory: docker
run: |
set -euo pipefail
# install ss
sudo chmod 1777 /tmp
sudo apt update && sudo apt install -y iproute2
RUNNER_WORKSPACE_PREFIX=${RUNNER_WORKSPACE_PREFIX:-${HOME}}
COMPOSE_PROJECT_NAME="${GITHUB_RUN_ID}-${DOC_ENGINE}"
COMPOSE_PROJECT_NAME="${GITHUB_RUN_ID}-${DOC_ENGINE}-${API_PROXY_SCHEME}"
echo "COMPOSE_PROJECT_NAME=${COMPOSE_PROJECT_NAME}" >> ${GITHUB_ENV}
echo "RAGFLOW_CONTAINER=${COMPOSE_PROJECT_NAME}-ragflow-cpu-1" >> ${GITHUB_ENV}
ARTIFACTS_DIR=${RUNNER_WORKSPACE_PREFIX}/artifacts/${GITHUB_REPOSITORY}/${GITHUB_RUN_ID}/${DOC_ENGINE}
ARTIFACTS_DIR=${RUNNER_WORKSPACE_PREFIX}/artifacts/${GITHUB_REPOSITORY}/${GITHUB_RUN_ID}/${DOC_ENGINE}/${API_PROXY_SCHEME}
echo "ARTIFACTS_DIR=${ARTIFACTS_DIR}" >> ${GITHUB_ENV}
rm -rf "${ARTIFACTS_DIR}" && mkdir -p "${ARTIFACTS_DIR}"
@@ -329,7 +359,7 @@ jobs:
# Engine-specific offset partitions keep concurrent engine jobs from
# choosing the same host ports when they land on the same self-hosted runner.
# A lock plus reservation file closes the check/start race between parallel jobs.
PORT_BASES=(1200 1201 23817 23820 5432 5455 9000 9001 6379 6380 6601 9380 9381 9382 9384 9383 9385 80 443 4222)
PORT_BASES=(1200 1201 23817 23820 5432 5455 9000 9001 6379 6380 6601 9380 9381 9382 9384 9383 9385 80 443 4222 8222)
PARTITION_SIZE=6000
case "${DOC_ENGINE}" in
elasticsearch) PARTITION_BASE=1000 ;;
@@ -397,7 +427,8 @@ jobs:
MINIO_PORT=$((9000 + PORT_OFFSET))
MINIO_CONSOLE_PORT=$((9001 + PORT_OFFSET))
REDIS_PORT=$((6379 + PORT_OFFSET))
NATS_PORT=$((4222 + PORT_OFFSET))
EXPOSE_NATS_PORT=$((4222 + PORT_OFFSET))
NATS_MONITORING_PORT=$((8222 + PORT_OFFSET))
TEI_PORT=$((6380 + PORT_OFFSET))
KIBANA_PORT=$((6601 + PORT_OFFSET))
SVR_HTTP_PORT=$((9380 + PORT_OFFSET))
@@ -411,7 +442,7 @@ jobs:
# Persist computed ports into .env so docker-compose uses the correct host bindings.
# Remove previous CI overrides first; docker compose uses the last duplicate key.
sed -i '/^ES_PORT=/d;/^OS_PORT=/d;/^INFINITY_THRIFT_PORT=/d;/^INFINITY_HTTP_PORT=/d;/^INFINITY_PSQL_PORT=/d;/^EXPOSE_MYSQL_PORT=/d;/^MINIO_PORT=/d;/^MINIO_CONSOLE_PORT=/d;/^REDIS_PORT=/d;/^TEI_PORT=/d;/^KIBANA_PORT=/d;/^SVR_HTTP_PORT=/d;/^ADMIN_SVR_HTTP_PORT=/d;/^SVR_MCP_PORT=/d;/^GO_HTTP_PORT=/d;/^GO_ADMIN_PORT=/d;/^SANDBOX_EXECUTOR_MANAGER_PORT=/d;/^SVR_WEB_HTTP_PORT=/d;/^SVR_WEB_HTTPS_PORT=/d;/^NATS_PORT=/d;/^COMPOSE_PROFILES=/d;/^TEI_MODEL=/d;/^RAGFLOW_IMAGE=/d;/^DOC_ENGINE=/d' .env
sed -i '/^ES_PORT=/d;/^OS_PORT=/d;/^INFINITY_THRIFT_PORT=/d;/^INFINITY_HTTP_PORT=/d;/^INFINITY_PSQL_PORT=/d;/^EXPOSE_MYSQL_PORT=/d;/^MINIO_PORT=/d;/^MINIO_CONSOLE_PORT=/d;/^REDIS_PORT=/d;/^EXPOSE_NATS_PORT=/d;/^NATS_MONITORING_PORT=/d;/^TEI_PORT=/d;/^KIBANA_PORT=/d;/^SVR_HTTP_PORT=/d;/^ADMIN_SVR_HTTP_PORT=/d;/^SVR_MCP_PORT=/d;/^GO_HTTP_PORT=/d;/^GO_ADMIN_PORT=/d;/^SANDBOX_EXECUTOR_MANAGER_PORT=/d;/^SVR_WEB_HTTP_PORT=/d;/^SVR_WEB_HTTPS_PORT=/d;/^COMPOSE_PROFILES=/d;/^TEI_MODEL=/d;/^RAGFLOW_IMAGE=/d;/^DOC_ENGINE=/d;/^API_PROXY_SCHEME=/d' .env
{
echo ""
echo "ES_PORT=${ES_PORT}"
@@ -423,7 +454,8 @@ jobs:
echo "MINIO_PORT=${MINIO_PORT}"
echo "MINIO_CONSOLE_PORT=${MINIO_CONSOLE_PORT}"
echo "REDIS_PORT=${REDIS_PORT}"
echo "NATS_PORT=${NATS_PORT}"
echo "EXPOSE_NATS_PORT=${EXPOSE_NATS_PORT}"
echo "NATS_MONITORING_PORT=${NATS_MONITORING_PORT}"
echo "TEI_PORT=${TEI_PORT}"
echo "KIBANA_PORT=${KIBANA_PORT}"
echo "SVR_HTTP_PORT=${SVR_HTTP_PORT}"
@@ -434,26 +466,37 @@ jobs:
echo "SANDBOX_EXECUTOR_MANAGER_PORT=${SANDBOX_EXECUTOR_MANAGER_PORT}"
echo "SVR_WEB_HTTP_PORT=${SVR_WEB_HTTP_PORT}"
echo "SVR_WEB_HTTPS_PORT=${SVR_WEB_HTTPS_PORT}"
echo "COMPOSE_PROFILES=${DOC_ENGINE},cpu,tei-cpu,deepdoc"
echo "COMPOSE_PROFILES=${DOC_ENGINE},cpu,ragflow-go,tei-cpu,deepdoc"
echo "TEI_MODEL=BAAI/bge-small-en-v1.5"
echo "RAGFLOW_IMAGE=${RAGFLOW_IMAGE}"
echo "DOC_ENGINE=${DOC_ENGINE}"
echo "API_PROXY_SCHEME=${API_PROXY_SCHEME}"
} >> .env
echo "HOST_ADDRESS=http://host.docker.internal:${SVR_HTTP_PORT}" >> ${GITHUB_ENV}
if [[ "${API_PROXY_SCHEME}" == "go" ]]; then
RAGFLOW_HTTP_PORT=${SVR_WEB_HTTP_PORT}
else
RAGFLOW_HTTP_PORT=${SVR_HTTP_PORT}
fi
echo "RAGFLOW_HTTP_PORT=${RAGFLOW_HTTP_PORT}" >> ${GITHUB_ENV}
echo "HOST_ADDRESS=http://host.docker.internal:${RAGFLOW_HTTP_PORT}" >> ${GITHUB_ENV}
sed -i \
-e 's#${NATS_PORT:-4222}:4222#${EXPOSE_NATS_PORT}:4222#' \
-e 's#"8222:8222"#"${NATS_MONITORING_PORT}:8222"#' \
docker-compose-base.yml
# Patch entrypoint.sh for coverage
sed -i '/"\$PY" api\/ragflow_server.py \${INIT_SUPERUSER_ARGS} &/c\ echo "Ensuring coverage is installed..."\n "$PY" -m pip install coverage -i https://mirrors.aliyun.com/pypi/simple\n export COVERAGE_FILE=/ragflow/logs/.coverage\n echo "Starting ragflow_server with coverage..."\n "$PY" -m coverage run --source=./api/apps --omit="*/tests/*,*/migrations/*" -a api/ragflow_server.py ${INIT_SUPERUSER_ARGS} &' ./entrypoint.sh
- name: Start ragflow:nightly for Infinity
if: env.HAS_GO == 'true' || env.HAS_PYTHON == 'true' || env.HAS_WEB == 'true'
run: |
sudo docker compose -f docker/docker-compose.yml -p ${COMPOSE_PROJECT_NAME} down -v || true
sudo docker ps -a --filter "label=com.docker.compose.project=${COMPOSE_PROJECT_NAME}" -q | xargs -r sudo docker rm -f
sudo docker compose -f docker/docker-compose.yml -p ${COMPOSE_PROJECT_NAME} up -d
- name: Run sdk tests against Infinity
if: env.HAS_PYTHON == 'true'
if: env.API_PROXY_SCHEME == 'python'
run: |
export http_proxy=""; export https_proxy=""; export no_proxy=""; export HTTP_PROXY=""; export HTTPS_PROXY=""; export NO_PROXY=""
svc_ready=0
@@ -474,7 +517,6 @@ jobs:
source .venv/bin/activate && set -o pipefail; DOC_ENGINE=infinity pytest -s --tb=short --level=${HTTP_API_TEST_LEVEL} --junitxml=pytest-infinity-sdk.xml --cov=sdk/python/ragflow_sdk --cov-branch --cov-report=xml:coverage-infinity-sdk.xml test/testcases/test_sdk_api 2>&1 | tee infinity_sdk_test.log
- name: Run New RESTFUL api tests against Infinity
if: env.HAS_PYTHON == 'true'
run: |
export http_proxy=""; export https_proxy=""; export no_proxy=""; export HTTP_PROXY=""; export HTTPS_PROXY=""; export NO_PROXY=""
svc_ready=0
@@ -494,7 +536,7 @@ jobs:
source .venv/bin/activate && set -o pipefail; DOC_ENGINE=infinity pytest -s --tb=short --level=${HTTP_API_TEST_LEVEL} test/testcases/restful_api 2>&1 | tee infinity_restful_api_test.log
- name: RAGFlow CLI retrieval test Infinity
if: env.HAS_PYTHON == 'true'
if: env.API_PROXY_SCHEME == 'python'
env:
PYTHONPATH: ${{ github.workspace }}
run: |
@@ -548,9 +590,8 @@ jobs:
source docker/.env
set +a
HOST_ADDRESS="http://host.docker.internal:${SVR_HTTP_PORT}"
USER_HOST="$(echo "${HOST_ADDRESS}" | sed -E 's#^https?://([^:/]+).*#\1#')"
USER_PORT="${SVR_HTTP_PORT}"
USER_PORT="${RAGFLOW_HTTP_PORT}"
ADMIN_HOST="${USER_HOST}"
ADMIN_PORT="${ADMIN_SVR_HTTP_PORT}"
@@ -606,7 +647,7 @@ jobs:
run_cli "${LOG_FILE}" $CLI --type user --host "$USER_HOST" --port "$USER_PORT" --username "$EMAIL" --password "$PASS" command "Benchmark 16 100 search 'what are these documents about' on datasets '$DATASET'"
- name: Stop ragflow to save coverage Infinity
if: ${{ !cancelled() && env.HAS_PYTHON == 'true' }}
if: ${{ !cancelled() && env.API_PROXY_SCHEME == 'python' }}
run: |
# Send SIGINT to ragflow_server.py to trigger coverage save
PID=$(sudo docker exec ${RAGFLOW_CONTAINER} ps aux | grep "ragflow_server.py" | grep -v grep | awk '{print $2}' | head -n 1)
@@ -621,7 +662,7 @@ jobs:
sudo docker compose -f docker/docker-compose.yml -p ${COMPOSE_PROJECT_NAME} stop
- name: Generate server coverage report Infinity
if: ${{ !cancelled() && env.HAS_PYTHON == 'true' }}
if: ${{ !cancelled() && env.API_PROXY_SCHEME == 'python' }}
run: |
# .coverage file should be in docker/ragflow-logs/.coverage
if [ -f docker/ragflow-logs/.coverage ]; then
@@ -647,18 +688,20 @@ jobs:
fail_ci_if_error: false
- name: Collect ragflow log Infinity
if: ${{ !cancelled() && env.HAS_PYTHON == 'true' }}
if: ${{ !cancelled() }}
run: |
if [ -d docker/ragflow-logs ]; then
cp -r docker/ragflow-logs ${ARTIFACTS_DIR}/ragflow-logs-infinity
echo "ragflow log" && tail -n 200 docker/ragflow-logs/ragflow_server.log || true
destination="${ARTIFACTS_DIR}/ragflow-logs-infinity"
sudo cp -r docker/ragflow-logs "${destination}"
sudo chown -R "$(id -u):$(id -g)" "${destination}"
echo "ragflow log" && sudo tail -n 200 docker/ragflow-logs/ragflow_server.log || true
else
echo "No docker/ragflow-logs directory found; skipping log collection"
fi
sudo rm -rf docker/ragflow-logs || true
- name: Stop ragflow:nightly for Infinity
if: ${{ always() && (env.HAS_GO == 'true' || env.HAS_PYTHON == 'true' || env.HAS_WEB == 'true') }}
if: ${{ always() }}
run: |
# Sometimes `docker compose down` fail due to hang container, heavy load etc. Need to remove such containers to release resources(for example, listen ports).
sudo docker compose -f docker/docker-compose.yml -p ${COMPOSE_PROJECT_NAME} down -v || true
@@ -673,13 +716,18 @@ jobs:
ragflow_tests_elasticsearch:
name: ragflow_tests_elasticsearch
name: ragflow_tests_elasticsearch (${{ matrix.api_proxy_scheme }})
needs: ragflow_preflight
if: ${{ github.event_name != 'pull_request' || (github.event.pull_request.draft == false && contains(github.event.pull_request.labels.*.name, 'ci') && (github.event.action != 'labeled' || github.event.label.name == 'ci')) }}
runs-on: [ "self-hosted", "ragflow-test" ]
strategy:
fail-fast: false
matrix:
api_proxy_scheme: ${{ fromJSON(needs.ragflow_preflight.outputs.api_proxy_schemes) }}
env:
DOC_ENGINE: elasticsearch
RAGFLOW_IMAGE: infiniflow/ragflow:${{ github.run_id }}-elasticsearch
API_PROXY_SCHEME: ${{ matrix.api_proxy_scheme }}
RAGFLOW_IMAGE: infiniflow/ragflow:${{ github.run_id }}-elasticsearch-${{ matrix.api_proxy_scheme }}
HTTP_API_TEST_LEVEL: ${{ needs.ragflow_preflight.outputs.http_api_test_level }}
HAS_GO: ${{ needs.ragflow_preflight.outputs.has_go_changes }}
HAS_PYTHON: ${{ needs.ragflow_preflight.outputs.has_python_changes }}
@@ -700,7 +748,7 @@ jobs:
fetch-tags: true
- name: Build ragflow go server
if: env.HAS_GO == 'true'
if: env.API_PROXY_SCHEME == 'go'
run: |
set -euo pipefail
BUILDER_CONTAINER=ragflow_build_${GITHUB_RUN_ID}_${DOC_ENGINE}_$(od -An -N4 -tx4 /dev/urandom | tr -d ' ')
@@ -716,13 +764,13 @@ jobs:
-e TZ="${TZ}" \
-e UV_INDEX=https://mirrors.aliyun.com/pypi/simple \
-v "${PWD}:/ragflow" \
-v "${PWD}/internal/cpp/resource:/usr/share/infinity/resource" \
-v "${PWD}/internal/binding/cpp/resource:/usr/share/infinity/resource" \
infiniflow/infinity_builder:ubuntu22_clang20
sudo docker exec "${BUILDER_CONTAINER}" bash -c 'git config --global safe.directory "*" && cd /ragflow && ./build.sh --cpp'
./build.sh --go
- name: Run Go unit tests
if: env.HAS_GO == 'true'
if: env.API_PROXY_SCHEME == 'go'
# Runs after `./build.sh --go`, which guarantees the C++ static
# library (librag_tokenizer_c_api.a) is present on disk. The Go
# test binaries link against it transitively through
@@ -752,30 +800,35 @@ jobs:
fi
- name: Build ragflow:nightly
if: env.HAS_GO == 'true' || env.HAS_PYTHON == 'true' || env.HAS_WEB == 'true'
run: |
set -euo pipefail
sudo docker pull ubuntu:24.04
sudo DOCKER_BUILDKIT=1 docker build --build-arg NEED_MIRROR=1 --build-arg HTTPS_PROXY=${HTTPS_PROXY} --build-arg HTTP_PROXY=${HTTP_PROXY} -f Dockerfile -t ${RAGFLOW_IMAGE} .
RUNNER_WORKSPACE_PREFIX=${RUNNER_WORKSPACE_PREFIX:-${HOME}}
BUILD_LOCK_FILE=${RUNNER_WORKSPACE_PREFIX}/artifacts/${GITHUB_REPOSITORY}/docker-build.lock
mkdir -p "$(dirname "${BUILD_LOCK_FILE}")"
(
flock -w 10800 9 || { echo "Timed out waiting for the shared Docker build slot" >&2; exit 1; }
echo "Acquired Docker build slot for ${DOC_ENGINE}/${API_PROXY_SCHEME}"
sudo docker pull ubuntu:24.04
sudo DOCKER_BUILDKIT=1 docker build --build-arg NEED_MIRROR=1 --build-arg HTTPS_PROXY=${HTTPS_PROXY} --build-arg HTTP_PROXY=${HTTP_PROXY} -f Dockerfile -t ${RAGFLOW_IMAGE} .
) 9>"${BUILD_LOCK_FILE}"
- name: Prepare Python test environment
if: env.HAS_PYTHON == 'true'
run: |
uv sync --python 3.13 --group test --frozen
uv pip install -e sdk/python
- name: Prepare function test environment
if: env.HAS_GO == 'true' || env.HAS_PYTHON == 'true' || env.HAS_WEB == 'true'
working-directory: docker
run: |
set -euo pipefail
# install ss
sudo chmod 1777 /tmp
sudo apt update && sudo apt install -y iproute2
RUNNER_WORKSPACE_PREFIX=${RUNNER_WORKSPACE_PREFIX:-${HOME}}
COMPOSE_PROJECT_NAME="${GITHUB_RUN_ID}-${DOC_ENGINE}"
COMPOSE_PROJECT_NAME="${GITHUB_RUN_ID}-${DOC_ENGINE}-${API_PROXY_SCHEME}"
echo "COMPOSE_PROJECT_NAME=${COMPOSE_PROJECT_NAME}" >> ${GITHUB_ENV}
echo "RAGFLOW_CONTAINER=${COMPOSE_PROJECT_NAME}-ragflow-cpu-1" >> ${GITHUB_ENV}
ARTIFACTS_DIR=${RUNNER_WORKSPACE_PREFIX}/artifacts/${GITHUB_REPOSITORY}/${GITHUB_RUN_ID}/${DOC_ENGINE}
ARTIFACTS_DIR=${RUNNER_WORKSPACE_PREFIX}/artifacts/${GITHUB_REPOSITORY}/${GITHUB_RUN_ID}/${DOC_ENGINE}/${API_PROXY_SCHEME}
echo "ARTIFACTS_DIR=${ARTIFACTS_DIR}" >> ${GITHUB_ENV}
rm -rf "${ARTIFACTS_DIR}" && mkdir -p "${ARTIFACTS_DIR}"
@@ -786,7 +839,7 @@ jobs:
# Engine-specific offset partitions keep concurrent engine jobs from
# choosing the same host ports when they land on the same self-hosted runner.
# A lock plus reservation file closes the check/start race between parallel jobs.
PORT_BASES=(1200 1201 23817 23820 5432 5455 9000 9001 6379 6380 6601 9380 9381 9382 9384 9383 9385 80 443 4222)
PORT_BASES=(1200 1201 23817 23820 5432 5455 9000 9001 6379 6380 6601 9380 9381 9382 9384 9383 9385 80 443 4222 8222)
PARTITION_SIZE=6000
case "${DOC_ENGINE}" in
elasticsearch) PARTITION_BASE=1000 ;;
@@ -854,7 +907,8 @@ jobs:
MINIO_PORT=$((9000 + PORT_OFFSET))
MINIO_CONSOLE_PORT=$((9001 + PORT_OFFSET))
REDIS_PORT=$((6379 + PORT_OFFSET))
NATS_PORT=$((4222 + PORT_OFFSET))
EXPOSE_NATS_PORT=$((4222 + PORT_OFFSET))
NATS_MONITORING_PORT=$((8222 + PORT_OFFSET))
TEI_PORT=$((6380 + PORT_OFFSET))
KIBANA_PORT=$((6601 + PORT_OFFSET))
SVR_HTTP_PORT=$((9380 + PORT_OFFSET))
@@ -868,7 +922,7 @@ jobs:
# Persist computed ports into .env so docker-compose uses the correct host bindings.
# Remove previous CI overrides first; docker compose uses the last duplicate key.
sed -i '/^ES_PORT=/d;/^OS_PORT=/d;/^INFINITY_THRIFT_PORT=/d;/^INFINITY_HTTP_PORT=/d;/^INFINITY_PSQL_PORT=/d;/^EXPOSE_MYSQL_PORT=/d;/^MINIO_PORT=/d;/^MINIO_CONSOLE_PORT=/d;/^REDIS_PORT=/d;/^TEI_PORT=/d;/^KIBANA_PORT=/d;/^SVR_HTTP_PORT=/d;/^ADMIN_SVR_HTTP_PORT=/d;/^SVR_MCP_PORT=/d;/^GO_HTTP_PORT=/d;/^GO_ADMIN_PORT=/d;/^SANDBOX_EXECUTOR_MANAGER_PORT=/d;/^SVR_WEB_HTTP_PORT=/d;/^SVR_WEB_HTTPS_PORT=/d;/^NATS_PORT=/d;/^COMPOSE_PROFILES=/d;/^TEI_MODEL=/d;/^RAGFLOW_IMAGE=/d;/^DOC_ENGINE=/d' .env
sed -i '/^ES_PORT=/d;/^OS_PORT=/d;/^INFINITY_THRIFT_PORT=/d;/^INFINITY_HTTP_PORT=/d;/^INFINITY_PSQL_PORT=/d;/^EXPOSE_MYSQL_PORT=/d;/^MINIO_PORT=/d;/^MINIO_CONSOLE_PORT=/d;/^REDIS_PORT=/d;/^EXPOSE_NATS_PORT=/d;/^NATS_MONITORING_PORT=/d;/^TEI_PORT=/d;/^KIBANA_PORT=/d;/^SVR_HTTP_PORT=/d;/^ADMIN_SVR_HTTP_PORT=/d;/^SVR_MCP_PORT=/d;/^GO_HTTP_PORT=/d;/^GO_ADMIN_PORT=/d;/^SANDBOX_EXECUTOR_MANAGER_PORT=/d;/^SVR_WEB_HTTP_PORT=/d;/^SVR_WEB_HTTPS_PORT=/d;/^COMPOSE_PROFILES=/d;/^TEI_MODEL=/d;/^RAGFLOW_IMAGE=/d;/^DOC_ENGINE=/d;/^API_PROXY_SCHEME=/d' .env
{
echo ""
echo "ES_PORT=${ES_PORT}"
@@ -880,7 +934,8 @@ jobs:
echo "MINIO_PORT=${MINIO_PORT}"
echo "MINIO_CONSOLE_PORT=${MINIO_CONSOLE_PORT}"
echo "REDIS_PORT=${REDIS_PORT}"
echo "NATS_PORT=${NATS_PORT}"
echo "EXPOSE_NATS_PORT=${EXPOSE_NATS_PORT}"
echo "NATS_MONITORING_PORT=${NATS_MONITORING_PORT}"
echo "TEI_PORT=${TEI_PORT}"
echo "KIBANA_PORT=${KIBANA_PORT}"
echo "SVR_HTTP_PORT=${SVR_HTTP_PORT}"
@@ -891,26 +946,37 @@ jobs:
echo "SANDBOX_EXECUTOR_MANAGER_PORT=${SANDBOX_EXECUTOR_MANAGER_PORT}"
echo "SVR_WEB_HTTP_PORT=${SVR_WEB_HTTP_PORT}"
echo "SVR_WEB_HTTPS_PORT=${SVR_WEB_HTTPS_PORT}"
echo "COMPOSE_PROFILES=${DOC_ENGINE},cpu,tei-cpu,deepdoc"
echo "COMPOSE_PROFILES=${DOC_ENGINE},cpu,ragflow-go,tei-cpu,deepdoc"
echo "TEI_MODEL=BAAI/bge-small-en-v1.5"
echo "RAGFLOW_IMAGE=${RAGFLOW_IMAGE}"
echo "DOC_ENGINE=${DOC_ENGINE}"
echo "API_PROXY_SCHEME=${API_PROXY_SCHEME}"
} >> .env
echo "HOST_ADDRESS=http://host.docker.internal:${SVR_HTTP_PORT}" >> ${GITHUB_ENV}
if [[ "${API_PROXY_SCHEME}" == "go" ]]; then
RAGFLOW_HTTP_PORT=${SVR_WEB_HTTP_PORT}
else
RAGFLOW_HTTP_PORT=${SVR_HTTP_PORT}
fi
echo "RAGFLOW_HTTP_PORT=${RAGFLOW_HTTP_PORT}" >> ${GITHUB_ENV}
echo "HOST_ADDRESS=http://host.docker.internal:${RAGFLOW_HTTP_PORT}" >> ${GITHUB_ENV}
sed -i \
-e 's#${NATS_PORT:-4222}:4222#${EXPOSE_NATS_PORT}:4222#' \
-e 's#"8222:8222"#"${NATS_MONITORING_PORT}:8222"#' \
docker-compose-base.yml
# Patch entrypoint.sh for coverage
sed -i '/"\$PY" api\/ragflow_server.py \${INIT_SUPERUSER_ARGS} &/c\ echo "Ensuring coverage is installed..."\n "$PY" -m pip install coverage -i https://mirrors.aliyun.com/pypi/simple\n export COVERAGE_FILE=/ragflow/logs/.coverage\n echo "Starting ragflow_server with coverage..."\n "$PY" -m coverage run --source=./api/apps --omit="*/tests/*,*/migrations/*" -a api/ragflow_server.py ${INIT_SUPERUSER_ARGS} &' ./entrypoint.sh
- name: Start ragflow:nightly for Elasticsearch
if: env.HAS_GO == 'true' || env.HAS_PYTHON == 'true' || env.HAS_WEB == 'true'
run: |
sudo docker compose -f docker/docker-compose.yml -p ${COMPOSE_PROJECT_NAME} down -v || true
sudo docker ps -a --filter "label=com.docker.compose.project=${COMPOSE_PROJECT_NAME}" -q | xargs -r sudo docker rm -f
sudo docker compose -f docker/docker-compose.yml -p ${COMPOSE_PROJECT_NAME} up -d
- name: Run sdk tests against Elasticsearch
if: env.HAS_PYTHON == 'true'
if: env.API_PROXY_SCHEME == 'python'
run: |
export http_proxy=""; export https_proxy=""; export no_proxy=""; export HTTP_PROXY=""; export HTTPS_PROXY=""; export NO_PROXY=""
svc_ready=0
@@ -931,7 +997,6 @@ jobs:
source .venv/bin/activate && set -o pipefail; pytest -s --tb=short --level=${HTTP_API_TEST_LEVEL} --junitxml=pytest-es-sdk.xml --cov=sdk/python/ragflow_sdk --cov-branch --cov-report=xml:coverage-es-sdk.xml test/testcases/test_sdk_api 2>&1 | tee es_sdk_test.log
- name: Run New RESTFUL api tests against Elasticsearch
if: env.HAS_PYTHON == 'true'
run: |
export http_proxy=""; export https_proxy=""; export no_proxy=""; export HTTP_PROXY=""; export HTTPS_PROXY=""; export NO_PROXY=""
svc_ready=0
@@ -951,7 +1016,7 @@ jobs:
source .venv/bin/activate && set -o pipefail; pytest -s --tb=short --level=${HTTP_API_TEST_LEVEL} test/testcases/restful_api 2>&1 | tee es_restful_api_test.log
- name: RAGFlow CLI retrieval test Elasticsearch
if: env.HAS_PYTHON == 'true'
if: env.API_PROXY_SCHEME == 'python'
env:
PYTHONPATH: ${{ github.workspace }}
run: |
@@ -1005,9 +1070,8 @@ jobs:
source docker/.env
set +a
HOST_ADDRESS="http://host.docker.internal:${SVR_HTTP_PORT}"
USER_HOST="$(echo "${HOST_ADDRESS}" | sed -E 's#^https?://([^:/]+).*#\1#')"
USER_PORT="${SVR_HTTP_PORT}"
USER_PORT="${RAGFLOW_HTTP_PORT}"
ADMIN_HOST="${USER_HOST}"
ADMIN_PORT="${ADMIN_SVR_HTTP_PORT}"
@@ -1063,7 +1127,7 @@ jobs:
run_cli "${LOG_FILE}" $CLI --type user --host "$USER_HOST" --port "$USER_PORT" --username "$EMAIL" --password "$PASS" command "Benchmark 16 100 search 'what are these documents about' on datasets '$DATASET'"
- name: Stop ragflow to save coverage Elasticsearch
if: ${{ !cancelled() && env.HAS_PYTHON == 'true' }}
if: ${{ !cancelled() && env.API_PROXY_SCHEME == 'python' }}
run: |
# Send SIGINT to ragflow_server.py to trigger coverage save
PID=$(sudo docker exec ${RAGFLOW_CONTAINER} ps aux | grep "ragflow_server.py" | grep -v grep | awk '{print $2}' | head -n 1)
@@ -1078,7 +1142,7 @@ jobs:
sudo docker compose -f docker/docker-compose.yml -p ${COMPOSE_PROJECT_NAME} stop
- name: Generate server coverage report Elasticsearch
if: ${{ !cancelled() && env.HAS_PYTHON == 'true' }}
if: ${{ !cancelled() && env.API_PROXY_SCHEME == 'python' }}
run: |
# .coverage file should be in docker/ragflow-logs/.coverage
if [ -f docker/ragflow-logs/.coverage ]; then
@@ -1099,18 +1163,20 @@ jobs:
fi
- name: Collect ragflow log Elasticsearch
if: ${{ !cancelled() && env.HAS_PYTHON == 'true' }}
if: ${{ !cancelled() }}
run: |
if [ -d docker/ragflow-logs ]; then
cp -r docker/ragflow-logs ${ARTIFACTS_DIR}/ragflow-logs-es
echo "ragflow log" && tail -n 200 docker/ragflow-logs/ragflow_server.log || true
destination="${ARTIFACTS_DIR}/ragflow-logs-es"
sudo cp -r docker/ragflow-logs "${destination}"
sudo chown -R "$(id -u):$(id -g)" "${destination}"
echo "ragflow log" && sudo tail -n 200 docker/ragflow-logs/ragflow_server.log || true
else
echo "No docker/ragflow-logs directory found; skipping log collection"
fi
sudo rm -rf docker/ragflow-logs || true
- name: Stop ragflow:nightly for Elasticsearch
if: ${{ always() && (env.HAS_GO == 'true' || env.HAS_PYTHON == 'true' || env.HAS_WEB == 'true') }}
if: ${{ always() }}
run: |
# Sometimes `docker compose down` fail due to hang container, heavy load etc. Need to remove such containers to release resources(for example, listen ports).
sudo docker compose -f docker/docker-compose.yml -p ${COMPOSE_PROJECT_NAME} down -v || true
@@ -1120,4 +1186,4 @@ jobs:
fi
if [[ -n ${PORT_RESERVATION:-} ]]; then
rm -f "${PORT_RESERVATION}"
fi
fi

View File

@@ -100,7 +100,9 @@ jobs:
trap 'rm -f "$changed_files"' EXIT
git diff --name-only ${{ github.event.pull_request.base.sha }}...${{ github.event.pull_request.head.sha }} \
| while read -r file; do
[[ -f "$file" ]] && printf '%s\0' "$file"
if [[ -f "$file" ]]; then
printf '%s\0' "$file"
fi
done > "$changed_files"
echo "Changed files to run lefthook on:"
if [[ -s "$changed_files" ]]; then
@@ -178,7 +180,7 @@ jobs:
-e TZ="${TZ}" \
-e UV_INDEX=https://mirrors.aliyun.com/pypi/simple \
-v "${PWD}:/ragflow" \
-v "${PWD}/internal/cpp/resource:/usr/share/infinity/resource" \
-v "${PWD}/internal/binding/cpp/resource:/usr/share/infinity/resource" \
infiniflow/infinity_builder:ubuntu22_clang20
sudo docker exec "${BUILDER_CONTAINER}" bash -c 'git config --global safe.directory "*" && cd /ragflow && ./build.sh --cpp'
./build.sh --go
@@ -622,7 +624,7 @@ jobs:
-e TZ="${TZ}" \
-e UV_INDEX=https://mirrors.aliyun.com/pypi/simple \
-v "${PWD}:/ragflow" \
-v "${PWD}/internal/cpp/resource:/usr/share/infinity/resource" \
-v "${PWD}/internal/binding/cpp/resource:/usr/share/infinity/resource" \
infiniflow/infinity_builder:ubuntu22_clang20
sudo docker exec "${BUILDER_CONTAINER}" bash -c 'git config --global safe.directory "*" && cd /ragflow && ./build.sh --cpp'
./build.sh --go

18
.gitignore vendored
View File

@@ -22,6 +22,7 @@ Cargo.lock
.idea/
.vscode/
.cursor/settings.json
.opencode/
# Exclude Mac generated files
.DS_Store
@@ -225,9 +226,9 @@ uv-aarch64-unknown-linux-gnu.tar.gz
docker/launch_backend_service_windows.sh
# C++ build directories
internal/cpp/build/
internal/cpp/cmake-build-release/
internal/cpp/cmake-build-debug/
internal/binding/cpp/build/
internal/binding/cpp/cmake-build-release/
internal/binding/cpp/cmake-build-debug/
# Trae IDE config
.trae/
@@ -245,5 +246,16 @@ bin/*
# Parser test fixtures and python tools
internal/deepdoc/parser/pdf/testdata/
internal/deepdoc/parser/pdf/tools-py/
# IDE tooling artifacts
.codebuddy/
# Local build output
build/
internal/deepdoc/parser/docx/testdata/
internal/deepdoc/parser/docx/tool/
# test data compare tool
internal/ingestion/task/tool/generate_dataflow_golden.py
internal/ingestion/task/tool/README.md
internal/cpp/cmake-build-release

View File

@@ -72,9 +72,9 @@ docker/seekdb
# Native / compiled build dirs
target/
bin/
internal/cpp/build/
internal/cpp/cmake-build-release/
internal/cpp/cmake-build-debug/
internal/binding/cpp/build/
internal/binding/cpp/cmake-build-release/
internal/binding/cpp/cmake-build-debug/
# Optional: skip tests and docs from indexing
# test/

192
AGENTS.md
View File

@@ -1,109 +1,109 @@
# RAGFlow Project Instructions for GitHub Copilot
# RAGFlow Instructions
This file provides context, build instructions, and coding standards for the RAGFlow project.
It is structured to follow GitHub Copilot's [customization guidelines](https://docs.github.com/en/copilot/concepts/prompting/response-customization).
Use this file as the local operating guide for the current codebase. Prefer the code and the current CLAUDE.md over any older convention or remembered project shape.
## 1. Project Overview
RAGFlow is an open-source RAG (Retrieval-Augmented Generation) engine based on deep document understanding. It is a full-stack application with a Python backend and a React/TypeScript frontend.
## Core stance
- Treat legacy code as liability, not as a compatibility target.
- Prefer deletion over shims, deprecated branches, wrapper APIs, and dual-track migration notes.
- If old and new implementations coexist, converge to one path unless an external contract forces compatibility.
- Remove dead tests, commented-out code, stale docs, and "move later" notes instead of preserving them.
- Reduce public surface area when a helper can be made private or internal.
- Keep refactors centered on the owning abstraction, not on adjacent compatibility layers.
- **Backend**: Python 3.10+ (Flask/Quart)
- **Frontend**: TypeScript, React, UmiJS
- **Architecture**: Microservices based on Docker.
- `api/`: Backend API server.
- `rag/`: Core RAG logic (indexing, retrieval).
- `deepdoc/`: Document parsing and OCR.
- `web/`: Frontend application.
## Current stack
- Backend: Python 3.13+, Quart-based API server, Peewee ORM, async workers.
- Frontend: React + TypeScript + Vite in `web/`.
- Go: the repository also has a substantial Go module for servers, ingestion, parser/runtime, CLI, and supporting services.
- Runtime services commonly include MySQL/PostgreSQL, Redis, MinIO, and Elasticsearch/Infinity/OpenSearch depending on configuration.
## 2. Directory Structure
- `api/`: Backend API server (Flask/Quart).
- `apps/`: API Blueprints (Knowledge Base, Chat, etc.).
- `db/`: Database models and services.
- `rag/`: Core RAG logic.
- `llm/`: LLM, Embedding, and Rerank model abstractions.
- `deepdoc/`: Document parsing and OCR modules.
- `agent/`: Agentic reasoning components.
- `web/`: Frontend application (React + UmiJS).
- `docker/`: Docker deployment configurations.
- `sdk/`: Python SDK.
- `test/`: Backend tests.
## Code layout to expect
- `api/`: Python API server entrypoints, blueprints, services, and database code.
- `rag/`: ingestion, retrieval, LLM integration, and graph RAG logic.
- `deepdoc/`: parsing and OCR.
- `agent/`: workflow canvas, components, tools, and templates.
- `cmd/`: Go entrypoints. `ragflow_main` is the main server/admin/ingestor binary surface; `ragflow-cli` is the CLI entrypoint.
- `internal/`: main Go application code. Important subtrees:
- `internal/agent/`: Go agent runtime, canvas execution, components, tool bindings, workflow helpers.
- `internal/cli/`: CLI parsing, HTTP transport, command execution, response formatting.
- `internal/dao/`: Go data-access layer and persistence-facing helpers.
- `internal/deepdoc/`: Go DeepDOC integrations, especially native-backed PDF/DOCX parsing.
- `internal/engine/`: search/index backends such as Elasticsearch and Infinity.
- `internal/entity/`: shared Go entities and model definitions.
- `internal/handler/`: HTTP handlers and route-facing request logic.
- `internal/ingestion/`: Go ingestion pipeline, canvas adapter, components, wiring, service orchestration.
- `internal/ingestion/component/`: stage implementations such as file/parser/chunker/tokenizer/extractor.
- `internal/ingestion/pipeline/`: DSL translation, canvas-driven execution, checkpoints, resume/run logic.
- `internal/parser/`: parser and chunk libraries used by ingestion and other Go paths.
- `internal/parser/parser/`: typed parse-result parsers for markdown/html/pdf/docx/xlsx/text and related families.
- `internal/parser/chunk/`: chunk operator library and DSL/typed execution helpers.
- `internal/service/`: higher-level business services used by handlers and server flows.
- `internal/storage/`: storage backends and in-memory test doubles.
- `internal/router/`: HTTP route registration.
- `internal/server/`: server bootstrap/config wiring.
- `internal/cpp/`: C++ sources used by native-backed Go features.
- `web/`: frontend application.
- `docker/`: local and production compose files.
- `sdk/` and `test/`: SDK and automated tests.
## 3. Build Instructions
## Go-specific rules
- Treat `internal/ingestion`, `internal/parser`, and `internal/deepdoc` as actively refactored code. Prefer collapsing duplicate paths over preserving transitional wrappers.
- Do not add or preserve deprecated Go APIs just to ease migration inside the repo.
- Remove commented-out Go code instead of leaving recovery notes in place.
- Keep package comments and doc comments aligned with the current runtime path, not with migration history.
### Backend (Python)
The project uses **uv** for dependency management.
## Working rules
- Before editing, inspect the nearest code path that actually owns the behavior.
- Keep changes small and local unless the task is explicitly a broader refactor.
- Prefer one implementation path instead of preserving old and new versions side by side.
- Preserve behavior with focused tests when the behavior is still valid; do not keep tests that protect obsolete behavior.
- If a surface is only there for compatibility, remove it unless the user asks to keep it.
- Do not add new compatibility wording in comments or docs.
- When a maintainer takes over a community PR, a new commit generated by rewriting history (e.g. `merge`, `rebase -i`) must preserve the original author and add the maintainer as co-author (via a `Co-authored-by:` trailer) instead of overwriting the author with the maintainer alone.
1. **Setup Environment**:
```bash
uv sync --python 3.13 --all-extras
uv run python3 ragflow_deps/download_deps.py
```
2. **Run Server**:
- **Pre-requisite**: Start dependent services (MySQL, ES/Infinity, Redis, MinIO).
```bash
docker compose -f docker/docker-compose-base.yml up -d
```
- **Launch**:
```bash
source .venv/bin/activate
export PYTHONPATH=$(pwd)
bash docker/launch_backend_service.sh
```
### Frontend (TypeScript/React)
Located in `web/`.
1. **Install Dependencies**:
```bash
cd web
npm install
```
2. **Run Dev Server**:
```bash
npm run dev
```
Runs on port 8000 by default.
### Docker Deployment
To run the full stack using Docker:
## Commands
### Backend
```bash
cd docker
docker compose -f docker-compose.yml up -d
uv sync --python 3.13 --all-extras
uv run python3 ragflow_deps/download_deps.py
docker compose -f docker/docker-compose-base.yml up -d
source .venv/bin/activate
export PYTHONPATH=$(pwd)
bash docker/launch_backend_service.sh
uv run pytest
ruff check
ruff format
```
## 4. Testing Instructions
### Frontend
```bash
cd web
npm install
npm run dev
npm run build
npm run lint
npm run test
npm run type-check
```
### Backend Tests
- **Run All Tests**:
```bash
uv run pytest
```
- **Run Specific Test**:
```bash
uv run pytest test/test_api.py
```
### Go
```bash
uv run ragflow_deps/download_deps.py
bash build.sh --test ./path/to/package/...
bash build.sh --go
# or build specific binaries:
bash build.sh --all
```
### Frontend Tests
- **Run Tests**:
```bash
cd web
npm run test
```
## Validation preference
- Run the narrowest relevant test, lint, or build command after a change.
- For backend changes, prefer targeted pytest or ruff checks over full-suite runs.
- For frontend changes, prefer the touched-package lint, type-check, or test command.
- For Go changes, prefer package-scoped `bash build.sh --test ...` first.
- Do not default to raw `go test`, `go build`, or IDE Run/Debug for Go in this repo. They often miss the required CGO flags and native static libraries (`office_oxide`, `pdfium-static`, `pdf_oxide`) that `build.sh` wires correctly.
- If Go native builds fail, inspect `build.sh` and `internal/development.md` before changing code. Common environment issues are missing downloaded native deps and missing `lld` on Linux.
## 5. Coding Standards & Guidelines
- **Python Formatting**: Use `ruff` for linting and formatting.
```bash
ruff check
ruff format
```
- **Frontend Linting**:
```bash
cd web
npm run lint
```
- **Git Hooks**: Run this once after the first clone to enable local Git hooks.
```bash
lefthook install
lefthook run pre-commit --all-files
```
## Default review checklist
- Remove instead of retaining `deprecated`, `legacy`, or compatibility-only code.
- Collapse duplicate implementations to one path.
- Drop stale comments and documentation that describe a superseded design.
- Keep exported APIs only when the current code actually needs them.

308
CLAUDE.md
View File

@@ -1,308 +0,0 @@
# CLAUDE.md
This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
## Project Overview
RAGFlow is an open-source RAG (Retrieval-Augmented Generation) engine based on deep document understanding. It's a full-stack application with:
- Python backend (Quart-based async API server — Quart is the async reimplementation of Flask)
- React/TypeScript frontend (built with vitejs)
- Background task executor workers (separate Python processes, Redis-queue-driven)
- Peewee ORM for database models (not SQLAlchemy)
- Multiple data stores (MySQL/PostgreSQL, Elasticsearch/Infinity/OpenSearch/OceanBase, Redis, MinIO)
## Architecture
### Runtime Architecture
RAGFlow runs as **two separate Python process types**, orchestrated by `docker/launch_backend_service.sh`:
- **API Server** (`api/ragflow_server.py`): Quart-based async HTTP server
- **Task Executors** (`rag/svr/task_executor.py`): Background workers processing documents from Redis streams. Multiple instances run in parallel (controlled by `WS` env var). Each consumes from priority-ordered Redis streams (`te.1.common`, `te.0.common`), using consumer groups for load distribution.
Key consequence: task executors import a different code surface than the API server, so always check which process a module is meant for.
### Backend API (`/api/`)
- **App factory**: `api/apps/__init__.py` — creates the Quart app, configures auth (`login_required` decorator, JWT + API token + session fallback), and dynamically discovers/registers blueprints
- **Two API coexisting patterns**:
- **RESTful APIs** in `api/apps/restful_apis/` — newer pattern with Pydantic request validation, service layer in `api/apps/services/`, routes registered under `/api/v1`
- **Legacy APIs** in `api/apps/*_app.py` — older pattern using `@validate_request()`, routes registered under `/v1/<page_name>`
- **SDK APIs** in `api/apps/sdk/` — registered under `/v1/`
- **Services**: `api/db/services/` — business logic wrapping Peewee model operations. `api/apps/services/` — service layer for the RESTful APIs
- **Models**: `api/db/db_models.py` — Peewee ORM models with pooled MySQL/PostgreSQL connections, custom `JSONField`/`ListField` types, retry logic on connection loss
### Core Processing (`/rag/`)
- **Document ingestion pipeline**: `rag/flow/pipeline.py``Pipeline` (extends `agent.canvas.Graph`) orchestrates the ingestion DAG. Components: File (fetches binary from storage), Parser (dispatches to `deepdoc.parser` based on file type), TokenChunker/TitleChunker (splits into chunks), Tokenizer (computes full-text tokens + embedding vectors), Extractor (LLM-based extraction). Data flows via Pydantic `*FromUpstream` schemas.
- **Document parsing**: `deepdoc/` — PDF parsing (vision-based OCR, layout analysis, table structure recognition) and format-specific parsers (DOCX, XLSX, PPT, Markdown, HTML, images). All parsers normalize to a common structure (list of bbox dicts for PDFs, `{text, doc_type_kwd}` for others).
- **DeepDoc HTTP API service** (`deepdoc/server/`): OSS ONNX models (DLA, OCR, TSR) wrapped with LitServe as a standalone HTTP API on port 8124. The Go parser (`internal/parser/`) calls this service via `DeepDocClient`. Endpoints: `GET /health`, `GET /model`, `POST /predict/dla`, `POST /predict/tsr`, `POST /predict/ocr` (with `operator=det` or `operator=rec` form field). Docker image: `deepdoc_oss:latest`. See `deepdoc/server/README.md` for the full API reference.
- **LLM Integration**: `rag/llm/` — factory pattern with runtime class discovery. `chat_model.py` (30+ providers via OpenAI SDK and LiteLLM wrappers), `embedding_model.py`, `rerank_model.py`, `cv_model.py` (image-to-text), `sequence2txt_model.py` (ASR), `tts_model.py`. Use `LLMBundle` (from `api.db.services.llm_service`) as the unified interface.
- **Graph RAG**: `rag/graphrag/` — multi-phase pipeline: per-document subgraph extraction (LLM or spaCy NER), Leiden community detection, entity resolution, community summarization. Entities/relations/reports are indexed as chunks alongside regular text chunks, differentiated by `knowledge_graph_kwd`.
- **Search**: `rag/nlp/search.py``Dealer` class combines vector similarity + BM25 + re-ranking. `KGSearch` extends it for graph-aware retrieval (entity resolution, n-hop enrichment).
### Agent System (`/agent/`)
- **Execution engine**: `agent/canvas.py``Canvas` (extends `Graph`) executes the DAG. Components are run in topological order via `_run_batch`, each receiving upstream outputs as kwargs. Control-flow components (`Categorize`, `Switch`, `Iteration`, `Loop`) dynamically modify the execution path.
- **Component base**: `agent/component/base.py``ComponentBase` with `invoke(**kwargs)` / `invoke_async(**kwargs)` lifecycle. Variable references (`{component_id@output_var}` or `{sys.query}`) are resolved from the canvas graph at runtime.
- **Components**: Modular workflow components in `agent/component/` — Begin, LLM, Agent (tool-calling LLM), Categorize, Switch, Iteration, Loop, Message, Invoke (HTTP), and data manipulation nodes. Auto-discovered by `__init__.py`.
- **Templates**: Pre-built agent workflows as JSON DSL files in `agent/templates/`. Each contains a complete `components` DAG, `path`, and `globals`.
- **Tools**: `agent/tools/` — Retrieval, web search (DuckDuckGo, Google, Tavily, SearXNG), academic search (ArXiv, PubMed, Google Scholar, Wikipedia), code execution, SQL execution, email, GitHub, finance data, translation, weather. Tools implement `ToolBase` (extends `ComponentBase`) and produce OpenAI-compatible function descriptors.
- **Plugins**: `agent/plugin/` — plugin system using `pluginlib` for loading external LLM tool plugins from `embedded_plugins/`.
### Frontend (`/web/`)
- React/TypeScript with vitejs framework
- shadcn/ui components (Radix UI primitives + Tailwind CSS)
- `@tanstack/react-query` for server state (cache keys, mutations, invalidation)
- Zustand for local state (primarily agent canvas graph store)
- `react-router` v7 with lazy-loaded pages
- `react-i18next` for i18n (17 languages)
- Axios for HTTP with a layered pattern: endpoint definitions (`utils/api.ts`) → HTTP client (`utils/next-request.ts`) → service layer (`services/`) → query hooks (`hooks/use-*-request.ts`) → components
- `@xyflow/react` for the agent workflow canvas
- `react-hook-form` + `zod` for form validation
- Two API proxy prefixes: `webAPI = '/v1'` (legacy) and `restAPIv1 = '/api/v1'` (RESTful)
## Common Development Commands
### Backend Development
```bash
# Install Python dependencies
uv sync --python 3.13 --all-extras
uv run python3 ragflow_deps/download_deps.py
# Run once after the first clone to enable local Git hooks
lefthook install
# Start dependent services
docker compose -f docker/docker-compose-base.yml up -d
# Run backend (requires services to be running)
source .venv/bin/activate
export PYTHONPATH=$(pwd)
bash docker/launch_backend_service.sh
# Run tests
uv run pytest
# Linting
ruff check
ruff format
```
### Frontend Development
```bash
cd web
npm install
npm run dev # Development server
npm run build # Production build
npm run lint # ESLint
npm run test # Jest tests
```
### Docker Development
```bash
# Full stack with Docker (includes deepdoc vision service)
cd docker
docker compose -f docker-compose.yml up -d
# Check server status
docker logs -f ragflow-server
# Build the OSS deepdoc vision service standalone
docker build -f docker/Dockerfile_deepdoc_oss -t deepdoc_oss:latest .
docker run -p 8124:8124 deepdoc_oss:latest
# Rebuild images
docker build --platform linux/amd64 -f Dockerfile -t infiniflow/ragflow:nightly .
```
## Key Configuration Files
- `docker/.env` - Environment variables for Docker deployment
- `docker/service_conf.yaml.template` - Backend service configuration
- `pyproject.toml` - Python dependencies and project configuration
- `web/package.json` - Frontend dependencies and scripts
## Testing
- **Python**: pytest with markers (p1/p2/p3 priority levels)
- **Frontend**: Jest with React Testing Library
- **API Tests**: HTTP API and SDK tests in `test/` and `sdk/python/test/`
## Database Engines
RAGFlow supports switching between Elasticsearch (default) and Infinity:
- Set `DOC_ENGINE=infinity` in `docker/.env` to use Infinity
- Requires container restart: `docker compose down -v && docker compose up -d`
## Account Password Handling (Critical for Login Flow)
### Password Encryption Pipeline (Browser → Backend → DB Hash)
The login password verification chain is counterintuitive. Understanding this is essential when generating or verifying password hashes.
**Complete flow:**
```
Browser input: "demo"
→ Base64("demo") = "ZGVtbw=="
→ RSA encrypt with conf/public.pem
→ POST to /api/v1/auth/login
Backend DecryptPassword():
→ RSA decrypt with conf/private.pem (passphrase: "Welcome")
→ Returns "ZGVtbw==" (NOT "demo"!)
VerifyPassword("ZGVtbw==", storedHash) ← hash is of Base64(password), not raw password
```
**Consequences:**
- The string verified against the hash is **Base64(original password)**, never the raw password
- `DecryptPassword()` handles both RSA-encrypted (browser) and plaintext (curl/API key) inputs: if base64 decode fails, the input is returned as-is for backward compatibility
- Python backend has the same design: `api/utils/crypt.py:decrypt()` RSA-decrypts and returns the Base64-encoded string directly, no further decode
### How to Generate a Valid Password Hash
```bash
# For password "demo" (user input in browser):
# The actual verified string = Base64("demo") = "ZGVtbw=="
# Generate hash with: common.GenerateWerkzeugPasswordHash("ZGVtbw==")
# or use the scrypt template:
# scrypt:32768:8:1$<random-b64-salt>$<hex-hash-of-ZGVtbw==>
```
**To update a user's password in the running database:**
```bash
docker exec docker-mysql-1 mysql -u root -pinfini_rag_flow rag_flow \
-e "UPDATE user SET password='<hash>' WHERE email='<email>';"
```
### RSA Keys
- `conf/public.pem` — frontend uses this to encrypt Base64(password) before sending
- `conf/private.pem` — backend uses this to decrypt, passphrase `"Welcome"`
- Both referenced in `internal/common/password.go:DecryptPassword()`
### Obtaining an API Token for a Tenant
When testing APIs manually (curl, Go scripts, etc.), you need a valid auth token. The login endpoint returns **two different tokens**:
| Field | Format | Purpose |
|-------|--------|---------|
| `response.body.data.access_token` | Raw UUID | Stored in DB, NOT used for API auth |
| `response.Header["Authorization"]` | itsdangerous-signed token | Used as `Bearer <token>` for all subsequent API requests |
**How to obtain the correct token:**
```bash
# Step 1: Construct the encrypted password
# Raw password → Base64 → RSA encrypt with conf/public.pem
PASSWORD="demo"
PASSWORD_B64=$(echo -n "$PASSWORD" | base64)
# Step 2: POST to login (use RSA encryption — easiest via a Go/Python script)
# Response header contains: Authorization: <itsdangerous-signed-token>
# Step 3: Use the Authorization header value for all API requests
curl -H "Authorization: <itsdangerous-signed-token>" \
http://127.0.0.1:9222/api/v1/agents
```
**Go snippet (complete login + token extraction):**
```go
// Login
passwordB64 := base64.StdEncoding.EncodeToString([]byte(password))
pubData, _ := os.ReadFile("conf/public.pem")
block, _ := pem.Decode(pubData)
pubKey, _ := x509.ParsePKIXPublicKey(block.Bytes)
ciphertext, _ := rsa.EncryptPKCS1v15(rand.Reader, pubKey.(*rsa.PublicKey), []byte(passwordB64))
encryptedB64 := base64.StdEncoding.EncodeToString(ciphertext)
body, _ := json.Marshal(map[string]string{"email": email, "password": encryptedB64})
resp, _ := http.Post(baseURL+"/api/v1/auth/login", "application/json", bytes.NewReader(body))
// KEY: use the Authorization header, NOT body.access_token
authToken := resp.Header.Get("Authorization")
// Use for API calls
req, _ := http.NewRequest("GET", baseURL+"/api/v1/agents", nil)
req.Header.Set("Authorization", authToken)
```
**The raw `access_token` (UUID) in the response body** is the internal DB token used only by the `itsdangerous` middleware to verify the signed token — it is never passed directly in API Authorization headers.
---
## Agent Run E2E Tests
### Running the Tests
```bash
# Run all agent run e2e tests (in-memory SQLite + miniredis, no Docker needed)
cd /home/zhichyu/github.com/infiniflow/ragflow
go test -count=1 -v -run 'TestRunAgent_RealCanvas|TestRunAgent_RunTracker' ./internal/service/
```
### Test Architecture
All e2e tests live in `internal/service/agent_run_e2e_test.go`. They exercise the full production chain:
```
loadCanvasForUser → versionDAO.GetLatest → decodeCanvasFromDSL →
canvas.Compile → cc.Workflow.Invoke → answer extraction
```
**Test isolation**: Each test stands up its own in-memory SQLite DB (pushed as `dao.DB`), seeds User/Tenant/UserCanvas/UserCanvasVersion rows, and tears down in `t.Cleanup`. Tests use **miniredis** for Redis-backed CheckPointStore + RunTracker — no external services needed.
**Key test helpers:**
- `makeCanvasWithDSL(t, canvasID, userID, tenantID, versionID, dsl)` — seeds all required DB rows
- `drainAgentEvents(t, events)` — drains the `<-chan canvas.RunEvent` channel, buckets results into `messages`, `waiting`, `errors_`, `done`
- `newRunTrackerForTest(t, ttl)` — wires a `canvas.RunTracker` against in-memory miniredis
**Existing e2e tests:**
| Test | What it covers |
|------|---------------|
| `TestRunAgent_RealCanvas_BeginMessage` | Happy path: Begin→Message, verifies `"{{sys.query}}"` resolution |
| `TestRunAgent_RealCanvas_WaitForUserResume` | Resume path: Begin→Message→UserFillUp, two-run cycle |
| `TestRunAgent_RealCanvas_CompileFails` | Error path: unknown component name → sanitized error |
| `TestRunAgent_RealCanvas_InvokeFails` | Error path: unresolvable template ref |
| `TestRunAgent_RunTracker_AttachCheckpoint_CallSequence` | Production boot: Start→AttachCheckpoint→MarkSucceeded with Redis/miniredis |
**Test DSL data files** are in `internal/agent/dsl/testdata/`:
- `agent_msg.json` — Agent+Message with Begin, LLM-powered agent component
- `all.json` — Complex: Begin→UserFillUp→Switch→Loop→Message
- `switch.json`, `resume.json`, `browser.json`, `subagent.json`, etc.
**Handler-level SSE streaming tests** in `internal/handler/agent_test.go` use a `stubChatRunner` that emits pre-configured `canvas.RunEvent` values without a real DB or eino runner, verifying:
- SSE `Content-Type: text/event-stream`
- `data: {...}\n\n` framing
- Trailing `data: [DONE]\n\n` terminator
- OpenAI-compatible non-stream `choices` response shape
**Important**: `_ "ragflow/internal/agent/component"` (blank import in test) is required — it triggers `init()` to register all component factories. Without it, `canvas.Compile` fails to resolve any component type.
---
## Development Environment Requirements
- Python 3.10-3.13
- Node.js >=18.20.4
- Docker & Docker Compose
- uv package manager
- 16GB+ RAM, 50GB+ disk space
1. Think before acting. Read existing files before writing code.
2. Be concise in output but thorough in reasoning.
3. Prefer editing over rewriting whole files.
4. Do not re-read files you have already read.
5. Test your code before declaring done.
6. No sycophantic openers or closing fluff.
7. Keep solutions simple and direct.
8. User instructions always override this file.

1
CLAUDE.md Symbolic link
View File

@@ -0,0 +1 @@
AGENTS.md

View File

@@ -59,7 +59,7 @@ RUN mkdir -p /usr/share/infinity/resource && \
cp -r /tmp/resource/* /usr/share/infinity/resource && \
rm -rf /tmp/resource
ARG NGINX_VERSION=1.31.0-1~noble
ARG NGINX_VERSION=1.31.2-1~noble
RUN --mount=type=cache,id=ragflow_apt,target=/var/cache/apt,sharing=locked \
mkdir -p /etc/apt/keyrings && \
curl --retry 5 --retry-delay 2 --retry-all-errors -fsSL https://nginx.org/keys/nginx_signing.key | gpg --dearmor -o /etc/apt/keyrings/nginx-archive-keyring.gpg && \

View File

@@ -1,6 +1,6 @@
<div align="center">
<a href="https://cloud.ragflow.io/">
<img src="web/src/assets/logo-with-text.svg" width="520" alt="ragflow logo">
<img src="https://raw.githubusercontent.com/infiniflow/ragflow/main/web/src/assets/logo-with-text.svg" width="520" alt="ragflow logo">
</a>
</div>
@@ -25,7 +25,7 @@
<img alt="Static Badge" src="https://img.shields.io/badge/Get-Started-4e6b99">
</a>
<a href="https://hub.docker.com/r/infiniflow/ragflow" target="_blank">
<img src="https://img.shields.io/docker/pulls/infiniflow/ragflow?label=Docker%20Pulls&color=0db7ed&logo=docker&logoColor=white&style=flat-square" alt="docker pull infiniflow/ragflow:v0.26.3">
<img src="https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/infiniflow/ragflow-stats/main/badges/docker-pulls.json&style=flat-square&logo=docker&logoColor=white" alt="docker pull infiniflow/ragflow:v0.26.4">
</a>
<a href="https://github.com/infiniflow/ragflow/releases/latest">
<img src="https://img.shields.io/github/v/release/infiniflow/ragflow?color=blue&label=Latest%20Release" alt="Latest Release">
@@ -193,12 +193,12 @@ releases! 🌟
> All Docker images are built for x86 platforms. We don't currently offer Docker images for ARM64.
> If you are on an ARM64 platform, follow [this guide](https://ragflow.io/docs/dev/build_docker_image) to build a Docker image compatible with your system.
> The command below downloads the `v0.26.3` edition of the RAGFlow Docker image. See the following table for descriptions of different RAGFlow editions. To download a RAGFlow edition different from `v0.26.3`, update the `RAGFLOW_IMAGE` variable accordingly in **docker/.env** before using `docker compose` to start the server.
> The command below downloads the `v0.26.4` edition of the RAGFlow Docker image. See the following table for descriptions of different RAGFlow editions. To download a RAGFlow edition different from `v0.26.4`, update the `RAGFLOW_IMAGE` variable accordingly in **docker/.env** before using `docker compose` to start the server.
```bash
$ cd ragflow/docker
# git checkout v0.26.3
git checkout v0.26.4
# Optional: use a stable tag (see releases: https://github.com/infiniflow/ragflow/releases)
# This step ensures the **entrypoint.sh** file in the code matches the Docker image version.
@@ -320,7 +320,7 @@ docker build --platform linux/amd64 \
## 🔨 Launch service from source for development
> [!IMPORTANT]
> After cloning the repository for the first time, run `lefthook install` once from the repo root to enable local Git hooks.
> After cloning the repository for the first time, run `git config --local --unset core.hooksPath`, `uv tool install lefthook` and `lefthook install` once from the repo root to enable local Git hooks.
1. Install `uv`, or skip this step if it is already installed:
@@ -334,6 +334,8 @@ docker build --platform linux/amd64 \
cd ragflow/
uv sync --python 3.13 # install RAGFlow dependent python modules
uv run python3 ragflow_deps/download_deps.py
git config --local --unset core.hooksPath
uv tool install lefthook
lefthook install
```
3. Launch the dependent services (MinIO, Elasticsearch, Redis, and MySQL) using Docker Compose:

View File

@@ -1,6 +1,6 @@
<div align="center">
<a href="https://cloud.ragflow.io/">
<img src="web/src/assets/logo-with-text.svg" width="520" alt="ragflow logo">
<img src="https://raw.githubusercontent.com/infiniflow/ragflow/main/web/src/assets/logo-with-text.svg" width="520" alt="ragflow logo">
</a>
</div>
@@ -25,7 +25,7 @@
<img alt="Static Badge" src="https://img.shields.io/badge/Get-Started-4e6b99">
</a>
<a href="https://hub.docker.com/r/infiniflow/ragflow" target="_blank">
<img src="https://img.shields.io/docker/pulls/infiniflow/ragflow?label=Docker%20Pulls&color=0db7ed&logo=docker&logoColor=white&style=flat-square" alt="docker pull infiniflow/ragflow:v0.26.3">
<img src="https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/infiniflow/ragflow-stats/main/badges/docker-pulls.json&style=flat-square&logo=docker&logoColor=white" alt="docker pull infiniflow/ragflow:v0.26.4">
</a>
<a href="https://github.com/infiniflow/ragflow/releases/latest">
<img src="https://img.shields.io/github/v/release/infiniflow/ragflow?color=blue&label=Latest%20Release" alt="Latest Release">
@@ -193,12 +193,12 @@
> جميع الصور Docker مصممة لمنصات x86. لا نعرض حاليًا صور Docker لـ ARM64.
> إذا كنت تستخدم نظامًا أساسيًا ARM64، فاتبع [هذا الدليل](https://ragflow.io/docs/dev/build_docker_image) لإنشاء صورة Docker متوافقة مع نظامك.
> يقوم الأمر أدناه بتنزيل إصدار `v0.26.3` من الصورة RAGFlow Docker. راجع الجدول التالي للحصول على أوصاف لإصدارات RAGFlow المختلفة. لتنزيل إصدار RAGFlow مختلف عن `v0.26.3`، قم بتحديث المتغير `RAGFLOW_IMAGE` وفقًا لذلك في **docker/.env** قبل استخدام `docker compose` لبدء تشغيل الخادم.
> يقوم الأمر أدناه بتنزيل إصدار `v0.26.4` من الصورة RAGFlow Docker. راجع الجدول التالي للحصول على أوصاف لإصدارات RAGFlow المختلفة. لتنزيل إصدار RAGFlow مختلف عن `v0.26.4`، قم بتحديث المتغير `RAGFLOW_IMAGE` وفقًا لذلك في **docker/.env** قبل استخدام `docker compose` لبدء تشغيل الخادم.
```bash
$ cd ragflow/docker
# git checkout v0.26.3
git checkout v0.26.4
# Optional: use a stable tag (see releases: https://github.com/infiniflow/ragflow/releases)
# This step ensures the **entrypoint.sh** file in the code matches the Docker image version.
@@ -331,6 +331,8 @@ docker build --platform linux/amd64 \
cd ragflow/
uv sync --python 3.13 # install RAGFlow dependent python modules
uv run python3 ragflow_deps/download_deps.py
git config --local --unset core.hooksPath
uv tool install lefthook
lefthook install
```
3. قم بتشغيل الخدمات التابعة (MinIO وElasticsearch وRedis وMySQL) باستخدام Docker Compose:

View File

@@ -1,6 +1,6 @@
<div align="center">
<a href="https://cloud.ragflow.io/">
<img src="web/src/assets/logo-with-text.svg" width="520" alt="ragflow logo">
<img src="https://raw.githubusercontent.com/infiniflow/ragflow/main/web/src/assets/logo-with-text.svg" width="520" alt="ragflow logo">
</a>
</div>
@@ -25,10 +25,10 @@
<img alt="Badge statique" src="https://img.shields.io/badge/Get-Started-4e6b99">
</a>
<a href="https://hub.docker.com/r/infiniflow/ragflow" target="_blank">
<img src="https://img.shields.io/docker/pulls/infiniflow/ragflow?label=Docker%20Pulls&color=0db7ed&logo=docker&logoColor=white&style=flat-square" alt="docker pull infiniflow/ragflow:v0.26.3">
<img src="https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/infiniflow/ragflow-stats/main/badges/docker-pulls.json&style=flat-square&logo=docker&logoColor=white" alt="docker pull infiniflow/ragflow:v0.26.4">
</a>
<a href="https://github.com/infiniflow/ragflow/releases/latest">
<img src="https://img.shields.io/github/v/release/infiniflow/ragflow?color=blue&label=Dernière%20version" alt="Dernière version">
<img src="https://img.shields.io/github/v/release/infiniflow/ragflow?color=blue&label=Derniere%20version" alt="Dernière version">
</a>
<a href="https://github.com/infiniflow/ragflow/blob/main/LICENSE">
<img height="21" src="https://img.shields.io/badge/License-Apache--2.0-ffffff?labelColor=d4eaf7&color=2e6cc4" alt="licence">
@@ -190,12 +190,12 @@ Essayez notre service cloud sur [https://cloud.ragflow.io](https://cloud.ragflow
> Toutes les images Docker sont construites pour les plateformes x86. Nous ne proposons pas actuellement d'images Docker pour ARM64.
> Si vous êtes sur une plateforme ARM64, suivez [ce guide](https://ragflow.io/docs/dev/build_docker_image) pour construire une image Docker compatible avec votre système.
> La commande ci-dessous télécharge l'édition `v0.26.3` de l'image Docker RAGFlow. Consultez le tableau suivant pour les descriptions des différentes éditions de RAGFlow. Pour télécharger une édition de RAGFlow différente de `v0.26.3`, mettez à jour la variable `RAGFLOW_IMAGE` dans **docker/.env** avant d'utiliser `docker compose` pour démarrer le serveur.
> La commande ci-dessous télécharge l'édition `v0.26.4` de l'image Docker RAGFlow. Consultez le tableau suivant pour les descriptions des différentes éditions de RAGFlow. Pour télécharger une édition de RAGFlow différente de `v0.26.4`, mettez à jour la variable `RAGFLOW_IMAGE` dans **docker/.env** avant d'utiliser `docker compose` pour démarrer le serveur.
```bash
$ cd ragflow/docker
# git checkout v0.26.3
git checkout v0.26.4
# Optionnel : utiliser un tag stable (voir les versions : https://github.com/infiniflow/ragflow/releases)
# Cette étape garantit que le fichier **entrypoint.sh** dans le code correspond à la version de l'image Docker.
@@ -322,6 +322,8 @@ docker build --platform linux/amd64 \
cd ragflow/
uv sync --python 3.13 # install RAGFlow dependent python modules
uv run python3 ragflow_deps/download_deps.py
git config --local --unset core.hooksPath
uv tool install lefthook
lefthook install
```
3. Lancez les services dépendants (MinIO, Elasticsearch, Redis et MySQL) avec Docker Compose :

View File

@@ -1,6 +1,6 @@
<div align="center">
<a href="https://cloud.ragflow.io/">
<img src="web/src/assets/logo-with-text.svg" width="520" alt="Logo ragflow">
<img src="https://raw.githubusercontent.com/infiniflow/ragflow/main/web/src/assets/logo-with-text.svg" width="520" alt="Logo ragflow">
</a>
</div>
@@ -25,7 +25,7 @@
<img alt="Lencana Daring" src="https://img.shields.io/badge/Get-Started-4e6b99">
</a>
<a href="https://hub.docker.com/r/infiniflow/ragflow" target="_blank">
<img src="https://img.shields.io/docker/pulls/infiniflow/ragflow?label=Docker%20Pulls&color=0db7ed&logo=docker&logoColor=white&style=flat-square" alt="docker pull infiniflow/ragflow:v0.26.3">
<img src="https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/infiniflow/ragflow-stats/main/badges/docker-pulls.json&style=flat-square&logo=docker&logoColor=white" alt="docker pull infiniflow/ragflow:v0.26.4">
</a>
<a href="https://github.com/infiniflow/ragflow/releases/latest">
<img src="https://img.shields.io/github/v/release/infiniflow/ragflow?color=blue&label=Rilis%20Terbaru" alt="Rilis Terbaru">
@@ -191,12 +191,12 @@ Coba layanan cloud kami di [https://cloud.ragflow.io](https://cloud.ragflow.io).
> Semua gambar Docker dibangun untuk platform x86. Saat ini, kami tidak menawarkan gambar Docker untuk ARM64.
> Jika Anda menggunakan platform ARM64, [silakan gunakan panduan ini untuk membangun gambar Docker yang kompatibel dengan sistem Anda](https://ragflow.io/docs/dev/build_docker_image).
> Perintah di bawah ini mengunduh edisi v0.26.3 dari gambar Docker RAGFlow. Silakan merujuk ke tabel berikut untuk deskripsi berbagai edisi RAGFlow. Untuk mengunduh edisi RAGFlow yang berbeda dari v0.26.3, perbarui variabel RAGFLOW_IMAGE di docker/.env sebelum menggunakan docker compose untuk memulai server.
> Perintah di bawah ini mengunduh edisi v0.26.4 dari gambar Docker RAGFlow. Silakan merujuk ke tabel berikut untuk deskripsi berbagai edisi RAGFlow. Untuk mengunduh edisi RAGFlow yang berbeda dari v0.26.4, perbarui variabel RAGFLOW_IMAGE di docker/.env sebelum menggunakan docker compose untuk memulai server.
```bash
$ cd ragflow/docker
# git checkout v0.26.3
git checkout v0.26.4
# Opsional: gunakan tag stabil (lihat releases: https://github.com/infiniflow/ragflow/releases)
# This steps ensures the **entrypoint.sh** file in the code matches the Docker image version.
@@ -303,6 +303,8 @@ docker build --platform linux/amd64 \
cd ragflow/
uv sync --python 3.13 # install RAGFlow dependent python modules
uv run python3 ragflow_deps/download_deps.py
git config --local --unset core.hooksPath
uv tool install lefthook
lefthook install
```
3. Jalankan aplikasi yang diperlukan (MinIO, Elasticsearch, Redis, dan MySQL) menggunakan Docker Compose:

View File

@@ -1,6 +1,6 @@
<div align="center">
<a href="https://cloud.ragflow.io/">
<img src="web/src/assets/logo-with-text.svg" width="350" alt="ragflow logo">
<img src="https://raw.githubusercontent.com/infiniflow/ragflow/main/web/src/assets/logo-with-text.svg" width="520" alt="ragflow logo">
</a>
</div>
@@ -25,7 +25,7 @@
<img alt="Static Badge" src="https://img.shields.io/badge/Get-Started-4e6b99">
</a>
<a href="https://hub.docker.com/r/infiniflow/ragflow" target="_blank">
<img src="https://img.shields.io/docker/pulls/infiniflow/ragflow?label=Docker%20Pulls&color=0db7ed&logo=docker&logoColor=white&style=flat-square" alt="docker pull infiniflow/ragflow:v0.26.3">
<img src="https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/infiniflow/ragflow-stats/main/badges/docker-pulls.json&style=flat-square&logo=docker&logoColor=white" alt="docker pull infiniflow/ragflow:v0.26.4">
</a>
<a href="https://github.com/infiniflow/ragflow/releases/latest">
<img src="https://img.shields.io/github/v/release/infiniflow/ragflow?color=blue&label=Latest%20Release" alt="Latest Release">
@@ -172,12 +172,12 @@
> 現在、公式に提供されているすべての Docker イメージは x86 アーキテクチャ向けにビルドされており、ARM64 用の Docker イメージは提供されていません。
> ARM64 アーキテクチャのオペレーティングシステムを使用している場合は、[このドキュメント](https://ragflow.io/docs/dev/build_docker_image)を参照して Docker イメージを自分でビルドしてください。
> 以下のコマンドは、RAGFlow Docker イメージの v0.26.3 エディションをダウンロードします。異なる RAGFlow エディションの説明については、以下の表を参照してください。v0.26.3 とは異なるエディションをダウンロードするには、docker/.env ファイルの RAGFLOW_IMAGE 変数を適宜更新し、docker compose を使用してサーバーを起動してください。
> 以下のコマンドは、RAGFlow Docker イメージの v0.26.4 エディションをダウンロードします。異なる RAGFlow エディションの説明については、以下の表を参照してください。v0.26.4 とは異なるエディションをダウンロードするには、docker/.env ファイルの RAGFLOW_IMAGE 変数を適宜更新し、docker compose を使用してサーバーを起動してください。
```bash
$ cd ragflow/docker
# git checkout v0.26.3
git checkout v0.26.4
# 任意: 安定版タグを利用 (一覧: https://github.com/infiniflow/ragflow/releases)
# この手順は、コード内の entrypoint.sh ファイルが Docker イメージのバージョンと一致していることを確認します。
@@ -304,6 +304,8 @@ docker build --platform linux/amd64 \
cd ragflow/
uv sync --python 3.13 # install RAGFlow dependent python modules
uv run python3 ragflow_deps/download_deps.py
git config --local --unset core.hooksPath
uv tool install lefthook
lefthook install
```
3. Docker Compose を使用して依存サービスMinIO、Elasticsearch、Redis、MySQLを起動する:

View File

@@ -1,6 +1,6 @@
<div align="center">
<a href="https://cloud.ragflow.io/">
<img src="web/src/assets/logo-with-text.svg" width="520" alt="ragflow logo">
<img src="https://raw.githubusercontent.com/infiniflow/ragflow/main/web/src/assets/logo-with-text.svg" width="520" alt="ragflow logo">
</a>
</div>
@@ -25,7 +25,7 @@
<img alt="Static Badge" src="https://img.shields.io/badge/Get-Started-4e6b99">
</a>
<a href="https://hub.docker.com/r/infiniflow/ragflow" target="_blank">
<img src="https://img.shields.io/docker/pulls/infiniflow/ragflow?label=Docker%20Pulls&color=0db7ed&logo=docker&logoColor=white&style=flat-square" alt="docker pull infiniflow/ragflow:v0.26.3">
<img src="https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/infiniflow/ragflow-stats/main/badges/docker-pulls.json&style=flat-square&logo=docker&logoColor=white" alt="docker pull infiniflow/ragflow:v0.26.4">
</a>
<a href="https://github.com/infiniflow/ragflow/releases/latest">
<img src="https://img.shields.io/github/v/release/infiniflow/ragflow?color=blue&label=Latest%20Release" alt="Latest Release">
@@ -174,12 +174,12 @@
> 모든 Docker 이미지는 x86 플랫폼을 위해 빌드되었습니다. 우리는 현재 ARM64 플랫폼을 위한 Docker 이미지를 제공하지 않습니다.
> ARM64 플랫폼을 사용 중이라면, [시스템과 호환되는 Docker 이미지를 빌드하려면 이 가이드를 사용해 주세요](https://ragflow.io/docs/dev/build_docker_image).
> 아래 명령어는 RAGFlow Docker 이미지의 v0.26.3 버전을 다운로드합니다. 다양한 RAGFlow 버전에 대한 설명은 다음 표를 참조하십시오. v0.26.3와 다른 RAGFlow 버전을 다운로드하려면, docker/.env 파일에서 RAGFLOW_IMAGE 변수를 적절히 업데이트한 후 docker compose를 사용하여 서버를 시작하십시오.
> 아래 명령어는 RAGFlow Docker 이미지의 v0.26.4 버전을 다운로드합니다. 다양한 RAGFlow 버전에 대한 설명은 다음 표를 참조하십시오. v0.26.4와 다른 RAGFlow 버전을 다운로드하려면, docker/.env 파일에서 RAGFLOW_IMAGE 변수를 적절히 업데이트한 후 docker compose를 사용하여 서버를 시작하십시오.
```bash
$ cd ragflow/docker
# git checkout v0.26.3
git checkout v0.26.4
# Optional: use a stable tag (see releases: https://github.com/infiniflow/ragflow/releases)
# 이 단계는 코드의 entrypoint.sh 파일이 Docker 이미지 버전과 일치하도록 보장합니다.
@@ -299,6 +299,8 @@ docker build --platform linux/amd64 \
cd ragflow/
uv sync --python 3.13 # install RAGFlow dependent python modules
uv run python3 ragflow_deps/download_deps.py
git config --local --unset core.hooksPath
uv tool install lefthook
lefthook install
```

View File

@@ -1,6 +1,6 @@
<div align="center">
<a href="https://cloud.ragflow.io/">
<img src="web/src/assets/logo-with-text.svg" width="520" alt="ragflow logo">
<img src="https://raw.githubusercontent.com/infiniflow/ragflow/main/web/src/assets/logo-with-text.svg" width="520" alt="ragflow logo">
</a>
</div>
@@ -25,10 +25,10 @@
<img alt="Badge Estático" src="https://img.shields.io/badge/Get-Started-4e6b99">
</a>
<a href="https://hub.docker.com/r/infiniflow/ragflow" target="_blank">
<img src="https://img.shields.io/docker/pulls/infiniflow/ragflow?label=Docker%20Pulls&color=0db7ed&logo=docker&logoColor=white&style=flat-square" alt="docker pull infiniflow/ragflow:v0.26.3">
<img src="https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/infiniflow/ragflow-stats/main/badges/docker-pulls.json&style=flat-square&logo=docker&logoColor=white" alt="docker pull infiniflow/ragflow:v0.26.4">
</a>
<a href="https://github.com/infiniflow/ragflow/releases/latest">
<img src="https://img.shields.io/github/v/release/infiniflow/ragflow?color=blue&label=Última%20Relese" alt="Última Versão">
<img src="https://img.shields.io/github/v/release/infiniflow/ragflow?color=blue&label=%C3%9Altima%20Release" alt="Última Release">
</a>
<a href="https://github.com/infiniflow/ragflow/blob/main/LICENSE">
<img height="21" src="https://img.shields.io/badge/License-Apache--2.0-ffffff?labelColor=d4eaf7&color=2e6cc4" alt="licença">
@@ -191,12 +191,12 @@ Experimente o nosso serviço na nuvem em [https://cloud.ragflow.io](https://clou
> Todas as imagens Docker são construídas para plataformas x86. Atualmente, não oferecemos imagens Docker para ARM64.
> Se você estiver usando uma plataforma ARM64, por favor, utilize [este guia](https://ragflow.io/docs/dev/build_docker_image) para construir uma imagem Docker compatível com o seu sistema.
> O comando abaixo baixa a edição`v0.26.3` da imagem Docker do RAGFlow. Consulte a tabela a seguir para descrições de diferentes edições do RAGFlow. Para baixar uma edição do RAGFlow diferente da `v0.26.3`, atualize a variável `RAGFLOW_IMAGE` conforme necessário no **docker/.env** antes de usar `docker compose` para iniciar o servidor.
> O comando abaixo baixa a edição`v0.26.4` da imagem Docker do RAGFlow. Consulte a tabela a seguir para descrições de diferentes edições do RAGFlow. Para baixar uma edição do RAGFlow diferente da `v0.26.4`, atualize a variável `RAGFLOW_IMAGE` conforme necessário no **docker/.env** antes de usar `docker compose` para iniciar o servidor.
```bash
$ cd ragflow/docker
# git checkout v0.26.3
git checkout v0.26.4
# Opcional: use uma tag estável (veja releases: https://github.com/infiniflow/ragflow/releases)
# Esta etapa garante que o arquivo entrypoint.sh no código corresponda à versão da imagem do Docker.
@@ -320,6 +320,8 @@ docker build --platform linux/amd64 \
cd ragflow/
uv sync --python 3.13 # instala os módulos Python dependentes do RAGFlow
uv run python3 ragflow_deps/download_deps.py
git config --local --unset core.hooksPath
uv tool install lefthook
lefthook install
```
3. Inicie os serviços dependentes (MinIO, Elasticsearch, Redis e MySQL) usando Docker Compose:

View File

@@ -1,6 +1,6 @@
<div align="center">
<a href="https://cloud.ragflow.io/">
<img src="web/src/assets/logo-with-text.svg" width="520" alt="ragflow logo">
<img src="https://raw.githubusercontent.com/infiniflow/ragflow/main/web/src/assets/logo-with-text.svg" width="520" alt="ragflow logo">
</a>
</div>
@@ -25,10 +25,10 @@
<img alt="Çevrimiçi Demo" src="https://img.shields.io/badge/Get-Started-4e6b99">
</a>
<a href="https://hub.docker.com/r/infiniflow/ragflow" target="_blank">
<img src="https://img.shields.io/docker/pulls/infiniflow/ragflow?label=Docker%20Pulls&color=0db7ed&logo=docker&logoColor=white&style=flat-square" alt="docker pull infiniflow/ragflow:v0.26.3">
<img src="https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/infiniflow/ragflow-stats/main/badges/docker-pulls.json&style=flat-square&logo=docker&logoColor=white" alt="docker pull infiniflow/ragflow:v0.26.4">
</a>
<a href="https://github.com/infiniflow/ragflow/releases/latest">
<img src="https://img.shields.io/github/v/release/infiniflow/ragflow?color=blue&label=Son%20Sürüm" alt="Son Sürüm">
<img src="https://img.shields.io/github/v/release/infiniflow/ragflow?color=blue&label=Son%20S%C3%BCr%C3%BCm" alt="Son Sürüm">
</a>
<a href="https://github.com/infiniflow/ragflow/blob/main/LICENSE">
<img height="21" src="https://img.shields.io/badge/Lisans-Apache--2.0-ffffff?labelColor=d4eaf7&color=2e6cc4" alt="lisans">
@@ -191,12 +191,12 @@ Bulut hizmetimizi [https://cloud.ragflow.io](https://cloud.ragflow.io) adresinde
> Tüm Docker imajları x86 platformları için oluşturulmuştur. Şu anda ARM64 için Docker imajı sunmuyoruz.
> ARM64 platformundaysanız, sisteminizle uyumlu bir Docker imajı oluşturmak için [bu kılavuzu](https://ragflow.io/docs/dev/build_docker_image) takip edin.
> Aşağıdaki komut RAGFlow Docker imajının `v0.26.3` sürümünü indirir. Farklı RAGFlow sürümleri için aşağıdaki tabloya bakın. `v0.26.3` dışında bir sürüm indirmek için, `docker compose` ile sunucuyu başlatmadan önce **docker/.env** dosyasındaki `RAGFLOW_IMAGE` değişkenini güncelleyin.
> Aşağıdaki komut RAGFlow Docker imajının `v0.26.4` sürümünü indirir. Farklı RAGFlow sürümleri için aşağıdaki tabloya bakın. `v0.26.4` dışında bir sürüm indirmek için, `docker compose` ile sunucuyu başlatmadan önce **docker/.env** dosyasındaki `RAGFLOW_IMAGE` değişkenini güncelleyin.
```bash
$ cd ragflow/docker
# git checkout v0.26.3
git checkout v0.26.4
# İsteğe bağlı: Kararlı bir etiket kullanın (sürümler: https://github.com/infiniflow/ragflow/releases)
# Bu adım, koddaki **entrypoint.sh** dosyasının Docker imaj sürümüyle eşleşmesini sağlar.
@@ -326,6 +326,8 @@ docker build --platform linux/amd64 \
cd ragflow/
uv sync --python 3.13 # RAGFlow'un bağımlı Python modüllerini yükler
uv run python3 ragflow_deps/download_deps.py
git config --local --unset core.hooksPath
uv tool install lefthook
lefthook install
```
3. Bağımlı hizmetleri (MinIO, Elasticsearch, Redis ve MySQL) Docker Compose kullanarak başlatın:

View File

@@ -1,6 +1,6 @@
<div align="center">
<a href="https://cloud.ragflow.io/">
<img src="web/src/assets/logo-with-text.svg" width="350" alt="ragflow logo">
<img src="https://raw.githubusercontent.com/infiniflow/ragflow/main/web/src/assets/logo-with-text.svg" width="520" alt="ragflow logo">
</a>
</div>
@@ -25,7 +25,7 @@
<img alt="Static Badge" src="https://img.shields.io/badge/Get-Started-4e6b99">
</a>
<a href="https://hub.docker.com/r/infiniflow/ragflow" target="_blank">
<img src="https://img.shields.io/docker/pulls/infiniflow/ragflow?label=Docker%20Pulls&color=0db7ed&logo=docker&logoColor=white&style=flat-square" alt="docker pull infiniflow/ragflow:v0.26.3">
<img src="https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/infiniflow/ragflow-stats/main/badges/docker-pulls.json&style=flat-square&logo=docker&logoColor=white" alt="docker pull infiniflow/ragflow:v0.26.4">
</a>
<a href="https://github.com/infiniflow/ragflow/releases/latest">
<img src="https://img.shields.io/github/v/release/infiniflow/ragflow?color=blue&label=Latest%20Release" alt="Latest Release">
@@ -191,12 +191,12 @@
> 所有 Docker 映像檔都是為 x86 平台建置的。目前,我們不提供 ARM64 平台的 Docker 映像檔。
> 如果您使用的是 ARM64 平台,請使用 [這份指南](https://ragflow.io/docs/dev/build_docker_image) 來建置適合您系統的 Docker 映像檔。
> 執行以下指令會自動下載 RAGFlow Docker 映像 `v0.26.3`。請參考下表查看不同 Docker 發行版的說明。如需下載不同於 `v0.26.3` 的 Docker 映像,請在執行 `docker compose` 啟動服務之前先更新 **docker/.env** 檔案內的 `RAGFLOW_IMAGE` 變數。
> 執行以下指令會自動下載 RAGFlow Docker 映像 `v0.26.4`。請參考下表查看不同 Docker 發行版的說明。如需下載不同於 `v0.26.4` 的 Docker 映像,請在執行 `docker compose` 啟動服務之前先更新 **docker/.env** 檔案內的 `RAGFLOW_IMAGE` 變數。
```bash
$ cd ragflow/docker
# git checkout v0.26.3
git checkout v0.26.4
# 可選使用穩定版標籤查看發佈https://github.com/infiniflow/ragflow/releases
# 此步驟確保程式碼中的 entrypoint.sh 檔案與 Docker 映像版本一致。
@@ -331,6 +331,8 @@ docker build --platform linux/amd64 \
cd ragflow/
uv sync --python 3.13 # install RAGFlow dependent python modules
uv run python3 ragflow_deps/download_deps.py
git config --local --unset core.hooksPath
uv tool install lefthook
lefthook install
```
3. 透過 Docker Compose 啟動依賴的服務MinIO, Elasticsearch, Redis, and MySQL

View File

@@ -1,6 +1,6 @@
<div align="center">
<a href="https://cloud.ragflow.io/">
<img src="web/src/assets/logo-with-text.svg" width="350" alt="ragflow logo">
<img src="https://raw.githubusercontent.com/infiniflow/ragflow/main/web/src/assets/logo-with-text.svg" width="520" alt="ragflow logo">
</a>
</div>
@@ -25,7 +25,7 @@
<img alt="Static Badge" src="https://img.shields.io/badge/Get-Started-4e6b99">
</a>
<a href="https://hub.docker.com/r/infiniflow/ragflow" target="_blank">
<img src="https://img.shields.io/docker/pulls/infiniflow/ragflow?label=Docker%20Pulls&color=0db7ed&logo=docker&logoColor=white&style=flat-square" alt="docker pull infiniflow/ragflow:v0.26.3">
<img src="https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/infiniflow/ragflow-stats/main/badges/docker-pulls.json&style=flat-square&logo=docker&logoColor=white" alt="docker pull infiniflow/ragflow:v0.26.4">
</a>
<a href="https://github.com/infiniflow/ragflow/releases/latest">
<img src="https://img.shields.io/github/v/release/infiniflow/ragflow?color=blue&label=Latest%20Release" alt="Latest Release">
@@ -192,12 +192,12 @@
> 请注意,目前官方提供的所有 Docker 镜像均基于 x86 架构构建,并不提供基于 ARM64 的 Docker 镜像。
> 如果你的操作系统是 ARM64 架构,请参考[这篇文档](https://ragflow.io/docs/dev/build_docker_image)自行构建 Docker 镜像。
> 运行以下命令会自动下载 RAGFlow Docker 镜像 `v0.26.3`。请参考下表查看不同 Docker 发行版的描述。如需下载不同于 `v0.26.3` 的 Docker 镜像,请在运行 `docker compose` 启动服务之前先更新 **docker/.env** 文件内的 `RAGFLOW_IMAGE` 变量。
> 运行以下命令会自动下载 RAGFlow Docker 镜像 `v0.26.4`。请参考下表查看不同 Docker 发行版的描述。如需下载不同于 `v0.26.4` 的 Docker 镜像,请在运行 `docker compose` 启动服务之前先更新 **docker/.env** 文件内的 `RAGFLOW_IMAGE` 变量。
```bash
$ cd ragflow/docker
# git checkout v0.26.3
git checkout v0.26.4
# 可选使用稳定版本标签查看发布https://github.com/infiniflow/ragflow/releases
# 这一步确保代码中的 entrypoint.sh 文件与 Docker 镜像的版本保持一致。
@@ -331,6 +331,8 @@ docker build --platform linux/amd64 \
cd ragflow/
uv sync --python 3.13 # install RAGFlow dependent python modules
uv run python3 ragflow_deps/download_deps.py
git config --local --unset core.hooksPath
uv tool install lefthook
lefthook install
```

View File

@@ -28,7 +28,7 @@ It consists of a server-side Service and a command-line client (CLI), both imple
```bash
python admin/server/admin_server.py
```
The service will start and listen for incoming connections from the CLI on the configured port.
The service will start and listen for incoming connections from the CLI on the configured port.
#### Using docker image
@@ -48,7 +48,7 @@ It consists of a server-side Service and a command-line client (CLI), both imple
1. Ensure the Admin Service is running.
2. Install ragflow-cli.
```bash
pip install ragflow-cli==0.26.3
pip install ragflow-cli==0.26.4
```
3. Launch the CLI client:
```bash
@@ -58,9 +58,9 @@ It consists of a server-side Service and a command-line client (CLI), both imple
The default password is admin.
**Parameters:**
- -h: RAGFlow admin server host address
- -p: RAGFlow admin server port

View File

@@ -25,14 +25,14 @@ import requests
class HttpClient:
def __init__(
self,
host: str = "127.0.0.1",
port: int = 9381,
api_version: str = "v1",
api_key: Optional[str] = None,
connect_timeout: float = 5.0,
read_timeout: float = 60.0,
verify_ssl: bool = False,
self,
host: str = "127.0.0.1",
port: int = 9381,
api_version: str = "v1",
api_key: Optional[str] = None,
connect_timeout: float = 5.0,
read_timeout: float = 60.0,
verify_ssl: bool = False,
) -> None:
self.host = host
self.port = port
@@ -71,19 +71,19 @@ class HttpClient:
return headers
def request(
self,
method: str,
path: str,
*,
use_api_base: bool = True,
auth_kind: Optional[str] = "api",
headers: Optional[Dict[str, str]] = None,
json_body: Optional[Dict[str, Any]] = None,
data: Any = None,
files: Any = None,
params: Optional[Dict[str, Any]] = None,
stream: bool = False,
iterations: int = 1,
self,
method: str,
path: str,
*,
use_api_base: bool = True,
auth_kind: Optional[str] = "api",
headers: Optional[Dict[str, str]] = None,
json_body: Optional[Dict[str, Any]] = None,
data: Any = None,
files: Any = None,
params: Optional[Dict[str, Any]] = None,
stream: bool = False,
iterations: int = 1,
) -> requests.Response | dict:
url = self.build_url(path, use_api_base=use_api_base)
merged_headers = self._headers(auth_kind, headers)
@@ -144,18 +144,18 @@ class HttpClient:
# )
def request_json(
self,
method: str,
path: str,
*,
use_api_base: bool = True,
auth_kind: Optional[str] = "api",
headers: Optional[Dict[str, str]] = None,
json_body: Optional[Dict[str, Any]] = None,
data: Any = None,
files: Any = None,
params: Optional[Dict[str, Any]] = None,
stream: bool = False,
self,
method: str,
path: str,
*,
use_api_base: bool = True,
auth_kind: Optional[str] = "api",
headers: Optional[Dict[str, str]] = None,
json_body: Optional[Dict[str, Any]] = None,
data: Any = None,
files: Any = None,
params: Optional[Dict[str, Any]] = None,
stream: bool = False,
) -> Dict[str, Any]:
response = self.request(
method,

View File

@@ -336,8 +336,8 @@ reset_default_asr: RESET DEFAULT ASR ";"
reset_default_tts: RESET DEFAULT TTS ";"
list_user_datasets: LIST DATASETS ";"
create_user_dataset_with_parser: CREATE DATASET quoted_string WITH EMBEDDING quoted_string PARSER quoted_string ";"
create_user_dataset_with_pipeline: CREATE DATASET quoted_string WITH EMBEDDING quoted_string PIPELINE quoted_string ";"
create_user_dataset_with_parser: CREATE DATASET quoted_string WITH EMBEDDING quoted_string PARSER quoted_string ";"
create_user_dataset_with_pipeline: CREATE DATASET quoted_string WITH EMBEDDING quoted_string PIPELINE quoted_string ";"
drop_user_dataset: DROP DATASET quoted_string ";"
list_user_dataset_files: LIST FILES OF DATASET quoted_string ";"
list_user_dataset_documents: LIST DOCUMENTS OF DATASET quoted_string ";"
@@ -640,15 +640,13 @@ class RAGFlowCLITransformer(Transformer):
dataset_name = items[2].children[0].strip("'\"")
embedding = items[5].children[0].strip("'\"")
parser_type = items[7].children[0].strip("'\"")
return {"type": "create_user_dataset", "dataset_name": dataset_name, "embedding": embedding,
"parser_type": parser_type}
return {"type": "create_user_dataset", "dataset_name": dataset_name, "embedding": embedding, "parser_type": parser_type}
def create_user_dataset_with_pipeline(self, items):
dataset_name = items[2].children[0].strip("'\"")
embedding = items[5].children[0].strip("'\"")
pipeline = items[7].children[0].strip("'\"")
return {"type": "create_user_dataset", "dataset_name": dataset_name, "embedding": embedding,
"pipeline": pipeline}
return {"type": "create_user_dataset", "dataset_name": dataset_name, "embedding": embedding, "pipeline": pipeline}
def drop_user_dataset(self, items):
dataset_name = items[2].children[0].strip("'\"")
@@ -666,7 +664,7 @@ class RAGFlowCLITransformer(Transformer):
dataset_names = []
dataset_names.append(items[4].children[0].strip("'\""))
for i in range(5, len(items)):
if items[i] and hasattr(items[i], 'children') and items[i].children:
if items[i] and hasattr(items[i], "children") and items[i].children:
dataset_names.append(items[i].children[0].strip("'\""))
return {"type": "list_user_datasets_metadata", "dataset_names": dataset_names}
@@ -675,7 +673,7 @@ class RAGFlowCLITransformer(Transformer):
doc_ids = []
if len(items) > 6 and items[6] == "DOCUMENTS":
for i in range(7, len(items)):
if items[i] and hasattr(items[i], 'children') and items[i].children:
if items[i] and hasattr(items[i], "children") and items[i].children:
doc_id = items[i].children[0].strip("'\"")
doc_ids.append(doc_id)
return {"type": "list_user_documents_metadata_summary", "dataset_name": dataset_name, "document_ids": doc_ids}
@@ -698,17 +696,17 @@ class RAGFlowCLITransformer(Transformer):
dataset_name = None
vector_size = None
for i, item in enumerate(items):
if hasattr(item, 'data') and item.data == 'quoted_string':
if hasattr(item, "data") and item.data == "quoted_string":
dataset_name = item.children[0].strip("'\"")
if hasattr(item, 'type') and item.type == 'NUMBER':
if i > 0 and items[i-1].type == 'SIZE' and items[i-2].type == 'VECTOR':
if hasattr(item, "type") and item.type == "NUMBER":
if i > 0 and items[i - 1].type == "SIZE" and items[i - 2].type == "VECTOR":
vector_size = int(item)
return {"type": "create_dataset_table", "dataset_name": dataset_name, "vector_size": vector_size}
def drop_dataset_table(self, items):
dataset_name = None
for item in items:
if hasattr(item, 'data') and item.data == 'quoted_string':
if hasattr(item, "data") and item.data == "quoted_string":
dataset_name = item.children[0].strip("'\"")
return {"type": "drop_dataset_table", "dataset_name": dataset_name}
@@ -792,7 +790,7 @@ class RAGFlowCLITransformer(Transformer):
def update_chunk(self, items):
def get_quoted_value(item):
if hasattr(item, 'children') and item.children:
if hasattr(item, "children") and item.children:
return item.children[0].strip("'\"")
return str(item).strip("'\"")
@@ -813,16 +811,16 @@ class RAGFlowCLITransformer(Transformer):
for i in range(2, len(items)):
item = items[i]
# Check for FROM token to stop
if hasattr(item, 'type') and item.type == 'FROM':
if hasattr(item, "type") and item.type == "FROM":
break
if hasattr(item, 'children') and item.children:
if hasattr(item, "children") and item.children:
tag = item.children[0].strip("'\"")
tags.append(tag)
# Find dataset_name: quoted_string after DATASET
dataset_name = None
for i, item in enumerate(items):
# Check if item is a DATASET token
if hasattr(item, 'type') and item.type == 'DATASET':
if hasattr(item, "type") and item.type == "DATASET":
# Next item should be quoted_string
dataset_name = items[i + 1].children[0].strip("'\"")
break
@@ -835,10 +833,10 @@ class RAGFlowCLITransformer(Transformer):
# Check if it's "REMOVE ALL CHUNKS"
for item in items:
if hasattr(item, 'type') and item.type == 'ALL':
if hasattr(item, "type") and item.type == "ALL":
# Find doc_id
for j, inner_item in enumerate(items):
if hasattr(inner_item, 'type') and inner_item.type == 'DOCUMENT':
if hasattr(inner_item, "type") and inner_item.type == "DOCUMENT":
doc_id = items[j + 1].children[0].strip("'\"")
return {"type": "remove_chunks", "doc_id": doc_id, "delete_all": True}
@@ -846,12 +844,12 @@ class RAGFlowCLITransformer(Transformer):
chunk_ids = []
doc_id = None
for i, item in enumerate(items):
if hasattr(item, 'type') and item.type == 'DOCUMENT':
if hasattr(item, "type") and item.type == "DOCUMENT":
doc_id = items[i + 1].children[0].strip("'\"")
elif hasattr(item, 'children') and item.children:
elif hasattr(item, "children") and item.children:
val = item.children[0].strip("'\"")
# Skip if it's "FROM" or "DOCUMENT"
if val.upper() in ['FROM', 'DOCUMENT']:
if val.upper() in ["FROM", "DOCUMENT"]:
continue
chunk_ids.append(val)

View File

@@ -1,6 +1,6 @@
[project]
name = "ragflow-cli"
version = "0.26.3"
version = "0.26.4"
description = "Admin Service's client of [RAGFlow](https://github.com/infiniflow/ragflow). The Admin Service provides user management and system monitoring. "
authors = [{ name = "Lynn", email = "lynn_inf@hotmail.com" }]
license = { text = "Apache License, Version 2.0" }
@@ -8,7 +8,7 @@ readme = "README.md"
requires-python = ">=3.13,<3.14"
dependencies = [
"requests>=2.30.0,<3.0.0",
"beartype>=0.20.0,<1.0.0",
"beartype>=0.22.9,<1.0.0",
"pycryptodomex>=3.10.0",
"lark>=1.1.0",
"requests-toolbelt>=1.0.0",

View File

@@ -36,6 +36,7 @@ from user import login_user
warnings.filterwarnings("ignore", category=getpass.GetPassWarning)
def encrypt(input_string):
pub = "-----BEGIN PUBLIC KEY-----\nMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEArq9XTUSeYr2+N1h3Afl/z8Dse/2yD0ZGrKwx+EEEcdsBLca9Ynmx3nIB5obmLlSfmskLpBo0UACBmB5rEjBp2Q2f3AG3Hjd4B+gNCG6BDaawuDlgANIhGnaTLrIqWrrcm4EMzJOnAOI1fgzJRsOOUEfaS318Eq9OVO3apEyCCt0lOQK6PuksduOjVxtltDav+guVAA068NrPYmRNabVKRNLJpL8w4D44sfth5RvZ3q9t+6RTArpEtc5sh5ChzvqPOzKGMXW83C95TxmXqpbK6olN4RevSfVjEAgCydH6HN6OhtOQEcnrU97r9H0iZOWwbw3pVrZiUkuRD1R56Wzs2wIDAQAB\n-----END PUBLIC KEY-----"
pub_key = RSA.importKey(pub)
@@ -49,9 +50,6 @@ def encode_to_base64(input_string):
return base64_encoded.decode("utf-8")
class RAGFlowCLI(Cmd):
def __init__(self):
super().__init__()
@@ -240,9 +238,9 @@ class RAGFlowCLI(Cmd):
print(r"""
____ ___ ______________ ________ ____
/ __ \/ | / ____/ ____/ /___ _ __ / ____/ / / _/
/ /_/ / /| |/ / __/ /_ / / __ \ | /| / / / / / / / /
/ _, _/ ___ / /_/ / __/ / / /_/ / |/ |/ / / /___/ /____/ /
/_/ |_/_/ |_\____/_/ /_/\____/|__/|__/ \____/_____/___/
/ /_/ / /| |/ / __/ /_ / / __ \ | /| / / / / / / / /
/ _, _/ ___ / /_/ / __/ / / /_/ / |/ |/ / / /___/ /____/ /
/_/ |_/_/ |_\____/_/ /_/\____/|__/|__/ \____/_____/___/
""")
self.cmdloop()
@@ -254,15 +252,13 @@ class RAGFlowCLI(Cmd):
result = self.parse_command(command)
self.execute_command(result)
def parse_connection_args(self, args: List[str]) -> Dict[str, Any]:
parser = argparse.ArgumentParser(description="RAGFlow CLI Client", add_help=False)
parser.add_argument("-h", "--host", default="127.0.0.1", help="Admin or RAGFlow service host")
parser.add_argument("-p", "--port", type=int, default=9381, help="Admin or RAGFlow service port")
parser.add_argument("-w", "--password", default="admin", type=str, help="Superuser password")
parser.add_argument("-t", "--type", default="admin", type=str, help="CLI mode, admin or user")
parser.add_argument("-u", "--username", default=None,
help="Username (email). In admin mode defaults to admin@ragflow.io, in user mode required.")
parser.add_argument("-u", "--username", default=None, help="Username (email). In admin mode defaults to admin@ragflow.io, in user mode required.")
parser.add_argument("command", nargs="?", help="Single command")
try:
parsed_args, remaining_args = parser.parse_known_args(args)
@@ -274,7 +270,7 @@ class RAGFlowCLI(Cmd):
if remaining_args:
if remaining_args[0] == "command":
command_str = ' '.join(remaining_args[1:]) + ';'
command_str = " ".join(remaining_args[1:]) + ";"
auth = True
if remaining_args[1] == "register":
auth = False
@@ -282,28 +278,14 @@ class RAGFlowCLI(Cmd):
if username is None:
print("Error: username (-u) is required in user mode")
return {"error": "Username required"}
return {
"host": parsed_args.host,
"port": parsed_args.port,
"password": parsed_args.password,
"type": parsed_args.type,
"username": username,
"command": command_str,
"auth": auth
}
return {"host": parsed_args.host, "port": parsed_args.port, "password": parsed_args.password, "type": parsed_args.type, "username": username, "command": command_str, "auth": auth}
else:
return {"error": "Invalid command"}
else:
auth = True
if username is None:
auth = False
return {
"host": parsed_args.host,
"port": parsed_args.port,
"type": parsed_args.type,
"username": username,
"auth": auth
}
return {"host": parsed_args.host, "port": parsed_args.port, "type": parsed_args.type, "username": username, "auth": auth}
except SystemExit:
return {"error": "Invalid connection arguments"}
@@ -321,6 +303,7 @@ class RAGFlowCLI(Cmd):
# print(f"Parsed command: {command_dict}")
run_command(self.ragflow_client, command_dict)
def main():
cli = RAGFlowCLI()

View File

@@ -71,6 +71,7 @@ class RAGFlowClient:
user_password: str = command.get("password")
if not user_password:
import getpass
user_password = getpass.getpass("Password: ")
try:
token = login_user(self.http_client, self.server_type, email, user_password)
@@ -86,8 +87,7 @@ class RAGFlowClient:
def ping_server(self, command):
iterations = command.get("iterations", 1)
if iterations > 1:
response = self.http_client.request("GET", "/system/ping", use_api_base=True, auth_kind="web",
iterations=iterations)
response = self.http_client.request("GET", "/system/ping", use_api_base=True, auth_kind="web", iterations=iterations)
return response
else:
response = self.http_client.request("GET", "/system/ping", use_api_base=True, auth_kind="web")
@@ -106,8 +106,7 @@ class RAGFlowClient:
enc_password = encrypt_password(password)
print(f"Register user: {nickname}, email: {username}, password: ******")
payload = {"email": username, "nickname": nickname, "password": enc_password}
response = self.http_client.request(method="POST", path="/users",
json_body=payload, use_api_base=True, auth_kind="web")
response = self.http_client.request(method="POST", path="/users", json_body=payload, use_api_base=True, auth_kind="web")
res_json = response.json()
if response.status_code == 200:
if res_json["code"] == 0:
@@ -135,8 +134,7 @@ class RAGFlowClient:
service_id: int = command["number"]
response = self.http_client.request("GET", f"/admin/services/{service_id}", use_api_base=True,
auth_kind="admin")
response = self.http_client.request("GET", f"/admin/services/{service_id}", use_api_base=True, auth_kind="admin")
res_json = response.json()
if response.status_code == 200:
res_data = res_json["data"]
@@ -226,9 +224,7 @@ class RAGFlowClient:
password_tree: Tree = command["password"]
password: str = password_tree.children[0].strip("'\"")
print(f"Alter user: {user_name}, password: ******")
response = self.http_client.request("PUT", f"/admin/users/{user_name}/password",
json_body={"new_password": encrypt_password(password)}, use_api_base=True,
auth_kind="admin")
response = self.http_client.request("PUT", f"/admin/users/{user_name}/password", json_body={"new_password": encrypt_password(password)}, use_api_base=True, auth_kind="admin")
res_json = response.json()
if response.status_code == 200:
print(res_json["message"])
@@ -247,9 +243,7 @@ class RAGFlowClient:
print(f"Create user: {user_name}, password: ******, role: {role}")
# enpass1 = encrypt(password)
enc_password = encrypt_password(password)
response = self.http_client.request(method="POST", path="/admin/users",
json_body={"username": user_name, "password": enc_password, "role": role},
use_api_base=True, auth_kind="admin")
response = self.http_client.request(method="POST", path="/admin/users", json_body={"username": user_name, "password": enc_password, "role": role}, use_api_base=True, auth_kind="admin")
res_json = response.json()
if response.status_code == 200:
self._print_table_simple(res_json["data"])
@@ -266,9 +260,7 @@ class RAGFlowClient:
activate_status: str = activate_tree.children[0].strip("'\"")
if activate_status.lower() in ["on", "off"]:
print(f"Alter user {user_name} activate status, turn {activate_status.lower()}.")
response = self.http_client.request("PUT", f"/admin/users/{user_name}/activate",
json_body={"activate_status": activate_status}, use_api_base=True,
auth_kind="admin")
response = self.http_client.request("PUT", f"/admin/users/{user_name}/activate", json_body={"activate_status": activate_status}, use_api_base=True, auth_kind="admin")
res_json = response.json()
if response.status_code == 200:
print(res_json["message"])
@@ -283,14 +275,12 @@ class RAGFlowClient:
user_name_tree: Tree = command["user_name"]
user_name: str = user_name_tree.children[0].strip("'\"")
response = self.http_client.request("PUT", f"/admin/users/{user_name}/admin", use_api_base=True,
auth_kind="admin")
response = self.http_client.request("PUT", f"/admin/users/{user_name}/admin", use_api_base=True, auth_kind="admin")
res_json = response.json()
if response.status_code == 200:
print(res_json["message"])
else:
print(
f"Fail to grant {user_name} admin authorization, code: {res_json['code']}, message: {res_json['message']}")
print(f"Fail to grant {user_name} admin authorization, code: {res_json['code']}, message: {res_json['message']}")
def revoke_admin(self, command):
if self.server_type != "admin":
@@ -298,14 +288,12 @@ class RAGFlowClient:
user_name_tree: Tree = command["user_name"]
user_name: str = user_name_tree.children[0].strip("'\"")
response = self.http_client.request("DELETE", f"/admin/users/{user_name}/admin", use_api_base=True,
auth_kind="admin")
response = self.http_client.request("DELETE", f"/admin/users/{user_name}/admin", use_api_base=True, auth_kind="admin")
res_json = response.json()
if response.status_code == 200:
print(res_json["message"])
else:
print(
f"Fail to revoke {user_name} admin authorization, code: {res_json['code']}, message: {res_json['message']}")
print(f"Fail to revoke {user_name} admin authorization, code: {res_json['code']}, message: {res_json['message']}")
def create_role(self, command):
if self.server_type != "admin":
@@ -319,10 +307,7 @@ class RAGFlowClient:
desc_str = desc_tree.children[0].strip("'\"")
print(f"create role name: {role_name}, description: {desc_str}")
response = self.http_client.request("POST", "/admin/roles",
json_body={"role_name": role_name, "description": desc_str},
use_api_base=True,
auth_kind="admin")
response = self.http_client.request("POST", "/admin/roles", json_body={"role_name": role_name, "description": desc_str}, use_api_base=True, auth_kind="admin")
res_json = response.json()
if response.status_code == 200:
self._print_table_simple(res_json["data"])
@@ -336,9 +321,7 @@ class RAGFlowClient:
role_name_tree: Tree = command["role_name"]
role_name: str = role_name_tree.children[0].strip("'\"")
print(f"drop role name: {role_name}")
response = self.http_client.request("DELETE", f"/admin/roles/{role_name}",
use_api_base=True,
auth_kind="admin")
response = self.http_client.request("DELETE", f"/admin/roles/{role_name}", use_api_base=True, auth_kind="admin")
res_json = response.json()
if response.status_code == 200:
self._print_table_simple(res_json["data"])
@@ -355,24 +338,18 @@ class RAGFlowClient:
desc_str: str = desc_tree.children[0].strip("'\"")
print(f"alter role name: {role_name}, description: {desc_str}")
response = self.http_client.request("PUT", f"/admin/roles/{role_name}",
json_body={"description": desc_str},
use_api_base=True,
auth_kind="admin")
response = self.http_client.request("PUT", f"/admin/roles/{role_name}", json_body={"description": desc_str}, use_api_base=True, auth_kind="admin")
res_json = response.json()
if response.status_code == 200:
self._print_table_simple(res_json["data"])
else:
print(
f"Fail to update role {role_name} with description: {desc_str}, code: {res_json['code']}, message: {res_json['message']}")
print(f"Fail to update role {role_name} with description: {desc_str}, code: {res_json['code']}, message: {res_json['message']}")
def list_roles(self, command):
if self.server_type != "admin":
print("This command is only allowed in ADMIN mode")
response = self.http_client.request("GET", "/admin/roles",
use_api_base=True,
auth_kind="admin")
response = self.http_client.request("GET", "/admin/roles", use_api_base=True, auth_kind="admin")
res_json = response.json()
if response.status_code == 200:
self._print_table_simple(res_json["data"])
@@ -386,9 +363,7 @@ class RAGFlowClient:
role_name_tree: Tree = command["role_name"]
role_name: str = role_name_tree.children[0].strip("'\"")
print(f"show role: {role_name}")
response = self.http_client.request("GET", f"/admin/roles/{role_name}/permission",
use_api_base=True,
auth_kind="admin")
response = self.http_client.request("GET", f"/admin/roles/{role_name}/permission", use_api_base=True, auth_kind="admin")
res_json = response.json()
if response.status_code == 200:
self._print_table_simple(res_json["data"])
@@ -409,15 +384,12 @@ class RAGFlowClient:
action_str: str = action_tree.children[0].strip("'\"")
actions.append(action_str)
print(f"grant role_name: {role_name_str}, resource: {resource_str}, actions: {actions}")
response = self.http_client.request("POST", f"/admin/roles/{role_name_str}/permission",
json_body={"actions": actions, "resource": resource_str}, use_api_base=True,
auth_kind="admin")
response = self.http_client.request("POST", f"/admin/roles/{role_name_str}/permission", json_body={"actions": actions, "resource": resource_str}, use_api_base=True, auth_kind="admin")
res_json = response.json()
if response.status_code == 200:
self._print_table_simple(res_json["data"])
else:
print(
f"Fail to grant role {role_name_str} with {actions} on {resource_str}, code: {res_json['code']}, message: {res_json['message']}")
print(f"Fail to grant role {role_name_str} with {actions} on {resource_str}, code: {res_json['code']}, message: {res_json['message']}")
def revoke_permission(self, command):
if self.server_type != "admin":
@@ -433,15 +405,12 @@ class RAGFlowClient:
action_str: str = action_tree.children[0].strip("'\"")
actions.append(action_str)
print(f"revoke role_name: {role_name_str}, resource: {resource_str}, actions: {actions}")
response = self.http_client.request("DELETE", f"/admin/roles/{role_name_str}/permission",
json_body={"actions": actions, "resource": resource_str}, use_api_base=True,
auth_kind="admin")
response = self.http_client.request("DELETE", f"/admin/roles/{role_name_str}/permission", json_body={"actions": actions, "resource": resource_str}, use_api_base=True, auth_kind="admin")
res_json = response.json()
if response.status_code == 200:
self._print_table_simple(res_json["data"])
else:
print(
f"Fail to revoke role {role_name_str} with {actions} on {resource_str}, code: {res_json['code']}, message: {res_json['message']}")
print(f"Fail to revoke role {role_name_str} with {actions} on {resource_str}, code: {res_json['code']}, message: {res_json['message']}")
def alter_user_role(self, command):
if self.server_type != "admin":
@@ -452,15 +421,12 @@ class RAGFlowClient:
user_name_tree: Tree = command["user_name"]
user_name_str: str = user_name_tree.children[0].strip("'\"")
print(f"alter_user_role user_name: {user_name_str}, role_name: {role_name_str}")
response = self.http_client.request("PUT", f"/admin/users/{user_name_str}/role",
json_body={"role_name": role_name_str}, use_api_base=True,
auth_kind="admin")
response = self.http_client.request("PUT", f"/admin/users/{user_name_str}/role", json_body={"role_name": role_name_str}, use_api_base=True, auth_kind="admin")
res_json = response.json()
if response.status_code == 200:
self._print_table_simple(res_json["data"])
else:
print(
f"Fail to alter user: {user_name_str} to role {role_name_str}, code: {res_json['code']}, message: {res_json['message']}")
print(f"Fail to alter user: {user_name_str} to role {role_name_str}, code: {res_json['code']}, message: {res_json['message']}")
def show_user_permission(self, command):
if self.server_type != "admin":
@@ -469,14 +435,12 @@ class RAGFlowClient:
user_name_tree: Tree = command["user_name"]
user_name_str: str = user_name_tree.children[0].strip("'\"")
print(f"show_user_permission user_name: {user_name_str}")
response = self.http_client.request("GET", f"/admin/users/{user_name_str}/permission", use_api_base=True,
auth_kind="admin")
response = self.http_client.request("GET", f"/admin/users/{user_name_str}/permission", use_api_base=True, auth_kind="admin")
res_json = response.json()
if response.status_code == 200:
self._print_table_simple(res_json["data"])
else:
print(
f"Fail to show user: {user_name_str} permission, code: {res_json['code']}, message: {res_json['message']}")
print(f"Fail to show user: {user_name_str} permission, code: {res_json['code']}, message: {res_json['message']}")
def generate_key(self, command: dict[str, Any]) -> None:
if self.server_type != "admin":
@@ -485,14 +449,12 @@ class RAGFlowClient:
username_tree: Tree = command["user_name"]
user_name: str = username_tree.children[0].strip("'\"")
print(f"Generating API key for user: {user_name}")
response = self.http_client.request("POST", f"/admin/users/{user_name}/keys", use_api_base=True,
auth_kind="admin")
response = self.http_client.request("POST", f"/admin/users/{user_name}/keys", use_api_base=True, auth_kind="admin")
res_json: dict[str, Any] = response.json()
if response.status_code == 200:
self._print_table_simple(res_json["data"])
else:
print(
f"Failed to generate key for user {user_name}, code: {res_json['code']}, message: {res_json['message']}")
print(f"Failed to generate key for user {user_name}, code: {res_json['code']}, message: {res_json['message']}")
def list_keys(self, command: dict[str, Any]) -> None:
if self.server_type != "admin":
@@ -501,8 +463,7 @@ class RAGFlowClient:
username_tree: Tree = command["user_name"]
user_name: str = username_tree.children[0].strip("'\"")
print(f"Listing API keys for user: {user_name}")
response = self.http_client.request("GET", f"/admin/users/{user_name}/keys", use_api_base=True,
auth_kind="admin")
response = self.http_client.request("GET", f"/admin/users/{user_name}/keys", use_api_base=True, auth_kind="admin")
res_json: dict[str, Any] = response.json()
if response.status_code == 200:
self._print_table_simple(res_json["data"])
@@ -520,8 +481,7 @@ class RAGFlowClient:
print(f"Dropping API key for user: {user_name}")
# URL encode the key to handle special characters
encoded_key: str = urllib.parse.quote(key, safe="")
response = self.http_client.request("DELETE", f"/admin/users/{user_name}/keys/{encoded_key}", use_api_base=True,
auth_kind="admin")
response = self.http_client.request("DELETE", f"/admin/users/{user_name}/keys/{encoded_key}", use_api_base=True, auth_kind="admin")
res_json: dict[str, Any] = response.json()
if response.status_code == 200:
print(res_json["message"])
@@ -534,23 +494,19 @@ class RAGFlowClient:
var_name = _strip_tree_value(command["var_name"])
var_value = _strip_tree_value(command["var_value"])
response = self.http_client.request("PUT", "/admin/variables",
json_body={"var_name": var_name, "var_value": var_value}, use_api_base=True,
auth_kind="admin")
response = self.http_client.request("PUT", "/admin/variables", json_body={"var_name": var_name, "var_value": var_value}, use_api_base=True, auth_kind="admin")
res_json = response.json()
if response.status_code == 200:
print(res_json["message"])
else:
print(
f"Fail to set variable {var_name} to {var_value}, code: {res_json['code']}, message: {res_json['message']}")
print(f"Fail to set variable {var_name} to {var_value}, code: {res_json['code']}, message: {res_json['message']}")
def show_variable(self, command):
if self.server_type != "admin":
print("This command is only allowed in ADMIN mode")
var_name = _strip_tree_value(command["var_name"])
response = self.http_client.request(method="GET", path="/admin/variables", json_body={"var_name": var_name},
use_api_base=True, auth_kind="admin")
response = self.http_client.request(method="GET", path="/admin/variables", json_body={"var_name": var_name}, use_api_base=True, auth_kind="admin")
res_json = response.json()
if response.status_code == 200:
self._print_table_simple(res_json["data"])
@@ -604,8 +560,7 @@ class RAGFlowClient:
if self.server_type != "admin":
print("This command is only allowed in ADMIN mode")
license = command["license"]
response = self.http_client.request("POST", "/admin/license", json_body={"license": license}, use_api_base=True,
auth_kind="admin")
response = self.http_client.request("POST", "/admin/license", json_body={"license": license}, use_api_base=True, auth_kind="admin")
res_json = response.json()
if response.status_code == 200:
print("Set license successfully")
@@ -617,9 +572,7 @@ class RAGFlowClient:
print("This command is only allowed in ADMIN mode")
value1 = command["value1"]
value2 = command["value2"]
response = self.http_client.request("POST", "/admin/license/config",
json_body={"value1": value1, "value2": value2}, use_api_base=True,
auth_kind="admin")
response = self.http_client.request("POST", "/admin/license/config", json_body={"value1": value1, "value2": value2}, use_api_base=True, auth_kind="admin")
res_json = response.json()
if response.status_code == 200:
print("Set license successfully")
@@ -690,8 +643,7 @@ class RAGFlowClient:
user_name: str = username_tree.children[0].strip("'\"")
print(f"Listing all datasets of user: {user_name}")
response = self.http_client.request("GET", f"/admin/users/{user_name}/datasets", use_api_base=True,
auth_kind="admin")
response = self.http_client.request("GET", f"/admin/users/{user_name}/datasets", use_api_base=True, auth_kind="admin")
res_json = response.json()
if response.status_code == 200:
table_data = res_json["data"]
@@ -708,8 +660,7 @@ class RAGFlowClient:
username_tree: Tree = command["user_name"]
user_name: str = username_tree.children[0].strip("'\"")
print(f"Listing all agents of user: {user_name}")
response = self.http_client.request("GET", f"/admin/users/{user_name}/agents", use_api_base=True,
auth_kind="admin")
response = self.http_client.request("GET", f"/admin/users/{user_name}/agents", use_api_base=True, auth_kind="admin")
res_json = response.json()
if response.status_code == 200:
table_data = res_json["data"]
@@ -733,8 +684,7 @@ class RAGFlowClient:
# Step 1: Add provider
provider_payload = {"provider_name": provider_name}
provider_response = self.http_client.request("PUT", "/providers", json_body=provider_payload,
use_api_base=True, auth_kind="web")
provider_response = self.http_client.request("PUT", "/providers", json_body=provider_payload, use_api_base=True, auth_kind="web")
provider_res = provider_response.json()
if provider_response.status_code == 200 and provider_res.get("code") == 0:
print(f"Success to add provider {provider_name}")
@@ -747,15 +697,8 @@ class RAGFlowClient:
return
# Step 2: Add instance
instance_payload = {
"instance_name": "default",
"api_key": api_key,
"region": "default",
"base_url": ""
}
instance_response = self.http_client.request("POST", f"/providers/{provider_name}/instances",
json_body=instance_payload, use_api_base=True,
auth_kind="web")
instance_payload = {"instance_name": "default", "api_key": api_key, "region": "default", "base_url": ""}
instance_response = self.http_client.request("POST", f"/providers/{provider_name}/instances", json_body=instance_payload, use_api_base=True, auth_kind="web")
instance_res = instance_response.json()
if instance_response.status_code == 200 and instance_res.get("code") == 0:
print(f"Success to add instance for provider {provider_name}")
@@ -771,8 +714,7 @@ class RAGFlowClient:
print("This command is only allowed in USER mode")
return
provider_name: str = command["provider_name"]
response = self.http_client.request("DELETE", f"/providers/{provider_name}", use_api_base=True,
auth_kind="web")
response = self.http_client.request("DELETE", f"/providers/{provider_name}", use_api_base=True, auth_kind="web")
res_json = response.json()
if response.status_code == 200 and res_json.get("code") == 0:
print(f"Success to drop model provider {provider_name}")
@@ -810,8 +752,7 @@ class RAGFlowClient:
"model_type": model_type,
"model_name": model_name,
}
response = self.http_client.request("PATCH", "/models/default", json_body=payload, use_api_base=True,
auth_kind="web")
response = self.http_client.request("PATCH", "/models/default", json_body=payload, use_api_base=True, auth_kind="web")
res_json = response.json()
if response.status_code == 200 and res_json.get("code") == 0:
print(f"Success to set default {model_type} to {model_id}")
@@ -830,8 +771,7 @@ class RAGFlowClient:
return
payload = {"model_type": model_type}
response = self.http_client.request("PATCH", "/models/default", json_body=payload, use_api_base=True,
auth_kind="web")
response = self.http_client.request("PATCH", "/models/default", json_body=payload, use_api_base=True, auth_kind="web")
res_json = response.json()
if response.status_code == 200 and res_json.get("code") == 0:
print(f"Success to reset default {model_type}")
@@ -861,8 +801,7 @@ class RAGFlowClient:
iterations = command.get("iterations", 1)
if iterations > 1:
response = self.http_client.request("GET", "/datasets", use_api_base=True, auth_kind="web",
iterations=iterations)
response = self.http_client.request("GET", "/datasets", use_api_base=True, auth_kind="web", iterations=iterations)
return response
else:
response = self.http_client.request("GET", "/datasets", use_api_base=True, auth_kind="web")
@@ -876,16 +815,12 @@ class RAGFlowClient:
def create_user_dataset(self, command):
if self.server_type != "user":
print("This command is only allowed in USER mode")
payload = {
"name": command["dataset_name"],
"embedding_model": command["embedding"]
}
payload = {"name": command["dataset_name"], "embedding_model": command["embedding"]}
if "parser_id" in command:
payload["chunk_method"] = command["parser"]
if "pipeline" in command:
payload["pipeline_id"] = command["pipeline"]
response = self.http_client.request("POST", "/datasets", json_body=payload, use_api_base=True,
auth_kind="web")
response = self.http_client.request("POST", "/datasets", json_body=payload, use_api_base=True, auth_kind="web")
res_json = response.json()
if response.status_code == 200:
self._print_table_simple(res_json["data"])
@@ -981,8 +916,7 @@ class RAGFlowClient:
dataset_ids = [dataset_id for _, dataset_id in valid_datasets]
kb_ids_param = ",".join(dataset_ids)
response = self.http_client.request("GET", f"/kb/get_meta?kb_ids={kb_ids_param}",
use_api_base=False, auth_kind="web")
response = self.http_client.request("GET", f"/kb/get_meta?kb_ids={kb_ids_param}", use_api_base=False, auth_kind="web")
res_json = response.json()
if response.status_code != 200:
print(f"Fail to get metadata, code: {res_json.get('code')}, message: {res_json.get('message')}")
@@ -996,11 +930,7 @@ class RAGFlowClient:
table_data = []
for field_name, values_dict in meta.items():
for value, docs in values_dict.items():
table_data.append({
"field": field_name,
"value": value,
"doc_ids": ", ".join(docs)
})
table_data.append({"field": field_name, "value": value, "doc_ids": ", ".join(docs)})
self._print_table_simple(table_data)
def list_user_documents_metadata_summary(self, command_dict):
@@ -1018,8 +948,7 @@ class RAGFlowClient:
payload = {"kb_id": kb_id}
if doc_ids:
payload["doc_ids"] = doc_ids
response = self.http_client.request("POST", "/document/metadata/summary", json_body=payload,
use_api_base=False, auth_kind="web")
response = self.http_client.request("POST", "/document/metadata/summary", json_body=payload, use_api_base=False, auth_kind="web")
res_json = response.json()
if response.status_code == 200:
summary = res_json.get("data", {}).get("summary", {})
@@ -1086,16 +1015,11 @@ class RAGFlowClient:
"quote": True,
"keyword": False,
"tts": False,
"system": "You are an intelligent assistant. Your primary function is to answer questions based strictly on the provided knowledge base.\n\n **Essential Rules:**\n - Your answer must be derived **solely** from this knowledge base: `{knowledge}`.\n - **When information is available**: Summarize the content to give a detailed answer.\n - **When information is unavailable**: Your response must contain this exact sentence: \"The answer you are looking for is not found in the knowledge base!\"\n - **Always consider** the entire conversation history.",
"system": 'You are an intelligent assistant. Your primary function is to answer questions based strictly on the provided knowledge base.\n\n **Essential Rules:**\n - Your answer must be derived **solely** from this knowledge base: `{knowledge}`.\n - **When information is available**: Summarize the content to give a detailed answer.\n - **When information is unavailable**: Your response must contain this exact sentence: "The answer you are looking for is not found in the knowledge base!"\n - **Always consider** the entire conversation history.',
"refine_multiturn": False,
"use_kg": False,
"reasoning": False,
"parameters": [
{
"key": "knowledge",
"optional": False
}
],
"parameters": [{"key": "knowledge", "optional": False}],
"toc_enhance": False,
},
"similarity_threshold": 0.2,
@@ -1136,8 +1060,7 @@ class RAGFlowClient:
# Build payload
payload = {"kb_id": dataset_id, "vector_size": vector_size}
# Call API
response = self.http_client.request("POST", "/kb/doc_engine_table", json_body=payload,
use_api_base=False, auth_kind="web")
response = self.http_client.request("POST", "/kb/doc_engine_table", json_body=payload, use_api_base=False, auth_kind="web")
res_json = response.json()
if response.status_code == 200 and res_json.get("code") == 0:
print(f"Success to create table for dataset: {dataset_name}")
@@ -1155,8 +1078,7 @@ class RAGFlowClient:
return
# Call API to delete table
payload = {"kb_id": dataset_id}
response = self.http_client.request("DELETE", "/kb/doc_engine_table", json_body=payload,
use_api_base=False, auth_kind="web")
response = self.http_client.request("DELETE", "/kb/doc_engine_table", json_body=payload, use_api_base=False, auth_kind="web")
res_json = response.json()
if response.status_code == 200 and res_json.get("code") == 0:
print(f"Success to drop table for dataset: {dataset_name}")
@@ -1168,8 +1090,7 @@ class RAGFlowClient:
print("This command is only allowed in USER mode")
return
# Call API to create metadata table
response = self.http_client.request("POST", "/tenant/doc_engine_metadata_table",
use_api_base=False, auth_kind="web")
response = self.http_client.request("POST", "/tenant/doc_engine_metadata_table", use_api_base=False, auth_kind="web")
res_json = response.json()
if response.status_code == 200 and res_json.get("code") == 0:
print("Success to create metadata table")
@@ -1181,8 +1102,7 @@ class RAGFlowClient:
print("This command is only allowed in USER mode")
return
# Call API to delete metadata table
response = self.http_client.request("DELETE", "/tenant/doc_engine_metadata_table",
use_api_base=False, auth_kind="web")
response = self.http_client.request("DELETE", "/tenant/doc_engine_metadata_table", use_api_base=False, auth_kind="web")
res_json = response.json()
if response.status_code == 200 and res_json.get("code") == 0:
print("Success to drop metadata table")
@@ -1225,8 +1145,7 @@ class RAGFlowClient:
def _list_chat_sessions(self, dialog_id):
"""List all sessions (conversations) for a given dialog."""
response = self.http_client.request("GET", f"/chats/{dialog_id}/conversations", use_api_base=True,
auth_kind="web")
response = self.http_client.request("GET", f"/chats/{dialog_id}/conversations", use_api_base=True, auth_kind="web")
res_json = response.json()
if response.status_code == 200 and res_json["code"] == 0:
return res_json["data"]
@@ -1242,14 +1161,12 @@ class RAGFlowClient:
if dialog_id is None:
return
payload = {"name": "New conversation"}
response = self.http_client.request("POST", f"/chats/{dialog_id}/conversations", json_body=payload,
use_api_base=True, auth_kind="web")
response = self.http_client.request("POST", f"/chats/{dialog_id}/conversations", json_body=payload, use_api_base=True, auth_kind="web")
res_json = response.json()
if response.status_code == 200 and res_json["code"] == 0:
print(f"Success to create chat session for chat: {chat_name}")
else:
print(
f"Fail to create chat session for chat {chat_name}, code: {res_json['code']}, message: {res_json['message']}")
print(f"Fail to create chat session for chat {chat_name}, code: {res_json['code']}, message: {res_json['message']}")
def drop_chat_session(self, command):
if self.server_type != "user":
@@ -1270,14 +1187,12 @@ class RAGFlowClient:
print(f"Chat session '{session_id}' not found in chat '{chat_name}'")
return
payload = {"ids": to_drop_session_ids}
response = self.http_client.request("DELETE", f"/chats/{dialog_id}/conversations", json_body=payload,
use_api_base=True, auth_kind="web")
response = self.http_client.request("DELETE", f"/chats/{dialog_id}/conversations", json_body=payload, use_api_base=True, auth_kind="web")
res_json = response.json()
if response.status_code == 200 and res_json["code"] == 0:
print(f"Success to drop chat session '{session_id}' from chat: {chat_name}")
else:
print(
f"Fail to drop chat session '{session_id}' from chat {chat_name}, code: {res_json['code']}, message: {res_json['message']}")
print(f"Fail to drop chat session '{session_id}' from chat {chat_name}, code: {res_json['code']}, message: {res_json['message']}")
def list_chat_sessions(self, command):
if self.server_type != "user":
@@ -1305,13 +1220,9 @@ class RAGFlowClient:
# Prepare payload for completion API
# Note: stream parameter is not sent, server defaults to stream=True
payload = {
"session_id": session_id,
"messages": [{"role": "user", "content": message}]
}
payload = {"session_id": session_id, "messages": [{"role": "user", "content": message}]}
response = self.http_client.request("POST", "/chat/completions", json_body=payload,
use_api_base=True, auth_kind="web", stream=True)
response = self.http_client.request("POST", "/chat/completions", json_body=payload, use_api_base=True, auth_kind="web", stream=True)
if response.status_code != 200:
print(f"Fail to chat on session, status code: {response.status_code}")
@@ -1322,17 +1233,16 @@ class RAGFlowClient:
for line in response.iter_lines():
if not line:
continue
line_str = line.decode('utf-8')
if not line_str.startswith('data:'):
line_str = line.decode("utf-8")
if not line_str.startswith("data:"):
continue
data_str = line_str[5:].strip()
if data_str == '[DONE]':
if data_str == "[DONE]":
break
try:
data_json = json.loads(data_str)
if data_json.get("code") != 0:
print(
f"\nFail to chat on session, code: {data_json.get('code')}, message: {data_json.get('message', '')}")
print(f"\nFail to chat on session, code: {data_json.get('code')}, message: {data_json.get('message', '')}")
return
# Check if it's the final message
if data_json.get("data") is True:
@@ -1416,14 +1326,12 @@ class RAGFlowClient:
print(f"Documents {document_names} not found in {dataset_name}")
payload = {"doc_ids": document_ids, "run": 1}
response = self.http_client.request("POST", "/documents/ingest", json_body=payload, use_api_base=True,
auth_kind="web")
response = self.http_client.request("POST", "/documents/ingest", json_body=payload, use_api_base=True, auth_kind="web")
res_json = response.json()
if response.status_code == 200 and res_json["code"] == 0:
print(f"Success to parse {to_parse_doc_names} of {dataset_name}")
else:
print(
f"Fail to parse documents {res_json["data"]["docs"]}, code: {res_json['code']}, message: {res_json['message']}")
print(f"Fail to parse documents {res_json['data']['docs']}, code: {res_json['code']}, message: {res_json['message']}")
def parse_dataset(self, command_dict):
if self.server_type != "user":
@@ -1442,8 +1350,7 @@ class RAGFlowClient:
document_ids.append(doc["id"])
payload = {"doc_ids": document_ids, "run": 1}
response = self.http_client.request("POST", "/documents/ingest", json_body=payload, use_api_base=True,
auth_kind="web")
response = self.http_client.request("POST", "/documents/ingest", json_body=payload, use_api_base=True, auth_kind="web")
res_json = response.json()
if response.status_code == 200 and res_json["code"] == 0:
pass
@@ -1483,15 +1390,7 @@ class RAGFlowClient:
encoder = MultipartEncoder(fields=fields)
headers = {"Content-Type": encoder.content_type}
response = self.http_client.request(
"POST",
f"/datasets/{dataset_id}/documents?return_raw_files=true",
headers=headers,
data=encoder,
json_body=None,
params=None,
stream=False,
auth_kind="web",
use_api_base=True
"POST", f"/datasets/{dataset_id}/documents?return_raw_files=true", headers=headers, data=encoder, json_body=None, params=None, stream=False, auth_kind="web", use_api_base=True
)
res = response.json()
if res.get("code") == 0:
@@ -1526,22 +1425,18 @@ class RAGFlowClient:
}
iterations = command_dict.get("iterations", 1)
if iterations > 1:
response = self.http_client.request("POST", "/retrieval", json_body=payload, use_api_base=True,
auth_kind="web", iterations=iterations)
response = self.http_client.request("POST", "/retrieval", json_body=payload, use_api_base=True, auth_kind="web", iterations=iterations)
return response
else:
response = self.http_client.request("POST", "/retrieval", json_body=payload, use_api_base=True,
auth_kind="web")
response = self.http_client.request("POST", "/retrieval", json_body=payload, use_api_base=True, auth_kind="web")
res_json = response.json()
if response.status_code == 200:
if res_json["code"] == 0:
self._print_table_simple(res_json["data"]["chunks"])
else:
print(
f"Fail to search datasets: {dataset_names}, code: {res_json['code']}, message: {res_json['message']}")
print(f"Fail to search datasets: {dataset_names}, code: {res_json['code']}, message: {res_json['message']}")
else:
print(
f"Fail to search datasets: {dataset_names}, code: {res_json['code']}, message: {res_json['message']}")
print(f"Fail to search datasets: {dataset_names}, code: {res_json['code']}, message: {res_json['message']}")
def get_chunk(self, command_dict):
if self.server_type != "user":
@@ -1549,8 +1444,7 @@ class RAGFlowClient:
return
chunk_id = command_dict["chunk_id"]
response = self.http_client.request("GET", f"/chunk/get?chunk_id={chunk_id}", use_api_base=False,
auth_kind="web")
response = self.http_client.request("GET", f"/chunk/get?chunk_id={chunk_id}", use_api_base=False, auth_kind="web")
res_json = response.json()
if response.status_code == 200:
if res_json["code"] == 0:
@@ -1568,8 +1462,7 @@ class RAGFlowClient:
file_path = command_dict["file_path"]
payload = {"file_path": file_path}
response = self.http_client.request("POST", "/kb/insert_from_file", json_body=payload,
use_api_base=False, auth_kind="web")
response = self.http_client.request("POST", "/kb/insert_from_file", json_body=payload, use_api_base=False, auth_kind="web")
res_json = response.json()
if response.status_code == 200:
if res_json["code"] == 0:
@@ -1589,8 +1482,7 @@ class RAGFlowClient:
file_path = command_dict["file_path"]
payload = {"file_path": file_path}
response = self.http_client.request("POST", "/tenant/insert_metadata_from_file", json_body=payload,
use_api_base=False, auth_kind="web")
response = self.http_client.request("POST", "/tenant/insert_metadata_from_file", json_body=payload, use_api_base=False, auth_kind="web")
res_json = response.json()
if response.status_code == 200:
if res_json["code"] == 0:
@@ -1617,8 +1509,7 @@ class RAGFlowClient:
return
# Get doc_id from chunk_id via GET /chunk/get
response = self.http_client.request("GET", f"/chunk/get?chunk_id={chunk_id}", use_api_base=False,
auth_kind="web")
response = self.http_client.request("GET", f"/chunk/get?chunk_id={chunk_id}", use_api_base=False, auth_kind="web")
res_json = response.json()
if response.status_code != 200:
print(f"Fail to get chunk info, code: {res_json.get('code')}, message: {res_json.get('message')}")
@@ -1655,14 +1546,8 @@ class RAGFlowClient:
else:
print(f"Fail to update chunk, HTTP {response.status_code}")
def _get_documents_by_ids(self, ids:list[str]):
response = self.http_client.request(
"POST",
"/document/infos",
json_body={"doc_ids": ids},
use_api_base=False,
auth_kind="web"
)
def _get_documents_by_ids(self, ids: list[str]):
response = self.http_client.request("POST", "/document/infos", json_body={"doc_ids": ids}, use_api_base=False, auth_kind="web")
if response.status_code != 200:
return f"Fail to get document info, HTTP {response.status_code}", None
@@ -1687,6 +1572,7 @@ class RAGFlowClient:
# Parse JSON string to dict
import json
try:
meta_fields = json.loads(meta_json_str)
except json.JSONDecodeError as e:
@@ -1713,13 +1599,7 @@ class RAGFlowClient:
"meta_fields": meta_fields,
}
response = self.http_client.request(
"PATCH",
f"/datasets/{dataset_id}/documents/{doc_id}",
json_body=payload,
use_api_base=True,
auth_kind="web"
)
response = self.http_client.request("PATCH", f"/datasets/{dataset_id}/documents/{doc_id}", json_body=payload, use_api_base=True, auth_kind="web")
res_json = response.json()
if response.status_code == 200:
@@ -1747,8 +1627,7 @@ class RAGFlowClient:
"tags": tags,
}
response = self.http_client.request("POST", f"/kb/{dataset_id}/rm_tags", json_body=payload,
use_api_base=False, auth_kind="web")
response = self.http_client.request("POST", f"/kb/{dataset_id}/rm_tags", json_body=payload, use_api_base=False, auth_kind="web")
res_json = response.json()
if response.status_code == 200:
if res_json.get("code") == 0:
@@ -1771,8 +1650,7 @@ class RAGFlowClient:
elif command_dict.get("chunk_ids"):
payload["chunk_ids"] = command_dict["chunk_ids"]
response = self.http_client.request("POST", "/chunk/rm", json_body=payload,
use_api_base=False, auth_kind="web")
response = self.http_client.request("POST", "/chunk/rm", json_body=payload, use_api_base=False, auth_kind="web")
res_json = response.json()
if response.status_code == 200:
if res_json.get("code") == 0:
@@ -1803,15 +1681,14 @@ class RAGFlowClient:
if "available_int" in command_dict:
payload["available_int"] = command_dict["available_int"]
response = self.http_client.request("POST", "/chunk/list", json_body=payload, use_api_base=False,
auth_kind="web")
response = self.http_client.request("POST", "/chunk/list", json_body=payload, use_api_base=False, auth_kind="web")
res_json = response.json()
if response.status_code == 200:
if res_json["code"] == 0:
chunks = res_json["data"]["chunks"]
if chunks:
for i, chunk in enumerate(chunks):
print(f"\n--- Chunk {i+1} ---")
print(f"\n--- Chunk {i + 1} ---")
for key, value in chunk.items():
print(f" {key}: {value}")
else:
@@ -1845,7 +1722,7 @@ class RAGFlowClient:
all_done = True
for doc in docs:
if doc.get("run") != "DONE":
print(f"Document {doc["name"]} is not done, status: {doc.get("run")}")
print(f"Document {doc['name']} is not done, status: {doc.get('run')}")
all_done = False
break
if all_done:
@@ -1856,16 +1733,10 @@ class RAGFlowClient:
def _list_documents(self, dataset_name: str, dataset_id: str):
# Use the new RESTful API: GET /api/v1/datasets/<dataset_id>/documents
response = self.http_client.request(
"GET",
f"/datasets/{dataset_id}/documents",
use_api_base=True,
auth_kind="web"
)
response = self.http_client.request("GET", f"/datasets/{dataset_id}/documents", use_api_base=True, auth_kind="web")
res_json = response.json()
if response.status_code != 200:
print(
f"Fail to list files from dataset {dataset_name}, code: {res_json['code']}, message: {res_json['message']}")
print(f"Fail to list files from dataset {dataset_name}, code: {res_json['code']}, message: {res_json['message']}")
return None
return res_json["data"]["docs"]
@@ -2254,22 +2125,14 @@ def run_benchmark(client: RAGFlowClient, command_dict: dict):
total_duration = result["duration"]
qps = iterations / total_duration if total_duration > 0 else None
print(f"command: {command}, Concurrency: {concurrency}, iterations: {iterations}")
print(
f"total duration: {total_duration:.4f}s, QPS: {qps}, COMMAND_COUNT: {iterations}, SUCCESS: {success_count}, FAILURE: {iterations - success_count}")
print(f"total duration: {total_duration:.4f}s, QPS: {qps}, COMMAND_COUNT: {iterations}, SUCCESS: {success_count}, FAILURE: {iterations - success_count}")
pass
else:
results: List[Optional[dict]] = [None] * concurrency
mp_context = mp.get_context("spawn")
start_time = time.perf_counter()
with ProcessPoolExecutor(max_workers=concurrency, mp_context=mp_context) as executor:
future_map = {
executor.submit(
run_command,
client,
command
): idx
for idx in range(concurrency)
}
future_map = {executor.submit(run_command, client, command): idx for idx in range(concurrency)}
for future in as_completed(future_map):
idx = future_map[future]
results[idx] = future.result()
@@ -2291,7 +2154,6 @@ def run_benchmark(client: RAGFlowClient, command_dict: dict):
total_command_count = iterations * concurrency
qps = total_command_count / total_duration if total_duration > 0 else None
print(f"command: {command}, Concurrency: {concurrency} , iterations: {iterations}")
print(
f"total duration: {total_duration:.4f}s, QPS: {qps}, COMMAND_COUNT: {total_command_count}, SUCCESS: {success_count}, FAILURE: {total_command_count - success_count}")
print(f"total duration: {total_duration:.4f}s, QPS: {qps}, COMMAND_COUNT: {total_command_count}, SUCCESS: {success_count}, FAILURE: {total_command_count - success_count}")
pass

View File

@@ -29,6 +29,7 @@ def encrypt_password(password_plain: str) -> str:
import base64
from Cryptodome.PublicKey import RSA
from Cryptodome.Cipher import PKCS1_v1_5 as Cipher_pkcs1_v1_5
def crypt(line):
"""
decrypt(crypt(input_string)) == base64(input_string), which frontend and ragflow_cli use.
@@ -36,13 +37,11 @@ def encrypt_password(password_plain: str) -> str:
pub = "-----BEGIN PUBLIC KEY-----\nMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEArq9XTUSeYr2+N1h3Afl/z8Dse/2yD0ZGrKwx+EEEcdsBLca9Ynmx3nIB5obmLlSfmskLpBo0UACBmB5rEjBp2Q2f3AG3Hjd4B+gNCG6BDaawuDlgANIhGnaTLrIqWrrcm4EMzJOnAOI1fgzJRsOOUEfaS318Eq9OVO3apEyCCt0lOQK6PuksduOjVxtltDav+guVAA068NrPYmRNabVKRNLJpL8w4D44sfth5RvZ3q9t+6RTArpEtc5sh5ChzvqPOzKGMXW83C95TxmXqpbK6olN4RevSfVjEAgCydH6HN6OhtOQEcnrU97r9H0iZOWwbw3pVrZiUkuRD1R56Wzs2wIDAQAB\n-----END PUBLIC KEY-----"
rsa_key = RSA.importKey(pub)
cipher = Cipher_pkcs1_v1_5.new(rsa_key)
password_base64 = base64.b64encode(line.encode('utf-8')).decode("utf-8")
password_base64 = base64.b64encode(line.encode("utf-8")).decode("utf-8")
encrypted_password = cipher.encrypt(password_base64.encode())
return base64.b64encode(encrypted_password).decode('utf-8')
return base64.b64encode(encrypted_password).decode("utf-8")
except Exception as exc:
raise AuthException(
"Password encryption unavailable; install pycryptodomex (uv sync --python 3.13 --group test)."
) from exc
raise AuthException("Password encryption unavailable; install pycryptodomex (uv sync --python 3.13 --group test).") from exc
return crypt(password_plain)

2
admin/client/uv.lock generated
View File

@@ -156,7 +156,7 @@ wheels = [
[[package]]
name = "ragflow-cli"
version = "0.26.3"
version = "0.26.4"
source = { virtual = "." }
dependencies = [
{ name = "beartype" },

View File

@@ -15,6 +15,7 @@
#
import time
start_ts = time.time()
import os
@@ -38,26 +39,24 @@ from common.versions import get_ragflow_version
stop_event = threading.Event()
if __name__ == '__main__':
if __name__ == "__main__":
faulthandler.enable()
init_root_logger("admin_service")
logging.info(r"""
____ ___ ______________ ___ __ _
/ __ \/ | / ____/ ____/ /___ _ __ / | ____/ /___ ___ (_)___
____ ___ ______________ ___ __ _
/ __ \/ | / ____/ ____/ /___ _ __ / | ____/ /___ ___ (_)___
/ /_/ / /| |/ / __/ /_ / / __ \ | /| / / / /| |/ __ / __ `__ \/ / __ \
/ _, _/ ___ / /_/ / __/ / / /_/ / |/ |/ / / ___ / /_/ / / / / / / / / / /
/_/ |_/_/ |_\____/_/ /_/\____/|__/|__/ /_/ |_\__,_/_/ /_/ /_/_/_/ /_/
/_/ |_/_/ |_\____/_/ /_/\____/|__/|__/ /_/ |_\__,_/_/ /_/ /_/_/_/ /_/
""")
app = Flask(__name__)
app.register_blueprint(admin_bp)
app.config["SESSION_PERMANENT"] = False
app.config["SESSION_TYPE"] = "filesystem"
app.config["MAX_CONTENT_LENGTH"] = int(
os.environ.get("MAX_CONTENT_LENGTH", 1024 * 1024 * 1024)
)
app.config["MAX_CONTENT_LENGTH"] = int(os.environ.get("MAX_CONTENT_LENGTH", 1024 * 1024 * 1024))
Session(app)
logging.info(f'RAGFlow admin version: {get_ragflow_version()}')
logging.info(f"RAGFlow admin version: {get_ragflow_version()}")
show_configs()
login_manager = LoginManager()
login_manager.init_app(app)

View File

@@ -70,9 +70,7 @@ def setup_auth(login_manager):
logging.warning(f"Authentication attempt with invalid token format: {len(access_token)} chars")
return None
user = UserService.query(
access_token=access_token, status=StatusEnum.VALID.value
)
user = UserService.query(access_token=access_token, status=StatusEnum.VALID.value)
if user:
if not user[0].access_token or not user[0].access_token.strip():
logging.warning(f"User {user[0].email} has empty access_token in database")
@@ -123,22 +121,16 @@ def add_tenant_for_admin(user_info: dict, role: str):
"embd_id": settings.EMBEDDING_MDL,
"asr_id": settings.ASR_MDL,
"parser_ids": settings.PARSERS,
"img2txt_id": settings.IMAGE2TEXT_MDL,
"img2txt_id": settings.VISION_MDL,
"rerank_id": settings.RERANK_MDL,
}
usr_tenant = {
"tenant_id": user_info["id"],
"user_id": user_info["id"],
"invited_by": user_info["id"],
"role": role
}
usr_tenant = {"tenant_id": user_info["id"], "user_id": user_info["id"], "invited_by": user_info["id"], "role": role}
# tenant_llm = get_init_tenant_llm(user_info["id"])
TenantService.insert(**tenant)
UserTenantService.insert(**usr_tenant)
# TenantLLMService.insert_many(tenant_llm)
logging.info(
f"Added tenant for email: {user_info['email']}, A default tenant has been set; changing the default models after login is strongly recommended.")
logging.info(f"Added tenant for email: {user_info['email']}, A default tenant has been set; changing the default models after login is strongly recommended.")
def check_admin_auth(func):
@@ -160,13 +152,13 @@ def check_admin_auth(func):
def login_admin(email: str, password: str):
"""
:param email: admin email
:param password: string before decrypt
:param password: string before decrypt (RSA encrypted + base64 encoded)
"""
users = UserService.query(email=email)
if not users:
raise UserNotFoundError(email)
psw = decrypt(password)
user = UserService.query_user(email, psw)
decrypted = decrypt(password)
user = UserService.query_user(email, decrypted)
if not user:
raise AdminException("Email and password do not match!")
if not user.is_superuser:
@@ -212,28 +204,17 @@ def login_verify(f):
@wraps(f)
def decorated(*args, **kwargs):
auth = request.authorization
if not auth or 'username' not in auth.parameters or 'password' not in auth.parameters:
return jsonify({
"code": 401,
"message": "Authentication required",
"data": None
}), 200
if not auth or "username" not in auth.parameters or "password" not in auth.parameters:
return jsonify({"code": 401, "message": "Authentication required", "data": None}), 200
username = auth.parameters['username']
password = auth.parameters['password']
username = auth.parameters["username"]
password = auth.parameters["password"]
try:
if not check_admin(username, password):
return jsonify({
"code": 500,
"message": "Access denied",
"data": None
}), 200
return jsonify({"code": 500, "message": "Access denied", "data": None}), 200
except Exception:
logging.exception("An error occurred during admin login verification.")
return jsonify({
"code": 500,
"message": "An internal server error occurred."
}), 200
return jsonify({"code": 500, "message": "An internal server error occurred."}), 200
return f(*args, **kwargs)

View File

@@ -34,8 +34,7 @@ class BaseConfig(BaseModel):
detail_func_name: str
def to_dict(self) -> dict[str, Any]:
return {'id': self.id, 'name': self.name, 'host': self.host, 'port': self.port,
'service_type': self.service_type}
return {"id": self.id, "name": self.name, "host": self.host, "port": self.port, "service_type": self.service_type}
class ServiceConfigs:
@@ -63,11 +62,11 @@ class MetaConfig(BaseConfig):
def to_dict(self) -> dict[str, Any]:
result = super().to_dict()
if 'extra' not in result:
result['extra'] = dict()
extra_dict = result['extra'].copy()
extra_dict['meta_type'] = self.meta_type
result['extra'] = extra_dict
if "extra" not in result:
result["extra"] = dict()
extra_dict = result["extra"].copy()
extra_dict["meta_type"] = self.meta_type
result["extra"] = extra_dict
return result
@@ -77,21 +76,20 @@ class MySQLConfig(MetaConfig):
def to_dict(self) -> dict[str, Any]:
result = super().to_dict()
if 'extra' not in result:
result['extra'] = dict()
extra_dict = result['extra'].copy()
extra_dict['username'] = self.username
extra_dict['password'] = self.password
result['extra'] = extra_dict
if "extra" not in result:
result["extra"] = dict()
extra_dict = result["extra"].copy()
extra_dict["username"] = self.username
extra_dict["password"] = self.password
result["extra"] = extra_dict
return result
class PostgresConfig(MetaConfig):
def to_dict(self) -> dict[str, Any]:
result = super().to_dict()
if 'extra' not in result:
result['extra'] = dict()
if "extra" not in result:
result["extra"] = dict()
return result
@@ -100,11 +98,11 @@ class RetrievalConfig(BaseConfig):
def to_dict(self) -> dict[str, Any]:
result = super().to_dict()
if 'extra' not in result:
result['extra'] = dict()
extra_dict = result['extra'].copy()
extra_dict['retrieval_type'] = self.retrieval_type
result['extra'] = extra_dict
if "extra" not in result:
result["extra"] = dict()
extra_dict = result["extra"].copy()
extra_dict["retrieval_type"] = self.retrieval_type
result["extra"] = extra_dict
return result
@@ -113,11 +111,11 @@ class InfinityConfig(RetrievalConfig):
def to_dict(self) -> dict[str, Any]:
result = super().to_dict()
if 'extra' not in result:
result['extra'] = dict()
extra_dict = result['extra'].copy()
extra_dict['db_name'] = self.db_name
result['extra'] = extra_dict
if "extra" not in result:
result["extra"] = dict()
extra_dict = result["extra"].copy()
extra_dict["db_name"] = self.db_name
result["extra"] = extra_dict
return result
@@ -127,12 +125,12 @@ class ElasticsearchConfig(RetrievalConfig):
def to_dict(self) -> dict[str, Any]:
result = super().to_dict()
if 'extra' not in result:
result['extra'] = dict()
extra_dict = result['extra'].copy()
extra_dict['username'] = self.username
extra_dict['password'] = self.password
result['extra'] = extra_dict
if "extra" not in result:
result["extra"] = dict()
extra_dict = result["extra"].copy()
extra_dict["username"] = self.username
extra_dict["password"] = self.password
result["extra"] = extra_dict
return result
@@ -141,11 +139,11 @@ class MessageQueueConfig(BaseConfig):
def to_dict(self) -> dict[str, Any]:
result = super().to_dict()
if 'extra' not in result:
result['extra'] = dict()
extra_dict = result['extra'].copy()
extra_dict['mq_type'] = self.mq_type
result['extra'] = extra_dict
if "extra" not in result:
result["extra"] = dict()
extra_dict = result["extra"].copy()
extra_dict["mq_type"] = self.mq_type
result["extra"] = extra_dict
return result
@@ -155,30 +153,28 @@ class RedisConfig(MessageQueueConfig):
def to_dict(self) -> dict[str, Any]:
result = super().to_dict()
if 'extra' not in result:
result['extra'] = dict()
extra_dict = result['extra'].copy()
extra_dict['database'] = self.database
extra_dict['password'] = self.password
result['extra'] = extra_dict
if "extra" not in result:
result["extra"] = dict()
extra_dict = result["extra"].copy()
extra_dict["database"] = self.database
extra_dict["password"] = self.password
result["extra"] = extra_dict
return result
class RabbitMQConfig(MessageQueueConfig):
def to_dict(self) -> dict[str, Any]:
result = super().to_dict()
if 'extra' not in result:
result['extra'] = dict()
if "extra" not in result:
result["extra"] = dict()
return result
class RAGFlowServerConfig(BaseConfig):
def to_dict(self) -> dict[str, Any]:
result = super().to_dict()
if 'extra' not in result:
result['extra'] = dict()
if "extra" not in result:
result["extra"] = dict()
return result
@@ -187,9 +183,9 @@ class TaskExecutorConfig(BaseConfig):
def to_dict(self) -> dict[str, Any]:
result = super().to_dict()
if 'extra' not in result:
result['extra'] = dict()
result['extra']['message_queue_type'] = self.message_queue_type
if "extra" not in result:
result["extra"] = dict()
result["extra"]["message_queue_type"] = self.message_queue_type
return result
@@ -198,11 +194,11 @@ class FileStoreConfig(BaseConfig):
def to_dict(self) -> dict[str, Any]:
result = super().to_dict()
if 'extra' not in result:
result['extra'] = dict()
extra_dict = result['extra'].copy()
extra_dict['store_type'] = self.store_type
result['extra'] = extra_dict
if "extra" not in result:
result["extra"] = dict()
extra_dict = result["extra"].copy()
extra_dict["store_type"] = self.store_type
result["extra"] = extra_dict
return result
@@ -212,12 +208,12 @@ class MinioConfig(FileStoreConfig):
def to_dict(self) -> dict[str, Any]:
result = super().to_dict()
if 'extra' not in result:
result['extra'] = dict()
extra_dict = result['extra'].copy()
extra_dict['user'] = self.user
extra_dict['password'] = self.password
result['extra'] = extra_dict
if "extra" not in result:
result["extra"] = dict()
extra_dict = result["extra"].copy()
extra_dict["user"] = self.user
extra_dict["password"] = self.password
result["extra"] = extra_dict
return result
@@ -229,106 +225,105 @@ def load_configurations(config_path: str) -> list[BaseConfig]:
for k, v in raw_configs.items():
match k:
case "ragflow":
name: str = f'ragflow_{ragflow_count}'
host: str = v['host']
http_port: int = v['http_port']
config = RAGFlowServerConfig(id=id_count, name=name, host=host, port=http_port,
service_type="ragflow_server",
detail_func_name="check_ragflow_server_alive")
name: str = f"ragflow_{ragflow_count}"
host: str = v["host"]
http_port: int = v["http_port"]
config = RAGFlowServerConfig(id=id_count, name=name, host=host, port=http_port, service_type="ragflow_server", detail_func_name="check_ragflow_server_alive")
configurations.append(config)
id_count += 1
case "es":
name: str = 'elasticsearch'
url = v['hosts']
name: str = "elasticsearch"
url = v["hosts"]
parsed = urlparse(url)
host: str = parsed.hostname
port: int = parsed.port
username: str = v.get('username')
password: str = v.get('password')
config = ElasticsearchConfig(id=id_count, name=name, host=host, port=port, service_type="retrieval",
retrieval_type="elasticsearch",
username=username, password=password,
detail_func_name="get_es_cluster_stats")
username: str = v.get("username")
password: str = v.get("password")
config = ElasticsearchConfig(
id=id_count,
name=name,
host=host,
port=port,
service_type="retrieval",
retrieval_type="elasticsearch",
username=username,
password=password,
detail_func_name="get_es_cluster_stats",
)
configurations.append(config)
id_count += 1
case "infinity":
name: str = 'infinity'
url = v['uri']
parts = url.split(':', 1)
name: str = "infinity"
url = v["uri"]
parts = url.split(":", 1)
host = parts[0]
port = int(parts[1])
database: str = v.get('db_name', 'default_db')
config = InfinityConfig(id=id_count, name=name, host=host, port=port, service_type="retrieval",
retrieval_type="infinity",
db_name=database, detail_func_name="get_infinity_status")
database: str = v.get("db_name", "default_db")
config = InfinityConfig(id=id_count, name=name, host=host, port=port, service_type="retrieval", retrieval_type="infinity", db_name=database, detail_func_name="get_infinity_status")
configurations.append(config)
id_count += 1
case "minio_0":
name: str = 'minio_0'
url = v['host']
parts = url.split(':', 1)
name: str = "minio_0"
url = v["host"]
parts = url.split(":", 1)
host = parts[0]
port = int(parts[1])
user = v.get('user')
password = v.get('password')
config = MinioConfig(id=id_count, name=name, host=host, port=port, user=user, password=password,
service_type="file_store",
store_type="minio", detail_func_name="check_minio_alive")
user = v.get("user")
password = v.get("password")
config = MinioConfig(id=id_count, name=name, host=host, port=port, user=user, password=password, service_type="file_store", store_type="minio", detail_func_name="check_minio_alive")
configurations.append(config)
id_count += 1
case "minio":
name: str = 'minio'
url = v['host']
parts = url.split(':', 1)
name: str = "minio"
url = v["host"]
parts = url.split(":", 1)
host = parts[0]
port = int(parts[1])
user = v.get('user')
password = v.get('password')
config = MinioConfig(id=id_count, name=name, host=host, port=port, user=user, password=password,
service_type="file_store",
store_type="minio", detail_func_name="check_minio_alive")
user = v.get("user")
password = v.get("password")
config = MinioConfig(id=id_count, name=name, host=host, port=port, user=user, password=password, service_type="file_store", store_type="minio", detail_func_name="check_minio_alive")
configurations.append(config)
id_count += 1
case "redis":
name: str = 'redis'
url = v['host']
parts = url.split(':', 1)
name: str = "redis"
url = v["host"]
parts = url.split(":", 1)
host = parts[0]
port = int(parts[1])
password = v.get('password')
db: int = v.get('db')
config = RedisConfig(id=id_count, name=name, host=host, port=port, password=password, database=db,
service_type="message_queue", mq_type="redis", detail_func_name="get_redis_info")
password = v.get("password")
db: int = v.get("db")
config = RedisConfig(id=id_count, name=name, host=host, port=port, password=password, database=db, service_type="message_queue", mq_type="redis", detail_func_name="get_redis_info")
configurations.append(config)
id_count += 1
case "mysql":
name: str = 'mysql'
host: str = v.get('host')
port: int = v.get('port')
username = v.get('user')
password = v.get('password')
config = MySQLConfig(id=id_count, name=name, host=host, port=port, username=username, password=password,
service_type="meta_data", meta_type="mysql", detail_func_name="get_mysql_status")
name: str = "mysql"
host: str = v.get("host")
port: int = v.get("port")
username = v.get("user")
password = v.get("password")
config = MySQLConfig(
id=id_count, name=name, host=host, port=port, username=username, password=password, service_type="meta_data", meta_type="mysql", detail_func_name="get_mysql_status"
)
configurations.append(config)
id_count += 1
case "admin":
pass
case "task_executor":
name: str = 'task_executor'
host: str = v.get('host', '')
port: int = v.get('port', 0)
message_queue_type: str = v.get('message_queue_type')
config = TaskExecutorConfig(id=id_count, name=name, host=host, port=port, message_queue_type=message_queue_type,
service_type="task_executor", detail_func_name="check_task_executor_alive")
name: str = "task_executor"
host: str = v.get("host", "")
port: int = v.get("port", 0)
message_queue_type: str = v.get("message_queue_type")
config = TaskExecutorConfig(
id=id_count, name=name, host=host, port=port, message_queue_type=message_queue_type, service_type="task_executor", detail_func_name="check_task_executor_alive"
)
configurations.append(config)
id_count += 1
case "rabbitmq":
name: str = 'rabbitmq'
host: str = v.get('host')
port: int = v.get('port')
config = RabbitMQConfig(id=id_count, name=name, host=host, port=port,
service_type="message_queue", mq_type="rabbitmq", detail_func_name="check_rabbitmq_alive")
name: str = "rabbitmq"
host: str = v.get("host")
port: int = v.get("port")
config = RabbitMQConfig(id=id_count, name=name, host=host, port=port, service_type="message_queue", mq_type="rabbitmq", detail_func_name="check_rabbitmq_alive")
configurations.append(config)
id_count += 1
case _:

View File

@@ -4,14 +4,17 @@ class AdminException(Exception):
self.code = code
self.message = message
class UserNotFoundError(AdminException):
def __init__(self, username):
super().__init__(f"User '{username}' not found", 404)
class UserAlreadyExistsError(AdminException):
def __init__(self, username):
super().__init__(f"User '{username}' already exists", 409)
class CannotDeleteAdminError(AdminException):
def __init__(self):
super().__init__("Cannot delete admin account", 403)
super().__init__("Cannot delete admin account", 403)

View File

@@ -17,16 +17,8 @@ from flask import jsonify
def success_response(data=None, message="Success", code=0):
return jsonify({
"code": code,
"message": message,
"data": data
}), 200
return jsonify({"code": code, "message": message, "data": data}), 200
def error_response(message="Error", code=-1, data=None):
return jsonify({
"code": code,
"message": message,
"data": data
}), 400
return jsonify({"code": code, "message": message, "data": data}), 400

View File

@@ -153,6 +153,8 @@ def change_password(username):
def alter_user_activate_status(username):
try:
data = request.get_json()
if current_user.email == username:
return error_response(f"can't alter current user status: {username}", 409)
if not data or "activate_status" not in data:
return error_response("Activation status is required", 400)
activate_status = data["activate_status"]

View File

@@ -489,10 +489,7 @@ class SandboxMgr:
"""List all available sandbox providers."""
result = []
for provider_id, metadata in SandboxMgr.PROVIDER_REGISTRY.items():
result.append({
"id": provider_id,
**metadata
})
result.append({"id": provider_id, **metadata})
return result
@staticmethod
@@ -635,6 +632,7 @@ class SandboxMgr:
config_json = json.dumps(config)
SettingsMgr.update_by_name(f"sandbox.{provider_type}", config_json)
from agent.sandbox.client import reload_provider
reload_provider()
return {"provider_type": provider_type, "config": config}
@@ -727,11 +725,7 @@ def main() -> dict:
# Build detailed result message
success = execution_result.exit_code == 0 and "TEST_PASSED" in execution_result.stdout
message_parts = [
f"Test {success and 'PASSED' or 'FAILED'}",
f"Exit code: {execution_result.exit_code}",
f"Execution time: {execution_result.execution_time:.2f}s"
]
message_parts = [f"Test {success and 'PASSED' or 'FAILED'}", f"Exit code: {execution_result.exit_code}", f"Execution time: {execution_result.execution_time:.2f}s"]
if execution_result.stdout.strip():
stdout_preview = execution_result.stdout.strip()[:200]
@@ -751,12 +745,13 @@ def main() -> dict:
"execution_time": execution_result.execution_time,
"stdout": execution_result.stdout,
"stderr": execution_result.stderr,
}
},
}
except AdminException:
raise
except Exception as e:
import traceback
error_details = traceback.format_exc()
raise AdminException(f"Connection test failed: {str(e)}\\n\\nStack trace:\\n{error_details}")

View File

@@ -25,18 +25,19 @@ import time
from concurrent.futures import ThreadPoolExecutor
from copy import deepcopy
from functools import partial
from typing import Any, Union, Tuple
from typing import Any, Tuple, Union
from agent.component import component_class
from agent.component.base import ComponentBase
from agent.dsl_migration import normalize_chunker_dsl
from api.db.joint_services.tenant_model_service import get_tenant_default_model_by_type
from api.db.services.file_service import FileService
from api.db.services.llm_service import LLMBundle
from api.db.services.task_service import has_canceled
from api.db.joint_services.tenant_model_service import get_tenant_default_model_by_type
from common.constants import LLMType
from common.misc_utils import get_uuid, hash_str2int
from common.llm_request_context import set_llm_request_context, reset_llm_request_context
from common.exceptions import TaskCanceledException
from common.misc_utils import get_uuid, hash_str2int
from common.token_utils import token_usage_sink, langfuse_run_attrs
from rag.prompts.generator import chunks_format
from rag.utils.redis_conn import REDIS_CONN
@@ -161,6 +162,7 @@ class Graph:
def close(self):
from common.mcp_tool_call_conn import MCPToolCallSession
seen = set()
for cpn in self.components.values():
obj = cpn.get("obj")
@@ -198,7 +200,9 @@ class Graph:
return self._tenant_id
def get_value_with_variable(self, value: str) -> Any:
pat = re.compile(r"\{* *\{([a-zA-Z:0-9]+@[A-Za-z0-9_.-]+|sys\.[A-Za-z0-9_.]+|env\.[A-Za-z0-9_.]+)\} *\}*")
# Reference the canonical pre-compiled regex from ComponentBase so
# the source-pattern and the runtime-pattern can never drift apart.
pat = ComponentBase.variable_ref_patt_re
out_parts = []
last = 0
@@ -228,7 +232,13 @@ class Graph:
exp = exp.strip("{").strip("}").strip(" ").strip("{").strip("}")
if exp.find("@") < 0:
return self.globals[exp]
cpn_id, var_nm = exp.split("@")
# Split from the left with maxsplit=1 so the trailing var_nm can
# legitimately contain '@' characters (defensive: although the
# upstream regex in `get_value_with_variable` constrains `var_nm`
# to `[A-Za-z0-9_.-]+`, direct callers of this method may pass
# any string and should not raise `ValueError: too many values
# to unpack`). `cpn_id` is system-generated and never contains '@'.
cpn_id, var_nm = exp.split("@", 1)
cpn = self.get_component(cpn_id)
if not cpn:
raise Exception(f"Can't find variable: '{cpn_id}@{var_nm}'")
@@ -275,7 +285,10 @@ class Graph:
if exp.find("@") < 0:
self.globals[exp] = value
return
cpn_id, var_nm = exp.split("@")
# See `get_variable_value` above for rationale on `maxsplit=1`.
# Without it, a var_nm containing '@' would raise
# `ValueError: too many values to unpack` instead of being preserved.
cpn_id, var_nm = exp.split("@", 1)
cpn = self.get_component(cpn_id)
if not cpn:
raise Exception(f"Can't find variable: '{cpn_id}@{var_nm}'")
@@ -428,6 +441,14 @@ class Canvas(Graph):
_lf_attrs["session_id"] = str(_session_id)[:200]
sink_token = token_usage_sink.set(self._run_token_usage)
attrs_token = langfuse_run_attrs.set(_lf_attrs)
# Forward the originating session/user to upstream LLM providers (as the
# OpenAI `user` field) for the duration of this run, and reset afterwards so
# the value never leaks to later calls in the same task. Reuse the same
# session/user already derived above so both integrations stay consistent.
_req_ctx_token = set_llm_request_context(
session_id=_session_id,
user_id=_user_id,
)
try:
async for ev in self._run_impl(**kwargs):
yield ev
@@ -444,6 +465,7 @@ class Canvas(Graph):
except ValueError:
logging.debug("Failed to reset Langfuse run attributes ContextVar", exc_info=True)
langfuse_run_attrs.set(None)
reset_llm_request_context(_req_ctx_token)
async def _run_impl(self, **kwargs):
self.globals["sys.date"] = datetime.datetime.now(datetime.timezone.utc).strftime("%Y-%m-%d %H:%M:%S")
@@ -455,7 +477,15 @@ class Canvas(Graph):
path_set = set(self.path)
for k, cpn in self.components.items():
if k in path_set:
self.components[k]["obj"].reset(True)
# Begin is intentionally kept as `only_output=True` to preserve existing behavior.
# (Begin/UserFillUp may populate `_param.inputs` during invocation; we leave that unchanged here.)
# All other path components must clear both
# inputs and outputs so the next run resolves refs against
# this run's runtime values (e.g. Await-response capture
# propagating to a downstream Agent's user_prompt), not
# against stale values from the previous canvas run.
is_begin = self.components[k]["obj"].component_name.lower() == "begin"
self.components[k]["obj"].reset(only_output=is_begin)
if kwargs.get("webhook_payload"):
for k, cpn in self.components.items():
@@ -529,11 +559,13 @@ class Canvas(Graph):
if use_async:
await cpn_obj.invoke_async(**(call_kwargs or {}))
return
# run_in_executor does not propagate context variables; copy the
# current context so the token usage sink / Langfuse attributes set
# by run() remain visible to LLMBundle calls inside sync components.
ctx = contextvars.copy_context()
await loop.run_in_executor(self._thread_pool, lambda: ctx.run(partial(sync_fn, **(call_kwargs or {}))))
# run_in_executor does not carry context variables into the worker
# thread; copy the current context so the LLM request context (the
# `user` forwarding), token usage sink, and Langfuse attributes set
# by run() remain visible to sync components.
bound_call = partial(sync_fn, **(call_kwargs or {}))
call_ctx = contextvars.copy_context()
await loop.run_in_executor(self._thread_pool, partial(call_ctx.run, bound_call))
i = f
while i < t:

View File

@@ -22,8 +22,9 @@ from typing import Dict, Type
_package_path = os.path.dirname(__file__)
__all_classes: Dict[str, Type] = {}
def _import_submodules() -> None:
for filename in os.listdir(_package_path): # noqa: F821
for filename in os.listdir(_package_path): # noqa: F821
if filename.startswith("__") or not filename.endswith(".py") or filename.startswith("base"):
continue
module_name = filename[:-3]
@@ -34,13 +35,14 @@ def _import_submodules() -> None:
except ImportError as e:
print(f"Warning: Failed to import module {module_name}: {str(e)}")
def _extract_classes_from_module(module: ModuleType) -> None:
for name, obj in inspect.getmembers(module):
if (inspect.isclass(obj) and
obj.__module__ == module.__name__ and not name.startswith("_")):
if inspect.isclass(obj) and obj.__module__ == module.__name__ and not name.startswith("_"):
__all_classes[name] = obj
globals()[name] = obj
_import_submodules()
__all__ = list(__all_classes.keys()) + ["__all_classes"]

View File

@@ -27,7 +27,7 @@ import json_repair
from agent.component.llm import LLM, LLMParam
from agent.tools.base import LLMToolPluginCallSession, ToolBase, ToolMeta, ToolParamBase
from api.db.joint_services.tenant_model_service import get_model_config_from_provider_instance, get_model_type_by_name
from api.db.joint_services.tenant_model_service import resolve_model_config, resolve_model_type
from api.db.services.llm_service import LLMBundle
from api.db.services.mcp_server_service import MCPServerService
from common.connection_utils import timeout
@@ -82,9 +82,9 @@ class Agent(LLM, ToolBase):
original_name = cpn.get_meta()["function"]["name"]
indexed_name = f"{original_name}_{idx}"
self.tools[indexed_name] = cpn
model_types = get_model_type_by_name(self._canvas.get_tenant_id(), self._param.llm_id)
model_types = resolve_model_type(self._canvas.get_tenant_id(), self._param.llm_id)
model_type = "chat" if "chat" in model_types else model_types[0]
chat_model_config = get_model_config_from_provider_instance(self._canvas.get_tenant_id(), model_type, self._param.llm_id)
chat_model_config = resolve_model_config(self._canvas.get_tenant_id(), model_type, self._param.llm_id)
self.chat_mdl = LLMBundle(
self._canvas.get_tenant_id(),
chat_model_config,

View File

@@ -15,20 +15,21 @@
#
import asyncio
import builtins
import json
import logging
import os
import re
import time
from abc import ABC
import builtins
import json
import os
import logging
from typing import Any, List, Union
import pandas as pd
from agent import settings
from common.connection_utils import timeout
from common.misc_utils import thread_pool_exec
_logger = logging.getLogger(__name__)
@@ -101,6 +102,7 @@ class ComponentParamBase(ABC):
return None
logging.warning("ComponentParamBase.__str__: JSON fallback via str() for type=%s", type(obj).__name__)
return str(obj)
return json.dumps(self.as_dict(), ensure_ascii=False, default=_serialize_default)
def as_dict(self):
@@ -135,15 +137,11 @@ class ComponentParamBase(ABC):
update_from_raw_conf = conf.get(_IS_RAW_CONF, True)
if update_from_raw_conf:
deprecated_params_set = self._get_or_init_deprecated_params_set()
feeded_deprecated_params_set = (
self._get_or_init_feeded_deprecated_params_set()
)
feeded_deprecated_params_set = self._get_or_init_feeded_deprecated_params_set()
user_feeded_params_set = self._get_or_init_user_feeded_params_set()
setattr(self, _IS_RAW_CONF, False)
else:
feeded_deprecated_params_set = (
self._get_or_init_feeded_deprecated_params_set(conf)
)
feeded_deprecated_params_set = self._get_or_init_feeded_deprecated_params_set(conf)
user_feeded_params_set = self._get_or_init_user_feeded_params_set(conf)
def _recursive_update_param(param, config, depth, prefix):
@@ -179,15 +177,11 @@ class ComponentParamBase(ABC):
else:
# recursive set obj attr
sub_params = _recursive_update_param(
attr, config_value, depth + 1, prefix=f"{prefix}{config_key}."
)
sub_params = _recursive_update_param(attr, config_value, depth + 1, prefix=f"{prefix}{config_key}.")
setattr(param, config_key, sub_params)
if not allow_redundant and redundant_attrs:
raise ValueError(
f"cpn `{getattr(self, '_name', type(self))}` has redundant parameters: `{[redundant_attrs]}`"
)
raise ValueError(f"cpn `{getattr(self, '_name', type(self))}` has redundant parameters: `{[redundant_attrs]}`")
return param
@@ -218,9 +212,7 @@ class ComponentParamBase(ABC):
param_validation_path_prefix = home_dir + "/param_validation/"
param_name = type(self).__name__
param_validation_path = "/".join(
[param_validation_path_prefix, param_name + ".json"]
)
param_validation_path = "/".join([param_validation_path_prefix, param_name + ".json"])
validation_json = None
@@ -253,11 +245,7 @@ class ComponentParamBase(ABC):
break
if not value_legal:
raise ValueError(
"Please check runtime conf, {} = {} does not match user-parameter restriction".format(
variable, value
)
)
raise ValueError("Please check runtime conf, {} = {} does not match user-parameter restriction".format(variable, value))
elif variable in validation_json:
self._validate_param(attr, validation_json)
@@ -335,11 +323,7 @@ class ComponentParamBase(ABC):
def _range(value, ranges):
in_range = False
for left_limit, right_limit in ranges:
if (
left_limit - settings.FLOAT_ZERO
<= value
<= right_limit + settings.FLOAT_ZERO
):
if left_limit - settings.FLOAT_ZERO <= value <= right_limit + settings.FLOAT_ZERO:
in_range = True
break
@@ -355,16 +339,11 @@ class ComponentParamBase(ABC):
def _warn_deprecated_param(self, param_name, description):
if self._deprecated_params_set.get(param_name):
logging.warning(
f"{description} {param_name} is deprecated and ignored in this version."
)
logging.warning(f"{description} {param_name} is deprecated and ignored in this version.")
def _warn_to_deprecate_param(self, param_name, description, new_param):
if self._deprecated_params_set.get(param_name):
logging.warning(
f"{description} {param_name} will be deprecated in future release; "
f"please use {new_param} instead."
)
logging.warning(f"{description} {param_name} will be deprecated in future release; please use {new_param} instead.")
return True
return False
@@ -372,8 +351,14 @@ class ComponentParamBase(ABC):
class ComponentBase(ABC):
component_name: str
thread_limiter = asyncio.Semaphore(int(os.environ.get("MAX_CONCURRENT_CHATS", 10)))
variable_ref_patt = r"\{* *\{([a-zA-Z:0-9]+@[A-Za-z0-9_.-]+|sys\.[A-Za-z0-9_.]+|env\.[A-Za-z0-9_.]+)\} *\}*"
# Match `cpn_id@var_nm` / `sys.var_nm` / `env.var_nm` style template refs.
# `cpn_id` allows underscores (frontend ids like `userfillup_abc`,
# `retrieval_xyz`) and colons (legacy DSL ids like `UserFillUp:CateInput`,
# `Retrieval:KBSearch`).
variable_ref_patt = r"\{* *\{([a-zA-Z0-9_:]+@[A-Za-z0-9_.-]+|sys\.[A-Za-z0-9_.]+|env\.[A-Za-z0-9_.]+)\} *\}*"
variable_ref_patt_re = re.compile(variable_ref_patt, flags=re.IGNORECASE | re.DOTALL)
iteration_alias_patt = r"\{* *\{(item|index|result)\} *\}*"
iteration_alias_patt_re = re.compile(iteration_alias_patt, flags=re.IGNORECASE | re.DOTALL)
def __str__(self):
"""
@@ -385,9 +370,7 @@ class ComponentBase(ABC):
return """{{
"component_name": "{}",
"params": {}
}}""".format(self.component_name,
self._param
)
}}""".format(self.component_name, self._param)
def __init__(self, canvas, id, param: ComponentParamBase):
from agent.canvas import Graph # Local import to avoid cyclic dependency
@@ -403,7 +386,7 @@ class ComponentBase(ABC):
def check_if_canceled(self, message: str = "") -> bool:
if self.is_canceled():
task_id = getattr(self._canvas, 'task_id', 'unknown')
task_id = getattr(self._canvas, "task_id", "unknown")
log_message = f"Task {task_id} has been canceled"
if message:
log_message += f" during {message}"
@@ -491,7 +474,9 @@ class ComponentBase(ABC):
input_elements = self.get_input_elements()
_logger.debug(
"[Base] Component '%s' (%s) resolving inputs. Input element keys: %s",
self._id, self.component_name, list(input_elements.keys()),
self._id,
self.component_name,
list(input_elements.keys()),
)
for var, o in input_elements.items():
v = self.get_param(var)
@@ -502,9 +487,9 @@ class ComponentBase(ABC):
resolved = self._canvas.get_variable_value(v)
self.set_input_value(var, resolved)
_logger.debug("[Base] var '%s': resolved ref '%s' -> %s", var, v, json.dumps(resolved, ensure_ascii=False, default=str)[:200])
elif isinstance(v, str) and re.search(self.variable_ref_patt, v):
elif isinstance(v, str) and self.variable_ref_patt_re.search(v):
elements = self.get_input_elements_from_text(v)
kv = {k: e.get('value', '') for k, e in elements.items()}
kv = {k: e.get("value", "") for k, e in elements.items()}
self.set_input_value(var, self.string_format(v, kv))
_logger.debug("[Base] var '%s': resolved text refs '%s' -> %s", var, v, json.dumps(kv, ensure_ascii=False, default=str)[:200])
else:
@@ -538,16 +523,21 @@ class ComponentBase(ABC):
def get_input_elements_from_text(self, txt: str) -> dict[str, dict[str, str]]:
res = {}
for r in re.finditer(self.variable_ref_patt, txt, flags=re.IGNORECASE | re.DOTALL):
for r in self.variable_ref_patt_re.finditer(txt):
exp = r.group(1)
cpn_id, var_nm = exp.split("@") if exp.find("@") > 0 else ("", exp)
# Use maxsplit=1 to be defensive: although `exp` here comes
# from `variable_ref_patt` (which constrains `var_nm` to
# `[A-Za-z0-9_.-]+`), a future regex relaxation or a non-
# pattern caller should not raise `ValueError: too many values
# to unpack` if the trailing part happens to contain '@'.
cpn_id, var_nm = exp.split("@", 1) if exp.find("@") > 0 else ("", exp)
res[exp] = {
"name": (self._canvas.get_component_name(cpn_id) + f"@{var_nm}") if cpn_id else exp,
"value": self._canvas.get_variable_value(exp),
"_retrieval": self._canvas.get_variable_value(f"{cpn_id}@_references") if cpn_id else None,
"_cpn_id": cpn_id
"_cpn_id": cpn_id,
}
for r in re.finditer(self.iteration_alias_patt, txt, flags=re.IGNORECASE | re.DOTALL):
for r in self.iteration_alias_patt_re.finditer(txt):
exp = r.group(1)
if exp in res:
continue
@@ -559,7 +549,7 @@ class ComponentBase(ABC):
"name": (self._canvas.get_component_name(cpn_id) + f"@{var_nm}"),
"value": self._canvas.get_variable_value(ref),
"_retrieval": self._canvas.get_variable_value(f"{cpn_id}@_references"),
"_cpn_id": cpn_id
"_cpn_id": cpn_id,
}
return res
@@ -601,33 +591,27 @@ class ComponentBase(ABC):
return self._canvas.get_component(pid)["obj"]
def get_upstream(self) -> List[str]:
cpn_nms = self._canvas.get_component(self._id)['upstream']
cpn_nms = self._canvas.get_component(self._id)["upstream"]
return cpn_nms
def get_downstream(self) -> List[str]:
cpn_nms = self._canvas.get_component(self._id)['downstream']
cpn_nms = self._canvas.get_component(self._id)["downstream"]
return cpn_nms
@staticmethod
def string_format(content: str, kv: dict[str, str]) -> str:
for n, v in kv.items():
def repl(_match, val=v):
return str(val) if val is not None else ""
content = re.sub(
r"\{%s\}" % re.escape(n),
repl,
content
)
content = re.sub(r"\{%s\}" % re.escape(n), repl, content)
return content
def exception_handler(self):
if not self._param.exception_method:
return None
return {
"goto": self._param.exception_goto,
"default_value": self._param.exception_default_value
}
return {"goto": self._param.exception_goto, "default_value": self._param.exception_default_value}
def get_exception_default_value(self):
if self._param.exception_method != "comment":

View File

@@ -17,17 +17,17 @@ from agent.component.fillup import UserFillUpParam, UserFillUp
class BeginParam(UserFillUpParam):
"""
Define the Begin component parameters.
"""
def __init__(self):
super().__init__()
self.mode = "conversational"
self.prologue = "Hi! I'm your smart assistant. What can I do for you?"
def check(self):
self.check_valid_value(self.mode, "The 'mode' should be either `conversational` or `task`", ["conversational", "task","Webhook"])
self.check_valid_value(self.mode, "The 'mode' should be either `conversational` or `task`", ["conversational", "task", "Webhook"])
def get_input_form(self) -> dict[str, dict]:
return getattr(self, "inputs")

View File

@@ -33,7 +33,7 @@ from urllib.request import Request, urlopen
from agent.component.base import ComponentBase
from agent.component.llm import LLMParam
from api.db import FileType
from api.db.joint_services.tenant_model_service import get_model_config_from_provider_instance, get_model_type_by_name
from api.db.joint_services.tenant_model_service import resolve_model_config, resolve_model_type
from api.db.services import duplicate_name
from api.db.services.file_service import FileService
from api.utils.file_utils import filename_type
@@ -371,9 +371,7 @@ class Browser(ComponentBase, ABC):
sort_keys=True,
ensure_ascii=False,
)
raw_canvas_id = (
f"dsl_{hashlib.sha1(graph_text.encode('utf-8')).hexdigest()[:12]}"
)
raw_canvas_id = f"dsl_{hashlib.sha1(graph_text.encode('utf-8')).hexdigest()[:12]}"
canvas_id = self._safe_path_segment(raw_canvas_id)
node_id = self._safe_path_segment(self._id)
return os.path.join(root, tenant, canvas_id, node_id)
@@ -402,9 +400,9 @@ class Browser(ComponentBase, ABC):
def _build_browser_llm(self):
from browser_use.llm import ChatBrowserUse, ChatOpenAI
chat_model_config = get_model_config_from_provider_instance(
chat_model_config = resolve_model_config(
self._canvas.get_tenant_id(),
get_model_type_by_name(self._canvas.get_tenant_id(), self._param.llm_id),
resolve_model_type(self._canvas.get_tenant_id(), self._param.llm_id),
self._param.llm_id,
)
cfg = self._as_model_config_dict(chat_model_config)
@@ -488,10 +486,7 @@ class Browser(ComponentBase, ABC):
# Keep browser-use watchdog fallback in sync with our resolved path.
os.environ["BROWSER_USE_BROWSER_BINARY_PATH"] = executable_path
else:
logging.warning(
"Browser no local browser executable found. "
"Set BROWSER_USE_EXECUTABLE_PATH or preinstall chromium in image to avoid runtime playwright install."
)
logging.warning("Browser no local browser executable found. Set BROWSER_USE_EXECUTABLE_PATH or preinstall chromium in image to avoid runtime playwright install.")
if profile_dir:
browser_kwargs["user_data_dir"] = profile_dir
# browser-use expects profile_directory to be a profile name
@@ -682,21 +677,13 @@ class Browser(ComponentBase, ABC):
try:
self._prepare_input_values()
user_prompt = self._resolve_text(kwargs.get("prompts", self._param.prompts))
with tempfile.TemporaryDirectory(prefix="browser_use_upload_") as upload_dir, tempfile.TemporaryDirectory(
prefix="browser_use_download_"
) as download_dir:
with tempfile.TemporaryDirectory(prefix="browser_use_upload_") as upload_dir, tempfile.TemporaryDirectory(prefix="browser_use_download_") as download_dir:
uploaded_files = self._prepare_upload_files(upload_dir)
upload_lines = [
f"- file_id={item['file_id']}, name={item['name']}, local_path={item['local_path']}"
for item in uploaded_files
]
upload_lines = [f"- file_id={item['file_id']}, name={item['name']}, local_path={item['local_path']}" for item in uploaded_files]
task_text = user_prompt
if upload_lines:
task_text += (
"\n\nYou can upload files from these local paths when operating web pages:\n"
+ "\n".join(upload_lines)
)
task_text += "\n\nYou can upload files from these local paths when operating web pages:\n" + "\n".join(upload_lines)
upload_local_paths = [item.get("local_path", "") for item in uploaded_files if item.get("local_path")]
if persist_session:
@@ -707,11 +694,7 @@ class Browser(ComponentBase, ABC):
profile_dir = tempfile.mkdtemp(prefix="browser_use_profile_")
except OSError:
profile_dir = None
history = asyncio.run(
self._run_browser_use_async(
task_text, download_dir, upload_local_paths, profile_dir
)
)
history = asyncio.run(self._run_browser_use_async(task_text, download_dir, upload_local_paths, profile_dir))
target_dir_id = FileService.get_root_folder(self._canvas.get_tenant_id())["id"]
downloaded_files = self._save_downloads(download_dir, target_dir_id)

View File

@@ -21,17 +21,17 @@ from abc import ABC
from common.constants import LLMType
from api.db.services.llm_service import LLMBundle
from api.db.joint_services.tenant_model_service import get_model_config_from_provider_instance
from api.db.joint_services.tenant_model_service import resolve_model_config
from agent.component.llm import LLMParam, LLM
from common.connection_utils import timeout
from rag.llm.chat_model import ERROR_PREFIX
class CategorizeParam(LLMParam):
"""
Define the categorize component parameters.
"""
def __init__(self):
super().__init__()
self.category_description = {}
@@ -50,12 +50,7 @@ class CategorizeParam(LLMParam):
raise ValueError(f"[Categorize] 'To' of category {k} can not be empty!")
def get_input_form(self) -> dict[str, dict]:
return {
"query": {
"type": "line",
"name": "Query"
}
}
return {"query": {"type": "line", "name": "Query"}}
def update_prompt(self):
cate_lines = []
@@ -63,13 +58,12 @@ class CategorizeParam(LLMParam):
for line in desc.get("examples", []):
if not line:
continue
cate_lines.append("USER: \"" + re.sub(r"\n", " ", line, flags=re.DOTALL) + "\""+c)
cate_lines.append('USER: "' + re.sub(r"\n", " ", line, flags=re.DOTALL) + '"' + c)
descriptions = []
for c, desc in self.category_description.items():
if desc.get("description"):
descriptions.append(
"\n------\nCategory: {}\nDescription: {}".format(c, desc["description"]))
descriptions.append("\n------\nCategory: {}\nDescription: {}".format(c, desc["description"]))
self.sys_prompt = """
You are an advanced classification system that categorizes user questions into specific types. Analyze the input question and classify it into ONE of the following categories:
@@ -84,10 +78,7 @@ Here's description of each category:
- Return only the category name without explanations
- Use "Other" only when no other category fits
""".format(
"\n - ".join(list(self.category_description.keys())),
"\n".join(descriptions)
)
""".format("\n - ".join(list(self.category_description.keys())), "\n".join(descriptions))
if cate_lines:
self.sys_prompt += """
@@ -106,7 +97,7 @@ class Categorize(LLM, ABC):
logging.warning(f"[Categorize] input element not detected for query key: {query_key}")
return elements
@timeout(int(os.environ.get("COMPONENT_EXEC_TIMEOUT", 10*60)))
@timeout(int(os.environ.get("COMPONENT_EXEC_TIMEOUT", 10 * 60)))
async def _invoke_async(self, **kwargs):
if self.check_if_canceled("Categorize processing"):
return
@@ -124,13 +115,13 @@ class Categorize(LLM, ABC):
msg[-1]["content"] = query_value
self.set_input_value(query_key, msg[-1]["content"])
self._param.update_prompt()
chat_model_config = get_model_config_from_provider_instance(self._canvas.get_tenant_id(), LLMType.CHAT, self._param.llm_id)
chat_model_config = resolve_model_config(self._canvas.get_tenant_id(), LLMType.CHAT, self._param.llm_id)
chat_mdl = LLMBundle(self._canvas.get_tenant_id(), chat_model_config)
user_prompt = """
---- Real Data ----
{}
""".format(" | ".join(["{}: \"{}\"".format(c["role"].upper(), re.sub(r"\n", "", c["content"], flags=re.DOTALL)) for c in msg]))
""".format(" | ".join(['{}: "{}"'.format(c["role"].upper(), re.sub(r"\n", "", c["content"], flags=re.DOTALL)) for c in msg]))
if self.check_if_canceled("Categorize processing"):
return
@@ -158,7 +149,7 @@ class Categorize(LLM, ABC):
self.set_output("category_name", max_category)
self.set_output("_next", cpn_ids)
@timeout(int(os.environ.get("COMPONENT_EXEC_TIMEOUT", 10*60)))
@timeout(int(os.environ.get("COMPONENT_EXEC_TIMEOUT", 10 * 60)))
def _invoke(self, **kwargs):
return asyncio.run(self._invoke_async(**kwargs))

View File

@@ -19,55 +19,47 @@ import os
from agent.component.base import ComponentBase, ComponentParamBase
from api.utils.api_utils import timeout
class DataOperationsParam(ComponentParamBase):
"""
Define the Data Operations component parameters.
"""
def __init__(self):
super().__init__()
self.query = []
self.operations = "literal_eval"
self.select_keys = []
self.filter_values=[]
self.updates=[]
self.remove_keys=[]
self.rename_keys=[]
self.outputs = {
"result": {
"value": [],
"type": "Array of Object"
}
}
def check(self):
self.check_valid_value(self.operations, "Support operations", ["select_keys", "literal_eval","combine","filter_values","append_or_update","remove_keys","rename_keys"])
self.filter_values = []
self.updates = []
self.remove_keys = []
self.rename_keys = []
self.outputs = {"result": {"value": [], "type": "Array of Object"}}
class DataOperations(ComponentBase,ABC):
def check(self):
self.check_valid_value(self.operations, "Support operations", ["select_keys", "literal_eval", "combine", "filter_values", "append_or_update", "remove_keys", "rename_keys"])
class DataOperations(ComponentBase, ABC):
component_name = "DataOperations"
def get_input_form(self) -> dict[str, dict]:
return {
k: {"name": o.get("name", ""), "type": "line"}
for input_item in (self._param.query or [])
for k, o in self.get_input_elements_from_text(input_item).items()
}
return {k: {"name": o.get("name", ""), "type": "line"} for input_item in (self._param.query or []) for k, o in self.get_input_elements_from_text(input_item).items()}
@timeout(int(os.environ.get("COMPONENT_EXEC_TIMEOUT", 10*60)))
@timeout(int(os.environ.get("COMPONENT_EXEC_TIMEOUT", 10 * 60)))
def _invoke(self, **kwargs):
self.input_objects=[]
self.input_objects = []
inputs = getattr(self._param, "query", None)
if not isinstance(inputs, (list, tuple)):
inputs = [inputs]
for input_ref in inputs:
input_object=self._canvas.get_variable_value(input_ref)
input_object = self._canvas.get_variable_value(input_ref)
self.set_input_value(input_ref, input_object)
if input_object is None:
continue
if isinstance(input_object,dict):
if isinstance(input_object, dict):
self.input_objects.append(input_object)
elif isinstance(input_object,list):
elif isinstance(input_object, list):
self.input_objects.extend(x for x in input_object if isinstance(x, dict))
else:
continue
@@ -85,13 +77,12 @@ class DataOperations(ComponentBase,ABC):
self._remove_keys()
else:
self._rename_keys()
def _select_keys(self):
filter_criteria: list[str] = self._param.select_keys
results = [{key: value for key, value in data_dict.items() if key in filter_criteria} for data_dict in self.input_objects]
self.set_output("result", results)
def _recursive_eval(self, data):
if isinstance(data, dict):
return {k: self._recursive_eval(v) for k, v in data.items()}
@@ -99,23 +90,19 @@ class DataOperations(ComponentBase,ABC):
return [self._recursive_eval(item) for item in data]
if isinstance(data, str):
try:
if (
data.strip().startswith(("{", "[", "(", "'", '"'))
or data.strip().lower() in ("true", "false", "none")
or data.strip().replace(".", "").isdigit()
):
if data.strip().startswith(("{", "[", "(", "'", '"')) or data.strip().lower() in ("true", "false", "none") or data.strip().replace(".", "").isdigit():
return ast.literal_eval(data)
except (ValueError, SyntaxError, TypeError, MemoryError):
return data
else:
return data
return data
def _literal_eval(self):
self.set_output("result", self._recursive_eval(self.input_objects))
def _combine(self):
result={}
result = {}
for obj in self.input_objects:
for key, value in obj.items():
if key not in result:
@@ -126,15 +113,13 @@ class DataOperations(ComponentBase,ABC):
else:
result[key].append(value)
else:
result[key] = (
[result[key], value] if not isinstance(value, list) else [result[key], *value]
)
result[key] = [result[key], value] if not isinstance(value, list) else [result[key], *value]
self.set_output("result", result)
def norm(self,v):
def norm(self, v):
s = "" if v is None else str(v)
return s
def match_rule(self, obj, rule):
key = rule.get("key")
op = (rule.get("operator") or "equals").lower()
@@ -155,10 +140,10 @@ class DataOperations(ComponentBase,ABC):
if op == "end with":
return v.endswith(target)
return False
def _filter_values(self):
results=[]
rules = (getattr(self._param, "filter_values", None) or [])
results = []
rules = getattr(self._param, "filter_values", None) or []
for obj in self.input_objects:
if not rules:
results.append(obj)
@@ -166,11 +151,10 @@ class DataOperations(ComponentBase,ABC):
if all(self.match_rule(obj, r) for r in rules):
results.append(obj)
self.set_output("result", results)
def _append_or_update(self):
results=[]
updates = getattr(self._param, "updates", []) or []
results = []
updates = getattr(self._param, "updates", []) or []
for obj in self.input_objects:
new_obj = dict(obj)
for item in updates:
@@ -187,7 +171,7 @@ class DataOperations(ComponentBase,ABC):
results = []
remove_keys = getattr(self._param, "remove_keys", []) or []
for obj in (self.input_objects or []):
for obj in self.input_objects or []:
new_obj = dict(obj)
for k in remove_keys:
if not isinstance(k, str):
@@ -200,7 +184,7 @@ class DataOperations(ComponentBase,ABC):
results = []
rename_pairs = getattr(self._param, "rename_keys", []) or []
for obj in (self.input_objects or []):
for obj in self.input_objects or []:
new_obj = dict(obj)
for pair in rename_pairs:
if not isinstance(pair, dict):

View File

@@ -13,6 +13,7 @@ from xml.sax.saxutils import escape
from agent.component.base import ComponentParamBase
from api.utils.api_utils import timeout
from api.utils.file_response import agent_attachment_preview_path
from common import settings
from common.misc_utils import get_uuid
from .message import Message
@@ -136,6 +137,7 @@ class DocGenerator(Message, ABC):
"mime_type": mime_type,
"size": file_size,
"base64": file_base64,
"preview_url": agent_attachment_preview_path(doc_id, ext=output_format, mime_type=mime_type),
"include_download_info_in_content": self._param.include_download_info_in_content,
}
self.set_output("doc_id", doc_id)
@@ -163,6 +165,7 @@ class DocGenerator(Message, ABC):
logging.info("Starting document generation, content length: %s chars", len(content))
if content:
def _replace_variable(match_obj: re.Match[str]) -> str:
match = match_obj.group(1)
try:
@@ -186,7 +189,7 @@ class DocGenerator(Message, ABC):
flags=re.DOTALL,
)
return content
return self._strip_thinking(content)
def _get_output_directory(self) -> str:
os.makedirs(self._default_output_directory, exist_ok=True)

View File

@@ -39,71 +39,52 @@ class ExcelProcessorParam(ComponentParamBase):
"""
Define the ExcelProcessor component parameters.
"""
def __init__(self):
super().__init__()
# Input configuration
self.input_files = [] # Variable references to uploaded files
self.operation = "read" # read, merge, transform, output
# Processing options
self.sheet_selection = "all" # all, first, or comma-separated sheet names
self.merge_strategy = "concat" # concat, join
self.join_on = "" # Column name for join operations
# Transform options (for LLM-guided transformations)
self.transform_instructions = ""
self.transform_data = "" # Variable reference to transformation data
# Output options
self.output_format = "xlsx" # xlsx, csv
self.output_filename = "output"
# Component outputs
self.outputs = {
"data": {
"type": "object",
"value": {}
},
"summary": {
"type": "str",
"value": ""
},
"markdown": {
"type": "str",
"value": ""
}
}
self.outputs = {"data": {"type": "object", "value": {}}, "summary": {"type": "str", "value": ""}, "markdown": {"type": "str", "value": ""}}
def check(self):
self.check_valid_value(
self.operation,
"[ExcelProcessor] Operation",
["read", "merge", "transform", "output"]
)
self.check_valid_value(
self.output_format,
"[ExcelProcessor] Output format",
["xlsx", "csv"]
)
self.check_valid_value(self.operation, "[ExcelProcessor] Operation", ["read", "merge", "transform", "output"])
self.check_valid_value(self.output_format, "[ExcelProcessor] Output format", ["xlsx", "csv"])
return True
class ExcelProcessor(ComponentBase, ABC):
"""
Excel processing component for RAGFlow agents.
Operations:
- read: Parse Excel files into structured data
- merge: Combine multiple Excel files
- transform: Apply data transformations based on instructions
- output: Generate Excel file output
"""
component_name = "ExcelProcessor"
def get_input_form(self) -> dict[str, dict]:
"""Define input form for the component."""
res = {}
for ref in (self._param.input_files or []):
for ref in self._param.input_files or []:
for k, o in self.get_input_elements_from_text(ref).items():
res[k] = {"name": o.get("name", ""), "type": "file"}
if self._param.transform_data:
@@ -111,13 +92,13 @@ class ExcelProcessor(ComponentBase, ABC):
res[k] = {"name": o.get("name", ""), "type": "object"}
return res
@timeout(int(os.environ.get("COMPONENT_EXEC_TIMEOUT", 10*60)))
@timeout(int(os.environ.get("COMPONENT_EXEC_TIMEOUT", 10 * 60)))
def _invoke(self, **kwargs):
if self.check_if_canceled("ExcelProcessor processing"):
return
operation = self._param.operation.lower()
if operation == "read":
self._read_excels()
elif operation == "merge":
@@ -137,7 +118,7 @@ class ExcelProcessor(ComponentBase, ABC):
value = self._canvas.get_variable_value(file_ref)
if value is None:
return None, None
# Handle different value formats
if isinstance(value, dict):
# File reference from Begin/UserFillUp component
@@ -154,12 +135,13 @@ class ExcelProcessor(ComponentBase, ABC):
# Could be base64 encoded or a path
if value.startswith("data:"):
import base64
# Extract base64 content
_, encoded = value.split(",", 1)
return base64.b64decode(encoded), "uploaded.xlsx"
return None, None
def _get_file_content_from_list(self, item) -> tuple[bytes, str]:
"""Extract file content from a list item."""
if isinstance(item, dict):
@@ -170,15 +152,15 @@ class ExcelProcessor(ComponentBase, ABC):
"""Parse Excel content into a dictionary of DataFrames (one per sheet)."""
try:
excel_file = BytesIO(content)
if filename.lower().endswith(".csv"):
df = pd.read_csv(excel_file)
return {"Sheet1": df}
else:
# Read all sheets
xlsx = pd.ExcelFile(excel_file, engine='openpyxl')
xlsx = pd.ExcelFile(excel_file, engine="openpyxl")
sheet_selection = self._param.sheet_selection
if sheet_selection == "all":
sheets_to_read = xlsx.sheet_names
elif sheet_selection == "first":
@@ -187,12 +169,12 @@ class ExcelProcessor(ComponentBase, ABC):
# Comma-separated sheet names
requested = [s.strip() for s in sheet_selection.split(",")]
sheets_to_read = [s for s in requested if s in xlsx.sheet_names]
dfs = {}
for sheet in sheets_to_read:
dfs[sheet] = pd.read_excel(xlsx, sheet_name=sheet)
return dfs
except Exception as e:
logging.error(f"Error parsing Excel file {filename}: {e}")
return {}
@@ -202,36 +184,36 @@ class ExcelProcessor(ComponentBase, ABC):
all_data = {}
summaries = []
markdown_parts = []
for file_ref in (self._param.input_files or []):
for file_ref in self._param.input_files or []:
if self.check_if_canceled("ExcelProcessor reading"):
return
# Get variable value
value = self._canvas.get_variable_value(file_ref)
self.set_input_value(file_ref, str(value)[:200] if value else "")
if value is None:
continue
# Handle file content
content, filename = self._get_file_content(file_ref)
if content is None:
continue
# Parse Excel
dfs = self._parse_excel_to_dataframes(content, filename)
for sheet_name, df in dfs.items():
key = f"{filename}_{sheet_name}" if len(dfs) > 1 else filename
all_data[key] = df.to_dict(orient="records")
# Build summary
summaries.append(f"**{key}**: {len(df)} rows, {len(df.columns)} columns ({', '.join(df.columns.tolist()[:5])}{'...' if len(df.columns) > 5 else ''})")
# Build markdown table
markdown_parts.append(f"### {key}\n\n{df.head(10).to_markdown(index=False)}\n")
# Set outputs
self.set_output("data", all_data)
self.set_output("summary", "\n".join(summaries) if summaries else "No Excel files found")
@@ -240,29 +222,29 @@ class ExcelProcessor(ComponentBase, ABC):
def _merge_excels(self):
"""Merge multiple Excel files/sheets into one."""
all_dfs = []
for file_ref in (self._param.input_files or []):
for file_ref in self._param.input_files or []:
if self.check_if_canceled("ExcelProcessor merging"):
return
value = self._canvas.get_variable_value(file_ref)
self.set_input_value(file_ref, str(value)[:200] if value else "")
if value is None:
continue
content, filename = self._get_file_content(file_ref)
if content is None:
continue
dfs = self._parse_excel_to_dataframes(content, filename)
all_dfs.extend(dfs.values())
if not all_dfs:
self.set_output("data", {})
self.set_output("summary", "No data to merge")
return
# Merge strategy
if self._param.merge_strategy == "concat":
merged_df = pd.concat(all_dfs, ignore_index=True)
@@ -273,7 +255,7 @@ class ExcelProcessor(ComponentBase, ABC):
merged_df = merged_df.merge(df, on=self._param.join_on, how="outer")
else:
merged_df = pd.concat(all_dfs, ignore_index=True)
self.set_output("data", {"merged": merged_df.to_dict(orient="records")})
self.set_output("summary", f"Merged {len(all_dfs)} sources into {len(merged_df)} rows, {len(merged_df.columns)} columns")
self.set_output("markdown", merged_df.head(20).to_markdown(index=False))
@@ -285,14 +267,14 @@ class ExcelProcessor(ComponentBase, ABC):
if not transform_ref:
self.set_output("summary", "No transform data reference provided")
return
data = self._canvas.get_variable_value(transform_ref)
self.set_input_value(transform_ref, str(data)[:300] if data else "")
if data is None:
self.set_output("summary", "Transform data is empty")
return
# Convert to DataFrame
if isinstance(data, dict):
# Could be {"sheet": [rows]} format
@@ -315,7 +297,7 @@ class ExcelProcessor(ComponentBase, ABC):
else:
self.set_output("data", {"raw": str(data)})
self.set_output("markdown", str(data))
self.set_output("summary", "Transformed data ready for processing")
def _output_excel(self):
@@ -325,14 +307,14 @@ class ExcelProcessor(ComponentBase, ABC):
if not transform_ref:
self.set_output("summary", "No data reference for output")
return
data = self._canvas.get_variable_value(transform_ref)
self.set_input_value(transform_ref, str(data)[:300] if data else "")
if data is None:
self.set_output("summary", "No data to output")
return
try:
# Prepare DataFrames
if isinstance(data, dict):
@@ -346,10 +328,10 @@ class ExcelProcessor(ComponentBase, ABC):
else:
self.set_output("summary", "Invalid data format for Excel output")
return
# Generate output
doc_id = get_uuid()
if self._param.output_format == "csv":
# For CSV, only output first sheet
first_df = list(dfs.values())[0]
@@ -358,7 +340,7 @@ class ExcelProcessor(ComponentBase, ABC):
else:
# Excel output
excel_io = BytesIO()
with pd.ExcelWriter(excel_io, engine='openpyxl') as writer:
with pd.ExcelWriter(excel_io, engine="openpyxl") as writer:
for sheet_name, df in dfs.items():
# Sanitize sheet name (max 31 chars, no special chars)
safe_name = sheet_name[:31].replace("/", "_").replace("\\", "_")
@@ -366,23 +348,19 @@ class ExcelProcessor(ComponentBase, ABC):
excel_io.seek(0)
binary_content = excel_io.read()
filename = f"{self._param.output_filename}.xlsx"
# Store file
settings.STORAGE_IMPL.put(self._canvas._tenant_id, doc_id, binary_content)
# Set attachment output
self.set_output("attachment", {
"doc_id": doc_id,
"format": self._param.output_format,
"file_name": filename
})
self.set_output("attachment", {"doc_id": doc_id, "format": self._param.output_format, "file_name": filename})
total_rows = sum(len(df) for df in dfs.values())
self.set_output("summary", f"Generated {filename} with {len(dfs)} sheet(s), {total_rows} total rows")
self.set_output("data", {k: v.to_dict(orient="records") for k, v in dfs.items()})
logging.info(f"ExcelProcessor: Generated {filename} as {doc_id}")
except Exception as e:
logging.error(f"ExcelProcessor output error: {e}")
self.set_output("summary", f"Error generating output: {str(e)}")

View File

@@ -29,4 +29,4 @@ class ExitLoop(ComponentBase, ABC):
pass
def thoughts(self) -> str:
return ""
return ""

View File

@@ -25,7 +25,6 @@ _INITIAL_USER_INPUT_CONSUMED_KEY = "sys.__initial_user_input_consumed__"
class UserFillUpParam(ComponentParamBase):
def __init__(self):
super().__init__()
self.enable_tips = True
@@ -55,11 +54,7 @@ class UserFillUp(ComponentBase):
return {}
if isinstance(query, dict):
matched = {
key: value if isinstance(value, dict) else {"value": value}
for key, value in query.items()
if key in fields
}
matched = {key: value if isinstance(value, dict) else {"value": value} for key, value in query.items() if key in fields}
if matched:
self._canvas.globals[_INITIAL_USER_INPUT_CONSUMED_KEY] = True
return matched
@@ -81,7 +76,13 @@ class UserFillUp(ComponentBase):
return FileService.get_files(files, layout_recognize=layout_recognize)
if isinstance(value, dict):
return value.get("value")
raw = value.get("value")
if value.get("type") == "object" and isinstance(raw, str) and raw.strip():
try:
return json.loads(raw)
except Exception:
return raw
return raw
return value
@@ -108,11 +109,18 @@ class UserFillUp(ComponentBase):
ans = v
if not ans:
ans = ""
content = re.sub(r"\{%s\}"%k, ans, content)
content = re.sub(r"\{%s\}" % k, ans, content)
self.set_output("tips", content)
layout_recognize = self._param.layout_recognize or None
merged_inputs = self._merge_runtime_inputs(kwargs.get("inputs", {}))
if not merged_inputs:
# No fresh user answer was supplied on this entry. Clear any values
# retained from a previous response so the canvas wait-check treats
# the form as unsatisfied and pauses for input again. Without this,
# an Await Response node inside a Loop would only pause on the first
# iteration and silently reuse the earlier answer afterwards.
self._clear_form_values()
for k, v in merged_inputs.items():
if self.check_if_canceled("UserFillUp processing"):
return
@@ -120,5 +128,18 @@ class UserFillUp(ComponentBase):
self.set_output(k, resolved)
self.set_input_value(k, resolved)
def _clear_form_values(self):
for field in self.get_input_elements().values():
if not isinstance(field, dict):
continue
field_type = str(field.get("type", "")).lower()
# An optional file input is already treated as satisfied when empty
# (see Canvas._is_input_field_satisfied), so clearing it would not
# force a re-prompt and would only drop a previously uploaded file.
# Leave it untouched to avoid unexpected data loss.
if "file" in field_type and field.get("optional"):
continue
field["value"] = None
def thoughts(self) -> str:
return "Waiting for your input..."

View File

@@ -24,6 +24,7 @@ class VariableModel(BaseModel):
model_config = ConfigDict(extra="forbid")
"""
class IterationParam(ComponentParamBase):
"""
Define the Iteration component parameters.
@@ -32,15 +33,10 @@ class IterationParam(ComponentParamBase):
def __init__(self):
super().__init__()
self.items_ref = ""
self.variable={}
self.variable = {}
def get_input_form(self) -> dict[str, dict]:
return {
"items": {
"type": "json",
"name": "Items"
}
}
return {"items": {"type": "json", "name": "Items"}}
def check(self):
return True
@@ -62,10 +58,7 @@ class Iteration(ComponentBase, ABC):
arr = self._canvas.get_variable_value(self._param.items_ref)
if not isinstance(arr, list):
self.set_output("_ERROR", self._param.items_ref + " must be an array, but its type is "+str(type(arr)))
self.set_output("_ERROR", self._param.items_ref + " must be an array, but its type is " + str(type(arr)))
def thoughts(self) -> str:
return "Need to process {} items.".format(len(self._canvas.get_variable_value(self._param.items_ref)))

View File

@@ -14,6 +14,7 @@
# limitations under the License.
#
from abc import ABC
from agent.component.base import ComponentBase, ComponentParamBase
@@ -21,6 +22,7 @@ class IterationItemParam(ComponentParamBase):
"""
Define the IterationItem component parameters.
"""
def check(self):
return True
@@ -40,7 +42,7 @@ class IterationItem(ComponentBase, ABC):
arr = self._canvas.get_variable_value(parent._param.items_ref)
if not isinstance(arr, list):
self._idx = -1
raise Exception(parent._param.items_ref + " must be an array, but its type is "+str(type(arr)))
raise Exception(parent._param.items_ref + " must be an array, but its type is " + str(type(arr)))
if self._idx > 0:
if self.check_if_canceled("IterationItem processing"):
@@ -79,7 +81,11 @@ class IterationItem(ComponentBase, ABC):
for k, o in p._param.outputs.items():
if "ref" not in o:
continue
_cid, var = o["ref"].split("@")
# Use maxsplit=1 so an `@` legitimately embedded in `var`
# (e.g. a user-defined output key that happens to contain
# '@') does not raise `ValueError: too many values to unpack`.
# `_cid` is system-generated and never contains '@'.
_cid, var = o["ref"].split("@", 1)
if _cid != cid:
continue
res = p.output(k)

View File

@@ -3,10 +3,12 @@ import os
from agent.component.base import ComponentBase, ComponentParamBase
from api.utils.api_utils import timeout
class ListOperationsParam(ComponentParamBase):
"""
Define the List Operations component parameters.
"""
def __init__(self):
super().__init__()
self.query = ""
@@ -20,24 +22,8 @@ class ListOperationsParam(ComponentParamBase):
# first field). Mirrors internal/agent/component/list_operations.go
# parseSortByFieldList + opSort's SortBy path.
self.sort_by = ""
self.filter = {
"operator": "=",
"value": ""
}
self.outputs = {
"result": {
"value": [],
"type": "Array of ?"
},
"first": {
"value": "",
"type": "?"
},
"last": {
"value": "",
"type": "?"
}
}
self.filter = {"operator": "=", "value": ""}
self.outputs = {"result": {"value": [], "type": "Array of ?"}, "first": {"value": "", "type": "?"}, "last": {"value": "", "type": "?"}}
@staticmethod
def _normalize_operation_name(operation):
@@ -45,7 +31,7 @@ class ListOperationsParam(ComponentParamBase):
if op.lower() == "topn":
return "head"
return op or "nth"
def check(self):
self.check_empty(self.query, "query")
self.operations = self._normalize_operation_name(self.operations)
@@ -57,14 +43,14 @@ class ListOperationsParam(ComponentParamBase):
def get_input_form(self) -> dict[str, dict]:
return {}
class ListOperations(ComponentBase,ABC):
class ListOperations(ComponentBase, ABC):
component_name = "ListOperations"
@timeout(int(os.environ.get("COMPONENT_EXEC_TIMEOUT", 10*60)))
@timeout(int(os.environ.get("COMPONENT_EXEC_TIMEOUT", 10 * 60)))
def _invoke(self, **kwargs):
self.input_objects=[]
self.input_objects = []
inputs = getattr(self._param, "query", None)
self.inputs = self._canvas.get_variable_value(inputs)
if not isinstance(self.inputs, list):
@@ -83,7 +69,6 @@ class ListOperations(ComponentBase,ABC):
elif self._param.operations == "drop_duplicates":
self._drop_duplicates()
def _coerce_n(self):
try:
return int(getattr(self._param, "n", 0))
@@ -99,12 +84,10 @@ class ListOperations(ComponentBase,ABC):
def _set_outputs(self, outputs):
self._param.outputs["result"]["value"] = outputs
self._param.outputs["first"]["value"] = outputs[0] if outputs else None
self._param.outputs["last"]["value"] = outputs[-1] if outputs else None
self._param.outputs["last"]["value"] = outputs[-1] if outputs else None
def _raise_strict_range_error(self, operation, n):
raise ValueError(
f"{operation} requires n to be within the valid range in strict mode, got {n}."
)
raise ValueError(f"{operation} requires n to be within the valid range in strict mode, got {n}.")
def _nth(self):
n = self._coerce_n()
@@ -160,9 +143,9 @@ class ListOperations(ComponentBase,ABC):
self._set_outputs(outputs)
def _filter(self):
self._set_outputs([i for i in self.inputs if self._eval(self._norm(i),self._param.filter["operator"],self._param.filter["value"])])
self._set_outputs([i for i in self.inputs if self._eval(self._norm(i), self._param.filter["operator"], self._param.filter["value"])])
def _norm(self,v):
def _norm(self, v):
s = "" if v is None else str(v)
return s
@@ -222,7 +205,7 @@ class ListOperations(ComponentBase,ABC):
outs.append(item)
self._set_outputs(outs)
def _hashable(self,x):
def _hashable(self, x):
if isinstance(x, dict):
return tuple(sorted((k, self._hashable(v)) for k, v in x.items()))
if isinstance(x, (list, tuple)):

View File

@@ -25,7 +25,7 @@ from functools import partial
from common.constants import LLMType
from api.db.services.dialog_service import _stream_with_think_delta
from api.db.services.llm_service import LLMBundle
from api.db.joint_services.tenant_model_service import get_model_config_from_provider_instance, get_model_type_by_name
from api.db.joint_services.tenant_model_service import resolve_model_config, resolve_model_type
from agent.component.base import ComponentBase, ComponentParamBase
from common.connection_utils import timeout
from rag.prompts.generator import tool_call_summary, message_fit_in, citation_prompt, structured_output_prompt
@@ -78,6 +78,8 @@ class LLMParam(ComponentParamBase):
conf["presence_penalty"] = float(self.presence_penalty)
if float(self.frequency_penalty) > 0 and get_attr("frequencyPenaltyEnabled"):
conf["frequency_penalty"] = float(self.frequency_penalty)
if hasattr(self, "thinking") and self.thinking and self.thinking != "default":
conf["thinking"] = self.thinking
return conf
@@ -86,9 +88,9 @@ class LLM(ComponentBase):
def __init__(self, canvas, component_id, param: ComponentParamBase):
super().__init__(canvas, component_id, param)
model_types = get_model_type_by_name(self._canvas.get_tenant_id(), self._param.llm_id)
model_types = resolve_model_type(self._canvas.get_tenant_id(), self._param.llm_id)
model_type = "chat" if "chat" in model_types else model_types[0]
chat_model_config = get_model_config_from_provider_instance(self._canvas.get_tenant_id(), model_type, self._param.llm_id)
chat_model_config = resolve_model_config(self._canvas.get_tenant_id(), model_type, self._param.llm_id)
self.chat_mdl = LLMBundle(self._canvas.get_tenant_id(), chat_model_config, max_retries=self._param.max_retries, retry_interval=self._param.delay_after_error)
self.imgs = []
@@ -316,14 +318,14 @@ class LLM(ComponentBase):
len(sys_file_imgs),
max(0, prev_img_count + len(sys_file_imgs) - len(self.imgs)),
)
model_types = get_model_type_by_name(self._canvas.get_tenant_id(), self._param.llm_id)
if self.imgs and LLMType.IMAGE2TEXT.value in model_types:
model_type = LLMType.IMAGE2TEXT.value
model_types = resolve_model_type(self._canvas.get_tenant_id(), self._param.llm_id)
if self.imgs and LLMType.VISION.value in model_types:
model_type = LLMType.VISION.value
elif LLMType.CHAT.value in model_types:
model_type = LLMType.CHAT.value
else:
model_type = model_types[0]
model_config = get_model_config_from_provider_instance(self._canvas.get_tenant_id(), model_type, self._param.llm_id)
model_config = resolve_model_config(self._canvas.get_tenant_id(), model_type, self._param.llm_id)
if self.imgs:
self.chat_mdl = LLMBundle(self._canvas.get_tenant_id(), model_config, max_retries=self._param.max_retries, retry_interval=self._param.delay_after_error)

View File

@@ -25,16 +25,11 @@ class LoopParam(ComponentParamBase):
def __init__(self):
super().__init__()
self.loop_variables = []
self.loop_termination_condition=[]
self.loop_termination_condition = []
self.maximum_loop_count = 0
def get_input_form(self) -> dict[str, dict]:
return {
"items": {
"type": "json",
"name": "Items"
}
}
return {"items": {"type": "json", "name": "Items"}}
def check(self):
return True
@@ -83,10 +78,10 @@ class Loop(ComponentBase, ABC):
for item in self._param.loop_variables:
if self._is_incomplete_loop_variable(item):
raise ValueError("Loop Variable is not complete.")
if item["input_mode"]=="variable":
self.set_output(item["variable"],self._canvas.get_variable_value(item["value"]))
elif item["input_mode"]=="constant":
self.set_output(item["variable"],item["value"])
if item["input_mode"] == "variable":
self.set_output(item["variable"], self._canvas.get_variable_value(item["value"]))
elif item["input_mode"] == "constant":
self.set_output(item["variable"], item["value"])
else:
if item["type"] == "number":
self.set_output(item["variable"], 0)
@@ -101,6 +96,5 @@ class Loop(ComponentBase, ABC):
else:
self.set_output(item["variable"], "")
def thoughts(self) -> str:
return "Loop from canvas."
return "Loop from canvas."

View File

@@ -21,9 +21,11 @@ class LoopItemParam(ComponentParamBase):
"""
Define the LoopItem component parameters.
"""
def check(self):
return True
class LoopItem(ComponentBase, ABC):
component_name = "LoopItem"
@@ -31,7 +33,6 @@ class LoopItem(ComponentBase, ABC):
super().__init__(canvas, id, param)
self._idx = 0
def _invoke(self, **kwargs):
if self.check_if_canceled("LoopItem processing"):
return
@@ -45,7 +46,7 @@ class LoopItem(ComponentBase, ABC):
return
self._idx += 1
def evaluate_condition(self,var, operator, value):
def evaluate_condition(self, var, operator, value):
if isinstance(var, str):
if operator == "contains":
return value in var
@@ -140,11 +141,7 @@ class LoopItem(ComponentBase, ABC):
else:
raise ValueError("Invalid input mode.")
conditions.append(self.evaluate_condition(var, operator, value))
should_end = (
all(conditions) if logical_operator == "and"
else any(conditions) if logical_operator == "or"
else None
)
should_end = all(conditions) if logical_operator == "and" else any(conditions) if logical_operator == "or" else None
if should_end is None:
raise ValueError("Invalid logical operator,should be 'and' or 'or'.")
@@ -164,4 +161,4 @@ class LoopItem(ComponentBase, ABC):
return False
def thoughts(self) -> str:
return "Next turn..."
return "Next turn..."

View File

@@ -14,8 +14,10 @@
# limitations under the License.
#
import asyncio
try:
import nest_asyncio
nest_asyncio.apply()
except Exception:
pass
@@ -45,20 +47,14 @@ class MessageParam(ComponentParamBase):
"""
Define the Message component parameters.
"""
def __init__(self):
super().__init__()
self.content = []
self.stream = True
self.output_format = None # default output format
self.auto_play = False
self.outputs = {
"content": {
"type": "str"
},
"downloads": {
"type": "list"
}
}
self.outputs = {"content": {"type": "str"}, "downloads": {"type": "list"}}
def check(self):
self.check_empty(self.content, "[Message] Content")
@@ -71,9 +67,7 @@ class Message(ComponentBase):
@staticmethod
def _is_download_info(value: Any) -> bool:
return isinstance(value, dict) and all(
key in value for key in ("doc_id", "filename", "mime_type")
)
return isinstance(value, dict) and all(key in value for key in ("doc_id", "filename", "mime_type"))
@staticmethod
def _download_info_includes_content(value: Any) -> bool:
@@ -157,7 +151,7 @@ class Message(ComponentBase):
delimiter: str = None,
downloads: list[dict[str, Any]] | None = None,
) -> tuple[str, dict[str, str | list | Any]]:
for k,v in self.get_input_elements_from_text(script).items():
for k, v in self.get_input_elements_from_text(script).items():
if k in kwargs:
continue
v = v["value"]
@@ -191,7 +185,7 @@ class Message(ComponentBase):
buf += t
return buf
async def _stream(self, rand_cnt:str):
async def _stream(self, rand_cnt: str):
s = 0
all_content = ""
cache = {}
@@ -200,8 +194,8 @@ class Message(ComponentBase):
if self.check_if_canceled("Message streaming"):
return
all_content += rand_cnt[s: r.start()]
yield rand_cnt[s: r.start()]
all_content += rand_cnt[s : r.start()]
yield rand_cnt[s : r.start()]
s = r.end()
exp = r.group(1)
if exp in cache:
@@ -235,9 +229,7 @@ class Message(ComponentBase):
continue
elif inspect.isawaitable(v):
v = await v
v = self._stringify_message_value(
v, downloads=downloads, fallback_to_str=True
)
v = self._stringify_message_value(v, downloads=downloads, fallback_to_str=True)
yield v
self.set_input_value(exp, v)
all_content += v
@@ -247,21 +239,19 @@ class Message(ComponentBase):
if self.check_if_canceled("Message streaming"):
return
all_content += rand_cnt[s: ]
yield rand_cnt[s: ]
all_content += rand_cnt[s:]
yield rand_cnt[s:]
self.set_output("downloads", downloads)
self.set_output("content", all_content)
self._convert_content(all_content)
await self._save_to_memory(all_content)
def _is_jinjia2(self, content:str) -> bool:
patt = [
r"\{%.*%\}", "{{", "}}"
]
def _is_jinjia2(self, content: str) -> bool:
patt = [r"\{%.*%\}", "{{", "}}"]
return any([re.search(p, content) for p in patt])
@timeout(int(os.environ.get("COMPONENT_EXEC_TIMEOUT", 10*60)))
@timeout(int(os.environ.get("COMPONENT_EXEC_TIMEOUT", 10 * 60)))
def _invoke(self, **kwargs):
if self.check_if_canceled("Message processing"):
return
@@ -303,19 +293,19 @@ class Message(ComponentBase):
def _parse_markdown_table_lines(self, table_lines: list):
"""
Parse a list of Markdown table lines into a pandas DataFrame.
Args:
table_lines: List of strings, each representing a row in the Markdown table
(excluding separator lines like |---|---|)
Returns:
pandas DataFrame with the table data, or None if parsing fails
"""
import pandas as pd
if not table_lines:
return None
rows = []
headers = None
@@ -350,36 +340,58 @@ class Message(ComponentBase):
return cell
return cell
for line in table_lines:
# Split by | and clean up
cells = [cell.strip() for cell in line.split('|')]
cells = [cell.strip() for cell in line.split("|")]
# Remove empty first and last elements from split (caused by leading/trailing |)
cells = [c for c in cells if c]
if headers is None:
headers = cells
else:
cells = [_coerce_excel_cell_type(c) for c in cells]
rows.append(cells)
if headers and rows:
# Ensure all rows have same number of columns as headers
normalized_rows = []
for row in rows:
while len(row) < len(headers):
row.append('')
normalized_rows.append(row[:len(headers)])
row.append("")
normalized_rows.append(row[: len(headers)])
return pd.DataFrame(normalized_rows, columns=headers)
return None
@staticmethod
def _strip_thinking(content: str) -> str:
"""Remove <think>...</think> reasoning blocks before document export.
Reasoning models (e.g. DeepSeek-R1, OpenAI o1) embed chain-of-thought
inside ``<think>`` tags. These blocks must not leak into exported
Word/PDF/Excel documents.
"""
if not isinstance(content, str) or not content:
return content
# Remove complete think blocks (DOTALL so newlines are matched)
cleaned = re.sub(r"<think>.*?</think>", "", content, flags=re.DOTALL)
# Remove any dangling unclosed <think> opening tag + trailing content
cleaned = re.sub(r"<think>.*$", "", cleaned, flags=re.DOTALL)
# Remove leftover standalone tags
cleaned = re.sub(r"</?think>", "", cleaned)
# Collapse 3+ consecutive newlines left behind by removed blocks
cleaned = re.sub(r"\n{3,}", "\n\n", cleaned)
return cleaned.strip()
def _convert_content(self, content):
if not self._param.output_format:
return
content = self._strip_thinking(content)
import pypandoc
doc_id = get_uuid()
if self._param.output_format.lower() not in {"markdown", "html", "pdf", "docx", "xlsx"}:
@@ -408,49 +420,47 @@ class Message(ComponentBase):
# Debug: log the content being parsed
logging.info(f"XLSX Parser: Content length={len(content) if content else 0}, first 500 chars: {content[:500] if content else 'None'}")
# Try to parse ALL Markdown tables from the content
# Each table will be written to a separate sheet
tables = [] # List of (sheet_name, dataframe)
if isinstance(content, str):
lines = content.strip().split('\n')
lines = content.strip().split("\n")
logging.info(f"XLSX Parser: Total lines={len(lines)}, lines starting with '|': {sum(1 for line in lines if line.strip().startswith('|'))}")
current_table_lines = []
current_table_title = None
pending_title = None
in_table = False
table_count = 0
for i, line in enumerate(lines):
stripped = line.strip()
# Check for potential table title (lines before a table)
# Look for patterns like "Table 1:", "## Table", or markdown headers
if not in_table and stripped and not stripped.startswith('|'):
if not in_table and stripped and not stripped.startswith("|"):
# Check if this could be a table title
lower_stripped = stripped.lower()
if (lower_stripped.startswith('table') or
stripped.startswith('#') or
':' in stripped):
pending_title = stripped.lstrip('#').strip()
if stripped.startswith('|') and '|' in stripped[1:]:
if lower_stripped.startswith("table") or stripped.startswith("#") or ":" in stripped:
pending_title = stripped.lstrip("#").strip()
if stripped.startswith("|") and "|" in stripped[1:]:
# Check if this is a separator line (|---|---|)
cleaned = stripped.replace(' ', '').replace('|', '').replace('-', '').replace(':', '')
if cleaned == '':
cleaned = stripped.replace(" ", "").replace("|", "").replace("-", "").replace(":", "")
if cleaned == "":
continue # Skip separator line
if not in_table:
# Starting a new table
in_table = True
current_table_lines = []
current_table_title = pending_title
pending_title = None
current_table_lines.append(stripped)
elif in_table and not stripped.startswith('|'):
elif in_table and not stripped.startswith("|"):
# End of current table - save it
if current_table_lines:
df = self._parse_markdown_table_lines(current_table_lines)
@@ -460,24 +470,22 @@ class Message(ComponentBase):
if current_table_title:
# Clean and truncate title for sheet name
sheet_name = current_table_title[:31]
sheet_name = sheet_name.replace('/', '_').replace('\\', '_').replace('*', '').replace('?', '').replace('[', '').replace(']', '').replace(':', '')
sheet_name = sheet_name.replace("/", "_").replace("\\", "_").replace("*", "").replace("?", "").replace("[", "").replace("]", "").replace(":", "")
else:
sheet_name = f"Table_{table_count}"
tables.append((sheet_name, df))
# Reset for next table
in_table = False
current_table_lines = []
current_table_title = None
# Check if this line could be a title for the next table
if stripped:
lower_stripped = stripped.lower()
if (lower_stripped.startswith('table') or
stripped.startswith('#') or
':' in stripped):
pending_title = stripped.lstrip('#').strip()
if lower_stripped.startswith("table") or stripped.startswith("#") or ":" in stripped:
pending_title = stripped.lstrip("#").strip()
# Don't forget the last table if content ends with a table
if in_table and current_table_lines:
df = self._parse_markdown_table_lines(current_table_lines)
@@ -485,11 +493,11 @@ class Message(ComponentBase):
table_count += 1
if current_table_title:
sheet_name = current_table_title[:31]
sheet_name = sheet_name.replace('/', '_').replace('\\', '_').replace('*', '').replace('?', '').replace('[', '').replace(']', '').replace(':', '')
sheet_name = sheet_name.replace("/", "_").replace("\\", "_").replace("*", "").replace("?", "").replace("[", "").replace("]", "").replace(":", "")
else:
sheet_name = f"Table_{table_count}"
tables.append((sheet_name, df))
# Fallback: if no tables found, create single sheet with content
if not tables:
df = pd.DataFrame({"Content": [content if content else ""]})
@@ -497,7 +505,7 @@ class Message(ComponentBase):
# Write all tables to Excel, each in a separate sheet
excel_io = BytesIO()
with pd.ExcelWriter(excel_io, engine='openpyxl') as writer:
with pd.ExcelWriter(excel_io, engine="openpyxl") as writer:
used_names = set()
for sheet_name, df in tables:
# Ensure unique sheet names
@@ -505,14 +513,14 @@ class Message(ComponentBase):
counter = 1
while sheet_name in used_names:
suffix = f"_{counter}"
sheet_name = original_name[:31-len(suffix)] + suffix
sheet_name = original_name[: 31 - len(suffix)] + suffix
counter += 1
used_names.add(sheet_name)
df.to_excel(writer, sheet_name=sheet_name, index=False)
excel_io.seek(0)
binary_content = excel_io.read()
logging.info(f"Generated Excel with {len(tables)} sheet(s): {[t[0] for t in tables]}")
else: # pdf, docx
@@ -543,10 +551,7 @@ class Message(ComponentBase):
os.remove(tmp_name)
settings.STORAGE_IMPL.put(self._canvas._tenant_id, doc_id, binary_content)
self.set_output("attachment", {
"doc_id":doc_id,
"format":self._param.output_format,
"file_name":f"{doc_id[:8]}.{self._param.output_format}"})
self.set_output("attachment", {"doc_id": doc_id, "format": self._param.output_format, "file_name": f"{doc_id[:8]}.{self._param.output_format}"})
logging.info(f"Converted content uploaded as {doc_id} (format={self._param.output_format})")
@@ -560,15 +565,10 @@ class Message(ComponentBase):
user_id = self._param.user_id if hasattr(self._param, "user_id") else ""
if user_id:
import re
# is variable
if re.match(r"^{.*}$", user_id):
user_id = self._canvas.get_variable_value(user_id)
message_dict = {
"user_id": user_id,
"agent_id": self._canvas._id,
"session_id": self._canvas.task_id,
"user_input": self._canvas.get_sys_query(),
"agent_response": content
}
message_dict = {"user_id": user_id, "agent_id": self._canvas._id, "session_id": self._canvas.task_id, "user_input": self._canvas.get_sys_query(), "agent_response": content}
return await queue_save_to_memory_task(self._param.memory_ids, message_dict)

View File

@@ -1,194 +0,0 @@
#
# Copyright 2025 The InfiniFlow Authors. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
"""
PipelineChunker Component
Run RAGFlow Pipeline-style chunkers (rag.app.*) against uploaded files inside an
Agent workflow. Emits plain text chunks for downstream Agent nodes — no
embedding, no persistence. Wraps existing chunker functions; does not
re-implement chunking logic.
"""
import importlib
import logging
import os
from abc import ABC
from agent.component.base import ComponentBase, ComponentParamBase
from api.db.services.file_service import FileService
from common.connection_utils import timeout
# Parser id -> dotted module path under rag.app. Imported lazily so we don't
# pull deepdoc/OCR/VLM machinery at component-discovery time.
_PARSER_MODULES: dict[str, str] = {
"general": "rag.app.naive",
"naive": "rag.app.naive",
"paper": "rag.app.paper",
"book": "rag.app.book",
"presentation": "rag.app.presentation",
"manual": "rag.app.manual",
"laws": "rag.app.laws",
"qa": "rag.app.qa",
"table": "rag.app.table",
"resume": "rag.app.resume",
"picture": "rag.app.picture",
"one": "rag.app.one",
"audio": "rag.app.audio",
"email": "rag.app.email",
"tag": "rag.app.tag",
}
def _load_chunker(parser_id: str):
"""Resolve a parser id to the underlying ``rag.app.<module>.chunk`` callable."""
module_path = _PARSER_MODULES[parser_id.lower()]
return importlib.import_module(module_path).chunk
class PipelineChunkerParam(ComponentParamBase):
"""
Define the PipelineChunker component parameters.
"""
def __init__(self):
"""Initialise PipelineChunker defaults and declare component outputs."""
super().__init__()
self.inputs = [] # variable references to uploaded files
self.parser_id = "naive"
self.lang = "English"
self.from_page = 0
self.to_page = 100000000
self.parser_config = {}
self.outputs = {
"chunks": {"type": "list", "value": []},
"chunks_full": {"type": "list", "value": []},
"summary": {"type": "str", "value": ""},
}
def check(self):
"""Validate parser id, page range, and parser_config shape."""
self.check_valid_value(
self.parser_id.lower(),
"[PipelineChunker] parser_id",
list(_PARSER_MODULES.keys()),
)
self.check_nonnegative_number(self.from_page, "[PipelineChunker] from_page")
self.check_nonnegative_number(self.to_page, "[PipelineChunker] to_page")
if isinstance(self.from_page, (int, float)) and isinstance(self.to_page, (int, float)) and self.from_page > self.to_page:
raise ValueError("[PipelineChunker] from_page must be <= to_page")
if not isinstance(self.parser_config, dict):
raise ValueError("[PipelineChunker] parser_config must be a dict.")
return True
class PipelineChunker(ComponentBase, ABC):
"""
Run a Pipeline-style chunker (naive, paper, qa, manual, book, ...) against
one or more uploaded files and surface the resulting chunks to downstream
Agent nodes.
"""
component_name = "PipelineChunker"
def get_input_form(self) -> dict[str, dict]:
"""Expose each referenced file input as a file-typed form element."""
res = {}
for ref in self._param.inputs or []:
for k, o in self.get_input_elements_from_text(ref).items():
res[k] = {"name": o.get("name", ""), "type": "file"}
return res
def _get_file_content(self, file_ref: str) -> tuple[bytes | None, str | None]:
"""Resolve a canvas variable reference to ``(content_bytes, filename)``."""
value = self._canvas.get_variable_value(file_ref)
if value is None:
return None, None
if isinstance(value, list) and value:
value = value[0]
if isinstance(value, dict):
file_id = value.get("id") or value.get("file_id")
created_by = value.get("created_by") or self._canvas.get_tenant_id()
filename = value.get("name") or value.get("filename") or "uploaded"
if file_id:
try:
return FileService.get_blob(created_by, file_id), filename
except Exception as e:
logging.exception(
f"[PipelineChunker] FileService.get_blob failed for "
f"file_id={file_id} created_by={created_by} filename={filename}: {e}"
)
return None, None
return None, None
@timeout(int(os.environ.get("COMPONENT_EXEC_TIMEOUT", 10 * 60)))
def _invoke(self, **kwargs):
"""Run the configured chunker over every referenced file and publish outputs."""
if self.check_if_canceled("PipelineChunker processing"):
return
chunker = _load_chunker(self._param.parser_id)
tenant_id = self._canvas.get_tenant_id()
chunk_kwargs = dict(
lang=self._param.lang,
tenant_id=tenant_id,
from_page=self._param.from_page,
to_page=self._param.to_page,
parser_config=self._param.parser_config or {},
callback=lambda prog=0, msg="": logging.info(f"[PipelineChunker] {prog}: {msg}"),
)
all_chunks: list[dict] = []
per_file_counts: list[str] = []
for file_ref in self._param.inputs or []:
if self.check_if_canceled("PipelineChunker processing"):
return
content, filename = self._get_file_content(file_ref)
self.set_input_value(file_ref, filename or "")
if content is None:
logging.warning(f"[PipelineChunker] could not resolve file ref: {file_ref}")
per_file_counts.append(f"{filename or file_ref}: error (could not resolve file)")
continue
try:
file_chunks = chunker(filename, binary=content, **chunk_kwargs) or []
except Exception as e:
logging.exception(e)
per_file_counts.append(f"{filename}: error (chunking failed)")
continue
all_chunks.extend(file_chunks)
per_file_counts.append(f"{filename}: {len(file_chunks)} chunks")
text_only = [(c.get("content_with_weight") or c.get("text") or "") for c in all_chunks if isinstance(c, dict)]
text_only = [t for t in text_only if t]
self.set_output("chunks", text_only)
self.set_output("chunks_full", all_chunks)
self.set_output(
"summary",
f"Parser: {self._param.parser_id} | Files: {len(self._param.inputs or [])} | Chunks: {len(text_only)}" + (" | " + "; ".join(per_file_counts) if per_file_counts else ""),
)
def thoughts(self) -> str:
"""Return a short status line for UI display."""
return f"Chunking with `{self._param.parser_id}` strategy..."

View File

@@ -52,18 +52,10 @@ class StringTransform(Message, ABC):
def get_input_form(self) -> dict[str, dict]:
if self._param.method == "split":
return {
"line": {
"name": "String",
"type": "line"
}
}
return {k: {
"name": o["name"],
"type": "line"
} for k, o in self.get_input_elements_from_text(self._param.script).items()}
return {"line": {"name": "String", "type": "line"}}
return {k: {"name": o["name"], "type": "line"} for k, o in self.get_input_elements_from_text(self._param.script).items()}
@timeout(int(os.environ.get("COMPONENT_EXEC_TIMEOUT", 10*60)))
@timeout(int(os.environ.get("COMPONENT_EXEC_TIMEOUT", 10 * 60)))
def _invoke(self, **kwargs):
if self.check_if_canceled("StringTransform processing"):
return
@@ -73,7 +65,7 @@ class StringTransform(Message, ABC):
else:
self._merge(kwargs)
def _split(self, line:str|None = None):
def _split(self, line: str | None = None):
if self.check_if_canceled("StringTransform split processing"):
return
@@ -84,13 +76,13 @@ class StringTransform(Message, ABC):
self.set_input_value(self._param.split_ref, var)
res = []
for i,s in enumerate(re.split(r"(%s)"%("|".join([re.escape(d) for d in self._param.delimiters])), var, flags=re.DOTALL)):
for i, s in enumerate(re.split(r"(%s)" % ("|".join([re.escape(d) for d in self._param.delimiters])), var, flags=re.DOTALL)):
if i % 2 == 1:
continue
res.append(s)
self.set_output("result", res)
def _merge(self, kwargs:dict[str, str] = {}):
def _merge(self, kwargs: dict[str, str] = {}):
if self.check_if_canceled("StringTransform merge processing"):
return
@@ -104,7 +96,7 @@ class StringTransform(Message, ABC):
except Exception:
pass
for k,v in kwargs.items():
for k, v in kwargs.items():
if v is None:
v = ""
script = re.sub(k, lambda match: v, script)
@@ -113,5 +105,3 @@ class StringTransform(Message, ABC):
def thoughts(self) -> str:
return f"It's {self._param.method}ing."

View File

@@ -40,8 +40,7 @@ class SwitchParam(ComponentParamBase):
"""
self.conditions = []
self.end_cpn_ids = []
self.operators = ['contains', 'not contains', 'start with', 'end with', 'empty', 'not empty', '=', '', '>',
'<', '', '']
self.operators = ["contains", "not contains", "start with", "end with", "empty", "not empty", "=", "", ">", "<", "", ""]
def check(self):
self.check_empty(self.conditions, "[Switch] conditions")
@@ -51,12 +50,8 @@ class SwitchParam(ComponentParamBase):
self.check_empty(self.end_cpn_ids, "[Switch] the ELSE/Other destination can not be empty.")
def get_input_form(self) -> dict[str, dict]:
return {
"urls": {
"name": "URLs",
"type": "line"
}
}
return {"urls": {"name": "URLs", "type": "line"}}
class Switch(ComponentBase, ABC):
component_name = "Switch"
@@ -137,7 +132,7 @@ class Switch(ComponentBase, ABC):
except Exception:
return True if input <= value else False
raise ValueError(f'Not supported operator: {operator}')
raise ValueError(f"Not supported operator: {operator}")
def thoughts(self) -> str:
return "Im weighing a few options and will pick the next step shortly."

View File

@@ -38,13 +38,9 @@ class VariableAggregatorParam(ComponentParamBase):
if not g.get("group_name"):
raise ValueError("[VariableAggregator] group_name can not be empty!")
if not g.get("variables"):
raise ValueError(
f"[VariableAggregator] variables of group `{g.get('group_name')}` can not be empty"
)
raise ValueError(f"[VariableAggregator] variables of group `{g.get('group_name')}` can not be empty")
if not isinstance(g.get("variables"), list):
raise ValueError(
f"[VariableAggregator] variables of group `{g.get('group_name')}` should be a list of strings"
)
raise ValueError(f"[VariableAggregator] variables of group `{g.get('group_name')}` should be a list of strings")
def get_input_form(self) -> dict[str, dict]:
return {
@@ -67,11 +63,11 @@ class VariableAggregator(ComponentBase):
# record candidate selectors within this group
self.set_input_value(f"{gname}.variables", list(group.get("variables", [])))
for selector in group.get("variables", []):
val = self._canvas.get_variable_value(selector['value'])
val = self._canvas.get_variable_value(selector["value"])
if val:
self.set_output(gname, val)
break
@staticmethod
def _to_object(value: Any) -> Any:
# Try to convert value to serializable object if it has to_object()

View File

@@ -19,32 +19,30 @@ import numbers
from agent.component.base import ComponentBase, ComponentParamBase
from api.utils.api_utils import timeout
class VariableAssignerParam(ComponentParamBase):
"""
Define the Variable Assigner component parameters.
"""
def __init__(self):
super().__init__()
self.variables=[]
self.variables = []
def check(self):
return True
def get_input_form(self) -> dict[str, dict]:
return {
"items": {
"type": "json",
"name": "Items"
}
}
class VariableAssigner(ComponentBase,ABC):
def get_input_form(self) -> dict[str, dict]:
return {"items": {"type": "json", "name": "Items"}}
class VariableAssigner(ComponentBase, ABC):
component_name = "VariableAssigner"
_NO_PARAMETER_OPERATORS = {"clear", "remove_first", "remove_last"}
@timeout(int(os.environ.get("COMPONENT_EXEC_TIMEOUT", 10*60)))
@timeout(int(os.environ.get("COMPONENT_EXEC_TIMEOUT", 10 * 60)))
def _invoke(self, **kwargs):
if not isinstance(self._param.variables,list):
if not isinstance(self._param.variables, list):
return
else:
for item in self._param.variables:
@@ -56,105 +54,105 @@ class VariableAssigner(ComponentBase,ABC):
raise ValueError("Variable is not complete.")
if operator not in self._NO_PARAMETER_OPERATORS and parameter is None:
raise ValueError("Variable is not complete.")
variable_value=self._canvas.get_variable_value(variable)
new_variable=self._operate(variable_value,operator,parameter)
variable_value = self._canvas.get_variable_value(variable)
new_variable = self._operate(variable_value, operator, parameter)
self._canvas.set_variable_value(variable, new_variable)
def _operate(self,variable,operator,parameter):
def _operate(self, variable, operator, parameter):
if operator == "overwrite":
return self._overwrite(parameter)
elif operator == "clear":
return self._clear(variable)
elif operator == "set":
return self._set(variable,parameter)
return self._set(variable, parameter)
elif operator == "append":
return self._append(variable,parameter)
return self._append(variable, parameter)
elif operator == "extend":
return self._extend(variable,parameter)
return self._extend(variable, parameter)
elif operator == "remove_first":
return self._remove_first(variable)
elif operator == "remove_last":
return self._remove_last(variable)
elif operator == "+=":
return self._add(variable,parameter)
return self._add(variable, parameter)
elif operator == "-=":
return self._subtract(variable,parameter)
return self._subtract(variable, parameter)
elif operator == "*=":
return self._multiply(variable,parameter)
return self._multiply(variable, parameter)
elif operator == "/=":
return self._divide(variable,parameter)
return self._divide(variable, parameter)
else:
return
def _overwrite(self,parameter):
def _overwrite(self, parameter):
return self._canvas.get_variable_value(parameter)
def _clear(self,variable):
if isinstance(variable,list):
def _clear(self, variable):
if isinstance(variable, list):
return []
elif isinstance(variable,str):
elif isinstance(variable, str):
return ""
elif isinstance(variable,dict):
elif isinstance(variable, dict):
return {}
elif isinstance(variable,bool):
elif isinstance(variable, bool):
return False
elif isinstance(variable,int):
elif isinstance(variable, int):
return 0
elif isinstance(variable,float):
elif isinstance(variable, float):
return 0.0
else:
return None
def _set(self,variable,parameter):
def _set(self, variable, parameter):
if variable is None:
return self._canvas.get_value_with_variable(parameter)
elif isinstance(variable,str):
elif isinstance(variable, str):
return self._canvas.get_value_with_variable(parameter)
elif isinstance(variable,bool):
elif isinstance(variable, bool):
return parameter
elif isinstance(variable,int):
elif isinstance(variable, int):
return parameter
elif isinstance(variable,float):
elif isinstance(variable, float):
return parameter
else:
return parameter
def _append(self,variable,parameter):
parameter=self._canvas.get_variable_value(parameter)
def _append(self, variable, parameter):
parameter = self._canvas.get_variable_value(parameter)
if variable is None:
variable=[]
if not isinstance(variable,list):
variable = []
if not isinstance(variable, list):
return "ERROR:VARIABLE_NOT_LIST"
elif len(variable)!=0 and not isinstance(parameter,type(variable[0])):
elif len(variable) != 0 and not isinstance(parameter, type(variable[0])):
return "ERROR:PARAMETER_NOT_LIST_ELEMENT_TYPE"
else:
variable.append(parameter)
return variable
def _extend(self,variable,parameter):
parameter=self._canvas.get_variable_value(parameter)
def _extend(self, variable, parameter):
parameter = self._canvas.get_variable_value(parameter)
if variable is None:
variable=[]
if not isinstance(variable,list):
variable = []
if not isinstance(variable, list):
return "ERROR:VARIABLE_NOT_LIST"
elif not isinstance(parameter,list):
elif not isinstance(parameter, list):
return "ERROR:PARAMETER_NOT_LIST"
elif len(variable)!=0 and len(parameter)!=0 and not isinstance(parameter[0],type(variable[0])):
elif len(variable) != 0 and len(parameter) != 0 and not isinstance(parameter[0], type(variable[0])):
return "ERROR:PARAMETER_NOT_LIST_ELEMENT_TYPE"
else:
return variable + parameter
def _remove_first(self,variable):
if not isinstance(variable,list):
def _remove_first(self, variable):
if not isinstance(variable, list):
return "ERROR:VARIABLE_NOT_LIST"
if len(variable)==0:
if len(variable) == 0:
return variable
return variable[1:]
def _remove_last(self,variable):
if not isinstance(variable,list):
def _remove_last(self, variable):
if not isinstance(variable, list):
return "ERROR:VARIABLE_NOT_LIST"
if len(variable)==0:
if len(variable) == 0:
return variable
return variable[:-1]
@@ -163,32 +161,32 @@ class VariableAssigner(ComponentBase,ABC):
return False
return isinstance(value, numbers.Number)
def _add(self,variable,parameter):
def _add(self, variable, parameter):
if self.is_number(variable) and self.is_number(parameter):
return variable + parameter
else:
return "ERROR:VARIABLE_NOT_NUMBER or PARAMETER_NOT_NUMBER"
def _subtract(self,variable,parameter):
def _subtract(self, variable, parameter):
if self.is_number(variable) and self.is_number(parameter):
return variable - parameter
else:
return "ERROR:VARIABLE_NOT_NUMBER or PARAMETER_NOT_NUMBER"
def _multiply(self,variable,parameter):
def _multiply(self, variable, parameter):
if self.is_number(variable) and self.is_number(parameter):
return variable * parameter
else:
return "ERROR:VARIABLE_NOT_NUMBER or PARAMETER_NOT_NUMBER"
def _divide(self,variable,parameter):
def _divide(self, variable, parameter):
if self.is_number(variable) and self.is_number(parameter):
if parameter==0:
if parameter == 0:
return "ERROR:DIVIDE_BY_ZERO"
else:
return variable/parameter
return variable / parameter
else:
return "ERROR:VARIABLE_NOT_NUMBER or PARAMETER_NOT_NUMBER"
return "ERROR:VARIABLE_NOT_NUMBER or PARAMETER_NOT_NUMBER"
def thoughts(self) -> str:
return "Assign variables from canvas."
return "Assign variables from canvas."

View File

@@ -56,7 +56,7 @@ def normalize_chunker_dsl(dsl: dict) -> dict:
for old_name, new_name in COMPONENT_RENAMES.items():
prefix = f"{old_name}:"
if component_id.startswith(prefix):
new_component_id = f"{new_name}:{component_id[len(prefix):]}"
new_component_id = f"{new_name}:{component_id[len(prefix) :]}"
break
component_id_map[component_id] = new_component_id
@@ -66,12 +66,7 @@ def normalize_chunker_dsl(dsl: dict) -> dict:
def repl(match: re.Match[str]) -> str:
component_id = match.group(2)
return (
match.group(1)
+ component_id_map.get(component_id, component_id)
+ match.group(3)
+ match.group(4)
)
return match.group(1) + component_id_map.get(component_id, component_id) + match.group(3) + match.group(4)
return VARIABLE_REF_PATTERN.sub(repl, text)
@@ -96,15 +91,9 @@ def normalize_chunker_dsl(dsl: dict) -> dict:
obj["component_name"] = COMPONENT_RENAMES.get(component_name, component_name)
if isinstance(new_component.get("downstream"), list):
new_component["downstream"] = [
component_id_map.get(component_id, component_id)
for component_id in new_component["downstream"]
]
new_component["downstream"] = [component_id_map.get(component_id, component_id) for component_id in new_component["downstream"]]
if isinstance(new_component.get("upstream"), list):
new_component["upstream"] = [
component_id_map.get(component_id, component_id)
for component_id in new_component["upstream"]
]
new_component["upstream"] = [component_id_map.get(component_id, component_id) for component_id in new_component["upstream"]]
parent_id = new_component.get("parent_id")
if isinstance(parent_id, str):
@@ -115,10 +104,7 @@ def normalize_chunker_dsl(dsl: dict) -> dict:
normalized["components"] = rewritten_components
if isinstance(normalized.get("path"), list):
normalized["path"] = [
component_id_map.get(component_id, component_id)
for component_id in normalized["path"]
]
normalized["path"] = [component_id_map.get(component_id, component_id) for component_id in normalized["path"]]
graph = normalized.get("graph")
if isinstance(graph, dict):

View File

@@ -1 +1 @@
PLUGIN_TYPE_LLM_TOOLS = "llm_tools"
PLUGIN_TYPE_LLM_TOOLS = "llm_tools"

View File

@@ -7,6 +7,7 @@ class BadCalculatorPlugin(LLMToolPlugin):
A sample LLM tool plugin, will add two numbers with 100.
It only presents for demo purpose. Do not use it in production.
"""
_version_ = "1.0.0"
@classmethod
@@ -17,19 +18,9 @@ class BadCalculatorPlugin(LLMToolPlugin):
"description": "A tool to calculate the sum of two numbers (will give wrong answer)",
"displayDescription": "$t:bad_calculator.description",
"parameters": {
"a": {
"type": "number",
"description": "The first number",
"displayDescription": "$t:bad_calculator.params.a",
"required": True
},
"b": {
"type": "number",
"description": "The second number",
"displayDescription": "$t:bad_calculator.params.b",
"required": True
}
}
"a": {"type": "number", "description": "The first number", "displayDescription": "$t:bad_calculator.params.a", "required": True},
"b": {"type": "number", "description": "The second number", "displayDescription": "$t:bad_calculator.params.b", "required": True},
},
}
def invoke(self, a: int, b: int) -> str:

View File

@@ -38,14 +38,8 @@ def llm_tool_metadata_to_openai_tool(llm_tool_metadata: LLMToolMetadata) -> dict
"description": llm_tool_metadata["description"],
"parameters": {
"type": "object",
"properties": {
k: {
"type": p["type"],
"description": p["description"]
}
for k, p in llm_tool_metadata["parameters"].items()
},
"required": [k for k, p in llm_tool_metadata["parameters"].items() if p["required"]]
}
}
"properties": {k: {"type": p["type"], "description": p["description"]} for k, p in llm_tool_metadata["parameters"].items()},
"required": [k for k, p in llm_tool_metadata["parameters"].items() if p["required"]],
},
},
}

View File

@@ -15,10 +15,8 @@ class PluginManager:
self._llm_tool_plugins = {}
def load_plugins(self) -> None:
loader = pluginlib.PluginLoader(
paths=[str(Path(os.path.dirname(__file__), "embedded_plugins"))]
)
loader = pluginlib.PluginLoader(paths=[str(Path(os.path.dirname(__file__), "embedded_plugins"))])
for type, plugins in loader.plugins.items():
for name, plugin in plugins.items():
logging.info(f"Loaded {type} plugin {name} version {plugin.version}")

View File

@@ -111,7 +111,10 @@ def _load_provider_from_settings() -> None:
except Exception as e:
logger.error(f"Failed to load sandbox provider from settings: {e}")
import traceback
traceback.print_exc()
def _load_provider_config_from_settings(provider_type: str) -> Dict[str, Any]:
provider_config_settings = SystemSettingsService.get_by_name(f"sandbox.{provider_type}")
if not provider_config_settings:
@@ -147,12 +150,7 @@ def reload_provider() -> None:
_load_provider_from_settings()
def execute_code(
code: str,
language: str = "python",
timeout: int = 30,
arguments: Optional[Dict[str, Any]] = None
) -> ExecutionResult:
def execute_code(code: str, language: str = "python", timeout: int = 30, arguments: Optional[Dict[str, Any]] = None) -> ExecutionResult:
"""
Execute code in the configured sandbox.
@@ -173,9 +171,7 @@ def execute_code(
provider_manager = get_provider_manager()
if not provider_manager.is_configured():
raise RuntimeError(
"No sandbox provider configured. Please configure sandbox settings in the admin panel."
)
raise RuntimeError("No sandbox provider configured. Please configure sandbox settings in the admin panel.")
provider = provider_manager.get_provider()
provider_name = provider_manager.get_provider_name() or getattr(provider, "__class__", type(provider)).__name__
@@ -192,13 +188,7 @@ def execute_code(
try:
# Execute the code
result = provider.execute_code(
instance_id=instance.instance_id,
code=code,
language=language,
timeout=timeout,
arguments=arguments
)
result = provider.execute_code(instance_id=instance.instance_id, code=code, language=language, timeout=timeout, arguments=arguments)
return result

View File

@@ -22,4 +22,3 @@ router = APIRouter()
router.get("/")(healthz_handler)
router.get("/healthz")(healthz_handler)
router.post("/run")(run_code_handler)

View File

@@ -230,9 +230,7 @@ async def execute_code(req: CodeExecutionRequest):
if returncode != 0:
raise RuntimeError(f"Directory creation failed: {stderr}")
tar_proc = await asyncio.create_subprocess_exec(
"tar", "czf", "-", "-C", workdir, code_name, runner_name, str(bundle["args_name"]), stdout=asyncio.subprocess.PIPE
)
tar_proc = await asyncio.create_subprocess_exec("tar", "czf", "-", "-C", workdir, code_name, runner_name, str(bundle["args_name"]), stdout=asyncio.subprocess.PIPE)
tar_stdout, _ = await tar_proc.communicate()
docker_proc = await asyncio.create_subprocess_exec(
@@ -334,8 +332,16 @@ async def _collect_artifacts(container: str, task_id: str, host_workdir: str) ->
# List files in the artifacts directory inside the container
returncode, stdout, _ = await async_run_command(
"docker", "exec", container, "find", artifacts_path,
"-maxdepth", "1", "-type", "f", timeout=5,
"docker",
"exec",
container,
"find",
artifacts_path,
"-maxdepth",
"1",
"-type",
"f",
timeout=5,
)
if returncode != 0 or not stdout.strip():
return []
@@ -359,7 +365,14 @@ async def _collect_artifacts(container: str, task_id: str, host_workdir: str) ->
# Check file size inside the container
returncode, size_str, _ = await async_run_command(
"docker", "exec", container, "stat", "-c", "%s", file_path, timeout=5,
"docker",
"exec",
container,
"stat",
"-c",
"%s",
file_path,
timeout=5,
)
if returncode != 0:
logger.warning(f"Failed to stat artifact {fname}")
@@ -374,7 +387,12 @@ async def _collect_artifacts(container: str, task_id: str, host_workdir: str) ->
# Read file content via docker exec (docker cp doesn't work with gVisor tmpfs)
returncode, content_b64, stderr = await async_run_command(
"docker", "exec", container, "base64", file_path, timeout=30,
"docker",
"exec",
container,
"base64",
file_path,
timeout=30,
)
if returncode != 0:
logger.warning(f"Failed to read artifact {fname}: {stderr}")
@@ -382,12 +400,14 @@ async def _collect_artifacts(container: str, task_id: str, host_workdir: str) ->
content_b64 = content_b64.replace("\n", "").strip()
items.append(ArtifactItem(
name=fname,
mime_type=mime_type,
size=file_size,
content_b64=content_b64,
))
items.append(
ArtifactItem(
name=fname,
mime_type=mime_type,
size=file_size,
content_b64=content_b64,
)
)
logger.info(f"Collected artifact: {fname} ({file_size} bytes, {mime_type})")
return items

View File

@@ -232,11 +232,7 @@ class AliyunCodeInterpreterProvider(SandboxProvider):
# Wrap code to call main() function
# Matches self_managed provider behavior: call main(**arguments)
args_json = json.dumps(arguments or {})
wrapped_code = (
build_python_wrapper(code, args_json)
if normalized_lang == "python"
else build_javascript_wrapper(code, args_json)
)
wrapped_code = build_python_wrapper(code, args_json) if normalized_lang == "python" else build_javascript_wrapper(code, args_json)
logger.debug(f"Aliyun Code Interpreter: Wrapped code (first 200 chars): {wrapped_code[:200]}")
start_time = time.time()

View File

@@ -33,6 +33,7 @@ class SandboxProviderConfigError(Exception):
@dataclass
class SandboxInstance:
"""Represents a sandbox execution instance"""
instance_id: str
provider: str
status: str # running, stopped, error
@@ -46,6 +47,7 @@ class SandboxInstance:
@dataclass
class ExecutionResult:
"""Result of code execution in a sandbox"""
stdout: str
stderr: str
exit_code: int
@@ -96,14 +98,7 @@ class SandboxProvider(ABC):
pass
@abstractmethod
def execute_code(
self,
instance_id: str,
code: str,
language: str,
timeout: int = 10,
arguments: Optional[Dict[str, Any]] = None
) -> ExecutionResult:
def execute_code(self, instance_id: str, code: str, language: str, timeout: int = 10, arguments: Optional[Dict[str, Any]] = None) -> ExecutionResult:
"""
Execute code in a sandbox instance.

View File

@@ -97,16 +97,10 @@ class E2BProvider(SandboxProvider):
metadata={
"language": language,
"region": self.region,
}
},
)
def execute_code(
self,
instance_id: str,
code: str,
language: str,
timeout: int = 10
) -> ExecutionResult:
def execute_code(self, instance_id: str, code: str, language: str, timeout: int = 10) -> ExecutionResult:
"""
Execute code in the E2B instance.
@@ -130,9 +124,7 @@ class E2BProvider(SandboxProvider):
# POST /sandbox/{sandboxID}/execute
raise RuntimeError(
"E2B provider is not yet fully implemented. "
"Please use the self-managed provider or implement the E2B API integration. "
"See https://github.com/e2b-dev/e2b for API documentation."
"E2B provider is not yet fully implemented. Please use the self-managed provider or implement the E2B API integration. See https://github.com/e2b-dev/e2b for API documentation."
)
def destroy_instance(self, instance_id: str) -> bool:
@@ -208,7 +200,7 @@ class E2BProvider(SandboxProvider):
"min": 5,
"max": 300,
"description": "API request timeout for code execution",
}
},
}
def _normalize_language(self, language: str) -> str:

View File

@@ -49,6 +49,8 @@ LOCAL_PYTHON_THREAD_ENV_VARS = (
"BLIS_NUM_THREADS",
"VECLIB_MAXIMUM_THREADS",
)
class LocalProvider(SandboxProvider):
"""
Execute code as a local child process.

View File

@@ -108,17 +108,10 @@ class SelfManagedProvider(SandboxProvider):
"language": language,
"endpoint": self.endpoint,
"pool_size": self.pool_size,
}
},
)
def execute_code(
self,
instance_id: str,
code: str,
language: str,
timeout: int = 10,
arguments: Optional[Dict[str, Any]] = None
) -> ExecutionResult:
def execute_code(self, instance_id: str, code: str, language: str, timeout: int = 10, arguments: Optional[Dict[str, Any]] = None) -> ExecutionResult:
"""
Execute code in the sandbox.
@@ -144,11 +137,7 @@ class SelfManagedProvider(SandboxProvider):
# Prepare request
code_b64 = base64.b64encode(code.encode("utf-8")).decode("utf-8")
payload = {
"code_b64": code_b64,
"language": normalized_lang,
"arguments": arguments or {}
}
payload = {"code_b64": code_b64, "language": normalized_lang, "arguments": arguments or {}}
url = f"{self.endpoint}/run"
exec_timeout = timeout or self.timeout
@@ -156,19 +145,12 @@ class SelfManagedProvider(SandboxProvider):
start_time = time.time()
try:
response = requests.post(
url,
json=payload,
timeout=exec_timeout,
headers={"Content-Type": "application/json"}
)
response = requests.post(url, json=payload, timeout=exec_timeout, headers={"Content-Type": "application/json"})
execution_time = time.time() - start_time
if response.status_code != 200:
raise RuntimeError(
f"HTTP {response.status_code}: {response.text}"
)
raise RuntimeError(f"HTTP {response.status_code}: {response.text}")
result = response.json()
structured_result = result.get("result") or {}
@@ -188,14 +170,12 @@ class SelfManagedProvider(SandboxProvider):
"result_present": structured_result.get("present", False),
"result_value": structured_result.get("value"),
"result_type": structured_result.get("type"),
}
},
)
except requests.Timeout:
execution_time = time.time() - start_time
raise TimeoutError(
f"Execution timed out after {exec_timeout} seconds"
)
raise TimeoutError(f"Execution timed out after {exec_timeout} seconds")
except requests.RequestException as e:
raise RuntimeError(f"HTTP request failed: {str(e)}")
@@ -388,7 +368,8 @@ class SelfManagedProvider(SandboxProvider):
if endpoint:
# Check if it's a valid HTTP/HTTPS URL or localhost
import re
url_pattern = r'^(https?://|http://localhost|http://[\d\.]+:[a-z]+:[/]|http://[\w\.]+:)'
url_pattern = r"^(https?://|http://localhost|http://[\d\.]+:[a-z]+:[/]|http://[\w\.]+:)"
if not re.match(url_pattern, endpoint):
return False, f"Invalid endpoint format: {endpoint}. Must start with http:// or https://"

View File

@@ -135,9 +135,7 @@ class SSHProvider(SandboxProvider):
timeout=min(self.timeout, 10),
)
if exit_code != 0:
raise RuntimeError(
f"Failed to create remote artifacts directory: {stderr or stdout or 'unknown error'}"
)
raise RuntimeError(f"Failed to create remote artifacts directory: {stderr or stdout or 'unknown error'}")
except Exception:
sftp.close()
client.close()
@@ -211,9 +209,7 @@ class SSHProvider(SandboxProvider):
"status": "ok" if exit_code == 0 else "error",
"timeout": exec_timeout,
"command": command,
"artifacts": self._collect_artifacts(
sftp, posixpath.join(remote_work_dir, "artifacts")
),
"artifacts": self._collect_artifacts(sftp, posixpath.join(remote_work_dir, "artifacts")),
"result_present": structured_result.get("present", False),
"result_value": structured_result.get("value"),
"result_type": structured_result.get("type"),
@@ -269,18 +265,13 @@ class SSHProvider(SandboxProvider):
timeout=min(self.timeout, 10),
)
if exit_code != 0:
raise SandboxProviderConfigError(
f"SSH connectivity check failed on {self.username}@{self.host}:{self.port}: "
f"{stderr or 'remote command returned non-zero exit status'}"
)
raise SandboxProviderConfigError(f"SSH connectivity check failed on {self.username}@{self.host}:{self.port}: {stderr or 'remote command returned non-zero exit status'}")
finally:
client.close()
except SandboxProviderConfigError:
raise
except Exception as exc:
raise SandboxProviderConfigError(
f"Failed to connect to SSH host {self.username}@{self.host}:{self.port}: {exc}"
) from exc
raise SandboxProviderConfigError(f"Failed to connect to SSH host {self.username}@{self.host}:{self.port}: {exc}") from exc
def get_supported_languages(self) -> List[str]:
return ["python", "javascript", "nodejs"]
@@ -470,9 +461,7 @@ class SSHProvider(SandboxProvider):
# Match the Go provider's fail-closed posture (see
# internal/agent/sandbox/ssh.go::hostKeyCallback).
logging.warning("SSH: failed to load configured known_hosts file; refusing connection")
raise SandboxProviderConfigError(
"Failed to load configured SSH known_hosts file."
) from exc
raise SandboxProviderConfigError("Failed to load configured SSH known_hosts file.") from exc
# Reject unknown hosts: this is the default fail-closed posture
# to prevent silent MITM. Operators must either ship a populated
# known_hosts file or accept the warning (paramiko will fail the
@@ -522,9 +511,7 @@ class SSHProvider(SandboxProvider):
except Exception as exc:
errors.append(str(exc))
raise SandboxProviderConfigError(
"Failed to load SSH private key. " + "; ".join(error for error in errors if error)
)
raise SandboxProviderConfigError("Failed to load SSH private key. " + "; ".join(error for error in errors if error))
def _create_remote_workspace(self, client: paramiko.SSHClient) -> str:
base_dir = self.work_dir.rstrip("/") or "/tmp"
@@ -535,9 +522,7 @@ class SSHProvider(SandboxProvider):
timeout=min(self.timeout, 10),
)
if exit_code != 0:
raise RuntimeError(
f"Failed to create remote workspace on {self.host}: {stderr or stdout or 'unknown error'}"
)
raise RuntimeError(f"Failed to create remote workspace on {self.host}: {stderr or stdout or 'unknown error'}")
remote_work_dir = stdout.strip().splitlines()[-1] if stdout.strip() else ""
if not remote_work_dir:
@@ -577,10 +562,7 @@ class SSHProvider(SandboxProvider):
else:
raise RuntimeError(f"Unsupported language for SSH provider: {language}")
return (
f"cd {shlex.quote(remote_work_dir)} && "
f"{shlex.quote(executable)} {shlex.quote(remote_script_path)}"
)
return f"cd {shlex.quote(remote_work_dir)} && {shlex.quote(executable)} {shlex.quote(remote_script_path)}"
def _run_remote_command(
self,
@@ -700,7 +682,5 @@ def _get_paramiko_module():
try:
import paramiko
except ImportError as exc:
raise SandboxProviderConfigError(
"paramiko is required for the SSH sandbox provider. Install the project dependencies to enable it."
) from exc
raise SandboxProviderConfigError("paramiko is required for the SSH sandbox provider. Install the project dependencies to enable it.") from exc
return paramiko

View File

@@ -36,7 +36,7 @@ if __name__ == "__main__":
def build_javascript_wrapper(code: str, args_json: str) -> str:
return f'''{code}
return f"""{code}
const __ragflowArgs = {args_json};
@@ -55,7 +55,7 @@ const __ragflowArgs = {args_json};
}}
console.log('{RESULT_MARKER_PREFIX}' + Buffer.from(payload, 'utf8').toString('base64'));
}})();
'''
"""
def extract_structured_result(stdout: str) -> tuple[str, dict[str, Any]]:

View File

@@ -151,7 +151,7 @@ class TestAliyunCodeInterpreterIntegration:
""",
language="python",
timeout=30,
arguments={"name": "World", "count": 2}
arguments={"name": "World", "count": 2},
)
assert result.exit_code == 0
@@ -211,7 +211,7 @@ class TestAliyunCodeInterpreterIntegration:
}""",
language="javascript",
timeout=30,
arguments={"name": "World", "count": 2}
arguments={"name": "World", "count": 2},
)
assert result.exit_code == 0

View File

@@ -32,12 +32,7 @@ class TestSandboxDataclasses:
def test_sandbox_instance_creation(self):
"""Test SandboxInstance dataclass creation."""
instance = SandboxInstance(
instance_id="test-123",
provider="self_managed",
status="running",
metadata={"language": "python"}
)
instance = SandboxInstance(instance_id="test-123", provider="self_managed", status="running", metadata={"language": "python"})
assert instance.instance_id == "test-123"
assert instance.provider == "self_managed"
@@ -46,24 +41,13 @@ class TestSandboxDataclasses:
def test_sandbox_instance_default_metadata(self):
"""Test SandboxInstance with None metadata."""
instance = SandboxInstance(
instance_id="test-123",
provider="self_managed",
status="running",
metadata=None
)
instance = SandboxInstance(instance_id="test-123", provider="self_managed", status="running", metadata=None)
assert instance.metadata == {}
def test_execution_result_creation(self):
"""Test ExecutionResult dataclass creation."""
result = ExecutionResult(
stdout="Hello, World!",
stderr="",
exit_code=0,
execution_time=1.5,
metadata={"status": "success"}
)
result = ExecutionResult(stdout="Hello, World!", stderr="", exit_code=0, execution_time=1.5, metadata={"status": "success"})
assert result.stdout == "Hello, World!"
assert result.stderr == ""
@@ -73,13 +57,7 @@ class TestSandboxDataclasses:
def test_execution_result_default_metadata(self):
"""Test ExecutionResult with None metadata."""
result = ExecutionResult(
stdout="output",
stderr="error",
exit_code=1,
execution_time=0.5,
metadata=None
)
result = ExecutionResult(stdout="output", stderr="error", exit_code=1, execution_time=0.5, metadata=None)
assert result.metadata == {}
@@ -145,7 +123,7 @@ class TestSelfManagedProvider:
assert provider.pool_size == 10
assert not provider._initialized
@patch('requests.get')
@patch("requests.get")
def test_initialize_success(self, mock_get):
"""Test successful initialization."""
mock_response = Mock()
@@ -153,12 +131,7 @@ class TestSelfManagedProvider:
mock_get.return_value = mock_response
provider = SelfManagedProvider()
result = provider.initialize({
"endpoint": "http://test-endpoint:9385",
"timeout": 60,
"max_retries": 5,
"pool_size": 20
})
result = provider.initialize({"endpoint": "http://test-endpoint:9385", "timeout": 60, "max_retries": 5, "pool_size": 20})
assert result is True
assert provider.endpoint == "http://test-endpoint:9385"
@@ -168,7 +141,7 @@ class TestSelfManagedProvider:
assert provider._initialized
mock_get.assert_called_once_with("http://test-endpoint:9385/healthz", timeout=5)
@patch('requests.get')
@patch("requests.get")
def test_initialize_failure(self, mock_get):
"""Test initialization failure."""
mock_get.side_effect = Exception("Connection error")
@@ -181,7 +154,7 @@ class TestSelfManagedProvider:
def test_initialize_default_config(self):
"""Test initialization with default config."""
with patch('requests.get') as mock_get:
with patch("requests.get") as mock_get:
mock_response = Mock()
mock_response.status_code = 200
mock_get.return_value = mock_response
@@ -222,30 +195,18 @@ class TestSelfManagedProvider:
with pytest.raises(RuntimeError, match="Provider not initialized"):
provider.create_instance("python")
@patch('requests.post')
@patch("requests.post")
def test_execute_code_success(self, mock_post):
"""Test successful code execution."""
mock_response = Mock()
mock_response.status_code = 200
mock_response.json.return_value = {
"status": "success",
"stdout": '{"result": 42}',
"stderr": "",
"exit_code": 0,
"time_used_ms": 100.0,
"memory_used_kb": 1024.0
}
mock_response.json.return_value = {"status": "success", "stdout": '{"result": 42}', "stderr": "", "exit_code": 0, "time_used_ms": 100.0, "memory_used_kb": 1024.0}
mock_post.return_value = mock_response
provider = SelfManagedProvider()
provider._initialized = True
result = provider.execute_code(
instance_id="test-123",
code="def main(): return {'result': 42}",
language="python",
timeout=10
)
result = provider.execute_code(instance_id="test-123", code="def main(): return {'result': 42}", language="python", timeout=10)
assert result.stdout == '{"result": 42}'
assert result.stderr == ""
@@ -254,7 +215,7 @@ class TestSelfManagedProvider:
assert result.metadata["status"] == "success"
assert result.metadata["instance_id"] == "test-123"
@patch('requests.post')
@patch("requests.post")
def test_execute_code_maps_structured_result_into_metadata(self, mock_post):
"""Test successful code execution with structured result envelope."""
mock_response = Mock()
@@ -277,19 +238,14 @@ class TestSelfManagedProvider:
provider = SelfManagedProvider()
provider._initialized = True
result = provider.execute_code(
instance_id="test-123",
code="def main(): return {'items': ['a', 'b']}",
language="python",
timeout=10
)
result = provider.execute_code(instance_id="test-123", code="def main(): return {'items': ['a', 'b']}", language="python", timeout=10)
assert result.stdout == "debug line\n"
assert result.metadata["result_present"] is True
assert result.metadata["result_value"] == {"items": ["a", "b"]}
assert result.metadata["result_type"] == "json"
@patch('requests.post')
@patch("requests.post")
def test_execute_code_timeout(self, mock_post):
"""Test code execution timeout."""
mock_post.side_effect = requests.Timeout()
@@ -298,14 +254,9 @@ class TestSelfManagedProvider:
provider._initialized = True
with pytest.raises(TimeoutError, match="Execution timed out"):
provider.execute_code(
instance_id="test-123",
code="while True: pass",
language="python",
timeout=5
)
provider.execute_code(instance_id="test-123", code="while True: pass", language="python", timeout=5)
@patch('requests.post')
@patch("requests.post")
def test_execute_code_http_error(self, mock_post):
"""Test code execution with HTTP error."""
mock_response = Mock()
@@ -317,22 +268,14 @@ class TestSelfManagedProvider:
provider._initialized = True
with pytest.raises(RuntimeError, match="HTTP 500"):
provider.execute_code(
instance_id="test-123",
code="invalid code",
language="python"
)
provider.execute_code(instance_id="test-123", code="invalid code", language="python")
def test_execute_code_not_initialized(self):
"""Test executing code when provider not initialized."""
provider = SelfManagedProvider()
with pytest.raises(RuntimeError, match="Provider not initialized"):
provider.execute_code(
instance_id="test-123",
code="print('hello')",
language="python"
)
provider.execute_code(instance_id="test-123", code="print('hello')", language="python")
def test_destroy_instance(self):
"""Test destroying an instance (no-op for self-managed)."""
@@ -344,7 +287,7 @@ class TestSelfManagedProvider:
assert result is True
@patch('requests.get')
@patch("requests.get")
def test_health_check_success(self, mock_get):
"""Test successful health check."""
mock_response = Mock()
@@ -358,7 +301,7 @@ class TestSelfManagedProvider:
assert result is True
mock_get.assert_called_once_with("http://localhost:9385/healthz", timeout=5)
@patch('requests.get')
@patch("requests.get")
def test_health_check_failure(self, mock_get):
"""Test health check failure."""
mock_get.side_effect = Exception("Connection error")
@@ -439,20 +382,20 @@ class TestProviderInterface:
provider = SelfManagedProvider()
# Check all abstract methods are implemented
assert hasattr(provider, 'initialize')
assert hasattr(provider, "initialize")
assert callable(provider.initialize)
assert hasattr(provider, 'create_instance')
assert hasattr(provider, "create_instance")
assert callable(provider.create_instance)
assert hasattr(provider, 'execute_code')
assert hasattr(provider, "execute_code")
assert callable(provider.execute_code)
assert hasattr(provider, 'destroy_instance')
assert hasattr(provider, "destroy_instance")
assert callable(provider.destroy_instance)
assert hasattr(provider, 'health_check')
assert hasattr(provider, "health_check")
assert callable(provider.health_check)
assert hasattr(provider, 'get_supported_languages')
assert hasattr(provider, "get_supported_languages")
assert callable(provider.get_supported_languages)

View File

@@ -76,9 +76,7 @@ def test_python_builtins_import_is_rejected():
assert is_safe is False
# Pin the specific reason: rejection must come from the new ``builtins``
# entry in ``DANGEROUS_IMPORTS``, not from some unrelated parse error.
assert any("builtins" in issue for issue, _ in issues), (
f"expected an issue mentioning 'builtins', got {issues!r}"
)
assert any("builtins" in issue for issue, _ in issues), f"expected an issue mentioning 'builtins', got {issues!r}"
def test_python_attribute_eval_call_is_rejected():
@@ -94,9 +92,7 @@ def test_python_attribute_eval_call_is_rejected():
# not from the ``import builtins`` line above. We assert ``exec`` is in at
# least one finding so the test fails if visit_Call's attribute branch is
# ever reverted.
assert any("exec" in issue for issue, _ in issues), (
f"expected an issue mentioning 'exec', got {issues!r}"
)
assert any("exec" in issue for issue, _ in issues), f"expected an issue mentioning 'exec', got {issues!r}"
def test_javascript_safe_code_still_passes():

View File

@@ -36,17 +36,14 @@ print("✓ Provider has all required methods")
print("\n[3/5] Testing SDK imports...")
try:
# Check if agentrun SDK is available using importlib
if (
importlib.util.find_spec("agentrun.sandbox") is None
or importlib.util.find_spec("agentrun.utils.config") is None
or importlib.util.find_spec("agentrun.utils.exception") is None
):
if importlib.util.find_spec("agentrun.sandbox") is None or importlib.util.find_spec("agentrun.utils.config") is None or importlib.util.find_spec("agentrun.utils.exception") is None:
raise ImportError("agentrun SDK not found")
# Verify imports work (assign to _ to indicate they're intentionally unused)
from agentrun.sandbox import CodeInterpreterSandbox, TemplateType, CodeLanguage
from agentrun.utils.config import Config
from agentrun.utils.exception import ServerError
_ = (CodeInterpreterSandbox, TemplateType, CodeLanguage, Config, ServerError)
print("✓ SDK modules imported successfully")

File diff suppressed because one or more lines are too long

File diff suppressed because one or more lines are too long

File diff suppressed because one or more lines are too long

File diff suppressed because one or more lines are too long

File diff suppressed because one or more lines are too long

File diff suppressed because one or more lines are too long

File diff suppressed because one or more lines are too long

File diff suppressed because one or more lines are too long

File diff suppressed because one or more lines are too long

File diff suppressed because one or more lines are too long

File diff suppressed because one or more lines are too long

File diff suppressed because one or more lines are too long

File diff suppressed because one or more lines are too long

File diff suppressed because one or more lines are too long

View File

@@ -18,29 +18,29 @@ import os
from agent.canvas import Canvas
from common import settings
if __name__ == '__main__':
if __name__ == "__main__":
parser = argparse.ArgumentParser()
dsl_default_path = os.path.join(
os.path.dirname(os.path.realpath(__file__)),
"dsl_examples",
"retrieval_and_generate.json",
)
parser.add_argument('-s', '--dsl', default=dsl_default_path, help="input dsl", action='store', required=True)
parser.add_argument('-t', '--tenant_id', default=False, help="Tenant ID", action='store', required=True)
parser.add_argument('-m', '--stream', default=False, help="Stream output", action='store_true', required=False)
parser.add_argument("-s", "--dsl", default=dsl_default_path, help="input dsl", action="store", required=True)
parser.add_argument("-t", "--tenant_id", default=False, help="Tenant ID", action="store", required=True)
parser.add_argument("-m", "--stream", default=False, help="Stream output", action="store_true", required=False)
args = parser.parse_args()
settings.init_settings()
canvas = Canvas(open(args.dsl, "r").read(), args.tenant_id)
if canvas.get_prologue():
print(f"==================== Bot =====================\n> {canvas.get_prologue()}", end='')
print(f"==================== Bot =====================\n> {canvas.get_prologue()}", end="")
query = ""
while True:
canvas.reset(True)
query = input("\n==================== User =====================\n> ")
ans = canvas.run(query=query)
print("==================== Bot =====================\n> ", end='')
print("==================== Bot =====================\n> ", end="")
for ans in canvas.run(query=query):
print(ans, end='\n', flush=True)
print(ans, end="\n", flush=True)
print(canvas.path)

View File

@@ -22,8 +22,9 @@ from typing import Dict, Type
_package_path = os.path.dirname(__file__)
__all_classes: Dict[str, Type] = {}
def _import_submodules() -> None:
for filename in os.listdir(_package_path): # noqa: F821
for filename in os.listdir(_package_path): # noqa: F821
if filename.startswith("__") or not filename.endswith(".py") or filename.startswith("base"):
continue
module_name = filename[:-3]
@@ -34,15 +35,16 @@ def _import_submodules() -> None:
except ImportError as e:
print(f"Warning: Failed to import module {module_name}: {str(e)}")
def _extract_classes_from_module(module: ModuleType) -> None:
for name, obj in inspect.getmembers(module):
if (inspect.isclass(obj) and
obj.__module__ == module.__name__ and not name.startswith("_")):
if inspect.isclass(obj) and obj.__module__ == module.__name__ and not name.startswith("_"):
__all_classes[name] = obj
globals()[name] = obj
_import_submodules()
__all__ = list(__all_classes.keys()) + ["__all_classes"]
del _package_path, _import_submodules, _extract_classes_from_module
del _package_path, _import_submodules, _extract_classes_from_module

View File

@@ -13,44 +13,85 @@
# See the License for the specific language governing permissions and
# limitations under the License.
#
import logging
import os
import time
from abc import ABC
import pandas as pd
from agent.component.base import ComponentBase, ComponentParamBase
from agent.tools.base import ToolMeta, ToolParamBase, ToolBase
from common.connection_utils import timeout
class AkShareParam(ComponentParamBase):
class AkShareParam(ToolParamBase):
"""
Define the AkShare component parameters.
"""
def __init__(self):
self.meta: ToolMeta = {
"name": "akshare_stock_news",
"description": "AkShare retrieves the latest news articles for a given Chinese A-share stock from East Money (东方财富).",
"parameters": {
"query": {
"type": "string",
"description": "The stock symbol/code to fetch news for, e.g. '600519'.",
"default": "{sys.query}",
"required": True,
}
},
}
super().__init__()
self.top_n = 10
def check(self):
self.check_positive_integer(self.top_n, "Top N")
def get_input_form(self) -> dict[str, dict]:
return {"query": {"name": "Stock symbol", "type": "line"}}
class AkShare(ComponentBase, ABC):
class AkShare(ToolBase, ABC):
component_name = "AkShare"
def _run(self, history, **kwargs):
import akshare as ak
ans = self.get_input()
ans = ",".join(ans["content"]) if "content" in ans else ""
if not ans:
return AkShare.be_output("")
@timeout(int(os.environ.get("COMPONENT_EXEC_TIMEOUT", 12)))
def _invoke(self, **kwargs):
if self.check_if_canceled("AkShare processing"):
return
try:
ak_res = []
stock_news_em_df = ak.stock_news_em(symbol=ans)
stock_news_em_df = stock_news_em_df.head(self._param.top_n)
ak_res = [{"content": '<a href="' + i["新闻链接"] + '">' + i["新闻标题"] + '</a>\n 新闻内容: ' + i[
"新闻内容"] + " \n发布时间:" + i["发布时间"] + " \n文章来源: " + i["文章来源"]} for index, i in stock_news_em_df.iterrows()]
except Exception as e:
return AkShare.be_output("**ERROR**: " + str(e))
symbol = kwargs.get("query")
if not symbol:
self.set_output("formalized_content", "")
return ""
if not ak_res:
return AkShare.be_output("")
last_e = None
for _ in range(self._param.max_retries + 1):
if self.check_if_canceled("AkShare processing"):
return
return pd.DataFrame(ak_res)
try:
import akshare as ak
df = ak.stock_news_em(symbol=symbol).head(self._param.top_n)
if self.check_if_canceled("AkShare processing"):
return
items = ['<a href="{}">{}</a>\n 新闻内容: {} \n发布时间:{} \n文章来源: {}'.format(i["新闻链接"], i["新闻标题"], i["新闻内容"], i["发布时间"], i["文章来源"]) for _, i in df.iterrows()]
res = "\n\n".join(items)
self.set_output("formalized_content", res)
return res
except Exception as e:
if self.check_if_canceled("AkShare processing"):
return
last_e = e
logging.exception(f"AkShare error: {e}")
time.sleep(self._param.delay_after_error)
if last_e:
self.set_output("_ERROR", str(last_e))
return f"AkShare error: {last_e}"
assert False, self.output()
def thoughts(self) -> str:
return "Looking up the latest stock news for: {}".format(self.get_input().get("query", "-_-!"))

View File

@@ -28,7 +28,7 @@ class ArXivParam(ToolParamBase):
"""
def __init__(self):
self.meta:ToolMeta = {
self.meta: ToolMeta = {
"name": "arxiv_search",
"description": """arXiv is a free distribution service and an open-access archive for nearly 2.4 million scholarly articles in the fields of physics, mathematics, computer science, quantitative biology, quantitative finance, statistics, electrical engineering and systems science, and economics. Materials on this site are not peer-reviewed by arXiv.""",
"parameters": {
@@ -36,26 +36,20 @@ class ArXivParam(ToolParamBase):
"type": "string",
"description": "The search keywords to execute with arXiv. The keywords should be the most important words/terms(includes synonyms) from the original request.",
"default": "{sys.query}",
"required": True
"required": True,
}
}
},
}
super().__init__()
self.top_n = 12
self.sort_by = 'submittedDate'
self.sort_by = "submittedDate"
def check(self):
self.check_positive_integer(self.top_n, "Top N")
self.check_valid_value(self.sort_by, "ArXiv Search Sort_by",
['submittedDate', 'lastUpdatedDate', 'relevance'])
self.check_valid_value(self.sort_by, "ArXiv Search Sort_by", ["submittedDate", "lastUpdatedDate", "relevance"])
def get_input_form(self) -> dict[str, dict]:
return {
"query": {
"name": "Query",
"type": "line"
}
}
return {"query": {"name": "Query", "type": "line"}}
class ArXiv(ToolBase, ABC):
@@ -71,29 +65,20 @@ class ArXiv(ToolBase, ABC):
return ""
last_e = ""
for _ in range(self._param.max_retries+1):
for _ in range(self._param.max_retries + 1):
if self.check_if_canceled("ArXiv processing"):
return
try:
sort_choices = {"relevance": arxiv.SortCriterion.Relevance,
"lastUpdatedDate": arxiv.SortCriterion.LastUpdatedDate,
'submittedDate': arxiv.SortCriterion.SubmittedDate}
sort_choices = {"relevance": arxiv.SortCriterion.Relevance, "lastUpdatedDate": arxiv.SortCriterion.LastUpdatedDate, "submittedDate": arxiv.SortCriterion.SubmittedDate}
arxiv_client = arxiv.Client()
search = arxiv.Search(
query=kwargs["query"],
max_results=self._param.top_n,
sort_by=sort_choices[self._param.sort_by]
)
search = arxiv.Search(query=kwargs["query"], max_results=self._param.top_n, sort_by=sort_choices[self._param.sort_by])
results = list(arxiv_client.results(search))
if self.check_if_canceled("ArXiv processing"):
return
self._retrieve_chunks(results,
get_title=lambda r: r.title,
get_url=lambda r: r.pdf_url,
get_content=lambda r: r.summary)
self._retrieve_chunks(results, get_title=lambda r: r.title, get_url=lambda r: r.pdf_url, get_content=lambda r: r.summary)
return self.output("formalized_content")
except Exception as e:
if self.check_if_canceled("ArXiv processing"):

View File

@@ -28,10 +28,9 @@ from common.mcp_tool_call_conn import MCPToolBinding, MCPToolCallSession, ToolCa
from timeit import default_timer as timer
from common.misc_utils import thread_pool_exec
class ToolParameter(TypedDict):
type: str
description: str
@@ -81,7 +80,9 @@ class LLMToolPluginCallSession(ToolCallSession):
resp = fallback_output
else:
resp = fallback_output
logging.warning(f"[ToolCall] resp is None, fallback to output name={name} output_keys={list(fallback_output.keys()) if isinstance(fallback_output, dict) else type(fallback_output).__name__}")
logging.warning(
f"[ToolCall] resp is None, fallback to output name={name} output_keys={list(fallback_output.keys()) if isinstance(fallback_output, dict) else type(fallback_output).__name__}"
)
except Exception as e:
logging.warning(f"[ToolCall] resp is None and output fallback failed name={name} err={e}")
@@ -96,28 +97,25 @@ class LLMToolPluginCallSession(ToolCallSession):
class ToolParamBase(ComponentParamBase):
def __init__(self):
#self.meta:ToolMeta = None
# self.meta:ToolMeta = None
super().__init__()
self._init_inputs()
self._init_attr_by_meta()
def _init_inputs(self):
self.inputs = {}
for k,p in self.meta["parameters"].items():
for k, p in self.meta["parameters"].items():
self.inputs[k] = deepcopy(p)
def _init_attr_by_meta(self):
for k,p in self.meta["parameters"].items():
for k, p in self.meta["parameters"].items():
if not hasattr(self, k):
setattr(self, k, p.get("default"))
def get_meta(self):
params = {}
for k, p in self.meta["parameters"].items():
params[k] = {
"type": p["type"],
"description": p["description"]
}
params[k] = {"type": p["type"], "description": p["description"]}
if "enum" in p:
params[k]["enum"] = p["enum"]
@@ -129,12 +127,8 @@ class ToolParamBase(ComponentParamBase):
"function": {
"name": function_name,
"description": desc,
"parameters": {
"type": "object",
"properties": params,
"required": [k for k, p in self.meta["parameters"].items() if p["required"]]
}
}
"parameters": {"type": "object", "properties": params, "required": [k for k, p in self.meta["parameters"].items() if p["required"]]},
},
}
@@ -209,20 +203,8 @@ class ToolBase(ComponentBase):
title = get_title(r)
url = get_url(r)
score = get_score(r) if get_score else 1
chunks.append({
"chunk_id": id,
"content": content,
"doc_id": id,
"docnm_kwd": title,
"similarity": score,
"url": url
})
aggs.append({
"doc_name": title,
"doc_id": id,
"count": 1,
"url": url
})
chunks.append({"chunk_id": id, "content": content, "doc_id": id, "docnm_kwd": title, "similarity": score, "url": url})
aggs.append({"doc_name": title, "doc_id": id, "count": 1, "url": url})
self._canvas.add_reference(chunks, aggs)
self.set_output("formalized_content", "\n".join(kb_prompt({"chunks": chunks, "doc_aggs": aggs}, 200000, True)))

View File

@@ -13,10 +13,13 @@
# See the License for the specific language governing permissions and
# limitations under the License.
#
import logging
import os
from abc import ABC
import asyncio
from crawl4ai import AsyncWebCrawler
from agent.tools.base import ToolParamBase, ToolBase
from crawl4ai import AsyncWebCrawler, BrowserConfig, CrawlerRunConfig, CacheMode
from agent.tools.base import ToolMeta, ToolParamBase, ToolBase
from common.connection_utils import timeout
class CrawlerParam(ToolParamBase):
@@ -25,6 +28,18 @@ class CrawlerParam(ToolParamBase):
"""
def __init__(self):
self.meta: ToolMeta = {
"name": "web_crawler",
"description": "This tool can be used to crawl a web page and return its content as HTML, Markdown, or the extracted main text.",
"parameters": {
"query": {
"type": "string",
"description": "The absolute URL (including the http:// or https:// scheme) of the web page to crawl.",
"default": "{sys.query}",
"required": True,
}
},
}
super().__init__()
self.proxy = None
self.extract_type = "markdown"
@@ -32,37 +47,70 @@ class CrawlerParam(ToolParamBase):
def check(self):
self.check_valid_value(self.extract_type, "Type of content from the crawler", ["html", "markdown", "content"])
def get_input_form(self) -> dict[str, dict]:
return {"query": {"name": "URL", "type": "line"}}
class Crawler(ToolBase, ABC):
component_name = "Crawler"
def _run(self, history, **kwargs):
@timeout(int(os.environ.get("COMPONENT_EXEC_TIMEOUT", 10 * 60)))
def _invoke(self, **kwargs):
from common.ssrf_guard import assert_url_is_safe, pin_dns_global
ans = self.get_input()
ans = " - ".join(ans["content"]) if "content" in ans else ""
if self.check_if_canceled("Crawler processing"):
return
url = kwargs.get("query")
if not url:
self.set_output("formalized_content", "")
return ""
try:
_ssrf_hostname, _ssrf_ip = assert_url_is_safe(ans)
_ssrf_hostname, _ssrf_ip = assert_url_is_safe(url)
except ValueError:
return Crawler.be_output("URL not valid")
msg = "URL not valid"
self.set_output("_ERROR", msg)
return msg
try:
# pin_dns_global is used (not thread-local) because crawl4ai resolves
# DNS in asyncio executor threads that don't share thread-local state.
with pin_dns_global(_ssrf_hostname, _ssrf_ip):
result = asyncio.run(self.get_web(ans))
result = asyncio.run(self.get_web(url))
return Crawler.be_output(result)
if self.check_if_canceled("Crawler processing"):
return
result = result or ""
self.set_output("formalized_content", result)
return result
except Exception as e:
return Crawler.be_output(f"An unexpected error occurred: {str(e)}")
if self.check_if_canceled("Crawler processing"):
return
logging.exception(f"Crawler error: {e}")
msg = f"An unexpected error occurred: {str(e)}"
self.set_output("_ERROR", msg)
return msg
async def get_web(self, url):
if self.check_if_canceled("Crawler async operation"):
return
proxy = self._param.proxy if self._param.proxy else None
async with AsyncWebCrawler(verbose=True, proxy=proxy) as crawler:
result = await crawler.arun(url=url, bypass_cache=True)
browser_config = BrowserConfig(
verbose=True,
proxy_config=proxy,
)
run_config = CrawlerRunConfig(
cache_mode=CacheMode.BYPASS,
)
async with AsyncWebCrawler(config=browser_config) as crawler:
result = await crawler.arun(url=url, config=run_config)
if self.check_if_canceled("Crawler async operation"):
return

View File

@@ -27,18 +27,53 @@ class DeepLParam(ComponentParamBase):
super().__init__()
self.auth_key = "xxx"
self.parameters = []
self.source_lang = 'ZH'
self.target_lang = 'EN-GB'
self.source_lang = "ZH"
self.target_lang = "EN-GB"
def check(self):
self.check_valid_value(self.source_lang, "Source language",
['AR', 'BG', 'CS', 'DA', 'DE', 'EL', 'EN', 'ES', 'ET', 'FI', 'FR', 'HU', 'ID', 'IT',
'JA', 'KO', 'LT', 'LV', 'NB', 'NL', 'PL', 'PT', 'RO', 'RU', 'SK', 'SL', 'SV', 'TR',
'UK', 'ZH'])
self.check_valid_value(self.target_lang, "Target language",
['AR', 'BG', 'CS', 'DA', 'DE', 'EL', 'EN-GB', 'EN-US', 'ES', 'ET', 'FI', 'FR', 'HU',
'ID', 'IT', 'JA', 'KO', 'LT', 'LV', 'NB', 'NL', 'PL', 'PT-BR', 'PT-PT', 'RO', 'RU',
'SK', 'SL', 'SV', 'TR', 'UK', 'ZH'])
self.check_valid_value(
self.source_lang,
"Source language",
["AR", "BG", "CS", "DA", "DE", "EL", "EN", "ES", "ET", "FI", "FR", "HU", "ID", "IT", "JA", "KO", "LT", "LV", "NB", "NL", "PL", "PT", "RO", "RU", "SK", "SL", "SV", "TR", "UK", "ZH"],
)
self.check_valid_value(
self.target_lang,
"Target language",
[
"AR",
"BG",
"CS",
"DA",
"DE",
"EL",
"EN-GB",
"EN-US",
"ES",
"ET",
"FI",
"FR",
"HU",
"ID",
"IT",
"JA",
"KO",
"LT",
"LV",
"NB",
"NL",
"PL",
"PT-BR",
"PT-PT",
"RO",
"RU",
"SK",
"SL",
"SV",
"TR",
"UK",
"ZH",
],
)
class DeepL(ComponentBase, ABC):
@@ -57,8 +92,7 @@ class DeepL(ComponentBase, ABC):
try:
translator = deepl.Translator(self._param.auth_key)
result = translator.translate_text(ans, source_lang=self._param.source_lang,
target_lang=self._param.target_lang)
result = translator.translate_text(ans, source_lang=self._param.source_lang, target_lang=self._param.target_lang)
return DeepL.be_output(result.text)
except Exception as e:

View File

@@ -28,7 +28,7 @@ class DuckDuckGoParam(ToolParamBase):
"""
def __init__(self):
self.meta:ToolMeta = {
self.meta: ToolMeta = {
"name": "duckduckgo_search",
"description": "DuckDuckGo is a search engine focused on privacy. It offers search capabilities for web pages, images, and provides translation services. DuckDuckGo also features a private AI chat interface, providing users with an AI assistant that prioritizes data protection.",
"parameters": {
@@ -36,7 +36,7 @@ class DuckDuckGoParam(ToolParamBase):
"type": "string",
"description": "The search keywords to execute with DuckDuckGo. The keywords should be the most important words/terms(includes synonyms) from the original request.",
"default": "{sys.query}",
"required": True
"required": True,
},
"channel": {
"type": "string",
@@ -45,7 +45,7 @@ class DuckDuckGoParam(ToolParamBase):
"default": "general",
"required": False,
},
}
},
}
super().__init__()
self.top_n = 10
@@ -56,18 +56,7 @@ class DuckDuckGoParam(ToolParamBase):
self.check_valid_value(self.channel, "Web Search or News", ["text", "news"])
def get_input_form(self) -> dict[str, dict]:
return {
"query": {
"name": "Query",
"type": "line"
},
"channel": {
"name": "Channel",
"type": "options",
"value": "general",
"options": ["general", "news"]
}
}
return {"query": {"name": "Query", "type": "line"}, "channel": {"name": "Channel", "type": "options", "value": "general", "options": ["general", "news"]}}
class DuckDuckGo(ToolBase, ABC):
@@ -83,7 +72,7 @@ class DuckDuckGo(ToolBase, ABC):
return ""
last_e = ""
for _ in range(self._param.max_retries+1):
for _ in range(self._param.max_retries + 1):
if self.check_if_canceled("DuckDuckGo processing"):
return
@@ -99,10 +88,7 @@ class DuckDuckGo(ToolBase, ABC):
if self.check_if_canceled("DuckDuckGo processing"):
return
self._retrieve_chunks(duck_res,
get_title=lambda r: r["title"],
get_url=lambda r: r.get("href", r.get("url")),
get_content=lambda r: r["body"])
self._retrieve_chunks(duck_res, get_title=lambda r: r["title"], get_url=lambda r: r.get("href", r.get("url")), get_content=lambda r: r["body"])
self.set_output("json", duck_res)
return self.output("formalized_content")
else:
@@ -116,10 +102,7 @@ class DuckDuckGo(ToolBase, ABC):
if self.check_if_canceled("DuckDuckGo processing"):
return
self._retrieve_chunks(duck_res,
get_title=lambda r: r["title"],
get_url=lambda r: r.get("href", r.get("url")),
get_content=lambda r: r["body"])
self._retrieve_chunks(duck_res, get_title=lambda r: r["title"], get_url=lambda r: r.get("href", r.get("url")), get_content=lambda r: r["body"])
self.set_output("json", duck_res)
return self.output("formalized_content")
except Exception as e:

Some files were not shown because too many files have changed in this diff Show More