Commit Graph

15 Commits

Author SHA1 Message Date
Rene Arredondo
dc8b6d767c fix(agent): inject uploaded attachments into LLM context (#15215) (#15220)
## Summary

Fixes #15215 — attachments uploaded to an agent were not reaching the
LLM.

When a user uploads a file in an agent chat, `canvas.run` parses it into
the `sys.files` global (text content for documents, `data:image/...`
URIs
for images — see `agent/canvas.py:752-768`). But the LLM/Agent
component's
`_prepare_prompt_variables` only substitutes variables the user's prompt
template explicitly references via `{var}` placeholders. The default
prompt is `[{"role": "user", "content": "{sys.query}"}]` with no
`{sys.files}`, so the parsed attachment content never reaches the model.

In the reporter's logs, this is why the agent saw only the bare query
`附件 摘要 attachment summary` and went searching the dataset instead of
reading the uploaded PDF.

## Fix

`agent/component/llm.py` — added `_collect_sys_files()` and an
auto-injection step in `_prepare_prompt_variables`:

- If `sys.files` is non-empty **and** neither `sys_prompt` nor any entry
  in `prompts` already contains `{sys.files}` (no double-injection),
  split the entries into text vs. `data:image/...` URIs.
- Image URIs are merged into `self.imgs`, which the existing logic uses
  to switch the chat model to `IMAGE2TEXT` and pass `images=...` to
  `async_chat`.
- Text content is appended to the last `user` role message in `msg`,
  mirroring how `dialog_service.async_chat_solo` handles attachments for
  the non-agent chat path (`api/db/services/dialog_service.py:318-321`).

Both `LLM._invoke_async` and `Agent._invoke_async` (tool-using) go
through `_prepare_prompt_variables`, so plain LLM nodes and Agent nodes
are fixed in both streaming and non-streaming paths.

## Test plan

- [ ] Upload a PDF attachment to an agent with the default `{sys.query}`
prompt and ask "summarize the attachment" — the model should answer
      from the file content rather than searching the knowledge base.
- [ ] Upload an image attachment to an agent and ask about its contents
—
      the model should switch to the vision-capable LLM and answer from
      the image.
- [ ] Verify that an agent whose prompt **does** include `{sys.files}`
      still works and does **not** include the file content twice.
- [ ] Verify that an agent run with no attachments behaves unchanged.
- [ ] Run `uv run pytest` to make sure no existing tests regress.

### Type of change

- [x] Bug Fix (non-breaking change which fixes an issue)
- [ ] New Feature (non-breaking change which adds functionality)
- [ ] Documentation Update
- [ ] Refactoring
- [ ] Performance Improvement
- [ ] Other (please describe):

---------

Co-authored-by: yzc <yuzhichang@gmail.com>
2026-06-30 15:48:59 +08:00
Hz_
0de8f3e127 feat: add missing qwen models to all_models.json (#16379)
Add 19 missing qwen models and 3 aliases to all_models.json.

Models added: qwen-image-2.0-pro (2026-06-22, 2026-04-22), qwen3.5-ocr,
qwen3.7-max-2026-05-17, qwen3.5-livetranslate-flash-realtime,
qwen3.5-omni-plus/flash-realtime, qwen-deep-research-2025-12-15,
qwen-flash-character-2026-02-26, qwen-plus-2025-11-05,
qwen-deep-search-planning, qwen3-s2s-flash-realtime-2025-09-22,
qwen-max-1201/longcontext/0107, qwen-1.8b-longcontext-chat

Aliases: qwen3.5-plus-2026-04-20, qwen-turbo-0919, qwen-1.8b-chat
2026-06-26 15:35:30 +08:00
Jin Hai
65afaa1292 Model config: add tools (#16371)
### What problem does this PR solve?

```
{
      "name": "glm-4-flash",
      "max_tokens": 128000,
      "model_types": [
        "chat"
      ],
      "tools": {
        "support": true
      }
}
```

```
RAGFlow(admin)> list provider 'zhipu-ai' models;
+------------+---------------+------------+---------------+----------------+-----------+-----------+
| dimensions | max_dimension | max_tokens | model_type    | name           | thinking  | tools     |
+------------+---------------+------------+---------------+----------------+-----------+-----------+
|            |               | 204800     | [chat]        | glm-5          | supported | supported |
|            |               | 204800     | [chat]        | glm-5-turbo    | supported | supported |
```

### Type of change

- [x] Bug Fix (non-breaking change which fixes an issue)

---------

Signed-off-by: Jin Hai <haijin.chn@gmail.com>
2026-06-26 11:37:51 +08:00
Jin Hai
7214a23614 Go: fix duplicate models (#16197)
### What problem does this PR solve?

1. Remove unused file
2. Remove duplicate models
3. Resort the function order

### Type of change

- [x] Bug Fix (non-breaking change which fixes an issue)
- [x] Refactoring

Signed-off-by: Jin Hai <haijin.chn@gmail.com>
2026-06-19 09:57:58 +08:00
Haruko386
b53b5bf12c Json add paddleOCR models (#16156)
close #15853

### What problem does this PR solve?

As title

### Type of change

- [x] Other (add models):
2026-06-18 17:57:41 +08:00
Hz_
8047857de0 fix(go): all_models.json (#16075)
### What problem does this PR solve?

This PR fixes Go admin server startup failure caused by duplicate model
aliases in conf/all_models.json.

The model provider loader builds a global lookup table from both model
name and alias values. Some aliases duplicated another model's name or
another
alias, for example amazon.titan-embed-text-v1, which caused startup to
fail with a duplicate alias error. This PR removes conflicting duplicate
aliases
  while keeping all model definitions intact.
2026-06-16 15:31:17 +08:00
Hz_
4a33455a20 feat(go-models): add more providers (#16017)
### What problem does this PR solve?

add more providers.
2026-06-16 12:54:19 +08:00
Haruko386
cafd8a1125 Json: add many models to all_models.json (#16013)
### What problem does this PR solve?

As title

### Type of change

- [x] Other (please describe): add some models
2026-06-15 15:25:49 +08:00
Hz_
eb6ea284a8 feat(go-models): Add google models to all_models.json (#16007)
### What problem does this PR solve?

Add google models to all_models.json
2026-06-15 11:37:56 +08:00
Haruko386
4115282c5f Json[model-provider] add nvidia, moonshot, minimax, claude, GPT models (#15970)
### What problem does this PR solve?

As title

### Type of change

- [x] Other (please describe): add models
2026-06-12 19:16:10 +08:00
Hz_
30724140d2 feat(go): Add Z.ai model entries to all_models.json Add missing Qwen commercial models and provider aliases (#15929)
### What problem does this PR solve?

- Add Z.ai model definitions to `conf/all_models.json`.
- Add missing Qwen / DashScope commercial API-only models, including:
    - Qwen3.7 / Qwen3.6 / Qwen3.5 Max, Plus, Flash families
    - Qwen Coder and Math commercial models
- Qwen VL, OCR, Omni, ASR, TTS, translation, image generation, and image
editing models
- Add verified provider-specific aliases for supported Qwen models:
  - DashScope / Alibaba Cloud Model Studio model IDs
  - OpenRouter `qwen/...` aliases
  - Amazon Bedrock `qwen.qwen3-*` model IDs
- Add `thinking` metadata for Qwen models that officially support
thinking mode.
- Remove aliases that exactly duplicate their own canonical `name`.
2026-06-12 14:33:01 +08:00
Haruko386
e3be39d0de Json: add some models (#15947)
### What problem does this PR solve?

As title

### Type of change

- [x] Other (please describe): add models
2026-06-12 14:32:21 +08:00
Hz_
515acf4f60 fix(go): Fix case-insensitive model alias lookup (#15911)
## Summary

- Normalize model alias index keys to lowercase
- Detect lowercase alias collisions during provider manager
initialization
- Fix ListModels metadata mapping for mixed-case provider aliases
2026-06-10 20:36:43 +08:00
Hz_
38755c705a feat(go): Add DeepSeek models and Gitee alias metadata tests (#15885)
This PR expands conf/all_models.json with DeepSeek model entries and
provider aliases.

Changes:

- Added DeepSeek model entries across `V4`, `V3.2`, `V3.1`, `V3`, `R1`,
`Coder`, `Math`, `VL`, `OCR`, `Prover`, `MoE`, and `LLM` series.
- Normalized model name values to lowercase canonical IDs.
- Added alias values for official DeepSeek/Hugging Face names and
provider-specific names from OpenRouter, VolcEngine, SiliconFlow,
HuaweiCloud, and QiniuCloud.
- Preserved model metadata such as max_tokens, model_types, and thinking
where applicable.
- Added Gitee ListModels tests to verify DeepSeek aliases map back to
model metadata from all_models.json.
- Added an optional Gitee integration test gated by
GITEE_LIST_MODELS_INTEGRATION=1.

Test:

/usr/local/go/bin/go clean -cache
/usr/local/go/bin/go test ./internal/entity/models -run
'TestGiteeListModels(MapsAllDeepSeekAliasesToModelMetadata|KeepsOwnedBySuffixAfterAliasMetadataLookup|
Integration)'
2026-06-10 13:59:23 +08:00
Jin Hai
55abf4f565 Go: new CLI command, list all models and show model (#15786)
### What problem does this PR solve?

```
RAGFlow(user)> list models;
+---------------------------+------------+-------------+--------------------+---------------------------------------------+
| alias                     | max_tokens | model_types | name               | thinking                                    |
+---------------------------+------------+-------------+--------------------+---------------------------------------------+
|                           | 1048576    | [chat]      | deepseek-v4-flash  | map[clear_thinking:true default_value:true] |
|                           | 1048576    | [chat]      | deepseek-v4-pro    | map[clear_thinking:true default_value:true] |
|                           | 1024000    | [chat]      | minimax-m3         | map[clear_thinking:true default_value:true] |
|                           | 64000      | [vision]    | glm-4.5v           | map[clear_thinking:true default_value:true] |
| [baai/bge-m3]             | 8192       | [embedding] | bge-m3             |                                             |
| [baai/bge-reranker-v2-m3] | 1024       | [rerank]    | bge-reranker-v2-m3 |                                             |
|                           |            | [tts]       | step-audio-tts-3b  |                                             |
| [qwen/qwen3-asr-1.7b]     |            | [asr]       | qwen3-asr-1.7b     |                                             |
| [paddleocr-vl-1.5]        |            | [ocr]       | paddleocr-vl-0.9b  |                                             |
+---------------------------+------------+-------------+--------------------+---------------------------------------------+
RAGFlow(user)> show model 'minimax-m3';
+--------------+---------------------------------------------+
| field        | value                                       |
+--------------+---------------------------------------------+
| name         | minimax-m3                                  |
| max_tokens   | 1024000                                     |
| model_types  | [chat]                                      |
| thinking     | map[clear_thinking:true default_value:true] |
| class        |                                             |
| alias        |                                             |
| ModelTypeMap |                                             |
+--------------+---------------------------------------------+
RAGFlow(user)> show model 'baai/bge-m3';
+--------------+---------------+
| field        | value         |
+--------------+---------------+
| model_types  | [embedding]   |
| thinking     |               |
| class        |               |
| alias        | [baai/bge-m3] |
| ModelTypeMap |               |
| name         | bge-m3        |
| max_tokens   | 8192          |
+--------------+---------------+
```

---------

Signed-off-by: Jin Hai <haijin.chn@gmail.com>
2026-06-08 21:38:15 +08:00