### Summary
Add Querit Contents as a built-in page content tool for RAGFlow Agents
and Canvas workflows.
Querit Contents crawls one or more known URLs and returns their page
content and optional metadata. It complements the existing Querit Search
tool: Search discovers relevant pages, while Contents retrieves pages
already selected by an Agent or workflow.
This integration supports two usage modes:
- A standalone `QueritContents` node in Canvas workflows.
- An embedded content tool available to RAGFlow Agents.
Make `ResolveModelContentLength` honor the per-model custom **context window length** (`content_length`) — stored in the Python-legacy `tenant_model.extra["max_tokens"]` field, whose semantic meaning is the context window, NOT the generation cap — **before** any provider-catalog read, and remove the parallel service-layer implementation so every consumer shares one resolution path.
Switch the agent LLM component's `fitMessages` from the local map-based `messageFitInRaw`/`countAllTokens`/`stringContent`/`setContent` helpers to the new shared `internal/component/messagefit` package (#18091).
- Prefer canonical `dataset_ids` over legacy `kb_ids` in Canvas
retrieval components.
- Cover selected and explicitly cleared dataset IDs with regression
tests.
- Add cross-language retrieval with dataset-specific embedding and
rerank models.
- Support vector and keyword similarity controls, metadata filtering,
TOC enhancement, and child-chunk expansion.
- Route Canvas retrieval across datasets and memories with compatible
embedding validation.
- Start the Agent model-stream collector before ReAct execution.
- Preserve all streamed reasoning/content deltas and drain the collector
on errors.
- Add coverage for delayed thinking streams and tool-call execution.
## Summary
- Normalize Canvas component names before resolving Go Agent tools and
parameters.
- Add regression coverage for CodeExec and other Canvas tool mappings.
## Testing
- `CGO_ENABLED=0 go test -count=1 ./internal/agent/tool
./internal/agent/component`
Port Python rag/advanced_rag agentic search to Go: ES-backed dataset-nav
service, agentic-search harness, and agent tools.
Includes agentic-search port plan and self-review docs.
## Summary
Classify the Go test suite by dependency level using build tags so the
default
`go test ./...` run stays self-contained, and add local convenience
commands
plus a documented convention.
- Add build tags to 5 real-service tests that were previously un-tagged
and only
soft-isolated via `t.Skip`: `kg_test.go` (integration), `minio_test.go`
(integration), `template_integration_test.go` (integration),
`stagehand_runtime_integration_test.go` (integration),
`pipeline_e2e_test.go` (e2e).
The default unit run no longer compiles/attempts these.
- Reclassify the full-pipeline `real_consumer` tests from `integration`
to `e2e`.
- Add `build.sh` shortcuts: `--test-integration`, `--test-e2e`,
`--test-manual`,
`--test-all` (integration + e2e; `manual` is excluded and is local
opt-in only,
never run in CI).
- Document the tier scheme (unit / integration / e2e / manual +
orthogonal cgo) in
`AGENTS.md`.
## Tier definitions
| Tier | Build tag | Runs by default? |
|---|---|---|
| Unit | (none) | Yes — in-memory SQLite / miniredis / httptest stubs |
| Integration | `integration` | No (`-tags integration`) — single real
service |
| E2E | `e2e` | No (`-tags e2e`) — full ingest→index→retrieve pipeline |
| Manual | `manual` | No (`-tags manual`) — very slow; never in CI |
## Verification
- `gofmt -l` clean on all changed files; `bash -n build.sh` OK.
- `go list` confirms the default set excludes the tagged files, and
`-tags integration` / `-tags e2e` include them.
- Full regression: unit / integration / e2e each **97 ok, 0 FAIL**.
- Fixed a regression where `pipeline_knowledge_compiler_test.go` relied
on a
transitive import side-effect from `template_integration_test.go` to
register
the `File`/`Parser`/`TokenChunker` components; it now blank-imports the
component packages directly.
## Test plan
- [ ] `./build.sh --test` (unit) passes
- [ ] `./build.sh --test-integration` passes (needs real services; skips
otherwise)
- [ ] `./build.sh --test-e2e` passes (needs real services; skips
otherwise)
---------
Co-authored-by: CodeBuddy <noreply@cnb.cool>
## Summary
This PR hardens the Go ingestion **Extractor** and **LLM retry** paths
and closes several Python->Go parity gaps in the keyword/question/tag
extraction flow.
- **Generic retry utility** (`internal/common/retry.go`):
`RetryWithBackoff` with exponential backoff (default 3 retries, 2s
initial delay, capped at 1m), context-aware sleep, and a `maxRetries<=0`
fast path. Covered by `internal/common/retry_test.go`.
- **LLM retry reuse**: `agent/component/llm_retry.go` now delegates to
`common.RetryWithBackoff` instead of an inline loop (behavior preserved:
ctx cancellation short-circuits the backoff).
- **Extractor LLM calls** (`extractor.go`):
- `call()` now retries transient LLM failures via `RetryWithBackoff`
(retry exhaustion fails the chunk instead of silently skipping).
- Sets `temperature = 0.2`, matching Python `generator.py:230,245`.
- Runs keyword and question extraction **concurrently** per chunk when
both are enabled (`task_executor.py:444-448`), with mutex-guarded map
writes to avoid data races.
- Substitutes `{field_name}` placeholders (including `{chunks}` -> chunk
text) in `prompt`/`system_prompt` before the call, mirroring Python
`string_format` (`extractor.py:102-103`); unmatched placeholders are
left as-is.
- Falls back to the **tenant default chat model** when `llm_id` is empty
(`task_executor.py:573-574`).
- Strips `` **greedily** (`strings.LastIndex`) in
`cleanExtractionResult`.
- **Auto-tagging** (`extractor_tag.go`): drops the `in.llmID != ""`
guards so an empty `llm_id` no longer skips tagging (uses the tenant
default model), and strips `` greedily in `parseTaggerResponse`.
- **Docs**: fixes a misleading `PresentationChunker` docstring that
claimed per-slide `image`/`position` output (the PPTX path emits none —
unlike PDF), and removes a stale `docs/migration_python_go_diff.md`
reference in `media_dispatch.go`.
## Test plan
- `bash build.sh --test ./internal/common/...` — passes (new retry
utility + tests).
- `bash build.sh --test ./internal/ingestion/component/...` — passes
(extractor/chunker/schema).
- `gofmt` and lefthook pre-commit checks pass.
Note: the personal `docs/migration_python_go_diff.md` working notebook
in the tree is intentionally **not** part of this PR.
---------
Co-authored-by: CodeBuddy <noreply@codebuddy.ai>
## Summary
- Return the configured `empty_response` when retrieval has no query or
no chunks.
- Preserve `formalized_content` for downstream Message nodes.
## Testing
- `bash build.sh --go`
- `ok ragflow/internal/agent/tool`
- `ok ragflow/internal/agent/component`
## Summary
- Route ReAct graph step-limit errors through the Agent `_ERROR` output.
- Align the Go Agent default max rounds with Python by using five
rounds.
- Preserve cancellation, timeout, and configuration errors as real
failures.
## Testing
- `bash build.sh --test ./internal/agent/component/...`
<img width="2050" height="1409" alt="image"
src="https://github.com/user-attachments/assets/108e2d27-e115-4595-91a2-21f90c2531ae"
/>
## Summary
Three groups of changes across the Go ingestion pipeline:
### 1. DOCX parsing improvements
- **docx_parser.go**: Enhanced DOCX parsing with better structure
extraction and media handling
- **docx_parser_cgo_test.go**, **docx_parser_test.go**: Companion tests
- **office_parsers_no_cgo.go**: Stub sync for non-CGO builds
### 2. Email (.eml) parsing: base64 Content-Transfer-Encoding decoding
- **email_parser.go** (`decodeCTE`): Added Content-Transfer-Encoding
decoding for base64 and quoted-printable. Go's `mime/multipart.Reader`
does not decode Content-Transfer-Encoding automatically, so attachments
with `Content-Transfer-Encoding: base64` remained base64-encoded in the
output. The new `decodeCTE` helper is called after reading each
multipart part's raw bytes in `readMailBody`, mirroring Python's
`part.get_payload(decode=True)`.
- **email_parser_test.go**: Two new tests — simple base64 attachment and
nested multipart/alternative with base64 attachment.
### 3. Extractor LLM driver fix + ModelDriver consolidation
- **extractor.go**: Fixed a bug where the Extractor component used
`ModelFactory.CreateModelDriver()`, which creates bare model instances
without API keys or provider configuration. Switched to
`models.GetPreconfiguredDriver()` which resolves the actual
pre-configured driver from `ProviderManager`, matching the codepath used
by `llm.go`. This fixes auto keyword/question extraction in DSL
pipelines that require LLM calls.
- **get_driver.go** (new): Extracted shared `GetPreconfiguredDriver()`
from `llm.go:newChatModelDriver()` so both `llm.go` and `extractor.go`
use the same codepath.
- **get_driver_test.go** (new): Tests for the shared driver resolution.
- **llm.go**: Replaced inline driver resolution with
`models.GetPreconfiguredDriver()`.
### 4. Chunker fixes and observability
- **group.go** (`extractLineRecords`): Fixed to also read `markdown` and
`html` payload keys — previously it only read `text`/`content`, causing
GroupTitleChunker to silently return empty results for markdown-format
parser output.
- **common.go** (`compileDelimPattern`): Aligned with Python's
`_compile_delimiter_pattern` — only backtick-wrapped delimiters produce
an active regex pattern; plain delimiters are not compiled into the
split regex.
- **token.go** (`applyChildrenDelim`): Set `DocType` and `CKType` to
`"text"` on created ChunkDocs so the token-size merge path correctly
identifies and merges text segments.
- **parser.go**, **extractor.go**, **tokenizer.go**, **group.go**,
**hierarchy.go**: Added debug-level logging for pipeline diagnostics.
- **parser_dispatch_test.go**, **group_test.go**: New tests.
## Verification
- All Go tests pass: `bash build.sh --test ./internal/parser/parser/...`
and `bash build.sh --test ./internal/ingestion/component/...`
- Build succeeds: `bash build.sh --go`
## Summary
- Load the Agent history window with Python-compatible defaults.
- Include prior conversation messages in ReAct prompts without
duplicating the current user input.
## Testing
- `bash build.sh --test ./internal/agent/...`
- `bash build.sh --test ./internal/service/...`
<img width="2014" height="1121" alt="image"
src="https://github.com/user-attachments/assets/15327e2a-3286-42ec-9415-c06ada758156"
/>
## Summary
- Preserve request-scoped system variables such as files and user IDs
during Canvas execution.
- Persist conversation history, turn counts, and tool memory in the
session DSL across turns.
- Parse agent uploads into `sys.files` and align system variable
rendering with Python.
## Testing
- `bash build.sh --test ./internal/agent/...`
- `bash build.sh --test ./internal/service/...`
<img width="1896" height="1232" alt="image"
src="https://github.com/user-attachments/assets/b420cd97-53c3-470f-a3e1-d39cea26a213"
/>
## Summary
- Add `GetInputForm()` for `ListOperationsComponent` to expose `Query`
input field in debug UI
- Add `GetInputForm()` for `VariableAggregatorComponent` to expose
`Variables` input field in debug UI
## Test
- Verify input form fields render correctly for both components in the
debug UI
### Summary
As title
- [x] fix tool & component `Retrieval KB`
- [x] fix agent cannot use `Retrieval KB` in agent chat
#### Main cause
Model provider do not set up:
```Go
if chatModelConfig.Tools != nil {
reqBody["tools"] = chatModelConfig.Tools
}
```
#### Working Now
<img width="3774" height="2128" alt="image"
src="https://github.com/user-attachments/assets/400a349d-0211-43e5-a7ec-7a014acf77a6"
/>
```
____________________________________
< This PR takes me all day to do it. >
------------------------------------
\
\ (\__/)
(•ㅅ•)
/ づ
```
## Summary
- return the input form collected by
`DataOperationsComponent.GetInputForm`
- add a regression test covering system variables, component outputs,
and duplicate references
