## Summary
- Route ReAct graph step-limit errors through the Agent `_ERROR` output.
- Align the Go Agent default max rounds with Python by using five
rounds.
- Preserve cancellation, timeout, and configuration errors as real
failures.
## Testing
- `bash build.sh --test ./internal/agent/component/...`
<img width="2050" height="1409" alt="image"
src="https://github.com/user-attachments/assets/108e2d27-e115-4595-91a2-21f90c2531ae"
/>
## Summary
- Register `TavilySearch` and `TavilyExtract` Canvas component names in
the Go Agent tool registry.
- Add regression coverage for building both tools by their Python
component names.
## Testing
- `bash build.sh --test ./internal/agent/tool/...`
- `bash build.sh --test ./internal/agent/component/...`
### Summary
As title:
Some operations on datasets have been fixed as they previously lacked
authentication, preventing team members from proper or correct usage.
---------
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
### Summary
As title
verif ied from CLI
```
RAGFlow(api/default)> asr with 'paraformer@test@funasr' audio './internal/test.wav' param '{"language": "en"}'
+----------------------------------------------------------------------------------------------------------------------+
| text |
+----------------------------------------------------------------------------------------------------------------------+
| The examination and testimony of the experts enabled the commission to conclude that five shots may have been fired. |
+----------------------------------------------------------------------------------------------------------------------+
```
---
<img width="2876" height="1377" alt="image"
src="https://github.com/user-attachments/assets/4cee62c6-1c71-43ce-b398-4d9fbff33c3c"
/>
```bash
INFO: 127.0.0.1:51910 - "POST /v1/audio/transcriptions HTTP/1.1" 200 OK██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00, 2.19it/s]
INFO: 127.0.0.1:60934 - "GET /v1/models HTTP/1.1" 200 OK%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00, 2.10it/s]
```
---------
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
## Summary
Three groups of changes across the Go ingestion pipeline:
### 1. DOCX parsing improvements
- **docx_parser.go**: Enhanced DOCX parsing with better structure
extraction and media handling
- **docx_parser_cgo_test.go**, **docx_parser_test.go**: Companion tests
- **office_parsers_no_cgo.go**: Stub sync for non-CGO builds
### 2. Email (.eml) parsing: base64 Content-Transfer-Encoding decoding
- **email_parser.go** (`decodeCTE`): Added Content-Transfer-Encoding
decoding for base64 and quoted-printable. Go's `mime/multipart.Reader`
does not decode Content-Transfer-Encoding automatically, so attachments
with `Content-Transfer-Encoding: base64` remained base64-encoded in the
output. The new `decodeCTE` helper is called after reading each
multipart part's raw bytes in `readMailBody`, mirroring Python's
`part.get_payload(decode=True)`.
- **email_parser_test.go**: Two new tests — simple base64 attachment and
nested multipart/alternative with base64 attachment.
### 3. Extractor LLM driver fix + ModelDriver consolidation
- **extractor.go**: Fixed a bug where the Extractor component used
`ModelFactory.CreateModelDriver()`, which creates bare model instances
without API keys or provider configuration. Switched to
`models.GetPreconfiguredDriver()` which resolves the actual
pre-configured driver from `ProviderManager`, matching the codepath used
by `llm.go`. This fixes auto keyword/question extraction in DSL
pipelines that require LLM calls.
- **get_driver.go** (new): Extracted shared `GetPreconfiguredDriver()`
from `llm.go:newChatModelDriver()` so both `llm.go` and `extractor.go`
use the same codepath.
- **get_driver_test.go** (new): Tests for the shared driver resolution.
- **llm.go**: Replaced inline driver resolution with
`models.GetPreconfiguredDriver()`.
### 4. Chunker fixes and observability
- **group.go** (`extractLineRecords`): Fixed to also read `markdown` and
`html` payload keys — previously it only read `text`/`content`, causing
GroupTitleChunker to silently return empty results for markdown-format
parser output.
- **common.go** (`compileDelimPattern`): Aligned with Python's
`_compile_delimiter_pattern` — only backtick-wrapped delimiters produce
an active regex pattern; plain delimiters are not compiled into the
split regex.
- **token.go** (`applyChildrenDelim`): Set `DocType` and `CKType` to
`"text"` on created ChunkDocs so the token-size merge path correctly
identifies and merges text segments.
- **parser.go**, **extractor.go**, **tokenizer.go**, **group.go**,
**hierarchy.go**: Added debug-level logging for pipeline diagnostics.
- **parser_dispatch_test.go**, **group_test.go**: New tests.
## Verification
- All Go tests pass: `bash build.sh --test ./internal/parser/parser/...`
and `bash build.sh --test ./internal/ingestion/component/...`
- Build succeeds: `bash build.sh --go`
## Summary
- Load the Agent history window with Python-compatible defaults.
- Include prior conversation messages in ReAct prompts without
duplicating the current user input.
## Testing
- `bash build.sh --test ./internal/agent/...`
- `bash build.sh --test ./internal/service/...`
<img width="2014" height="1121" alt="image"
src="https://github.com/user-attachments/assets/15327e2a-3286-42ec-9415-c06ada758156"
/>
## Summary
Align the Go ingestion pipeline with Python's `image` → `img_id`
persistence semantics, and unify the chunk-id computation across all
paths.
### Changes
**1. Image upload at chunker stage **
- Add `ImageUploader` type and `DefaultImageUploader` in
`internal/ingestion/component/image_uploader.go` — the write-side
counterpart to `FetchBinary`, storing raw image bytes at `(bucket=kbID,
key=chunkID)`, no re-encoding.
- Add `uploadOneImage` — pure upload primitive (bytes in, `img_id` out),
does not touch chunk maps.
- Add `uploadChunkImages` / `uploadChunkImage` — caller-side helper:
decodes `image` from a chunk, uploads bytes, writes `ck["img_id"]`,
`delete(ck,"image")` , bounded by a process-wide semaphore (default 10,
env `MAX_CONCURRENT_MINIO`).
- Wire via `imageUploadDecorator` in `register.go`: every chunker runs
the upload pass at invocation time, writing `ck["id"]` before upload and
dropping image bytes right after — peak memory = single chunk image
lifetime.
**2. Unify chunk-id computation**
- Consolidate three separate id-computation paths (`component.ChunkID`,
`task.ChunkID`, inline `FormatUint` in API) into one:
`common.ChunkID(docID, text string)`, using `%016x` +
`xxhash.Sum64String(text+docID)` (matching Python `hexdigest()`).
- The chunker decorator writes `ck["id"]` via `common.ChunkID`; the
persist stage (`ProcessChunksForPipeline`) falls back to the same
function (`if !exists id`).
- The API AddChunk path now also calls `common.ChunkID` instead of the
divergent `FormatUint(xxhash.Sum64(...))` — fixing a pre-existing
inconsistency.
- Delete `internal/ingestion/component/chunk_id.go` and
`internal/ingestion/task/chunk_builder.go` (both were pure forwarding
shells).
**3. Preserve `img_id` (never deleted)**
- `img_id` is a persistent index field (Infinity, OB) and the only
consumer-side reference for image retrieval; it is NEVER removed from
the chunk map. Only `image` (raw data URL) is dropped after upload.
**4. PPT parser support**
Previously PPT parsing failed. Add support to parse.
### Key design decisions
| Decision | Choice |
|----------|--------|
| Upload timing | Chunker stage (not persist), so image bytes are
dropped immediately — bounds peak memory to one chunk image |
| Upload concurrency | Process-wide semaphore, default 10 (matches
Python `minio_limiter`), env `MAX_CONCURRENT_MINIO` |
| Image encoding | Store as-is, no JPEG re-encoding (unlike Python) |
| `img_id` format | `"<kb_id>-<chunk_id>"` — matches Python
task_executor path |
| id function | Single `common.ChunkID(docID, text)`, concatenation
`text+docID` inside hash (matching Python) |
| `removeInternalChunkFields` | Retains `delete(ck,"image")` as
defensive fallback for non-chunker paths |
### Files touched
| File | Change |
|------|--------|
| `internal/common/format.go` | Add `ChunkID(docID, text)` |
| `internal/common/format_test.go` | Add ChunkID golden-value test |
| `internal/ingestion/component/image_uploader.go` | Add `ImageUploader`
type + `DefaultImageUploader` |
| `internal/ingestion/component/chunker/image_upload.go` | Add
`uploadOneImage`, `uploadChunkImages`, `uploadChunkImage`,
`decodeChunkImage`, semaphore |
| `internal/ingestion/component/chunker/image_upload_test.go` | Tests:
upload/drop, skip, no-image, concurrency, missing-id error |
| `internal/ingestion/component/chunker/register.go` | Add
`imageUploadDecorator` (writes `ck["id"]`, runs upload) |
| `internal/ingestion/task/chunk_process.go` | Use `common.ChunkID` for
persist fallback |
| `internal/service/chunk/chunk.go` | Use `common.ChunkID` instead of
`FormatUint` |
| `internal/ingestion/component/chunk_id.go` | **Deleted** (moved to
`common`) |
| `internal/ingestion/task/chunk_builder.go` | **Deleted** (shell, no
callers left) |
| `internal/ingestion/task/chunk_builder_test.go` | **Deleted** (test
migrated to `common/format_test.go`) |
### Verification
```
bash build.sh --test ./internal/service/chunk/... ./internal/common/... ./internal/ingestion/component/... ./internal/ingestion/task/...
→ ok service/chunk / common / component / chunker / schema / task
```
## Summary
`Pipeline.Run` already accepts an `override_params` map (keyed by
`cpnID`, override-wins) that is merged into each component's params at
compile time via `canvas.WithOverrideParams`. `PatchDSL` performed a
byte-for-byte equivalent merge by baking `ParserConfig` into the DSL
text before compilation, so `runPipelineWithDSL` applied the same
`Doc.ParserConfig` twice with no effect.
## Change
- Delete the now-dead `PatchDSL` function
(`internal/ingestion/pipeline/pipeline.go`); it had a single call site.
- In `runPipelineWithDSL`, compile the DSL unchanged and pass the
extracted component params to `Run` as `override_params`.
## Behavioral note (important)
Previously `PatchDSL` was also the channel that delivered the tenant LLM
id into Extractor components: it baked `parserConfig` — after
`common.InjectExtractorLLMID(parserConfig, Tenant.LLMID)` had injected
`llm_id` — into the DSL. The old `Run` override_params re-cast
`Doc.ParserConfig` (which intentionally omits `llm_id`). So removing
`PatchDSL` while keeping that re-cast would have silently dropped LLM id
injection. This change passes the local `parserConfig` (the
LLMID-injected copy) to `Run`, preserving the prior behavior.
The DSL string returned by `runPipelineWithDSL` is only consumed by
`recordPipelineLog` (structure storage) and `ExtractPayload` (which
reads the terminal structure, not params), so returning the original DSL
is equivalent.
## Test plan
- `bash build.sh --test ./internal/ingestion/task/...
./internal/ingestion/pipeline/...` — both packages pass.
- `gofmt` clean; pre-commit hooks pass.
Co-authored-by: Claude <noreply@anthropic.com>
