## Summary
Align the Go ingestion pipeline with Python's `image` → `img_id`
persistence semantics, and unify the chunk-id computation across all
paths.
### Changes
**1. Image upload at chunker stage **
- Add `ImageUploader` type and `DefaultImageUploader` in
`internal/ingestion/component/image_uploader.go` — the write-side
counterpart to `FetchBinary`, storing raw image bytes at `(bucket=kbID,
key=chunkID)`, no re-encoding.
- Add `uploadOneImage` — pure upload primitive (bytes in, `img_id` out),
does not touch chunk maps.
- Add `uploadChunkImages` / `uploadChunkImage` — caller-side helper:
decodes `image` from a chunk, uploads bytes, writes `ck["img_id"]`,
`delete(ck,"image")` , bounded by a process-wide semaphore (default 10,
env `MAX_CONCURRENT_MINIO`).
- Wire via `imageUploadDecorator` in `register.go`: every chunker runs
the upload pass at invocation time, writing `ck["id"]` before upload and
dropping image bytes right after — peak memory = single chunk image
lifetime.
**2. Unify chunk-id computation**
- Consolidate three separate id-computation paths (`component.ChunkID`,
`task.ChunkID`, inline `FormatUint` in API) into one:
`common.ChunkID(docID, text string)`, using `%016x` +
`xxhash.Sum64String(text+docID)` (matching Python `hexdigest()`).
- The chunker decorator writes `ck["id"]` via `common.ChunkID`; the
persist stage (`ProcessChunksForPipeline`) falls back to the same
function (`if !exists id`).
- The API AddChunk path now also calls `common.ChunkID` instead of the
divergent `FormatUint(xxhash.Sum64(...))` — fixing a pre-existing
inconsistency.
- Delete `internal/ingestion/component/chunk_id.go` and
`internal/ingestion/task/chunk_builder.go` (both were pure forwarding
shells).
**3. Preserve `img_id` (never deleted)**
- `img_id` is a persistent index field (Infinity, OB) and the only
consumer-side reference for image retrieval; it is NEVER removed from
the chunk map. Only `image` (raw data URL) is dropped after upload.
**4. PPT parser support**
Previously PPT parsing failed. Add support to parse.
### Key design decisions
| Decision | Choice |
|----------|--------|
| Upload timing | Chunker stage (not persist), so image bytes are
dropped immediately — bounds peak memory to one chunk image |
| Upload concurrency | Process-wide semaphore, default 10 (matches
Python `minio_limiter`), env `MAX_CONCURRENT_MINIO` |
| Image encoding | Store as-is, no JPEG re-encoding (unlike Python) |
| `img_id` format | `"<kb_id>-<chunk_id>"` — matches Python
task_executor path |
| id function | Single `common.ChunkID(docID, text)`, concatenation
`text+docID` inside hash (matching Python) |
| `removeInternalChunkFields` | Retains `delete(ck,"image")` as
defensive fallback for non-chunker paths |
### Files touched
| File | Change |
|------|--------|
| `internal/common/format.go` | Add `ChunkID(docID, text)` |
| `internal/common/format_test.go` | Add ChunkID golden-value test |
| `internal/ingestion/component/image_uploader.go` | Add `ImageUploader`
type + `DefaultImageUploader` |
| `internal/ingestion/component/chunker/image_upload.go` | Add
`uploadOneImage`, `uploadChunkImages`, `uploadChunkImage`,
`decodeChunkImage`, semaphore |
| `internal/ingestion/component/chunker/image_upload_test.go` | Tests:
upload/drop, skip, no-image, concurrency, missing-id error |
| `internal/ingestion/component/chunker/register.go` | Add
`imageUploadDecorator` (writes `ck["id"]`, runs upload) |
| `internal/ingestion/task/chunk_process.go` | Use `common.ChunkID` for
persist fallback |
| `internal/service/chunk/chunk.go` | Use `common.ChunkID` instead of
`FormatUint` |
| `internal/ingestion/component/chunk_id.go` | **Deleted** (moved to
`common`) |
| `internal/ingestion/task/chunk_builder.go` | **Deleted** (shell, no
callers left) |
| `internal/ingestion/task/chunk_builder_test.go` | **Deleted** (test
migrated to `common/format_test.go`) |
### Verification
```
bash build.sh --test ./internal/service/chunk/... ./internal/common/... ./internal/ingestion/component/... ./internal/ingestion/task/...
→ ok service/chunk / common / component / chunker / schema / task
```
### Summary
1. list and get by id API for builtin DSL
2. add DSL default component param values validation
3. remove all hard code keys for parser config
### Summary
```
RAGFlow(api/default)> CHAT WITH 'glm-4-flash@new_test@zhipu-ai' MESSAGE '30 words describes LLM';
Answer: Hello! I'm ChatGLM, an AI assistant. Feel free to ask me any questions or request help with any tasks.
Input tokens: 5
Output tokens: 28
Time: 12.748241
```
Signed-off-by: Jin Hai <haijin.chn@gmail.com>
## Summary
- Preserve request-scoped system variables such as files and user IDs
during Canvas execution.
- Persist conversation history, turn counts, and tool memory in the
session DSL across turns.
- Parse agent uploads into `sys.files` and align system variable
rendering with Python.
## Testing
- `bash build.sh --test ./internal/agent/...`
- `bash build.sh --test ./internal/service/...`
<img width="1896" height="1232" alt="image"
src="https://github.com/user-attachments/assets/b420cd97-53c3-470f-a3e1-d39cea26a213"
/>
### What problem does this PR solve?
Await Response incorrectly consumed the initial `sys.query` as its
input, so the first user interaction was skipped.
This change makes Await Response wait for actual user input while
preserving the existing initial-query behavior for the Begin node.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
### Summary
1. Remove dead code (replaced by builtin ingestion pipeline)
2. Refactor (move document parsing progress from http api into ingestion
executor)
### Summary
In Go and python implementation, the dataset / KB id isn't validated if
it is accessible by this user.
---------
Signed-off-by: Jin Hai <haijin.chn@gmail.com>
### Summary
```
RAGFlow(admin)> show version;
+--------------+-----------------------+
| field | value |
+--------------+-----------------------+
| version | v0.26.4-84-g547bc8614 |
| version_type | open source |
+--------------+-----------------------+
```
---------
Signed-off-by: Jin Hai <haijin.chn@gmail.com>
### Summary
1. update docker compose file to start NATS healthy
2. Add two commands
```
RAGFlow(admin)> live;
SUCCESS
RAGFlow(admin)> health;
+---------------+-------+
| field | value |
+---------------+-------+
| storage | ok |
| message_queue | ok |
| status | ok |
| db | ok |
| redis | ok |
| doc_engine | ok |
+---------------+-------+
```
---------
Signed-off-by: Jin Hai <haijin.chn@gmail.com>
feat(ingestion): mirror Go pipeline progress into the document table;
harden resume guards
- pipeline: bind the owning document via WithDocumentID; after each
TrackProgress event aggregate ingestion_task_log progress and mirror
progress/run/progress_msg back into the document table, so GET
/api/v1/datasets/{dataset_id}/documents reflects live Go pipeline
progress without a bespoke endpoint.
- canvas: extend the S3 resume guard to reject legacy no-op nodes (e.g.
ExitLoop) so component_total equals the count of progress-reporting
components and the aggregate percent can reach 100%.
- runtime/canvas: route progress through TrackProgress; add interrupt
test coverage (r3_interrupt_test.go).
- dao/entity: add IngestionTask.DocumentID column and AggregateProgress
support used by the mirror; IngestionTaskLog keeps a Checkpoint column
alongside the progress fields.
feat(deepdoc): cache DocAnalyzer inference results in Redis (1h TTL)
- Redis-backed DocAnalyzerCache decorator over inference.Client; cache
key = "ddoc:cache:<method>:" + sha256 of the JPEG-encoded image bytes
(deterministic).
- TTL = 1h; hits skip the inner HTTP call and return cached JSON; inner
errors are not cached.
refactor(deepdoc): align figure cropping with Python cropout + bounded
page caches
- CropSectionByDLA mirrors Python cropout: best-overlap DLA
figure/equation region, fallback to section bbox per page, vertical
concat on gray background.
- sliding-window page-image cache bounds peak memory to the recent
window instead of the whole PDF.
- rename DLADebug -> DLARegions across parser/chunker/tests.
refactor(parser): drop lib_type selector; align NewXxxParser with
NewPDFParser
- remove config["lib_type"] lookup and the libType param/field/switch
from all nine constructors; surface the CGO-required error at
ParseWithResult time instead of construction time; drop resolveLibType,
its test, and the four lib_type constants.
feat(utility): add a reusable workerpool for bounded concurrent
execution
- internal/utility/workerpool.go (+ tests).
refactor: translate Chinese prose comments to English in non-harness Go
files.
chore: upgrade github.com/cloudwego/eino from v0.9.9 to v0.9.12.
### Summary
Handle searching dataset without embedding model
In this PR, Searching datasets with different embedding models or
searching dataset with/without embedding models are not allowed. We will
improve the behavior later.
