Files
ragflow/internal/ingestion/component/image_uploader.go
Jack 2f7c2eb53c Feat(ingestion): align image to MinIO upload, unify chunk-id computation and add PPT parsing support (#17111)
## Summary

Align the Go ingestion pipeline with Python's `image` → `img_id`
persistence semantics, and unify the chunk-id computation across all
paths.

### Changes

**1. Image upload at chunker stage **

- Add `ImageUploader` type and `DefaultImageUploader` in
`internal/ingestion/component/image_uploader.go` — the write-side
counterpart to `FetchBinary`, storing raw image bytes at `(bucket=kbID,
key=chunkID)`, no re-encoding.
- Add `uploadOneImage` — pure upload primitive (bytes in, `img_id` out),
does not touch chunk maps.
- Add `uploadChunkImages` / `uploadChunkImage` — caller-side helper:
decodes `image` from a chunk, uploads bytes, writes `ck["img_id"]`,
`delete(ck,"image")` , bounded by a process-wide semaphore (default 10,
env `MAX_CONCURRENT_MINIO`).
- Wire via `imageUploadDecorator` in `register.go`: every chunker runs
the upload pass at invocation time, writing `ck["id"]` before upload and
dropping image bytes right after — peak memory = single chunk image
lifetime.

**2. Unify chunk-id computation**

- Consolidate three separate id-computation paths (`component.ChunkID`,
`task.ChunkID`, inline `FormatUint` in API) into one:
`common.ChunkID(docID, text string)`, using `%016x` +
`xxhash.Sum64String(text+docID)` (matching Python `hexdigest()`).
- The chunker decorator writes `ck["id"]` via `common.ChunkID`; the
persist stage (`ProcessChunksForPipeline`) falls back to the same
function (`if !exists id`).
- The API AddChunk path now also calls `common.ChunkID` instead of the
divergent `FormatUint(xxhash.Sum64(...))` — fixing a pre-existing
inconsistency.
- Delete `internal/ingestion/component/chunk_id.go` and
`internal/ingestion/task/chunk_builder.go` (both were pure forwarding
shells).

**3. Preserve `img_id` (never deleted)**

- `img_id` is a persistent index field (Infinity, OB) and the only
consumer-side reference for image retrieval; it is NEVER removed from
the chunk map. Only `image` (raw data URL) is dropped after upload.

**4. PPT parser support**
Previously PPT parsing failed. Add support to parse.

### Key design decisions

| Decision | Choice |
|----------|--------|
| Upload timing | Chunker stage (not persist), so image bytes are
dropped immediately — bounds peak memory to one chunk image |
| Upload concurrency | Process-wide semaphore, default 10 (matches
Python `minio_limiter`), env `MAX_CONCURRENT_MINIO` |
| Image encoding | Store as-is, no JPEG re-encoding (unlike Python) |
| `img_id` format | `"<kb_id>-<chunk_id>"` — matches Python
task_executor path |
| id function | Single `common.ChunkID(docID, text)`, concatenation
`text+docID` inside hash (matching Python) |
| `removeInternalChunkFields` | Retains `delete(ck,"image")` as
defensive fallback for non-chunker paths |

### Files touched

| File | Change |
|------|--------|
| `internal/common/format.go` | Add `ChunkID(docID, text)` |
| `internal/common/format_test.go` | Add ChunkID golden-value test |
| `internal/ingestion/component/image_uploader.go` | Add `ImageUploader`
type + `DefaultImageUploader` |
| `internal/ingestion/component/chunker/image_upload.go` | Add
`uploadOneImage`, `uploadChunkImages`, `uploadChunkImage`,
`decodeChunkImage`, semaphore |
| `internal/ingestion/component/chunker/image_upload_test.go` | Tests:
upload/drop, skip, no-image, concurrency, missing-id error |
| `internal/ingestion/component/chunker/register.go` | Add
`imageUploadDecorator` (writes `ck["id"]`, runs upload) |
| `internal/ingestion/task/chunk_process.go` | Use `common.ChunkID` for
persist fallback |
| `internal/service/chunk/chunk.go` | Use `common.ChunkID` instead of
`FormatUint` |
| `internal/ingestion/component/chunk_id.go` | **Deleted** (moved to
`common`) |
| `internal/ingestion/task/chunk_builder.go` | **Deleted** (shell, no
callers left) |
| `internal/ingestion/task/chunk_builder_test.go` | **Deleted** (test
migrated to `common/format_test.go`) |

### Verification

```
bash build.sh --test ./internal/service/chunk/... ./internal/common/... ./internal/ingestion/component/... ./internal/ingestion/task/...
→ ok   service/chunk / common / component / chunker / schema / task
```
2026-07-20 19:33:51 +08:00

45 lines
1.8 KiB
Go

//
// Copyright 2026 The InfiniFlow Authors. All Rights Reserved.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
//
package component
import (
"context"
"fmt"
)
// ImageUploader stores a chunk's raw image bytes and returns the img_id
// reference to persist in the index. It is the write-side counterpart to
// FetchBinary: injected into the chunker so the chunker can upload cropped /
// extracted images on the spot and drop the bytes, instead of carrying them
// across the component boundary.
type ImageUploader func(ctx context.Context, kbID, chunkID string, data []byte) (imgID string, err error)
// DefaultImageUploader stores image bytes at (bucket=kbID, key=chunkID) and
// returns img_id "<kb_id>-<chunk_id>". Mirrors Python image2id's storage_put +
// f"{bucket}-{objname}" (rag/utils/base64_image.py:80-82), minus re-encoding:
// the bytes are stored in whatever format the chunker produced.
func DefaultImageUploader(_ context.Context, kbID, chunkID string, data []byte) (string, error) {
stg := resolveStorage()
if stg == nil {
return "", fmt.Errorf("no storage backend registered")
}
if err := stg.Put(kbID, chunkID, data); err != nil {
return "", fmt.Errorf("store chunk image (%q,%q): %w", kbID, chunkID, err)
}
return kbID + "-" + chunkID, nil
}