Files
ragflow/internal/ingestion/component/chunker/presentation.go
Jack 76aaecc284 fix(ingestion): improve Extractor/LLM robustness and Python->Go parity (#17470)
## Summary

This PR hardens the Go ingestion **Extractor** and **LLM retry** paths
and closes several Python->Go parity gaps in the keyword/question/tag
extraction flow.

- **Generic retry utility** (`internal/common/retry.go`):
`RetryWithBackoff` with exponential backoff (default 3 retries, 2s
initial delay, capped at 1m), context-aware sleep, and a `maxRetries<=0`
fast path. Covered by `internal/common/retry_test.go`.
- **LLM retry reuse**: `agent/component/llm_retry.go` now delegates to
`common.RetryWithBackoff` instead of an inline loop (behavior preserved:
ctx cancellation short-circuits the backoff).
- **Extractor LLM calls** (`extractor.go`):
- `call()` now retries transient LLM failures via `RetryWithBackoff`
(retry exhaustion fails the chunk instead of silently skipping).
  - Sets `temperature = 0.2`, matching Python `generator.py:230,245`.
- Runs keyword and question extraction **concurrently** per chunk when
both are enabled (`task_executor.py:444-448`), with mutex-guarded map
writes to avoid data races.
- Substitutes `{field_name}` placeholders (including `{chunks}` -> chunk
text) in `prompt`/`system_prompt` before the call, mirroring Python
`string_format` (`extractor.py:102-103`); unmatched placeholders are
left as-is.
- Falls back to the **tenant default chat model** when `llm_id` is empty
(`task_executor.py:573-574`).
- Strips `` **greedily** (`strings.LastIndex`) in
`cleanExtractionResult`.
- **Auto-tagging** (`extractor_tag.go`): drops the `in.llmID != ""`
guards so an empty `llm_id` no longer skips tagging (uses the tenant
default model), and strips `` greedily in `parseTaggerResponse`.
- **Docs**: fixes a misleading `PresentationChunker` docstring that
claimed per-slide `image`/`position` output (the PPTX path emits none —
unlike PDF), and removes a stale `docs/migration_python_go_diff.md`
reference in `media_dispatch.go`.

## Test plan

- `bash build.sh --test ./internal/common/...` — passes (new retry
utility + tests).
- `bash build.sh --test ./internal/ingestion/component/...` — passes
(extractor/chunker/schema).
- `gofmt` and lefthook pre-commit checks pass.

Note: the personal `docs/migration_python_go_diff.md` working notebook
in the tree is intentionally **not** part of this PR.

---------

Co-authored-by: CodeBuddy <noreply@codebuddy.ai>
2026-07-28 19:22:18 +08:00

113 lines
3.7 KiB
Go

//
// Copyright 2026 The InfiniFlow Authors. All Rights Reserved.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
//
// PresentationChunker emits one chunk per upstream slide or page.
// It is the faithful Go port of the Python `presentation` chunk method
// (rag/app/presentation.py), whose docstring states: "Every page will
// be treated as a chunk."
//
// Unlike TokenChunker (which merges slides into a single chunk) or
// OneChunker (which collapses many slides into one), PresentationChunker
// keeps each slide as the unit of chunking. The upstream parser produces
// one record per slide with text and slide_number; this chunker passes
// each through unchanged. Note that the PPTX/PPT path does not emit image
// or position information (unlike the PDF path), so slide chunks carry no
// image and no bbox-based preview positioning.
package chunker
import (
"context"
"fmt"
"ragflow/internal/agent/runtime"
"ragflow/internal/ingestion/component/schema"
"gorm.io/gorm"
)
const ComponentNamePresentationChunker = "PresentationChunker"
type presentationChunkerParam struct{}
func (p *presentationChunkerParam) Update(conf map[string]any) {}
func (presentationChunkerParam) Defaults() presentationChunkerParam {
return presentationChunkerParam{}
}
func (presentationChunkerParam) Validate() error { return nil }
type PresentationChunkerComponent struct {
name string
param presentationChunkerParam
}
func NewPresentationChunker(params map[string]any) (runtime.Component, error) {
p := presentationChunkerParam{}.Defaults()
(&p).Update(params)
if err := p.Validate(); err != nil {
return nil, err
}
return &PresentationChunkerComponent{
name: ComponentNamePresentationChunker,
param: p,
}, nil
}
func (c *PresentationChunkerComponent) Inputs() map[string]string { return ChunkerInputs }
func (c *PresentationChunkerComponent) Outputs() map[string]string { return ChunkerOutputs }
func (c *PresentationChunkerComponent) Invoke(ctx context.Context, db *gorm.DB, inputs map[string]any) (map[string]any, error) {
return c.invoke(ctx, inputs)
}
func (c *PresentationChunkerComponent) invoke(_ context.Context, inputs map[string]any) (map[string]any, error) {
if inputs == nil {
return emptyOutputs(), nil
}
upstream, err := decodeChunkerFromUpstream(inputs)
if err != nil {
return map[string]any{
"output_format": "chunks",
"chunks": []map[string]any{},
"_ERROR": fmt.Sprintf("Input error: %v", err),
}, nil
}
// The presentation template only configures pdf and slides
// parser setups, matching Python's restriction to .pptx/.ppt/.pdf.
// The upstream parser therefore always emits per-slide JSON, never
// flat text/markdown/html, so every payload keeps one-chunk-per-slide.
items := slideItems(upstream.JSONResult, upstream.Chunks)
if len(items) == 0 {
return emptyOutputs(), nil
}
return chunkOutputs(items), nil
}
// slideItems returns the per-slide records, preferring JSONResult and
// falling back to Chunks. Each record (slide/page) becomes one chunk.
func slideItems(items, chunks []schema.ChunkDoc) []schema.ChunkDoc {
if len(items) > 0 {
return items
}
return chunks
}
func init() {
MustRegisterChunker(ComponentNamePresentationChunker)
}