mirror of
https://github.com/infiniflow/ragflow.git
synced 2026-07-25 01:43:27 +08:00
## Summary Adds page-range parsing support to the Go-native pipeline path and introduces strict `parse_type` validation for both dataset and document update endpoints. ## What changed ### Pages range parsing - **`internal/utility/pdf_pages.go`** — `NormalizePDFPages`: normalizes raw page ranges (list of `[from,to]` 1-indexed inclusive ranges) into sorted, merged, deduplicated `[][]int`. Invalid ranges are dropped. - **`internal/ingestion/pipeline/pdf_pages.go`** — `NormalizeParserConfigPages`: walks any parser_config map and normalizes `"pages"` values under every component → filetype setup, so the persisted config always carries clean, merged ranges. - **`internal/deepdoc/parser/pdf/parser.go`** — integrates `resolvePagesToProcess` to filter parsed PDF pages by the configured ranges. - Pipeline integration (parser pages): `internal/parser/parser/pdf_parser_common.go`, `chunk_process.go`, plus associated e2e and unit tests. ### Parse type validation (shared logic) - **`internal/service/parser_mode.go`** (new) — `ValidateParseTypeMode`: shared function that validates `parse_type` (1=BuiltIn/parser_id, 2=Pipeline/pipeline_id) and ensures the corresponding field is present. Used by both dataset and document update endpoints. - **`internal/service/dataset/crud.go`** / `update.go` — replaces inline `isPipelineMode`/`isBuiltinMode` computation with the shared `service.ValidateParseTypeMode`. - **`internal/service/document/document_dataset_update.go`** — adds strict `parse_type` validation in `validateDatasetDocumentUpdate`, simplifies the reparse logic to a two-way switch (isBuiltin/isPipeline) now that parse_type is always valid. - **`internal/service/document/document.go`** — adds `ParseType` field to `UpdateDatasetDocumentRequest`. - **`internal/service/document/document_dataset_update.go`** — `updateDocumentParserConfig` fallback path when DSL loading fails. - **`internal/service/parser_mode_test.go`** (new) — test coverage for nil, invalid, and missing-field scenarios. ### Frontend - **`web/src/interfaces/request/document.ts`** — adds `parseType` to `IChangeParserRequestBody`. - **`web/src/hooks/use-document-request.ts`** — `useSetDocumentPipelineParser` sends `parse_type` in the PATCH payload. - **`web/src/pages/dataset/dataset/use-change-document-parser.ts`** — Go/Python branching for the document parser config dialog. - **`web/src/components/document-pipeline-dialog/use-document-pipeline-form.ts`** — `buildSubmitData` returns `parseType` (bugfix: was dropped from the return value). ### Test changes - **Removed**: 2 tests that verified the old "mutually exclusive" error (replaced by `ValidateParseTypeMode` coverage). - **Modified**: 6 tests across document and dataset packages to include `ParseType` in request structs. - **Added**: new e2e tests for pages parsing (`pages_e2e_test.go`, `pdf_parser_pages_e2e_test.go`) and unit tests for `NormalizePDFPages`, `NormalizeParserConfigPages`, `resolvePagesToProcess`. ## Backward compatibility - The `parse_type` field is **required** when `parser_id` or `pipeline_id` is sent. This changes the contract for both dataset and document PATCH endpoints, but aligns the Go backend with the existing frontend behavior (the frontend already sends `parse_type`). Callers that omit `parse_type` when updating parser/pipeline selections will receive a clear error message. - Existing callers that only update fields like `name`, `enabled`, or `meta_fields` are unaffected. - Test updates ensure all known call sites are compliant.
56 lines
1.8 KiB
Go
56 lines
1.8 KiB
Go
//go:build cgo
|
|
|
|
package parser
|
|
|
|
import (
|
|
"context"
|
|
"errors"
|
|
"fmt"
|
|
"ragflow/internal/common"
|
|
|
|
deepdocpdf "ragflow/internal/deepdoc/parser/pdf"
|
|
deepdoctype "ragflow/internal/deepdoc/parser/type"
|
|
)
|
|
|
|
func (p *PDFParser) ParseWithResult(ctx context.Context, filename string, data []byte) ParseResult {
|
|
if err := p.validateParseMethod(); err != nil {
|
|
return ParseResult{Err: err}
|
|
}
|
|
common.Info(fmt.Sprintf("------------file: %s, parse_method: %s", filename, p.ParseMethod))
|
|
switch normalizePDFParseMethod(p.ParseMethod) {
|
|
case "plain_text":
|
|
return parsePDFWithPlainText(filename, data, p)
|
|
case "mineru":
|
|
return parsePDFWithMinerU(ctx, filename, data, p)
|
|
case "paddleocr":
|
|
return parsePDFWithPaddleOCR(ctx, filename, data, p)
|
|
case "docling":
|
|
return parsePDFWithDocling(ctx, filename, data, p)
|
|
case "opendataloader":
|
|
return parsePDFWithOpenDataLoader(ctx, filename, data, p)
|
|
case "somark":
|
|
return parsePDFWithSoMark(filename, data, p)
|
|
case "tcadp":
|
|
return parsePDFWithTCADP(filename, data, p)
|
|
}
|
|
cfg := deepdoctype.DefaultParserConfig()
|
|
cfg.SkipOCR = false
|
|
cfg.Pages = p.Pages
|
|
parser := deepdocpdf.NewParser(cfg)
|
|
res := parsePDFWithDeepDocOptions(ctx, filename, data, pdfPostProcessOptions{
|
|
outputFormat: p.OutputFormat,
|
|
zoom: cfg.Zoom,
|
|
enableMultiColumn: p.EnableMultiColumn,
|
|
flattenMediaToText: p.FlattenMediaToText,
|
|
removeTOC: p.RemoveTOC,
|
|
removeHeaderFooter: p.RemoveHeaderFooter,
|
|
}, parser.Parse)
|
|
if res.Err != nil && errors.Is(res.Err, deepdocpdf.ErrNoPDFData) {
|
|
return ParseResult{Err: fmt.Errorf("%w: %s", ErrPDFEngineUnavailable, filename)}
|
|
}
|
|
if res.Err != nil && res.Err.Error() == "deepdoc/pdf: cgo required" {
|
|
return ParseResult{Err: fmt.Errorf("%w: %s", ErrPDFEngineUnavailable, filename)}
|
|
}
|
|
return res
|
|
}
|