feat: parser pages range and parse type validation for dataset/document (#17293)

## Summary

Adds page-range parsing support to the Go-native pipeline path and
introduces strict `parse_type` validation for both dataset and document
update endpoints.

## What changed

### Pages range parsing
- **`internal/utility/pdf_pages.go`** — `NormalizePDFPages`: normalizes
raw page ranges (list of `[from,to]` 1-indexed inclusive ranges) into
sorted, merged, deduplicated `[][]int`. Invalid ranges are dropped.
- **`internal/ingestion/pipeline/pdf_pages.go`** —
`NormalizeParserConfigPages`: walks any parser_config map and normalizes
`"pages"` values under every component → filetype setup, so the
persisted config always carries clean, merged ranges.
- **`internal/deepdoc/parser/pdf/parser.go`** — integrates
`resolvePagesToProcess` to filter parsed PDF pages by the configured
ranges.
- Pipeline integration (parser pages):
`internal/parser/parser/pdf_parser_common.go`, `chunk_process.go`, plus
associated e2e and unit tests.

### Parse type validation (shared logic)
- **`internal/service/parser_mode.go`** (new) — `ValidateParseTypeMode`:
shared function that validates `parse_type` (1=BuiltIn/parser_id,
2=Pipeline/pipeline_id) and ensures the corresponding field is present.
Used by both dataset and document update endpoints.
- **`internal/service/dataset/crud.go`** / `update.go` — replaces inline
`isPipelineMode`/`isBuiltinMode` computation with the shared
`service.ValidateParseTypeMode`.
- **`internal/service/document/document_dataset_update.go`** — adds
strict `parse_type` validation in `validateDatasetDocumentUpdate`,
simplifies the reparse logic to a two-way switch (isBuiltin/isPipeline)
now that parse_type is always valid.
- **`internal/service/document/document.go`** — adds `ParseType` field
to `UpdateDatasetDocumentRequest`.
- **`internal/service/document/document_dataset_update.go`** —
`updateDocumentParserConfig` fallback path when DSL loading fails.
- **`internal/service/parser_mode_test.go`** (new) — test coverage for
nil, invalid, and missing-field scenarios.

### Frontend
- **`web/src/interfaces/request/document.ts`** — adds `parseType` to
`IChangeParserRequestBody`.
- **`web/src/hooks/use-document-request.ts`** —
`useSetDocumentPipelineParser` sends `parse_type` in the PATCH payload.
- **`web/src/pages/dataset/dataset/use-change-document-parser.ts`** —
Go/Python branching for the document parser config dialog.
-
**`web/src/components/document-pipeline-dialog/use-document-pipeline-form.ts`**
— `buildSubmitData` returns `parseType` (bugfix: was dropped from the
return value).

### Test changes
- **Removed**: 2 tests that verified the old "mutually exclusive" error
(replaced by `ValidateParseTypeMode` coverage).
- **Modified**: 6 tests across document and dataset packages to include
`ParseType` in request structs.
- **Added**: new e2e tests for pages parsing (`pages_e2e_test.go`,
`pdf_parser_pages_e2e_test.go`) and unit tests for `NormalizePDFPages`,
`NormalizeParserConfigPages`, `resolvePagesToProcess`.

## Backward compatibility
- The `parse_type` field is **required** when `parser_id` or
`pipeline_id` is sent. This changes the contract for both dataset and
document PATCH endpoints, but aligns the Go backend with the existing
frontend behavior (the frontend already sends `parse_type`). Callers
that omit `parse_type` when updating parser/pipeline selections will
receive a clear error message.
- Existing callers that only update fields like `name`, `enabled`, or
`meta_fields` are unaffected.
- Test updates ensure all known call sites are compliant.
This commit is contained in:
Jack
2026-07-23 19:57:27 +08:00
committed by GitHub
parent fce2a94fe0
commit d12fd3b79d
45 changed files with 1819 additions and 496 deletions

View File

@@ -6,6 +6,7 @@ import (
"image"
"log/slog"
"math"
"sort"
"sync"
lyt "ragflow/internal/deepdoc/parser/pdf/layout"
@@ -108,6 +109,46 @@ func documentPages(pageCount int) []int {
return pages
}
// resolvePagesToProcess converts the 1-indexed inclusive Config.Pages ranges
// into a sorted, de-duplicated slice of 0-indexed page numbers clamped to
// [0, pageCount-1]. Empty/nil ranges fall back to all pages (the historical
// behavior), so callers that leave Pages unset are unaffected.
func resolvePagesToProcess(ranges [][]int, pageCount int) []int {
if len(ranges) == 0 {
return documentPages(pageCount)
}
seen := make(map[int]struct{}, pageCount)
out := make([]int, 0, pageCount)
for _, r := range ranges {
if len(r) != 2 {
continue
}
from0 := r[0] - 1
to0 := r[1] - 1
if from0 < 0 {
from0 = 0
}
if from0 > pageCount-1 {
continue
}
if to0 > pageCount-1 {
to0 = pageCount - 1
}
if to0 < from0 {
continue
}
for pg := from0; pg <= to0; pg++ {
if _, dup := seen[pg]; dup {
continue
}
seen[pg] = struct{}{}
out = append(out, pg)
}
}
sort.Ints(out)
return out
}
// extractOutlines extracts the PDF outlines, returning nil on error.
func (p *Parser) extractOutlines(engine pdf.PDFEngine) []pdf.Outline {
outlines, outlineErr := engine.Outlines()
@@ -497,7 +538,15 @@ func (p *Parser) processPages(ctx context.Context, engine pdf.PDFEngine, docAnal
}
tb := NewTableBuilderFor(docAnalyzer)
pages := documentPages(pageCount)
pages := resolvePagesToProcess(p.Config.Pages, pageCount)
if len(p.Config.Pages) > 0 {
slog.Info("deepdoc pdf parse: page ranges applied",
"configured_ranges", p.Config.Pages,
"page_count", pageCount,
"pages_to_parse", pages)
} else {
slog.Debug("deepdoc pdf parse: parsing all pages", "page_count", pageCount)
}
pageResults, pageErr := p.runPageWorkers(ctx, engine, pages, docAnalyzer, tb)
if pageErr != nil {