4 Commits

Author	SHA1	Message	Date
Jack	d12fd3b79d	feat: parser pages range and parse type validation for dataset/document (#17293) ... ## Summary Adds page-range parsing support to the Go-native pipeline path and introduces strict `parse_type` validation for both dataset and document update endpoints. ## What changed ### Pages range parsing - **`internal/utility/pdf_pages.go`** — `NormalizePDFPages`: normalizes raw page ranges (list of `[from,to]` 1-indexed inclusive ranges) into sorted, merged, deduplicated `[][]int`. Invalid ranges are dropped. - **`internal/ingestion/pipeline/pdf_pages.go`** — `NormalizeParserConfigPages`: walks any parser_config map and normalizes `"pages"` values under every component → filetype setup, so the persisted config always carries clean, merged ranges. - **`internal/deepdoc/parser/pdf/parser.go`** — integrates `resolvePagesToProcess` to filter parsed PDF pages by the configured ranges. - Pipeline integration (parser pages): `internal/parser/parser/pdf_parser_common.go`, `chunk_process.go`, plus associated e2e and unit tests. ### Parse type validation (shared logic) - **`internal/service/parser_mode.go`** (new) — `ValidateParseTypeMode`: shared function that validates `parse_type` (1=BuiltIn/parser_id, 2=Pipeline/pipeline_id) and ensures the corresponding field is present. Used by both dataset and document update endpoints. - **`internal/service/dataset/crud.go`** / `update.go` — replaces inline `isPipelineMode`/`isBuiltinMode` computation with the shared `service.ValidateParseTypeMode`. - **`internal/service/document/document_dataset_update.go`** — adds strict `parse_type` validation in `validateDatasetDocumentUpdate`, simplifies the reparse logic to a two-way switch (isBuiltin/isPipeline) now that parse_type is always valid. - **`internal/service/document/document.go`** — adds `ParseType` field to `UpdateDatasetDocumentRequest`. - **`internal/service/document/document_dataset_update.go`** — `updateDocumentParserConfig` fallback path when DSL loading fails. - **`internal/service/parser_mode_test.go`** (new) — test coverage for nil, invalid, and missing-field scenarios. ### Frontend - **`web/src/interfaces/request/document.ts`** — adds `parseType` to `IChangeParserRequestBody`. - **`web/src/hooks/use-document-request.ts`** — `useSetDocumentPipelineParser` sends `parse_type` in the PATCH payload. - **`web/src/pages/dataset/dataset/use-change-document-parser.ts`** — Go/Python branching for the document parser config dialog. - **`web/src/components/document-pipeline-dialog/use-document-pipeline-form.ts`** — `buildSubmitData` returns `parseType` (bugfix: was dropped from the return value). ### Test changes - **Removed**: 2 tests that verified the old "mutually exclusive" error (replaced by `ValidateParseTypeMode` coverage). - **Modified**: 6 tests across document and dataset packages to include `ParseType` in request structs. - **Added**: new e2e tests for pages parsing (`pages_e2e_test.go`, `pdf_parser_pages_e2e_test.go`) and unit tests for `NormalizePDFPages`, `NormalizeParserConfigPages`, `resolvePagesToProcess`. ## Backward compatibility - The `parse_type` field is **required** when `parser_id` or `pipeline_id` is sent. This changes the contract for both dataset and document PATCH endpoints, but aligns the Go backend with the existing frontend behavior (the frontend already sends `parse_type`). Callers that omit `parse_type` when updating parser/pipeline selections will receive a clear error message. - Existing callers that only update fields like `name`, `enabled`, or `meta_fields` are unaffected. - Test updates ensure all known call sites are compliant.	2026-07-23 19:57:27 +08:00
Jack	8669c469d5	fix(ingestion): align laws DSL with Python — heading fallback, colon-title, short-line filter, remove_toc, and image extension mapping (#17200) ... ## Summary This PR aligns the Go ingestion pipeline's **Laws** DSL template with the Python implementation by fixing heading-detection gaps, adds image-extension support, refactors the **Extractor** component's LLM resolution, hardens heading detection for CJK text, and makes the Extractor accept the Python DSL prompt key names (`sys_prompt`/`prompts`) alongside the Go names. ## Changes ### 1. Picture file-type detection (`internal/utility/file.go`) Adds explicit mapping for common image extensions (png, jpg, jpeg, gif, bmp, tiff, tif, webp, svg, ico, avif, heic, apng) → `FileTypeVISUAL`, with regression tests. ### 2. Laws DSL heading-detection alignment (`internal/ingestion/component/chunker/`) Four fixes to `resolveTitleLevels`: | Fix | What changed | Why | |-----|-------------|-----| | **DOCX `ck_type` fallback** | `ckType` field on `lineRecord`, propagated from `ChunkDoc.CKType` in `recordsFromStructured`. When `ck_type=="heading"`, assign `fallbackLevel`. | office_oxide extracts DOCX heading metadata, but the info was lost before reaching the heading detector. Word headings whose text doesn't match any regex (e.g. "Introduction") were treated as body. | | **`make_colon_as_title` promotion** | `isColonTitle()`: promotes lines ending with `:`/`：` that have sentence-ending punctuation before the colon and ≥32 runes between them. | Mirrors Python's `make_colon_as_title` in `rag/nlp/__init__.py`. Triple guard prevents false positives. | | **Short/numeric line filter** | Lines with ≤1 rune or purely numeric are pinned to body level. | Mirrors Python `tree_merge`'s filter of `sections` where `len(...) <= 1` or `re.match(r"[0-9]+$", ...)`. | | **PDF `remove_toc`** | `"remove_toc": true` added to the PDF parser setup in `ingestion_pipeline_laws.json`. | The Go PDF parser already supports TOC removal; the Book template already enables it. | ### 3. Extractor llm_id resolution (`internal/ingestion/component/extractor.go`) Refactored to handle both **bare tenant_model UUIDs** and **composite model@provider** strings via the shared `resolveModelConfig` (`dispatch_model.go`): - **`resolveExtractorChatConfig`** — UUID path calls `resolveModelConfigByID` directly (one DB hit); composite path goes through `resolveModelConfig`. Added `isBareTenantModelID` pre-check for clear errors when a UUID doesn't exist. - **`resolveExtractorChatTarget`** — propagates resolution errors instead of silently returning empty driver. - **`Chat()`** — removed `driver = "dummy"` fallback. Missing driver is now an explicit error. - **Removed dead code**: `splitExtractorLLID`, `findExtractorSoleActiveInstance`. ### 4. `InjectExtractorLLMID` — fallback when no user config (`internal/common/parser_config.go`) Injects the tenant's global default LLM into extractor components **only when their `llm_id` is empty**. Preserves user-selected UUID or model@provider values. Priority: user-configured llm_id > tenant global default > error (no silent dummy fallback). ### 5. `ResponseHeaderTimeout` increase (`internal/entity/models/base_model.go`) `ResponseHeaderTimeout` 60s → 120s in `NewDriverHTTPClient`. Reasoning models with large extraction prompts can take longer than 60s to produce the first response token. ### 6. CJK rune-aware heading detection (`internal/ingestion/component/chunker/title.go`) Two byte-vs-rune bugs that only manifest on CJK text: | Fix | What changed | Why | |-----|-------------|-----| | **`isColonTitle` byte offset** | `body[lastPunct+1:]` → `body[lastPunct+runeLen:]` via `utf8.DecodeRuneInString` | `strings.LastIndexAny` returns a byte index; `+1` skips only 1 byte, corrupting multi-byte CJK punctuation (e.g. `。` = 3 bytes) and inflating the rune count past the 32-rune threshold → false-positive heading promotion. | | **Short-line filter byte count** | `len(text) <= 1` → `utf8.RuneCountInString(text) <= 1` | Go `len` is UTF-8 bytes; a single CJK char (3 bytes) passed the filter, but Python's `len` returns 1 → mismatch. | ### 7. `extractor_tag.go` — log error when llm fails When `resolveExtractorChatTarget` returned an error, `runAutoTags` will log error. ### 8. Python DSL prompt-key compatibility (`internal/ingestion/component/extractor.go`) The Resume DSL template uses Python-side key names (`sys_prompt`, `prompts`). `NewExtractorComponent` now accepts them as fallbacks alongside the Go names: - `system_prompt` (Go) ← `sys_prompt` (Python) as fallback - `prompt` (Go string) ← `prompts` (Python array `[{"role","content"}]`, takes `[0].content`) as fallback Mirrors the alias pattern already in `internal/agent/component/llm.go`. `resolveInputs` accepts per-call `sys_prompt` override too. ## Remaining gaps vs Python | Gap | Scope | Impact | |-----|-------|--------| | **TOC removal for TXT/MD/HTML** | Python's `remove_contents_table` works on all text formats; Go's `remove_toc` is PDF-only. | Low — plain-text documents rarely contain structured TOCs. | | **Regex pattern details** | Minor differences in quantifiers, missing H5/H6 markdown patterns, missing 4-level numbering pattern. | Low — Go's variants are stricter; DOCX headings are covered by `ck_type` fallback. | ## Testing - `TestHierarchyTitleChunker_CKTypeHeadingFallback` — DOCX `ck_type` heading promotion - `TestHierarchyTitleChunker_ColonTitlePromotion` / `_ColonTitleShortLine_Negative` — colon-title promotion + guard - `TestIsColonTitle_CJKEdgeCase` / `TestIsColonTitle_ASCII_NoRegression` — CJK byte-offset fix + ASCII regression - `TestHierarchyTitleChunker_ColonTitlePromotion_CJK_EdgeCase` — CJK colon edge case through full pipeline - `TestHierarchyTitleChunker_ShortSingleCJKLineFilter` — single CJK char filtered to body - `TestHierarchyTitleChunker_ShortNumericLineFilter` — purely numeric lines filtered - `TestGetFileType_ImageExtensions` / `_ExistingFormats_NoRegression` — image extension mapping - `TestInjectExtractorLLMID_SkipWhenUUID` / `_SkipWhenComposite` / `_InjectWhenEmpty` — llm_id injection guard - `TestIsBareTenantModelID` — UUID detection - `TestResolveExtractorChatTarget_AtSplitFallback` / `_NoDriver` — @ split fallback without DB - `TestNewExtractorComponent_SysPromptAlias` / `_PromptsArray` / `_PromptsArray_PromptWins` / `_SystemPromptWinsOverSysPrompt` — Python key compatibility - `TestBuildDOCXJSONSections_List` / `_TextBox` / `_MixedWithList` — DOCX list/text_box parsing - Full ingestion test suite passes (chunker, pipeline, task, service, component packages)	2026-07-22 19:14:32 +08:00
qinling0210	edeb47cd54	Implement Tagger in Extractor component in GO (#17128) ... ### Summary Implement Tagger in Extractor component in GO	2026-07-21 17:52:06 +08:00
Jack	10c00a9614	Feat: add built in DSL file API (#17003) ... ### Summary 1. list and get by id API for builtin DSL 2. add DSL default component param values validation 3. remove all hard code keys for parser config	2026-07-18 11:00:08 +08:00