mirror of
https://github.com/infiniflow/ragflow.git
synced 2026-07-25 09:53:29 +08:00
bb6b43b5c9
## Summary Three groups of changes across the Go ingestion pipeline: ### 1. DOCX parsing improvements - **docx_parser.go**: Enhanced DOCX parsing with better structure extraction and media handling - **docx_parser_cgo_test.go**, **docx_parser_test.go**: Companion tests - **office_parsers_no_cgo.go**: Stub sync for non-CGO builds ### 2. Email (.eml) parsing: base64 Content-Transfer-Encoding decoding - **email_parser.go** (`decodeCTE`): Added Content-Transfer-Encoding decoding for base64 and quoted-printable. Go's `mime/multipart.Reader` does not decode Content-Transfer-Encoding automatically, so attachments with `Content-Transfer-Encoding: base64` remained base64-encoded in the output. The new `decodeCTE` helper is called after reading each multipart part's raw bytes in `readMailBody`, mirroring Python's `part.get_payload(decode=True)`. - **email_parser_test.go**: Two new tests — simple base64 attachment and nested multipart/alternative with base64 attachment. ### 3. Extractor LLM driver fix + ModelDriver consolidation - **extractor.go**: Fixed a bug where the Extractor component used `ModelFactory.CreateModelDriver()`, which creates bare model instances without API keys or provider configuration. Switched to `models.GetPreconfiguredDriver()` which resolves the actual pre-configured driver from `ProviderManager`, matching the codepath used by `llm.go`. This fixes auto keyword/question extraction in DSL pipelines that require LLM calls. - **get_driver.go** (new): Extracted shared `GetPreconfiguredDriver()` from `llm.go:newChatModelDriver()` so both `llm.go` and `extractor.go` use the same codepath. - **get_driver_test.go** (new): Tests for the shared driver resolution. - **llm.go**: Replaced inline driver resolution with `models.GetPreconfiguredDriver()`. ### 4. Chunker fixes and observability - **group.go** (`extractLineRecords`): Fixed to also read `markdown` and `html` payload keys — previously it only read `text`/`content`, causing GroupTitleChunker to silently return empty results for markdown-format parser output. - **common.go** (`compileDelimPattern`): Aligned with Python's `_compile_delimiter_pattern` — only backtick-wrapped delimiters produce an active regex pattern; plain delimiters are not compiled into the split regex. - **token.go** (`applyChildrenDelim`): Set `DocType` and `CKType` to `"text"` on created ChunkDocs so the token-size merge path correctly identifies and merges text segments. - **parser.go**, **extractor.go**, **tokenizer.go**, **group.go**, **hierarchy.go**: Added debug-level logging for pipeline diagnostics. - **parser_dispatch_test.go**, **group_test.go**: New tests. ## Verification - All Go tests pass: `bash build.sh --test ./internal/parser/parser/...` and `bash build.sh --test ./internal/ingestion/component/...` - Build succeeds: `bash build.sh --go`
92 lines
2.2 KiB
Go
92 lines
2.2 KiB
Go
//go:build !cgo
|
|
|
|
package parser
|
|
|
|
import (
|
|
"errors"
|
|
"fmt"
|
|
)
|
|
|
|
// ErrOfficeCGORequired is returned by ParseWithResult on every
|
|
// office-parser family (DOC / DOCX / PPT / PPTX)
|
|
// when the build is not CGO-enabled. The CGO build's
|
|
// implementation captures the office_oxide PlainText / ToMarkdown
|
|
// output; this stub mirrors that surface so the package compiles
|
|
// and existing tests pass. The error is surfaced at parse time
|
|
// rather than at construction time, matching the NewPDFParser
|
|
// shape used by the rest of the package.
|
|
var ErrOfficeCGORequired = errors.New("parser: office family requires CGO (office_oxide)")
|
|
|
|
func (p *DOCXParser) ParseWithResult(filename string, _ []byte) ParseResult {
|
|
return ParseResult{
|
|
File: map[string]any{"name": filename},
|
|
Err: fmt.Errorf("%w: docx", ErrOfficeCGORequired),
|
|
}
|
|
}
|
|
|
|
func (p *DOCParser) ParseWithResult(filename string, _ []byte) ParseResult {
|
|
return ParseResult{
|
|
File: map[string]any{"name": filename},
|
|
Err: fmt.Errorf("%w: doc", ErrOfficeCGORequired),
|
|
}
|
|
}
|
|
|
|
func (p *PPTParser) ParseWithResult(filename string, _ []byte) ParseResult {
|
|
return ParseResult{
|
|
File: map[string]any{"name": filename},
|
|
Err: fmt.Errorf("%w: ppt", ErrOfficeCGORequired),
|
|
}
|
|
}
|
|
|
|
func (p *PPTXParser) ParseWithResult(filename string, _ []byte) ParseResult {
|
|
return ParseResult{
|
|
File: map[string]any{"name": filename},
|
|
Err: fmt.Errorf("%w: pptx", ErrOfficeCGORequired),
|
|
}
|
|
}
|
|
|
|
type DOCParser struct{}
|
|
|
|
func NewDOCParser() *DOCParser {
|
|
return &DOCParser{}
|
|
}
|
|
|
|
func (p *DOCParser) String() string {
|
|
return "DOCParser(no-cgo)"
|
|
}
|
|
|
|
type DOCXParser struct{}
|
|
|
|
func NewDOCXParser() *DOCXParser {
|
|
return &DOCXParser{}
|
|
}
|
|
|
|
func (p *DOCXParser) ConfigureFromSetup(setup map[string]any) {
|
|
// No-op in the no-CGO stub: the real implementation in
|
|
// docx_parser.go reads output_format from setup.
|
|
}
|
|
|
|
func (p *DOCXParser) String() string {
|
|
return "DOCXParser(no-cgo)"
|
|
}
|
|
|
|
type PPTParser struct{}
|
|
|
|
func NewPPTParser() *PPTParser {
|
|
return &PPTParser{}
|
|
}
|
|
|
|
func (p *PPTParser) String() string {
|
|
return "PPTParser(no-cgo)"
|
|
}
|
|
|
|
type PPTXParser struct{}
|
|
|
|
func NewPPTXParser() *PPTXParser {
|
|
return &PPTXParser{}
|
|
}
|
|
|
|
func (p *PPTXParser) String() string {
|
|
return "PPTXParser(no-cgo)"
|
|
}
|