mirror of
https://github.com/infiniflow/ragflow.git
synced 2026-07-11 14:15:40 +08:00
feat(ingestion): mirror Go pipeline progress into the document table;
harden resume guards
- pipeline: bind the owning document via WithDocumentID; after each
TrackProgress event aggregate ingestion_task_log progress and mirror
progress/run/progress_msg back into the document table, so GET
/api/v1/datasets/{dataset_id}/documents reflects live Go pipeline
progress without a bespoke endpoint.
- canvas: extend the S3 resume guard to reject legacy no-op nodes (e.g.
ExitLoop) so component_total equals the count of progress-reporting
components and the aggregate percent can reach 100%.
- runtime/canvas: route progress through TrackProgress; add interrupt
test coverage (r3_interrupt_test.go).
- dao/entity: add IngestionTask.DocumentID column and AggregateProgress
support used by the mirror; IngestionTaskLog keeps a Checkpoint column
alongside the progress fields.
feat(deepdoc): cache DocAnalyzer inference results in Redis (1h TTL)
- Redis-backed DocAnalyzerCache decorator over inference.Client; cache
key = "ddoc:cache:<method>:" + sha256 of the JPEG-encoded image bytes
(deterministic).
- TTL = 1h; hits skip the inner HTTP call and return cached JSON; inner
errors are not cached.
refactor(deepdoc): align figure cropping with Python cropout + bounded
page caches
- CropSectionByDLA mirrors Python cropout: best-overlap DLA
figure/equation region, fallback to section bbox per page, vertical
concat on gray background.
- sliding-window page-image cache bounds peak memory to the recent
window instead of the whole PDF.
- rename DLADebug -> DLARegions across parser/chunker/tests.
refactor(parser): drop lib_type selector; align NewXxxParser with
NewPDFParser
- remove config["lib_type"] lookup and the libType param/field/switch
from all nine constructors; surface the CGO-required error at
ParseWithResult time instead of construction time; drop resolveLibType,
its test, and the four lib_type constants.
feat(utility): add a reusable workerpool for bounded concurrent
execution
- internal/utility/workerpool.go (+ tests).
refactor: translate Chinese prose comments to English in non-harness Go
files.
chore: upgrade github.com/cloudwego/eino from v0.9.9 to v0.9.12.
44 lines
1.7 KiB
Go
44 lines
1.7 KiB
Go
//go:build cgo
|
|
|
|
// Package pdfsync provides a single process-wide mutex that serializes ALL
|
|
// access to the PDFium C library, regardless of which Go binding issues the
|
|
// call.
|
|
//
|
|
// Both the cgo pdfium binding (package pdfium) and the Rust pdf_oxide binding
|
|
// (package pdfoxide) are linked into the same final binary against the same
|
|
// PDFium static library. build.sh collapses the duplicate FPDF_* symbols with
|
|
// --allow-multiple-definition, so the process contains exactly ONE PDFium
|
|
// instance. PDFium is documented as NOT thread-safe for any call — concurrent
|
|
// calls corrupt the global heap and crash with SIGSEGV even when the calls
|
|
// operate on different documents.
|
|
//
|
|
// Therefore every native entry point in either binding must hold this one
|
|
// shared lock. Two independent mutexes would give the illusion of safety
|
|
// while still allowing a cgo pdfium call and a Rust pdf_oxide call to
|
|
// interleave onto the same PDFium instance. A single shared mutex is correct
|
|
// in both the one-instance and the (hypothetical, build-breaking) two-instance
|
|
// cases; two mutexes are only correct in the two-instance case.
|
|
package pdfsync
|
|
|
|
import "sync"
|
|
|
|
// Mu serializes all PDFium C API calls across every binding linked into the
|
|
// process. Acquire it (via With / WithErr) around every FPDF_* call and every
|
|
// pdf_oxide Rust call that re-enters the same PDFium instance.
|
|
var Mu sync.Mutex
|
|
|
|
// With runs f while holding the shared PDFium mutex.
|
|
func With(f func()) {
|
|
Mu.Lock()
|
|
defer Mu.Unlock()
|
|
f()
|
|
}
|
|
|
|
// WithErr runs f while holding the shared PDFium mutex and propagates its
|
|
// error unchanged.
|
|
func WithErr(f func() error) error {
|
|
Mu.Lock()
|
|
defer Mu.Unlock()
|
|
return f()
|
|
}
|