Files
ragflow/internal/deepdoc/parser/pdf/pdfsync_shared_mutex_test.go

76 lines
2.0 KiB
Go
Raw Normal View History

feat(agent): Go ingestion pipeline progress mirroring and DeepDOC parser hardening (#16795) feat(ingestion): mirror Go pipeline progress into the document table; harden resume guards - pipeline: bind the owning document via WithDocumentID; after each TrackProgress event aggregate ingestion_task_log progress and mirror progress/run/progress_msg back into the document table, so GET /api/v1/datasets/{dataset_id}/documents reflects live Go pipeline progress without a bespoke endpoint. - canvas: extend the S3 resume guard to reject legacy no-op nodes (e.g. ExitLoop) so component_total equals the count of progress-reporting components and the aggregate percent can reach 100%. - runtime/canvas: route progress through TrackProgress; add interrupt test coverage (r3_interrupt_test.go). - dao/entity: add IngestionTask.DocumentID column and AggregateProgress support used by the mirror; IngestionTaskLog keeps a Checkpoint column alongside the progress fields. feat(deepdoc): cache DocAnalyzer inference results in Redis (1h TTL) - Redis-backed DocAnalyzerCache decorator over inference.Client; cache key = "ddoc:cache:<method>:" + sha256 of the JPEG-encoded image bytes (deterministic). - TTL = 1h; hits skip the inner HTTP call and return cached JSON; inner errors are not cached. refactor(deepdoc): align figure cropping with Python cropout + bounded page caches - CropSectionByDLA mirrors Python cropout: best-overlap DLA figure/equation region, fallback to section bbox per page, vertical concat on gray background. - sliding-window page-image cache bounds peak memory to the recent window instead of the whole PDF. - rename DLADebug -> DLARegions across parser/chunker/tests. refactor(parser): drop lib_type selector; align NewXxxParser with NewPDFParser - remove config["lib_type"] lookup and the libType param/field/switch from all nine constructors; surface the CGO-required error at ParseWithResult time instead of construction time; drop resolveLibType, its test, and the four lib_type constants. feat(utility): add a reusable workerpool for bounded concurrent execution - internal/utility/workerpool.go (+ tests). refactor: translate Chinese prose comments to English in non-harness Go files. chore: upgrade github.com/cloudwego/eino from v0.9.9 to v0.9.12.
2026-07-10 10:36:10 +08:00
//go:build cgo && manual
package pdf
import (
"os"
"path/filepath"
"sync"
"testing"
"ragflow/internal/deepdoc/parser/pdf/pdfium"
"ragflow/internal/deepdoc/parser/pdf/pdfoxide"
)
// TestPdfsyncSharedMutexAcrossBindings hammers both the cgo pdfium binding and
// the Rust pdf_oxide binding concurrently on the same document. Both bindings
// link a single PDFium instance (build.sh collapses the duplicate FPDF_*
// symbols with --allow-multiple-definition), so each binding must serialize
// through the SAME process-wide pdfsync.Mu. Two independent mutexes would
// still interleave calls onto the shared PDFium heap and SIGSEGV. The test
// therefore only passes if it completes without crashing — it directly
// exercises the shared-mutex invariant described in pdfsync/pdfsync.go.
func TestPdfsyncSharedMutexAcrossBindings(t *testing.T) {
pdfPath := filepath.Join("testdata", "pdfs", "01_english_simple.pdf")
data, err := os.ReadFile(pdfPath)
if err != nil {
t.Fatal(err)
}
const goroutines = 16
const iterations = 3
var wg sync.WaitGroup
for i := 0; i < goroutines; i++ {
wg.Add(1)
go func(id int) {
defer wg.Done()
usePdfium := id%2 == 0
for j := 0; j < iterations; j++ {
if usePdfium {
if _, _, err := pdfium.PageSize(data, 0); err != nil {
t.Errorf("pdfium.PageSize: %v", err)
return
}
img, err := pdfium.RenderPage(data, 0, 72)
if err != nil {
t.Errorf("pdfium.RenderPage: %v", err)
return
}
if img.Bounds().Dx() <= 0 {
t.Error("pdfium.RenderPage returned zero-width image")
return
}
} else {
doc, err := pdfoxide.OpenBytes(data)
if err != nil {
t.Errorf("pdfoxide.OpenBytes: %v", err)
return
}
res, err := doc.RenderPage(0, 72)
doc.Close()
if err != nil {
t.Errorf("pdfoxide.RenderPage: %v", err)
return
}
if res.Width <= 0 {
t.Error("pdfoxide.RenderPage returned zero-width image")
return
}
}
}
}(i)
}
wg.Wait()
}