mirror of
https://github.com/infiniflow/ragflow.git
synced 2026-07-11 22:25:41 +08:00
12787996d1
feat(ingestion): mirror Go pipeline progress into the document table;
harden resume guards
- pipeline: bind the owning document via WithDocumentID; after each
TrackProgress event aggregate ingestion_task_log progress and mirror
progress/run/progress_msg back into the document table, so GET
/api/v1/datasets/{dataset_id}/documents reflects live Go pipeline
progress without a bespoke endpoint.
- canvas: extend the S3 resume guard to reject legacy no-op nodes (e.g.
ExitLoop) so component_total equals the count of progress-reporting
components and the aggregate percent can reach 100%.
- runtime/canvas: route progress through TrackProgress; add interrupt
test coverage (r3_interrupt_test.go).
- dao/entity: add IngestionTask.DocumentID column and AggregateProgress
support used by the mirror; IngestionTaskLog keeps a Checkpoint column
alongside the progress fields.
feat(deepdoc): cache DocAnalyzer inference results in Redis (1h TTL)
- Redis-backed DocAnalyzerCache decorator over inference.Client; cache
key = "ddoc:cache:<method>:" + sha256 of the JPEG-encoded image bytes
(deterministic).
- TTL = 1h; hits skip the inner HTTP call and return cached JSON; inner
errors are not cached.
refactor(deepdoc): align figure cropping with Python cropout + bounded
page caches
- CropSectionByDLA mirrors Python cropout: best-overlap DLA
figure/equation region, fallback to section bbox per page, vertical
concat on gray background.
- sliding-window page-image cache bounds peak memory to the recent
window instead of the whole PDF.
- rename DLADebug -> DLARegions across parser/chunker/tests.
refactor(parser): drop lib_type selector; align NewXxxParser with
NewPDFParser
- remove config["lib_type"] lookup and the libType param/field/switch
from all nine constructors; surface the CGO-required error at
ParseWithResult time instead of construction time; drop resolveLibType,
its test, and the four lib_type constants.
feat(utility): add a reusable workerpool for bounded concurrent
execution
- internal/utility/workerpool.go (+ tests).
refactor: translate Chinese prose comments to English in non-harness Go
files.
chore: upgrade github.com/cloudwego/eino from v0.9.9 to v0.9.12.
54 lines
2.3 KiB
Go
54 lines
2.3 KiB
Go
//
|
|
// Copyright 2026 The InfiniFlow Authors. All Rights Reserved.
|
|
//
|
|
// Licensed under the Apache License, Version 2.0 (the "License");
|
|
// you may not use this file except in compliance with the License.
|
|
// You may obtain a copy of the License at
|
|
//
|
|
// http://www.apache.org/licenses/LICENSE-2.0
|
|
//
|
|
// Unless required by applicable law or agreed to in writing, software
|
|
// distributed under the License is distributed on an "AS IS" BASIS,
|
|
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
|
// See the License for the specific language governing permissions and
|
|
// limitations under the License.
|
|
//
|
|
|
|
package entity
|
|
|
|
type IngestionTask struct {
|
|
ID string `gorm:"column:id;primaryKey;size:32" json:"id"`
|
|
UserID string `gorm:"column:user_id;size:32;not null" json:"user_id"`
|
|
DocumentID string `gorm:"column:document_id;size:32;not null;index" json:"document_id"`
|
|
DatasetID string `gorm:"column:dataset_id;size:32;not null" json:"dataset_id"`
|
|
Schema JSONMap `gorm:"column:schema;type:longtext" json:"schema"`
|
|
Status string `gorm:"column:status;size:32;not null;" json:"status"`
|
|
// ComponentTotal is the number of components in the task's DSL graph.
|
|
// It is the authoritative denominator for progress percentage so the
|
|
// frontend does not have to count DSL nodes itself. Written once the
|
|
// pipeline compiles the canvas (see pipeline.Run).
|
|
ComponentTotal int `gorm:"column:component_total;default:0" json:"component_total"`
|
|
BaseModel
|
|
}
|
|
|
|
// TableName specify table name
|
|
func (IngestionTask) TableName() string {
|
|
return "ingestion_task"
|
|
}
|
|
|
|
type IngestionTaskLog struct {
|
|
ID int `gorm:"column:id;primaryKey;autoIncrement" json:"id"`
|
|
TaskID string `gorm:"column:task_id;size:32;not null;index" json:"task_id"`
|
|
Checkpoint JSONMap `gorm:"column:checkpoint;type:longtext;not null" json:"checkpoint"`
|
|
ComponentIndex int `gorm:"column:component_index" json:"component_index"`
|
|
Phase int `gorm:"column:phase" json:"phase"`
|
|
Component string `gorm:"column:component;size:64;index" json:"component"`
|
|
Message string `gorm:"column:message;type:text" json:"message"`
|
|
BaseModel
|
|
}
|
|
|
|
// TableName specify table name
|
|
func (IngestionTaskLog) TableName() string {
|
|
return "ingestion_task_log"
|
|
}
|