mirror of
https://github.com/infiniflow/ragflow.git
synced 2026-07-23 08:56:42 +08:00
feat(moodle): support deleted-file sync (#14548)
Fixes #14551 ### What problem does this PR solve? The Moodle connector did not let the sync runner clean up indexed documents that were deleted from the source. Other connectors such as dropbox, seafile, webdav, and rss already do this through a slim snapshot pass. This PR adds the same support for Moodle. When `sync_deleted_files` is on, the runner now asks the Moodle connector for a lightweight list of every module id that could be indexed. The runner then compares this list with the index and removes any indexed document whose id is not in the list. The slim pass does not download files. It only goes through courses and modules and yields ids. The id format matches the ids that the loader produces, so the match is exact. ### Type of change - [x] New Feature (non-breaking change which adds functionality) ### Notes - `MoodleConnector` now also implements `SlimConnectorWithPermSync`. - New `retrieve_all_slim_docs_perm_sync` yields slim docs with the same ids the loader uses (`moodle_resource_<id>`, `moodle_forum_<id>`, `moodle_page_<id>`, `moodle_book_<id>`, `moodle_assign_<id>`, `moodle_quiz_<id>`). - The `Moodle` sync class now returns `(document_generator, file_list)` so the runner can do the cleanup. If the slim snapshot fails, `file_list` is set back to `None` and the run continues without cleanup. - The web data source map exposes `syncDeletedFiles` for Moodle so the option shows up in the UI. ### How was this tested? - `ruff check` passes on the changed Python files. - Manual review of the produced slim ids against the ids the loader builds in `_process_resource`, `_process_forum`, `_process_page`, `_process_book`, and `_process_activity`. - Behavior parity with the merged dropbox (#14476), seafile (#14499), webdav (#14491), and rss (#14493) PRs.
This commit is contained in:
@@ -976,19 +976,40 @@ class Moodle(SyncBase):
|
||||
|
||||
# Determine the time range for synchronization based on reindex or poll_range_start
|
||||
poll_start = task.get("poll_range_start")
|
||||
file_list = None
|
||||
|
||||
if task["reindex"] == "1" or poll_start is None:
|
||||
document_generator = self.connector.load_from_state()
|
||||
_begin_info = "totally"
|
||||
else:
|
||||
# Freeze the poll end time BEFORE the slim snapshot so that the
|
||||
# snapshot and the poll cover the same point in time. Without
|
||||
# this, a module created between the snapshot and the poll
|
||||
# could be polled as new and at the same time be missing from
|
||||
# the slim list, which would mark it as stale and delete it.
|
||||
end_ts = datetime.now(timezone.utc).timestamp()
|
||||
|
||||
if self.conf.get("sync_deleted_files"):
|
||||
file_list = []
|
||||
try:
|
||||
for slim_batch in self.connector.retrieve_all_slim_docs_perm_sync():
|
||||
file_list.extend(slim_batch)
|
||||
except Exception:
|
||||
logging.exception(
|
||||
"Moodle slim snapshot failed; skipping stale-document cleanup "
|
||||
"(connector_id=%s, kb_id=%s)",
|
||||
task.get("connector_id"),
|
||||
task.get("kb_id"),
|
||||
)
|
||||
file_list = None
|
||||
document_generator = self.connector.poll_source(
|
||||
poll_start.timestamp(),
|
||||
datetime.now(timezone.utc).timestamp(),
|
||||
end_ts,
|
||||
)
|
||||
_begin_info = f"from {poll_start}"
|
||||
|
||||
self.log_connection("Moodle", self.conf["moodle_url"], task)
|
||||
return document_generator
|
||||
return document_generator, file_list
|
||||
|
||||
|
||||
class BOX(SyncBase):
|
||||
|
||||
Reference in New Issue
Block a user