zlei6/ragflow
1
0
Fork 0
mirror of https://github.com/infiniflow/ragflow.git synced 2026-07-13 08:28:22 +08:00
Code Issues Packages Projects Releases Wiki Activity

feat(connectors): ETag-based bypass for incremental S3 ingestion (#14628) (#14677)

Browse Source 
### What problem does this PR solve?

S3-family connector syncs currently re-download every in-window object
just so we can compute `xxhash128(blob)` and compare against
`Document.content_hash`. Anything that bumps `LastModified` without
changing bytes (`aws s3 cp` touches, bucket re-encryption, etc.) pays
full bandwidth and re-parses files that didn't actually change. #14628
covers the broader incremental-ingestion redesign; this PR is the first
slice.

The fix is a pre-listing short-circuit. `BlobStorageConnector` (S3 / R2
/ GCS / OCI / S3-compat) now implements a new `FingerprintConnector`
interface: `list_keys()` paginates `list_objects_v2` and yields
`KeyRecord(key, fingerprint)` where `fingerprint = xxhash128(ETag)`. The
orchestrator joins those against the connector's existing `{doc_id:
content_hash}` map and only calls `get_value(key)` when the fingerprint
differs. Unchanged keys are skipped entirely — no `GetObject`, no
re-parse.

No DDL. xxhash128(ETag) is 32 hex chars and reuses the existing
`Document.content_hash` column per @yingfeng's suggestion; the connector
decides at listing time whether to populate it. Local uploads and
connectors that don't opt in fall through to the existing post-download
`xxhash128(blob)` path with no behavior change.

This is PR-1 of a 4-PR series — full design lives on #14628. Subsequent
PRs extend tier 1 to local FS / WebDAV / Dropbox / Seafile / RDBMS
(PR-2), wire up tier 2 cursor connectors with `SyncLogs.next_checkpoint`
(PR-3), and unify deletion via `KeyRecord(deleted=True)` reconciliation
(PR-4). Holding those back keeps this PR additive and reviewable on its
own.

#### Files touched

- `common/data_source/models.py` — new `KeyRecord`; optional
`fingerprint` on `Document`
- `common/data_source/interfaces.py` — `IncrementalCapability` enum,
`FingerprintConnector` ABC
- `common/data_source/blob_connector.py` — `BlobStorageConnector`
implements `FingerprintConnector`; per-object download factored into
`_build_document_from_obj()` so `_yield_blob_objects`, `list_keys`,
`get_value` all share it
- `rag/svr/sync_data_source.py` —
`_BlobLikeBase._fingerprint_filtered_generator` does the bypass loop;
`_run_task_logic` plumbs `doc.fingerprint` into the upload dict
- `api/db/services/document_service.py` —
`list_id_content_hash_map_by_kb_and_source_type()` helper
- `api/db/services/connector_service.py` + `file_service.py` —
fingerprint flows through `duplicate_and_parse → upload_document` and
lands in `content_hash`
- `test/unit_test/common/test_blob_connector_fingerprint.py` — 14 tests
covering ETag normalization (single-part, multipart, quoted, empty),
`list_keys()` not calling `GetObject`, `get_value()` materializing with
fingerprint, deterministic/stable fingerprints, and the bypass loop
asserting `GetObject` is *not* called on a match

#### Worth flagging for review

Old `_BlobLikeBase._generate` called `poll_source(start, now)` with a
`LastModified` window when `poll_range_start` was set. New code uses
`_fingerprint_filtered_generator` (full bucket listing + fingerprint
compare) outside of explicit `reindex=1`. Strictly better for
unchanged-bucket cases since it skips `GetObject`, but it does mean
every sync now does a full `list_objects_v2` paginate. Should still be
cheap for most buckets — flagging in case anyone has a very large bucket
where the time-window filter was meaningful.

On migration: existing rows have `content_hash = xxhash128(blob)` from
the old code. The first sync after this lands sees ETag-derived
fingerprints that don't match, re-fetches every object once, and writes
the new fingerprint. From the second sync onward the bypass works as
expected. "Slow day one, fast every day after." A `fingerprint_backfill:
trust` opt-out is sketched in the design doc but not in this PR.

#### Test plan

- [x] `uv run ruff check` — clean on all 8 touched files
- [x] `uv run pytest
test/unit_test/common/test_blob_connector_fingerprint.py -v` — 14 passed
- [x] Broader unit-test suite — no regressions in anything I touched
- [ ] Manual smoke against a real S3 bucket — configure a connector, run
sync twice, expect the second sync to log `bypassed=N, fetched=0` and no
`GetObject` calls in CloudTrail / bucket access logs
- [ ] Manual smoke with `reindex=1` — confirm the full re-download path
still works

### Type of change

- [x] New Feature (non-breaking change which adds functionality)

---------

Co-authored-by: Yingfeng <yingfeng.zhang@gmail.com>
This commit is contained in:
Hunnyboy1217
2026-05-09 05:03:56 -07:00
committed by GitHub
parent 7931b693dc
commit 782084780e
8 changed files with 658 additions and 53 deletions
Download Patch File Download Diff File Expand all files Collapse all files

151
common/data_source/blob_connector.py
View File

@@ -1,9 +1,12 @@
"""Blob storage connector"""
import logging
import os
from collections.abc import Iterator
from datetime import datetime, timezone
from typing import Any, Optional
import xxhash
from common.data_source.utils import (
create_s3_client,
detect_bucket_region,
@@ -18,9 +21,14 @@ from common.data_source.exceptions import (
CredentialExpiredError,
InsufficientPermissionsError
)
from common.data_source.interfaces import LoadConnector, PollConnector
from common.data_source.interfaces import (
FingerprintConnector,
LoadConnector,
PollConnector,
)
from common.data_source.models import (
Document,
KeyRecord,
SecondsSinceUnixEpoch,
GenerateDocumentsOutput,
GenerateSlimDocumentOutput,
@@ -28,7 +36,20 @@ from common.data_source.models import (
)
class BlobStorageConnector(LoadConnector, PollConnector):
def _normalize_etag(raw_etag: Optional[str]) -> Optional[str]:
"""Return a 32-char hex fingerprint derived from an S3 ETag.
S3 ETags are MD5 (32 hex chars) for single-part uploads and "<md5>-<n>"
(34+ chars) for multipart. We always hash so the column format is uniform
regardless of upload type or provider quirks; equality of the hashed value
is sufficient for change detection.
"""
if not raw_etag:
return None
return xxhash.xxh128(raw_etag.strip('"').encode()).hexdigest()
class BlobStorageConnector(LoadConnector, PollConnector, FingerprintConnector):
"""Blob storage connector"""
def __init__(
@@ -48,6 +69,11 @@ class BlobStorageConnector(LoadConnector, PollConnector):
self.size_threshold: int | None = BLOB_STORAGE_SIZE_THRESHOLD
self.bucket_region: Optional[str] = None
self.european_residency: bool = european_residency
# Populated by list_keys() so a subsequent get_value(key) can find the
# raw S3 object metadata (LastModified, ETag, Key, Size) without a second
# head_object call. Lifetime is one list_keys() pass.
self._listing_cache: dict[str, dict[str, Any]] = {}
self._filename_counts: dict[str, int] = {}
def set_allow_images(self, allow_images: bool) -> None:
"""Set whether to process images"""
@@ -122,6 +148,44 @@ class BlobStorageConnector(LoadConnector, PollConnector):
return None
def _build_document_from_obj(
self,
obj: dict[str, Any],
filename_counts: dict[str, int],
) -> Optional[Document]:
"""Materialize a Document for one S3 object, downloading its body."""
key = obj["Key"]
file_name = os.path.basename(key)
last_modified = obj["LastModified"].replace(tzinfo=timezone.utc)
size_bytes = extract_size_bytes(obj)
if (
self.size_threshold is not None
and isinstance(size_bytes, int)
and size_bytes > self.size_threshold
):
logging.warning(
f"{file_name} exceeds size threshold of {self.size_threshold}. Skipping."
)
return None
blob = download_object(
self.s3_client, self.bucket_name, key, self.size_threshold
)
if blob is None:
return None
return Document(
id=f"{self.bucket_type}:{self.bucket_name}:{key}",
blob=blob,
source=DocumentSource(self.bucket_type.value),
semantic_identifier=self._get_semantic_id(key, file_name, filename_counts),
extension=get_file_ext(file_name),
doc_updated_at=last_modified,
size_bytes=size_bytes if size_bytes else 0,
fingerprint=_normalize_etag(obj.get("ETag")),
)
def _yield_blob_objects(
self,
start: datetime,
@@ -132,51 +196,64 @@ class BlobStorageConnector(LoadConnector, PollConnector):
batch: list[Document] = []
for obj in all_objects:
last_modified = obj["LastModified"].replace(tzinfo=timezone.utc)
file_name = os.path.basename(obj["Key"])
key = obj["Key"]
size_bytes = extract_size_bytes(obj)
if (
self.size_threshold is not None
and isinstance(size_bytes, int)
and size_bytes > self.size_threshold
):
logging.warning(
f"{file_name} exceeds size threshold of {self.size_threshold}. Skipping."
)
continue
try:
blob = download_object(
self.s3_client, self.bucket_name, key, self.size_threshold
)
if blob is None:
doc = self._build_document_from_obj(obj, filename_counts)
if doc is None:
continue
semantic_id = self._get_semantic_id(key, file_name, filename_counts)
batch.append(
Document(
id=f"{self.bucket_type}:{self.bucket_name}:{key}",
blob=blob,
source=DocumentSource(self.bucket_type.value),
semantic_identifier=semantic_id,
extension=get_file_ext(file_name),
doc_updated_at=last_modified,
size_bytes=size_bytes if size_bytes else 0,
)
)
batch.append(doc)
if len(batch) == self.batch_size:
yield batch
batch = []
except Exception:
logging.exception(f"Error decoding object {key}")
logging.exception(f"Error decoding object {obj.get('Key')}")
if batch:
yield batch
def list_keys(self) -> Iterator[KeyRecord]:
"""Enumerate the full bucket keyspace with per-object fingerprints.
Cheap path: relies on list_objects_v2 which returns ETag in the listing,
so no GetObject call is needed. Caches each object's metadata so a
subsequent get_value(key) call can rebuild the Document without a second
round-trip to S3.
"""
if self.s3_client is None:
raise ConnectorMissingCredentialError("Blob storage")
all_objects, filename_counts = self._collect_blob_objects(
start=datetime(1970, 1, 1, tzinfo=timezone.utc),
end=datetime.now(timezone.utc),
)
self._filename_counts = filename_counts
self._listing_cache = {}
for obj in all_objects:
doc_id = f"{self.bucket_type}:{self.bucket_name}:{obj['Key']}"
self._listing_cache[doc_id] = obj
yield KeyRecord(
key=doc_id,
fingerprint=_normalize_etag(obj.get("ETag")),
)
def get_value(self, key: str) -> Document:
"""Materialize the Document for a key previously yielded by list_keys().
Must be called within the same list_keys() pass that produced the key,
since the metadata cache lives on the connector instance and is reset
each list_keys() call.
"""
obj = self._listing_cache.get(key)
if obj is None:
raise KeyError(
f"get_value({key!r}) called before list_keys() yielded the key, "
"or after a subsequent list_keys() reset the cache"
)
doc = self._build_document_from_obj(obj, self._filename_counts)
if doc is None:
raise RuntimeError(f"Failed to materialize Document for key {key!r}")
return doc
def _collect_blob_objects(
self,
start: datetime,

52
common/data_source/interfaces.py
View File

@@ -2,7 +2,7 @@
import abc
import uuid
from abc import ABC, abstractmethod
from enum import IntFlag, auto
from enum import IntEnum, IntFlag, auto
from types import TracebackType
from typing import Any, Dict, Generator, TypeVar, Generic, Callable, TypeAlias
from collections.abc import Iterator
@@ -10,12 +10,26 @@ from anthropic import BaseModel
from common.data_source.models import (
Document,
KeyRecord,
SlimDocument,
ConnectorCheckpoint,
ConnectorFailure,
SecondsSinceUnixEpoch, GenerateSlimDocumentOutput
)
class IncrementalCapability(IntEnum):
"""How a connector handles incremental sync.
FULL_RESYNC -- every sync re-pulls; no per-key state.
CURSOR -- "give me everything since cursor X"; opaque cursor persisted across syncs.
FINGERPRINT -- list_keys() returns (key, fingerprint) cheaply; bodies fetched lazily.
"""
FULL_RESYNC = 0
CURSOR = 1
FINGERPRINT = 2
GenerateDocumentsOutput = Iterator[list[Document]]
class LoadConnector(ABC):
@@ -415,3 +429,39 @@ class IndexingHeartbeatInterface(ABC):
just to act as a keep-alive.
"""
class FingerprintConnector(ABC):
"""Tier 1 connector: cheap full listing with per-key fingerprint.
Sources that can enumerate their entire keyspace via a metadata-only call
(e.g. S3 list_objects_v2 returning ETag + LastModified) implement this to
let the orchestrator skip GetObject for keys whose fingerprint hasn't
changed since the last sync.
The fingerprint is an opaque equality token: two equal fingerprints mean
the content is unchanged from the orchestrator's point of view. Format is
a 32-char hex string so it fits the existing Document.content_hash column;
connectors are responsible for normalizing whatever the source exposes
(typically by hashing it with xxhash128).
"""
INCREMENTAL_CAPABILITY: IncrementalCapability = IncrementalCapability.FINGERPRINT
@abstractmethod
def list_keys(self) -> Iterator[KeyRecord]:
"""Yield one KeyRecord per object currently in the source.
Must enumerate the full current keyspace -- the orchestrator diffs the
result against persisted state to detect adds, updates, and deletes.
"""
raise NotImplementedError
@abstractmethod
def get_value(self, key: str) -> Document:
"""Fetch the body for a single key, returning a fully populated Document.
Called only when list_keys()'s fingerprint differs from the persisted
content_hash for that key (or when no persisted fingerprint exists).
"""
raise NotImplementedError

19
common/data_source/models.py
View File

@@ -99,6 +99,25 @@ class Document(BaseModel):
primary_owners: Optional[list] = None
metadata: Optional[dict[str, Any]] = None
doc_metadata: Optional[dict[str, Any]] = None
# Opaque, connector-supplied fingerprint stored in Document.content_hash for
# change-detection. 32-char hex string; format is per-source (xxhash128 of
# bytes for local uploads, xxhash128(ETag) for blob storage, etc.). When set
# on a yielded Document, the orchestrator persists it as content_hash and
# skips the post-download xxhash128(blob) recomputation.
fingerprint: Optional[str] = None
class KeyRecord(BaseModel):
"""One entry returned by a FingerprintConnector.list_keys() call.
A KeyRecord is the cheap-listing primitive: connector enumerates all keys
it has, attaches a fingerprint when the source exposes one, and the
orchestrator only fetches content when the fingerprint differs from what's
persisted.
"""
key: str
fingerprint: Optional[str] = None
deleted: bool = False
class BasicExpertInfo(BaseModel):