Connector Document Sync#

RAGFlow connectors ingest content from external sources (blob storage, SharePoint, Notion, GitHub, relational databases) and must reliably translate each external object into a stable RAGFlow document ID, then avoid re-downloading and re-processing objects that haven't changed. Two mechanisms handle this: deterministic ID construction and fingerprint-based change detection.

Document ID Construction#

Every connector builds a document ID from native source identifiers, not from content. The pattern is consistent across connectors:

SharePoint namespaces by drive ID because Graph API driveItem.id values are only unique within a single drive — a site can expose multiple document libraries 8.

Notion appends the page UUID to the semantic identifier to disambiguate pages with duplicate names 9.

RDBMS defaults to an MD5 hash of the serialized row content when no id_column is configured 10, making IDs unstable if row content changes. Always configure id_column for RDBMS connectors when possible.

Fingerprint-Based Change Detection#

The FingerprintConnector Interface#

FingerprintConnector is the interface for sources that can enumerate their keyspace cheaply via metadata-only calls — without downloading object bodies. It requires two methods:

• list_keys() — yields KeyRecord(key, fingerprint, deleted) for every object in the source 11
• get_value(key) — downloads and returns a fully-populated Document for a single key; called only when the fingerprint has changed 12

BlobStorageConnector is the only current FingerprintConnector implementation 13. It uses S3's list_objects_v2 (which returns ETag in the listing without a GetObject call) to populate KeyRecord.fingerprint cheaply 14.

Fingerprint Format#

Blob storage normalizes the S3 ETag (which varies between single-part MD5 and multipart <md5>-<n> formats) to a uniform 32-char hex string via _normalize_etag() using xxhash.xxh128. This ensures equality comparison works regardless of upload method or provider quirks.

The Document.fingerprint field is an optional str 15. When set, the orchestrator persists it as content_hash and skips recomputing xxhash128(blob) after download.

Orchestration: _fingerprint_filtered_generator#

The sync worker _BlobLikeBase._fingerprint_filtered_generator() orchestrates the bypass logic:

1. Pre-loads {doc_id: content_hash} from the database for all existing documents in the KB from this connector 16
2. Iterates connector.list_keys() — metadata only, no downloads 17
3. Resolves the stored fingerprint against both the legacy format (hash128(connector_id:key)) and the current format (hash128(kb_id:connector_id:key)) for backward compatibility 18
4. Skips get_value() when key_record.fingerprint == stored 19
5. Fetches via get_value() only for new or changed objects 20
6. Logs bypass/fetch/fail counts; uses WARNING level if any fetch failed 21

The fingerprint path is disabled on explicit full reindex (task["reindex"] == "1"), which forces all objects through load_from_state() 22.

Incremental Sync (Non-Fingerprint Connectors)#

Connectors that don't implement FingerprintConnector use time-based incremental sync:

• Blob — when fingerprint path is unavailable (e.g. full reindex), falls back to filtering by LastModified within a (start, end] window 23; SharePoint similarly filters by lastModifiedDateTime 24
• Notion — uses Notion's search API sorted by last_edited_time descending; stops when page timestamps fall before start 25
• GitHub — checkpoint-based pagination (GithubConnectorCheckpoint) with per-repo stage tracking; stops when pr.updated_at <= start 26
• RDBMS — optional timestamp_column cursor; prepare_sync_state() snapshots the current MAX(timestamp_column) before fetching, and persist_sync_state() writes it back to the connector config 27; without timestamp_column, falls back to full table scan 28

Key Source Files#