Connector Architecture#

RAGFlow's Connector system provides scheduled, automated ingestion of external data sources — such as S3, Google Drive, Notion, Slack, and 30+ others — into knowledge bases. Connectors run on a configurable polling cadence, track sync state across runs, and tag every ingested document with a source_type so the system can distinguish connector-managed documents from manually uploaded files.

Core Data Models#

Three database models form the backbone of the system (all defined in api/db/db_models.py):

Key Connector fields 1:

• source — matches a value from the FileSource / DocumentSource enums (e.g., "google_drive", "s3", "notion").
• input_type — always POLL for user-created connectors; possible values are poll, event, load_state, slim_retrieval 2.
• config — source-specific credentials and settings stored as JSON.
• refresh_freq / prune_freq — polling interval in minutes for sync and prune tasks respectively.
• status — lifecycle state, values from TaskStatus: "0" (unstart), "1" (running), "2" (cancel), "3" (done), "5" (schedule).

Supported Data Sources#

The FileSource enum in common/constants.py enumerates all recognized source identifiers. FileSource.LOCAL (empty string "") is reserved for manually uploaded files. Connector sources include:

• Cloud storage: S3, R2, Google Cloud Storage, Azure Blob, OCI Storage, Dropbox, Box, OneDrive
• Collaboration/docs: Google Drive, Notion, Confluence, SharePoint, WebDAV, Seafile
• Messaging: Slack, Teams, Discord, Gmail, Outlook, IMAP
• Dev tools: GitHub, GitLab, Bitbucket, Jira
• Databases: MySQL, PostgreSQL, BigQuery, DingTalk AI Table
• Other SaaS: Airtable, Asana, Zendesk, Salesforce, Moodle, RSS, REST API

OAuth is supported for Google Drive, Gmail, and Box via dedicated endpoints in the connector API 3.

source_type Field and Document Provenance#

Every Document row carries a source_type field (default: "local"). Every File row carries the same field (default: "").

When a connector indexes a document, source_type is set to the composite string "{connector.source}/{connector.id}" 4. This two-part format lets the system:

1. Filter all documents belonging to a specific connector instance — used in rebuild() 5 and stale-document cleanup 6.
2. Distinguish connector-managed documents from local uploads at query time.

Service Layer#

ConnectorService in api/db/services/connector_service.py is the primary service class. Key methods:

• schedule_tasks(connector_id) — enqueues SYNC and optionally PRUNE tasks for each linked KB 7. Prune is only scheduled when config.sync_deleted_files is truthy.
• cancel_tasks(connector_id) — transitions all SCHEDULE/RUNNING tasks for the connector to CANCEL 8.
• rebuild(kb_id, connector_id, tenant_id) — clears all sync logs and connector-sourced documents for a KB, then re-schedules a full sync 5.
• cleanup_stale_documents_for_task(...) — removes documents no longer present in the upstream source by diffing the retained file list against all documents with the connector's source_type 6.
• accessible(connector_id, user_id) — tenant-scoped authorization check 9.

Task types are defined in ConnectorTaskType: SYNC and PRUNE.

REST API#

All connector endpoints live in api/apps/restful_apis/connector_api.py:

UI Integration#

The dataset settings page exposes connector linking via link-data-source.tsx. Each linked connector is shown as a DataSourceItem with controls to toggle auto_parse, trigger a rebuild, or unlink the connector from the KB 18. The auto_parse field on Connector2Kb determines whether newly synced documents are automatically sent through the parsing pipeline 19.