Alert Deduplication and Ingestion#

Keep ingests alerts through two webhook endpoints, routes them through an async processing pipeline, fingerprints them per-provider, and deduplicates before writing to the database. The two key axes of behavior are (1) whether a provider_type is specified in the request and (2) whether a custom deduplication rule exists for that provider.

Ingestion Endpoints#

Both routes are defined in keep/api/routes/alerts.py and immediately return HTTP 202, handing off processing to a background thread pool or ARQ job queue (when Redis is enabled).

• Generic endpoint accepts AlertDto, list[AlertDto], or a raw dict — no provider-specific body parsing. It passes provider_type=None to process_event 1.
• Provider-specific endpoint resolves a provider class, calls provider_class.parse_event_raw_body(event) before dispatch, and passes the resolved provider_type string 2.

Processing Pipeline (process_event)#

process_event in keep/api/tasks/process_event_task.py orchestrates the full flow:

1. Pre-format extraction rules — tenant-level enrichment rules run against the raw payload before any formatting 3.
2. Provider formatting — if provider_type is not None and the event is a raw dict, FormData, or list, the provider class's format_alert() produces AlertDto(s) 4. If the event is already an AlertDto, formatting is skipped.
3. Internal preparation — __internal_prepartion stamps each alert: defaults source to ["keep"] if absent, overrides fingerprint if the caller supplied one, and records the api_key_name used 5.
4. __handle_formatted_events — runs maintenance-window checks, deduplication, DB save, Elasticsearch indexing, workflow triggers, rules engine, and Pusher notifications 6.

Silent failures: process_event runs after a 202 has already been returned to the caller. Any exception caught in the outer try/except is logged and retried via ARQ, but the caller is not notified 7.

Fingerprinting#

Each provider subclass declares FINGERPRINT_FIELDS: list[str] = [] at the class level. BaseProvider.get_alert_fingerprint(alert, fingerprint_fields) computes a SHA-256 over the values of those fields in order. If fingerprint_fields is empty, it falls back to alert.name 8.

After the provider's _format_alert() runs, BaseProvider.format_alert() checks for a custom deduplication rule and, if found, re-computes the fingerprint using custom_deduplication_rule.fingerprint_fields 9. This override happens inside format_alert() itself, so it applies consistently across all ingestion paths.

Deduplication#

Rule Resolution (AlertDeduplicator.get_deduplication_rules)#

1. If provider_type is None, it is normalized to "keep" so custom rules can match 10.
2. A custom rule is looked up by (tenant_id, provider_id, provider_type). If found and full_deduplication=True, it is used as-is 11.
3. If no custom rule exists, the default full-deduplication rule is returned: ignore_fields=["lastReceived"], full_deduplication=True — effectively hashing the entire alert payload minus lastReceived 12.
4. If a custom rule exists but full_deduplication=False, the default rule's ignore_fields are merged into it 13.

Hash Comparison (_apply_deduplication_rule)#

For each alert, ignore_fields are stripped from a deep copy, and SHA-256 is taken over the remaining JSON 14. The result is compared against the last stored hash for that fingerprint:

provider_type=None Behavior#

When alerts arrive via POST /alerts/event (no provider type), the deduplicator normalizes provider_type to "keep". Custom deduplication rules configured for the keep provider — including field-based fingerprint_fields — are therefore correctly applied to these alerts. Without a custom rule, they fall back to the whole-payload hash (minus lastReceived).

Key Files#