Provider Alert Retrieval

Provider Alert Retrieval#

Provider alert retrieval is the mechanism by which Keep pulls historical/current alerts from integrated monitoring systems (Datadog, Prometheus, Zabbix, etc.) into the platform. Each provider that supports alert polling implements _get_alerts(), a protected method on BaseProvider.

The `_get_alerts` / `get_alerts` Contract#

BaseProvider defines _get_alerts() as a non-abstract method that unconditionally raises NotImplementedError. It is not decorated with @abc.abstractmethod, even though BaseProvider uses ABCMeta . The only two enforced abstract methods are dispose() and validate_config() . Alert retrieval is therefore optional — providers that don't implement it silently inherit the raising base version.

The public entry point is get_alerts(), which wraps _get_alerts() with:

OpenTelemetry tracing — creates a span named {ClassName}-get_alerts.
Automatic enrichment — stamps every returned AlertDto with providerId and providerType before returning.

get_alerts() does not catch NotImplementedError. The caller get_alerts_by_fingerprint() does catch it and returns an empty dict, making the degradation path explicit at that layer .

Call chain:

get_alerts_by_fingerprint()
  └─ get_alerts() ← traces + enriches
       └─ _get_alerts() ← provider-specific implementation

Error Handling Patterns Across Providers#

There is no enforced contract for how _get_alerts() handles errors. In practice, implementations fall into two camps:

Per-item silent skip (resilient / partial results)#

Errors during individual alert processing are caught, logged, and skipped. The method returns whatever alerts succeeded.

Provider	Pattern	Key lines
Datadog	`try/except` per event → log → `continue`	lines 1056–1208
SignalFx	`try/except` per incident → log → `pass`	lines 157–162
Grafana	Per-item handling across three independent alert sources	(multi-source aggregation)

Fail-fast (all-or-nothing)#

Any error — API or per-item — propagates immediately and the entire call fails.

Provider	Pattern	Key lines
Prometheus	`raise_for_status()` + batch format; no per-item catch	lines 152–168
Dynatrace	`raise_for_status()` → raises `Exception`; list comprehension for items	lines 103–129
Zabbix	`__send_request` raises `ProviderMethodException`; no per-item catch	lines 466–511
VictoriaMetrics	`raise_for_status()` + outer `except` re-raises	lines 434–471

Implications for Developers#

Adding a new provider with alert support: Implement _get_alerts() -> list[AlertDto]. Choose per-item or fail-fast based on whether partial results are preferable to a total failure.
Debugging missing alerts: If a provider returns fewer alerts than expected, check whether it uses per-item skipping and look for logged errors around _get_alerts.
No enforcement mechanism: There is no compile-time or startup check that a provider implements _get_alerts. If the method is missing, callers using get_alerts_by_fingerprint() will silently receive an empty dict ; callers using get_alerts() directly will raise at runtime.
Alert enrichment is automatic: providerId and providerType are set by the get_alerts() wrapper — implementations of _get_alerts() do not need to populate these fields .