Prometheus Metrics Integration#

Kyverno's metrics subsystem is built on OpenTelemetry and exposes metrics via two backends: Prometheus (pull-based, default for most deployments) and OTLP/gRPC (push-based, for OpenTelemetry collectors). The entry point for the entire subsystem is InitMetrics in pkg/metrics/init.go.

Architecture Overview#

The subsystem has three main layers:

1. Configuration — MetricsConfiguration (pkg/config/metricsconfig.go) holds namespace filters, histogram bucket boundaries, per-metric exposure config, and the metricsRefreshInterval. It is loaded from a Kubernetes ConfigMap and guards all fields with a sync.RWMutex 1.

2. MeterProvider — An OTel sdkmetric.MeterProvider is created by either NewPrometheusConfig or NewOTLPGRPCConfig. Both functions apply the same aggregationSelector (customizing histogram bucket boundaries) and exemplarFilter, and attach an OTel resource identifying the service — kyverno-svc-metrics for Prometheus, kyverno (the MeterName constant) for gRPC 2.

3. MetricsConfigManager — The MetricsConfigManager interface (implemented by MetricsConfig) is a process-wide singleton set via SetManager. It owns instrument handles for all metric categories: policy changes, client queries, kyverno info, admission, policy engine, controllers, circuit breakers, events, HTTP, cleanup, TTL, and policy-type-specific metrics 3. Instruments are (re-)registered by calling initializeMetrics.

Initialization Flow#

InitMetrics (pkg/metrics/init.go) orchestrates startup:

1. Creates a MetricsConfig via NewMetricsConfigManager and registers it as the global manager with SetManager 4.
2. If disableMetricsExport is set, falls back to the default no-op OTel MeterProvider and returns early 5.
3. Selects a backend based on otelProvider:
   • "grpc" → NewOTLPGRPCConfig: creates an OTLP exporter with a 2-second periodic reader; optionally fetches TLS credentials from a Kubernetes ConfigMap 6.
   • "prometheus" → NewPrometheusConfig: creates a Prometheus pull exporter; also registers promhttp.Handler() on config.MetricsPath 7.
4. Sets the new provider globally via otel.SetMeterProvider and calls initializeMetrics to bind all instruments 8.

Periodic MeterProvider Re-creation (Prometheus Only)#

When the provider is "prometheus" and metricsRefreshInterval > 0, InitMetrics starts a background goroutine that periodically tears down and rebuilds the entire MeterProvider 9:

ticker fires
  → shutdown existing sdkmetric.MeterProvider
  → call NewPrometheusConfig (creates a fresh exporter + provider)
  → otel.SetMeterProvider(new provider)
  → initializeMetrics(new provider) ← re-registers all instruments

The goroutine exits cleanly when the passed context.Context is cancelled 10. Errors at any step are logged and the iteration is skipped (continue), leaving the previous provider in place until the next tick 11.

Why re-create? The Prometheus exporter accumulates in-memory metric state. Periodic recreation resets that state — useful for long-running deployments where stale label cardinality or accumulated histogram data can bloat memory.

Configuring the interval: Set metricsRefreshInterval in the kyverno-metrics ConfigMap. The value is a Go time.Duration string (e.g., "1h"). When absent or zero, the refresh goroutine is never started 12 13.

MetricsConfiguration Reference#

All configuration lives in pkg/config/metricsconfig.go 14. Key ConfigMap keys:

The configuration supports hot-reload via Load(*corev1.ConfigMap); a notify() callback chain is invoked on each reload 15. BuildMeterProviderViews translates the metricsExposure map into OTel sdkmetric.View objects applied when a MeterProvider is constructed.

Key Source Files#