Prometheus Metrics Integration#
Kyverno's metrics subsystem is built on OpenTelemetry and exposes metrics via two backends: Prometheus (pull-based, default for most deployments) and OTLP/gRPC (push-based, for OpenTelemetry collectors). The entry point for the entire subsystem is InitMetrics in pkg/metrics/init.go.
Architecture Overview#
The subsystem has three main layers:
-
Configuration β
MetricsConfiguration(pkg/config/metricsconfig.go) holds namespace filters, histogram bucket boundaries, per-metric exposure config, and themetricsRefreshInterval. It is loaded from a Kubernetes ConfigMap and guards all fields with async.RWMutex. -
MeterProvider β An OTel
sdkmetric.MeterProvideris created by eitherNewPrometheusConfigorNewOTLPGRPCConfig. Both functions apply the sameaggregationSelector(customizing histogram bucket boundaries) andexemplarFilter, and attach an OTel resource identifying the service βkyverno-svc-metricsfor Prometheus,kyverno(theMeterNameconstant) for gRPC . -
MetricsConfigManager β The
MetricsConfigManagerinterface (implemented byMetricsConfig) is a process-wide singleton set viaSetManager. It owns instrument handles for all metric categories: policy changes, client queries, kyverno info, admission, policy engine, controllers, circuit breakers, events, HTTP, cleanup, TTL, and policy-type-specific metrics . Instruments are (re-)registered by callinginitializeMetrics.
Initialization Flow#
InitMetrics (pkg/metrics/init.go) orchestrates startup:
- Creates a
MetricsConfigviaNewMetricsConfigManagerand registers it as the global manager withSetManager. - If
disableMetricsExportis set, falls back to the default no-op OTelMeterProviderand returns early . - Selects a backend based on
otelProvider:"grpc"βNewOTLPGRPCConfig: creates an OTLP exporter with a 2-second periodic reader; optionally fetches TLS credentials from a Kubernetes ConfigMap ."prometheus"βNewPrometheusConfig: creates a Prometheus pull exporter; also registerspromhttp.Handler()onconfig.MetricsPath.
- Sets the new provider globally via
otel.SetMeterProviderand callsinitializeMetricsto bind all instruments .
Periodic MeterProvider Re-creation (Prometheus Only)#
When the provider is "prometheus" and metricsRefreshInterval > 0, InitMetrics starts a background goroutine that periodically tears down and rebuilds the entire MeterProvider :
ticker fires
β shutdown existing sdkmetric.MeterProvider
β call NewPrometheusConfig (creates a fresh exporter + provider)
β otel.SetMeterProvider(new provider)
β initializeMetrics(new provider) β re-registers all instruments
The goroutine exits cleanly when the passed context.Context is cancelled . Errors at any step are logged and the iteration is skipped (continue), leaving the previous provider in place until the next tick .
Why re-create? The Prometheus exporter accumulates in-memory metric state. Periodic recreation resets that state β useful for long-running deployments where stale label cardinality or accumulated histogram data can bloat memory.
Configuring the interval: Set metricsRefreshInterval in the kyverno-metrics ConfigMap. The value is a Go time.Duration string (e.g., "1h"). When absent or zero, the refresh goroutine is never started .
MetricsConfiguration Reference#
All configuration lives in pkg/config/metricsconfig.go . Key ConfigMap keys:
| ConfigMap key | Type | Default | Purpose |
|---|---|---|---|
metricsRefreshInterval | time.Duration | 0 (disabled) | Interval for Prometheus MeterProvider recreation |
namespaces | JSON | include all | Namespace allow/deny list for metric exposure |
bucketBoundaries | JSON array | 15 buckets (0.005β30s) | Histogram bucket boundaries |
metricsExposure | JSON map | all enabled | Per-metric enable/disable and label filtering |
The configuration supports hot-reload via Load(*corev1.ConfigMap); a notify() callback chain is invoked on each reload . BuildMeterProviderViews translates the metricsExposure map into OTel sdkmetric.View objects applied when a MeterProvider is constructed.
Key Source Files#
| File | Purpose |
|---|---|
pkg/metrics/init.go | InitMetrics β subsystem entry point, provider selection, refresh goroutine |
pkg/metrics/metrics.go | MetricsConfig, MetricsConfigManager, NewPrometheusConfig, NewOTLPGRPCConfig, initializeMetrics, ShutDownController |
pkg/config/metricsconfig.go | MetricsConfiguration interface and metricsConfig implementation β ConfigMap parsing, refresh interval, bucket boundaries, views |