Flux HelmRelease Configuration in Cozystack#

Cozystack generates Flux HelmRelease objects from two paths: the PackageReconciler (for operator-managed Package CRs) and the cozy-apiserver (for Application resources). Both paths share the same set of configurable knobs and apply the same defaults. 1

How HelmReleases Are Generated#

The PackageReconciler.buildHelmReleaseSpec() function assembles every generated HelmRelease with:

• Polling interval (spec.interval): How often Flux re-evaluates a healthy release. Default: 5 minutes. 2
• Retry interval (Install.Strategy.RetryInterval / Upgrade.Strategy.RetryInterval): How long Flux waits between failed install/upgrade attempts. Default: 30 seconds. 3 This is decoupled from the polling interval so failures recover fast without impacting healthy releases. 4
• Install / Upgrade timeout: Maximum wall-clock time allowed for each Helm action. Default: 10 minutes each. 5

Both paths expose CLI flags to override these defaults: --helmrelease-interval, --helmrelease-retry-interval, --helmrelease-install-timeout, --helmrelease-upgrade-timeout, --helmrelease-max-history. 6

Per-application timeout overrides are possible via the release.cozystack.io/helm-install-timeout and release.cozystack.io/helm-upgrade-timeout annotations on an ApplicationDefinition. 7

Remediation Strategy: RetryOnFailure#

All operator-generated HelmReleases use ActionStrategyRetryOnFailure for both install and upgrade. This means a failed install is retried as an upgrade rather than uninstalled first (the Flux default, RemediateOnFailure). Preserving the previous attempt's resources is intentional: uninstalling core components like Cilium (CNI) or CSI drivers would leave the cluster in a broken state for the entire retry window.

Handcrafted tenant HelmRelease templates (e.g., monitoring, seaweedfs-db) achieve the same effect by setting remediation.retries: -1 on both install and upgrade, making Flux retry indefinitely. The timeout field (typically 10m) acts as the real backstop. 8

Timeout Values Across the Codebase#

Dependency Sequencing (dependsOn)#

HelmReleases use dependsOn to gate deployment on upstream readiness. For example, the tenant monitoring HelmRelease depends on victoria-metrics-operator being Ready before creating VMRule/VMAlert CRs, preventing install failures due to the operator's admission webhook being briefly unavailable on cold start.

Health Checks and waitStrategy#

For releases that provision custom resources (databases, operators), Cozystack uses CEL-based healthCheckExprs with waitStrategy: poller to prevent a HelmRelease from reporting Ready before the underlying resource is actually healthy. 10

The seaweedfs-db template illustrates this pattern: without the poller strategy, the HelmRelease flips Ready as soon as Helm applies the CNPG Cluster CR — before Postgres has bootstrapped. The CEL expressions distinguish CNPG's normal ClusterIsNotReady bootstrap state (treated as inProgress) from actual failure states.

The PackageReconciler auto-couples the poller wait strategy whenever healthCheckExprs are set, removing the footgun of configuring expressions without the required strategy. 11

CRD Policy#

Upgrade.CRDs is controlled via the ComponentInstall.UpgradeCRDs field and defaults to Skip (the Helm upstream default). Setting it to CreateReplace enables rolling CRD updates during upgrades but carries a data-loss risk if upstream CRDs drop fields. 12

Pod Restart Backoff Interaction#

Kubernetes CrashLoopBackOff and HelmRelease retries operate independently but interact through the HelmRelease timeout budget:

• Kubernetes backoff (exponential, up to 5 minutes between restarts) delays pod convergence. If a pod takes multiple restart cycles to come up, the accumulated backoff can consume the entire 10-minute HelmRelease timeout.
• HelmRelease retries only see Deployment/DaemonSet Ready conditions — they have no visibility into individual pod restart state.

Common patterns where this surfaces:

• Database race: A client pod starts before its database Service has ready endpoints. Under Cilium with kubeProxyReplacement: true, this produces EHOSTUNREACH ("no route to host") rather than ECONNREFUSED, which may not trigger client-side retry logic designed for connection refused errors.
• Slow certificate issuance: If cert-manager takes 8+ minutes to issue a certificate, a pod without a startupProbe may be killed by its liveness probe while still initializing, triggering repeated restarts that outlast the HelmRelease timeout.

The recommended fix for slow-startup pods is a startupProbe with a generous failureThreshold — not a longer HelmRelease timeout. This grants the container a dedicated initialization grace period without widening the remediation window for genuine failures.