Volume Snapshot Backup#
Overview#
Volume snapshot backup is one of CloudNativePG's backup methods (BackupMethodVolumeSnapshot), managed entirely by the operator β not the instance manager . It takes point-in-time CSI VolumeSnapshot objects for each PVC attached to a selected Pod, then tracks their lifecycle to completion. The feature supports both online (PostgreSQL stays up) and offline (Pod fenced) modes.
Entry point: pkg/reconciler/backup/volumesnapshot/reconciler.go β the Reconciler struct and its Reconcile method.
Backup Lifecycle (6 Steps)#
The reconciler drives every snapshot backup through six sequential steps :
- Prepare β puts PostgreSQL into backup mode (online) or fences the Pod (offline). Skipped if snapshots already exist.
- Create snapshots β creates one
VolumeSnapshotper PVC, then requeues after 10 s to let the external-snapshotter controller pick them up. - Wait for provisioning β polls until every snapshot has been claimed by a
VolumeSnapshotContent; annotates each withSnapshotStartTime/SnapshotEndTimeonce provisioned. - Finalize β exits backup mode / unfences the Pod, sets
BackupPhaseFinalizing, and patches backup status with snapshot metadata. - Wait for ready β polls until every snapshot's
ReadyToUseistrue. - Complete β sets
BackupPhaseCompleted, annotates snapshots with start/end times, WAL position, and backup-label data.
Each step returns a ctrl.Result to requeue when work remains, or nil to advance to the next step in the same reconcile loop.
Configuration Precedence (Backup vs. Cluster)#
GetVolumeSnapshotConfiguration in api/v1/backup_funcs.go applies a simple override: start with cluster.Spec.Backup.VolumeSnapshot, then overwrite with any field set on the Backup spec:
Backup.Spec.Onlineoverridescluster.Spec.Backup.VolumeSnapshot.OnlineBackup.Spec.OnlineConfigurationoverrides the cluster-level online configuration
GetOnlineOrDefault provides the final effective online flag, defaulting to true (online) when nothing is explicitly set.
Snapshot Error Handling & Retry Logic#
CSI Errors β Deadline-Based Retry (PR #11132)#
Previously, handleSnapshotErrors tried to classify CSI errors as permanent vs. transient via text-pattern matching, but provider-specific messages (OCI 409 "backup in progress", GCP 502, etc.) fell through and caused premature failures . The fix in PR #11132 removed all classification logic: every VolumeSnapshot.Status.Error is now retried uniformly until a deadline expires .
The deadline is controlled by the annotation backup.cnpg.io/volumeSnapshotDeadline on the Backup resource (default: 10 minutes) . The reconciler records the first failure timestamp in backup.Status.PluginMetadata and requeues every 10 s until the deadline is exceeded .
Network/Transport Errors β Immediate Requeue (PR #11069)#
Transport-level failures to the instance manager (dial timeout, connection refused, TLS error) are detected by isNetworkErrorRetryable and cause an immediate 5 s requeue , not a backup failure. This matters most during the finalize step, where snapshots are already provisioned and a single network blip would otherwise discard an otherwise complete backup.
Snapshot Collision Handling (PR #11071)#
The operator lists existing snapshots through a cached client, so the cache can lag behind a snapshot created in a previous reconcile cycle. When Create returns AlreadyExists, handleSnapshotCreateError applies three-tier logic:
- Same backup owner (label
backup.cnpg.io/namematches) β log and return nil; the reconciler requeues and picks it up via label selector. - Cache still hides the object (
GetreturnsNotFoundafterAlreadyExists) β treat as harmless; requeue and re-evaluate when the cache catches up. - Foreign snapshot (owned by a different backup) β surface as an error to avoid silently adopting unrelated objects.
Key Files & References#
| Area | File |
|---|---|
| Core reconciler | pkg/reconciler/backup/volumesnapshot/reconciler.go |
| Network-error retryability | pkg/reconciler/backup/volumesnapshot/errors.go |
| Configuration precedence & backup API helpers | api/v1/backup_funcs.go |
| PR: collision tolerance | #11071 |
| PR: transport-error retry | #11069 |
| PR: universal CSI-error retry | #11132 |
| Bug: OCI false failure | #8859 |