Repository Cache Synchronization#

Porch's package lifecycle involves several asynchronous components — the CRD-based PackageRevision controller, the API registry, in-memory caches, and external git backends — that can race against each other. Synchronization failures manifest as fatal concurrent map panics, blueprints that aren't immediately visible after publishing, stale lifecycle states being written back by a slow render, or partial database state visible to concurrent readers.

Synchronization work spans six distinct layers:

In-Memory Cache Race Conditions (crcache)#

Concurrent Map Panics#

cachedRepository maintains two maps — cachedPackageRevisions and cachedPackages — protected by a sync.RWMutex 1. Under concurrent load, parallel PackageRevision create/update requests caused fatal "concurrent map iteration and map write" panics. PR #258 2 addressed this by:

• Wrapping all reads with RLock() / RUnlock() in getCachedPackages() and identifyLatestRevisions()
• Upgrading to Lock() / Unlock() for lastVersion writes in refreshAllCachedPackages()
• Using snapshot-based iteration in createMainPackageRevision() and refreshAllCachedPackages() to avoid holding a lock while ranging over maps that a concurrent writer might modify

A TODO in repository.go acknowledges the residual risk of multiple simultaneous refresh operations when the read lock is dropped before calling refreshAllCachedPackages() 1.

Repository Initialization Races#

Cache.OpenRepository() delegates to SafeRepoMap.LoadOrCreate() to atomically retrieve or create a cached repository entry 3. This replaced an earlier double-check locking pattern (PR #378, 4) which prevented duplicate goroutines and resource leaks when concurrent callers raced to initialize the same repository.

The current FindAllUpstreamReferencesInRepositories method demonstrates the expected usage pattern: it acquires cachedRepo.mutex.RLock() before ranging over cachedPackageRevisions and releases it before returning 3.

Timing Gap: Cache Visibility After Package Publish#

When a package revision transitions to Published, a "main" branch twin is created. Without an explicit cache refresh, subsequent lookups returned stale data — making the published blueprint invisible to callers immediately after the approve operation (issue #678, PR #87) 5.

The fix added a RefreshCache() method to cachedRepository that calls refreshAllCachedPackages() synchronously, and wired a call to it from the engine's UpdatePackageRevision() immediately after a lifecycle transition to Published 6.

For watcher-based clients, the engine's WatcherManager broadcasts changes via NotifyPackageRevisionChange() after mutating operations, providing near-real-time visibility without polling 6.

Retry & Recovery for Transient Remote Failures#

Remote git operations fail transiently with errors like "ref file is empty" when the backend is momentarily inconsistent. PR #281 7 introduced fetchRemoteRepositoryWithRetry() in pkg/externalrepo/git/git.go, which wraps the underlying fetch using util.RetryOnErrorConditional 8.

Retry parameters:

• Attempts: up to r.repoOperationRetryAttempts (typically 4)
• Backoff: (retryNumber + 1) × baseRetryDelay where baseRetryDelay = 200ms (exponential)

Covered operations: OpenRepository, Version, listPackageRevisions, DeletePackageRevision, updateDeletionProposedCache

Retryable push errors recognized by pattern matching 8:

• "remote ref", "failed to update ref", "pre-receive hook declined"
• "non-fast-forward update", "stale file handle", "broken pipe", "status code: 500"

For delete operations, a conditional retry specifically targets conflictingRequiredRemoteRefError 8. The generic RetryOnErrorConditional utility lives in pkg/util/util.go.

API Registry: Concurrent CUD Serialization#

Concurrent Create/Update/Delete requests on the same PackageRevision can corrupt state before they reach the cache layer. PR #118 9 added a per-revision mutex map in pkg/registry/porch/packagecommon.go:

• pkgRevOperationMutexes — a map of sync.Mutex keyed by namespace/name (or namespace/repo/package/workspace for Creates), guarded by mutexMapMutex 10
• getMutexForPackage(key) retrieves or creates the per-package mutex
• On Update/Delete, TryLock() is called; if it fails, the server returns HTTP 409 with "another request is already in progress on package revision [key]" 10

For finalizer-clearing updates during deletion, pkg/meta/store.go uses retry.RetryOnConflict() from the Kubernetes client library to handle transient conflicts 9.

PackageRevision Controller: Reconcile Loop Races#

The v1alpha2 PackageRevision controller (introduced in PR #514, 11) runs three sequential sub-reconcilers per cycle:

reconcileSource → reconcileRender → reconcileLifecycle

Because rendering is async and can take seconds, several timing hazards exist:

Stale Lifecycle State#

After reconcileRender completes, a concurrent API patch may have advanced the Spec.Lifecycle field (e.g., Draft → Proposed). Without a re-read, reconcileLifecycle would see the pre-render snapshot and revert the transition. PR #964 12 fixed this by re-reading the PackageRevision from the API server after reconcileRender before calling reconcileLifecycle 13.

Stale Render Detection#

checkRenderStale compares the render-request annotation value recorded at render-start against the current value fetched post-render 14. If a concurrent user push changed the annotation during rendering, the render is marked stale and requeued — preventing the stale result from overwriting the newer push.

FunctionConfig Cache#

The FunctionConfigReconciler serializes cache updates with a write lock in UpdateExecCache() and uses a read lock in GetProcessorFromCache() (PR #964, 12). On controller startup, controllers/main.go pre-populates the FunctionConfig store via mgr.GetAPIReader().List() to prevent an empty exec cache immediately after a pod restart 12.

Shared Cache / SSA Field Ownership#

PR #514 shares a single cache instance between the repository controller and the PR controller 11. Concurrent field updates are prevented by Server-Side Apply with distinct field managers for controller state, render state, and Kptfile-derived fields.

Database Layer: Transactional Resource Writes#

When using the dbcache backend, PackageRevisionResources writes in pkg/cache/dbcache/dbpackagerevisionresourcessql.go wrap DELETE + INSERT in a PostgreSQL transaction (PR #964, 12). Without this, concurrent writers could leave a partially-deleted or partially-inserted resource set visible to concurrent readers — an incomplete state that caused flaky E2E tests (issue #967).

Key Source Files#