Apply Locking and Parallelism#

Atlantis enforces two distinct lock layers before executing an apply command:

1. Global apply lock — a persistent, operator-managed lock (via atlantis apply lock) that disables all applies cluster-wide. Implemented by ApplyClient behind the ApplyLockChecker / ApplyLocker interfaces in server/core/locking/apply_locking.go.

2. Pull-level working-dir lock — an in-memory, per-PR mutex that serializes concurrent apply commands on the same pull request and blocks applies while a plan is in flight. Implemented by WorkingDirLocker.TryLockPull in server/events/working_dir_locker.go. Introduced in PR #6605 to prevent applying a plan whose PR head commit or base branch has changed since the plan was generated.

The top-level entry point is ApplyCommandRunner.Run in server/events/apply_command_runner.go.

Execution Order in ApplyCommandRunner.Run#

Lock acquisition happens before project/workspace resolution — an ordering with significant implications for multi-instance setups 1:

Because step 5 precedes step 7, a targeted -p <project> apply acquires the pull lock before Atlantis knows which projects it owns — the root cause of a multi-instance ordering bug described below 3.

Lock Granularity and Intra-PR Parallelism#

Two nested lock scopes control concurrency within a single pull request 4:

Pull-level lock: only one apply (or plan) may hold this at a time. It also scans existing workspace locks for an in-flight plan and blocks the apply if one is found 5, preventing apply from racing against a concurrent atlantis plan.

Workspace-level lock: scoped to repo/pull/workspace/path/projectName. PR #6086 added projectName to the key, so projects with different names sharing the same workspace/path can lock independently and run in parallel within a single apply invocation.

Parallel apply within one invocation: once the pull-level lock is held, matched projects can run concurrently. isParallelEnabled returns true when ParallelApplyEnabled is set on the first project context 2. Each project then contends only on its own workspace-level lock.

Net effect: across separate apply invocations on the same PR, applies are strictly serialized by the pull-level lock; within a single apply invocation across multiple projects, they can run in parallel via workspace-level locks.

Multi-Instance and Multi-Replica Complications#

Lock-Before-Project-Resolution Bug#

In deployments where multiple Atlantis instances each own a disjoint set of projects, a targeted atlantis apply -p project-b is broadcast to all instances. An instance that doesn't own project-b still acquires the pull-level lock before discovering it has no matching projects. If another instance is concurrently running an apply on the same PR, the non-owning instance posts a lock-conflict error — even when --silence-no-projects should suppress it 3.

PR #6811 fix (ShouldSilenceTargetedApply): for -p-targeted applies with --silence-no-projects enabled, checks whether the named project exists in this instance's config before acquiring the pull lock. If the project is unowned and no plan is in flight, the command is silently skipped. This is complementary to skipIgnoredTargetedDirBeforeApplyLocks (which handles -d directory-targeted applies); the two cover different targeting modes 6.

Cross-Replica Concurrency with Shared Working Directories#

DefaultWorkingDirLocker is an in-memory map guarded by a sync.Mutex 7 — it cannot coordinate across processes. When multiple Atlantis replicas share a working directory (e.g., via a shared filesystem), two replicas can simultaneously operate on the same PR, risking corruption (e.g., incompatible Terraform provider lockfiles).

PR #6825 fix (LeasedWorkingDirLocker): wraps the local locker with a Redis-backed lease at the pull level:

• TryAcquirePullLease — atomic acquire with TTL (owner-only).
• RenewPullLease — periodic renewal; if renewal fails near expiry, the process fail-stops (panics) before another replica can take over.
• ReleasePullLease — owner-only release.

Project-level locks remain local (in-memory), so intra-instance parallelism across projects is preserved. The Redis lease layer is wired in only when Redis is configured; single-instance deployments use the local locker unchanged 8.

Redis Lock Storage Details#

Project-level locks in Redis use the key format pr/{repoFullName}/{path}/{workspace}/{projectName} 9. PR #6086 added projectName to this key; migrateOldLockKeys 10 migrates old keys on startup using SCAN (compatible with Redis Cluster) rather than KEYS. Ownership-checked cleanup (UnlockIfOwnedByPull) uses an atomic Lua script to delete a lock only when its pull number matches 11.

Key Files#

Related PRs:

• PR #6605 — introduced the pull-level lock to prevent stale plan application
• PR #6086 — added projectName to workspace lock key for independent parallel applies
• PR #6811 — silences unowned named-project applies before pull locking in multi-instance setups
• PR #6825 — Redis-backed leased locker to prevent cross-replica concurrency on shared working directories

Issue:

• Issue #6786 — documents the lock-before-project-resolution ordering bug in multi-instance setups