Store Lifecycle Management#

The PD (Placement Driver) module coordinates the full lifecycle of HugeStore nodes — from initial registration through graceful shutdown — via two primary services: StoreNodeService and TaskScheduleService. Store state is persisted in RocksDB through StoreInfoMeta, and exposed over REST via StoreAPI.

State Machine#

A store progresses through these states (defined in Metapb.StoreState):

Registration#

StoreNodeService.register() handles both first-time and re-registration:

• First-time: a random 64-bit storeId is generated by newStoreNode(), the store is persisted with state Pending.
• Re-registration: if the store was Offline or Up, or appears in initialStoreList, it transitions directly to Up 1. Other states land in Pending.
• Tombstone guard: stores in Tombstone state are rejected at registration time 2.
• Duplicate detection: if the Raft address changed within 0.8×STORE_HEART_BEAT_INTERVAL (24 s of the 30 s default), the registration is rejected as a probable duplicate 3.

Heartbeats & TTL-based Active Membership#

StoreNodeService.heartBeat() is the main keep-alive path:

• Every heartbeat calls StoreInfoMeta.keepStoreAlive(), which writes the store's serialized proto under an activeStore key with a configurable TTL (pdConfig.store.keepAliveTimeout).
• Active-store reads use getInstanceListWithTTL(), which returns only entries whose TTL has not expired — so membership naturally shrinks when heartbeats stop.
• A store in Offline state that sends a heartbeat is automatically promoted back to Up 4.
• A store in Exiting state that reports partitionCount == 0 is auto-promoted to Tombstone via heartbeat, signalling that all its replicas have migrated out 5.

TaskScheduleService.patrolStores() runs every 60 seconds 6 and catches any Up/Unknown store that has silently disappeared from the active set, marking it Offline 7.

State Transitions via updateStore#

StoreNodeService.updateStore() is the controlled path for external state changes and reacts to each new state:

• Exiting: removes the store from the active set if it was already inactive; triggers storeTurnoff() to start shard reallocation 8.
• Offline: removes from active set immediately 9.
• Tombstone: removes from active set and calls storeTurnoff() 10. Guarded: transition is blocked if it would drop active stores below pdConfig.minStoreCount 11.
• Up: calls keepStoreAlive() and re-evaluates cluster health 12.

Every state change fires onStoreStatusChanged() 13, which notifies all registered StoreStatusListeners — including TaskScheduleService, which records the lastStoreTurnoffTime when a store goes Tombstone 14.

Shard Reallocation & Replica Migration#

storeTurnoff(store) iterates every ShardGroup that contains a shard on the departing store, removes that shard, and calls reallocShards():

• reallocShards computes the desired shardCount from config, adds or removes shards from active stores via modulo assignment, persists the new ShardGroup, and fires ConfChangeType.CONF_CHANGE_TYPE_ADJUST to the affected partition so the Raft group reconfigures itself 15.
• TaskScheduleService.patrolPartitions() independently checks that every shard group matches the configured replica count and calls reallocShards() for any that don't 16.
• Load balancing (balancePartitionShard) is intentionally held for at least 30 minutes after the last store turn-off (TurnOffAndBalanceInterval) to allow replicas to stabilise before redistribution .

Cluster Health Check#

After every Up/heartbeat event, checkStoreStatus() verifies:

1. Active store count ≥ pdConfig.minStoreCount (otherwise Cluster_Not_Ready).
2. For each shard group, more than half of its shards are on active stores (otherwise Cluster_Not_Ready).

Shard group state changes propagate via updateShardGroupState() → updateClusterStatus() → ClusterStats 18.

REST API Entry Points#

StoreAPI exposes the lifecycle externally:

Key Source Files#