Distributed Consensus and Recovery#

HugeGraph's storage layer (hugegraph-store) achieves distributed data consistency and durability through Raft-based state machine replication built on the Apache SoFA JRaft library, combined with RocksDB checkpoint-based snapshots that include integrity verification before activation.

Architecture Overview#

Each partition of graph data is managed by a PartitionEngine 1, which owns a JRaft RaftGroupService and Node. All writes flow through the Raft consensus protocol before being applied to the local RocksDB instance, ensuring all replicas maintain the same state.

Write Request
     │
     ▼
PartitionEngine.addRaftTask()
     │ (serialized as RaftOperation)
     ▼
JRaft Node.apply()
     │ (replicated to quorum)
     ▼
PartitionStateMachine.onApply()
     │ (applied by leader and all followers)
     ▼
BusinessHandler / DataManager (RocksDB write)

Core Components#

PartitionEngine#

PartitionEngine is the orchestration layer for each Raft group (one per partition). Key responsibilities:

• Initialization: Configures NodeOptions (log path, metadata path, snapshot path, election/RPC timeouts) and starts a RaftGroupService per partition group 2.
• Write path: addRaftTask() serializes operations and calls Node.apply(). Only the current leader accepts tasks; followers reject with NOT_LEADER.
• Error recovery: On onError(), calls restartRaftNode(), which shuts down and re-initializes the RaftGroupService.
• Peer management: changePeers() orchestrates the learner → follower promotion workflow: add as learner → wait for snapshot sync → promote to full peer.
• Snapshot triggering: doSnapshot() and doSnapshotSync() invoke Node.snapshot() directly, gated by an AtomicBoolean to prevent concurrent snapshot operations.

PartitionStateMachine#

PartitionStateMachine extends JRaft's StateMachineAdapter. It:

• Applies log entries: onApply() iterates committed log entries and dispatches each to registered RaftTaskHandler implementations 3. Both leader (local call) and follower (deserialized byte array) paths are handled.
• Saves snapshots: onSnapshotSave() executes asynchronously under a ReentrantLock to prevent concurrent saves, then delegates to SnapshotHandler.
• Loads snapshots: onSnapshotLoad() refuses to load on the leader 4, reads SnapshotMeta to recover the committedIndex, then delegates to SnapshotHandler.

SnapshotHandler#

SnapshotHandler bridges JRaft's snapshot API and RocksDB:

• Save (onSnapshotSave()): Calls businessHandler.saveSnapshot() to create a RocksDB checkpoint into <snapshotDir>/data, enumerates all files, and registers each with the JRaft SnapshotWriter. CRC64 checksums are computed for .sst and .log files 5 — for files ≤ 8 KB the full file is checksummed; for larger files only the head and tail 4 KB blocks are hashed. A should_not_load marker file is written to prevent the local node from redundantly reloading its own snapshot 6.
• Load (onSnapshotLoad()): Skips loading if the should_not_load flag is present 7, then calls businessHandler.loadSnapshot(), reloads partition metadata from the local DB, and replays any pending async tasks 8.

RocksDBSession — Checkpoint and Verification#

RocksDBSession implements the low-level checkpoint operations:

• saveSnapshot() 9: Creates a RocksDB Checkpoint at a _temp path, then atomically renames it to the final snapshot path. A read lock on cfHandleLock is held throughout.
• verifySnapshot() 10: Opens the snapshot in read-only mode (RocksDB.openReadOnly()) and validates that every ColumnFamilyHandle is non-null. Returns false immediately on any null handle or RocksDBException.
• loadSnapshot() 11: Under a write lock, hard-links (or copies) snapshot files to a _temp directory, calls verifySnapshot() on the temp path 12, and only on success atomically moves the verified directory to the final database path. This ensures a corrupt snapshot never replaces a healthy database.

Membership Changes and Replication#

New replicas join as learners first. The ReplicatorStateListener inner class monitors JRaft replicator state changes; when a learner transitions to ONLINE (snapshot fully transferred), it triggers any pending Change_Shard task to promote the learner to a full peer 13.

Log storage is backed by either RocksDBLogStorage or RocksDBSegmentLogStorage, selected via configuration 14.

Key Source Files#