Distributed Startup Resilience#

HugeGraph's distributed deployment (PD → Store → Server) relies on a layered startup sequence enforced by Docker Compose health checks and a pair of shell wait scripts. Several design choices in this chain have meaningful resilience implications.

Startup Sequence Overview#

The reference HA topology is a 3 PD × 3 Store × 3 Server cluster defined in docker-compose-3pd-3store-3server.yml. Startup order is enforced by depends_on: condition: service_healthy chains:

PD nodes (Raft cluster) → Store nodes → Server nodes

Inside each server container, docker-entrypoint.sh orchestrates:

1. wait-storage.sh — polls PD/Store readiness
2. init-store.sh — one-time graph initialization (gated by an init_complete flag file)
3. start-hugegraph.sh
4. wait-partition.sh — post-startup partition assignment check

Health Check Design#

PD health check — unconditional 200 OK#

GET /v1/health on the PD service is implemented in StoreAPI.checkHealthy() and returns an empty string with HTTP 200 unconditionally. The identical pattern is used by hg-store: HgStoreStatusController.checkHealthy() also returns "" with no state inspection.

Docker Compose uses these endpoints as the sole readiness signal for PD and Store nodes 1:

# PD healthcheck
test: ["CMD-SHELL", "curl -fsS http://localhost:8620/v1/health >/dev/null || exit 1"]

# Store healthcheck
test: ["CMD-SHELL", "curl -fsS http://localhost:8520/v1/health >/dev/null || exit 1"]

Implication: A PD or Store node passes its health check as soon as the HTTP listener is up — even if Raft leader election has not completed. Downstream services may start connecting before the cluster is operationally ready.

No Raft awareness#

Neither /v1/health endpoint queries Raft state. The codebase does expose Raft/partition-level information (e.g., PartitionAPI checks engine.isLeader()) but this is not plumbed into the liveness/readiness path. A Raft follower that has not yet joined a quorum will report healthy.

Server health check — /versions#

The server uses GET /versions as its health probe. This endpoint (implemented in VersionAPI.list()) simply returns build metadata and has no dependency on graph or storage state.

Wait Scripts#

wait-storage.sh — multi-endpoint with failover#

wait-storage.sh derives PD REST endpoints from pd.peers (gRPC port 8686 → REST port 8620) and tries each peer in a loop 2:

check_any_pd() {
  for peer in $(echo "$PD_REST_LIST" | tr ',' ' '); do
    if curl ... http://${peer}/v1/health ...; then
      echo "$peer"; return 0
    fi
  done
  return 1
}

Once any PD is healthy, it then polls GET /v1/stores waiting for at least one store in "state":"Up" 3. The whole block times out after WAIT_STORAGE_TIMEOUT_S=300 seconds 4.

Key limitation: The script picks the first available PD and sticks with it for the store-state poll. If that PD is a follower and not the cluster leader, GET /v1/stores may return stale or incomplete data depending on PD's read consistency.

wait-partition.sh — hardcoded single endpoint, no failover#

wait-partition.sh polls $STORE_REST — a single store endpoint — and waits for partitionCount to be non-zero 5. The STORE_REST variable is hardcoded to store0:8520 in the Docker Compose server template 6:

environment:
  STORE_REST: store0:8520

In docker-entrypoint.sh, if STORE_REST is unset, it defaults to store:8520 7. There is no fallback to store1 or store2 if store0 is unavailable. A failure at store0 stalls all three server containers.

The script is also called after the server starts and only produces a WARN log on timeout rather than aborting 8:

./bin/wait-partition.sh || log "WARN: partitions not assigned yet"

Key Issues at a Glance#

Relevant Source Files#