Stale Connection and Address Caching#

HugeGraph's store client layers three independent caches between a caller and a live store-node address. None of them has an active eviction or refresh path. In static deployments this is harmless; in Kubernetes — where pod IPs change on restarts and headless-service DNS records rotate — all three caches can independently hold stale data, producing persistent connection failures.

gRPC Channel Cache (AbstractGrpcClient)#

The root of the problem is the static channel pool in AbstractGrpcClient. Every subclass shares a single static Map<String, ManagedChannel[]> channels keyed by the address string host:port. Channels are lazily created on the first call to getChannels(target) and then returned directly from the map on every subsequent call — there is no TTL, no health-based eviction, and no shutdown()/rebuild path.

Channels are constructed with 1:

ManagedChannelBuilder.forTarget(target).usePlaintext().build()

forTarget resolves the address exactly once at channel-creation time. Because ManagedChannel is long-lived and gRPC's internal resolver is not forced to re-query DNS, a pod IP change renders every channel in the pool broken while the map key (the old host:port string) remains valid.

GrpcNodeHealthyClient maintains its own separate channel map with identical semantics (no eviction) 2. The only client that performs explicit cleanup is GrpcStoreStateClient, which implements Closeable and calls channel.shutdown() on all entries, but only on graceful close — not in response to address changes 3.

Node Metadata Cache (HgStoreNodeManager)#

HgStoreNodeManager is a process-wide singleton that maintains two maps: nodeIdMap (Long → HgStoreNode) and addressMap (String → HgStoreNode) 4. Nodes are inserted via addNode with a check-then-insert pattern that skips re-registration if the node ID already exists, even if the address has changed. There is no updateNode, removeNode, or TTL mechanism.

When a node is unreachable, applyNode logs a warning and fires a NOT_EXIST notification 5, but neither the metadata maps nor the gRPC channel pool are invalidated as a result.

GrpcStoreNodeBuilder sets the node address once at build time 6; GrpcStoreNodeImpl stores it in an immutable field 7. There is no public API to update a node's address after construction.

Partition-Fault Notifications (Limited Refresh)#

NotifyingExecutor handles PARTITION_FAULT_TYPE_NOT_LEADER responses from the store and forwards partition-leader changes to HgStoreNodeManager.notifying() 8. This is the only active refresh path in the client, but it only re-routes partition ownership — it does not delete the stale HgStoreNode entry or close the broken ManagedChannel for the old address.

NodeTxExecutor wraps all store calls in a retry loop 9 with a fixed backoff (1 s × 3, then incrementally), but retries reuse the same cached node/channel, so they can never succeed if the underlying address is stale.

DNS Caching via Java SecurityManager#

When OPEN_SECURITY_CHECK=true (the default in start-hugegraph.sh), the JVM is started with 10:

-Djava.security.manager=org.apache.hugegraph.security.HugeSecurityManager

Under a Java SecurityManager, the JVM's InetAddress cache defaults to infinite TTL for successful lookups (networkaddress.cache.ttl = -1), as opposed to the 30-second default used without a SecurityManager. HugeSecurityManager does not override this default and does not set networkaddress.cache.ttl 11. As a result, every DNS name resolved after JVM start is pinned forever — even if the Kubernetes headless-service DNS entry is updated.

The combination means: even if the gRPC channel were re-created (e.g., after a manual restart), the new channel would resolve the same stale IP until the JVM is restarted.

Summary of Affected Components#

Key Files for Deeper Investigation#

• AbstractGrpcClient.java — channel pool, getChannels, createChannel
• HgStoreNodeManager.java — node metadata maps, addNode, applyNode
• GrpcStoreNodeBuilder.java — address set at build time
• GrpcStoreNodeImpl.java — immutable address field
• NotifyingExecutor.java — partition-leader change handling
• NodeTxExecutor.java — retry loop
• HugeSecurityManager.java — checkConnect, no DNS TTL override
• hugegraph-server.sh — SecurityManager activation