gRPC Client Connection Management#

Overview#

HugeGraph's HStore client maintains a pool of gRPC ManagedChannel objects and stubs in AbstractGrpcClient. All concrete gRPC clients (blocking, async) extend this class. Understanding its pooling model, a known stub-initialization indexing bug, and the lack of stale-transport recovery is essential for debugging connectivity failures against HStore nodes.

Channel Pool#

AbstractGrpcClient holds channels in a static ConcurrentHashMap<String, ManagedChannel[]> keyed by the target address string 1. On first access for a given target, getChannels() creates an array of concurrency channels (default: 1 << 5 = 32) in parallel using a shared executor 2. Channels are built with ManagedChannelBuilder.forTarget(target).usePlaintext() — no TLS, no keep-alive, no idle-timeout 3.

Each call to getBlockingStub(String) or getAsyncStub(String) picks a slot via a global AtomicLong counter masked to the pool size 4, 5.

Key constraint: the channels map is static — it is shared across all AbstractGrpcClient subclass instances for the lifetime of the JVM.

Stub-Initialization Indexing Bug#

When the stub map for a target is first populated, the loop iterates concurrency slots but captures the already-incremented index from the outer call instead of iterating i 6:

IntStream.range(0, concurrency).forEach(i -> {
    ManagedChannel channel = channels[index]; // ← uses outer `index`, not `i`
    ...
    value[i] = new HgPair<>(channel, stub);
});

As a result, all concurrency stub slots are bound to the same single channel from position index. Subsequent round-robin selection picks different stub objects but they all share one underlying transport. This can cause head-of-line blocking and makes channel-level isolation ineffective. The async stub initialization has the same defect 7 8.

Stale Transport After Pod Replacement#

When a Kubernetes Store pod is replaced behind the same DNS name, the existing ManagedChannel objects become stale. The stack of issues that prevents recovery:

1. No channel eviction on transport failure#

Neither getChannels(), getBlockingStub(), nor getAsyncStub() removes or shuts down a cached channel on error. Once an entry is inserted into the channels map it stays there 2.

2. NotifyingExecutor does not distinguish UNAVAILABLE#

When a transport throws, NotifyingExecutor.handleErr() reports HgNodeStatus.NOT_WORK and rethrows. It inspects only FeedbackRes proto status codes, never StatusRuntimeException.getStatus().getCode(). There is no UNAVAILABLE-specific path that would evict channels, stubs, or the node entry 9.

3. retryingInvoke retries the same cached state#

NodeTxExecutor.retryingInvoke() retries the same Supplier with exponential backoff (1 s for the first three attempts, then incremental). It does not invalidate or rebuild any session, stub, channel, or DNS state between attempts 10.

4. notice() only invalidates the partition cache#

On transport failure, HgStoreNodePartitionerImpl.notice() calls pdClient.invalidPartitionCache() 11. It does not invalidate HgStoreNodeManager.nodeIdMap, cached addresses, stubs, or ManagedChannel instances.

5. applyNode returns the cached node unchanged#

When a retry re-queries PD and receives the same node ID, HgStoreNodeManager.applyNode() immediately returns the existing nodeIdMap entry without contacting the provider again. Address changes for the same node ID are invisible 12.

6. JVM DNS cache amplifies the problem (Kubernetes deployments)#

The HugeGraph Server container installs a SecurityManager for Gremlin sandboxing. Under Java 11, a SecurityManager causes InetAddress to cache successful DNS resolutions forever by default. Since gRPC's DnsNameResolver resolves via InetAddress.getAllByName(), even a gRPC-internal reconnect re-resolves to the old pod IP. The failure does not self-heal until the Server JVM is restarted 8.

Failure Signature#

HgStoreClientException {sessionInfo: {storeNodeSession: {storeNode:
  {address: "hugegraph-store-1...svc:8500", nodeId: ...}}},
  reason: "UNAVAILABLE: io exception"}

PD shows the Store as Up. Schema operations succeed. /versions and /graphs continue responding. Only HStore data writes fail 8.

Mitigations#

Immediate runtime workaround (Kubernetes)#

Set a finite JVM DNS cache TTL via JVM system property (validated on Temurin 11) in the Server container's JAVA_OPTS:

-Dsun.net.inetaddr.ttl=30

Note: -Dnetworkaddress.cache.ttl=30 does not work when passed as a system property under Java 11 — it must be set via Java security properties. Use sun.net.inetaddr.ttl as the chart-level workaround 8.

Store-client hardening (longer term)#

See the full proposal in issue #3124:

• Detect Status.Code.UNAVAILABLE explicitly in NotifyingExecutor.
• Evict and shut down the target's ManagedChannel[] entries atomically.
• Clear the corresponding blocking/async stub maps.
• Invalidate or refresh the HgStoreNode entry in nodeIdMap so address changes for the same node ID are observable.
• Fix the stub-initialization loop to bind each stub value[i] to channels[i], not channels[index].

Readiness probe gap#

The current /versions health check does not exercise the HStore data path. A pod remains Ready during this failure and continues receiving traffic 8.

Key Source Files#