gRPC Buffer Memory Management#
Overview#
When the Cortex querier receives series data from ingesters or store-gateways over gRPC, the unmarshalled protobuf objects (labels strings and chunk Data byte slices) reference the same backing memory as the gRPC receive buffer. Because these series objects are held for the entire duration of a query, the Go garbage collector cannot reclaim the ~32KB+ receive buffers, causing heap to grow under concurrent query load.
The fix applied across two PRs is to detach label and chunk data from gRPC buffers immediately after receiving each response, accepting small per-call allocation costs in exchange for prompt GC of the wire buffers.
The Problem: Buffer Pinning via Unsafe Casts#
Protobuf unmarshalling in Go reuses memory buffers. The ingester streaming path's hot function FromLabelAdaptersToLabels converts []LabelAdapter to labels.Labels via an unsafe.Pointer cast — zero allocation cost, but the resulting labels.Labels shares the backing memory of the gRPC receive buffer. Similarly, chunk Data fields are byte slices backed by the same buffer .
As long as any series in the query holds a reference to a label string or chunk byte slice, the entire gRPC receive buffer stays alive on the heap .
The Fix: Detach Before Holding#
Two complementary techniques break the reference chain:
1. Safe Label Copy — FromLabelAdaptersToLabelsWithCopy#
FromLabelAdaptersToLabelsWithCopy calls CopyLabels, which pre-allocates a single byte buffer sized to hold all label name/value strings, copies every string into it, and uses yoloString to get string views into the new buffer — one allocation per series regardless of label count .
This replaces the unsafe-cast variant in callers where data lifetime exceeds the gRPC buffer lifetime.
2. Chunk Data Deep Copy — detachChunksFromBuffer#
For chunk byte slices, the pattern is a simple append([]byte(nil), c.Data...) per chunk, allocating independent storage for each chunk's raw data .
Where These Are Applied#
| Query Path | PR | File | Change |
|---|---|---|---|
| Ingester streaming | #7670 | distributor_queryable.go | Replaced FromLabelAdaptersToLabels → FromLabelAdaptersToLabelsWithCopy + added detachChunksFromBuffer |
| Store-gateway streaming | #7519 | blocks_store_queryable.go | Added detachSeriesFromBuffer() using labelpb.ReAllocZLabelsStrings + chunk data append copy |
Note on streamingSelect#
The current indexed snapshot of streamingSelect still shows FromLabelAdaptersToLabels (the unsafe variant) at line 176. PR #7670 is the merged fix — if you're debugging heap growth, confirm the live branch includes this change.
Performance Tradeoff#
The benchmark in PR #7670 (100 series × 1KB chunks) shows :
- Without detach: ~23 KB/op, ~400 allocations — but gRPC buffers pinned for query lifetime
- With detach: ~161 KB/op, ~900 allocations — gRPC buffers freed immediately after response processing
The short-lived allocations are far preferable to unbounded heap growth under concurrent load.
Key Source References#
pkg/cortexpb/compat.go—FromLabelAdaptersToLabels(unsafe),FromLabelAdaptersToLabelsWithCopy(safe),CopyLabelspkg/querier/distributor_queryable.go—streamingSelect, ingester chunk/label processing- PR #7670 — Ingester path buffer detachment
- PR #7519 — Store-gateway path buffer detachment (precursor)