Tenant Metadata Management#

Overview#

Cortex maintains per-tenant metadata fetchers and store state in two components — the querier's BucketScanBlocksFinder and the store-gateway's BucketStores. Each lazily creates per-tenant resources (metadata fetchers, Prometheus registries, on-disk caches, BucketStore instances) and must clean them up as tenants transition through lifecycle states. Without active eviction, these resources grow without bound and cause memory, metric, and disk leaks over process lifetime .

Tenant State Model#

The users.Scanner interface exposes a single method:

ScanUsers(ctx) → (active []string, deleting []string, deleted []string, err error)

The three states are:

State	Condition
active	Bucket data present and no deletion mark
deleting	Bucket data present and deletion mark exists
deleted	Deletion mark exists, no bucket data remaining

Deletion marks are written to __markers__/<userID>/tenant-deletion-mark.json (and checked at the legacy path <userID>/markers/ for backward compatibility) . The mark records a deletion_time and optional finished_time .

Multiple scanner implementations are available: a direct bucket lister (listScanner), a cached-index reader (userIndexScanner), a shard-filtered wrapper (shardedScanner), and a TTL in-memory cache (cachedScanner) .

BucketScanBlocksFinder — Querier Fetcher Lifecycle#

BucketScanBlocksFinder periodically scans the bucket to discover blocks. It reuses one block.MetadataFetcher per tenant for performance and metric grouping.

Fetcher creation is lazy: getOrCreateMetaFetcher checks the d.fetchers map under a mutex; on miss, createMetaFetcher instantiates the fetcher with a per-tenant prometheus.Registry, registers it into the shared MetadataFetcherMetrics, and caches the on-disk meta state under <CacheDir>/<userID>/meta-syncer/.

Resource leak (fixed in PR #7573): fetchers were never evicted from d.fetchers, so deleted tenants accumulated Prometheus registry entries, map entries, and on-disk caches indefinitely .

Fix — evictInactiveUserFetchers: Called from scanBucket after ScanUsers succeeds, provided ctx.Err() == nil. The eviction logic:

Builds a set of currently-active tenant IDs from ScanUsers.
Under d.fetchersMx lock: calls d.fetchersMetrics.RemoveUserRegistry(userID) and deletes from d.fetchers for each inactive tenant.
Outside the lock (to keep disk I/O off the critical section): removes <CacheDir>/<userID>/meta-syncer/ via os.RemoveAll; best-effort removes the now-empty parent directory .

Single-binary safety: In single-binary mode, CacheDir is shared with the store-gateway's sync directory. The eviction removes only the meta-syncer subdirectory, never the full per-tenant directory tree, so co-located block data is preserved .

Scan scope: scanBucket considers only active users returned by ScanUsers . Tenants in the deleting or deleted states are excluded from fresh scans and their fetchers become eligible for eviction on the next cycle.

BucketStores — Store-Gateway Lifecycle#

BucketStores is the store-gateway's multi-tenant wrapper around Thanos BucketStore. It maintains a stores map and lazy-creates one BucketStore per tenant.

Sync scope: syncUsersBlocks scans for active and deleting users via scanUsers (both states still have bucket data to serve). The shard's FilterUsers further filters to tenants owned by this store-gateway instance.

Lazy store creation: getOrCreateStore allocates a BucketStore, a metadata fetcher (either BucketIndexMetadataFetcher or block.MetaFetcher), per-tenant Prometheus registries, and optionally a per-tenant token bucket — all stored in the stores map.

Eviction of excluded tenants: After each sync, deleteLocalFilesForExcludedTenants walks the sync directory and for any tenant not in the current shard:

Calls closeEmptyBucketStore, which:
- Checks that the store has no loaded blocks (isEmptyBucketStore — time range is (MaxInt64, MinInt64)) .
- Removes the entry from stores, unregisters metaFetcherMetrics and bucketStoreMetrics, removes the token bucket, and calls bs.Close() .
Removes the on-disk sync directory for that tenant .

Stores with blocks still loaded return errBucketStoreNotEmpty and are skipped — the next sync cycle will unload the blocks, and the store will be removed on a subsequent iteration .

Key Design Patterns#

Pattern	Detail
Error isolation	Querier eviction runs if `ctx.Err() == nil`, independent of per-tenant scan errors — preventing a single bad tenant from blocking cleanup of other tenants .
Empty-store guard	Store-gateway skips closing stores that still have blocks loaded; the next sync cycle will first unload blocks, then allow close .
Metrics cleanup	Both components call `RemoveUserRegistry(userID)` on eviction to prevent cardinality growth in `MetadataFetcherMetrics` / `BucketStoreMetrics` .
Disk cleanup	Both components delete per-tenant on-disk directories on eviction; the querier removes only the `meta-syncer` subdirectory to preserve co-located data in single-binary mode .
Graceful shutdown	Querier eviction is skipped when `ctx` is cancelled, avoiding unnecessary disk work during shutdown .

Key Source Files#

File	Role
`pkg/querier/blocks_finder_bucket_scan.go`	Querier fetcher lifecycle: lazy creation, eviction
`pkg/storegateway/bucket_stores.go`	Store-gateway BucketStore lifecycle: creation, close, cleanup
`pkg/storage/tsdb/users/`	`Scanner` interface and tenant state classification
`pkg/storage/tsdb/users/` (deletion marks)	`TenantDeletionMark`, `TenantDeletionMarkExists`