CSP API Rate Limiting#

CB-Tumblebug enforces per-CSP and global concurrency limits when performing bulk operations — primarily spec fetching and resource registration — to stay within cloud provider API rate limits. The configuration is centralized in src/core/model/csp/csp.go and consumed by the bulk operation logic in src/core/resource/spec.go.

Configuration: RateLimitConfig#

The RateLimitConfig struct captures three categories of limits per CSP:

A global cap of GlobalMaxConcurrentConnections = 10 applies across all CSPs simultaneously.

Per-CSP values are looked up via GetRateLimitConfig(providerName), which resolves derived CSP names (e.g., openstack-new01 → openstack) before table lookup. Unknown CSPs fall back to a conservative default (MaxConcurrentRegistrations=3, delays 1000–3000ms).

Per-CSP Limits at a Glance#

1

Bulk Spec Fetching: Two-Level Semaphore#

fetchSpecsForAllConnConfigsInternal uses a two-level concurrency model:

1. Provider goroutine — one goroutine per CSP provider, all run in parallel.
2. Connection semaphore — within each provider, a buffered channel limits concurrent connections 2:
   • Default: 50 concurrent connections per provider 3
   • Alibaba override: 5 — empirical testing showed concurrency ≥10 causes ~90% timeout on DescribeAvailableResource 4

Each connection task runs with a 20-minute timeout. Capacity test results are inline in the code comments 5:

Price Fetching: Three-Level Concurrency#

Bulk price fetching in spec.go adds a third layer 6:

1. Provider-level: max 3 providers processed simultaneously via semaphore
2. Per-provider connection-level: 8 concurrent connections for GCP 7 and Azure; 8 for AWS; configurable for Alibaba (TB_ALIBABA_REGION_CONCURRENCY, default 10)
3. Spider path: 15 concurrent connections for all other providers

Alibaba: Dedicated Rate-Limit Handling#

Alibaba has the most complex rate-limit handling, in src/core/csp/alibaba/price.go:

• Worker pool: default 5 goroutines (TB_ALIBABA_PRICE_WORKERS)
• Per-call delay: 200ms between DescribePrice calls (TB_ALIBABA_PRICE_INTERVAL_MS)
• Retry with backoff: up to 3 retries (TB_ALIBABA_PRICE_MAX_RETRY), base 1200ms backoff + jitter, triggered on ErrorCode: Throttling or TooManyRequests 10
• Transient network retry: single retry for connection reset, I/O timeout, TLS handshake timeout, EOF 11

Adding a New CSP#

To configure rate limits for a new provider 12:

1. Add a name constant to csp.go and append it to AllCSPs.
2. Add a RateLimitConfig entry to rateLimitConfigs with appropriate limits.
3. If the CSP is a derived instance (e.g., private OpenStack), call RegisterCloudPlatform at startup so it resolves to the base platform's config.