Dosu Logo
Home
Documents
Provider Architecture
Provider Architecture
Type
Document
Status
Published
Created
Jan 8, 2026
Updated
Apr 10, 2026
Updated by
Dosu Bot

Overview#

The system supports multiple data providers for exchange rates and security prices. Providers are selected and configured at runtime, allowing flexibility and resilience in data fetching. The system previously supported the Synth provider, which has since been removed.

Historical Context: Synth Provider#

The Synth provider was originally used to fetch exchange rates and security prices. It was integrated into the provider registry and could be selected via environment variables. The provider, its implementation, related tests, UI elements (such as logo fetching), and environment variables (e.g., SYNTH_API_KEY) were removed in a major update. This removal also included updating UI components to use generic or provider-agnostic options and updating documentation and tests to reflect the change. The rationale for removal was to consolidate on TwelveData for market data, simplify maintenance, and address API credit usage concerns, though the PRs do not state this explicitly. References to Synth were replaced with more general references to third-party providers in the UI and documentation

.

Current Providers#

The system supports seven securities providers: TwelveData, Yahoo Finance, Tiingo, EODHD, Alpha Vantage, MFAPI.in, and BinancePublic. The provider registry (Provider::Registry) manages available providers for each concept (e.g., :exchange_rates, :securities). For exchange rates, the available providers are :twelve_data and :yahoo_finance. For securities, the available providers are :twelve_data, :yahoo_finance, :tiingo, :eodhd, :alpha_vantage, :mfapi, and :binance_public
.

Provider selection is concept-based. Exchange rates use a single provider selected via the EXCHANGE_RATE_PROVIDER environment variable (defaulting to twelve_data). Securities support multiple providers simultaneously: users enable providers through checkboxes in the settings interface, with selections stored in SECURITIES_PROVIDERS (a comma-separated list). The legacy SECURITIES_PROVIDER environment variable is supported for backward compatibility
.

Provider Configuration#

Providers are configured via environment variables and, optionally, application settings. The main configuration variables are:

# Select the provider for exchange rates (single)
EXCHANGE_RATE_PROVIDER=twelve_data

# Enable multiple securities providers (comma-separated)
SECURITIES_PROVIDERS=twelve_data,yahoo_finance,tiingo,binance_public

# TwelveData API key and base URL
TWELVE_DATA_API_KEY=yourapikeyhere
TWELVE_DATA_URL=https://api.twelvedata.com

# TwelveData rate limiting configuration
TWELVE_DATA_MIN_REQUEST_INTERVAL=1.0 # Minimum delay (seconds) between API requests (default: 1.0)
TWELVE_DATA_MAX_REQUESTS_PER_MINUTE=7 # Maximum API credits per minute (default: 7)

# Yahoo Finance retry and throttling configuration
YAHOO_FINANCE_MAX_RETRIES=5 # Maximum number of retries for Yahoo Finance requests
YAHOO_FINANCE_RETRY_INTERVAL=1.0 # Interval (seconds) between retries
YAHOO_FINANCE_MIN_REQUEST_INTERVAL=0.5 # Minimum interval (seconds) between Yahoo Finance requests

# Tiingo API key and configuration
TIINGO_API_KEY=yourapikeyhere
TIINGO_URL=https://api.tiingo.com
TIINGO_MIN_REQUEST_INTERVAL=1.5 # Minimum delay (seconds) between API requests (default: 1.5)
TIINGO_MAX_REQUESTS_PER_HOUR=1000 # Maximum requests per hour (default: 1000)

# EODHD API key
EODHD_API_KEY=yourapikeyhere

# Alpha Vantage API key
ALPHA_VANTAGE_API_KEY=yourapikeyhere

# BinancePublic base URL (optional, defaults to data-api.binance.vision)
BINANCE_PUBLIC_URL=https://data-api.binance.vision

If environment variables are not set, the system uses application settings (e.g., Setting.twelve_data_api_key). API keys are encrypted at rest via Setting::EncryptedSettingFields. TwelveData, Tiingo, EODHD, and Alpha Vantage require API keys. Yahoo Finance, MFAPI.in, and BinancePublic do not require API keys. For Yahoo Finance, cookie and crumb authentication is handled automatically for all API endpoints; no manual credential setup is required. All requests to Yahoo Finance API endpoints (both /v8/finance/chart and /v10/finance/quoteSummary) require valid authentication cookies and CSRF crumbs, which are managed and cached internally. User-agent rotation and throttling are also applied automatically to reduce rate limiting and improve reliability.

TwelveData Rate Limiting#

TwelveData's API enforces credit-based rate limits (e.g., 8 credits/minute for free-tier users). To avoid exhausting API credits during concurrent account syncs, the provider implements a dual-layer throttling system:

  • Layer 1: Per-instance minimum interval — Enforces a configurable delay between API requests (default: 1.0 second) via TWELVE_DATA_MIN_REQUEST_INTERVAL. This paces individual requests to prevent burst traffic.
  • Layer 2: Global per-minute credit counter — Tracks API credit consumption via cache/Redis to stay within the configured limit (default: 7 credits/minute) set by TWELVE_DATA_MAX_REQUESTS_PER_MINUTE. When approaching the rate limit, requests are automatically throttled inline until the next minute window.

The system detects rate limit errors automatically, including HTTP 429 responses returned by TwelveData in both status codes and JSON error responses (code 429). Cross-rate endpoint calls via time_series/cross correctly charge 5 credits (instead of 1) to the throttling system.

The default of 7 credits/minute provides a safety buffer for free-tier users (who have 8 credits/minute), reducing the likelihood of hitting rate limits even with concurrent account syncs. Because requests are throttled inline (sleeping when necessary), the ExchangeRate::Provided cache lock TTL is set to 5 minutes to account for worst-case throttle waits.

Provider Rate Limiting#

Providers implement rate limiting via the Provider::RateLimitable module, which provides a standard interface for interval-based request throttling and error transformation. Each provider defines its own rate limits:

  • TwelveData: Dual-layer throttling with per-instance minimum interval (default: 1.0 second) and global per-minute credit counter (default: 7 credits/minute).
  • Yahoo Finance: Minimum interval throttling (default: 0.5 seconds) with retry logic for HTTP 429 responses.
  • Tiingo: Minimum interval (default: 1.5 seconds), hourly request counter (default: 1000 requests/hour), and unique symbol tracking (500 symbols/month).
  • EODHD: Daily request counter (20 requests/day).
  • Alpha Vantage: Daily request counter (25 requests/day).
  • MFAPI.in: Interval throttling only, no hard limits.
  • BinancePublic: Minimum interval throttling (default: 0.1 seconds), no hard limits.

The rate limiter uses atomic operations (increment-then-check) to prevent race conditions during concurrent account syncs. Providers that exceed rate limits raise RateLimitError, which the importer catches and reports.

Yahoo Finance Provider Details#

Yahoo Finance integration uses cookie and crumb authentication universally for all API endpoints, including:

  • /v8/finance/chart (used for health checks, fetching security prices, and chart data)
  • /v10/finance/quoteSummary (used for security information)

Authentication Process#

Authentication credentials (cookie and crumb) are fetched using the fetch_cookie_and_crumb method and cached automatically, with a maximum cache duration to avoid stale tokens. The crumb parameter is added to all API requests, and the cookie is included in request headers via the authenticated_client method.

When an "Unauthorized" error is received from the API, the provider automatically clears the cached crumb and fetches a fresh cookie/crumb pair, then retries the request once with the new authentication credentials. This helps prevent rate-limiting issues from Yahoo Finance that can occur when proper cookie and crumb headers are missing from requests.

Rate Limiting and Reliability#

To reduce rate limiting, the provider rotates through a pool of modern browser user-agent strings (Chrome 145, Firefox 148, Safari 26) for each request. The user-agent pool was updated in February 2026 to reflect current browser versions and avoid raising red flags with Yahoo's API.

Additionally, requests are throttled to enforce a minimum interval between calls (default: 0.5 seconds).

Retry logic is enhanced: the provider will retry failed requests up to the configured maximum (default: 5), with exponential backoff and support for retrying on HTTP 429 (rate limit) responses. All retry and throttling parameters can be configured via environment variables.

If authentication fails (e.g., due to invalid crumb or cookie), a specific authentication error is raised and reported. These mechanisms are transparent to the user and do not require manual intervention.

BinancePublic Provider Details#

BinancePublic provides free, keyless access to cryptocurrency securities via Binance's public market-data host (data-api.binance.vision). The provider supports search via cached exchangeInfo (refreshed every 24 hours) and daily historical prices via paginated klines (up to 1000 rows per call).

Supported Currencies#

BinancePublic exposes pairs in USD (from USDT), EUR, JPY, BRL, and TRY. Users can match their family currency and skip an extra FX conversion hop. Binance has no GBP trading pairs; GBP-family users fall back to USDT→USD via the app's FX conversion.

Exchange MIC#

Binance is registered under the official ISO 10383 operating MIC BNCX (assigned January 2026, country AE). This MIC is used in search results and configuration files.

Authentication and Rate Limiting#

No API key is required. The provider implements minimal interval throttling (default: 0.1 seconds) with no hard limits. Requests are retried automatically (up to 3 times) with exponential backoff.

Logos#

Crypto asset logos are served via Brandfetch's crypto API endpoint. The URL format is:

https://cdn.brandfetch.io/crypto/{BASE_ASSET}/icon/fallback/lettermark/w/{SIZE}/h/{SIZE}?c={CLIENT_ID}

For search_securities, logos are set directly using Security.brandfetch_crypto_url(base), which generates the Brandfetch crypto URL. For fetch_security_info, logo_url is intentionally returned as nil — crypto logos are generated at save time by Security#generate_logo_url_from_brandfetch using the /crypto/{base} route.

Unknown or obscure coins may return 400 from Brandfetch and render as broken images in the dropdown, similar to the behavior for stocks with obscure tickers. This change aligns crypto logo handling with the existing Brandfetch infrastructure used for stocks, simplifying the overall logo generation flow.

Rationale for Multiple Providers#

Supporting multiple providers offers flexibility, coverage, and resilience. Different providers may have different API models, data coverage, rate limits, and reliability characteristics. The multi-provider architecture allows users to enable multiple securities providers simultaneously, with the system querying all enabled providers concurrently and deduplicating results. Each security is linked to the provider that discovered it via the price_provider column, ensuring price fetches use the correct data source. If a provider becomes unavailable, securities linked to that provider are marked offline; re-enabling the provider brings them back online automatically. The removal of Synth and consolidation on TwelveData (with Yahoo Finance as a fallback, plus four additional providers) simplifies maintenance and configuration while retaining the ability to switch providers if needed

.

Data Fetching and UI Implications#

Data fetching logic is encapsulated in provider classes (e.g., Provider::TwelveData, Provider::YahooFinance, Provider::Tiingo, Provider::Eodhd, Provider::AlphaVantage, Provider::Mfapi, Provider::BinancePublic). These classes implement methods for fetching exchange rates and security prices, with retry logic, error handling, and caching. The provider registry enforces strict provider selection and raises errors if a requested provider is not found for a concept.

If no providers are configured or available, the system logs a warning and disables related features in the UI. For example, if no exchange rate provider is available, historical exchange rates cannot be fetched or displayed. Similarly, if no securities providers are available, security prices and details may be missing.

The multi-provider search interface queries all enabled providers concurrently via Concurrent::Promises, with per-provider timeout and result caching to avoid rate limit exhaustion. Search results are deduplicated on (ticker, exchange, provider) so the same ticker from different providers appears as separate options, allowing users to choose their preferred data source. The UI displays provider badges and currency information for each result, with country flags for visual disambiguation. When a user selects a security, the chosen provider is stored in the price_provider column.

Example: Selecting Providers#

To use Yahoo Finance as the provider for exchange rates and enable multiple securities providers, set the following environment variables:

EXCHANGE_RATE_PROVIDER=yahoo_finance
SECURITIES_PROVIDERS=yahoo_finance,tiingo,eodhd,binance_public
TIINGO_API_KEY=yourapikeyhere
EODHD_API_KEY=yourapikeyhere

To use TwelveData (the default) for both exchange rates and securities:

EXCHANGE_RATE_PROVIDER=twelve_data
SECURITIES_PROVIDERS=twelve_data
TWELVE_DATA_API_KEY=yourapikeyhere

The system will use the selected exchange rate provider for currency conversions and query all enabled securities providers concurrently during security searches. If a provider is unavailable or misconfigured, related features will be disabled in the UI, and warnings will be logged.

Provider Selection Flow#

Provider Architecture | Dosu