Yahoo Finance Integration#

Overview#

Provider::YahooFinance is a keyless provider that serves two concepts in Sure:

• Securities — price history and security info via Yahoo's /v8/finance/chart and /v10/finance/quoteSummary endpoints.
• Exchange Rates — FX pair data via the same chart endpoint using {FROM}{TO}=X symbols.

No API key is required. Authentication is handled by a cookie/crumb mechanism (see below). The provider is registered as yahoo_finance in Provider::Registry 1 and the base URL defaults to https://query1.finance.yahoo.com, overridable via YAHOO_FINANCE_URL 2.

Cookie/Crumb Authentication#

Yahoo Finance's chart and quoteSummary endpoints require a CSRF-style crumb token alongside a session cookie. The full flow is in fetch_cookie_and_crumb and request_cookie_and_crumb:

1. GET https://fc.yahoo.com — sets a session cookie in the response (even on HTTP 404). The cookie value is extracted before the first ; in set-cookie.
2. GET /v1/test/getcrumb with the cookie — returns a plain-text crumb string.
3. Cookie and crumb are cached under yahoo_finance_auth_crumb, capped at MAX_CRUMB_CACHE_DURATION (1 hour) or the cookie's own Max-Age, whichever is shorter 3 4.

Stale-crumb retry: Both fetch_authenticated_chart and fetch_security_info inspect the response body for {"code": "Unauthorized"} even when the HTTP status is 200. On detection, they call clear_crumb_cache and retry exactly once with fresh credentials before raising AuthenticationError.

Rate-limit vs. auth disambiguation: If the crumb endpoint returns the string "too many requests", the provider raises RateLimitError rather than AuthenticationError — distinguishing transient throttling from a genuine auth failure 5 6.

Price Fetching#

fetch_security_prices converts a date range to Unix timestamps (UTC) and calls fetch_authenticated_chart at /v8/finance/chart/{symbol} with interval=1d and includeAdjustedClose=true. The timestamp and indicators.quote[0].close arrays are zipped; nil close values (weekends, holidays) are skipped 7.

fetch_security_price (single-date variant) fetches a 10-day trailing window and returns the closest prior date's price.

Currency normalization: Yahoo returns some prices in minor units — normalize_currency_and_price converts GBp → GBP (×0.01) and ZAc → ZAR (×0.01) 8.

Symbol normalization: normalize_symbol appends exchange suffixes driven by EXCHANGE_CONFIG — e.g., XNSE → .NS, XBOM → .BO, XBOG → .CL.

Results are cached for 5 minutes (CACHE_DURATION) 9.

Rate Limiting & Request Throttling#

• Client-side throttle: throttle_request enforces a minimum gap between consecutive requests (default 0.5 s, via YAHOO_FINANCE_MIN_REQUEST_INTERVAL).
• HTTP 429 retry: Faraday retries up to YAHOO_FINANCE_MAX_RETRIES (default 5) times with exponential backoff and 50% jitter on 429 responses 10. Interval starts at YAHOO_FINANCE_RETRY_INTERVAL (default 1.0 s).
• User-agent rotation: Requests cycle through five modern browser UA strings (Chrome 145, Firefox 148, Safari 26, Edge 145) to reduce bot detection 11, following the approach from yfinance PR #2277.

Health Check System#

health_status returns :healthy, :rate_limited, :unavailable, or :unknown 12:

• Async: When the cached status is stale, health_status enqueues YahooFinanceHealthCheckJob 13 rather than blocking the caller. The job calls refresh_health_status 14.
• Distributed lock: A 15-second lock prevents thundering-herd health re-checks 15 16.
• Probe: perform_health_check fetches a cookie/crumb, then GETs /v8/finance/chart/AAPL with range=1d. HTTP 429 → :rate_limited; Unauthorized body or non-2xx → :unavailable; valid chart result → :healthy 17.

Status transitions are written to DebugLogEntry (category provider_health) 18. Freshness thresholds: healthy = 15 min, rate_limited = 30 min, unavailable = 5 min; all assessments are retained for 1 hour.

Error Types#

All errors are subclasses of Provider::YahooFinance::Error < Provider::Error 19:

When AuthenticationError or RateLimitError propagates through the health check, affected securities go offline: true with offline_reason: "health_check_failed" 20. To debug manually, use:

Provider::Registry.get_provider(:yahoo_finance).fetch_security_price(
  symbol: "NVDA", exchange_operating_mic: "XNAS", date: Date.current
)

Key Source Files#

Docker networking: web and worker containers use 8.8.8.8/1.1.1.1 DNS to avoid IPv6-first resolution failures against Yahoo Finance endpoints. If TCP errors persist, add extra_hosts entries for specific Yahoo IP addresses. See the Docker Self-Hosting guide for details 21.