Security Price Import Pipeline#

The pipeline fetches, gap-fills, and persists daily Security::Price rows for every online security. It runs as a scheduled background job and also fires inline during individual account syncs when a price is missing. Two jobs drive it:

1

During an individual account sync, Account::Syncer#import_market_data also calls into this pipeline on-demand for any prices not yet in the DB; errors there are swallowed and reported to Sentry so a market data failure doesn't abort the sync.

Scheduled Import: ImportMarketDataJob → MarketDataImporter#

ImportMarketDataJob#perform passes mode: and clear_cache: to MarketDataImporter.new(...).import_all, which runs two steps:

1. import_security_prices — iterates Security.online and calls security.import_provider_prices(start_date:, end_date:) for each record 2.
2. import_exchange_rates — fetches currency-pair rates needed when account or entry currency differs from the family's base currency.

Start-date logic (full vs. snapshot mode) 3 4:

• :full (default, daily job) — start date is the earliest trade date for that security, covering full history.
• :snapshot — always SNAPSHOT_DAYS = 31 days ago; used for lightweight refreshes.

End date is always today in EST to avoid querying a US-based API for a future date 5.

Per-Security Import: Security#import_provider_prices → Security::Price::Importer#

Security#import_provider_prices (defined in the Security::Provided concern) constructs a Security::Price::Importer and calls import_provider_prices on it. The importer:

1. Short-circuits if all expected non-provisional prices for the date range already exist in the DB 6.
2. Calls the provider via security_provider.fetch_security_prices(...) 7. On failure: logs, records a DebugLogEntry, stores @provider_error, and returns an empty hash.
3. Gap-fills with LOCF (Last Observation Carried Forward) — iterates fill_start_date..end_date and carries the previous valid price forward for any date without a provider value 8.
4. Tags provisional — LOCF-filled prices within the last PROVISIONAL_LOOKBACK_DAYS = 7 days are marked provisional: true and re-fetched on the next run 9.
5. Upserts in batches of 200 via Security::Price.upsert_all, keyed on (security_id, date, currency) 10.

Pre-listing / IPO gap handling: When the provider has no data before the user's start date (e.g., a 2023 Reddit trade before the 2024 IPO), the importer advances fill_start_date to the earliest valid provider price and stores that in security.first_provider_price_on, so future syncs skip the pre-listing range entirely 11.

clear_cache mode: Ignores existing DB prices in favour of fresh provider data. Used for manual refreshes and can rediscover earlier provider history 12.

Security::Price Model#

Security::Price validates presence of date, price, and currency, and enforces uniqueness on (date, security_id, currency) 13. The refetchable_provisional scope returns provisional rows within the last N days (default 7) — this is what the short-circuit check queries to decide whether re-fetching is needed.

Health Checks: Security::HealthChecker#

SecurityHealthCheckJob calls Security::HealthChecker.check_all, which processes two scopes 14:

• Never-checked — no daily cap; processed in full immediately.
• Due for re-check — not checked in the last 7 days; at most DAILY_BATCH_SIZE = 1000 per run, oldest-first.

For each security, run_check fetches today's price via provider.fetch_security_price(...). Alpha Vantage is exempt: the check is skipped to conserve its tight daily API quota 15.

Failure Handling and Offline Promotion#

The offline flag is the gate in MarketDataImporter.import_security_prices; offline securities are silently skipped via Security.online.find_each 19. The provider_status method surfaces :ok, :stale, :offline, :no_provider, or :provider_unavailable for UI and diagnostics.

Key Files#