Usage Cost Calculation#

Cost computation for observations flows through five sequential stages: usage normalization → model lookup → pricing tier selection → exact-match cost calculation → ClickHouse storage. Two ingestion paths feed into the same enrichment pipeline in IngestionService: the OTel path via OtelIngestionProcessor and the native SDK path.

ClickHouse stores four Map-type columns on the observations table to separate user-supplied from computed values 1:

Key source files:

Usage Key Normalization#

Both ingestion paths converge on canonical keys (input, output, total, input_cached_tokens, etc.) before hitting the cost calculator.

OTel Path#

extractUsageDetails() applies sources in this priority order — the first matching source wins 2:

1. langfuse.observation.usage_details span attribute (JSON blob) — skips all other sources when present, even if {}
2. Genkit (genkit-tracer scope) — reads genkit:output.usage.* keys
3. Vercel AI SDK (ai scope) — reads gen_ai.usage.* and enriches from ai.response.providerMetadata (OpenAI, Anthropic, Bedrock cache details)
4. Pydantic AI (pydantic-ai scope) — delegates to generic extraction
5. Generic gen_ai.usage.* / llm.token_count.* — fallback for OpenInference, TraceLoop, etc.

The generic extractor extractGenericGenAiUsageDetails() strips OTel prefixes and maps to canonical keys 2:

input is decremented by cache read + cache creation tokens to avoid double-counting 2.

Native SDK Path#

mapObservationEventsToRecords() populates provided_usage_details from three sources 1:

• Legacy usage.input/output/total fields
• Newer usageDetails map (null values filtered at merge)
• costDetails map → provided_cost_details

getUsageUnits() then coerces all values via Number(), silently dropping entries that are NaN or negative 1. If any user-provided usage key survives this validation, automatic tokenization is skipped entirely.

Model Lookup, Pricing Tier Selection, and Cost Calculation#

Model Lookup#

getGenerationUsage() only calls findModel when provided_model_name is set; without a model, automatic tokenization and pricing are both skipped 3.

findModel() resolves via a three-layer cache 3:

1. L1 — in-process TTL cache (default 10 s)
2. L2 — Redis (caches model + pricing tiers together)
3. Postgres — regex match_pattern field on the Model table

Unknown models are cached with a NOT_FOUND_TOKEN sentinel to prevent repeated DB hits.

Automatic Tokenization Fallback#

Auto-tokenization only runs when 3:

• A model was resolved
• provided_usage_details is empty
• Observation level is not ERROR

tokenCountAsync runs in a non-blocking worker thread pool, with a synchronous tokenCount fallback. Both dispatch on model.tokenizerId (e.g., OpenAI tiktoken, Claude). Failure produces empty usage_details; ingestion is never blocked.

Pricing Tier Selection#

matchPricingTier() evaluates non-default tiers in priority order with AND logic 4. Each condition:

1. Compiles usageDetailPattern into a regex
2. Sums all usage_details keys matching the pattern
3. Compares the sum to a threshold using an operator (gt, gte, lt, lte, eq, neq)

The first tier where all conditions pass is used; otherwise the default tier is the fallback. Pricing tiers are stored in the PricingTier table (Prisma) with a Price entry per usageType, each holding a Decimal price value 5.

Exact-Match Cost Calculation#

calculateUsageCosts() iterates usageUnits key-by-key and looks up the matching price via price.usageType === key — a strict exact string match 6. Cost for each key = price × units.

User override: if provided_cost_details contains any key, all automatic cost calculation is bypassed and the user values are used as-is 3.

Background Cost Recomputation#

When model pricing definitions change, historical observations need cost recalculation. AddGenerationsCostBackfill handles this as a Postgres-side background migration 7.

How it works:

• Processes observations in batches of 1000 (configurable), ordered by start_time DESC using the last batch row's start_time as a cursor 8
• Each batch uses a LATERAL JOIN to find the matching model by internal_model name, unit, project scope, and start_date ≤ observation.start_time 9
• Computes calculated_input_cost = prompt_tokens * input_price, calculated_output_cost = completion_tokens * output_price, and calculated_total_cost (using total_price * total_tokens when available, otherwise summing input + output) 10
• Respects user-provided costs: only fills calculated_* columns when all three of input_cost, output_cost, and total_cost are NULL 11
• Adds a temporary tmp_has_calculated_cost BOOLEAN column for progress tracking; drops it on completion 12
• Sets statement_timeout = '19min' and restores the original on exit 13
• Accepts maxRowsToProcess and maxDate CLI arguments for staged runs 14

The migration implements IBackgroundMigration and can also be executed directly as a CLI script 15.