Leaderboard Aggregation#

TruLens computes the dashboard leaderboard — records count, average latency, total tokens, total cost, and mean feedback scores per app/version — through two independent code paths controlled by the TRULENS_OTEL_TRACING environment variable. The two paths share an output schema but differ in where aggregation happens (Python vs. SQL), which JSON columns they read, and which bugs they carry.

Two Parallel Implementations#

Pre-OTel path (ORM-based)#

When OTel tracing is disabled, the entry point is DBConnector.get_leaderboard() in src/core/trulens/core/database/connector/base.py. This method:

1. Calls get_records_and_feedback() to fetch the full record payload for each row.
2. Parses app_json to extract app_name and app_version.
3. Uses df.groupby(["app_name", "app_version"])[col_agg_list].mean() to compute aggregated feedback scores and latency 1.

The cost and token data live in the cost_json column of the Record ORM table. Both cost_json and perf_json are declared as TYPE_JSON = Text (plain SQL TEXT) 2 3. Because these are Text columns, SQLAlchemy's [] subscript operator raises NotImplementedError — a critical issue detailed below.

The Cost schema model 4 defines the fields serialised into cost_json: n_tokens, cost, cost_currency (default "USD"), plus various request/token count fields. Currency addition raises ValueError when summing costs with different currencies 5.

OTel path (event-based)#

When OTel tracing is enabled, aggregation runs via _get_leaderboard_aggregates_otel() introduced in PR #2396. This method queries the Event table's RECORD_ROOT and EVAL_ROOT spans using SQL GROUP BY entirely at the database layer, then pivots feedback scores in Python. Key design points:

• Sum for total cost (sa.func.sum(...)) and total tokens.
• Avg for latency and per-metric feedback scores (sa.func.avg(...)).
• Currency handling: extracts SpanAttributes.COST.CURRENCY per span, defaults to USD via sa.func.coalesce(sa.func.max(currency_col), sa.literal("USD")), then splits the cost column post-query into "Total Cost (USD)" and "Total Cost (Snowflake Credits)" columns 6.
• Dialect-aware JSON extraction: uses _json_extract_otel(), which dispatches to json_extract_path_text for Snowflake and json_extract(col, '$.path') for all other dialects. The Event table's columns are typed as sa.JSON, so native subscripting works correctly.
• Latency computation is dialect-specific at query-build time: julianday arithmetic for SQLite/generic DBs, extract('epoch', ...) for PostgreSQL, timestampdiff for Snowflake 6.

Performance gains over the pre-OTel Python-side approach (benchmarked on 10k records): 2.7–5.2× faster depending on number of app versions 6.

Known Bugs#

Pre-OTel path: NotImplementedError on cost_json (critical)#

Bug (Issue #2729, 7): _get_leaderboard_aggregates_pre_otel() attempted to use SQLAlchemy [] subscript syntax on cost_json (e.g., Record.cost_json["n_tokens"].as_float()). Because cost_json is Text, not a native JSON column, SQLAlchemy raises NotImplementedError: Operator 'getitem' is not supported on this expression at statement-construction time — before any database query runs. The same method also referenced a non-existent Record.latency column. The failure is unconditional: the pre-OTel leaderboard path could never succeed.

Fix (PR #2730): Replaced the broken SQL subscript expressions with dialect-aware _json_path_expr() calls (reusing the existing helper) to extract cost_json.n_tokens and cost_json.cost at the SQL level, and computed latency from perf_json.start_time/end_time using julianday (SQLite) or extract('epoch') (PostgreSQL). Empty-result handling was also improved by returning an explicitly-typed empty DataFrame instead of letting downstream code fail on a missing-column error.

Introduced by: PR #2396, which added the SQL-level aggregation path but used the wrong column type semantics for the pre-OTel Record table.

OTel path: unaffected by the above#

The OTel path's _json_extract_otel() helper operates on sa.JSON-typed Event table columns, so subscript access works correctly. The _json_extract_otel method validates column existence and type before building the expression 8.

Key Source References#