Groundedness Evaluation#

Groundedness measures whether an LLM's response is supported by the source material it claims to be based on — a key RAG quality metric. TruLens implements this in LLMProvider.groundedness_measure_with_cot_reasons inside src/feedback/trulens/feedback/llm_provider.py. The function takes a source string (the retrieved context) and a statement string (the LLM response), and returns a (float, dict) tuple: a normalized 0–1 score and per-statement chain-of-thought reasons.

A second variant, groundedness_measure_with_cot_reasons_consider_answerability, also accepts a question argument and adjusts scoring for abstentions based on whether the source could actually answer the question (see Answerability and Abstention below).

Pipeline Overview#

The evaluation runs in four sequential stages:

statement ──► sentence splitting ──► trivial filtering ──► per-statement LLM scoring ──► average

1. Sentence Splitting#

The statement is split into individual hypotheses 1. Two modes are available, controlled by GroundednessConfigs.use_sent_tokenize 2:

• use_sent_tokenize=True (default): Uses NLTK's sent_tokenize (punkt tokenizer) — fast and cost-free 3.
• use_sent_tokenize=False: Calls the LLM with LLM_GROUNDEDNESS_SENTENCES_SPLITTER, which maps to Groundedness.sentences_splitter_prompt. Results are split on newlines. This is more accurate for complex text but incurs extra LLM cost and may hit context-window limits.

2. Trivial Statement Filtering#

When GroundednessConfigs.filter_trivial_statements=True (default), _remove_trivial_statements is called. It sends the full statement list to the LLM with the Trivial class prompts:

• System: "Identify and remove sentences that are stylistic, contain trivial pleasantries, or lack substantive information relevant to the main content. Respond only with a list of the remaining statements in the format of a python list of strings." 4
• User: ALL STATEMENTS: {statements} \n IMPORTANT STATEMENTS: 5

The LLM response is eval()-ed to produce the filtered list 6. If parsing fails, a warning is emitted and all statements are kept. If all statements are filtered out, the function returns (0.0, {"reason": "No non-trivial statements to evaluate"}) immediately 7.

Why this matters: Without filtering, filler sentences like "Hi. I'm here to help." would dilute scores for factual claims in the same response 8.

3. Per-Statement LLM Scoring#

Each remaining hypothesis is scored in parallel via ThreadPoolExecutor 9. For each hypothesis:

• The system prompt comes from Groundedness.generate_system_prompt. It instructs the LLM to act as an "INFORMATION OVERLAP classifier" scoring on a 0–3 Likert scale 10.
• The user prompt is LLM_GROUNDEDNESS_USER / Groundedness.user_prompt, which passes SOURCE: {premise} and Statement: {hypothesis} and requests a structured Criteria / Supporting Evidence / Score reply.
• Scoring criteria from Groundedness.criteria_template:
  • Directly supported → high score
  • Not supported → low score
  • Abstentions ("I don't know") → max score (treated as grounded)
  • Indirect/implicit evidence should not be penalized, but false positives should be guarded against

Scores are normalized from the raw 0–3 range to 0–1, and the score embedded in the reason string is also normalized for display 11.

4. Score Aggregation#

The final score is the mean of all per-statement scores 12. Reasons are concatenated as STATEMENT 0: ...\nSTATEMENT 1: ... in the returned dict 13.

Answerability and Abstention#

groundedness_measure_with_cot_reasons_consider_answerability adds two extra sub-evaluations per hypothesis 14:

1. Abstention detection via Abstention: Scores 0–1 whether the statement is a form of "I don't know."
2. Answerability check via Answerability: Scores 0–1 whether the source could answer the question.

Decision logic 15:

• Abstention + answerable source → score 0.0 ("Answerable abstention" — penalized)
• Abstention + unanswerable source → score 1.0 ("Unanswerable abstention" — grounded)
• Non-abstention → standard groundedness scoring

Configuration Reference#

GroundednessConfigs is defined at src/core/trulens/core/feedback/feedback.py:123 and passed to the Feedback constructor via groundedness_configs 16.

Key Source Files#