Failure Mode Analysis (FMA)#

This folder is a self-contained toolkit to run failure mode analyses on Dosu messages. It provides three entry points:

• agent/failure_mode_analysis/run_pipeline.py
• agent/failure_mode_analysis/run_custom_question.py
• agent/failure_mode_analysis/eval.py

Use -h/--help on any entry point to see full options and examples:

python -m agent.failure_mode_analysis.run_pipeline -h
python -m agent.failure_mode_analysis.run_custom_question -h
python -m agent.failure_mode_analysis.eval -h

Overview#

FMA runs a multi-step LLM pipeline over a dataset of Dosu messages to:

1. detect failure instances (IFI),
2. check whether key information was present (KIP), and
3. optionally produce explanations for retrieval and synthesis behavior.

You can run the pipeline over a sampled default dataset (date-ranged selection) or over explicit message_id values that you provide.

Dataset Selection#

By default, run_pipeline.py builds a dataset by sampling Dosu messages in a date range that meet these criteria:

• Public repositories only
• The GitHub thread has a maintainer reply following the Dosu message
• The Dosu message received fewer than 3 distinct types of negative feedback

For each selected message_id, FMA builds a row with:

• The conversation before the Dosu message (messages_before_dosu)
• The Dosu message content (dosu_message)
• The conversation after the Dosu message (messages_after_dosu)
• The Dosu agent tool log (optionally as JSON or Markdown)

You can also run FMA on explicit message IDs using --message-id. In this mode:

• Public repositories are still required
• Maintainer reply and feedback presence are NOT required
• All other formatting flags still apply

Date range defaults (when not overridden):

• Start: 2025-06-12T00:00:00Z
• End: 2025-07-09T23:59:59.999999Z

You can override with --date-range START END using ISO date (YYYY-MM-DD) or ISO datetime.

Pipeline Steps#

The FMA pipeline has four steps; the last two run only when IFI is true:

1. Is Failure Instance (ifi: bool)

   • True if the maintainer explicitly disagrees with the Dosu message; otherwise false.
   • You can override this definition via CLI (see below) to turn IFI into a semantic filter for custom failure notions.

2. Key Information Present (kip: bool)

   • Runs only if ifi is true.
   • True if the key information needed to correctly answer the OP’s question was present in the retrieved context.

3. Explanation Retrieval (explanation_retrieval: str)

   • Runs only if ifi is true (regardless of KIP).
   • Returns a dense natural-language explanation of how/where the key information was retrieved.

4. Explanation Synthesis (explanation_synthesis: str | None)

   • Runs only if ifi is true (regardless of KIP).
   • Returns a dense natural-language explanation of any synthesis issues in the agent’s trajectory, or None if there are none.

Supported modes:

• ifi — run IFI only
• ifi_kip — run IFI then KIP
• full — run IFI, KIP, Explanation Retrieval, and Explanation Synthesis

Chain-of-thought (rationales): add --cot to include ifi_rationale and kip_rationale fields in outputs.

Custom IFI definition: provide an inline string or a file to redefine what counts as a “failure”. Examples:

--ifi-definition "Set ifi=true if the Dosu message contains strong claims without evidence; else false."
--ifi-definition-file path/to/ifi_definition.txt

Running the Pipeline#

Basic examples:

# Full FMA on the first 60 valid rows in date range
python -m agent.failure_mode_analysis.run_pipeline \
  --mode full \
  --limit 60 \
  --date-range 2025-06-12 2025-07-09

# IFI + KIP only on 30 rows
python -m agent.failure_mode_analysis.run_pipeline \
  --mode ifi_kip \
  --limit 30

# IFI only with log truncation safeguards
python -m agent.failure_mode_analysis.run_pipeline \
  --mode ifi \
  --limit 20 \
  --truncate-logs --indicate-truncated

# Run on explicit message IDs (no --limit required)
python -m agent.failure_mode_analysis.run_pipeline \
  --mode full \
  --message-id <uuid1> <uuid2> <uuid3>

Selected outputs are written to agent/failure_mode_analysis/results/<run_id>/:

• run.json — run metadata (mode, model, limits, flags, date range, etc.)
• rows.jsonl or rows_cot.jsonl — per-row outputs (the latter when --cot is set)

Filtering saves:

• --save-non-error-rows — write only rows without JSON/step errors
• --save-non-error-ifi-true — write only rows without errors and with ifi=true

Evaluation (eval.py)#

Use eval.py to score IFI/KIP correctness against a small ground truth set. The expected CSV format is:

• message_id — UUID
• IFI — T or F
• KIP — T, F, or N/A (must be N/A when IFI is F)

We provide a ground truth set of 30 labelled rows at agent/failure_mode_analysis/ground_truths.csv.
While you may specify your own ground truth file, the message_ids of all the rows must belong to the same contiguous date range, and that date range must be specified.
We recommend simply running evals on just the default ground truth data, which can be done simply by not specifying a ground truth file in the CLI.

Examples:

# One run; print summary to stdout; use default labels
python -m agent.failure_mode_analysis.eval

# Multiple runs; save scores and per-row outputs; use default labels
python -m agent.failure_mode_analysis.eval \
  --runs 5 --save-scores --save-rows

Saved evaluation artifacts go to agent/failure_mode_analysis/results/evals/... by default, or to --out-dir if provided.

All parity flags like --truncate-logs, --md-tool-log, --remove-quotes, --remove-links, and --max-workers are supported here as well.

Custom Questions (run_custom_question.py)#

Ask an LLM any custom question about one or more Dosu messages. The script builds the relevant context (messages before/after and tool log) and renders a prompt.

Examples:

# Plain-text answer
python -m agent.failure_mode_analysis.run_custom_question \
  --message-id <uuid> \
  --question "Summarize the message in 30 words"

# JSON answer with an explicit schema (inline)
python -m agent.failure_mode_analysis.run_custom_question \
  --message-id <uuid1> <uuid2> \
  --question "Is this actionable? Return yes/no and why." \
  --response-format json \
  --json-schema '{"type":"object","properties":{"yes":{"type":"boolean"},"why":{"type":"string"}},"required":["yes","why"],"additionalProperties":false}' \
  --out agent/failure_mode_analysis/results/custom/answers.jsonl

# JSON answer with schema and question loaded from files
python -m agent.failure_mode_analysis.run_custom_question \
  --message-id <uuid> \
  --question-file path/to/question.txt \
  --response-format json \
  --json-schema-file path/to/schema.json

If you pass a single --message-id and no --out, results are printed to stdout. For multiple IDs or when --out is provided, results are written as JSONL.

Common Flags and Safeguards#

Some models can get stuck or degrade with long/repetitive contexts (especially logs). These options help:

• --truncate-logs — Truncate repetitive sequences in message contexts
• --indicate-truncated — Print the message_id of any truncated row
• --remove-quotes — Remove quote fields from tool logs
• --md-tool-log — Render tool logs as Markdown (instead of JSON)
• --remove-links — Strip URLs and unwrap Markdown links from contexts

LLM (Vertex AI)#

Inference runs through Google Vertex AI because we have Llama credits and do not yet have Llama configured on Azure. Defaults are set in agent/failure_mode_analysis/vertex_ai_client.py:

• project_id: dosu-staging
• region: us-east5
• model: meta/llama-4-maverick-17b-128e-instruct-maas
• max_tokens: 256 (to reduce looping on long contexts)
• frequency_penalty: None (override via code if needed)

You can change the model and hyperparameters by editing VertexAIConfig. In practice, google/gemini-2.5-pro has worked well for us.

Using the pipeline (run_pipeline.py)#

The pipeline runs IFI (Is Failure Instance), KIP (Key Information is Present), and optional explanation steps over a sampled dataset and writes results under results/.

Basic usage:

cd backend/agent/failure_mode_analysis
python run_pipeline.py --mode full --limit 50 --max-workers 4

Common flags:

• --mode (ifi | ifi_kip | full): which steps to run. Default: full.
• --limit (int, required): number of rows to process from the sampled range.
• --max-workers (int): max parallel workers. Default: 1.
• --date-range (START END): override default date range. Accepts ISO dates (YYYY-MM-DD) or datetimes. Default: 2025-06-12 to 2025-07-09.
• --truncate-logs: truncate repetitive patterns in messages_before_dosu to reduce tokens.
• --indicate-truncated: print message_id for truncated rows (works with --truncate-logs).
• --keep-quotes: keep quote fields in tool logs (removed by default).
• --md-tool-log: render tool logs as Markdown in prompts (instead of JSON).
• --cot: include rationale fields for IFI/KIP (chain-of-thought expected in JSON).

Outputs:

• Results are written to results/<run_id>/rows.jsonl (or rows_cot.jsonl when --cot is used).
• A run.json metadata file is also created in the same directory.

Setup#

Environment variables (add to .env.development.local or equivalent):

LANGSMITH_PROJECT=llama-rca
LANGSMITH_API_KEY=<your_langsmith_api_key>

WARNINGS:

• Results are written under agent/failure_mode_analysis/results/. This directory is NOT ignored in gitignore by default — avoid committing large runs unless intended.
• The dataset builders and entry points rely on your configured database connectio. Ensure the app can connect to your staging/production/local DB before running. To the best of my knowledge, all the queries associated with this folder are read only, but I always only had read only permissions when developing and testing. If you plan on using this against prod data, please use a connection that has a read only role.

File Map#

• agent/failure_mode_analysis/run_pipeline.py — Main FMA pipeline over a dataset or message IDs
• agent/failure_mode_analysis/eval.py — IFI/KIP evaluation against ground truth
• agent/failure_mode_analysis/run_custom_question.py — Ask arbitrary questions over message context
• agent/failure_mode_analysis/helpers/* — Prompt rendering, response schemas, CLI helpers
• agent/failure_mode_analysis/dataset/* — Dataset sampling, preprocessing, and formatting
• agent/failure_mode_analysis/prompts/* — Prompt templates used by the steps
• agent/failure_mode_analysis/vertex_ai_client.py — Vertex AI client and defaults
• agent/failure_mode_analysis/ground_truths.csv — Example ground truth for eval.py