CLI Diff Command Enhancements#

The insidellms diff command compares two run directories (each containing a records.jsonl file) and reports behavioral regressions, improvements, and other changes between them. Recent enhancements have introduced stricter validation options, most notably the --fail-on-changes flag, which is designed for robust CI gating and validation workflows.

--fail-on-changes Flag#

The --fail-on-changes flag causes the diff command to exit with a non-zero status if any differences are detected between the baseline and candidate runs. This includes regressions, other changes (such as metric mismatches or output differences), and missing or extra records. This strict mode is particularly useful for continuous integration (CI) pipelines, where any behavioral drift must be surfaced and gated before merging changes.

Other related flags include:

• --fail-on-regressions: Fails only if regressions are detected.
• --fail-on-trace-violations: Fails if trace violations increase.
• --fail-on-trace-drift: Fails if trace fingerprints differ, even if output text is unchanged.

The diff command supports both human-readable text output and machine-readable JSON output (with --format json and --output <file> options) for integration with CI systems and dashboards. The command summarizes counts and details for regressions, improvements, other changes, trace drifts, and trace violation increases, with configurable limits on displayed items.

Example: Strict Diff Validation#

insidellms diff ./runs/baseline ./runs/candidate --fail-on-changes

If any differences are found, the command exits with code 2, causing the CI job to fail and preventing the merge of a pull request that introduces behavioral drift [source].

Example: Strict Diff Validation#

Example: Trace Drift and Violation Gating#

The diff command now supports additional flags for even stricter CI/CD gating:

• --fail-on-trace-drift: Fails if any trace fingerprints differ between baseline and candidate runs, even if the output text is unchanged. This detects any behavioral drift at the trace level.
• --fail-on-trace-violations: Fails if the number of trace violations increases in the candidate run compared to the baseline. This ensures that new contract violations are surfaced and gated.

Example: Enforcing Trace Drift and Violation Checks#

insidellms diff ./runs/baseline ./runs/candidate --fail-on-trace-drift --fail-on-trace-violations

If any trace fingerprints change or if trace violations increase, the command exits with a non-zero status (exit code 3 or 4), causing the CI job to fail. This provides an additional layer of determinism and contract enforcement beyond output text comparison.

These flags can be combined with --fail-on-changes or used independently, depending on the desired level of strictness in your CI/CD pipeline.

CI Diff-Gating Workflow#

The CI diff-gating workflow leverages the deterministic nature of the insideLLMs harness and diff tooling to ensure reproducible, reliable validation of pull requests. The workflow is designed to catch any unintended changes in model behavior or outputs before code is merged.

Workflow Steps#

1. Baseline Generation: Run and save a baseline using the harness. The baseline's records.jsonl is either committed to the repository or stored as a CI artifact.
2. Candidate Run: On each pull request, run the harness again on the candidate code (the PR branch), producing a new records.jsonl.
3. Validation: Optionally, validate both baseline and candidate run directories using insidellms validate to ensure schema compliance.
4. Diff and Gate: Use the diff command with --fail-on-changes to compare the baseline and candidate runs. If any differences are detected, the CI job fails.

Example: CI Diff-Gating Commands#

# Run harness on baseline (typically on main branch or a known-good SHA)
insidellms harness ci/harness.yaml --run-dir .tmp/runs/base --skip-report

# Run harness on candidate (pull request branch)
insidellms harness ci/harness.yaml --run-dir .tmp/runs/head --skip-report

# Optionally validate outputs
insidellms validate .tmp/runs/base
insidellms validate .tmp/runs/head

# Diff and gate on any changes
insidellms diff .tmp/runs/base .tmp/runs/head --fail-on-changes

If the diff command detects any differences, it exits with a non-zero status, causing the CI pipeline to fail and blocking the PR [source].

GitHub Actions Integration#

Legacy Composite Action (Retired)#

The composite action at .github/actions/diff-gate/ has been retired and is no longer functional. As of the September 2026 repository audit remediation, this action now exits immediately with an error message and does not execute any harness runs or diff computations.

What changed:

• The composite action no longer runs baseline or candidate harnesses
• PR comment posting has been removed from the action (moved to a separate diff-gate-comment.yml workflow)
• The action's diff-gate.sh script has been replaced with a fail-closed stub that exits with code 1
• All documented inputs (config, baseline-ref, fail-on, python-version, extra-pip-args, harness-args, comment) no longer have any effect
• The documented outputs (exit-code and diff-json) have been replaced with "Unavailable" stubs

Why it was retired:
The workflow architecture has been restructured to separate diff computation from PR commenting for security and maintainability. This separation ensures that:

• Evaluation runs execute with minimal read-only permissions (contents: read)
• PR commenting is handled by a separate trusted workflow_run workflow on the default branch
• Untrusted candidate code (including fork PRs) never receives write tokens or repository secrets

Current implementation:
The diff-gating functionality has moved to the repository-root action (at the repository root action.yml) and is paired with a separate trusted commenting workflow. See:

• .github/workflows/diff-gate.yml — runs the harness and diff with read-only permissions
• .github/workflows/diff-gate-comment.yml — trusted workflow that posts PR comments based on validated reports
• docs/GITHUB_ACTION.md — documentation for the current repository-root action

Users of the legacy composite action must migrate to the repository-root action and separate comment workflow to restore diff-gating functionality.

Example: Legacy Composite Action (No Longer Works)#

The following example used the retired composite action and will no longer function:

# ❌ THIS NO LONGER WORKS — the action at .github/actions/diff-gate/ has been retired
name: Diff Gate
on:
  pull_request:

permissions:
  contents: read
  pull-requests: write

jobs:
  diff-gate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v6
        with:
          fetch-depth: 0

      - uses: ./.github/actions/diff-gate # ❌ This action is retired
        with:
          config: ci/harness.yaml
          fail-on: regressions

Retired Inputs (No Longer Functional)#

The following inputs were documented for the legacy composite action but no longer have any effect:

• config: Path to the harness YAML config file (default: ci/harness.yaml)
• baseline-ref: Git ref for the baseline run (default: PR base)
• fail-on: Failure mode—regressions (default), changes, or none
• python-version: Python version to use (default: 3.12)
• extra-pip-args: Extra arguments passed to pip install
• harness-args: Extra arguments passed to insidellms harness (default: --skip-report --deterministic-artifacts)
• comment: Whether to post a PR comment with the diff summary (default: true)

Retired Outputs (No Longer Available)#

The following outputs have been replaced with "Unavailable" stubs:

• exit-code: Exit code from the diff command (0 = pass, non-zero = fail)
• diff-json: Path to the JSON diff report file

Migration path: Use the repository-root action documented in docs/GITHUB_ACTION.md and the separate comment workflow pattern shown in .github/workflows/diff-gate.yml and .github/workflows/diff-gate-comment.yml.

Deterministic Harness-Based Validation#

The harness and diff workflow is deterministic by design, including run IDs and timestamps. This ensures that repeated runs with the same configuration and data produce identical outputs, making the diff-gating process reliable and reproducible [source].

Artifacts and Reporting#

Typical artifacts in this workflow include:

• records.jsonl: Canonical run log (one JSON object per example/model/probe).
• summary.json: Aggregated metrics and confidence intervals.
• report.html: Human-readable comparison report.
• diff.json: Machine-readable diff report (produced with --format json --output diff.json).

Integration with Testing and Validation Strategy#

The diff-gating workflow is part of a broader, deterministic testing and validation strategy in insideLLMs. The typical lifecycle is:

• Run/Harness: Execute experiments or harnesses to generate run artifacts.
• Validate: Check run directories for schema compliance.
• Report: Generate human- or machine-readable reports.
• Diff: Compare baseline and candidate runs for behavioral drift.
• Decide: Gate merges or deployments based on diff results.

Test coverage includes dedicated tests for diff gating and the --fail-on-changes flag, ensuring correctness and reliability of these features [source].

For more details, see the README CI Diff-Gating section and Results and Reports documentation.