CI Diff-Gating Workflow for Deterministic PR Validation#

The CI diff-gating workflow in insideLLMs enforces deterministic validation of pull requests by running a minimal, reproducible harness on both the base commit (typically the main branch) and the candidate PR commit. It then compares the behavioral outputs to detect any drift, including regressions, changes, or missing/extra records. This process ensures that all behavioral changes introduced by a PR are surfaced and reviewed before merging, supporting reproducibility and preventing unintended model or logic regressions.

This workflow is implemented at .github/workflows/diff-gate.yml and uses the repository root action (invoked via uses: ./) to automate baseline and candidate harness execution and diff computation. PR commenting is handled by a separate trusted workflow at .github/workflows/diff-gate-comment.yml that runs in a workflow_run context with pull-requests: write permission. The workflow was added as part of the synthesized branch effort to standardize CI quality gates.

Workflow Overview#

The workflow performs the following steps:

1. Baseline Run: The CI system checks out the PR base commit and runs the harness using the fixed configuration (ci/harness.yaml), saving results to the baseline run directory.
2. Candidate Run: The workflow returns to the PR head commit and runs the same harness, saving results to the candidate run directory.
3. Health Checking: Both manifests and record streams are validated for completeness and successful execution before comparison.
4. Diffing: The outputs of the two runs are compared using the insidellms diff command with the configured failure mode (default: --fail-on-any-difference, which is stricter than the legacy --fail-on-regressions).
5. Gating: If behavioral differences are detected or if either run is unhealthy, the CI job fails with an appropriate exit code, blocking the PR until the differences are reviewed and addressed.
6. Artifact Upload: The workflow uploads both run directories and a small PR report artifact for use by the comment workflow.
7. PR Comment: A separate workflow_run workflow (diff-gate-comment.yml) downloads the allowlisted PR report, validates it, and posts or updates a sticky comment with integer count summaries. This workflow only executes trusted code from the default branch and never evaluates candidate code or extracts full artifacts.

Example workflow commands executed by the action:

insidellms harness ci/harness.yaml --run-dir <baseline-dir>
insidellms harness ci/harness.yaml --run-dir <candidate-dir>
insidellms diff <baseline-dir> <candidate-dir> --fail-on-any-difference --format json

Deterministic Harness and Artifacts#

The harness configuration used for CI diff-gating (ci/harness.yaml) is designed for determinism. It uses a DummyModel (no API keys required) and a small, fixed dataset (ci/harness_dataset.jsonl) with a set of probes (logic, attack, instruction_following, code_generation). The harness spine—run, records, report, diff—is deterministic, including run IDs and timestamps, ensuring reliable and reproducible diffing in CI.

Example ci/harness.yaml excerpt:

models:
  - type: dummy
    args: {}
probes:
  - type: logic
    args: {}
  - type: attack
    args:
      attack_type: prompt_injection
  - type: instruction_following
    args: {}
  - type: code_generation
    args:
      language: python
dataset:
  format: jsonl
  path: ci/harness_dataset.jsonl
max_examples: 3
confidence_level: 0.95
report_title: CI Diff Gate Harness

View full ci/harness.yaml

Diff Gating Flags and Exit Codes#

The action uses --fail-on-any-difference by default when fail-on-changes: "true" is set (which is the repository default). This flag enforces strict gating on any behavioral drift between the base and candidate runs. The diff command will exit with exit code 2 if it detects any of the following:

• Regressions (worse scores or failed statuses)
• Improvements (better scores or status changes)
• Changes (metric mismatches, output changes, status changes)
• Records only present in the baseline or only in the candidate (missing/extra records)
• Trace drifts or trajectory changes (if trace comparison is enabled)

This is the strictest mode and surfaces all behavioral differences. The legacy --fail-on-changes flag has narrower behavior and does not fail on improvements or trace-only differences.

Exit codes:

• 0: No differences detected, both runs healthy
• 1: Unhealthy runs, invalid evidence, or health check failures
• 2: Differences detected according to the configured policy

If any differences are found or if runs are unhealthy, the CI job fails, blocking the PR until the differences are reviewed and resolved.

Types of Behavioral Changes Detected#

The diff command compares two run directories and reports:

• Regressions: Cases where the candidate run performs worse than the baseline (e.g., lower scores, failed statuses).
• Improvements: Cases where the candidate run performs better.
• Changes: Metric mismatches, output changes, or status changes that are not strictly regressions or improvements.
• Missing/Extra Records: Records present only in the baseline or only in the candidate.
• Trace Drifts and Violations: (If enabled) Differences in execution traces or contract violations.

All these are surfaced in the diff report, and with --fail-on-changes, any of them will fail the CI job.

Contributor Guidance: Handling Diff-Gating Failures#

If your PR fails due to the diff-gating workflow, it means that behavioral changes have been detected between your branch and the base branch. To resolve:

1. Review the diff output in the CI logs to identify the specific changes.
2. Determine whether the changes are intended and justified. If so, document the rationale in your PR description.
3. If the changes are unintended, update your code to restore behavioral consistency.
4. Re-run the CI workflow to ensure the diff-gating check passes.

This process ensures that all behavioral changes are visible, reviewed, and justified before merging.

GitHub Actions Integration#

The repository uses a split workflow architecture for diff-gating:

• Evaluation workflow (.github/workflows/diff-gate.yml): Runs candidate code with read-only permissions
• Comment workflow (.github/workflows/diff-gate-comment.yml): Runs trusted code with write permissions

Evaluation Workflow Architecture#

The main diff-gate workflow uses the repository root action (invoked via uses: ./) with minimal permissions:

name: Diff Gate

on:
  pull_request:
  workflow_dispatch:

permissions:
  contents: read

jobs:
  diff-gate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v7
        with:
          fetch-depth: 0
          persist-credentials: false

      - name: Run insideLLMs action
        id: insidellms
        uses: ./
        with:
          harness-config: ci/harness.yaml
          fail-on-changes: "true"

      - name: Upload safe comment report
        if: always()
        uses: actions/upload-artifact@v7
        with:
          name: insidellms-pr-report
          path: ${{ steps.insidellms.outputs.pr-report-json }}

      - name: Upload diff artifact
        if: always()
        uses: actions/upload-artifact@v7
        with:
          name: insidellms-diff-${{ github.sha }}
          path: |
            ${{ steps.insidellms.outputs.diff-json }}
            ${{ steps.insidellms.outputs.baseline-run-dir }}
            ${{ steps.insidellms.outputs.candidate-run-dir }}

The evaluation workflow:

• Has only contents: read permission (no pull-requests: write)
• Uses persist-credentials: false on checkout
• Executes candidate PR code, including fork code
• Uploads run directories and a small allowlisted PR report as artifacts
• Fails on unhealthy runs or detected differences

Action Inputs#

• harness-config: Path to the harness YAML config file (default: ci/harness.yaml)
• baseline-ref: Git ref for the baseline run (default: PR base)
• python-version: Python version to use (default: 3.11)
• install-extras: Optional extras to install (e.g., dev,nlp,visualization)
• run-args: Extra arguments forwarded to each harness invocation
• diff-args: Extra arguments forwarded to diff
• fail-on-changes: When true, use --fail-on-any-difference (default: true)
• post-pr-comment and comment-on-forks: Deprecated; emit a migration warning when enabled

Action Outputs#

• diff-json: Path to the diff report or run-health error report
• baseline-run-dir: Path to baseline run artifacts
• candidate-run-dir: Path to candidate run artifacts
• diff-exit-code: Gate exit code (1 for unhealthy runs/errors, or the diff policy exit code)
• baseline-commit: Resolved baseline commit
• is-fork-pr: Whether the pull request head is from a fork
• comment-status: Always returns disabled-inline-commenting
• pr-report-json: Path to the small allowlisted report for the trusted comment workflow

Comment Workflow#

The separate comment workflow (.github/workflows/diff-gate-comment.yml) runs on workflow_run completion with pull-requests: write permission. It:

• Only executes trusted code from the default branch
• Downloads the single small insidellms-pr-report artifact
• Validates fixed fields (PR number, head SHA, exit code, integer counts)
• Posts or updates a sticky PR comment with count summaries
• Never checks out candidate code or extracts full run artifacts
• Skips stale commits or runs without an unambiguous associated PR

Retired Composite Action#

The legacy composite action at .github/actions/diff-gate/ has been retired. It now exits with error code 1 and directs users to the repository root action and the split comment workflow. The retired action previously attempted to run harnesses and post PR comments inline, which created security concerns when evaluating untrusted fork code with write permissions.

Migration notice: If you were using .github/actions/diff-gate/ directly, switch to the repository root action (uses: ./) and adopt the split workflow pattern shown above. The root action and comment workflow are the maintained implementation.

For more details, see CI setup documentation and GitHub Action documentation.