CI/CD#

Contents#

• Workflow map
• Branches and tags
• Promotion flow
• stream_name routing
• Event truth table
• Centralized CI — projectbluefin/actions
• Schedule ownership
• Renovate auto-merge pipeline
• Weekly release pipeline
• Release pipeline pitfalls
• GHCR Package Access
• SBOM rules
• Condition quick reference

Workflow map#

Branches and tags#

push to lts does not trigger any build workflow (no push: lts trigger exists in any caller). The merge itself fires only lifecycle-caller.yml.

Promotion flow (main→lts)#

promote-testing-to-main.yml maintains an always-open auto/promote-testing-to-main PR targeting lts. Merging it cuts a release — see docs/skills/release.md.

Critical: The caller passes source_branch: main and target_branch: lts. Without these, the reusable workflow defaults to testing → main, trees are always identical, and no PR is ever created.

1. PRs squash-merge to main.
2. sync-main-to-testing.yml mirrors main → testing, triggering the promote workflow.
3. Promote workflow compares main vs lts trees; rebuilds squash branch if different.
4. Maintainers merge promotion PR (2 approvals required) → execute-release.yml fires → :testing copied to :lts.

Never squash-merge the promotion PR — breaks merge base for future promotions.
Never merge lts→main.

stream_name — how tags are determined#

The 3 callers delegate entirely to projectbluefin/actions/.github/workflows/reusable-build.yml@v1. The key input is stream_name:

stream_name: ${{ github.ref == 'refs/heads/lts' && 'lts' || 'testing' }}

There is no separate publish: false gate. Callers always publish when they run. On PRs, the detect-changes job may skip the build entirely if no image-relevant files changed.

Event truth table#

Centralized CI — projectbluefin/actions#

Common CI/CD logic lives in reusable GitHub Actions at https://github.com/projectbluefin/actions (@v1).

Reusable workflow used by bluefin-lts callers#

projectbluefin/actions/.github/workflows/reusable-build.yml@v1

Inputs used by each caller:

• brand_name — image name (bluefin-lts, bluefin-lts-hwe, bluefin-gdx)
• stream_name — testing or lts
• image_flavors — '["main"]'
• architecture — '["x86_64"]'

HWE and GDX kernel selection#

HWE (bluefin-lts-hwe) and GDX (bluefin-gdx) use the Fedora CoreOS stable kernel, not the CentOS kernel. The Justfile resolves the current Fedora CoreOS stable version at build time:

skopeo inspect docker://quay.io/fedora/fedora-coreos:stable
# → derives Fedora version (e.g., 44) → selects coreos-stable-44 akmods

This means HWE/GDX kernels automatically track upstream as CoreOS advances Fedora versions — no manual pin bumps needed. Set COREOS_STABLE_VERSION=NN to override for testing.

Regular builds (bluefin-lts) use centos-10 akmods and the CentOS Stream kernel.

Shared composite actions in bluefin-lts#

Schedule ownership#

release-reminder.yml is the only scheduled workflow — posts a reminder on the promotion PR after 7 days open. Do not add schedule: triggers to the build callers.

Renovate auto-merge pipeline#

renovate-automerge.yml triggers on workflow_run: completed: "PR Validation — testsuite" and only proceeds when conclusion == 'success'. pr-testsuite is lint-only (COPR guard + validate-pr — no E2E smoke), so it completes in ~10 min and drives automerge reliably.

Flow:

1. Renovate opens PR → pr-testsuite.yml runs lint (~10 min)
2. renovate-automerge.yml triggers on workflow_run success → calls reusable-renovate-automerge.yml@v1
3. PR merges to main → build workflows fire on push: main → :testing published

Required status check (ruleset 4940669): Lint & syntax only. Builds are informational.

projectbluefin/actions refs — always use @v1, never SHA-pin#

All projectbluefin/actions refs in .github/workflows/ must use the @v1 managed tag. Never SHA-pin them.

• SHA-pinned internal refs require a Renovate bump PR in every caller before a fix takes effect — propagation lag is days, not minutes
• @v1 always points to the latest stable release; fixes in projectbluefin/actions propagate instantly to all callers
• AGENTS.md explicitly exempts projectbluefin/ refs from the SHA-pin requirement: "projectbluefin/ refs (@v1, @main) are intentional managed tags and are exempted."
• A pre-commit hook (no-sha-pins-for-internal-actions in .pre-commit-config.yaml) blocks any future SHA pin on projectbluefin/ actions at commit time

Handling stale Renovate SHA-bump branches after a bulk @v1 conversion#

After merging a bulk PR that converts projectbluefin/actions SHA pins → @v1, Renovate's in-flight SHA-bump branch becomes stale: it tries to replace @v1 with a specific SHA (going backwards). Fix:

git fetch origin
git checkout -B renovate/projectbluefinactions origin/main
git push origin renovate/projectbluefinactions --force

This resets the branch to main (empty diff). The open Renovate PR will show no changes and can be closed. Renovate will not re-open it since there are no SHA pins left to track.

Required status check (ruleset 4940669): Lint & syntax only. Builds are informational.

Renovate common image tracking — critical pattern#

ghcr.io/projectbluefin/common delivers first-party fixes (e.g. rechunker-group-fix, boot services) that are safety-critical for users. These must land in :testing automatically without human intervention.

The right configuration (renovate.json5):

1. Custom manager for image-versions.yaml — Renovate needs a regex manager to discover the image: / tag: / digest: block. Without it, common is invisible to Renovate:

{
  customType: 'regex',
  managerFilePatterns: ['/^image-versions\\.yaml$/'],
  matchStrings: [
    'image: (?<packageName>[^\\s]+)\\n\\s+tag: (?<currentValue>[^\\s]+)\\n\\s+digest: (?<currentDigest>sha256:[a-f0-9]+)',
  ],
  datasourceTemplate: 'docker',
  versioningTemplate: 'docker',
},

2. automerge: true + schedule: "at any time" for common — removes the default weekly delay from config:best-practices:

{
  "matchPackageNames": ["ghcr.io/projectbluefin/common"],
  "automerge": true,
  "schedule": ["at any time"]
},

Never use "enabled": false for common — this silently drops all critical fixes without any PR. The rechunker-group-fix was stuck in common for >2 days before manual intervention (issue ublue-os/bluefin-lts#918).

Release pipeline pitfalls#

org.opencontainers.image.revision is the CentOS base SHA, not the LTS repo SHA.
The label is inherited from quay.io/centos-bootc/centos-bootc:c10s. Never compare it to a projectbluefin/bluefin-lts commit SHA. The resolve job captures locked_main_sha from the GitHub API separately for the SHA guard and update-lts-branch.

GitHub Actions transitive failure propagation.
When a transitive ancestor fails (e.g. run-upgrade-test), GitHub skips all downstream jobs — even ones that only needs: a job that succeeded. Jobs after promote must use if: always() && needs.X.result == 'success', not just if: needs.X.result == 'success'.

lts branch is always "ahead" of main.
execute-release.yml fast-forwards lts after a release. If lts has diverged, the fast-forward fails — see release.md for the force-update command.

continue-on-error: true is not valid on uses: jobs.
actionlint rejects it. Make a job non-blocking by using if: always() && ... conditions on the jobs that depend on it.

SHA guard fires if main advances during the upgrade-test window (~10 min).
Re-run the promote workflow once main is quiet: gh workflow run "Promote testing to main" --repo projectbluefin/bluefin-lts

Branch protection on main: Required check Lint & syntax + linear history enforced. Matches projectbluefin/bluefin.

Release-generation pitfalls#

• workflow_run chaining does not propagate from GITHUB_TOKEN-dispatched workflows reliably enough for LTS release generation.
• pr-validate.yml in projectbluefin/testsuite is NOT a reusable workflow (no workflow_call). Never call it with uses:; it is the testsuite's own linter.

bluefin vs bluefin-lts quick reference#

These repo-local differences are the ones AI edits most often miss:

If you copy workflow snippets from bluefin, translate those paths before saving.

detect-changes filter override#

bluefin-lts uses different paths from bluefin. Always pass the filters input when using detect-changes here:

- uses: projectbluefin/actions/bootc-build/detect-changes@v1
  id: detect
  with:
    filters: |
      image:
        - 'Containerfile'
        - 'build_scripts/**'
        - 'system_files/**'
        - 'image-versions.yaml'
        - 'Justfile'
      nvidia:
        - 'Containerfile'

Using the default (bluefin paths: build_files/**, image-versions.yml) would silently skip builds when real image changes land.

validate-pr glob override#

Default shellcheck-glob watches build_files/**/*.sh. LTS must override:

- uses: projectbluefin/actions/bootc-build/validate-pr@v1
  with:
    shellcheck-glob: "build_scripts/**/*.sh"

The same build_scripts/ + image-versions.yaml distinction should stay consistent in AGENTS.md, .github/CODEOWNERS, and skill-drift.yml.

Rechunker — chunka@v1 (projectbluefin/actions)#

Rechunking is handled internally by projectbluefin/actions/.github/workflows/reusable-build.yml@v1. The local reusable-build-image.yml was deleted in PR #73.

force-compression: true: LTS uses CentOS Stream 10, which must migrate existing registry layers from gzip to zstd:chunked. Fedora consumers (bluefin) leave this at the default false because their images are already zstd:chunked.

Rechunk is skipped for stream_name == testing (on-push builds to main). Only production builds (stream_name: lts) rechunk.

What the action does internally (reference only — do not duplicate inline):

• buildah build with upstream Containerfile.splitter at the pinned chunkah SHA
• Key flags: --prune /sysroot/, --max-layers 128, --label ostree.commit-, --label ostree.final-diffid-
• -v $(pwd):/run/src --security-opt=label=disable for buildah < v1.44 bind-mount stability
• sudo podman save | podman load to transfer rechunked image from root (buildah) to user (podman) storage

Do not reproduce the inline buildah invocation. All details live in projectbluefin/actions/bootc-build/chunka/action.yml. If a flag needs changing, update the shared action.

GHCR Package Access — always use github.token, never custom PATs#

Policy: Custom tokens (PATs, PACKAGES_TOKEN, etc.) are an antipattern in this project.
When a workflow can't access a package, fix the package permissions — do not create a token.

All CI steps use only github.token (or secrets.GITHUB_TOKEN) for GHCR access.
If you see a PACKAGES_TOKEN or any other secret used for registry login, that is a bug.

Required package configuration (org admin, one-time setup)#

Three GHCR packages must be linked to projectbluefin/bluefin-lts and grant Actions write access:

On each settings page:

1. Connected repository → set to projectbluefin/bluefin-lts
2. Manage Actions access → "Add repository" → projectbluefin/bluefin-lts → Write

Once done, github.token from any bluefin-lts workflow has full package read/write — no PAT needed.

Note: bluefin-lts is currently (incorrectly) linked to projectbluefin/bluefin and bluefin-gdx
has no linked repo. Until an org admin fixes this, GHCR pushes from bluefin-lts workflows will fail
with DENIED. The fix is the two-step UI action above — not a new secret.

SBOM rules#

• Generate/attest SBOMs only on refs/heads/lts and when inputs.publish is true.
• All SBOM steps must keep continue-on-error: true.
• Failed SBOM attestation must never block image publishing.
• LTS uses SPDX JSON artifacts on the amd64 manifest digest; signing uses keyless cosign (Sigstore OIDC).

SBOM permission gotcha#

reusable-build.yml calls sudo -E just gen-sbom which creates sbom_out/ owned by root.
The subsequent sign-and-publish step runs without sudo and fails with permission denied on sbom_out/$IMAGE/sbom.json.

Fix is in the Justfile gen-sbom recipe: after syft writes the file, ownership is returned to the invoking user:

chown -R "${SUDO_UID:-$(id -u)}:${SUDO_GID:-$(id -g)}" "sbom_out/" 2>/dev/null || true

If you ever touch gen-sbom in the Justfile, preserve this line.

Condition quick reference#

If nothing is pushed, nothing should sign.

uupd install — COPR removed, use GitHub releases#

Context: The ublue-os/packages COPR epel-10 chroot was removed. Any build using the old COPR repo will get a 404 and fail. Do not restore that pattern.

Fix: Install uupd from its GitHub release tarball. Version is pinned in image-versions.yaml:

downloads:
  # renovate: datasource=github-releases depName=ublue-os/uupd
  uupd: "v1.4.0"

In build_scripts/20-packages.sh:

# yq is NOT available in the CentOS build container — use grep/sed
UUPD_VERSION=$(grep '^\s*uupd:' /run/context/image-versions.yaml | sed 's/.*"\(.*\)".*/\1/')
curl -fsSL "https://github.com/ublue-os/uupd/releases/download/${UUPD_VERSION}/uupd_Linux_x86_64.tar.gz" \
    | tar -xzf - -C /usr/bin uupd
chmod 0755 /usr/bin/uupd

The tarball ships binary only — fetch service files separately:

UUPD_RAW="https://raw.githubusercontent.com/ublue-os/uupd/${UUPD_VERSION}"
curl -fsSL "${UUPD_RAW}/uupd.service" -o /usr/lib/systemd/system/uupd.service
curl -fsSL "${UUPD_RAW}/uupd.timer" -o /usr/lib/systemd/system/uupd.timer

Three rules from this failure:

1. yq is not in the CentOS Stream build container. Use grep/sed/awk.
2. image-versions.yaml must be in the context stage (COPY image-versions.yaml /image-versions.yaml in Containerfile).
3. build_scripts/40-services.sh must run after the service files exist — install order matters.

PR-based release gate model#

Design: The always-open auto/promote-testing-to-main PR is the release gate. Merge it (requires 2 maintainers) to cut a release. Gate checks run automatically after each promotion update.

Key GITHUB_TOKEN limitations (both apply here):

1. GITHUB_TOKEN pushes to a branch do NOT fire pull_request: synchronize events — GitHub blocks this to prevent loops.
2. GITHUB_TOKEN cannot trigger workflow_dispatch events via the API (HTTP 403).

Solution: Inline the gate as a gate job inside promote-testing-to-main.yml rather than dispatching separately:

jobs:
  promote:
    outputs:
      sync_needed: ${{ steps.compare.outputs.sync_needed }}
      pr_number: ${{ steps.upsert.outputs.pr_number }}
      testing_sha: ${{ steps.compare.outputs.testing_sha }}
    ...

  gate:
    needs: [promote]
    if: needs.promote.outputs.sync_needed == 'true'
    uses: projectbluefin/actions/.github/workflows/reusable-release-gate.yml@main
    with:
      pr_number: ${{ needs.promote.outputs.pr_number }}
      head_sha: ${{ needs.promote.outputs.testing_sha }}
      ...

reusable-release-gate.yml inputs: pr_number and head_sha are optional overrides. When provided, they replace context.payload.pull_request?.number and github.event.pull_request.head.sha respectively — enabling the gate to run outside a pull_request event context.

Gate output on PR #125: Sticky comment with <!-- release-status-marker --> is posted/updated on the promotion PR. Labels release/ready or release/blocked are auto-applied.

E2E gate: The gate checks for a post-testing-e2e workflow run on the PR's head SHA. When there is none (fresh image builds), the e2e check fails and the PR is labeled release/blocked. This is expected — maintainers can review and merge anyway via admin bypass.

execute-release.yml — startup_failure diagnosis and fix#

Root cause#

execute-release.yml calls reusable-release.yml@main from the release-notes job. That reusable workflow's nested image-release job has permissions: { contents: write, actions: read }. GitHub validates ALL permissions requested by nested jobs against the caller's permission grant at workflow startup, before any code runs. If the caller job does not grant a permission the callee requests, the entire workflow run gets startup_failure with no log output.

Error (visible ONLY by fetching the Actions web page, not via API/CLI):

Error calling workflow 'reusable-release.yml@main'.
The nested job 'image-release' is requesting 'actions: read', but is only allowed 'actions: none'.

Fix: Add actions: read to the release-notes job permissions block.

  release-notes:
    permissions:
      actions: read # required by reusable-release.yml's image-release nested job
      contents: write
      id-token: write
      packages: read
    uses: projectbluefin/actions/.github/workflows/reusable-release.yml@main

How to diagnose startup_failure in GitHub Actions#

GitHub API endpoints (/jobs, /logs) return nothing for startup_failure runs. gh run view gives only a generic "workflow file issue" message. Open the run URL directly in a browser and search for "requesting" or "is not allowed" in the page. The error format is:

Error calling workflow 'reusable-X.yml@main'.
The nested job 'Y' is requesting 'actions: read', but is only allowed 'actions: none'.

gh run view <RUN_ID> --repo projectbluefin/bluefin-lts will show startup_failure status but no log. The web UI is the only place the specific permission mismatch is shown.

YAML syntax gotcha in if: conditions with colons in strings#

If a commit message pattern contains : (colon-space), the if: condition will fail YAML parsing:

# BROKEN — ': ' in single-quoted string breaks YAML scalar
if: startsWith(github.event.head_commit.message, 'chore: promote testing to main')

# CORRECT — wrap entire condition in double quotes
if: "startsWith(github.event.head_commit.message, 'chore: promote testing to main')"

actionlint [expression] rule — untrusted inputs in run: steps#

actionlint flags github.event.head_commit.message (and other user-controlled inputs) when interpolated directly into a run: shell script. It is safe in if: conditions because those are evaluated by GitHub's expression engine, not the shell.

# BROKEN — actionlint [expression] error, injection risk
- run: |
    MSG="${{ github.event.head_commit.message }}"

# CORRECT — pass through env var
- env:
    COMMIT_MSG: ${{ github.event.head_commit.message }}
  run: |
    if echo "$COMMIT_MSG" | grep -q "^chore:"; then ...

# ALSO CORRECT — if: conditions are not shell, no injection risk
  if: "startsWith(github.event.head_commit.message, 'chore: promote testing to main')"

execute-release.yml trigger change (push vs pull_request)#

The pull_request: closed trigger was replaced with push: branches: [main] because:

• Bot-authored PRs that modify .github/workflows/ via GITHUB_TOKEN cannot fire pull_request events (GitHub security restriction).
• Push events fire for all merges including admin force-merges.
• A check-trigger job with the if: startsWith(...) condition gates the actual release jobs so non-promotion pushes are no-ops.

E2E known issues — QEMU environment artifacts#

These units fail in the QEMU CI VM but are harmless on real hardware.
The fix in each case is to add systemd.mask=<unit> to KERNEL_ARGS in
projectbluefin/testsuite/.github/workflows/e2e.yml.

After a testsuite fix merges — SHA bump runbook:

1. Get the new SHA: gh api repos/projectbluefin/testsuite/commits/main --jq '.sha'
2. Update the single uses: line in .github/workflows/run-testsuite.yml — all callers inherit it automatically
3. Commit: fix(ci): bump testsuite SHA to include <description> (PR #NNN)
4. Open a PR with Closes #<issue> for each open e2e failure issue
5. After merge, post-merge E2E re-runs; the failure issues auto-close on green

Current SHA (post testsuite#419): 726ed4d24e08a18d5c31f816519f4bd6f0463511

Trivy scan FATAL — CentOS 10 CPE indices missing#

Symptom: All three build jobs (Build Bluefin LTS, Build Bluefin LTS HWE, Build Bluefin GDX) fail at the image (main, …, testing, x86_64) step with exit code 1 and no obvious container build error. The actual error is Trivy crashing at the very end of the job (after a successful container build):

FATAL Fatal error run error: image scan error: … unable to find CPE indices.
See https://github.com/aquasecurity/trivy-db/issues/435

Root cause: Trivy 0.70.x exits 1 with FATAL when its database has no CPE index entries for a new OS family (CentOS Stream 10). The exit-code: '0' Trivy parameter only suppresses non-zero exit when vulnerabilities are found — it does not suppress exits caused by Trivy's own DB crash.

The bootc-build/scan-image@v1 action in projectbluefin/actions did not have continue-on-error: true on the Trivy steps, so a Trivy FATAL kills the entire build job.

Fix: projectbluefin/actions PR #201:

• continue-on-error: true on both Trivy scan steps (SARIF + JSON)
• Guard Python summarize step against missing trivy-results.json

After actions PR #201 merges: A maintainer must retag v1 in projectbluefin/actions:

git tag -f v1 <merge-commit-sha>
git push origin v1 --force

All consuming repos (bluefin-lts, bluefin, dakota) pick up the fix immediately via @v1.

Note: The dracut POSTTRANS failures (error: rpm-ostree kernel-install: … Invalid cross-device link) in kernel-swap.sh are non-fatal warnings — dnf exits 0 despite them and the build continues past them. They appear in logs but do not kill the build. PR #174 adds export DRACUT_TMPDIR=/boot as a belt-and-suspenders fix but the primary blocker is the Trivy issue above.