Universal Blue Build and Update System

The Universal Blue Build and Update System is the comprehensive infrastructure encompassing GitHub Actions CI/CD pipelines, OCI container delivery, and multi-layered automatic update architecture that powers Universal Blue and its desktop variants including Bluefin and Aurora. This system represents a distinct technical area separate from both the Universal Blue platform and the underlying rpm-ostree/bootc tooling, providing project-specific build orchestration and update mechanisms.

The infrastructure consists of three integrated components: (1) a GitHub Actions-based CI/CD pipeline that triggers builds on every main branch commit, with lts branch builds triggered exclusively via a dispatcher workflow that runs weekly (Tuesday 6 AM UTC / 1 AM EST / 2 AM EDT) or manual dispatch, typically delivering changes within 30 minutes to 2 hours; (2) OCI container-based delivery where operating systems are delivered as complete container images published to ghcr.io; and (3) a three-layer update architecture combining bootc for system updates, Flatpak for GUI applications, and Homebrew for CLI tools. The system implements intelligent build triggers using digest-based version tracking, comprehensive validation including secure boot verification, and advanced optimizations like rechunking for efficient delta updates.

Built on Fedora Silverblue and Kinoite foundations, the system underwent major architectural refactoring in 2025 from monolithic distributions to modular OCI containers, fully embracing bootc (bootable containers) as the primary upgrade mechanism. The architecture ensures system images remain pristine with no package-based degradation over time while enabling users to switch between variants and update streams seamlessly.

Build Infrastructure and CI/CD Pipeline

GitHub Actions Architecture

The Universal Blue project employs a reusable workflow system with a two-job structure: the check-build-required job determines if builds are needed, and the build_ublue job executes the actual build pipeline. Bluefin extends this with a matrix strategy for parallel builds across multiple variants and flavors.

The pipeline executes these key stages:

1. Checkout code
2. Install Cosign for image signing
3. Build container image using just build-container
4. Verify secure boot with just secureboot
5. Push to GitHub Container Registry
6. Sign container with Cosign
7. Generate and attach SBOM (Bluefin only)

Multi-Stage Container Build

The build process uses a four-stage Containerfile:

Variant-specific builds integrate ghcr.io/projectbluefin/common for shared configurations and ghcr.io/ublue-os/brew for Homebrew support.

Intelligent Build Triggers

The system implements digest-based triggering that compares SHA256 hashes in image-versions.yaml between branches to determine if builds are required. Build trigger behavior differs by branch:

Main branch triggers:

• Push events on every commit
• Base image digest changes (Fedora Silverblue/Kinoite upstream updates)
• Akmods digest changes (kernel module updates)
• Source file modifications (any code/config changes)
• Manual workflow_dispatch triggers

LTS branch triggers:

• A dispatcher workflow (scheduled-lts-release.yml) runs weekly on Tuesday at 6 AM UTC (1 AM EST / 2 AM EDT) on the default branch
• The dispatcher triggers all 5 build workflows on the lts branch via workflow_dispatch events
• Push events to lts branch trigger validation builds but do NOT publish production tags
• Pull requests to lts branch do NOT trigger workflows (prevents accidental publishes)
• Manual workflow_dispatch triggers (for emergency releases)
• Individual build workflows no longer have schedule: triggers—the centralized dispatcher eliminates 10 duplicate no-op builds that previously ran on main every Tuesday alongside the 5 dispatcher runs on lts, working around GitHub Actions limitation that scheduled triggers always run on the default branch

This architecture prevents accidental production tag publishing while maintaining validation builds on merges. The validation builds on lts push events provide a third layer of defense against branch pollution by verifying code integrity when promotions complete.

Dispatcher Workflow Architecture:

The dispatcher pattern solves a GitHub Actions constraint where scheduled triggers always execute on the default branch. The scheduled-lts-release.yml workflow runs on main and uses GitHub CLI to trigger all 5 build workflows on the lts branch via workflow_dispatch events. This ensures production releases build from the stable lts branch code rather than the more frequently updated main branch. See docs/plans/2026-03-02-fix-lts-tag-publishing.md for complete implementation details.

Branch Management and Promotion Strategy:

The system implements a three-layer defense strategy to protect the lts production branch from accidental pollution:

Layer 1: Manual Promotion Workflow - A create-lts-pr.yml workflow automates PR creation from main to lts when content differs. The workflow:

• Triggers automatically on every push to main
• Checks git diff --quiet origin/lts origin/main (content diff)
• Creates or updates a draft PR from main → lts when differences exist
• Requires manual merge (Create a merge commit) via GitHub UI as approval gate
• CRITICAL RULE: NEVER commit directly to lts — all changes must land in main first

NEVER squash-merge promotion PRs. Squash-merge creates orphan commits that permanently break the merge base between main and lts, causing every future promotion PR to accumulate all historical commits in its diff. Regular merge preserves the merge base and keeps future PRs clean.

Layer 2: Renovate Restriction - Renovate configuration restricts dependency updates to target only the main branch via baseBranchPatterns: ["main"]. This prevents automated dependency PRs from targeting the lts branch, ensuring the production branch receives updates only through manual promotion.

Layer 3: Validation Build Triggers - All 5 build workflows (build-dx-hwe.yml, build-dx.yml, build-gdx.yml, build-regular-hwe.yml, build-regular.yml) include lts in their push triggers. When promotions push to lts, validation builds execute to verify code integrity, but these builds do NOT publish production tags. Publishing only occurs via workflow_dispatch events (weekly dispatcher or manual triggers).

This three-layer architecture prevents branch pollution incidents where AI agents or automated tools merge lts → main (wrong direction) when detecting divergence in image-versions.yaml digests. Such divergence is intentional—main has newer versions for testing while lts maintains stable production versions.

Changes to image-versions.yaml itself are explicitly excluded to prevent build loops.

Build Validation and Safety Checks

The pipeline enforces multiple validation layers:

Bootc Container Lint: Validates bootable container standards during build using the bootc container lint command embedded in the Containerfile.

Secure Boot Verification: Comprehensive kernel signature verification using sbverify:

1. Extracts vmlinuz from built container image
2. Downloads public certificates from akmods repository
3. Verifies kernel signatures against both certificates

If sbverify is unavailable locally, the validation spawns an Alpine container with sbsigntool for verification.

Image Signing with Cosign

Container image signing occurs after successful push using Cosign, with SIGNING_SECRET stored in GitHub secrets. The signing process uses the digest from the push step to ensure signature covers the exact pushed image.

SBOM Generation

SBOM (Software Bill of Materials) generation occurs only on the lts branch when images are published. The system uses Syft to scan container images and generate SBOMs, which are attached as attestations to signed images for supply chain transparency. All SBOM-related steps include continue-on-error: true to ensure that SBOM failures (such as Sigstore/Rekor service outages) never block image publishing.

SBOM generation behavior:

SBOMs are generated exclusively when github.ref == 'refs/heads/lts' && inputs.publish evaluates to true:

• ✅ Generated when lts branch builds are triggered via the dispatcher workflow (weekly on Tuesday at 6 AM UTC / 1 AM EST / 2 AM EDT) or manual workflow dispatch
• ❌ Skipped on all main branch builds (including testing builds)
• ❌ Skipped on pull requests to any branch
• ❌ Skipped on validation builds (push events to lts branch that don't publish)

The sbom: input parameter has been removed from reusable-build-image.yml. SBOM behavior is controlled entirely by step-level conditions checking the branch reference and publish status. The workflow defaults publish to false for safety—callers must explicitly opt in to publishing.

For LTS production releases, SBOMs are generated weekly through the dispatcher pattern. The ublue-os/main repository has SBOM generation disabled but implementation preserved.

Rechunking for Efficient Updates

Bluefin implements a three-stage rechunking process using ghcr.io/ublue-os/legacy-rechunk.0.1 to optimize container layer boundaries for delta updates:

Phase 3 compares against the previous image version using PREV_REF to intelligently reorganize layer boundaries, reducing bandwidth requirements for updates.

Matrix Builds and Parallelization

Build parallelization strategies differ between repositories:

ublue-os/main: Executes 6 parallel builds (3 images × 2 variants):

• Images: base, kinoite, silverblue
• Variants: main, nvidia

Bluefin: Runs 4 builds per stream (2 base_names × 2 flavors):

• Base names: bluefin, bluefin-dx
• Flavors: main, nvidia-open

Separate workflow files handle each version stream (gts, stable, latest, beta) with stream-specific kernel pinning and scheduling.

Container Registry Delivery

Images are pushed to ghcr.io with retry logic to handle transient network failures:

Conditional Publishing:

• The main branch publishes images to :lts-testing tag on every push commit
• The lts branch publishes production tags (:lts, :lts.YYYYMMDD) only when triggered via workflow_dispatch events:
  • Weekly automated releases via the dispatcher workflow (Tuesday 6 AM UTC / 1 AM EST / 2 AM EDT)
  • Manual workflow dispatch for emergency releases
• Push events to lts branch trigger validation builds but do NOT publish production tags
• Pull requests to lts branch do NOT trigger workflows (prevents accidental publishes)
• Pull requests to main branch validate (build without publishing)
• The publish condition enforces this behavior: (github.event_name == 'workflow_dispatch' && (github.ref == 'refs/heads/lts' || github.ref == 'refs/heads/main')) || (github.event_name == 'push' && github.ref == 'refs/heads/main')

Bug Fix (PR #1154 & #1157): Critical bugs were fixed in the manifest generation and publishing steps:

• Manifest Step (PR #1154): The manifest generation step had inconsistent conditional logic compared to the build step. The build step correctly added -testing suffix for all non-production branches, but the manifest generation step only added it for pull requests and merge groups—omitting pushes to main. This caused main branch merges to accidentally push images to the production :lts tag instead of :lts-testing. PR #1154 resolved this by aligning manifest step logic with build step logic: both now use if [ "${REF_NAME}" != "${PRODUCTION_BRANCH}" ], ensuring consistent tagging behavior.
• Push Manifest and Sign Steps (PR #1157): Both the "Push Manifest" and "sign" steps used github.event_name != 'pull_request' conditionals, which caused them to fire even when inputs.publish was false. This resulted in "image not known" errors during validation builds on lts push events. PR #1157 fixed this by gating both steps on inputs.publish instead, ensuring they only execute when publishing is intended. The same fix was applied to signing steps in the build_push job to prevent signing non-existent images.

Changes on the main branch typically go live within 30 minutes to 2 hours. Production releases on the lts branch occur weekly via the automated dispatcher or on-demand via manual dispatch.

Version Management

The image-versions.yaml file tracks SHA256 digests for all base images and dependencies, ensuring reproducible builds and enabling the CI system to detect upstream changes. Digests are pinned in the Containerfile using ARG variables extracted during the build process.

OCI Container Delivery Mechanism

Container Format and Standards

Operating systems are delivered as OCI container images where system updates are complete new images, not individual packages. The standard OCI image format includes the kernel used to boot the system, enabling bootable containers.

Container Registry Integration

All images are published to ghcr.io/ublue-os/ with standard container registry authentication. bootc replaces ostree's HTTP transport with OCI container fetching, allowing the system to leverage existing container registry infrastructure, CDNs, and authentication mechanisms. Images are signed with Cosign for cryptographic verification.

Base Images Layer (ublue-os/main)

The ublue-os/main repository provides a common main image for all Universal Blue variants with key modifications to Fedora:

• Removes upstream Fedora kernel, replaces with signed kernels from akmods layer
• Replaces Mesa and Intel drivers with enhanced versions from negativo17's fedora-multimedia
• Removes Fedora Flatpak remote and replaces with Flathub
• Fetches zram-generator configuration directly from Fedora's rawhide branch using the raw endpoint at https://src.fedoraproject.org/rpms/zram-generator/raw/rawhide/f/zram-generator.conf (required because c10s lacks the -defaults subpackage)

As of September 2025, ublue-os/main builds only base, kinoite, and silverblue images, streamlining the build matrix.

GNOME Build Process (Bluefin LTS)

For EL10-based builds (Bluefin LTS), the system uses a parallel build architecture where both GNOME 49 and GNOME 50 variants are built independently through the same full build pipeline. Both versions source from COPR repositories (jreilly1821/c10s-gnome-49 and jreilly1821/c10s-gnome-50-fresh) that track Fedora dist-git.

The Containerfile accepts ARG GNOME_VERSION="49" (defaulting to GNOME 49), and build scripts conditionally branch on this value to select the appropriate COPR repository and compatibility packages. This architecture ensures both GNOME versions receive identical build treatment without layering complexity.

GNOME Version Selection

The build process uses the GNOME_VERSION build argument to determine which GNOME stack to install:

• GNOME 49 (default): The Justfile passes gnome_version="49" by default, producing :lts-testing and :lts-hwe-testing tags for both bluefin and bluefin-dx variants
• GNOME 50 (testing): The build-gnome50.yml workflow passes gnome-version: "50" to build :lts-testing-50 and :lts-hwe-testing-50 tags for both bluefin and bluefin-dx variants

Build Script Branching

The base image build script (10-packages-image-base.sh) conditionally enables the appropriate COPR and installs version-specific packages based on $GNOME_VERSION:

Critical packages are upgraded before the GNOME group install to prevent runtime crashes:

• glib2: EL10 ships 2.80.x; gnome-shell 49.x requires 2.82+ API symbols, 50.x requires 2.84+
• fontconfig: COPR pango 1.57+ links FcConfigSetDefaultSubstitute (added in fontconfig 2.17.0); EL10 base ships 2.15.0, causing symbol lookup errors
• gobject-introspection / gjs (GNOME 49 only): glib2 2.84+ ships both libgirepository-1.0 and libgirepository-2.0; upgrading only one causes both to load, triggering double-registration crashes in gnome-shell
• selinux-policy (GNOME 50 only): COPR 43.x is required for GDM 50 userdb varlink socket architecture; EL10 base 42.x lacks the necessary policy rules

The package build script (20-packages.sh) applies version locks to GNOME components based on $GNOME_VERSION:

# GNOME 49 versionlock
dnf versionlock add gnome-shell gdm gnome-session-wayland-session gobject-introspection gjs pango

# GNOME 50 versionlock
dnf versionlock add gnome-shell gdm mutter gnome-session-wayland-session \
    gnome-settings-daemon gnome-control-center gsettings-desktop-schemas \
    gtk4 libadwaita pango fontconfig

Build Workflow Architecture

The build-gnome50.yml workflow invokes reusable-build-image.yml with gnome-version: "50", which passes the version through to the Justfile's build recipe as the 7th positional argument. This triggers a full parallel build using --build-arg GNOME_VERSION=50 rather than layering on top of GNOME 49 images. The workflow runs only on the main branch (never on lts) and includes four jobs:

• build: Produces bluefin:lts-testing-50 (standard kernel)
• build-hwe: Produces bluefin:lts-hwe-testing-50 (HWE kernel)
• build-dx: Produces bluefin-dx:lts-testing-50 (DX variant with standard kernel)
• build-dx-hwe: Produces bluefin-dx:lts-hwe-testing-50 (DX variant with HWE kernel)

This mirrors the pattern used in standard build workflows, ensuring both bluefin and bluefin-dx variants receive GNOME 50 testing images.

EL10-Specific Workarounds:

All workarounds were validated on quay.io/centos-bootc/centos-bootc:stream10 with GDM greeter reaching successfully and gnome-shell starting under enforcing SELinux.

Modular Component Design

Following 2025 refactoring from monolithic to modular OCI containers, the system comprises:

Kernel Module Infrastructure (akmods)

The akmods infrastructure provides pre-compiled, signed kernel modules for immutable distributions. Modules are built daily and version-locked alongside kernel packages to prevent ABI mismatches, enabling out-of-tree drivers and hardware enablement without compromising system immutability.

Update Architecture and Technologies

Core Technologies

The update system combines two complementary technologies:

• rpm-ostree provides the storage layer with an immutable base system using OSTree commits
• bootc serves as the primary upgrade mechanism for atomic updates and rollbacks

bootc implements transactional, in-place OS updates using OCI/Docker container images and depends on ostree as a storage backend but replaces ostree's HTTP transport with OCI container fetching.

Immutability Model

The system enforces strict immutability: LockLayering=true in /etc/rpm-ostreed.conf locks local package layering by default, which prevents mutation of the base OSTree commit including package overlays, overrides, and initramfs changes. This ensures the system image remains pristine with no package-based degradation over time. Running rpm-ostree reset and rebooting always restores pure image mode.

Filesystem Layout

The OS is immutable by default using composefs with a specific filesystem organization:

bootc Commands and Operations

Key bootc commands for system management:

# System updates
bootc upgrade # Query source, queue update for next boot
bootc upgrade --apply # Upgrade and immediately reboot
bootc upgrade --check # Check for updates without side effects

# Switching images
bootc switch <image> # Change tracked image, preserves /etc and /var

# Status and recovery
bootc status # Display current system state
bootc rollback # Swap bootloader ordering to previous entry

bootc switch changes the tracked container image while preserving /etc and /var state, enabling seamless variant switching and blue/green deployments. bootc status displays the current system state including booted deployment and staged deployments, while bootc rollback swaps bootloader ordering to the previous boot entry for quick recovery.

System Update Flow

Updates are delivered as complete new images, not individual packages. The system checks for updates automatically every 6 hours and applies them on reboot, keeping the running system stable while staging changes. The A/B style upgrade system maintains both current and previous deployments, with staged updates downloading and preparing without affecting the running system.

Three-Layer Package Management Strategy

Overview

The system implements a three-layer package management strategy: immutable system layer, Flatpak for GUI applications, and Homebrew for CLI tools. This separation ensures system stability remains decoupled from application flexibility.

System Layer (bootc)

The immutable base OS is delivered as complete images with updates every 6 hours applied on reboot. bootc upgrade fetches updates via OCI container mechanisms, staging them for the next boot without affecting the running system.

Application Layer (Flatpak)

GUI applications are installed from Flathub with automatic updates enabled by default via systemd timers. Updates run daily at 4:00 AM, requiring no reboots. Applications are sandboxed via Flatpak with system-wide installations by default.

CLI Layer (Homebrew)

Homebrew is managed via the projectbluefin/common image and includes containerd support, enabling Docker installation directly from Homebrew. The system maintains a custom tap at ublue-os/homebrew-tap for project-specific packages not available in official Homebrew.

Update Orchestration with uupd

uupd Service

The uupd service coordinates system and Flatpak updates, replacing or supplementing rpm-ostreed-automatic and Flatpak timers. If uupd.timer exists, it is enabled by default; otherwise, individual timers are enabled.

Users can customize the update schedule:

sudo systemctl edit uupd.timer

Update Behavior

The ujust update command uses bootc upgrade by default but falls back to rpm-ostree upgrade if package layering is detected. This reflects the 2025 drastic refactoring to fully embrace bootc as the primary update mechanism.

Update Streams and Variants

Update Streams

Universal Blue provides multiple update streams with different stability and kernel configurations:

Kernel pinning in gts and stable streams ensures ZFS compatibility by maintaining consistent kernel versions.

Aurora and Bluefin Variants

The two primary desktop variants share the same update infrastructure:

• Bluefin built on silverblue-main (GNOME desktop)
• Aurora built on kinoite-main (KDE Plasma desktop)

Both variants support rebasing between each other using bootc switch, which preserves the home directory, Flatpak apps, and configurations during rebase. The variants share artwork and infrastructure components.

Usage Examples

System Updates

# Check for updates without applying
bootc upgrade --check

# Update system (applied on next reboot)
ujust update

# Update and reboot immediately
bootc upgrade --apply

Switching Between Variants

# Switch from Bluefin to Aurora
sudo bootc switch ghcr.io/ublue-os/aurora:latest

# Switch between update streams
sudo bootc switch ghcr.io/ublue-os/bluefin:stable

Managing Automatic Updates

# Toggle automatic updates on/off
ujust toggle-updates

# Check current update configuration
systemctl status uupd.timer

Application Management

# Install Flatpak application
flatpak install flathub org.mozilla.firefox

# Install CLI tool via Homebrew
brew install kubectl

# Update Flatpaks manually
flatpak update

Relevant Code Files

Key Repositories

Related Topics

• bootc (Bootable Containers) - Core technology for atomic OS updates using OCI container images
• rpm-ostree - Storage layer providing immutable base system with OSTree
• OSTree - Content-addressed filesystem and update system
• OCI (Open Container Initiative) - Container image standards used for OS delivery
• Fedora Silverblue - Upstream immutable Fedora GNOME desktop
• Fedora Kinoite - Upstream immutable Fedora KDE desktop
• GitHub Actions - CI/CD platform for automated builds
• GitHub Container Registry (ghcr.io) - OCI container registry for image distribution
• Cosign - Container image signing and verification
• Syft - Software Bill of Materials (SBOM) generation tool
• Flatpak - Application sandboxing and distribution system
• Homebrew - Package manager for CLI tools and applications
• Universal Blue - Parent platform and ecosystem
• Bluefin - Developer-focused desktop variant
• Aurora - KDE Plasma desktop variant

See Also

• Universal Blue Documentation
• Project Bluefin Documentation
• bootc Developer Documentation
• rpm-ostree Documentation
• Ask Project Bluefin Community