Supply Chain Security#

Project Bluefin implements defense-in-depth security across its container-based OS delivery pipeline. The security model spans six primary domains: automated security posture monitoring via OpenSSF Scorecard, cryptographic container signing with Cosign, Software Bill of Materials (SBOM) generation stored as OCI artifacts, digest-based build provenance, Secure Boot with dual-signature kernel module verification, and repository isolation to prevent supply chain attacks.

The architecture treats the build pipeline as the security perimeter. Every artifact—from base images to kernel modules—is cryptographically verified before integration. SHA256 digest pinning ensures reproducible builds, while automated validation prevents unauthorized modifications. Repository isolation restricts package sources, and branch protection enforces manual review for production releases.

These measures protect the entire pipeline from source to deployment across repositories including ublue-os/bluefin, ublue-os/main, and ublue-os/akmods.

OpenSSF Scorecard#

OpenSSF Scorecard evaluates open source projects against security best practices including dependency pinning, code review, branch protection, and vulnerability disclosure. Project Bluefin adopted Scorecard in March 2026 for continuous security posture monitoring.

Implementation#

The scorecard workflow was added on 2026-03-11, creating .github/workflows/scorecard.yml:

Triggers:

• Weekly: Tuesday at 13:43 UTC
• Push to main branch
• Branch protection rule changes

Permissions: Read-only by default with explicit grants:

• security-events: write - Upload to Code Scanning dashboard
• id-token: write - Publish to OpenSSF REST API

Action: ossf/scorecard-action@4eaacf0543bb3f2c246792bd56e8cdeffafb205a (v2.4.3)

Outputs:

• SARIF results uploaded to GitHub Code Scanning
• Results published to OpenSSF REST API with public badge
• Artifacts retained for 5 days

Scorecard is currently enabled in ublue-os/bluefin and projectbluefin/testhub. Both repositories display Scorecard badges in their README files.

Container Signing and Verification#

All container images are signed with Cosign after successful push to GitHub Container Registry (ghcr.io). The signing process uses the digest from the push step to ensure the signature covers the exact pushed image, eliminating time-of-check-to-time-of-use vulnerabilities.

Signing Command:

cosign sign -y --key env://COSIGN_PRIVATE_KEY ${IMAGE}@${DIGEST}

The private key is stored in GitHub secrets as SIGNING_SECRET. Signed images can be verified during builds to ensure supply chain integrity and prevent unauthorized base image modifications.

Implementation: .github/workflows/reusable-build.yml

SBOM (Software Bill of Materials)#

SBOMs provide transparency for supply chain auditing and vulnerability tracking. Bluefin generates SBOMs using Syft in SPDX JSON format.

Generation Process#

1. Create temporary container from built image
2. Export container filesystem to OCI directory structure
3. Run Syft with parallelism set to $(nproc)*2 for performance
4. Generate SBOM in SPDX JSON format

Storage as OCI Artifacts#

As of PR #4274 (merged 2026-03-11), SBOMs are stored as OCI artifacts instead of Rekor attestations.

Problem: SBOMs grew too large for Rekor, causing cosign attest to fail consistently.

Solution:

1. Use ORAS to attach SBOM directly to GHCR as OCI artifact with type application/vnd.spdx+json
2. Sign the SBOM artifact itself with Cosign: cosign sign -y --key env://COSIGN_PRIVATE_KEY ${IMAGE}@${SBOM_DIGEST}

This approach stores SBOMs in the same registry as the images, eliminating external service dependencies.

Publishing Controls#

SBOM generation in Bluefin is conditional:

• LTS Branch Only: Generated only on lts branch when inputs.publish is true
• Skipped: All main branch builds, pull requests, and validation builds
• Error Handling: All SBOM steps include continue-on-error: true to ensure external service outages never block image publishing
• ublue-os/main: SBOM generation currently disabled but implementation preserved

Implementation: .github/workflows/reusable-build.yml

Build Provenance and Attestation#

Digest-Based Tracking#

The image-versions.yaml file tracks SHA256 digests for all base images and dependencies, ensuring reproducible builds. Digests are pinned in Containerfile using ARG variables.

Tracked Components:

• Base image digests (upstream)
• Akmods package digests
• Build dependency versions

Files:

• ublue-os/main image-versions.yaml
• ublue-os/bluefin image-versions.yml

Automated Dependency Updates#

Renovate bot updates container image digests every 6 hours, auto-merging digest updates for reproducibility. Renovate is configured to target only the main branch via .github/renovate.json5.

Build Triggers#

Digest-based build triggers compare SHA256 hashes between branches, automatically triggering builds when:

• Base image digests change (upstream updates)
• Akmods digests change
• Source file modifications occur

Version Format#

Image versions follow the format: stream-fedora_version.YYYYMMDD[.point]

Example: stable-40.20260311.0

Reference: Bluefin Justfile versioning logic

Secure Boot and Kernel Module Signing#

Bluefin supports Secure Boot by default using custom Universal Blue keys. The enrollment password is universalblue. Public keys are available in the ublue-os/akmods repository.

Kernel Signature Verification#

Kernel signatures are verified during builds using sbverify against two public certificates. The verification runs as a CI pipeline step (just secureboot) after image build, before publishing.

Verification Process:

1. Extract vmlinuz from the built container image
2. Download public certificates from the akmods repository
3. Verify kernel signatures against both certificates
4. If sbverify unavailable locally, spawn Alpine container with sbsigntool

Implementation: ublue-os/main Justfile secureboot target

Kernel Module Dual-Signature#

Kernel modules undergo a dual-signature process using OpenSSL CMS format with two keys:

1. Decompress modules
2. Sign with both keys
3. Recompress modules
4. Rebuild RPM packages with signed modules
5. Verify module signatures immediately after signing
6. Reject unsigned modules - Cannot reach production

Implementation: ublue-os/akmods dual-sign.sh

Repository Isolation and Package Security#

An isolated COPR installation pattern prevents malicious package injection. Repositories are enabled only temporarily during package installation. Repository validation ensures no repositories remain globally enabled after builds.

Packages are split into FEDORA_PACKAGES and COPR_PACKAGES to prevent COPR supply chain attacks where untrusted packages override system packages.

Implementation Files:

• validate-repos.sh - Repository validation script
• copr-helpers.sh - Isolated COPR installation helpers
• 04-packages.sh - Package installation with COPR separation

Build Infrastructure Hardening#

Branch Protection Strategy#

A three-layer strategy protects the lts production branch:

1. Manual Promotion: create-lts-pr.yml workflow requires merge commit (never squash-merge)
2. Renovate Restriction: Renovate targets only main branch
3. Validation Builds: LTS push events trigger validation builds that verify integrity but never publish production tags

Publishing Controls#

The publish input in the reusable workflow defaults to false. Callers must explicitly opt in to image publishing. Production LTS releases are triggered only via workflow_dispatch:

• Weekly: Tuesday 6 AM UTC
• Manual trigger available

Push events to lts trigger validation builds only.

Push Retry Logic#

Images are pushed to ghcr.io with retry logic for transient failures:

• Bluefin: 3 attempts, 15-second delays via Wandalen/wretry.action
• ublue-os/main: 5 attempts, exponential backoff (5s, 10s, 15s, 20s, 25s)

Build Validation#

Bootc Container Lint: Validates bootable container standards during build using bootc container lint, embedded in Containerfile.

Rechunking Process: Optimizes container layer boundaries using ghcr.io/ublue-os/legacy-rechunk for delta updates in three phases: prune, create OSTree, rechunk against previous version.

CI/CD Pipeline Security#

Pipeline Stages#

1. Checkout code
2. Install Cosign for image signing
3. Build container image (just build-container)
4. Verify secure boot (just secureboot)
5. Push to GitHub Container Registry
6. Sign container with Cosign
7. Generate and attach SBOM (LTS branch only, when publishing)

Permissions Model#

Default permissions are read-only. Explicit grants are required for specific operations, following the principle of minimal privilege.

Immutability as Security#

The OS filesystem is read-only by default using composefs with /usr mounted read-only. This prevents runtime modification of system binaries. Composefs can enable fsverity for cryptographic integrity verification where the kernel verifies file contents match expected cryptographic hashes.

Local package layering is locked by default via LockLayering=true in /etc/rpm-ostreed.conf, preventing mutation of the base OSTree commit. Running rpm-ostree reset and rebooting always restores the system to pure image mode.

Relevant Code Files#

Related Topics#

• Bootc Architecture - Container-native OS delivery mechanism
• OSTree - Atomic update system for bootable filesystems
• Secure Boot - UEFI firmware security validation
• Sigstore - Keyless signing infrastructure
• SLSA Framework - Supply chain levels for software artifacts
• OCI Standards - Open Container Initiative specifications

External References#

• OpenSSF Scorecard Documentation
• Sigstore Cosign
• ORAS (OCI Registry as Storage)
• Syft SBOM Generator
• Bootc Documentation
• Project Bluefin Documentation