Enterprise Linux Management#

Enterprise Linux Management in the context of Bluefin refers to the deployment, authentication integration, policy enforcement, and fleet management challenges that arise when implementing Bluefin's immutable, container-based architecture in traditional enterprise IT environments. While Bluefin is designed as a next-generation Linux workstation for reliability, performance, and sustainability, significant gaps exist between its developer-focused design and conventional enterprise requirements for centralized control, directory services integration, and incremental patch management.

Bluefin's architecture is built on bootc (bootable containers), which provides transactional, atomic updates using OCI/Docker container images instead of traditional package-based systems. This image-based approach prohibits client-side package mutations, ensuring system immutability but creating friction with enterprise IT practices that expect mutable systems. Local package layering is locked by default via LockLayering=true in /etc/rpm-ostreed.conf, which prevents mutation of the base OSTree commit including package overlays and initramfs changes, fundamentally incompatible with traditional system hardening and compliance enforcement techniques.

The enterprise challenges span authentication integration (Active Directory, LDAP, enterprise WiFi), fleet management (centralized policy enforcement, inventory, compliance reporting), and update control (staggered rollouts, selective patching). Organizations seeking to deploy Bluefin at scale must either build custom bootc images incorporating enterprise requirements, accept the constraints of the immutable architecture, or maintain external management tooling outside the base system layer.

Core Architecture and Immutability#

Bootc: Container-Native OS Management#

Bootc implements an ostree-backed A/B upgrade system, where changes to the base image are staged, and the running system is not changed by default. The bootc upgrade command queries the container image source and queues an updated container image for the next boot, delivering complete new OS images rather than individual packages.

The system fundamentally differs from traditional package management in several critical ways:

1. Read-only base system: When deployed, container image files are read-only by default. With composefs enabled, the entire / is a read-only filesystem, and directories like /usr and /opt are lifecycled with the container image.

2. No client-side mutations: The bootc project takes a hard stance that system state should come from a container image. Unlike rpm-ostree which allows rpm-ostree install, bootc does not support client-side package layering—all changes must be made through container image builds.

3. Configuration binding: Code and configuration are strictly "lifecycle bound" together in the same container image, unlike traditional systems where packages and configuration are managed separately.

4. State separation: /etc and /var are machine-local state by default, with /var persisting user data that should not be affected by OS rollbacks. For example, a bootc update or rollback does not affect database data in /var/lib/postgres.

Update Streams and Cadence Control#

Bluefin offers images based on the current version of Fedora, as well as a CentOS-based image, providing flexibility in update aggressiveness through "streams":

• stable: The default stream for Bluefin follows the Fedora CoreOS release schedule, meaning kernel upgrades come about 2 weeks after they land in Fedora. This allows the Bluefin team to pin to a specific kernel in those circumstances to avoid regressions, called "gating" the kernel. Builds weekly or daily (stable-daily).

• latest: For users who want the very latest Fedora has to offer, an ungated Linux kernel, daily updates. This stream is purposely left unbranded and is not meant for general purpose use.

• lts: Bluefin LTS is built on CentOS Stream 10 for users who prefer a slower release cadence, about a three-to-five year lifespan on a single release.

The system checks for updates every six hours and applies them on reboot. This provides reliability but prevents staggered rollout strategies or emergency holds on specific systems. Updates are delivered as complete new images, not individual packages, making selective security patch deployment impossible.

Rollback and Recovery#

The bootc rollback command swaps the bootloader ordering to the previous boot entry, allowing administrators to quickly revert to a known-good state. The system maintains both the current and previous deployments, enabling instant rollback capabilities if issues arise.

Additionally, rpm-ostree reset and a reboot will always bring the system back to pure image mode, providing a panic button for infected or misconfigured systems.

Enterprise Authentication Challenges#

Active Directory and SSSD Integration#

While Bluefin lacks native Active Directory integration, users relying on Active Directory and SSSD for authentication have experienced critical login failures after system updates. Multiple issues complicate enterprise deployments:

1. SSSD Permission Issues: SSSD version 2.10.1 changed the service to run as root instead of the sssd user, causing permission errors when reading /etc/sssd/sssd.conf. Users encountered errors like "PAM unable to dlopen(/usr/lib64/security/pam_lastlog.so): Permission denied" that prevented AD authentication.

   Workaround: Run sudo /bin/find /etc/sssd -type d -exec /bin/chmod g+x {} \; to set group execute permissions, and use sudo systemctl edit sssd to restore capabilities.

2. Version Upgrade Complications: When upgrading from Fedora 42 to 43 while using SSSD authentication, users need to temporarily switch to local authentication with authselect select local before upgrading, then re-enable SSSD after booting with authselect select sssd with-mkhomedir. This is necessary because the migration from pam_lastlog to pam_lastlog2 doesn't properly update authselect configuration files.

3. Removed AD Support: Despite previous efforts to add AD integration, necessary packages for realm joining are not installed. When attempting to join a realm, users receive errors about missing packages: freeipa-client sssd-common oddjob oddjob-mkhomedir. A maintainer noted: "Yeah we can't support this until someone with an AD can drive this feature", indicating AD support was removed and won't return without community maintainer support.

LDAP Support#

Users have requested support for LDAP login through packages like sssd-ldap, oddjob-mkhomedir, and oddjob. While most packages needed for Active Directory support are already present, LDAP-specific packages are missing. There is uncertainty whether this belongs in the main image or downstream in Bluefin/Aurora.

Enterprise WiFi Authentication (802.1X/RADIUS)#

Bluefin has known compatibility issues with enterprise WiFi using RADIUS/802.1X authentication. When attempting to connect to RADIUS networks, NetworkManager may not prompt for credentials and instead falls back to previously connected networks due to the default iwd backend having compatibility problems with enterprise authentication standards.

Workaround: Switch from iwd to wpa_supplicant backend by creating /etc/NetworkManager/conf.d/nm.conf:

[device]
wifi.backend=wpa_supplicant

Or use ujust toggle-iwd to switch between iwd and wpa_supplicant for Wi-Fi networking. However, this change will remove all saved networks and require reconfiguration.

Fleet Management and Centralized Control#

Absence of Built-in Fleet Management#

Bluefin lacks native support for enterprise fleet management capabilities:

• Device inventory and discovery
• Centralized policy enforcement
• Remote monitoring and logging
• Compliance reporting
• Software deployment across multiple systems

There's a longstanding feature request for out-of-the-box fleet management using Cockpit. Cockpit client could provide a graphical interface for administrators to manage boxes they can SSH into, described as "a reasonable solution for a small number of installs or a lab". However, there's no indication of active development or implementation.

Similarly, bootc itself doesn't have an IPC mechanism today, though bootc aims to include some functionality similar to zincati (Fedora CoreOS's auto-update manager) in the future.

External Management Integration#

Bootc provides primitives for external fleet management:

1. Image Switching for Deployments: The bootc switch command supports blue/green deployments where some machines are rolled onto a new image independently of others. A common pattern is for a management agent to control OS updates via container image tags, such as quay.io/exampleos/someuser:v1.0 and v1.1.

   Organizations can also provide explicit digests via bootc switch quay.io/example@sha256:... where bootc upgrade will always be a no-op and upgrades are triggered by further switch operations.

2. Registry Integration: Bootc automatically honors configuration in /etc/containers such as containers-registries.conf. If you configure podman to pull images from your local mirror registry, bootc will automatically honor that, enabling centralized registry control.

3. Configuration Management: System paths in /etc persist across updates on Bluefin's immutable architecture. Standard configuration management tools (Ansible, Puppet, Salt) can manage files in /etc and /var for post-boot configuration and policy enforcement.

Update Control and Date Pinning#

Bluefin's release notes contain stream switching instructions, useful for nailing down regressions in specific package versions. Administrators can rebase to a specific date tag to "pin" to a specific day or version:

sudo bootc switch ghcr.io/ublue-os/bluefin:stable-20241027 --enforce-container-sigpolicy

However, organizations cannot pin individual systems to specific update dates within a centralized orchestration framework or create custom validation schedules without external tooling.

Policy Enforcement and Security#

Container Image Verification#

All Bluefin container images are signed with Cosign for cryptographic verification. The --enforce-container-sigpolicy flag enforces container signature policy when switching images.

Container image verification failures can prevent system updates in enterprise environments where /etc/containers/policy.json has been modified. Users who applied the 2024 cosign keypair rotation fix now have stale policy configurations pointing to incorrect key paths, causing "failed to invoke method OpenImage" errors.

Fix: Restore the pristine policy configuration by running sudo cp -a /usr/etc/containers/policy.json /etc/containers/policy.json.

Security Model#

Bootc's security model includes several layers:

• Filesystem Integrity: Bootc uses composefs by default for the root filesystem, which ensures the entire / is a read-only filesystem. For enhanced security, bootc supports fsverity to hard-require filesystem integrity verification via [composefs] enabled = verity in /usr/lib/ostree/prepare-root.conf.

• Transient Configuration: Enable etc.transient = true to make /etc configuration ephemeral, particularly important because /etc can contain arbitrary executable code like systemd unit files. Similarly, enable root.transient = true for a fully transient writable rootfs.

• Secure Boot: Secure Boot is supported by default using custom Universal Blue keys. Note that hibernation and Secure Boot are mutually exclusive in Bluefin LTS.

Kernel Arguments for Policy#

Bootc supports injecting kernel arguments that become machine-local state. Container images can include policy via /usr/lib/bootc/kargs.d TOML files, and changes to kargs.d files in container builds are honored post-install, preserving machine-local arguments.

Custom Image Development for Enterprise#

Organizations can build custom Bluefin variants using the modular OCI container architecture:

1. Base Layer: @projectbluefin/common provides core experience (ujust, MOTD, service units, GNOME configuration)

2. Custom Packages: Custom variants can layer additional packages and configurations

3. Private Registries: Images are published to ghcr.io with retry logic, enabling deployment to private registries (Harbor, Artifactory, or self-hosted container registry)

4. Provisioning: Bootc systems support standard provisioning tools like cloud-init, Ignition, systemd-firstboot, or custom systemd services

5. Installation Audit Trail: After installation, bootc writes .bootc-aleph.json containing provenance information (source image reference, digest, timestamp, bootc version, kernel version, SELinux state)

Relevant Configuration Files#

Enabling Local Package Layering#

Local Layering is adding individual packages onto the system. Generally speaking this is an anti-pattern in Bluefin as the end goal is to move away from the package based model entirely, however sometimes you just need something.

You can toggle this setting in /etc/rpm-ostreed.conf:

LockLayering=false

From the manpage:

LockLayering=
Controls whether any mutation of the base OSTree commit is supported (for
example, package overlays or overrides, initramfs overlays or regeneration).
Defaults to false.

rpm-ostree reset and a reboot will always bring the system back to pure image mode, making temporary compromises to get work done perfectly fine.

Local layering does significantly increase update time, but by default all Bluefin systems update in the background anyway and the result will mostly be invisible. Problems will generally occur if you are using a third party repository that doesn't align with what's happening in the Fedora archive at the time.

Enterprise Deployment Gap Summary#

Recommended Enterprise Deployment Model#

1. Build custom bootc images incorporating enterprise policy defaults, security hardening, directory service packages (SSSD, realm, oddjob), and organizational branding into the base system layer

2. Host images in private registries (Harbor, Artifactory, or self-hosted container registry) with proper authentication and access controls

3. Use configuration management (Ansible, Puppet, or similar) for post-boot configuration and policy enforcement in /etc and /var

4. Implement monitoring/inventory via standard Linux tools (Prometheus, Telegraf, fleet management dashboards)

5. Accept immutability constraints and design workflows around atomic updates rather than incremental changes

6. Test authentication integration early particularly 802.1X, VPN, and directory service connectivity before fleet deployment

7. Establish rollback procedures and test bootc rollback capabilities as part of deployment validation

8. Document image switching procedures for controlled blue/green deployments using bootc switch with explicit digests

Bluefin's architecture prioritizes developer experience and operational reliability over traditional enterprise flexibility. Organizations deploying at scale must accept these constraints or maintain separate management tooling outside the base system layer.

Related Topics#

• Bluefin Administrator's Guide - Day-to-day operation, update streams, and system administration
• Bluefin LTS - Long-term support variant built on CentOS Stream 10
• Bootc Documentation - Container-native OS update and deployment system
• Universal Blue - OCI-based Fedora variants and custom image building
• Immutable Operating Systems - Design philosophy and operational patterns for read-only base systems
• OSTree - Git-like version control system for operating system binaries