Incus Container Management#

Lead Section#

Incus is a modern, secure system container and virtual machine manager that provides a unified experience for running full Linux systems. It can scale from a single instance on a laptop to a cluster of up to 50 servers, making it suitable for both development and production workloads.

For Bluefin users, Incus is included in the Developer Experience (DX) variant alongside approximately 67 additional development packages including Docker, Podman, LXC, QEMU, libvirt, and virt-manager. Unlike Docker and Podman which focus on application containers, Incus manages full system containers and virtual machines with enterprise-grade features like clustering, live migration, and advanced networking.

Incus was created as a community-driven alternative to Canonical's LXD, maintained by many of the same people who originally created LXD. The project represents a fork from the Linux Containers organization that maintains the same powerful feature set while being truly community-driven and vendor-neutral.

History and Technology Stack#

Relationship to LXD and LXC#

Incus uses LXC (Linux Containers) as the underlying technology for implementing system containers, specifically using liblxc along with Linux kernel features like namespaces and cgroups. For virtual machines, Incus uses QEMU. This layered approach means Incus provides the management layer while LXC handles the low-level container runtime.

The technology lineage:

• LXC: Low-level container runtime using kernel features
• LXD: Management layer built on LXC (Canonical)
• Incus: Community-driven fork of LXD with vendor neutrality

Incus includes a tool called lxd-to-incus that enables in-place migration from LXD to Incus, transferring the entire database and storage to create an identical setup. It's been tested with LXD versions from 4.0 LTS through 5.21.

Release Strategy#

Incus has two release types:

• LTS releases: Incus 6.0 LTS is supported until June 2029 - recommended for production
• Feature releases: Monthly releases with latest features, supported until the next release

Architecture#

Client-Daemon Model#

Incus operates on a client-daemon architecture:

The Daemon (incusd): The Incus daemon runs on Linux and manages all instances, storage, and networking. It provides a RESTful API over HTTP that handles all communication, either over TLS for remote operations or via Unix socket for local access.

The Client (incus): The client tool can run on Linux, macOS, or Windows and communicates with the daemon through the REST API. All operations are available through both the command-line interface and the REST API directly.

Access Control: Local access through the Unix socket grants full administrative access to members of the incus-admin group, while remote network access supports multiple authentication and authorization methods including TLS certificates, OpenID Connect (OIDC), OpenFGA for fine-grained authorization, and scriptlet-based authorization.

Core Capabilities#

Instance Types#

Incus supports three types of instances:

1. System Containers#

System containers simulate a full operating system using Linux kernel features (namespaces and cgroups). They:

• Share the host's kernel
• Use minimal resources and are extremely lightweight
• Can only run Linux
• Are fast to start (near-instant)
• Are unprivileged by default for security
• Provide a near-VM experience without hardware virtualization requirements

2. Application Containers#

Incus can also run application containers by consuming OCI-compatible images from registries like Docker Hub. This is implemented through liblxc with help from umoci and skopeo, allowing single-application containers similar to Docker but within the Incus management framework.

3. Virtual Machines#

Virtual machines in Incus use QEMU and hardware virtualization to provide:

• A dedicated kernel per VM (can run non-Linux operating systems)
• Hardware isolation and full PCI device pass-through
• Support for workloads requiring specific kernel modules
• A built-in agent for seamless command execution and file transfers

System containers are faster and more lightweight but share the host kernel, while VMs provide complete isolation with their own kernel but use more resources.

Key Features#

Storage Management#

Incus supports multiple storage backends including dir, btrfs, LVM, ZFS, Ceph, and more. Features include:

• Storage pools with configurable volumes
• Storage buckets with S3 protocol support
• Snapshots and backup/recovery
• A new lvmcluster driver for shared storage in clusters

Networking#

Advanced networking capabilities include:

• Managed bridge networks with DHCP and DNS
• OVN (Open Virtual Network) integration for software-defined networking
• Network ACLs, forwards, and load balancers
• BGP support for routing
• Cross-host tunneling

Clustering#

Incus can cluster up to 50 servers together, providing:

• Live migration of instances between cluster members
• Automatic evacuation of failed nodes
• Distributed storage and networking
• Central management API for the entire cluster

Image Management#

Incus is image-based with official images for many distributions available at images.linuxcontainers.org. Images are automatically updated, cached, and can be customized with templates.

Projects and Multi-Tenancy#

Projects provide compartmentalization of instances, images, and profiles, enabling multi-tenant environments with isolated resources and separate access controls.

Security#

• Unprivileged containers by default (where container uid 0 maps to an unprivileged user on the host)
• Resource restrictions via cgroups
• AppArmor/SELinux profiles and seccomp filters
• Fine-grained authorization with multiple authentication backends

Installation on Bluefin#

Bluefin DX (Recommended)#

Incus is pre-installed in the Bluefin DX variant. The DX build includes three Incus-related packages:

• incus - The main Incus container/VM manager
• incus-agent - Agent for running inside Incus instances
• lxc - Linux container userspace tools

These packages are installed during build time in the immutable base image, preventing the common issue of package-based system degradation over time.

Automatic Group Configuration#

The bluefin-dx-groups service automatically:

• Creates the incus-admin group if it doesn't exist
• Adds all wheel group members to incus-admin, docker, and libvirt groups

Users can also manually configure groups using the ujust dx-group command.

SELinux Workaround#

Bluefin includes a dedicated systemd service for fixing Incus SELinux labels at system_files/dx/usr/lib/systemd/system/incus-workaround.service. This service:

1. Copies Incus binaries and libraries to /usr/local/bin/overrides/ and /usr/local/lib/overrides/
2. Bind mounts them over the originals
3. Runs restorecon to fix SELinux labels on /usr/bin/incus, /usr/bin/incus-agent, and /usr/lib/incus
4. Cleans up on service stop by unmounting and removing copies

This workaround is necessary because Bluefin uses an immutable OSTree-based filesystem where SELinux contexts on /usr cannot be modified directly. The bind mount + restorecon approach allows fixing labels in mutable /usr/local.

Incus-Distrobox (Alternative)#

For users who prefer containerized Incus, the ublue-os/toolboxes repository provides an incus-distrobox container that:

• Uses Ubuntu as the base with Incus packages from Zabbly
• Requires rootful distrobox with systemd init
• Mounts /var/lib/incus and /lib/modules from the host
• Adds users to the incus-admin group during initialization

Standard Bluefin#

Incus is not included in the base Bluefin installation. Users must either:

1. Upgrade to Bluefin DX using the rebase instructions
2. Use the incus-distrobox approach
3. Install via Homebrew (not recommended for core system containers)

Usage and Common Use Cases#

1. Development Environments#

Create isolated, reproducible dev environments that are faster and lighter than VMs but more complete than application containers. System containers provide a full distribution environment while remaining lightweight.

2. Testing and CI/CD#

Integration with tools like Terraform, Ansible, and Packer enables infrastructure-as-code workflows. GARM enables self-hosted GitHub Actions runners within Incus containers or VMs.

3. Multi-tenant Hosting#

Projects with resource restrictions enable hosting multiple isolated workloads with separate access controls.

4. Kubernetes Clusters#

Cluster API provider for Incus enables deploying K8s clusters on Incus infrastructure.

5. Private Cloud Infrastructure#

Clustering, live migration, and storage replication provide cloud-like capabilities on-premises.

6. Mixed Workload Environments#

Run both containers and VMs on the same infrastructure with unified management.

7. Application Isolation#

Run different versions of software, legacy applications, or security-sensitive workloads in isolated environments.

Comparison with Other Container Technologies#

Incus vs Podman#

Podman is the default container runtime on Bluefin, providing rootless container execution by default. Podman offers Docker CLI compatibility while integrating natively with systemd for container lifecycle management.

Key Differences:

• Podman: Application containers, rootless by default, OCI image focus, systemd integration
• Incus: System containers and VMs, infrastructure management, clustering, live migration

Incus vs Docker#

Docker is available on standard Bluefin via Homebrew installation. Bluefin LTS includes containerd from EPEL by default. Docker is set up as the default runtime for VS Code devcontainer workflows.

Key Differences:

• Docker: Application containers, development workflows, CI/CD pipelines
• Incus: Full system environments, VM management, clustering, multi-tenancy

Incus vs Distrobox/Toolbox#

Universal Blue provides development toolboxes via the ublue-os/toolboxes repository, a centralized collection of containers designed for Toolbox/Distrobox.

Key Differences:

• Distrobox: Quick dev environments, distribution package access, home directory integration
• Incus: Full system management, resource isolation, clustering, production workloads

Best Practices on Bluefin#

1. Preserve Immutability#

By containerizing development environments, the system image remains pristine, making upgrades less problematic with no package-based degradation over time. Use container-based approaches rather than attempting to layer packages on the immutable base.

Local package layering is locked by default via LockLayering=true in /etc/rpm-ostreed.conf, which prevents mutation of the base OSTree commit.

2. Leverage Container-First Philosophy#

Bluefin implements a container-first development philosophy designed around the principle that workflows remain OS agnostic using podman, docker, and flatpak. This approach embraces cloud-native development patterns where development happens in containers rather than directly on the host system.

3. Use Appropriate Technology#

Choose the right tool for the job:

• Flatpak: Graphical applications
• Podman/Docker: Application containers and services
• Distrobox: Quick distribution environments
• Incus: System containers, VMs, infrastructure management

4. SELinux Considerations#

When mounting directories from your home folder, you need to handle SELinux labeling. Use --security-opt label=disable when mounting directories from your home folder to prevent permission errors.

5. System Maintenance#

Relevant Code Files#

Related Topics#

• Containerized Development Environments: Distrobox, Dev Containers, and development workflows
• Container Runtimes: Podman, Docker, and OCI ecosystem
• Virtualization: QEMU, libvirt, virt-manager integration
• Immutable Operating Systems: OSTree, bootc, and image-based systems
• SELinux: Security contexts and labeling in immutable systems

See Also#

• Incus Official Documentation
• Bluefin DX Documentation
• Bluefin Administration Guide
• LXC Security Documentation
• ublue-os/toolboxes Repository