Build System#
How to build Dakota live ISOs locally and the key variables that control the build.
Two variant types ā understand this before changing anything#
Composefs variants embed an unpacked containers-storage graphroot at
/var/lib/containers/storage, bound to the explicitly mounted read-only
squashfs at /run/bootcmedia/var/lib/containers/storage ā the store files
share squashfs blocks with the live root (single-copy media). Non-composefs
variants embed one OCI layout at /usr/share/bootc-installer/oci, bound at
/run/bootcmedia/usr/share/bootc-installer/oci.
| Variant | composefs | squash before embed? |
|---|---|---|
dakota | true | yes ā squash to 1 layer |
utah (active), bluefin, bluefin-lts-hwe (retired) | false | no ā preserve ostree commit layers |
Payload preparation lives in one place: payload_to_oci_archive in scripts/payload-oci.sh, called by both scripts/iso-sd-boot.sh and scripts/build-live-squashfs.sh. It uses podman build only (no buildah): composefs payloads are built with --squash-all; non-composefs payloads keep their layers so ostree commit objects survive.
Changes to the producer must cover both active backends: Dakota (composefs) and Utah (ostree).
Quick start#
# Build and test dakota (composefs)
just debug=1 iso-sd-boot dakota
just plain-test-qemu dakota # must exit: ā
Installed system boot verified
# Build and test utah (non-composefs)
just debug=1 iso-sd-boot utah
just plain-test-qemu utah
Output: output/<target>-live.iso
Key variables#
| Variable | Default | Override example |
|---|---|---|
debug | 0 | debug=1 ā SSH enabled (liveuser/live, root/root) |
installer_channel | stable | installer_channel=dev ā continuous-dev Flatpak |
output_dir | output | output_dir=/var/data/iso |
workdir | output_dir | workdir=/mnt ā use XFS loopback on BTRFS hosts |
compression | fast | compression=release ā ~20% smaller, ~5Ć slower |
Never use debug=1 for production/release ISOs.
Never use installer_channel=dev in production builds ā see known regression in ci.md.
Disk space requirements#
Allow space for the prepared OCI archive, its extracted layout, the exported
live root, squashfs and ISO (at least 25 GB plus podman build storage). There
is no VFS import. Delete intermediates only after successful assembly.
ā ļø Never build from /tmp ā it is a 16 GB tmpfs. Always use a path on /var or another
large filesystem.
Rootless builds (no sudo)#
The justfile uses podman unshare which requires rootless podman (non-root user).
Never prefix just with sudo locally ā this breaks rootless podman with
please use unshare with rootless.
CI runs as root (sudo just ...) ā the justfile detects root via id -u and skips
podman unshare automatically.
BTRFS hosts ā use the XFS loopback#
BTRFS handles chunkified layers slowly even after squashing. Use the XFS loopback:
sudo just mount-xfs # creates 45 GB XFS at /mnt (idempotent)
sudo chown jorge:jorge /mnt # make accessible rootless
just workdir=/mnt iso-sd-boot dakota
Background builds#
just installer_channel=dev build-bg dakota
# Ctrl-C stops the log tail ā build continues running
# Check progress: tail -f output/build.log
Uses setsid sudo just ... & disown internally so the build survives terminal closure.
ā ļø build-bg requires sudo and a TTY. In agent/headless sessions (no TTY), sudo
fails silently with a terminal is required to read the password. Use direct backgrounding
instead:
LOG=output/build.log
mkdir -p output
setsid bash -c "just output_dir=output iso-sd-boot dakota > '${LOG}' 2>&1" &
disown $!
echo "Build started ā ${LOG}"
# Monitor:
tail -f output/build.log
Compression presets#
just compression=fast iso-sd-boot dakota # default ā fast CI/local
just compression=release iso-sd-boot dakota # production ISOs for R2
Use fast for CI and local testing. Use release for ISOs that go to R2.
Shared bound-media producer#
Both CI and local builds share the payload builder (payload_build_final,
retaining composefs --squash-all and preserving all ostree layers for Utah).
Composefs payloads are exported to a vfs containers-storage (payload_export_store)
and bound into the squashfs root (payload_bind_store), which writes schema-v1
/etc/bootc-installer/media-input.json from the installer-observed digests.
Non-composefs payloads keep the OCI archive path (payload_to_oci_archive +
payload_embed_oci_archive). No second archive, second squashfs, or runtime
export is used. See architecture.md
for the full binding and the native fisherman's mandatory read-only/hash preflight.
Use podman build with the FROM-only Containerfile, never create/commit (which
rewrites Entrypoint/config and breaks installed bootc).
Source layout: live/src/#
live/src/ is the single source of truth for the live ISO source tree.
| Path | Used by | Notes |
|---|---|---|
live/src/ | CI (build-iso.yml), live/Containerfile | Canonical for CI builds |
The live container (live/Containerfile) is used for both local and CI builds.
live/src/flatpaks is the definitive list of bundled Flatpaks.
A second copy of this tree previously lived at dakota/src/. It had no build
consumer, silently diverged from live/src/, and was removed; local justfile
recipes now call the live/src/ helpers directly.
# Quick headless QEMU test ā watch for DAKOTA_LIVE_READY on serial (Ctrl-A X to quit)
just boot-iso-serial dakota
# Full libvirt VM with SSH (requires debug=1 build)
just debug=1 iso-sd-boot dakota
just debug=1 boot-libvirt-debug dakota
# SSH: liveuser@<IP> password: live
# Cleanup: sudo virsh destroy dakota-debug && sudo virsh undefine dakota-debug --nvram
Justfile recipe reference#
| Recipe | Description |
|---|---|
iso-sd-boot <target> | Full build ā container + ISO assembly |
container <target> | Build the live-env container only |
build-bg <target> | Background build with live log tail |
mount-xfs | Create 45 GB XFS loopback at /mnt (sudo, idempotent) |
boot-iso-serial <target> | Boot ISO in QEMU, serial output (Ctrl-A X) |
boot-libvirt-debug <target> | Boot in libvirt, waits for DHCP + SSH |
e2e <target> | Build ISO + full LUKS E2E test |
Lessons#
Unified ISO: one nvidia image for all hardware (2026-06)#
Dakota ships a single ISO built from ghcr.io/projectbluefin/dakota-nvidia:stable.
The live environment runs the nvidia image. At install time, bootc-installer's
nvidia_imgref mechanism auto-detects the GPU:
- NVIDIA GPU present: installs
dakota-nvidia:stable,targetImgref=dakota-nvidia:stable - No NVIDIA GPU: installs offline from the same OCI layout,
targetImgref=dakota:stableā
firstbootc upgraderebases to the correct non-nvidia variant automatically
Dakota single-copy target: ā¤6.5 GB. The unpacked store shares squashfs blocks
with the live root, so the ISO carries the payload (plus Flatpaks) once ā
measured: an opaque layer blob adds +100% overhead, unpacked files add ~0%.
Record the actual ISO byte size (stat -c%s) and confirm there is exactly one
offline source and no OCI archive blob; never restore a second copy to shrink
the ISO or squash Utah's required ostree layers.
CI uses SUPERISO_COMPRESSION=release (2026-06)#
The build-iso.yml CI workflow sets SUPERISO_COMPRESSION=release in the squashfs build
step, producing zstd-15 compression. Local just iso-sd-boot defaults to compression=fast
(zstd-3). For production ISOs destined for R2, always use release compression:
just compression=release iso-sd-boot dakota
dakota/src/flatpaks diverged from live/src/flatpaks (2026-06, resolved)#
dakota/src/flatpaks contained be.alexandervanhee.gradia while live/src/flatpaks
did not, so CI builds omitted Gradia. The unreferenced dakota/src/ tree has since been
removed, leaving live/src/flatpaks as the only Flatpak list.
/tmp is a 16 GB tmpfs on this host. A Dakota build needs ~22 GB peak. The build
does not fail immediately ā it runs out of space mid-squash and produces a truncated
or corrupt ISO that fails to boot. Always use /var or an explicit output_dir.
buildah commit --squash vs podman create --entrypoint (2026-05)#
podman create --entrypoint /bin/sh && podman commit modifies the recorded Entrypoint
in the image config. Dakota/bootc images have no Entrypoint by design; a fake one causes
bootc install to fail with "cannot execute binary file". At the time the fix was
buildah commit --squash, which squashes layers without touching config. Superseded
by podman build (next lesson).
podman build replaces buildah for the payload squash (2026-10)#
buildah cannot be installed on a bootc image-mode host (no Homebrew formula, no
rpm-ostree install), so local ISO builds on Bluefin needed a hand-rolled shim. The
payload is now prepared by scripts/payload-oci.sh with podman alone:
FROM <payload>
COPY bootc-root-mount.toml /usr/lib/bootc/install/00-defaults.toml
built with podman build --pull=never [--squash-all] and written out with
podman push <tag> oci-archive:<file>:<payload-ref>.
Why this is safe where podman create && podman commit is not: a build inherits
the base image config (Entrypoint, Cmd, Env, Labels) from FROM; it never runs a
container with an overridden entrypoint, so nothing rewrites the config. COPY writes the
file straight to its destination, so no RUN step is needed either.
tests/test_live_build_invariants.py rejects buildah, podman create and
podman commit in the payload scripts. To check a built payload by hand:
skopeo inspect --config oci-archive:<payload.oci.tar>:<ref> | jq '.rootfs.diff_ids | length, .config.Entrypoint'
skopeo inspect --raw oci-archive:<payload.oci.tar>:<ref> | jq '.annotations["ostree.final-diffid"]'
Expect 1 diff_id and the same Entrypoint as the source image (skopeo inspect --config docker://<payload>).
dakota/src/install-flatpaks.sh was dead code (removed 2026-09)#
dakota/src/install-flatpaks.sh used to be a parallel copy of
live/src/install-flatpaks.sh, kept in sync by hand. It stopped being referenced by
any Containerfile once all variants moved onto live/Containerfile / live/src/, but
the file (and this note telling people to keep it in sync) stuck around ā silently
drifting to a release-tag scheme (continuous/continuous-dev) that no longer exists
upstream. Removed rather than fixed, since nothing built from it.
live/src/install-flatpaks.sh is the only copy; there is nothing left to mirror.
filesystem choice: always btrfs for dakota (2026-06)#
The dakota-nvidia:stable initramfs (built by freedesktop-sdk) includes btrfs.ko
but not xfs.ko. Verify with:
# Extract and inspect the installed initramfs
podman run --rm ghcr.io/projectbluefin/dakota-nvidia:stable bash -c '
python3 -c "
data = open(\"/usr/lib/modules/7.0.7/initramfs.img\", \"rb\").read()
idx = data.find(b\"TRAILER!!!\")
after = (idx + 10 + 511) & ~511
import subprocess, sys
payload = data[after:]
open(\"/tmp/p.zst\", \"wb\").write(payload)
"
zstdcat /tmp/p.zst | cpio -it 2>/dev/null | grep -iE "xfs|btrfs"
'
Always use filesystem: btrfs in fisherman recipes. Do NOT use btrfsSubvolumes: true
ā the subvolume setup interacts poorly with bootc's root-mount-spec config injection
(see docs/ci.md for the full root cause chain).
root-mount-spec injection: how and why (2026-06)#
bootc install to-filesystem auto-detects the root filesystem UUID using findmnt
inside a nested podman run. Inside that container, the udev database is not mounted,
so findmnt --output UUID returns empty and the install fails.
The iso-sd-boot recipe in the justfile injects root-mount-spec = 'LABEL=root' into
/usr/lib/bootc/install/00-defaults.toml inside the squashed OCI payload before
building the squashfs. This is safe because fisherman always formats with -L root.
If you see No filesystem uuid found in target root ā check that:
- The payload build ran (look for the
Squashingline and aCOPY bootc-root-mount.toml
step in the build log) /usr/lib/bootc/install/00-defaults.tomlin the payload containsroot-mount-spec- The injected config has no duplicate
[install]section headers
ostree.final-diffid: composefs vs ostree-native images (2026-06)#
Squashing the payload (podman build --squash-all) produces a regular filesystem tar (the merged OS files), not
an ostree commit blob.
- Composefs images (Dakota):
ostree.final-diffidmust be updated to point to the
new squashed layer's diff_id ā bootc reads it to locate the composefs commit layer. - Ostree-native images (bluefin/Silverblue): the annotation must be removed.
If it stays, bootc uses the "ostree-encapsulation" install path and fails with
Expected commit object, not Fileā it's looking for an ostree commit blob but the
squashed layer is a filesystem tar.
The iso-sd-boot recipe detects composefs mode from live/src/<target>/composefs and
either updates or removes the annotation accordingly.
/run overlay size: 28 GB RAM needed for bluefin offline install (2026-06)#
The live environment's /run tmpfs defaults to 20% of VM RAM (e.g. 5.6 GB on a 28 GB VM).
When fisherman runs podman run containers-storage:bluefin-nvidia, the VFS storage driver
creates a full 8.8 GB copy of the image in /var/lib/containers/storage/vfs/dir/ (the
overlay upper dir at /run/overlayfs). With a 5.6 GB /run, the copy immediately fails
with "no space left on device".
Fix: sudo mount -o remount,size=24G /run before running fisherman. This is automated
in configure-live.sh for bluefin (adds a systemd dropin that remounts /run at boot).
Alternatively, set rd.live.overlay.size=24576 in boot entries.
For QEMU testing of bluefin installs: use at least 28 GB RAM (-m 28672).
With 16 GB, the VFS container creation runs out of memory on the RAM-backed overlay.
Ghost lab: ISO testing via Argo Workflows (2026-06)#
ghost is the k3s control-plane + KubeVirt compute node. All ISO builds and
install testing run as local ad-hoc Argo Workflows submitted via the argo-mcp
pi extension. Direct SSH builds are forbidden ā they bypass cluster scheduling,
consume untracked resources, and have crashed the node.
ā NEVER ssh ghost to run builds.
ā NEVER rsync code to ghost and run just directly.
Every workload must be submitted as an Argo Workflow. Always inspect what
WorkflowTemplates exist in the cluster before submitting:
k8s_nodes_top # check headroom first
argo_list_workflow_templates namespace=argo # find the right template
argo_submit_workflow namespace=argo manifest=<yaml>
argo_logs_workflow name=<name> namespace=argo tailLines=100
Do NOT manually push ISOs to R2. Let CI handle all R2 uploads.
The latest pointer is the production artifact ā only CI may write it after
passing the boot verification gate. See docs/r2-promotion.md.
Variant image refs must be verified against the publishing workflow (2026-06)#
Never guess image names for a variant. Always verify what projectbluefin actually
publishes by reading the execute-release.yml workflow in the source repo:
gh api repos/projectbluefin/bluefin-lts/contents/.github/workflows/execute-release.yml \
--jq '.content' | base64 -d | grep 'image'
The pattern for confirming a tag exists:
skopeo list-tags docker://ghcr.io/projectbluefin/<image> | python3 -c \
"import json,sys; print([t for t in json.load(sys.stdin)['Tags'] if t in ('stable','lts','latest')])"
projectbluefin/bluefin-lts publishes: bluefin-lts:stable, bluefin-lts-hwe:stable, bluefin-lts-hwe-nvidia:stable.
There is no bluefin-lts-nvidia without -hwe. There is no bluefin-gdx in projectbluefin.
systemd-boot title comes from live_title file (2026-06)#
Each variant directory has a live_title file. build-iso.sh reads it via --title.
To change what users see in the boot menu, edit that file ā do not touch build-iso.sh.
libblkid/libuuid must come from base image, not Debian stage (2026-06)#
live/Containerfile stage 3 copies some libs from Debian bookworm for mkfs.xfs:
libinih.so.1 and liburcu.so.8 genuinely must come from Debian (wrong version in Fedora / absent in GNOME OS).
Do NOT copy libblkid.so.1 or libuuid.so.1 from Debian. Both the freedesktop-sdk
(dakota) and Fedora (bluefin) base images ship their own newer versions at
/usr/lib/x86_64-linux-gnu/. Overwriting them with Debian bookworm's older copy
(only BLKID_2_21) breaks sfdisk which requires BLKID_2_40 via libfdisk.so.1:
sfdisk: /usr/lib/x86_64-linux-gnu/libblkid.so.1: version `BLKID_2_40' not found
mkfs.xfs (Debian-compiled) works fine with the newer system libblkid because it
is backward-compatible with all earlier symbol versions. Only copy what is genuinely
missing from the target base image.
mksquashfs -e proc/sys/dev removes empty dirs in 4.7+ (2026-06)#
dmsquash-live-root (Debian bookworm dracut) needs a proc/ directory at the squashfs root to use the squashfs directly as the live rootfs. Without it: FATAL: Failed to find a root filesystem in squashfs.img.
dracut's usable_root() function also requires all three of proc/, sys/, and dev/ at the squashfs root. On modern GNOME OS (glibc 2.38+), there is no ld-2.XX.so file ā only ld-linux-x86-64.so.2 (which does not match the ld-*.so glob). So the second fallback check (proc sys dev all exist) is the only path that works.
Old mksquashfs (ā¤4.6, Ubuntu 22.04/24.04 system package) with -e proc excluded proc's CONTENTS but kept the empty directory. mksquashfs ā„4.7 (homebrew, newer distros) removes the directory itself. Same applies to sys and dev.
Fix in scripts/build-live-squashfs.sh and justfile:
mkdir -p "${SFS_ROOT}/proc" "${SFS_ROOT}/sys" "${SFS_ROOT}/dev" # ensure empty dirs exist
mksquashfs ... -wildcards -e "proc/*" -e "sys/*" -e "dev/*" ... # exclude contents only
Non-composefs variants (bluefin, bluefin-lts-hwe): recipe.json fields that must be set (2026-06)#
Two fields in recipe.json were missing/wrong for Fedora/CentOS variants, causing every install to fail:
1. filesystem was never set ā fisherman defaulted to xfs. All variants must install to btrfs.
The fix: configure-live.sh now sets recipe["filesystem"] = "btrfs" for all variants.
XFS is only a user option in the installer UI (images.json "filesystems" array), and only for non-composefs variants: fisherman rejects composefs on XFS (no fs-verity), so Dakota offers btrfs alone.
2. local_imgref pointed at the wrong store ā non-composefs variants embed the payload as
an OCI layout at oci:/var/lib/containers/oci-store, NOT as VFS containers-storage.
The old code set local_imgref = "containers-storage:<NVIDIA_IMGREF>", which pointed at a
VFS store that doesn't exist for these variants. Fisherman could not find the offline image.
Fix:
if composefs == "true":
recipe["image"] = f"containers-storage:{nvidia_imgref}"
recipe["local_imgref"] = f"containers-storage:{nvidia_imgref}"
else:
recipe["image"] = "oci:/var/lib/containers/oci-store"
recipe["local_imgref"] = "oci:/var/lib/containers/oci-store"
recipe["filesystem"] = "btrfs"
Rule: never test recipe changes with CI only. Build the container locally, run
podman run --rm <image> cat /etc/bootc-installer/recipe.json to verify the recipe,
then do a local QEMU install test before pushing to CI.
justfile iso-sd-boot must branch on composefs for OCI store embed (2026-06)#
configure-live.sh sets local_imgref differently per variant type:
- composefs (dakota):
containers-storage:<NVIDIA_IMGREF>ā VFS store at/var/lib/containers/storage - non-composefs (bluefin, lts):
oci:/var/lib/containers/oci-storeā OCI layout
The justfile iso-sd-boot recipe previously always used the VFS containers-storage path
regardless of composefs setting. For bluefin/lts-hwe, the installer looked for
oci:/var/lib/containers/oci-store but found nothing there ā causing:
fisherman: fatal: bootc install: pulling image: podman pull oci:/var/lib/containers/oci-store: exit status 125
Superseded (2026-10): the squash rules below are historical. Non-composefs payloads
now keep their layers (squashing destroys the ostree commits) and buildah is no longer
used. See the variant table at the top of this file and "podman build replaces buildah
for the payload squash". The composefs branching itself still applies.
Fix: iso-sd-boot now reads live/src/<target>/composefs and branches before squash:
composefs=trueā squash to 1 layer ā single-copy VFS store via
payload_export_store(podman-only:savestreams the tag from the default
store intoloadon a custom--root/--runroot/--storage-driver vfsstore ā
apull containers-storage:<tag>against the empty custom store cannot
resolve it ā thentagto the target identity,image inspectfor
digest/platform,saveā manifest.json for the config digest). Skopeo is
not in this path:skopeo copyinto a custom store needs a mount namespace
the build host denies (unshare(CLONE_NEWNS)ā EPERM), while podman brings
a working namespace setup. The runroot is a fixed short scratch dir
(/tmp/pstore-run.XXXXXX): podman caps runroot at 50 chars. The receipt
bindscontainersStoragePath: /var/lib/containers/storageā no second OCI
blob.composefs=falseā keep all layers (squashing destroys ostree commits) ā
podman pushto an OCI layout bound atociLayoutPath.
Critical: branch before squash. The composefs flag must be read before any squash
operation. Dakota (composefs) squashes; Utah/non-composefs must NOT squash.
This mirrors scripts/build-live-squashfs.sh exactly.
live/src/<target>/composefs is the single source of truth ā both scripts read it.
Installer bundle pin (2026-10)#
install-flatpaks.sh no longer follows the moving releases/latest/download/ redirect.
live/src/installer-bundle.lock names the exact tuna-os/bootc-installer release tag and
the SHA-256 of both bundles; the script downloads releases/download/<tag>/<file> and
fails the build (deleting the file) when the digest differs. To move to a new upstream
release run scripts/bump-installer-bundle.sh [<tag>] and commit the rewritten lock.
Installer layer cache busting (2026-06)#
The COPY src/flatpaks + RUN install-flatpaks.sh stage in the Containerfile is cached
by podman's layer cache. If neither file changes between builds, the old installer Flatpak
stays in the image indefinitely.
Fix: ARG CACHE_BUST=0 before the flatpak stage; justfile passes
--build-arg CACHE_BUST=$(date +%Y%m%d) to invalidate the layer once per calendar day.
To verify installer version: the metainfo XML in the flatpak may say an old version (not
updated by upstream). The real check is the fisherman binary at
var/lib/flatpak/app/org.bootcinstaller.Installer/.../files/bin/fisherman ā v3.x is a
Go binary containing [fisherman] version: strings and LUKS/flatpak/composefs logic.
Podman runroot 50-char limit + empty-placeholder bind gate (2026-11)#
Two failures combined to produce an ISO whose media-input.json carried empty
digests and then refused to assemble:
podman --runrootlonger than 50 characters fails every command
(Error: the specified runroot is longer than 50 characters). A staging dir
under a repo checkout (output/dakota-payload-store-run, 76 chars) exceeds
it ā and so doesmktempunder a longTMPDIR.payload_export_store
therefore uses a fixed short scratch runroot (/tmp/pstore-run.XXXXXX,
removed before returning) and asserts the length in unit tests.- The failure was silent: the caller captures the export in
$(...)
inside an||guard, whereset -eis suppressed, so failed
inspect/savecalls produced an empty receipt with exit 0. Every export
step now fails loudly (|| return 1with anERROR:line), the function
refuses empty receipt values,payload_bind_storerefuses empty inputs,
and both callers validate the receipt afterread. - The live image ships an empty
/var/lib/containers/storage
placeholder (configure-live.shmkdirs it alongsidestorage.conf), so
the squashfs root already contains that path after the overlay/cp step and
the old "refuse if exists" bind gate false-positived. The gate now takes
over an empty dir (rmdir + copytree) and still refuses symlinks, files,
and populated dirs. Unit harnesses replicate the placeholder in their
mount fixtures so the regression is covered.
Never edit a script while a build is running it. Bash reads the file
incrementally from a saved offset; changing bytes before the current offset
shifts everything after it, so the resumed read lands mid-line. Stage the
fix and apply it after the build finishes.
Bound store must be read-only at live boot (2026-11)#
Fisherman's store preflight requires the store dir on a read-only
filesystem, but the live root is an always-writable overlay ā CI E2E proved
every install refuses without it. Two-part fix, both required:
payload_export_storeredirects every*.lockin the vfs store to
symlinks under/run/bootc-installer-media-locks/(containers/storage
locks the graphroot even for reads; verified with skopeo inspect through
a read-only bind). Image bytes ā what the receipt digest covers ā are
untouched.configure-live.shinstallsvar-lib-containers-storage.mount, which
bind-mounts/run/bootcmedia/var/lib/containers/storageread-only over
the placeholder beforelive-media-setup.service, plus a tmpfiles entry
for the lock dir. Skipped when no store is embedded
(ConditionPathExists), so non-composefs variants are unaffected.
Flatpak remote descriptors are checked in (2026-10)#
install-flatpaks.sh registers Flathub (and any remote:app-id remote a variant
list names, currently tuna-os for Utah's Ghostty) from .flatpakrepo files under
live/src/remotes/, bind-mounted at /src/remotes by the Containerfile. A
.flatpakrepo carries Url= and GPGKey= ā the trust root every later download is
verified against ā and /var/lib/flatpak is copied onto installed systems via
flatpak_var_path, so it must be a reviewed file, not something fetched over the network
during the build. The build fails if a descriptor is missing.
To add a remote: drop live/src/remotes/<name>.flatpakrepo in, add [<name>]=1 to
KNOWN_REMOTES in the script, and reference it as <name>:<app-id> in the variant's
flatpaks list. tests/test_install_flatpaks.py pins the Flathub descriptor's SHA-256
and fails if flatpak remote-add is ever handed an https:// location again.
A Flathub or TunaOS key/URL rotation therefore shows up as a diff to a file in this
repository ā refresh it from the upstream URL, review the diff, and update the pinned
digest in the test.
CI bluefin OCI store verification (2026-06)#
build-iso-bluefin.yml previously checked var/lib/containers/storage/vfs-images (VFS path).
Bluefin/lts-hwe use composeFsBackend=false and embed at var/lib/containers/oci-store.
Fixed to check var/lib/containers/oci-store/index.json instead.
Non-composefs OCI must squash to 1 layer before embedding (2026-06-21)#
Superseded (2026-10): historical. Non-composefs payloads are no longer squashed ā
they keep their layers so ostree commit objects survive ā and buildah is no longer used.
See the variant table at the top of this file.
What failed: bluefin and bluefin-lts-hwe ISOs built at 12 GB instead of ~6 GB.
Why: buildah commit --format oci without --squash in the non-composefs path embeds all ~120 OCI layers as individual blobs into var/lib/containers/oci-store inside the squashfs. Each layer is ~60ā80 MB uncompressed; 120 layers ā ~8 GB OCI store inside a ~6 GB rootfs ā 12 GB squashfs ā 12 GB ISO.
Fix: Both justfile and scripts/build-live-squashfs.sh non-composefs paths must use --squash --format oci:
buildah commit --squash --format oci "${INJECT_CTR}" "oci:${OCI_DIR}:${OCI_IMAGE}"
Squashing reduces the embedded store to a single ~4 GB layer ā ~6 GB final ISO.
NEVER remove --squash from the composefs path. Dakota (composefs) squashes
for the single-copy VFS store; non-composefs targets (Utah) must NOT squash ā
squashing destroys the ostree commit objects the OCI layout must preserve.
VFS additionalimagestore requires squashing to prevent storage explosion (2026-06-23)#
Superseded (2026-10): historical. The
buildah commit --squashfix below no longer
applies; non-composefs payloads keep their layers and are prepared withpodman build
byscripts/payload-oci.sh. See the variant table at the top of this file.
What failed: Building stable and lts ISO targets failed in CI with no space left on device (ENOSPC) during the VFS import step of Build debug ISO.
Why: To support kernels without overlay-on-overlay, the non-composefs targets (stable, lts) use the vfs driver for their additional image store. If the payload image is NOT squashed before import, the VFS driver has to unpack and copy all ~120 layers sequentially. Because VFS lacks copy-on-write, this layer-on-layer unpacking causes an exponential disk space explosion (>100 GB), exhausting the runner's disk.
Fix: Ensure buildah commit --squash --format oci is used for the payload image in BOTH the justfile (inline iso-sd-boot) and scripts/build-live-squashfs.sh non-composefs paths before the skopeo copy ... containers-storage: import step. This squashes the payload to a single layer, preventing the VFS import explosion.
Mksquashfs silently skips bind-mounted dirs on overlayfs host (2026-06-23)#
Symptom: just iso-sd-boot ran successfully in CI, but the resulting live ISO was missing the entire embedded VFS store/OCI layout inside /usr/lib/containers/storage or /var/lib/containers/storage.
Why: In the justfile's iso-sd-boot target, the intermediate storage staging directory was bind-mounted (mount --bind) into the squashfs root. When the runner's build directory is on overlayfs (as in CI), mksquashfs respects filesystem boundaries (stops when the device ID changes) and silently skips the bind-mounted directory.
Fix: Avoid bind mounts when structuring filesystems inside SQUASHFS_ROOT. Use cp -a to copy the staged container storage directory directly into the squashfs root instead of bind-mounting.
Non-composefs bootcDirect QEMU E2E installs require the scratch disk (2026-06-23)#
Symptom: E2E test runs for stable and lts failed with no space left on device (ENOSPC) during bootc install to-filesystem inside the live VM.
Why: The justfile's luks-install-qemu and plain-install-qemu targets only mounted the /dev/vdb scratch disk over /var/tmp for composefs=true (dakota). However, bootc install on non-composefs targets also writes temporary layer/blob files to /var/tmp/container_images_... during its extraction/deployment phase. Without the scratch disk backing /var/tmp, bootc quickly exhausts the 4 GiB VM's RAM-backed overlay tmpfs.
Fix: Move the scratch disk mounting step outside the composefs checks in justfile's luks-install-qemu and plain-install-qemu targets so it is formatted and mounted over /var/tmp for all variants.
Architectural Decisions for OSTree (Stable/LTS) vs ComposeFS (Dakota) (2026-06-24)#
Our consolidation of OSTree (GRUB) and ComposeFS (systemd-boot) installer flows into a unified repository highlighted two distinct execution requirements:
-
Storage Drivers & Whiteouts:
- ComposeFS (
dakota) relies on a squashed, single-layervfsstorage driver. - OSTree (
stable,lts) requiresoverlaywithfuse-overlayfsmapping because the host live ISO runsdmsquash-liveoverlayfs, and CentOS 10/el10 kernels lack native overlay-on-overlay support. Standard OSTree layers must not be squashed during image construction to preserve commits integrity. Whiteouts are stripped out during the rsync staging phase via--no-specials --no-devices.
- ComposeFS (
-
Filesystem Selection & Boot UUID Timeout:
dakotaandstableinstall successfully using rawbtrfspartitions.ltstargets formatted as directxfsfail boot verification due to a/bootpartition ext4 UUID detection timeout in the CentOS 10 initramfs dracut loop.- Fix: LTS installations default to
xfs-in-lvmwhich mounts through LVM volume activation hooks, bypassing the udev device-by-uuid wait step and ensuring reliable boot discovery.
Debug ISO squashfs must mirror production mksquashfs flags (2026-06-21)#
What failed: build-iso.yml E2E step 1/4 (boot live ISO) ā dracut Warning: /sysroot has no proper rootfs layout ā Can't mount root filesystem. Production ISO was correct; boot failed because E2E uses the debug ISO (preferred by plain-boot-qemu-live).
Why: The debug ISO rebuild step in CI used -wildcards -e sys -e dev (without glob patterns). With -wildcards, mksquashfs treats -e sys as "exclude any path matching 'sys'" which removes the sys/ directory node entirely. dracut's usable_root() requires all three of proc/, sys/, dev/ as empty dirs. The production squashfs was built correctly (using -e "sys/*" -e "dev/*" and mkdir -p sys/ dev/) but the debug rebuild step had stale flags.
Fix in .github/workflows/build-iso.yml:
sudo mkdir -p /var/iso-build/debug-rootfs/proc \
/var/iso-build/debug-rootfs/sys \
/var/iso-build/debug-rootfs/dev \
/var/iso-build/debug-rootfs/run \
/var/iso-build/debug-rootfs/tmp
sudo mksquashfs ... -wildcards -e "proc/*" -e "sys/*" -e "dev/*" -e "run/*" -e "tmp/*"
Rule: Every squashfs packer must preserve empty proc/, sys/, dev/, run/ and tmp/, excluding only their contents with -wildcards -e "dir/*". Bare exclusions can remove directory nodes. Debian dracut 99base/init.sh retains /run/initramfs under /dev/.initramfs if the new root lacks /run, breaking post-boot media paths. Both active variants used this Debian fallback in the 2026-11-07 failing runs.
COMPOSEFS_BACKEND detection: never use sh -c 'python3 -c "..."' (2026-06-21)#
What failed silently: scripts/build-live-squashfs.sh always selected the non-composefs OCI layout path for ALL variants, including dakota (which needs the VFS composefs path). Dakota installs would fail because recipe.json says containers-storage:... but the offline store was embedded as an OCI layout at /var/lib/containers/oci-store.
Why: Shell quoting bug. The detection ran:
sh -c 'python3 -c "import json; print(json.load(open("/etc/bootc-installer/recipe.json"))..."'
Inside the container, sh sees the inner " after open( as closing the outer double-quoted string. python3 gets a malformed -c argument, exits with an error (suppressed by 2>/dev/null), output is empty, grep -qi true fails, COMPOSEFS_BACKEND stays false for ALL variants.
Fix: Avoid sh -c '...' wrapping and python3 entirely. The current probe in scripts/build-live-squashfs.sh runs grep directly as the container entrypoint and fails closed:
probe_rc=0
podman run --rm --entrypoint="" "${IMAGE}" \
grep -qi '"composeFsBackend": *true' /etc/bootc-installer/recipe.json \
|| probe_rc=$?
case "${probe_rc}" in
0) COMPOSEFS_BACKEND=true ;;
1) COMPOSEFS_BACKEND=false ;;
*) echo "ERROR: could not read composeFsBackend ..." >&2; exit 1 ;;
esac
Do not suppress stderr or pipe into grep -q: that collapses "recipe says false" and "probe could not run" into the same result. Only grep exit 1 means false; any other status (recipe.json missing, podman failure) aborts the build. In --target mode the value comes from variant_composefs in scripts/variant-config.sh instead.