CI/CD#
How the GitHub Actions workflows build, test, and publish Dakota ISOs.
ISOs produced#
Two ISOs are built and published to R2 (Dakota daily, Utah monthly; both also support workflow_dispatch):
| ISO | Workflow | R2 latest name | Image embedded | Trigger |
|---|---|---|---|---|
| Dakota | build-iso.yml | dakota-live-latest.iso | projectbluefin/dakota-nvidia:stable or a dispatched dakota-nvidia@sha256:... digest | Daily 03:00 UTC, workflow_dispatch, repository_dispatch (dakota-release, client_payload.image_ref) |
| Utah | build-iso-utah.yml | utah-live-latest.iso | projectbluefin/utah:stable | Monthly, 8th 04:00 UTC, workflow_dispatch |
The Bluefin and Bluefin LTS HWE ISOs are retired. As of
#228 this repo ships Dakota and Utah; build-iso-utah.yml runs monthly on the 8th at 04:00 UTC
(after Utah's Monday :stable release) and via workflow_dispatch.
See docs/variants.md for the full dormancy map and revival steps.
| ISO | Workflow | R2 latest name | State |
|---|---|---|---|
| Bluefin | build-iso-utah.yml | bluefin-live-latest.iso | retired (#228) β matrix entry disabled |
| Bluefin LTS HWE | build-iso-utah.yml | bluefin-lts-hwe-live-latest.iso | retired (#228) β matrix entry disabled |
| Utah | build-iso-utah.yml | utah-live-latest.iso | active monthly (8th 04:00 UTC) + dispatch |
The Dakota ISO is a unified NVIDIA ISO β the live environment boots the NVIDIA variant; the offline OCI store lets the installer deploy to non-NVIDIA hardware without a network pull (bootc auto-rebases on first upgrade).
To trigger a fresh Dakota publish:
gh workflow run build-iso.yml --ref main
Workflows#
| Workflow | File | Trigger |
|---|---|---|
| Dakota Build & Publish | build-iso.yml | Daily 03:00 UTC, workflow_dispatch, repository_dispatch (dakota-release, client_payload.image_ref) |
| Utah Build & Publish | build-iso-utah.yml | Monthly 8th 04:00 UTC, workflow_dispatch (bluefin entries retired in #228) |
| LUKS E2E Test | test-luks-install.yml | Push to main (docs ignored), weekly Mon 04:00 UTC, workflow_dispatch β dakota matrix only |
| Plain Install E2E | test-plain-install.yml | Push to main (docs ignored), merge queue (stable leg only), PRs (the Plain E2E gate check only), weekly Tue 04:00 UTC, workflow_dispatch β dakota matrix only |
| GUI Installer E2E | scheduled-gui-installer.yml | Weekly Wed 04:00 UTC, workflow_dispatch |
| ShellCheck Lint + Pinned fetches | lint.yml | PRs to main, push to main |
| Python Unit Tests | test.yml | PRs to main, push to main |
| R2 Download Stats | r2-download-stats.yml | Daily 06:00 UTC, workflow_dispatch β counts public ISO downloads from R2 Data Access Logs; needs CF_OBSERVABILITY_TOKEN secret |
CI alert cancellation handling (2026-11-14)#
Push-triggered E2E runs cancel older runs when a newer push arrives. The CI
alert checks all pages of runs for a newer push of the same workflow and branch
before opening or reopening an alert for a cancelled push. Scheduled runs do
not supersede push runs; a cancellation without a newer push still alerts
unless it had no jobs. API lookup failures fail the alert job rather than
silently suppressing an alert.
build-iso.yml#
Triggers: Daily 03:00 UTC (0 3 * * *), workflow_dispatch, repository_dispatch (dakota-release, client_payload.image_ref)
Job: build-and-publish (single job, no matrix)
Runner: ubuntu-24.04
Runs as: root via sudo
Path triggers: β removed; see lessons.live/**, scripts/**, .github/workflows/build-iso.yml
Pipeline steps#
- Free disk space β
ublue-os/remove-unwanted-softwarereclaims ~119 GB at/var/iso-build - Install deps β
apt-get install podman skopeo mtools xorriso squashfs-tools dosfstools isomd5sum - Log in to GHCR β
sudo podman login ghcr.io - Install cosign β
sigstore/cosign-installer(SHA-pinned) - Verify payload signature β the image ref defaults to
ghcr.io/projectbluefin/dakota-nvidia:stable; either dispatch may supply aghcr.io/projectbluefin/dakota-nvidia@sha256:<64 lowercase hex>digest, including the stable-promotion dispatch. Other image names and tags are rejected before verification. The digest form restricts the image name but does not independently prove promotion to stable.scripts/verify-image-signature.shchecks the keyless cosign signature against a projectbluefin workflow identity and the owning repository (projectbluefin/dakota), and prints the digest-pinned ref; the job fails if verification fails. - Pull payload image β removes any local
dakota-nvidia:stabletag, pulls the verified digest, and re-tags itdakota-nvidia:stablefor the later steps - Build live container β
podman build --pull=always --no-cache live/ --build-arg TARGET=dakota-nvidia --build-arg BASE_DIGEST=@sha256:β¦βlocalhost/dakota-nvidia-live:latest;BASE_DIGESTpins bothFROMlines inlive/Containerfileto the verified digest - Build live squashfs β
scripts/build-live-squashfs.shwithSUPERISO_COMPRESSION=releaseβ<target>.rootfs.sfs+<target>-boot.tar(~4.5 GB dakota, ~6 GB bluefin/lts-hwe) - Assemble ISO β
live/src/build-iso.shβdakota-live.iso(unpacked store in the live squashfs, deduped against the live root; no OCI archive blob, no--store) - Generate checksum β latest variant
- Plain-install E2E gates β live boot, strict bound-media preflight at 4 GiB, full offline install, installed boot and btrfs zstd/rootflags verification
- Boot verification β QEMU UEFI smoke boot on the production ISO
- Sign checksum β
cosign sign-blob(keyless, GitHub OIDC; the job hasid-token: write) writes<CHECKSUM>.sigstore.json(the Sigstore bundle) next to the checksum, and keeps the legacy<CHECKSUM>.sig/<CHECKSUM>.pempair beside it for now; runs under the upload's exact conditions and before it, so a signing failure publishes nothing (#360) - Upload to R2 + artifacts β only after ENOSPC, full install, installed-boot verification, production boot smoke, and signing all succeed
Do not add
--store, a second OCI archive, or an opaque payload blob alongside the store.
Composefs media embeds the unpacked store once (payload_export_store+
payload_bind_store) and emits a v1media-input.jsonbinding; non-composefs
media embeds one OCI layout (payload_embed_oci_archive). Missing or invalid
media must fail closed, not pull from the network. See the actual contract.
β οΈ installer_channel is locked to stable in CI#
Do NOT change installer_channel to dev in the live container build. There is an active
regression in the dev channel (tuna-os/fisherman#38) where the overlay storage
code path fails with:
open /var/tmp/oci-cache/index.json: no such file or directory
Production CI must stay on installer_channel=stable until the regression is fixed.
Both channels download from the release tag pinned in live/src/installer-bundle.lock
(releases/download/<tag>/), verified against the SHA-256 recorded there. Run
scripts/bump-installer-bundle.sh to move the pin to the current upstream release.
Disk layout in CI#
The build path is /var/iso-build (~119 GB free after disk-space action).
Peak usage ~22 GB (live squashfs ~6 GB + offline store ~6 GB + ISO ~5 GB + intermediate).
No XFS loopback needed in CI.
Boot verification logic#
CI accepts either:
DAKOTA_LIVE_READYwritten directly to/dev/ttyS0bylive-ready.serviceFinished live-ready.servicein the serial log (systemd journal console fallback)
Some dev channel builds don't write the serial marker but still reach GDM.
If both checks fail after 5 minutes, the job fails with tail -50 /tmp/serial.log.
R2 upload and backup rotation#
ISOs are uploaded to the testing bucket as latest pointers only β no dated objects.
Before uploading the new latest, the workflow rotates 3 backup slots:
backup-2 β backup-3
backup-1 β backup-2
latest β backup-1
new ISO β latest
Objects published per build:
dakota-live-latest.iso+-CHECKSUM+-CHECKSUM.sigstore.json+-CHECKSUM.sig+-CHECKSUM.pem(always overwritten; the.sigstore.jsonbundle is the primary artifact,.sig/.pemkept alongside for now)dakota-live-backup-{1,2,3}.iso+-CHECKSUM(rotated; slots beyond 3 are pruned; the
rotation moves only the ISO and checksum, so backup checksums carry no signature)
All are updated only after ENOSPC gate, full install, installed-boot verification,
and production ISO smoke boot all pass.
Verifying a downloaded ISO#
The checksum and the ISO are uploaded by the same job with the same credentials, so
the checksum on its own only guards against a corrupted download β anyone who can
write the bucket can replace both consistently (#220). The checksum is therefore
signed with keyless cosign; the signature binds it to a GitHub Actions run of this
repository's publish workflow, which a bucket write cannot forge.
NAME=dakota-live-latest.iso # or utah-live-latest.iso
for f in "$NAME" "$NAME-CHECKSUM" "$NAME-CHECKSUM.sigstore.json"; do
curl -fsSLO "https://projectbluefin.dev/$f"
done
# 1. The checksum was produced by build-iso.yml / build-iso-utah.yml on main.
# Primary: verify the Sigstore bundle cosign v3 publishes. The identity pin is
# the same as the command below; keep it exact (#259).
cosign verify-blob "$NAME-CHECKSUM" \
--bundle "$NAME-CHECKSUM.sigstore.json" \
--certificate-oidc-issuer https://token.actions.githubusercontent.com \
--certificate-identity-regexp \
'^https://github\.com/projectbluefin/dakota-iso/\.github/workflows/build-iso(-utah)?\.yml@refs/heads/main$'
# Alternative: the legacy separate signature + certificate are still published.
# cosign verify-blob "$NAME-CHECKSUM" \
# --signature "$NAME-CHECKSUM.sig" \
# --certificate "$NAME-CHECKSUM.pem" \
# --certificate-oidc-issuer https://token.actions.githubusercontent.com \
# --certificate-identity-regexp \
# '^https://github\.com/projectbluefin/dakota-iso/\.github/workflows/build-iso(-utah)?\.yml@refs/heads/main$'
# 2. The ISO matches the verified checksum.
sha256sum -c "$NAME-CHECKSUM"
Both commands must succeed. Pin the identity exactly as above: a regexp that only
anchors on the organisation would accept a signature from any projectbluefin
workflow (the mistake scripts/verify-image-signature.sh fixed in #259).
Only the latest checksum is signed. Backup slots lose the signature in rotation, and
promote-release.yml rewrites the checksum for the promoted filename, so named releases
(dakota-live-alpha6.iso etc.) have no .sigstore.json / .sig / .pem yet β verify
those by promoting only from a latest whose signature you have checked first.
README table auto-refresh#
After a successful Dakota upload, build-iso.yml auto-commits a README update:
the | \dakota` |row is rewritten with the current ISO size, publish date, and a link to the CI run. This requirescontents: write` permission on the job.
β οΈ Direct uploads from the local host hang (routing issue). Always use R2βR2
server-side copies via rclone for local promotion. See docs/r2-promotion.md.
build-iso-utah.yml#
Triggers: Monthly on the 8th, 04:00 UTC (0 4 8 * *), workflow_dispatch.
Matrix: utah (active); bluefin and bluefin-lts-hwe retired in #228.
Runner: ubuntu-24.04
This workflow uses the same bound-media producer and shipped fisherman consumer
as Dakota. A debug-only copy of the exact production squashfs enables SSH via
scripts/build-debug-iso.sh; production media and payload bytes remain unchanged.
It requires strict media preflight at 4 GiB, a complete offline install at 8 GiB,
installed-system boot and btrfs compression/rootflags verification, plus the
production ISO smoke boot, before signing or publishing. Bluefin and Bluefin LTS
matrix entries remain retired and commented out.
Boot verification: use AHCI, not SCSI CD (2026-06)#
The smoke boot uses OVMF + QEMU. GitHub Actions runners have no KVM β they run
-cpu qemu64 (software emulation). Without KVM, the SCSI bus enumeration in OVMF
is too slow: OVMF never discovers the SCSI CD and falls straight through to PXE boot,
failing with BdsDxe: No bootable option or device was found.
Do not use:
-device virtio-scsi-pci,id=scsi0
-device scsi-cd,drive=iso,bus=scsi0.0
Use instead:
-device ich9-ahci,id=ahci0
-device ide-cd,drive=iso,bus=ahci0.1
OVMF reliably auto-discovers AHCI optical drives on q35 regardless of CPU speed.
The local boot-iso-serial justfile recipe uses SCSI + KVM (-accel kvm -cpu host)
which works, but CI cannot use KVM.
Adding a new live variant to the matrix#
When a new image is ready to publish:
- Add the matrix entry to
build-iso-utah.yml - Commit the variant files in
<variant>/andlive/src/<variant>/in the same PR
test-luks-install.yml#
Matrix: installer_channel: [dev, stable] (fail-fast: false)
Timeout: 90 minutes
Triggers: PRs to main, weekly schedule, workflow_dispatch
Pipeline steps#
- Ensure
ci-screenshotsbranch exists - Free disk space
- Install deps (adds
qemu-system-x86 ovmf socat sshpass) - Configure podman storage (
configure_podman_storage.sh) - Build ISO with
debug=1and the matrixinstaller_channel - Boot live ISO in QEMU (daemonized) + wait for ready
- SSH into live env, write recipe, run
fishermanLUKS install - Patch BLS entries for dual console (
console=tty0 console=ttyS0) - Boot installed disk, send LUKS passphrase via QEMU monitor
- Verify boot success via serial log
- Save screenshots to
ci-screenshotsbranch + post PR comment
Configure podman storage script#
.github/scripts/configure_podman_storage.sh β intelligently selects the storage
driver based on the host filesystem:
- Clears existing podman storage to avoid driver mismatch errors
- On BTRFS: uses the native
btrfsdriver - On ext4/xfs: uses
overlay - Falls back to
vfsfor unknown filesystems
Screenshots#
LUKS test screenshots are saved to the ci-screenshots branch and linked in PR
comments. Key screenshots:
- Live boot (after
DAKOTA_LIVE_READY) - Plymouth LUKS passphrase prompt
- Final boot (after passphrase unlock)
test-plain-install.yml#
Matrix: installer_channel: [dev, stable] (fail-fast: false); in the merge queue, [stable] only
Timeout: 120 minutes
Triggers: push to main, merge queue, PRs to main (gate only), weekly schedule, workflow_dispatch
Merge-queue gate (#347)#
Nothing else that is required builds live/Containerfile, and Renovate updates
its base images. So the merge queue runs this workflow's stable leg, and the
Plain E2E gate job reports one stable check name that can be required:
- Pull request: the unit tests and the E2E are skipped (
test.ymlalready
runs the unit tests). The gate passes, so the PR can enter the queue. - Merge queue:
Plain E2E dakota (stable)runs. The gate passes only if it
ran and passed. A skipped, failed or cancelled leg fails it. - Push, schedule, dispatch: both legs, as before. The gate doesn't run, so a
cancelled push run gets no extra failed job forci-alert.ymlto count.
The dev leg stays out of the queue. It tests the upstream continuous-dev Flatpak,
not this repo, and a break there must not block every merge. A queued group now
waits about 35 minutes for the E2E, well within the queue's 120-minute timeout.
tests/test_plain_e2e_gate.py checks the triggers, the matrix expression and
the gate's script.
This workflow builds a debug Dakota ISO and runs the full plain-install QEMU path
(just ... plain-test-qemu dakota) to catch unencrypted installer regressions,
including the tight-memory ENOSPC class.
scheduled-gui-installer.yml#
Matrix: dakota Γ installer_channel: [dev, stable] (fail-fast: false)
Timeout: 120 minutes
Triggers: weekly Wednesday 04:00 UTC and workflow_dispatch only
This acceptance workflow builds a debug ISO for each installer channel and runs
just gui-e2e dakota. The recipe drives the desktop's already auto-launched
installer over the live user's AT-SPI bus, patches the installed boot entries
for serial verification, and boots the target disk. It always uploads serial
logs, screenshots, AT-SPI diagnostics, and guest installer logs. Its token has
only contents: read and packages: read permissions.
Adding a new workflow#
All workflow files go in .github/workflows/. Before adding:
- Run
actionlint - Check matrix
fail-fast: falsefor variant builds - Do not use
installer_channel=devin release builds. The scheduled GUI
acceptance workflow is the explicit exception because it detects installer
regressions before the stable bundle is published.
lint.yml β ShellCheck and pinned fetches#
Runs ShellCheck on every .sh file in the repository. Severity threshold is warning
(style/info is ignored). Uses ludeeus/action-shellcheck@2.0.0.
Any new shell script must pass ShellCheck before merge. For intentional suppression,
add an inline # shellcheck disable=SCxxxx comment with a justification.
The pinned-fetches job calls the shared
projectbluefin/actions/.github/workflows/reusable-pinned-fetches.yml gate, pinned to a
commit SHA (# v1) that Renovate keeps current, as in the other projectbluefin repos. It scans
the repository for fetches pinned to a moving target: download URLs on latest/branches,
curl | sh-style pipes, git clone with no commit SHA, container images by tag with no
@sha256 digest, and bootc switch/ostree rebase without a signature policy.
Intentional exceptions go in .github/pinned-fetches-allowlist.txt β one rule id or
substring per line, # comments allowed. Entries are bare substring matches, so keep them
as narrow as possible (the current localhost/ entry should be tightened before the
flip below).
The job runs in mode: warn (findings are reported, never blocking) pending the
#294 follow-up that flips it
to mode: fail once the findings are cleared or allowlisted.
The E2E fisherman clones (test-luks-install.yml, test-plain-install.yml) are pinned to
a commit on tuna-os/fisherman's dev branch. A Renovate git-refs custom manager in
renovate.json bumps that SHA; it matches checkout --detach <sha> # dev, so keep the
trailing # dev comment when editing the line.
test.yml β Python Unit Tests#
Runs pytest tests/ -v against Python 3.11.
β οΈ
pytest tests/passing does NOT mean the application works.
These are static-analysis and unit-level checks only. The real functional gates
aretest-luks-install.yml(LUKS E2E) andtest-plain-install.yml(plain install E2E).
Never claim a change is verified based onpytestalone.
Every tests/test_*.py file is collected automatically, so this page does not keep a
per-file list (a hand-maintained table here drifted out of date). To see what is covered,
browse tests/ β most modules open with a docstring saying what they guard β or run
pytest tests/ --collect-only -q.
Some tests skip themselves when optional tooling is missing. For example,
tests/test_build_iso_assembly.py needs xorriso/mtools; CI installs them so the tests
run, and the skip message names the apt packages to install locally.
Run locally with:
sudo apt-get install -y xorriso dosfstools mtools squashfs-tools
pip install pytest PyYAML jsonschema
pytest tests/ -v
Lessons#
Publish workflows must gate latest on the last real safety check (2026-06-16)#
If a workflow updates *-latest.iso, the publish step must come after the last
release-significant gate for that artifact β not before it. In practice:
build-iso.ymlmust wait for ENOSPC, full install, installed-boot verification,
and the final production ISO smoke boot before updatingdakota-live-latest.isobuild-iso-utah.ymlrequires the same media-preflight/install/installed-boot
sequence plus production QEMU smoke boot before updatingutah-live-latest.iso- Publish workflows define workflow-level
concurrencyso overlapping manual/scheduled
runs cannot race each other on thelatestpointers - Expensive E2E jobs (
plain-e2e,luks-e2e) depend onunit-testsso cheap failures
stop before QEMU burns runner time
Bound OCI media replaces runtime export (2026-11-06)#
Fisherman v0.5.3 makes media-input.json mandatory when live-iso-mode exists.
The older producer only embedded containers-storage and therefore failed before
installation. Both producers now embed the existing prepared archive as one OCI
layout; the native consumer verifies all SHA-256 blobs on the canonical read-only
squashfs before any disk operation. Do not remove the marker or weaken validation.
plain-enospc-gate no longer watches OCI export complete (there is no export).
It executes scripts/check-live-media.sh using shipped fisherman with a deliberately
nonexistent /never-a-device target. Only exit 1 with the exact disk-stat refusal
is accepted, proving strict media preflight completed without a tmpfs export;
EOF, missing receipt, writable media, corruption and every other failure fail.
This is not installation proof: full install and installed boot are separate gates.
Fixture-backed producer and hardware-target tests run in the unit suite. To test
the real pinned consumer without disks before a production build:
FISHERMAN_TEST_BINARY=/absolute/path/to/verified-v0.5.3/fisherman \
pytest tests/test_bound_media_consumer.py -v
Use the release asset and SHA-256 in installer-bundle.lock; bwrap with user
namespaces is required. Cases cover valid/missing/corrupt/writable media and
source/target mismatch. Passing this proves receipt preflight, not a full install.
Then run fresh Dakota and Utah install/installed-boot gates (including LUKS for
encryption) locally before dispatching publish workflows. Record actual ISO byte
sizes: compressed OCI blobs no longer deduplicate per-file against the live OS.
The VFS lessons below describe the historical implementation, not today's layout.
Double-embedded OCI store inflates ISO to 8 GB (2026-06)#
The live container (live/Containerfile) already bakes the OCI image into the squashfs
as VFS containers-storage via configure-live.sh and install-flatpaks.sh.
Building a separate store.squashfs.img with scripts/build-offline-store.sh and
passing --store store.squashfs.img to build-iso.sh embeds the same ~4 GB OCI
image twice in the final ISO β resulting in ~8 GB instead of ~5.3 GB.
Fix: Remove the "Build offline image store squashfs" CI step entirely.
Call build-iso.sh without --store. This is the correct architecture for the
unified VFS-embedded ISO.
Release compression for production ISOs (2026-06)#
scripts/build-live-squashfs.sh defaults to zstd level 3 (SUPERISO_COMPRESSION=fast).
CI sets SUPERISO_COMPRESSION=release (zstd-15, 1M blocks) in the squashfs build step β
this produces ~20% smaller ISOs at ~5Γ longer squashfs build time. Always use release
for ISOs published to R2. Use fast only for local testing.
installer_channel=dev regression: oci-cache/index.json not found (2026-05)#
After the continuous-dev release ~2026-05, fisherman's overlay storage path fails
with open /var/tmp/oci-cache/index.json: no such file or directory when composefs+btrfs
is the backend. Root cause: fisherman exports the OCI to scratch but bootc inside the
container cannot see it via the bind mount.
Fix: use installer_channel=stable. Keep build-iso.yml on stable until
tuna-os/fisherman#38 is resolved.
DAKOTA_LIVE_READY not seen when live-ready.service uses journal+console (2026-05)#
When StandardOutput=journal+console, the output goes to /dev/console (not /dev/ttyS0).
QEMU serial (-serial file:...) captures ttyS0 output only.
Fix: StandardOutput=tty + TTYPath=/dev/ttyS0 for direct serial writes.
CI falls back to SSH connectivity check if the marker is absent.
Offline install failed: VFS containers-storage missing from CI ISOs (2026-06)#
Symptom: fisherman: fatal: ... reference "containers-storage:ghcr.io/projectbluefin/dakota-nvidia:stable" does not resolve to an image ID
Root cause: The CI build called scripts/build-live-squashfs.sh without --oci-image, so
the live squashfs shipped with an empty /var/lib/containers/storage. The live recipe.json
has local_imgref=containers-storage:ghcr.io/projectbluefin/dakota-nvidia:stable, which
fisherman treats as authoritative. When the local store is empty, the install fails even if
the user has a working internet connection (fisherman does not fall back to docker://).
Why local builds were unaffected: just iso-sd-boot always did the squash+skopeo step and
baked the OCI into the squashfs. CI diverged from this path and the gap was undetected
because CI only validated boot, not install.
Fix: scripts/build-live-squashfs.sh now accepts --oci-image <ref>. When provided it:
- Squashes the payload to a single layer via
podman build --squash-all - Runs
skopeo copyinside the live container (for JSON tar-split compatibility) - Copies the populated VFS staging dir into the squashfs root with
cp -abefore mksquashfs
build-iso.yml now passes --oci-image ghcr.io/projectbluefin/dakota-nvidia:stable and
asserts the embedded store is non-empty before uploading the ISO to R2.
Invariant: The CI-built squashfs must contain a populated VFS store at
var/lib/containers/storage with the dakota-nvidia:stable image. The assertion step
catches any regression before upload.
See issue #78.
VFS store not captured by mksquashfs when using bind-mount into overlayfs (2026-06)#
Symptom: build-live-squashfs.sh --oci-image runs successfully, VFS store logs 9.1G, but
the squashfs is only ~4.2G (no VFS data) and the assertion fails.
Root cause: When SFS_ROOT is an overlayfs mount (the default on ext4/XFS CI runners),
the overlayfs filesystem has a different st_dev than a bind-mounted directory inside it.
mksquashfs respects filesystem boundaries (stops when st_dev changes) and silently skips
the bind-mounted VFS tree.
Fix: Copy the VFS staging dir into the squashfs root with cp -a instead of bind-mounting.
Writes to an overlayfs path go into the overlay upper layer; the resulting files inherit the
overlayfs st_dev and are included by mksquashfs.
Rule: Never use mount --bind to inject data into a directory that will be squash-packed with
mksquashfs when the mount point is overlayfs. Always copy.
VFS storage paths don't contain image names β assertion must check vfs-images/ (2026-06)#
Symptom: Assertion grep -c "ghcr.io" on unsquashfs -lc output returns 0 even when the
OCI store is correctly embedded.
Root cause: VFS containers-storage uses content-addressed hashes for all paths. Image
names like ghcr.io/projectbluefin/... are stored in JSON metadata inside the hash-named
directories, not in the directory paths themselves. unsquashfs -lc shows file paths only,
so grepping for ghcr.io always returns 0.
Correct assertion: Check for var/lib/containers/storage/vfs-images β this directory is
created by containers/storage for every imported image. If it has entries, the VFS store
was populated.
Note on mksquashfs deduplication: The VFS layer is a squashed copy of the same OS as the
live rootfs. mksquashfs deduplicates identical content blocks, so the squashfs size barely
increases despite embedding 9G of VFS data. Use inode/file counts to confirm inclusion,
not squashfs file size.
ENOSPC in skopeo OCI export β containers/storage tmpdir not redirected (2026-06)#
Symptom: Live ISO installs fail with:
reading blob sha256:...: write /var/tmp/container_images_XXXXXXXX: no space left on device
The installer correctly sets TMPDIR=/mnt/fisherman-target/.fisherman-scratch but the
blob staging file still lands at /var/tmp.
Root cause (3 layers):
configure-live.shwrites/etc/containers/storage.confwithdriver = "vfs"but no
tmpdirline.containers/storagedefaultsTMPDirto/var/tmp(hardcoded) when the config has no
tmpdirfield. Setting$TMPDIRin the subprocess env is not sufficient β containers/storage
reads the store config first and uses/var/tmpas the unconditional fallback./var/tmpon the live ISO is on the dracut overlayfs (~1.4 GiB writable layer) β too small
for multi-GiB OCI layer blobs.
Fix: skopeoExportOCI (fisherman) now reads the current effective storage.conf, injects
tmpdir = "<scratchDir>", writes the result to a temp file in the disk-backed scratch dir,
and passes it to skopeo via CONTAINERS_STORAGE_CONF. $TMPDIR is retained for belt-and-
suspenders coverage of containers/image's copy-side blob staging.
Why CI didn't catch it: The LUKS E2E test runs QEMU with 8 GiB RAM; the overlay tmpfs
is ~4 GiB β large enough for individual blobs in most runs. On 8 GiB user laptops with the
live environment loaded, free tmpfs headroom is much lower and ENOSPC triggers reliably.
Prevention: plain-test-qemu (new) runs with qemu-mem=4096 (4 GiB RAM), which gives
only ~2 GiB overlay tmpfs β reliably reproducing this class of bug. The test is gated
before R2 upload in build-iso.yml.
build-live-squashfs.sh WORK dir must be on large disk (2026-06)#
Symptom: Build live squashfs + boot tar step fails with:
write /usr/lib/locale/.../LC_COLLATE: no space left on device
mkdir /vfs-storage/vfs-layers/tmp: no space left on device
Root cause: build-live-squashfs.sh creates WORK at /var/tmp by default.
The squash-to-1-layer + VFS embedding writes ~12 GB of intermediates (payload.oci.tar
~6 GB + VFS staging ~6 GB). /var/tmp on GitHub ubuntu-24.04 runners sits on the
root filesystem which has ~14 GB free after jlumbroso/free-disk-space β not enough
if the image grows at all.
Fix: WORK now uses ${SUPERISO_TMPDIR:-/var/tmp}. In CI, build-iso.yml
sets SUPERISO_TMPDIR: /var/iso-build so all intermediates land on the 119 GB
disk-backed path. Locally the default /var/tmp still applies.
Prevention: If squashfs build ENOSPC recurs in CI, verify SUPERISO_TMPDIR
is set in the Build live squashfs + boot tar step env.
E2E plain install test requires sshd β production ISO has it disabled (2026-06)#
Symptom: Plain install E2E step fails with either:
kex_exchange_identification: read: Connection reset by peer(QEMU user-net accepts TCP, no listener inside guest)ERROR: serial marker seen but SSH not ready after 90 s(sshd never starts)
Root cause: sshd is only enabled in the live ISO when the container is built with
--build-arg DEBUG=1. The production build uses DEBUG=0, so no sshd. The E2E test
uses SSH to invoke fisherman; without sshd the test cannot proceed.
Fix: After building the production ISO, a CI step patches the production squashfs:
unsquashfsthe production rootfs (includes the embedded VFS store)- Add sshd.service symlink to
multi-user.target.wants - Append
PasswordAuthentication yes/PermitEmptyPasswords yesto sshd_config - Set
liveuserpassword tolivevia/etc/shadowpatch mksquashfsback with zstd-1 (fast, debug-only)- Assemble
output/dakota-debug-live.iso(uses same boot tar as production)
plain-boot-qemu-live in the justfile prefers output/{{target}}-debug-live.iso
when present, so CI runs against the debug ISO while R2 gets the production ISO.
Why the VFS store must stay: ghcr.io/projectbluefin/dakota-nvidia is private;
the live env inside QEMU has no GHCR credentials, so fisherman cannot pull from
network. The VFS store (embedded in the squashfs) is the only install source.
flatpak-spawn --host does not forward sandbox env to host process (2026-06)#
Symptom: fisherman sets TMPDIR=/mnt/fisherman-target/.fisherman-scratch and prints
# TMPDIR=<scratch> before running skopeo, but the blob staging file is still created
at /var/tmp/container_images_XXXXXXXX causing ENOSPC.
Root cause: The bootc-installer runs inside a Flatpak. When runner.go calls
flatpak-spawn --host skopeo copy ..., flatpak-spawn --host spawns the command in
the HOST mount namespace but does not automatically forward the Flatpak sandbox
environment to the spawned host process. skopeo inherits the host's default env
(no TMPDIR set) and uses /var/tmp for blob staging.
Setting cmd.Env for the flatpak-spawn subprocess propagates env vars to
flatpak-spawn itself, but not to the command it spawns on the host.
Fix (fisherman): runner.HostArgsWithEnv injects critical env vars via
--env=KEY=VALUE flags in the flatpak-spawn args:
flatpak-spawn --host --env=TMPDIR=/scratch --env=CONTAINERS_STORAGE_CONF=... skopeo copy ...
Released in bootc-installer v2.7.1.
How to identify: Look for # TMPDIR=<path> in fisherman output followed by
write /var/tmp/container_images_...: no space left on device. The TMPDIR debug
print confirms fisherman set the var correctly, but skopeo ignoring it confirms the
flatpak-spawn env forwarding gap.
E2E test split into 4 named steps with individual timeouts (2026-06)#
Why: The original single plain-test-qemu step had one monolithic timeout (90 min).
When it expired you had no idea which of the four stages (boot-live, install,
boot-installed, verify) was the bottleneck.
New structure:
| Step | Timeout | RAM | Purpose |
|---|---|---|---|
E2E 1/4 β Boot live ISO (4 GiB) | 10 min | 4 GiB | Live env boots, sshd responds |
E2E 2/4 β ENOSPC gate: OCI export only (4 GiB) | 10 min | 4 GiB | skopeo copies blob without ENOSPC (tight ~2 GiB overlay tmpfs) |
E2E 3/4 β Full install composefs (8 GiB) | 60 min | 8 GiB | btrfs+composefs install completes |
E2E 4/4 β Boot installed + verify Graphical target (8 GiB) | 20 min | 8 GiB | Installed system reaches systemd Graphical target |
Total worst-case ceiling: 100 min (vs. 90 min monolithic), with precise attribution.
Gates 1+2 use 4 GiB to keep the overlay tmpfs tight (~2 GiB) for ENOSPC testing.
Gate 3 switches to 8 GiB for realistic btrfs+composefs install performance.
Build trigger restored to daily + on-demand (2026-07)#
The original build-iso.yml ran on every push to live/**/scripts/**.
Push triggers were removed, and the build schedule was restored to run:
schedule: cron '0 3 * * *'β daily at 03:00 UTCworkflow_dispatchβ on demand for releases, hotfixes, or manual triggers
This keeps fresh Dakota image updates flowing daily while preserving on-demand
manual runs. For mid-cycle named releases (e.g. a new alpha), use the manual
promotion flow documented in docs/r2-promotion.md.
E2E on push to main: cancel superseded runs, skip doc-only pushes (2026-09-22)#
When multiple PRs are merged to main in close succession, each merge triggers
the heavy E2E workflows (test-luks-install.yml and test-plain-install.yml),
which run for 45β104 minutes per build.
Fix:
- Added workflow-level
concurrencywithgroup: e2e-${{ github.workflow }}-${{ github.event_name }}-${{ github.ref }}andcancel-in-progress: ${{ github.event_name == 'push' }}so subsequent merges cancel superseded builder runs without interrupting scheduled or manual dispatches. - Added
paths-ignore: ['docs/**', '**/*.md']to avoid building E2E images for documentation-only changes. - Invariant guarded by
test_e2e_workflows_define_push_concurrencyintests/test_live_build_invariants.py.
btrfs composefs install β root cause chain and fixes (2026-06)#
Context: dakota-nvidia uses GNOME OS / freedesktop-sdk. Its initramfs
(built by the freedesktop-sdk pipeline) includes btrfs.ko, erofs.ko, exfat.ko,
and squashfs.ko β but not xfs.ko. Any attempt to install with filesystem: xfs
succeeds but produces a system that drops to emergency mode on first boot because the
initramfs cannot load the XFS driver to mount the root partition.
Fix: Set filesystem: btrfs everywhere β live/src/etc/bootc-installer/images.json,
all justfile install recipes (luks, plain, enospc-gate). btrfs.ko IS in the initramfs.
Bug 2 β images.json overrides the fisherman CLI recipe
fisherman <recipe.json> reads the recipe for disk and image, but looks up
filesystem from /etc/bootc-installer/images.json on the host, not from the recipe
JSON. Changing "filesystem" in the CLI recipe has no effect. The canonical place to
set the default filesystem is images.json baked into the live container.
Bug 3 β bootc UUID auto-detect fails inside nested containers
bootc install to-filesystem uses findmnt --mountpoint /target --output UUID to
discover the root filesystem UUID. Inside fisherman's podman run --privileged inner
container, findmnt cannot read the udev database (/run/udev is not mounted), so
the UUID column is always empty and the install fails with:
error: Installing to filesystem: No filesystem uuid found in target root
Fix: Inject root-mount-spec = 'LABEL=root' into the bootc install config before
squashing the OCI payload. fisherman always formats root partitions with -L root, so
LABEL=root is a stable, reliable mount spec that bypasses UUID detection entirely.
How injection works (payload_build in scripts/payload-oci.sh):
FROM ${PAYLOAD_IMAGE}
COPY bootc-root-mount.toml /usr/lib/bootc/install/00-defaults.toml
built with podman build --pull=never (plus --squash-all for composefs payloads).
Both scripts/iso-sd-boot.sh and scripts/build-live-squashfs.sh use it.
Bug 4 β btrfsSubvolumes: true + root-mount-spec config = missing rootflags=subvol=@
When btrfsSubvolumes: true is set in the fisherman recipe, fisherman creates @,
@home, @snapshots subvolumes and remounts with subvol=@,compress=zstd:1. bootc
installs the OS into the @ subvolume. When booting, the initramfs needs
rootflags=subvol=@ in the kernel cmdline to mount the right subvolume.
bootc DOES add rootflags=subvol=@ automatically β but ONLY when it discovers the UUID
by itself (the else branch). When root-mount-spec is provided via config file, the
config branch returns kargs: Vec::new() (no rootflags). The kargs config field
in [install] section was tested but is not applied in the to-filesystem code path.
Fix: Remove btrfsSubvolumes entirely. Plain btrfs (root at subvolid=5) works
correctly with composefs. The OS is installed at the btrfs root, root=LABEL=root
mounts it, no subvolume flags needed.
Bug 5 β fisherman hostname-fix condition missed new error format
fisherman-install.sh detects hostname-write failures and patches the deployed /etc/hostname
manually. The original condition required BOTH "writing hostname" AND
"ostree admin --print-current-dir" in the fisherman log. New fisherman emits:
fisherman: fatal: writing hostname: finding composefs deploy etc: reading composefs deploy base
/mnt/fisherman-target/state/deploy: open /mnt/fisherman-target/state/deploy: no such file or directory
No ostree admin string β condition failed β install errored despite the OS being
fully installed.
Fix: fisherman-install.sh now matches "writing hostname" AND any of:
"ostree admin --print-current-dir"(old fisherman)"composefs deploy"or"state/deploy"or"no such file or directory"(new fisherman)
Lesson: nvidia-drm.modeset=1 required in all boot entries (2026-06)#
Symptom: Black screen on boot on all NVIDIA hardware. Live ISO appears to hang
after BIOS handoff with no display output.
Root cause: nvidia-drm.modeset=1 was never set in any of the boot entry
locations in live/src/build-iso.sh. Without KMS
enabled, the NVIDIA proprietary driver cannot take over the framebuffer from the
BIOS, leaving the screen dark even though the system is running fine.
Fix: Added nvidia-drm.modeset=1 to every boot cmdline entry
(BLS entries and loopback GRUB entries; at the time there were four, two of them
in the since-removed multi-arch path).
Regression test: TestBootCmdline::test_live_build_iso_has_nvidia_drm_modeset
and test_dakota_build_iso_has_nvidia_drm_modeset in tests/test_live_build_invariants.py
assert every boot entry line with root=live:/dev/sr0 contains this parameter.
Lesson: root=live:/dev/sr0 breaks USB flash drive boots (2026-06)#
Symptom: Black screen on boot on all non-NVIDIA real hardware. ISO boots
fine in QEMU but silently hangs on physical machines.
Root cause: ca7f1e4 switched from root=live:CDLABEL=DAKOTA_LIVE to
root=live:/dev/sr0 to fix CDLABEL detection in the Debian-built initramfs
on GnomeOS kernels. /dev/sr0 is an optical-drive device node β it only
exists when the ISO is presented via a SCSI CD emulation (QEMU virtio-scsi,
physical optical drive). USB flash drive boots present the ISO as /dev/sdX.
Dracut waits indefinitely for /dev/sr0, which never appears. With quiet
suppressing all output, the result is a permanent, silent black screen.
This affected every user booting from USB β the majority of real hardware
users β regardless of GPU.
Fix: Use root=live:LABEL=DAKOTA_LIVE (blkid-based label scan). LABEL=
works on any block device (USB /dev/sdX, optical /dev/sr0, QEMU
virtio-scsi) without requiring the cdrom_id udev helper that broke with
the Debian-built initramfs.
Why not CDLABEL=: CDLABEL= uses the cdrom_id udev helper which is
absent or broken in the Debian-built initramfs on GnomeOS native kernels.
LABEL= uses blkid which is always present.
Regression test: TestBootCmdline::test_{live,dakota}_build_iso_uses_label_not_cdlabel_or_sr0
asserts /dev/sr0 and CDLABEL= are banned from all boot entries.
E2E verified: just debug=1 plain-e2e dakota β full install + installed
system boot passed with LABEL=DAKOTA_LIVE.
Lesson: build-live-squashfs.sh missing root-mount-spec injection β installed system emergency mode (2026-06)#
Symptom: Installed system boots into emergency mode in CI with
initrd-parse-etc.service: Failed with result 'start-limit-hit'.
Local builds (just iso-sd-boot) succeed; CI builds (build-iso.yml) fail.
Root cause: justfile's iso-sd-boot recipe injects
root-mount-spec = 'LABEL=root' into the squashed payload OCI image via
buildah copy/run before embedding it into VFS containers-storage.
scripts/build-live-squashfs.sh (used by build-iso.yml) squashed the
OCI payload but skipped this injection. Without it, bootc install
falls back to UUID auto-detect, which calls findmnt inside a nested
container where the udev database is inaccessible β UUID is always empty.
The installed BLS entry gets root=UUID= with an empty UUID, so the
initrd can't find the root partition β bootc-root-setup.service fails
β initrd-parse-etc.service crash-loops β emergency mode.
Fix: Added the same buildah copy/run injection to
scripts/build-live-squashfs.sh immediately after buildah from and
before buildah commit --squash.
Rule: Any time justfile's iso-sd-boot injects config into the
squashed OCI payload, scripts/build-live-squashfs.sh must apply the
same injection. They must stay in sync.
Non-composefs variants use OCI layout, not VFS squash (2026-06)#
Dakota (composefs) embeds the payload as VFS containers-storage inside the squashfs.
This path does not work for Fedora/Silverblue images (bluefin, bluefin-lts-hwe)
because squashing destroys the ostree commit structure bootc requires, producing:
Expected commit object, not File
Fix: build-live-squashfs.sh resolves composeFsBackend per variant β in
--target mode from scripts/variant-config.sh (variant_composefs, the single
host-side authority), in positional mode from the recipe.json baked into the
live container β and:
- composefs = true (dakota): squash β VFS containers-storage (unchanged)
- composefs = false (bluefin/*): store OCI layout directly at
/var/lib/containers/oci-store
configure-live.sh writes the recipe accordingly:
- composefs:
"image": "containers-storage:<ref>" - non-composefs:
"image": "oci:/var/lib/containers/oci-store"
fisherman needs the bootcDirectOCI code path (originally landed via
projectbluefin/fisherman devβprod merge, bootc-installer PR #192) to handle oci:
source refs.
fisherman submodule pin can lag behind dev (2026-09)#
projectbluefin/bootc-installer and projectbluefin/fisherman are retired.
tuna-os/bootc-installer's fisherman submodule now points at tuna-os/fisherman
(bootc-installer#73). tuna-os/fisherman has no main/prod split β dev is both
its default and active line β so there's no branch-sync step to run. The remaining gap
is simpler: the submodule is a pinned SHA, so a fix merged to tuna-os/fisherman's
dev still needs the submodule pin in tuna-os/bootc-installer bumped and a new
Flatpak release cut before it reaches a dakota ISO.
ENOSPC report must measure each RAM-backed mount on its own (2026-11-18)#
What failed: A LUKS E2E run on main (dev installer leg) failed with
write /run/libpod/alive: no space left on device; df showed /run and the
live root (LiveOS_rootfs) both at 100%. The "largest entries" report from
scripts/fisherman-install.sh listed only /, /usr, /var/lib⦠and nothing
under /run, so the run said which mount filled but not what filled it.
Why: the report ran one du -x / /run /tmp β¦. du counts each inode once
across all its arguments, and / is the overlay over the read-only squashfs,
whose files report the inodes of the writable upper layer stored in /run. The
/ pass consumed those inodes first, so /run looked empty, and the squashfs
gigabytes pushed everything else out of the top 25.
Fix: never measure /; run one du -xah --max-depth=4 per RAM-backed mount
(/run, /tmp, /var/tmp, /dev/shm). The overlay upper layer lives under
/run, so files written to the live root show up there.
Workflow matrix must be kept in sync with variant config files (2026-06)#
build-iso-utah.yml has a strategy.matrix that hardcodes payload_image,
live_target, registry, and tag per variant. These same values live in the
variant config files (bluefin-lts-hwe/payload_ref, bluefin-lts-hwe/registry, etc.).
They can and do drift independently.
Rule: Every change to a variant config file (payload_ref, registry, live_target)
must be accompanied by the matching matrix update in build-iso-utah.yml in the same commit.
Long-term fix (Design Gate β requires human approval): Remove the duplication by having
the matrix read config values from the variant files at build time instead of hardcoding them.
Until that refactor lands, keep the matrix and config files in sync manually on every change.
Organization Standards for GitHub Actions (2026-06-25)#
Policy: The standard is the codebase itself. Use what is in production already.
Whenever a CI task requires a GitHub Action, your first step is to grep the organization's existing workflows to see what is already used in production. For example, Project Bluefin universally uses ublue-os/remove-unwanted-software to free disk space. Use the existing production standard to guarantee cross-repo consistency and prevent redundant third-party dependencies. You do not need to ask or look for a separate rules document if the codebase already shows a clear consensus.