E2E CI ā Plain Install Test#
Skill for the plain composefs install E2E gate in build-iso.yml.
When to Use#
- Debugging a live ISO boot failure (dracut emergency shell, no rootfs layout)
- Diagnosing ENOSPC during skopeo copy or fisherman install
- Investigating sshd connectivity issues in QEMU E2E
- Adding or modifying a CI E2E step
- Interpreting a CI failure in
build-iso.ymlortest-plain-install.yml
When NOT to Use#
- For R2 promotion ā use
docs/r2-promotion.md - For LUKS-specific failures ā use
docs/luks-testing.md - For build-time container failures ā use
docs/build.md
Core Process#
- Build the ISO with
debug=1; never enable test-only access in production. - Boot a fresh QEMU guest and wait for both the serial readiness marker and SSH.
- Run the test path against that fresh guest, then boot the installed disk.
- Preserve serial logs and guest diagnostics on failure.
Common Rationalizations#
- āThe production ISO booted, so the E2E is covered.ā ā It lacks debug-only
SSH and accessibility setup; validate the debug artifact separately. - āThe driver imported on the build host.ā ā Import it in the booted guest,
whose GNOME OS runtime supplies the actual binding and session bus.
Architecture#
The E2E gate splits into four named CI steps, each with its own timeout:
| Step | Timeout | RAM | Purpose |
|---|---|---|---|
E2E 1/4 ā Boot live ISO | 10 min | 4 GiB | Live env boots, sshd responds |
E2E 2/4 ā ENOSPC gate: OCI export only | 10 min | 4 GiB | skopeo copies blob without ENOSPC |
E2E 3/4 ā Full install composefs | 60 min | 8 GiB | btrfs+composefs install completes |
E2E 4/4 ā Boot installed + verify Graphical | 15 min | 8 GiB | Installed system reaches Graphical target |
Why split RAM between steps:
The ENOSPC bug triggers when overlay tmpfs is ~2 GiB (4 GiB RAM).
The bootc install to-filesystem --composefs-backend step extracts 6 GB via
btrfs+overlay through QEMU ā inherently slow at 4 GiB, ~3Ć faster at 8 GiB.
The debug ISO#
The production ISO (uploaded to R2) has sshd disabled ā it is built
with DEBUG=0. The E2E test uses a debug ISO that is patched at CI time:
unsquashfs production.rootfs.sfs ā debug-rootfs/
# add sshd.service symlink, password auth, liveuser:live password
mksquashfs debug-rootfs ā debug.rootfs.sfs (zstd-1, fast)
build-iso.sh ā output/dakota-debug-live.iso
plain-boot-qemu-live prefers output/{{target}}-debug-live.iso when
present, so CI uses the debug ISO and R2 gets the production ISO.
Never enable sshd in the production squashfs. The debug ISO is test-only.
QEMU disk: raw sparse + cache=unsafe#
Use raw sparse disk instead of qcow2 for sequential write workloads:
truncate -s 64G /var/tmp/dakota-plain-install.img # not qemu-img create -f qcow2
QEMU drive args:
-drive if=none,id=disk,file={{plain-qemu-disk}},format=raw,cache=unsafe
cache=unsafe skips fsync entirely. Fine for tests ā not for production.
Raw sparse + cache=unsafe delivers 200ā500 MB/s vs ~10ā50 MB/s for qcow2.
ENOSPC root cause and fix#
Symptom: write /var/tmp/container_images_XXXXXXXX: no space left on device
Root cause chain:
containers/imageTypeBigFiles path callsstore.TmpDir()firststore.TmpDir()in containers/storage returns/var/tmp(hardcoded default)/var/tmpis on the dracut overlayfs (~1.4 GiB at 4 GiB RAM)- A single squashed 5ā6 GiB OCI layer blob overflows it
Why TMPDIR env var doesn't help:
TypeBigFiles checks store.TmpDir() BEFORE os.Getenv("TMPDIR"). If the
store returns a non-empty string, TMPDIR is ignored entirely.
Why CONTAINERS_STORAGE_CONF with tmpdir = doesn't help:
Older containers/storage versions don't have tmpdir as a recognized TOML
field ā they silently reject it with Failed to decode the keys ["storage.tmpdir"].
The fix that works (fisherman v2.7.3):
Bind-mount a disk-backed scratch subdir over /var/tmp before the skopeo copy,
then umount it in a deferred call:
varTmpOverride := filepath.Join(tmpdir, "var-tmp-override")
os.MkdirAll(varTmpOverride, 0o1777)
exec.Command("mount", "--bind", varTmpOverride, "/var/tmp").Run()
defer exec.Command("umount", "/var/tmp").Run()
// now skopeo copy runs ā /var/tmp is disk-backed
This works across all containers/storage versions and is independent of env
vars, config keys, or source transport.
flatpak-spawn does not forward sandbox env to host#
Symptom: fisherman sets TMPDIR=/scratch and prints # TMPDIR=... before
running skopeo, but blobs still land in /var/tmp.
Root cause: When running inside a Flatpak, runner.HostArgs wraps the
command as flatpak-spawn --host skopeo .... flatpak-spawn --host spawns the
command in the host mount namespace but does NOT forward the Flatpak sandbox's
environment. cmd.Env applies to flatpak-spawn itself, not to skopeo.
Fix: Use runner.HostArgsWithEnv(name, args, envVars) which injects
--env=KEY=VALUE flags into the flatpak-spawn args when inside a Flatpak.
For non-Flatpak invocations the result is identical to HostArgs.
How to identify: # TMPDIR=<path> in fisherman output followed by
write /var/tmp/container_images_...: no space left on device. The TMPDIR
debug print confirms fisherman set the var; skopeo ignoring it confirms the
flatpak-spawn env forwarding gap.
sshd is only enabled in debug ISOs#
Symptom: kex_exchange_identification: read: Connection reset by peer OR
ERROR: serial marker seen but SSH not ready after 90 s
Root cause: configure-live.sh only enables sshd when DEBUG=1 is passed
as a build arg. Production ISOs have no sshd. QEMU user-mode networking accepts
the TCP connection on port 2223 and then resets it because port 22 is not open
inside the guest.
Fix: Build a debug ISO (see above) for E2E testing. Never change the
production ISO to enable sshd.
Serial marker fires before sshd is stable#
Symptom: Serial marker seen ā polling SSH... then sshd resets
connections for 10ā20 s before accepting.
Cause: live-ready.service fires After=display-manager.service,
which is before sshd finishes host-key generation. First connection after
serial marker gets kex_exchange_identification: read: Connection reset by peer.
Fix in plain-boot-qemu-live: After seeing the serial marker, poll SSH
in a 3-second retry loop (up to 90 s) before breaking out of the wait loop.
This ensures sshd is stable before plain-install-qemu SSHes in.
SUPERISO_TMPDIR ā squashfs build must use large disk#
Symptom: Build live squashfs + boot tar fails with ENOSPC.
Root cause: build-live-squashfs.sh creates its WORK dir at /var/tmp
by default. The squash-to-1-layer + VFS embedding writes ~12 GB of
intermediates; /var/tmp on ubuntu-24.04 CI runners has ~14 GB total.
If the image grows, this overflows.
Fix: Set SUPERISO_TMPDIR=/var/iso-build (119 GB) in the workflow env:
- name: Build live squashfs + boot tar
env:
SUPERISO_COMPRESSION: release
SUPERISO_TMPDIR: /var/iso-build
fisherman hostname failure ā fixed in v2.7.4#
Symptom (pre-v2.7.4):
fisherman: fatal: writing hostname: finding deployment dir: ostree admin --print-current-dir: exit status 1
Root cause: ostree admin --print-current-dir reads booted-deployment
state from the running kernel ā it always exits 1 against a freshly-installed
target that has never been booted. This hit the ostree branch of WriteHostname
in step 7 of the fisherman pipeline, crashing every real install.
Why CI didn't catch it for weeks:
- Unit tests mocked
DeploymentDirFnsoDefaultDeploymentDirwas never called - The e2e wrapper
scripts/fisherman-install.shsilently caught the crash,
re-mounted the disk, and patched/etc/hostnamedirectly - CI appeared green; the real installer on the ISO was broken
Fix (fisherman v0.2.1 / bootc-installer v2.7.4): DefaultDeploymentDir
now falls back to filepath.Glob(sysroot/ostree/deploy/*/deploy/*) when
--print-current-dir fails. Three regression tests lock this down.
If you see this error on v2.7.4+: It is a different failure ā investigate
whether the target disk structure is correct after bootc install to-filesystem.
scripts/fisherman-install.sh status: Still present as a safety net but
no longer load-bearing. Do not add new workaround logic to it ā fix the root
cause in fisherman instead.
XFS path not tested ā broken ISO shipped to R2 (2026-06-16)#
What failed: mkfs.xfs was copied into the live container without its shared library
dependencies (libinih.so.1, liburcu.so.8). The binary existed in the image but
failed at runtime on every XFS install with:
mkfs.xfs: error while loading shared libraries: libinih.so.1
Also: dakota-nvidia:stable initramfs has btrfs.ko but not xfs.ko. An XFS
install would complete but the installed system drops to emergency mode on first boot
(sysroot.mount fails ā cannot load XFS driver).
Current state: All variants (dakota, bluefin, bluefin-lts-hwe) default to
filesystem: btrfs in images.json and all E2E recipes. XFS is available only as a
user-selectable option in the installer UI, and only for non-composefs variants (fisherman
rejects composefs on XFS). plain-install-qemu uses btrfs.
Why CI didn't catch it initially:
plain-install-qemuhardcoded"filesystem": "btrfs"āmkfs.xfswas never called in CI- The unit test (
TestXfsprogs) only checked that the string"mkfs.xfs"appeared in the
Containerfile ā not that its deps were also present - No smoke test verified the binary executed inside the container before squashfs assembly
The fix (commit 3606cfc):
- Copy shared lib deps
libinih.so.1andliburcu.so.8alongside the binary in the Containerfile - Do not copy
libblkid.so.1orlibuuid.so.1from Debian ā the base image ships newer
versions; overwriting with Debian's breakssfdisk(seedocs/build.mdā libblkid lesson) TestXfsprogsnow asserts all required libs are present in the COPY blockbuild-iso.ymlrunsmkfs.xfs -Vinside the live container after build
Rule: For every binary copied into a container, assert its shared library deps are also
present. Use ldd <binary> on the source stage to enumerate them.
Rule: The E2E filesystem must match the installer default. images.json sets btrfs
as default ā E2E uses btrfs. If this ever changes, update the E2E recipe too.
QEMU resources ā use 8 GiB RAM / 8 vCPUs for local testing#
Symptom: Local E2E installs take 30ā60 min or time out.
Root cause: The composefs install (skopeo copy + bootc install to-filesystem --composefs-backend) is CPU and I/O bound. At 4 GiB / 4 vCPUs the flatpak copy phase alone takes 20+ minutes.
Fix: The justfile defaults are now qemu-mem=8192 and qemu-smp=8. All just plain-test-qemu / just luks-test-qemu / just e2e recipes pick these up automatically. Override as needed:
just qemu-mem=16384 qemu-smp=16 plain-test-qemu dakota
Do not hardcode -m 4096 -smp 4 in any new QEMU command ā use {{qemu-mem}} / {{qemu-smp}} justfile variables.
Live boot drops to dracut emergency shell ā "no proper rootfs layout"#
Symptom: QEMU boots ISO, dracut mounts squashfs, then immediately drops to dracut:/# with:
dracut Warning: /sysroot has no proper rootfs layout, ignoring and removing offending mount hook
Root cause: dracut-lib.sh:usable_root() accepts a root if either:
$1/lib*/ld-*.soglob matches (finds glibc dynamic linker), OR- All three of
$1/proc,$1/sys,$1/devexist
GNOME OS (glibc ā„2.38) ships ld-linux-x86-64.so.2 ā this does NOT match the ld-*.so glob. So path 2 is the only working path. The squashfs must have empty sys/ and dev/ at the root (alongside proc/).
mksquashfs 4.7+ removes bare-excluded directories. Using -e sys removes the sys/ dir entirely. The fix ā applied in justfile and scripts/build-live-squashfs.sh:
mkdir -p "${SFS_ROOT}/proc" "${SFS_ROOT}/sys" "${SFS_ROOT}/dev" "${SFS_ROOT}/run" "${SFS_ROOT}/tmp"
mksquashfs ... -wildcards -e "proc/*" -e "sys/*" -e "dev/*" -e "run/*" -e "tmp/*"
Never bare-exclude proc, sys, dev, run or tmp. Keep empty directories and exclude only their contents. Without /run, Debian dracut 99base/init.sh 353ā356 retains initramfs state under /dev/.initramfs, so /run/initramfs/live is lost after switch-root. A real squashfs regression must verify directory nodes and absent runtime contents; a staging-copy stub misses this failure.
systemd ordering cycle deadlock (rechunker-group-fix) (2026-06-24)#
Symptom:
Installed system boots, loops on waiting for device units (dev-ttyS0.device, dev-zram0.device, and dev-disk-by-uuid-...device), and drops to emergency mode after 90 seconds.
Root cause:
A circular systemd dependency (deadlock) exists on Universal Blue (LTS/stable) base images due to rechunker-group-fix.service running Before=systemd-sysusers.service without specifying DefaultDependencies=no. The full loop is:
rechunker-group-fix -> local-fs.target -> boot.mount -> device unit -> udevd -> sysusers -> rechunker-group-fix.
Depending on systemd's cycle breaking heuristics (e.g., whether sysusers or tmpfiles-setup-dev is deleted), this will either boot successfully (by luck) or deadlock (dropping to emergency shell).
The fix:
Write a systemd drop-in override during post-install (scripts/fisherman-install.sh) to set DefaultDependencies=no on rechunker-group-fix.service. This removes the default After=local-fs.target dependency, breaking the circular dependency loop.
Post-boot assertions need SSH into the installed system, not just the live env (2026-06-25, dakota#651)#
Symptom:
efibootmgr -v written by fisherman lives in host UEFI NVRAM and /proc/cmdline is a kernel-runtime artifact ā neither can be checked from a BLS entry file or from the live environment. The live-env-only SSH (debug-ssh-banner, configure-live.sh) doesn't help here because it only covers the live ISO, not the freshly-installed target disk before its first boot.
The fix:
scripts/fisherman-install.sh --enable-debug-ssh mounts the freshly-installed deployment (same mount/unlock logic already used for the hostname-write workaround) and patches root:root + a sshd.service system-preset + PermitRootLogin/PasswordAuthentication directly onto disk ā mirroring configure-live.sh's DEBUG=1 convention but applied to the installed system instead of the live squashfs. luks-boot-qemu-installed/plain-boot-qemu-installed forward a second SSH port (*-qemu-ssh-port-installed) to this now-enabled sshd. scripts/verify-post-install.sh then SSHes in as root once the installed system reaches its boot target and asserts efibootmgr -v, flatpak list --system --app, and (LUKS only) /proc/cmdline. Test-only ā never shipped in a production image.
Non-blocking by default: these assertions are observability-only until fisherman #1/#2 and common#385 land. On assertion failure scripts/verify-post-install.sh prints ā ļø ... NON-BLOCKING plus a ::warning:: annotation and exits 0; only STRICT_POST_BOOT=1 makes it exit 1 and fail the recipes, and CI does not set it. SSH unreachable is also non-blocking by default (::warning::, exit 0) until the debug-SSH path has a green E2E run on main; under STRICT_POST_BOOT=1 it exits 2. The LUKS installed VM reuses the live VM's OVMF vars file so installer-written Boot#### entries survive the reboot. The plain installed VM boots from the pristine OVMF vars template by default (a carried-over NVRAM had no valid BootOrder entry for the disk and dropped OVMF to the EFI shell); set PLAIN_CARRY_LIVE_NVRAM=1 to opt back into the carry-over when debugging installer-written entries.
Lesson: when a post-boot invariant can only be observed at runtime (firmware NVRAM state, live kernel cmdline, running daemon output), don't try to approximate it from static files during install ā enable a throwaway debug path into the actual booted target instead.
Debug SSH rejects the installed root password (2026-11-13, #317)#
The installed dev guest can reject the debug root password even when the
post-install shadow patch reports success. Debug installs now record a
fingerprint of the patched root shadow entry and compare it after boot; look
for E2E root shadow entry unchanged since install or changed since install
in the installed guest's serial log. No password hash is printed by this
diagnostic. An unchanged entry points investigation toward password verification;
a changed entry calls for tracing boot-time shadow updates. This diagnostic
does not itself fix the login failure.
composefs-native /etc lives under state/deploy/, not ostree/ (2026-10-27, #283)#
Symptom: plain E2E logs WARNING: deployment etc/ not found under ā¦/ostree ā post-install patches not applied, then ā Post-boot assertions SKIPPED: could not reach installed system over SSH. The hostname patch and the debug-SSH override were never written, so the post-boot assertions above never ran.
Root cause: fisherman-install.sh only searched ostree/bootc/deploy and ostree/deploy. A composefs-native install (bootc install --composefs-backend) keeps the writable /etc at state/deploy/<composefs-hash>/etc on the root filesystem ā the tree bind-mounted at /etc on boot, and the path fisherman's own DefaultComposeFsDeployEtcDir writes the hostname to. These installs also create an ostree/ directory, so "ostree/ exists" says nothing about the layout.
Fix: search state/deploy/*/etc first, then the two ostree/ layouts. When nothing matches, the script prints every etc/ within depth 4 plus the root listing, so the next layout change shows up in the CI log.
Note: state/deploy/<hash>/ holds only etc (and state), not usr ā the ssh.service vs sshd.service probe there falls back to sshd.service, which is what dakota ships.
Boot check missed login: in a large serial log ā SIGPIPE under pipefail (2026-11-13, #313)#
Symptom: plain E2E reports ā Timeout: installed system did not reach graphical target in 5 minutes while the serial log tail shows <hostname> login:. Small logs pass; large logs (lots of boot noise, e.g. the dev channel) fail.
Root cause: plain-verify-qemu runs under set -euo pipefail and tested the log with echo "$LOG" | grep -q PATTERN. grep -q exits at the first match; echo is still writing, dies of SIGPIPE, and the pipeline returns 141, which if treats as "no match".
Fix: both serial-log checks use a here-string, grep -q PATTERN <<<"$LOG", so there is no pipeline to fail. tests/test_plain_verify_boot_check.py runs the shipped conditions against a 2 MB+ log and fails on any echo "$VAR" | grep -q in the justfile.
Rule: in a pipefail script never pipe a variable or a command with unbounded output into grep -q (or head); grep a file or a here-string instead.
Red Flags#
- Claiming "tests pass" without specifying which gate (
plain-e2e,luks-test-qemu, or unit tests) - Testing with a QEMU instance that predates the current build
- Using
-smp 4 -m 4096in a QEMU command (too slow; use{{qemu-smp}}/{{qemu-mem}}justfile vars) - Bare-excluding runtime directories in mksquashfs: preserve
proc sys dev run tmpwith-wildcards -e "dir/*" - Declaring an install verified without booting the installed disk
- Using
installer_channel=devin production builds; the scheduled GUI
acceptance matrix deliberately tests both dev and stable - SSH connecting to a production ISO that has sshd disabled (build with
debug=1for testing) - Claiming post-boot assertions (UEFI/Flatpak/LUKS cmdline, dakota#651) pass without SSHing into the installed, booted disk ā a BLS entry or live-env check is not equivalent
- Treating a green
plain-test-qemu/luks-test-qemurun as proof the post-boot assertions passed ā they soft-fail unlessSTRICT_POST_BOOT=1; grep the log forā All post-boot assertions passed
Drive the auto-launched installer through AT-SPI#
scripts/atspi-installer-driver.py runs inside a fresh debug ISO guest as
liveuser. It discovers the existing org.bootcinstaller.Installer Flatpak
on that user's session bus, selects /dev/vda, completes the visible
confirmations, and waits for completion. It must not launch the Flatpak: the
acceptance test validates the normal desktop auto-launch path.
The driver uses gi.repository.Atspi, the GNOME OS PyGObject introspection
binding already included by the live image, rather than pyatspi, which the
image does not package. Before driving the UI, just gui-e2e dakota runs the
driver's --check-dependencies probe in the booted debug guest.
import gi
gi.require_version("Atspi", "2.0")
from gi.repository import Atspi
Only DEBUG=1 sets and locks
org.gnome.desktop.interface toolkit-accessibility=true and adds
GTK_MODULES=atk-bridge to the desktop entry, so the auto-launched Flatpak
exposes its tree on the live user's AT-SPI bus. Production ISOs receive none
of those forcing settings. just gui-e2e dakota then uses the existing plain
QEMU boot and verification recipes around the guest driver. On a failure,
retain the driver's tree/log diagnostics and the serial logs.
Diagnostics tree dump must walk deeper than page classification (2026-10-27)#
The installer's page labels sit around depth 22 of its AT-SPI tree, but the
driver walked only 12 levels, so failure dumps never showed the real page
(#244). The failure dump now uses DIAGNOSTICS_MAX_DEPTH = 40. Page
classification in run() keeps the default depth of 12 on purpose: inactive
installer pages also report SHOWING (#119), so walking deeper there would
mix hidden pages into classification. Change those two depths independently.
Source: Context7 /websites/api_pygobject_gnome confirms the
gi.require_version(...); from gi.repository import ... binding pattern.
The current GNOME OS image is the authority for the Atspi namespace used by
the driver.
Verification#
Before marking any E2E work complete:
- Live ISO boots to
DAKOTA_LIVE_READYon serial + SSH responds - fisherman install exits
EXIT:0in install.log - Installed system boots to Graphical target (not just QEMU started)
-
just plain-test-qemu <target>completed withā Installed system boot verified -
just plain-test-qemu <target>/just luks-test-qemu <target>log
ā All post-boot assertions passed(UEFI entry, Flatpak exclusion, and for
LUKS, cmdline UUID ā seescripts/verify-post-install.sh, dakota#651).
Exit status alone is not enough: failures are non-blocking unless run
withSTRICT_POST_BOOT=1 - ISO size ~5.3 GB (release) or ~6.5 GB (fast) ā not 8+ GB (double-embedded store)
- squashfs root contains
proc/,sys/,dev/as empty dirs - GPT type GUID =
28732ac1...(EFI System Partition, not Basic Data)