LUKS Testing#
End-to-end LUKS encrypted install testing. Reproduces projectbluefin/dakota#270.
Quick start (all-in-one)#
just debug=1 installer_channel=dev e2e dakota
# Builds ISO then runs the full LUKS E2E QEMU test
Or separately:
just debug=1 installer_channel=dev iso-sd-boot dakota
just luks-test-qemu dakota
QEMU flow (automated, CI-equivalent)#
luks-test-qemu dakota
āā luks-boot-qemu-live ā daemonize QEMU with live ISO + blank install disk
āā luks-install-qemu ā SSH fisherman LUKS install + BLS patch + shutdown
āā luks-boot-qemu-installed ā daemonize QEMU booting installed disk (no ISO)
āā luks-unlock-qemu ā send passphrase via QEMU monitor, verify boot
āā luks-verify-qemu ā SSH into the unlocked, booted system and assert:
UEFI boot entry present, installer Flatpak
excluded, LUKS cmdline UUID parseable
(projectbluefin/dakota#651)
Post-boot assertions (dakota#651)#
luks-install-qemu/plain-install-qemu call fisherman-install.sh --enable-debug-ssh,
which (mirroring live/src/configure-live.sh's DEBUG=1 convention) patches the
installed deployment directly on disk to set root:root and force-enable
sshd.service, purely for E2E use ā this is never part of a shipped image.
After the installed system boots (and, for LUKS, is unlocked),
scripts/verify-post-install.sh <ssh_port> <encryption_type> SSHes in as
root and checks:
efibootmgr -vshowsBootCurrentpointing at an installer-written NVRAM
entry ā anHD()device path with a\EFI\<vendor>\*.efiloader. OVMF's
auto-enumerated device entries (PciRoot(...), UiApp, EFI shell) and the
removable-media fallback\EFI\BOOT\BOOTX64.EFIdo not count
(fisherman #2 ā requires the/sys/firmware/efi/efivarsbind-mount in
fisherman'spodman runinvocation).flatpak list --system --appdoes not includeorg.bootcinstaller
(fisherman #1 āCopyFlatpaksmust not carry the live-ISO-only installer
app onto the installed target).- LUKS only:
/proc/cmdlinecontains a well-formed (8-4-4-4-12 hex) UUID in
rd.luks.uuid=[luks-]<uuid>orrd.luks.name=<uuid>=<name>
(projectbluefin/common#385).
Both luks-boot-qemu-installed and plain-boot-qemu-installed boot the
installed disk from the pristine OVMF_VARS template by default, so OVMF
enumerates the disk and finds the fallback loader
(EFI/BOOT/BOOTX64.EFI, guaranteed by fisherman's InstallSystemdBoot).
Carrying over the live VM's NVRAM (/var/tmp/dakota-qemu-live-vars.fd) leaves
a stale BootOrder pointing at the detached live ISO, so the firmware drops to
the EFI shell and the unlock automation types the passphrase into the shell ā
the LUKS E2E failure of 2026-11-09 (same #267 mode the plain recipe already
avoided). Set LUKS_CARRY_LIVE_NVRAM=1 (LUKS) or PLAIN_CARRY_LIVE_NVRAM=1
(plain) to opt back into the carry-over when debugging installer-written
Boot#### entries. The installed VM's SSH port is forwarded on 127.0.0.1
only.
The same script backs plain-verify-qemu (encryption type none, skips
assertion 3) and the new luks-verify-qemu recipe (encryption type
luks-passphrase).
Non-blocking by default (observability-only). Until fisherman #1/#2 and
common#385 land,verify-post-install.sh(and soplain-verify-qemu/
luks-verify-qemu) exits 0 on an assertion failure, printing a
ā ļø ... NON-BLOCKINGline (and a GitHub::warning::annotation in
Actions). SetSTRICT_POST_BOOT=1to make assertion failures exit 1 and
fail those observations. SSH reachability and btrfscompress=zstd[:1]in
the mounted/varfilesystem and persistentrootflags=are always hard
requirements. CI (test-luks-install.yml,test-plain-install.yml) does not
setSTRICT_POST_BOOT, so green alone does not prove the older observations ā
check the log forā All post-boot assertions passed.
Key paths#
| Variable | Default |
|---|---|
luks-qemu-disk | /var/tmp/dakota-luks-install.qcow2 |
luks-qemu-ssh-port | 2222 |
luks-qemu-ssh-port-installed | 2224 (forwarded post-reboot for post-boot assertions) |
luks-qemu-monitor-live | /tmp/dakota-qemu-live.sock |
luks-qemu-monitor-installed | /tmp/dakota-qemu-installed.sock |
luks-qemu-serial-live | /tmp/dakota-qemu-live-serial.log |
luks-qemu-serial-installed | /tmp/dakota-qemu-installed-serial.log |
luks-passphrase | testpassphrase |
Clean up before a fresh run:
sudo rm -f /var/tmp/dakota-luks-install.qcow2 /tmp/dakota-qemu-*.sock /tmp/dakota-qemu-*.log
Libvirt flow (interactive)#
# 1. Build debug ISO
just debug=1 installer_channel=dev iso-sd-boot dakota
# 2. Boot in libvirt, wait for SSH ready
just debug=1 boot-libvirt-debug dakota
# 3. SSH fisherman LUKS install + reboot
just luks-install dakota
# 4. Automated passphrase unlock
just luks-unlock dakota
# or interactive serial console (type: testpassphrase)
just luks-boot dakota
Cleanup:
sudo virsh destroy dakota-debug && sudo virsh undefine dakota-debug --nvram
Manual QEMU install (for debugging)#
QEMU=/home/linuxbrew/.linuxbrew/bin/qemu-system-x86_64
OVMF=/home/linuxbrew/.linuxbrew/Cellar/qemu/11.0.0/share/qemu/edk2-x86_64-code.fd
VARS=/var/tmp/OVMF_VARS.fd; dd if=/dev/zero bs=1k count=256 of=$VARS 2>/dev/null
qemu-img create -f qcow2 /var/tmp/dakota-install-disk.qcow2 50G
$QEMU -machine q35 -m 8192 -accel kvm -cpu host -smp 8 \
-drive if=pflash,format=raw,readonly=on,file=$OVMF \
-drive if=pflash,format=raw,file=$VARS \
-drive if=none,id=live,file=output/dakota-live.iso,media=cdrom,format=raw,readonly=on \
-device virtio-scsi-pci,id=scsi -device scsi-cd,drive=live \
-drive if=none,id=target,file=/var/tmp/dakota-install-disk.qcow2,format=qcow2 \
-device virtio-blk-pci,drive=target \
-net nic,model=virtio -net user,hostfwd=tcp::2222-:22 \
-device usb-ehci -device usb-tablet \
-display gtk,zoom-to-fit=on \
-serial file:/var/tmp/dakota-serial.log &
# Wait for live env
until grep -q DAKOTA_LIVE_READY /var/tmp/dakota-serial.log 2>/dev/null; do sleep 5; done
# Write recipe and run fisherman install
sshpass -p live ssh -o StrictHostKeyChecking=no -o PubkeyAuthentication=no \
-o PreferredAuthentications=password -p 2222 liveuser@localhost '
sudo /usr/local/bin/fisherman - << EOF
{"disk":"/dev/vda","filesystem":"btrfs","image":"","composeFsBackend":true,"bootloader":"systemd","hostname":"test","encryption":{"type":"luks-passphrase","passphrase":"testpassphrase"},"flatpaks":[]}
EOF
'
fisherman recipe fields#
{
"disk": "/dev/vda",
"filesystem": "btrfs",
"image": "",
"composeFsBackend": true,
"bootloader": "systemd",
"hostname": "dakota-luks-test",
"encryption": {
"type": "luks-passphrase",
"passphrase": "testpassphrase"
},
"flatpaks": []
}
encryption: null for unencrypted install.
On bound live media, empty image and omitted targetImgref select the host's
mandatory receipt; they do not enable a network fallback. The automated plain,
LUKS and libvirt callers explicitly copy both fields from that same receipt and
use the shipped pinned fisherman (no local replacement backend builds). The
receipt identifies the canonical read-only OCI layout and hardware-selected
update target; do not restore containers-storage: sources.
LUKS passphrase unlock (luks-unlock.py)#
live/src/luks-unlock.py handles automated passphrase injection. It:
- Polls QEMU monitor screendumps to detect Plymouth's passphrase prompt
- Injects keystrokes via
sendkeycommands - Monitors serial log for successful boot
Accepts two modes: qemu (QEMU monitor socket) and libvirt (serial PTY).
BLS console patch#
After install, the justfile patches BLS boot entries to add both serial and VT consoles:
console=tty0 console=ttyS0
This ensures CI can read the installed system's serial log AND the user gets a
graphical console.
Interactive QEMU rules#
- Always use
-display gtk,zoom-to-fit=onfor interactive sessions - Always create disks in
/var/tmp/ā never/tmp(16 GB tmpfs) - Install disk must be 50 GB+ ā create before launching QEMU
usb-tabletdevice is required for mouse tracking in GTK window
Lessons#
Boot installed disk: qcow2 + no cdrom = OVMF discovers virtio-blk (2026-06)#
When booting the installed disk in QEMU, two things are required for OVMF to
auto-discover and boot from the virtio-blk device without EFI NVRAM entries:
- Disk must be qcow2 (not raw). OVMF's fallback boot scan works correctly with
qcow2; raw format can cause scan failures depending on OVMF version. - No cdrom device attached. When a cdrom is also connected, OVMF tries it first
and may fail to enumerate the virtio-blk device for fallback boot.
The luks-boot-qemu-installed justfile recipe follows this pattern ā no -device scsi-cd,
disk as format=qcow2. Always use the justfile recipe for installed-disk boot.
If you installed to a raw image (e.g. via dd or a raw-format QEMU disk), convert first:
qemu-img convert -f raw -O qcow2 /var/tmp/install-disk.img /var/tmp/dakota-luks-install.qcow2
/var/tmp for disk images, never /tmp (2026-05)#
/tmp is a 16 GB tmpfs on this host. Two 50 GB sparse qcow2 images fill it
instantly, causing the QEMU VM to pause mid-install with an I/O error.
Always use /var/tmp/ for OVMF VARS files and install disk images.
setsid required for fisherman over SSH (2026-05)#
Running fisherman directly via SSH (ssh ... sudo fisherman recipe.json) gets
killed when the SSH session terminates (SIGHUP). Wrap with setsid:
sshpass -p live ssh ... 'setsid bash -c "sudo fisherman recipe.json > /tmp/install.log 2>&1" </dev/null >/dev/null 2>&1 &'
Poll interval: 15s not 120s (2026-05)#
Polling fisherman completion at 120s intervals causes 2-minute detection gaps.
Poll at 15s ā fisherman takes 5ā15 min but the completion check is cheap.
Live boot screenshot timing in QEMU (2026-06-02)#
The live boot screenshot was captured immediately after SSH or the serial ready check succeeded. However, SSH and systemd units start much earlier than GDM autologin and the installer GUI. Capturing the screenshot immediately resulted in a black screen.
Fix: Added wait-live mode to luks-unlock.py which monitors screen brightness and hash stability. It waits up to 5 minutes for the screen to become bright (brightness >= 1.5) and stable before capturing the PPM.
LUKS ENOSPC: fisherman skips /var/tmp bind-mount for encrypted targets (2026-06-14)#
The live environment mounts a tmpfs at /var/tmp sized at 80% of VM RAM
(~3.2 GiB with the default 4 GiB QEMU allocation). During composefs install,
fisherman exports the OCI image to an OCI layout via skopeo; containers-storage
writes intermediate blob temp files to /var/tmp regardless of TMPDIR.
For plain (non-LUKS) installs fisherman detects the space-constrained live
environment and bind-mounts /var/tmp ā <target>/.fisherman-scratch/var-tmp-override
before running skopeo, making writes disk-backed. For LUKS targets this
bind-mount does not happen, causing the ~9 GB blob extraction to fill the
~3.2 GiB tmpfs and fail with no space left on device.
Fix: luks-boot-qemu-live creates a 16G sparse scratch disk (luks-scratch-disk,
default /var/tmp/dakota-luks-scratch.img) and passes it to QEMU as a second
virtio-blk device (/dev/vdb). luks-install-qemu SSHes into the live VM,
formats /dev/vdb as ext4, and mounts it over /var/tmp before fisherman runs.
The scratch disk is cleaned up along with the install disk in the e2e recipe.
This fixes every LUKS E2E run since composefs was enabled (all had been silently
timing out at the 90-minute job ceiling before the fisherman wrapper surfaced
the real error).
Installed SSH failures need transport evidence (2026-11-11)#
Publication run 42950848237 completed installation, patched the selected
composefs deployment's root password, and reached graphical startup, but failed
installed SSH readiness. That rules out the previously observed silent password
replacement miss for this run; it does not establish whether networking, daemon
startup, or authentication failed. Do not infer a store-transfer regression from
the timing of a preceding merge alone.
verify-post-install.sh now retains the final SSH attempt's stderr and exit status
in a private temporary file, prints them only on readiness failure, and removes
the file on exit. It reports actual elapsed seconds and attempt count instead of
claiming a five-minute deadline (connection time adds to each sleep). It does not
print the password-bearing command or enable verbose credential logging. The
60-attempt policy and hard exit 2 are unchanged; mounted compression and persistent
rootflags remain required after connection. Stubbed transport tests cover refusal,
timeout, authentication failure, recovery and missing compression; these are not
installed-guest proof.
For #309, diagnosis and a fresh full install/installed boot/SSH/compression check
through the publication path are still required. Compare the failed serial log
and disk with successful run 42569645735 before selecting a guest-side fix. Do
not rerun production publication merely to gather diagnostics without approval.
Silent debug-SSH miss: shadow without a root: line (2026-11-09)#
scripts/fisherman-install.sh --enable-debug-ssh sets the installed system's
root password with sed s|^root:[^:]*:|...| on the deployed etc/shadow.
When the deployed shadow has no root: line (image drift ā e.g. before the
root:!unprovisioned: convention), sed is a silent no-op: the install
reports success and E2E 4/4 burns 5 minutes on Failed password for root
before failing SSH-unreachable. The wrapper now prints every deployment-etc
candidate, the selected DEPLOY_ETC, and the pre-patch root: field, and
exits 1 via grep -qF when the replacement does not land (or shadow is
absent) so the install step fails fast with the cause in the log instead of
timing out on SSH.
Plain install: debug-SSH patch boots empty, root login rejected (2026-11-14)#
On the plain (non-LUKS) E2E the post-install patch logged success, but the
installed system rejected every root SSH login (password check failed for user (root)) and firewalld logged INVALID_ZONE: not a valid zone file: no element found: line 1, column 0 for the public.xml the patch had just written. The
files existed on disk but were empty: the data never left the page cache (#317).
fisherman detaches its own mount of the target with umount -Rl. On a plain
install the wrapper mounts that same partition again, so it shares the still-live
filesystem and its umount is not the last one: nothing forces writeback. The
harness then powers the live VM off, and whether the data made it out was down
to timing (stable passed in the same run). The LUKS path opens a separate mapping,
so its unmount is final and flushes, which is why LUKS dev passed.
The wrapper now runs sync before unmounting the installed root. Any new step
that writes to the installed disk from the live session must finish before that
sync.
Leftover fisherman-root mapping skips the post-install patch (2026-11-12)#
fisherman unlocks the target as /dev/mapper/fisherman-root and closes it on
teardown, but that close can lose a race (Device fisherman-root is still in use.). scripts/fisherman-install.sh then tried to luksOpen the same
partition again, got Cannot use device /dev/vda2 which is in use, skipped the
debug-SSH patch, and still printed post-install patch complete ā so the run
failed five minutes later at the SSH gate with a misleading error (#314).
The wrapper now checks cryptsetup status fisherman-root first: it settles udev
and retries cryptsetup close up to five times, and if the mapping is still in
use it mounts through fisherman-root instead of opening the partition twice.
If the installed root cannot be opened or mounted at all, the wrapper exits 1
so the install step fails with the real cause in its log.
LUKS emergency shell: missing rd.luks.name= kernel arg (2026-06-23)#
After the ISO build was fixed, all 6 LUKS E2E variants (dakota dev/stable, lts
dev/stable, stable dev/stable) failed with:
[luks-unlock] RESULT: emergency shell ā issue #270 reproduced
Serial log reveals:
[ TIME ] Timed out waiting for device dev-disk-by-x2dlabel-root.device
[DEPEND] Dependency failed for sysroot.mount - /sysroot
The kernel cmdline in the BLS entry contained root=LABEL=root with no
rd.luks.name= argument, so the initramfs never unlocked the LUKS container
and could not find the root filesystem.
Root cause: The justfile has TWO install paths for LUKS tests:
| Path | Usage | Fisherman source | Has EnsureLuksArgs? |
|---|---|---|---|
| composefs (dakota) | "composeFsBackend": true + "image" set | Flatpak /usr/local/bin/fisherman | ā No |
| bootcDirect (stable, lts) | "image": "" + "targetImgref" set | Compiled from tmp_bootc_installer/ | ā Yes |
The tmp_bootc_installer/fisherman source (internal/post/luks_args.go) has
EnsureLuksArgs() which injects rd.luks.name=<UUID>=root into BLS entries.
This is compiled and used in the bootcDirect path (passing). The composefs
path uses the Flatpak-provided fisherman which lacks this function (failing).
Fix: Add rd.luks.name=<UUID>=root injection to the justfile's composefs
BLS patch step (justfile, composefs path, around line 859). Before patching
the BLS entry, query cryptsetup luksUUID /dev/vda2 for the LUKS UUID and
append it to the options line:
LUKS_UUID=$(cryptsetup luksUUID /dev/vda2 2>/dev/null || echo "")
sed -i "s|^options .*|& console=tty0 console=ttyS0 rd.luks.name=${LUKS_UUID}=root|" "$entry"
Verification: Run just luks-test-qemu dakota and confirm serial log shows:
Starting systemd-cryptsetup@root.service - Cryptography Setup for root...
Please enter passphrase for disk root::*****
Finished systemd-cryptsetup@root.service
Found device dev-disk-by-x2dlabel-root.device
Long-term fix: Ship EnsureLuksArgs in the projectbluefin/bootc-installer
Flatpak so both install paths are covered.