R2 Promotion#
Managing Dakota ISOs in Cloudflare R2: promoting builds, creating named releases,
and maintaining the latest pointers.
ā Critical rule: never manually overwrite latest#
dakota-live-latest.iso is the production artifact users download. Only CI
may write to it ā after the E2E gate passes.
Manual rclone copyto to the latest pointer is prohibited. It bypasses
the CI E2E test gate (plain-e2e) and ships untested ISOs to users.
This caused a production outage in June 2026 (broken XFS install, issue #85).
ā
Allowed: rclone copyto from current latest objects to create/update named releases (alpha3) after CI E2E passed
ā Prohibited: overwriting dakota-live-latest.iso from a local build
ā Prohibited: overwriting dakota-live-latest.iso-CHECKSUM from a local build
Bucket layout#
| Bucket | Purpose |
|---|---|
testing | All builds ā CI uploads here, all ISOs are public via projectbluefin.dev |
Endpoint: https://2a4147f637f7d9e6a67ca185357d3b0a.r2.cloudflarestorage.com
Account ID: 2a4147f637f7d9e6a67ca185357d3b0a
Latest pointers are long-lived. Dated build objects are no longer retained by policy.
rclone config (~/.config/rclone/rclone.conf)#
Credentials are not stored in this repo. Request them from a project maintainer
or retrieve them from the shared password manager.
[R2]
type = s3
provider = Cloudflare
region = auto
access_key_id = <your-r2-access-key-id>
secret_access_key = <your-r2-secret-access-key>
endpoint = https://2a4147f637f7d9e6a67ca185357d3b0a.r2.cloudflarestorage.com
acl = private
no_check_bucket = true
ā ļø no_check_bucket = true is required ā without it, CopyObject hangs on large files.
ā ļø acl = private is required per Cloudflare docs for object-level permission tokens.
Common operations#
# List bucket contents
rclone ls R2:testing | grep dakota | sort -k2
# Create a named release (e.g., alpha2)
rclone copyto -v \
R2:testing/dakota-live-latest.iso \
R2:testing/dakota-live-alpha2.iso
rclone copyto -v \
R2:testing/dakota-live-latest.iso-CHECKSUM \
R2:testing/dakota-live-alpha2.iso-CHECKSUM
Backup rotation#
Each CI build automatically rotates 3 backup slots before uploading the new latest:
backup-2 ā backup-3
backup-1 ā backup-2
latest ā backup-1
new ISO ā latest
Slots beyond 3 are auto-pruned by the workflow. You never need to manage backup rotation manually.
One-time cleanup (historical ā June 2026)#
The transition from dated publish (YYYYMMDD-SHA objects) to latest-only was performed in
June 2026. All dated objects were deleted at that time. The bucket no longer contains dated
objects by policy. If dated objects reappear (e.g. from an old workflow branch), delete them:
# Preview dated objects first (safe read-only)
rclone lsf R2:testing | grep -E '^[a-z0-9-]+-live-[0-9]{8}-[0-9a-f]{7}\.iso(-CHECKSUM)?$' | sort
# Delete them
rclone lsf R2:testing | grep -E '^[a-z0-9-]+-live-[0-9]{8}-[0-9a-f]{7}\.iso(-CHECKSUM)?$' | \
while read -r f; do
rclone deletefile "R2:testing/${f}"
done
Named ISOs#
| Name | Source | Notes |
|---|---|---|
dakota-live-alpha2.iso | 20260614-7ef17bd | Rebuilt with bootc-installer v2.7.4 (ENOSPC + composefs hostname fixes) |
dakota-live-alpha3.iso | 20260614-9939dd7 | First build with fixed installer (fisherman v0.2.1, bootc-installer v2.7.3) |
dakota-live-alpha4.iso | 20260618-f095551 | composefs installed-boot fix; root-mount-spec injection in build-live-squashfs.sh |
dakota-live-alpha5.iso | 20260701-7ee8e14 | New alpha5 release from the triggered build |
dakota-live-alpha6.iso | to be confirmed | Planned: not yet promoted. build-iso.yml rotates dakota-live-latest.iso daily, so the source build ID is whichever build is latest when promote-release.yml is dispatched from main; record it from the run summary afterwards |
dakota-live-latest.iso | Latest CI build | Auto-updated by daily build-iso.yml |
Public URLs#
https://projectbluefin.dev/dakota-live-latest.iso
https://projectbluefin.dev/dakota-live-latest.iso-CHECKSUM
https://projectbluefin.dev/dakota-live-latest.iso-CHECKSUM.sigstore.json
https://projectbluefin.dev/dakota-live-latest.iso-CHECKSUM.sig
https://projectbluefin.dev/dakota-live-latest.iso-CHECKSUM.pem
https://projectbluefin.dev/dakota-live-backup-1.iso
https://projectbluefin.dev/dakota-live-backup-2.iso
https://projectbluefin.dev/dakota-live-backup-3.iso
https://projectbluefin.dev/utah-live-latest.iso
https://projectbluefin.dev/utah-live-latest.iso-CHECKSUM
https://projectbluefin.dev/utah-live-latest.iso-CHECKSUM.sigstore.json
https://projectbluefin.dev/utah-live-latest.iso-CHECKSUM.sig
https://projectbluefin.dev/utah-live-latest.iso-CHECKSUM.pem
https://projectbluefin.dev/utah-live-backup-1.iso
https://projectbluefin.dev/utah-live-backup-2.iso
https://projectbluefin.dev/utah-live-backup-3.iso
The .sigstore.json bundle is the keyless cosign signature of the latest checksum
(the primary artifact since cosign v3); see
Verifying a downloaded ISO for the check. The .sig/.pem
pair is the same keyless signature in the older separate-file format, still published
alongside for now. Named releases and backup slots carry no signature.
The bluefin-live-* and bluefin-lts-hwe-live-* URLs are retired (#228): those matrix
entries are commented out in build-iso-utah.yml and are not promote-release.yml
choices, so nothing refreshes them. Any objects still in the bucket are stale.
Named releases follow the same pattern:
https://projectbluefin.dev/dakota-live-alpha5.iso
Verifying an ISO without downloading it#
# Fetch just the first 2 KB (GPT headers) and check partition type
curl --range 0-2047 https://projectbluefin.dev/dakota-live-latest.iso -o /var/tmp/head.bin
fdisk -l /var/tmp/head.bin
# Check MBR type byte (0xEE = protective/good, 0x00 = missing)
printf "MBR type: 0x%02x\n" "$(od -An -tx1 -j450 -N1 /var/tmp/head.bin | tr -d ' ')"
Expected: Disklabel type: gpt from gdisk/parted. Note that fdisk shows dos
for hybrid layouts ā this is normal. What matters is the GPT EFI System Partition type.
Inspect GPT without downloading (xorriso in container)#
podman run --rm \
-v ./output:/iso:ro \
debian:sid \
bash -c "
apt-get update -qq >/dev/null
apt-get install -y -qq xorriso >/dev/null 2>&1
xorriso -indev /iso/dakota-live.iso -report_system_area plain 2>/dev/null
"
# Must show: GPT type GUID: 28732ac1... (EFI System Partition OK)
CI upload (build-iso.yml)#
CI uploads latest objects only:
dakota-live-latest.isodakota-live-latest.iso-CHECKSUMdakota-live-latest.iso-CHECKSUM.sigstore.json/.sig/.pemā keyless cosign signature of the checksum (the.sigstore.jsonbundle is the primary artifact;.sig/.pemkept alongside)
Named releases are manual snapshots copied from latest after the CI gate passes.
Before promoting, verify the latest checksum signature
(procedure): the promoted checksum is rewritten
for the new filename and is not re-signed, so the signature on latest is the only
proof that what you are promoting came out of CI.
Rotating R2 credentials#
Use this procedure when credentials are compromised or as routine rotation.
Step 1 ā Create new token (Cloudflare dashboard, ~60 seconds):
dash.cloudflare.com ā R2 ā Manage R2 API Tokens ā Create token- Set permissions:
Object Read & Writeon thetestingbucket - Copy the new
access_key_idandsecret_access_key
Step 2 ā Update GitHub secrets:
gh secret set RCLONE_CONFIG_R2_ACCESS_KEY_ID --repo projectbluefin/dakota-iso
gh secret set RCLONE_CONFIG_R2_SECRET_ACCESS_KEY --repo projectbluefin/dakota-iso
# each prompts for the value ā paste, Enter, done
Step 3 ā Revoke old token in the Cloudflare dashboard.
Step 4 ā Verify:
gh secret list --repo projectbluefin/dakota-iso
# RCLONE_CONFIG_R2_ACCESS_KEY_ID and RCLONE_CONFIG_R2_SECRET_ACCESS_KEY
# should show today's timestamp
Note:
wranglercannot create or revoke R2 API tokens ā the OAuth token from
wrangler logindoes not carry theapi_tokens:editscope. Steps 1 and 3
require the Cloudflare dashboard. Only step 2 is CLI-automatable.
Lessons#
Direct uploads from local host hang/fail (2026-05)#
Uploading multi-GB ISOs directly from this host to R2 hangs indefinitely.
Root cause: likely a routing/MTU issue specific to this network.
Fix: always use R2āR2 server-side copies (rclone copyto R2:testing/src R2:testing/dst).
Server-side copies take 2ā5 min for 4ā5 GB files ā this is normal, do not assume failure.
no_check_bucket = true required in rclone config (2026-05)#
Without no_check_bucket = true, rclone's CopyObject call to the Cloudflare R2 API
hangs indefinitely on large files. This is a known Cloudflare R2 behavior.
Always include this in the rclone R2 config.
rclone endpoint uses account-ID hostname ā keep it configurable (2026-06)#
The R2 endpoint https://<account-id>.r2.cloudflarestorage.com embeds the
account ID as a hostname. Do not hardcode it in documentation ā the format is
required by rclone's S3 provider and cannot be shortened to a path-based form.
When setting up a new rclone config, prompt the user to fill in their account ID
rather than copying a pre-filled value from docs.
There is only one ISO ā delete ghost nvidia-live-* files if they reappear (2026-06)#
The CI produces a single unified ISO (dakota-live.iso) built from the
dakota-nvidia:stable image. There is no separate nvidia variant.
At one point an older CI layout uploaded dakota-nvidia-live-*.iso files.
Those were never updated after the unified ISO landed and became permanently
stale ghost files on R2. They were deleted in June 2026.
If dakota-nvidia-live-* files reappear in R2 (e.g. from an old workflow
branch being re-run), delete them:
for f in dakota-nvidia-live-latest.iso dakota-nvidia-live-latest.iso-CHECKSUM \
dakota-nvidia-live-alpha2.iso dakota-nvidia-live-alpha2.iso-CHECKSUM; do
rclone deletefile R2:testing/$f
done
Do not promote them or reference them in docs.
Overwriting a named release ā copy ISO and checksum together (2026-06)#
When a named release (e.g. alpha2) needs to be replaced, always copy both
the ISO and checksum in the same session so they stay paired.
rclone copyto -v R2:testing/dakota-live-latest.iso R2:testing/dakota-live-alpha2.iso
rclone copyto -v R2:testing/dakota-live-latest.iso-CHECKSUM R2:testing/dakota-live-alpha2.iso-CHECKSUM
GitHub-native named release promotion via promote-release.yml (2026-09-21)#
Named releases historically required a manual rclone copyto command run by an
operator with local R2 credentials. .github/workflows/promote-release.yml codifies
this as a GitHub Actions workflow using repository R2 secrets.
Key invariants codified in the workflow:
- Workflow dispatch restriction: GitHub only allows
workflow_dispatchon the
default branch (main). The workflow file must be merged tomainbefore it can
be dispatched. - Checksum rewrite: A blind
copytoof the checksum leavesdakota-live-latest.iso
inside the checksum payload, causingsha256sum -cto fail against the promoted filename.
The promote workflow reads the source checksum and writes a new file formatted with the
destination ISO name. - Source presence and size parity: Both
.isoand-CHECKSUMmust exist at the source,
and byte sizes are validated after copying to ensure no partial promotion occurs. - Publish race (known gap, not yet enforced):
build-iso.ymlrewrites
*-live-latest.isodaily and the ISO and checksum copies are not atomic. A publish
landing between them promotes a mixed pair that current verification cannot detect,
because it compares the two checksum files rather than hashing the promoted ISO.
Until that is fixed, dispatch promotions well clear of the daily build window.