CI Router#

Overview#

This is the load-first CI skill. Do not dump the whole CI history into context up front.
Load this file, identify the failure class, then load only the next skill you need.

When to Use#

Use this skill when the task mentions:

• GitHub Actions failures
• startup_failure, action_required, missing jobs, or flaky checks
• publish.yml, build.yml, execute-release.yml
• boot-check, smoke, testsuite, SBOM, or GHCR publish problems
• merge queue, promotion PRs, or stable release flow

When NOT to Use#

• Element build or packaging failures inside BST → debugging.md
• BST syntax, element kinds, or project layout → buildstream.md
• OCI image contents or layer assembly → oci-layers.md
• Normal PR review → pr-review.md

⚠️ Builder Discipline — Read Before Doing Anything#

ONE BST build at a time. Always.

Before merging, pushing, or dispatching any workflow, run the mandatory pre-flight:

gh run list --repo projectbluefin/dakota --limit 30 \
  --json databaseId,status,name,headBranch \
  | python3 -c "
import json, sys
runs = json.load(sys.stdin)
active = [r for r in runs if r['status'] in ('in_progress', 'queued', 'pending', 'waiting')]
if active:
    print(f'BLOCKED: {len(active)} active run(s). Cancel all before proceeding:')
    for r in active:
        print(f' gh run cancel {r[\"databaseId\"]} --repo projectbluefin/dakota # {r[\"name\"]} [{r[\"headBranch\"]}]')
else:
    print('OK: field is clear')
"

If output is not OK: field is clear — cancel every listed run first.

Cache-warm is not exempt. It shares the same ubuntu-24.04 runner pool and CAS write bandwidth as real builds. Two concurrent BST jobs do not halve wall time — they more than double it and risk 6-hour timeouts with zero elements cached.

The rationalizations that have caused real production failures:

• "Cache-warm is additive, it helps the build" → No. Cancel it.
• "This is almost done, just a few minutes" → Cancel it. You don't know that.
• "It's a different branch, it won't interfere" → Same runners and CAS. Cancel it.
• "I cancelled one, that's enough" → Cancel ALL. Re-run pre-flight.

Core Process#

1. Run the mandatory pre-flight above. Verify OK: field is clear.
2. Classify the failure before reading logs.
   • Which workflow? build, publish, promote, release, e2e, merge queue
   • Which phase? trigger, setup, reusable workflow call, build/export, boot, smoke, promotion
3. Load one next skill, not five.
   • Need workflow/trigger map → workflow-map.md
   • Need reusable workflow / permissions / cache-dir weirdness → ci-tooling.md
   • Need boot-check / smoke / testsuite behavior → e2e-ci.md
   • Need :testing → :stable / release flow → release-promotion.md
   • Need stale PR or queue cleanup → merge-queue.md
4. Read the actual workflow file before editing.
5. Verify tool behavior via Context7 for GitHub Actions or bootc when changing syntax/flags.
6. Write back the lesson to the narrowest skill file, not this router, unless the routing itself changed.

Skill Selection Table#

Workflow Quick Reference#

Trigger Behavior#

push paths-ignore: .github/workflows/**, docs/**, **.md, AGENTS.md — doc/workflow-only pushes do NOT trigger a build. This is intentional; it means a CI-only commit advancing the branch HEAD will leave no build artifact for that SHA.

Branch → tag mapping (verified from publish.yml source):

• testing or gh-readonly-queue/testing/* → :testing
• next or gh-readonly-queue/next/* → :next

PR path: validate + e2e (change-detected) — zero remote execution. ~15 min cached, ~30 min cold.

e2e change detection: e2e uses a should-run job that diffs the PR branch against its base. It runs when elements/, files/, patches/, Justfile, or project.conf change; otherwise the e2e job is skipped. Skipped satisfies the required status check.

Merge queue path: build fires on merge_group — full OCI build, real CI gate before merge. PRs target testing; next retains its own merge queue.

Daily build schedule: build.yml fires at 13:00 UTC daily (after nightly-next-build completes). This keeps CAS warm and ensures a fresh :testing tag each day even without a code push. cache-warm.yml was deleted — the daily build replaces it.

ARM build: build-aarch64.yml fires via workflow_run from publish.yml on the testing branch. This serializes ARM after x86 CAS writes complete, preventing contention. The previous Tuesday cron trigger was removed.

Remote Cache Architecture#

cache.projectbluefin.io:11002 handles all five BST remote services: artifact cache, source cache, CAS storage, remote execution, and action cache. All use the same endpoint with mTLS auth.

mTLS Authentication#

Push is conditional: Remote cache section is only added to buildstream-ci.conf if both are set. Without credentials, BST builds from source using local disk cache only — slower but functional. This is normal for external contributors' forks.

⚠️ Pre-Commit BST Syntax Gate#

For any change to project.conf, *.bst elements, or Justfile:

just bst show oci/bluefin.bst

Must exit clean before git commit. Catches invalid option names, types, and element references. Takes 5 seconds. Skipping wastes a 90-second CI build slot.

⚠️ Branch Base Rule#

Always branch from upstream/testing (the development trunk), never from local testing or main:

git checkout upstream/testing -b feature/my-change
git diff upstream/testing...HEAD --stat # verify before pushing

Recovery when a branch is already dirty:

git rebase --onto upstream/testing <last-unwanted-commit-sha> <branch-name>
git push --force-with-lease origin <branch-name>

Debugging CI Failures#

Where to Find Logs#

Common Failures#

Debugging Workflow#

1. Check config step output — confirms whether artifacts: / source-caches: sections are present
2. Search build log — look for [FAILURE] lines; on-error: continue collects all failures
3. Check if remote cache was hit — look for [get artifact] lines showing cache.projectbluefin.io:11002
4. Reproduce locally — just bst build oci/bluefin.bst uses the same bst2 container

Generated Files (Pre-Commit Required)#

Some files are generated locally and committed — they cannot be regenerated in CI because generation requires bst artifact list-contents, which only reads the local BST artifact cache (not remote execution cache).

# Regenerate
rm files/filemap.json files/fakecap-manifest.tsv
python3 scripts/gen-filemap.py
git add files/filemap.json files/fakecap-manifest.tsv
git commit -m "chore: regenerate chunkah filemap and fakecap manifest"

Treat these like Cargo.lock — commit the updates with your element changes.

Bot PR CI — GITHUB_TOKEN Suppression#

PRs created by a workflow using GITHUB_TOKEN do NOT fire pull_request events — GitHub suppresses workflow triggers from its own bot token to prevent recursive loops.

Fix: Use a GitHub App token (mergeraptor) for gh pr create in track-bst-sources.yml.

Ruleset#

testing (development trunk)#

Ruleset: testing-merge-queue-no-review

testing is the GitHub default branch and the merge target for all PRs.

main (release bookmark)#

Ruleset: main-bookmark-protection

main is written only by execute-release.yml via fast-forward after each successful stable promotion.

e2e change detection: e2e only tests PRs touching elements/, files/, patches/, Justfile, or project.conf. For all other paths (e.g. workflow pin bumps) the e2e job is skipped, which satisfies the required check. The should-run job uses git diff against the PR base — no paths: filter on the trigger.

Critical: Required status checks must only include checks that fire on pull_request. A check that only fires on merge_group will permanently block the "Add to merge queue" button.

Session Bootstrap Rule#

At the start of every dakota session, check GNOME OS upstream status:

gh pr list --repo gnome/gnome-build-meta --state open --limit 10
gh run list --repo projectbluefin/dakota --limit 5

Cross-References#

Lessons Learned#

Note: Lessons are ordered newest-first. Entries before 2026-06-23 may reference workflows that have since been deleted (promote-testing-to-main.yml, pr-release-gate.yml, sync-main-to-testing.yml, cache-warm.yml). Those workflows were deleted in the OCI-native redesign (issue 1073). Do not recreate them.

ARM warm-cache must be a parallel job with its own concurrency group (2026-06-22)#

The cache-warm.yml originally had only an x86_64 job. Adding ARM as a second
step inside the same job would serialise the two architectures and block x86 on
ARM failures. The correct pattern:

• Add a second top-level job (warm-cache-aarch64) — no needs: dependency.
• Use a separate concurrency.group (dakota-cache-warm-aarch64) so the two
  jobs never queue behind each other.
• Set continue-on-error: true on the ARM job — ARM failures never block x86.
• Use BST_FLAGS: --no-interactive --config /src/buildstream-ci.conf --option arch aarch64
  — the --config flag is required for the generated BST CI config (and the
  remote CAS push) to actually take effect.
• Use enable-remote-execution: false, enable-push: true — no RE service for
  ARM yet, but artifacts should still be pushed to the remote CAS.
• Use a distinct BST workspace cache key: bst-warm-aarch64-${{ hashFiles(...) }}
  — sharing a key with x86 will cause cross-arch cache pollution.

crun 1.21 (resolute) breaks just sbom on GHA — use --runtime runc (2026-06-08)#

update-podman: true in setup-runner installs crun 1.21 from Ubuntu 26.04
(resolute). This version has two new failure modes that break just sbom on
GHA runners:

1. seccomp BPF linkat EPERM — crun caches compiled seccomp BPF programs
   via linkat(). The GHA runner kernel's fs.protected_hardlinks or user-
   namespace restrictions block the hard-link from .cache/seccomp/ to the
   container bundle path:

   ⧉

   crun: linkat `.cache/seccomp/<hash>` to `<container-id>/seccomp.bpf`: Permission denied
   

2. systemd probe EACCES — crun probes systemd presence and caches the result
   in $XDG_RUNTIME_DIR/crun/.cache/systemd-missing-properties. On GHA the
   runtime dir is either uninitialised or was created by root in a prior privileged
   step, causing user 1001 to get EACCES:

   ⧉

   crun: opendir `/run/user/1001/crun/.cache/systemd-missing-properties`: Permission denied
   

Fix: add --runtime runc to both podman run calls in just sbom. runc is
always available on ubuntu-24.04 GHA runners (Docker installs it). runc has
neither the seccomp BPF caching nor the systemd probing.

Wrong partial fixes (both insufficient alone):

• Dropping --privileged (#745) — doesn't prevent either error
• Adding --security-opt seccomp=unconfined (#747) — fixes error 1 but not error 2

Do not add seccomp=unconfined as a workaround; use --runtime runc instead.
bst show and buildstream-sbom are read-only BST operations; runc is fully
sufficient.

# ✅ correct
podman run --rm --network=host --runtime runc ...

# ❌ wrong — triggers crun 1.21 failure modes
podman run --rm --network=host ...
podman run --rm --network=host --security-opt seccomp=unconfined ...

Continuous model — every merge ships immediately (2026-06-07)#

The pipeline was redesigned so every PR merge produces a new :testing image
without any e2e gate in the publish path. The schedule trigger was removed from
build.yml; builds now only fire on merge_group and workflow_dispatch.

New flow:

PR merge_group → build.yml → publish.yml → :$sha → :testing (no e2e)
                                                           │
                     weekly-testing-promotion.yml ─────────┘
                     (e2e gate here, then :stable)

Implication: :testing may briefly be broken if a PR introduces a regression.
The e2e gate at the weekly promotion prevents regressions from reaching :stable.

If breaks: look at the last few merge SHAs and bisect with
gh run list --workflow "Publish Bluefin dakota" --limit 10.

TOCTOU guard interaction: the weekly promotion's lock-sha step uses a GitHub
compare API ancestor check rather than exact equality. With continuous builds,
main will often be 1–2 commits ahead of :testing by Tuesday 06:00 UTC. An
exact-equality check would cause every promotion to fail. The ancestor check
allows promotion as long as :testing is a valid ancestor of main (i.e.,
histories have not diverged):

COMPARE=$(gh api "repos/${REPO}/compare/${SOURCE_SHA}...${CURRENT_SHA}" --jq '.status')
# "ahead" = main advanced past :testing = normal and fine
# anything else = diverged = abort

publish.yml startup_failure = is stale (2026-06-04)#

startup_failure on publish.yml nightly runs means the BST artifact or
CAS cache lookup failed before the job even started. When this happens on two
or more consecutive nights, :testing stops being updated. Symptoms visible
downstream: every dep-update PR shows "SSH never became ready" in e2e because
the QEMU VM tries to boot the stale image. Fix: investigate publish.yml
startup_failure first — check repo Secrets/Variables for CASD_CLIENT_CERT
and CASD_CLIENT_KEY expiry, and confirm the CAS server is reachable.

Also check if the workflow is disabled. A disabled_manually workflow
silently produces startup_failure with zero job output — jobs: [].
Check with:

gh api repos/projectbluefin/dakota/actions/workflows \
  --jq '.workflows[] | "\(.id) \(.state) \(.name)"'

Re-enable with:

gh api repos/projectbluefin/dakota/actions/workflows/<id>/enable --method PUT

Two confirmed causes of startup_failure with jobs: [] (2026-06-04):

1. Invalid top-level permissions: key — artifact-metadata: write is NOT a
   valid GITHUB_TOKEN permission scope. GitHub rejects the workflow at parse time
   before creating any jobs. actionlint does not catch this. Remove it.
   Valid scopes: actions, checks, contents, deployments, discussions,
   environments, id-token, issues, packages, pages, pull-requests,
   repository-projects, security-events, statuses, attestations.

2. Job-level permissions: on a reusable workflow call job — adding a
   permissions: block to a job that uses uses: (external reusable workflow)
   can cause GitHub to fail the entire workflow at startup. The working pattern
   (used by local e2e.yml) is to call the reusable workflow WITHOUT job-level
   permissions; it inherits from the top-level permissions: block instead.

After fixing startup_failure, publish may still fail if no BST artifact is in
CAS for the current main SHA. This happens when build.yml has only run on
branches (not main). Fix: dispatch build.yml on main first, wait for it to
complete (~5–6 hours), then dispatch publish.yml.

gh workflow run build.yml --repo projectbluefin/dakota --ref main
# wait for completion, then:
gh workflow run publish.yml --repo projectbluefin/dakota

Dep updates on testing not reaching main (2026-06-04)#

When dep-update PRs are merged directly to testing, publish.yml (which
builds from main) never sees them. Before dispatching a build or promotion,
check the gap:

git log --oneline upstream/main..upstream/testing -- elements/ files/ patches/

If commits exist, land them via a PR to main:

git checkout upstream/main -b fix/land-testing-deps
# Apply only element/files/patches diff — avoid docs/CI conflicts:
git diff upstream/main..upstream/testing -- elements/ files/ patches/ \
  > /tmp/testing-deps.patch
git apply --index /tmp/testing-deps.patch
git commit -m "chore(deps): land testing dep updates into main"
git push upstream fix/land-testing-deps
gh pr create --repo projectbluefin/dakota --base main --head fix/land-testing-deps ...

Do not cherry-pick the squash commits directly — they bundle docs/CI
changes that have already diverged between testing and main, producing
unresolvable conflicts in AGENTS.md, CODEOWNERS, and docs/skills/.

Same e2e failure on all PRs = infrastructure, not code (2026-06-04)#

If e2e / GNOME 50 — smoke fails with identical output across 4+ unrelated
PRs simultaneously, it is always an infrastructure issue — never a per-PR
code bug. The test suite tests :testing not the PR branch. Skip individual
PR debugging and go straight to:

gh run list --repo projectbluefin/dakota --workflow publish.yml --limit 10 \
  --json databaseId,conclusion,createdAt

If the last successful publish run is >24 hours old, :testing is stale.
Check projectbluefin/testsuite for open issues before filing a new one.

Remote CAS down = build dies immediately at element loading (2026-06-07)#

When cache.projectbluefin.io:11002 is unreachable, buildbox-casd exits after
6 connection retries (~18 seconds). BST reports this as a cryptic inner failure:

BUG: Message handling out of sync, unable to retrieve failure message for element plugins/buildstream-plugins-community.bst
FAILURE Loading elements
error: recipe `bst` failed with exit code 255

The real root cause is in the CASD log artifact:

[ERROR] Retry limit (5) exceeded for "GetCapabilities()"
[ERROR] 14: Failed to connect to remote host: Connection refused

Diagnosis:

gh run download <run-id> --repo projectbluefin/dakota \
  --name buildstream-logs-x86_64-default -D /tmp/bst-logs
cat /tmp/bst-logs/_casd/*.log | grep -E "connect|refused|ERROR" | tail -10

Fix: The remote CAS is infrastructure — it needs to be restarted on the server.
If the cache is truly down, the build cannot proceed (without the cache.storage-service,
BST has no local artifact store and cold-rebuilds everything which times out).
Re-trigger the build once the cache is back up:

gh workflow run "Build Bluefin dakota" --repo projectbluefin/dakota --ref main

Ghost-local workaround: Does not apply — ghost's userconfig has no remote CAS
configured, so ghost builds are unaffected by cache outages.

Ghost-specific build fixes belong in userconfig, NOT elements (2026-06-07)#

If a BST element fails to build on ghost but works in CI (remote execution), the
fix must go in ghost's local config — never in the element itself. Putting it
in the element invalidates the remote CAS artifact (cache-bust), forcing CI to
rebuild an element it was already handling correctly.

Real example (2026-06-07): Adding CARGO_PROFILE_RELEASE_LTO: "thin" to
uutils-coreutils.bst to fix a SIGABRT on ghost caused a 626-element cold rebuild
in CI. The build ran for 5h31m and timed out without completing (330-minute limit).

Wrong: elements/bluefin/foo.bst + environment: CARGO_PROFILE_RELEASE_LTO: "thin"
Right: ghost ~/.config/buildstream/userconfig.yaml project/element environment override

Ghost-specific environment overrides can go in userconfig under:

projects:
  dakota:
    elements:
      bluefin/uutils-coreutils.bst:
        environment:
          CARGO_PROFILE_RELEASE_LTO: "thin"

Diagnosing a slow in-progress build via GitHub API (2026-06-07)#

When a build has been running for 2–3+ hours and you want to know what's being
compiled without waiting for a timeout:

# 1. Find the in-progress build and its job IDs
gh api repos/projectbluefin/dakota/actions/runs/<run-id>/jobs | python3 -c "
import json, sys
from datetime import datetime, timezone
d = json.load(sys.stdin)
now = datetime.now(timezone.utc)
for job in d.get('jobs', []):
    if job.get('started_at'):
        s = datetime.fromisoformat(job['started_at'].replace('Z','+00:00'))
        mins = int((now - s).total_seconds() / 60)
        print(f\"{job['id']} | {job['status']} | {mins}m | {job['name'][:60]}\")
"

# 2. Fetch the live log (note: truncated at ~23K lines for long builds)
gh api repos/projectbluefin/dakota/actions/jobs/<job-id>/logs > /tmp/bst-live.log

# 3. Count cache hits vs elements being compiled
grep -c "SKIPPED" /tmp/bst-live.log # cache hits
grep "Running commands" /tmp/bst-live.log | tail -20 # what's actively building

# 4. See which upstream elements are being compiled (indicates junction drift)
grep "START.*Running commands" /tmp/bst-live.log | grep -oE "\[.*\]" | sort -u

Important: The live log endpoint is a snapshot, not a stream. For builds
running > ~90 minutes, the log may be stale by 60–90 minutes relative to current
wall-clock time. If the last log timestamp is behind by > 1 hour, the build is
still running but log data is not being returned. Use gh api repos/.../actions/runs/<id>/jobs to confirm status: in_progress.

Deciding whether to re-trigger: A build making steady progress on
gnome-build-meta core-deps/ elements is normal cache-warming after a GNOME
nightly — let it run. Only re-trigger if:

• The run hits a timeout error
• Elements are stuck "Waiting for the remote build to complete" for > 30 min (CAS issue)
• The build failed with a compilation error

gnome-build-meta nightly delta builds (2026-06-07)#

The GNOME upstream nightly (~08:00 UTC) updates a batch of core-deps/ elements
in gnome-build-meta. The first build that runs after a nightly must recompile
those elements from scratch. This is expected and not a problem.

Typical pattern:

• 1,000+ elements: SKIPPED (cache hits from the previous build)
• 10–30 core-deps/ elements: recompiled (changed in nightly)
• Each element compiles in 1–5 minutes; total extra time: ~60–120 minutes
• Build completes well within the 330-minute timeout

Elements commonly rebuilt after a nightly: protobuf, folks, sofia-sip,
procps, containers-common, libvirt-glib, spice-gtk, foundry, feedbackd,
jsonrpc-glib, libgit2-glib.

How to confirm it's a nightly delta (not a cache bust):

# Check which junction commit the failing build used:
grep "Fetching from.*gnome-build-meta" /tmp/bst-live.log

# Compare to the junction ref pinned in the element:
grep "ref:" elements/gnome-build-meta.bst

If the junction ref in elements/gnome-build-meta.bst matches what the build
fetched, the cache miss is upstream drift, not a local element change.

After a nightly delta completes, subsequent builds are fast again (< 90 min)
because all newly-compiled elements land in the remote CAS.

Diagnosing a build timeout (330-minute limit) (2026-06-07)#

A build that hits the 330-minute GitHub Actions timeout shows:

The action 'Build OCI image with BuildStream' has timed out after 330 minutes.

No element "failed" — the build was still running. Download the logs to find what
was active at timeout:

gh run download <run-id> --repo projectbluefin/dakota \
  --name buildstream-logs-x86_64-default -D /tmp/bst-logs

# Find elements that were waiting for remote execution when the timeout hit:
grep -rl "Waiting for the remote build to complete" /tmp/bst-logs/ | while read f; do
  tail -1 "$f" | grep -q "Waiting" && echo "$f"
done

# Find the latest-timestamped log files (actively building at timeout):
find /tmp/bst-logs -name "*.log" | grep -oP '\d{8}-\d{6}' | sort | tail -10

Root causes of timeouts:

After fixing the root cause, the re-triggered build will use the existing CAS
artifacts for all elements whose cache keys are unchanged — typically a warm build
completes in under 90 minutes.

Promotion pipeline hardening — bonedigger and release race (2026-06-07)#

bonedigger "workflow file issue": The lifecycle caller (bonedigger.yml) was
pinned to a common SHA that pre-dated lifecycle.yml existing in that repo. Also,
the brand_emoji input is not declared by the reusable workflow — passing an
undeclared input causes a GitHub workflow validation failure. Fix: update the SHA
pin to a commit where the file exists and remove undeclared inputs.

# Find commits that contain the target workflow file
gh api "repos/projectbluefin/common/commits?path=.github/workflows/lifecycle.yml&per_page=3" \
  --jq '.[].sha'

release.yml must not re-discover the publish run independently: If release.yml
queries gh run list --limit 1 after the promotion pipeline completes, a concurrent
publish run for a new SHA can land first and be picked up instead. Always pass the
promoted source_sha and promoted_digest as workflow_call inputs from the
promotion pipeline so release.yml filters by exact headSha.

Invalid OCI digest fallback: Never synthesize an OCI digest from a git SHA
(sha256:${git_sha}). If skopeo inspect fails, fail the job — a release with
a fake digest has wrong verification commands in the release notes.

cert-identity-regexp must be fully anchored: Cosign uses MatchString semantics,
so a regexp without a trailing $ matches any URL with that prefix. Always anchor:

^https://github\.com/projectbluefin/dakota/\.github/workflows/publish\.yml@refs/heads/(main|gh-readonly-queue/main/.+)$

SBOM artifact expiry fallback: Build artifacts expire after 30 days. For
workflow_dispatch out-of-band cuts, add a Syft fallback:

- name: Download SBOM
  id: sbom_artifact
  continue-on-error: true
  uses: actions/download-artifact@...
- name: Install Syft (fallback)
  if: steps.sbom_artifact.outcome == 'failure'
  id: syft
  uses: anchore/sbom-action/download-syft@<SHA> # v0
  with:
    syft-version: v1.44.0
- name: Generate SBOM with Syft (fallback)
  if: steps.sbom_artifact.outcome == 'failure'
  env:
    SYFT_CMD: ${{ steps.syft.outputs.cmd }}
  run: "${SYFT_CMD}" "ghcr.io/.../dakota@${DIGEST}" -o spdx-json=sbom-current/dakota.spdx.json

Use anchore/sbom-action/download-syft (SHA-pinned) instead of curl .../main/install.sh | sh.
The @main install script is a mutable supply-chain input even when the version flag is pinned.

release.yml publish run search must include merge-queue branches (2026-06-07)#

gh run list --branch main only returns runs whose triggering branch was exactly
main. Publish runs triggered by workflow_run from gh-readonly-queue/main/**
(i.e., merge queue builds) are associated with the queue branch, not main, in
the GitHub API. If the promoted :stable SHA came from a merge-queue run, the
--branch main filter silently misses it and release.yml exits with "no
successful publish run found."

Fix: Drop the --branch filter and filter by headBranch` in jq instead:

gh run list \
  --workflow "Publish Bluefin dakota" \
  --status success \
  --limit 100 \
  --json headSha,headBranch,createdAt,databaseId \
  | jq -r --arg sha "$SHA" '
      map(select(
        .headSha == $sha and
        (.headBranch == "main" or (.headBranch | test("^gh-readonly-queue/main/")))
      )) | .[0] // empty'

workflow_dispatch on publish.yml can promote non-main refs to (2026-06-07)#

publish.yml has no branch guard on the promote job. A manual dispatch from a
non-main branch flows through e2e and promotes to :testing, fast-forwarding the
testing branch to an unmerged commit.

Fix: Add a branch guard to the promote job. Since e2e-gate no longer
exists (continuous build model), the guard goes directly on promote:

promote:
  needs: [setup, publish]
  if: >-
    needs.publish.result == 'success' &&
    (github.event_name == 'workflow_run' || github.ref_name == 'main')

workflow_run events are always safe (they trigger from completed main/merge-queue
builds per the trigger filter in publish.yml). Only manual dispatches need the
github.ref_name == 'main' guard.

release.yml manual dispatch TOCTOU (2026-06-07)#

In workflow_dispatch mode with no source_sha, the original code resolved SHA
and digest in two separate skopeo inspect calls. If :stable moved between
them, the release would pair a wrong SHA with a wrong digest.

Fix: One skopeo inspect --format '{{index .Labels "org.opencontainers.image.revision"}} {{.Digest}}'
call extracts both values atomically. Write the digest to $GITHUB_ENV and read
it in the next step — no second skopeo call.

INSPECT=$(skopeo inspect --format \
  '{{index .Labels "org.opencontainers.image.revision"}} {{.Digest}}' \
  docker://ghcr.io/.../dakota:stable)
SHA=$(echo "${INSPECT}" | awk '{print $1}')
STABLE_DIGEST=$(echo "${INSPECT}" | awk '{print $2}')
echo "STABLE_DIGEST=${STABLE_DIGEST}" >> "$GITHUB_ENV"

Full pipeline to promote testing → stable manually:

# 1. Check for testing-only element commits not yet in main
git fetch upstream
git log --oneline upstream/main..upstream/testing -- elements/ files/ patches/
# If any: land them via PR (see "Dep updates on testing not reaching main" above)

# 2. Ensure publish.yml is enabled
gh api repos/projectbluefin/dakota/actions/workflows \
  --jq '.workflows[] | select(.name | contains("Publish")) | "\(.id) \(.state)"'

# 3. Dispatch publish.yml to build :testing from current main
gh workflow run publish.yml --repo projectbluefin/dakota

# 4. Once publish completes, dispatch promotion (pauses for production environment approval)
gh workflow run weekly-testing-promotion.yml --repo projectbluefin/dakota

Step 4 requires approval at: https://github.com/projectbluefin/dakota/deployments

The GitHub release (notes + card + SBOM) is created automatically by
release.yml after every successful publish.yml run — no manual step needed.

check-diff skip silently skips missing variant tags (2026-06-08)#

check-diff compares dakota:testing vs dakota:stable only. If they match,
has_diff=false and the entire promote matrix is skipped — including the
nvidia variant. This means if dakota-nvidia:stable was never created (e.g.,
nvidia :testing didn't exist during the first promotion that set :stable),
it will silently never get set on subsequent runs where the default image hasn't
changed.

How it breaks:

1. First promotion: NVIDIA :testing not found → has_nvidia=false → nvidia skipped
2. Next promotion: NVIDIA :testing now exists, but dakota:testing == dakota:stable
   → has_diff=false → entire promote job skipped → dakota-nvidia:stable never set

Fix (manual): Copy from the matching :testing digest directly:

# Confirm revision matches dakota:stable
skopeo inspect docker://ghcr.io/projectbluefin/dakota:stable \
  | jq '.Labels["org.opencontainers.image.revision"]'
skopeo inspect docker://ghcr.io/projectbluefin/dakota-nvidia:testing \
  | jq '.Labels["org.opencontainers.image.revision"]'

# Get the testing digest
DIGEST=$(skopeo inspect docker://ghcr.io/projectbluefin/dakota-nvidia:testing \
  | jq -r '.Digest')

# Copy to :stable (login with gh auth token first)
GH_TOKEN=$(gh auth token)
skopeo login ghcr.io --username <your-user> --password "$GH_TOKEN"
skopeo copy \
  "docker://ghcr.io/projectbluefin/dakota-nvidia@${DIGEST}" \
  "docker://ghcr.io/projectbluefin/dakota-nvidia:stable"

Underlying bug: check-diff should also detect missing variant stable tags
and set has_diff=true in that case, forcing the promote job to run even when
the default image hasn't changed.

Testing branch fast-forward is idempotent — GitHub API 422 on same SHA (2026-06-08)#

Symptom: publish.yml promote job fails with:

{"message":"Update is not a fast forward",...}
{"message":"Reference already exists",...}

Exit code 1 even though the image was published successfully.

Root cause: The original fast-forward step used a PATCH-then-POST fallback:

1. PATCH refs/heads/testing → GitHub returns 422 "Update is not a fast forward" when
   the ref is already at the target SHA (no-op case)
2. POST fallback → GitHub returns 422 "Reference already exists"

Both fail, causing the step to fail even though nothing needed updating.

Fix: Check the current SHA first; only PATCH or POST when actually needed:

CURRENT_SHA=$(gh api repos/${{ github.repository }}/git/refs/heads/testing \
  --jq .object.sha 2>/dev/null || echo "")
if [ "$CURRENT_SHA" = "$BUILD_SHA" ]; then
  echo "testing branch already at $BUILD_SHA — nothing to do"
elif [ -z "$CURRENT_SHA" ]; then
  gh api repos/${{ github.repository }}/git/refs --method POST \
    --field ref="refs/heads/testing" --field sha="$BUILD_SHA"
else
  gh api repos/${{ github.repository }}/git/refs/heads/testing \
    --method PATCH --field sha="$BUILD_SHA" --field force=false
fi

Merge-queue head_branch is never 'main' — use startsWith guard (2026-06-08)#

When a PR merges via GitHub's merge queue, github.event.workflow_run.head_branch
(and needs.setup.outputs.branch) is gh-readonly-queue/main/pr-N, never main.

Any if: condition that checks branch == 'main' will silently skip for all
merge-queue merges (i.e., every normal PR merge).

Correct pattern:

if: >-
  matrix.image_suffix == '' &&
  (needs.setup.outputs.branch == 'main' ||
   startsWith(needs.setup.outputs.branch, 'gh-readonly-queue/main/'))

/ stream — fully automated, no human gate (2026-06-08)#

The next branch (:next/:btw tags) is a continuously rolling GNOME OS
nightly stream. Junction bumps on next use auto-merge — no human review
required. This is intentional and differs from core junction bumps on main
(which require human review per track-bst-sources.yml).

track-next-junctions.yml schedules nightly junction tracking on the next
branch. PRs it opens get auto-merged once required checks pass.

export/publish jobs must skip storage-service — remote CAS quota too small for GNOME 51 (2026-06-09)#

Symptom: bst export in the publish job fails with:

OutOfSpaceException: Insufficient storage quota
errMsg = "Insufficient storage quota" (buildboxcommon_lrulocalcas.cpp:383)

The blob is ~8.5 GB (GNOME 51 root artifact is significantly larger than GNOME 50).

Root cause: cache.storage-service in the BST config routes the local casd
through cache.projectbluefin.io. The remote server's per-client storage quota
is exceeded when materialising the full artifact for export. Build jobs are fine
because they write blobs incrementally as they are built; export pulls the entire
artifact at once.

Fix (already in generate-bst-ci-config/action.yml):
cache.storage-service is only written when enable-push: true (build jobs).
Export/publish jobs (enable-push: false) use local disk for the casd.
The runner's BTRFS volume has sufficient space for export.

Do not revert this. Any future regression will show this same symptom on
the next/:btw stream, which produces the largest artifacts.

First cold build of next branch will timeout — retrigger until cache warms (2026-06-09)#

The next branch tracks gnome-build-meta master (GNOME 51+). The first build
after branching or a major gnome-build-meta ref bump is a full cold build of
the entire GNOME stack — ~700+ elements. This exceeds the 330-minute GHA timeout.

This is expected and normal. Each run pushes built artifacts to
cache.projectbluefin.io. Simply retrigger the build — each run picks up from
where the previous one left off:

gh workflow run build.yml --repo projectbluefin/dakota --ref next

Typically takes 2–3 runs to warm the full cache. Subsequent builds (after
junction bumps) are incremental (~3–25 min).

Indicator that cache is warm: build jobs complete in <5 minutes — all
artifacts are cache hits and no compilation occurs.

next branch needs manual cherry-picks of main fixes (2026-06-09)#

next is a long-lived parallel branch. Bug fixes merged to main
(e.g., sbom crun fixes, CI improvements) do not automatically land on next.

Before debugging a failure on next, check if the same fix is already on main:

git log upstream/next..upstream/main --oneline -- Justfile .github/

Cherry-pick selectively:

git checkout upstream/next -b fix/next-sync
git cherry-pick <sha1> <sha2> <sha3>
git push upstream fix/next-sync:next

Commits to watch for: any fix(sbom):, fix(ci):, or fix(publish): commits
on main that touch Justfile or .github/.

build only fires on junction bumps — not a guaranteed nightly (2026-06-09)#

build.yml has no schedule: trigger. The next branch builds when:

1. track-next-junctions.yml bumps gnome-build-meta master (20:00 UTC nightly,
   only if upstream advanced that day) → auto-merge PR → merge_group build
2. Manual workflow_dispatch

On days where gnome-build-meta master does not advance, no build fires.
For a guaranteed nightly, a schedule: trigger on next is needed in
build.yml. This is a known gap — track it if builds go stale.

publish.yml must include testing branch in workflow_run.branches (2026-06-10)#

publish.yml originally only listed main, gh-readonly-queue/main/**, next,
and gh-readonly-queue/next/** in workflow_run.branches. Auto-merge tracking
PRs target testing — their builds completed successfully but no image was ever
published. promote-testing-to-main.yml fires on push: branches: [testing] and
immediately does skopeo inspect dakota:testing, which silently failed every time
testing advanced without a prior main publish.

Fix (PR 766): add testing and gh-readonly-queue/testing/** to the
workflow_run.branches filter, extend the setup job if condition, and map
testing branch → testing_tag=testing. Match bluefin/bluefin-lts: every merge
to testing publishes :testing immediately.

track-bst-sources: branch from origin/$BASE_BRANCH, not origin/main (2026-06-10)#

track-bst-sources.yml created auto-merge tracking branches from origin/main
but targeted testing. When main and testing had diverged on workflow files, the
PR diff included those CI changes in reverse — the PR appeared to be deleting
them. PR 764 had 18 commits and would have removed testing from build.yml
triggers and deleted renovate-automerge.yml. It was closed as a corrupted PR.

Corrupted auto-track PR anatomy: CONFLICTING state, 10+ commits, diff shows
CI workflow regressions (removes triggers, deletes workflows). The element file
contents match the base branch — no real update present.

Fix (PR 766): determine BASE_BRANCH before git checkout, stash the
BST-tracked element changes, git checkout -B "$BRANCH" "origin/$BASE_BRANCH",
then git stash pop. The PR diff is now relative to the target branch only.

track-bst-sources: auto-merge silently never set — use --squash not --merge (2026-06-10)#

The repo has allowMergeCommit=false (only squash merges permitted). The
workflow called gh pr merge --auto --merge which hit the || echo ::warning::
fallback — auto-merge was never set on any tracking PR. They sat unmerged
indefinitely with no visible error.

Fix (PR 767): --merge → --squash. The renovate-automerge.yml already
used --squash correctly; track-bst-sources was the gap.

Diagnostic: if a tracking PR has auto-merge null and validate passed, check
allowMergeCommit on the repo before assuming a workflow bug.

permissions: {} at workflow level starves GITHUB_TOKEN for reusable job calls (2026-06-11)#

Setting permissions: {} at the workflow level and then specifying
permissions at the job level does NOT work when the job uses uses: to
call a reusable workflow. GitHub mints the GITHUB_TOKEN at the calling
workflow's top-level scope — job-level permissions: can only restrict, not
expand beyond that ceiling.

Symptom: startup_failure with jobs: [] (zero jobs started) on every
run of a thin caller that uses uses: with its own permissions: block.

Fix: Set the top-level permissions: to the superset of everything any job
in the workflow needs:

# WRONG — starves the token; jobs cannot escalate beyond {}
permissions: {}

jobs:
  promote:
    permissions:
      contents: write
      pull-requests: write
    uses: org/actions/.github/workflows/reusable.yml@SHA

# CORRECT — top-level is the budget; job-level can reduce but not expand
permissions:
  contents: write
  packages: read
  pull-requests: write
  issues: write

jobs:
  promote:
    uses: org/actions/.github/workflows/reusable.yml@SHA

Affected workflows fixed 2026-06-11: promote-testing-to-main.yml (#796),
execute-release.yml (#798).

pull_request: closed trigger causes startup_failure for all non-promotion merges (2026-06-11)#

When a workflow uses on: pull_request: types: [closed] and ALL jobs have
if: conditions that evaluate to false for non-promotion PRs, GitHub reports
the workflow run as startup_failure instead of a clean skip. This produces
alarming noise in every PR merge and masks real failures.

Symptom: execute-release.yml showed startup_failure on every single PR
merged to main from the day it was introduced — 25+ runs, none successful,
all with jobs: [].

Correct pattern (from bluefin-lts): Use push: branches: main +
workflow_dispatch, then add a lightweight check-trigger job that reads
the squash-merge commit message:

on:
  push:
    branches: [main]
  workflow_dispatch:

jobs:
  check-trigger:
    runs-on: ubuntu-latest
    outputs:
      is-promotion: ${{ steps.check.outputs.is-promotion }}
    steps:
      - id: check
        env:
          COMMIT_MSG: ${{ github.event.head_commit.message }}
          EVENT_NAME: ${{ github.event_name }}
        run: |
          if [ "$EVENT_NAME" = "workflow_dispatch" ]; then
            echo "is-promotion=true" >> "$GITHUB_OUTPUT"
          elif echo "$COMMIT_MSG" | grep -q "^ci: promote testing images to stable"; then
            echo "is-promotion=true" >> "$GITHUB_OUTPUT"
          else
            echo "is-promotion=false" >> "$GITHUB_OUTPUT"
          fi

  execute:
    needs: [check-trigger]
    if: needs.check-trigger.outputs.is-promotion == 'true'
    uses: ...

When is-promotion=false, check-trigger succeeds cleanly and subsequent
jobs are skipped — no startup_failure.

Fixed: execute-release.yml PR #800, 2026-06-11.

CODEOWNERS: no-owner override for auto-managed files (2026-06-11)#

Files auto-managed by a bot (e.g. elements/bluefin/common.bst bumped by
mergeraptor on every common release) should not trigger code-owner review
requests. Add a no-owner line for the specific file above the catch-all
path rule — CODEOWNERS is evaluated top-to-bottom and the first match wins:

# Auto-managed by mergeraptor — no review required
elements/bluefin/common.bst

# Everything else in elements/ needs a maintainer review
elements/ @projectbluefin/maintainers

Also add the bot to bypass_pull_request_allowances in the repo's branch
protection ruleset so it can satisfy the required_approving_review_count
without a human approval. Without this, auto-merge is set but never clears.

Fixed: PR #807, 2026-06-11.

CODEOWNERS: use team slugs, not individual handles (2026-06-11)#

Individual @handle entries in CODEOWNERS mean:

• New team members are never auto-requested for review
• Departed maintainers keep getting pinged
• Team membership changes require a CODEOWNERS PR

Fix: Use @org/team-slug instead:

# WRONG
* @castrojo @p5 @m2Giles @tulilirockz

# CORRECT
* @projectbluefin/maintainers

Fixed: PR #796, 2026-06-11.

Promotion PR noise: suppress CodeRabbit with @coderabbitai ignore (2026-06-11)#

CodeRabbit posts review summaries on every PR, including automated promotion
PRs that only touch .github/release-state.yaml. To suppress it, add this
HTML comment as the first line of the PR body:

<!-- @coderabbitai ignore -->

Added to reusable-promote.yml in projectbluefin/actions (commit f5cd16ce).

Promotion PR body: include release context for maintainer decision-making (2026-06-11)#

The old promotion PR body was a raw YAML dump. Maintainers had no context for
deciding whether to merge. The rich body template now includes:

Change indicator ⬆ appears when a junction ref changed since the last
promotion.

Location: reusable-promote.yml "Open or update promotion PR" step in
projectbluefin/actions.

GitHub Release body limit: 125k characters (2026-06-11)#

gh release create --notes-file release-notes.md fails with HTTP 422 if the
body exceeds GitHub's hard limit of 125,000 characters:

HTTP 422: Validation Failed
body is too long (maximum is 125000 characters)

The release notes generator in reusable-release.yml can produce bodies larger
than this limit when the SBOM diff or changelog is long (e.g. after 12+ days
between stable releases).

Fixed in projectbluefin/actions#191: reusable-release.yml now hard-caps
the release body at 120,000 characters with a … trailer before calling
gh release create. No manual intervention needed.

sign-and-publish reusable action: cert identity regexp must include consuming repo (2026-06-11)#

The sign-and-publish action in projectbluefin/actions has a default
certificate-identity-regexp that is repo-specific. If the pinned SHA
pre-dates when dakota was added to that regexp, every publish.yml run
fails at the cosign verification step — 100% failure rate.

Symptom: publish run fails at cosign verify with a cert identity mismatch.
All :testing builds stop. May appear as "65%" failure rate if some older
cached :testing images still serve.

Fix: Bump the projectbluefin/actions SHA to a commit that includes
the repo name in the default regexp. Or pass an explicit input:

- uses: projectbluefin/actions/.github/actions/sign-and-publish@<SHA>
  with:
    cosign_identity_regexp: >-
      ^https://github\.com/projectbluefin/(dakota|actions)/\.github/workflows/

Root cause (PR #792, 2026-06-11): Actions SHA 3025b5d31f34 excluded
dakota; bumping to 2a09e72e9be1 (actions#166) fixed it.

Rule: after bumping any projectbluefin/actions SHA, verify the first
publish run succeeds before assuming the bump is clean.

cliff.toml required at repo root for structured release notes (2026-06-11)#

reusable-release.yml calls git-cliff via the generate-release-notes
step. Without cliff.toml at the repo root, it falls back to a raw
git log heredoc — no commit grouping, no filtering, no section headers.

Add cliff.toml adapted from projectbluefin/common/cliff.toml with
Conventional Commits parser. Dakota-specific note: omit the
chore: promote skip rule. Dakota uses OCI digest promotion via the
execute-release.yml commit-message gate — there are no squash promotion
commits in the git history that need filtering out.

Key sections in cliff.toml:

[git]
conventional_commits = true
filter_unconventional = false
tag_pattern = "v[0-9].*"
skip_tags = ""

[git.commit_parsers]
# do NOT add: { message = "^chore: promote", skip = true }
# Dakota promotions don't produce commits like this

Added in PR #793, 2026-06-11. Closes projectbluefin/common#609.

gh pr merge --auto does NOT honour bypass_pull_request_allowances (2026-06-12)#

gh pr merge "$PR_URL" --auto --squash enables GitHub's auto-merge queue.
The queue evaluates branch-protection conditions using GitHub's internal process
and does not honour bypass_pull_request_allowances. So a bot app in the
bypass list that enables auto-merge still sees the PR sit open waiting for a
human approval that will never arrive automatically.

Only direct merges (without --auto) use the bypass.

Symptom: all auto-merge group PRs from track-bst-sources.yml (common,
distrobox, brew, shell extensions, etc.) were sitting open indefinitely despite
required_approving_count: 1 and the bot in bypass_pull_request_allowances.

Fix (PR #820): remove --auto from the merge call. Since there are no
required status checks, the direct merge completes immediately on PR creation.

# ✗ — queued auto-merge, bypass ignored
gh pr merge "$PR_URL" --auto --squash

# ✓ — direct merge, bypass honoured
gh pr merge "$PR_URL" --squash

Rule: use --auto only when you want to wait for required CI checks to pass
AND the merging actor has no bypass. For bypass actors merging bot-managed PRs
with no required checks, drop --auto.

Caller-level permissions: must be a superset of all reusable workflow job permissions (2026-06-12)#

When a thin-caller workflow calls a reusable workflow via uses:, the
caller's top-level permissions: block caps what GITHUB_TOKEN can do in
every job inside the reusable. If the reusable's job needs packages: read or
actions: read and the caller only declares contents: write, those scopes
are silently restricted to none — producing startup_failure with jobs: [].

Symptom: promote-testing-to-main.yml had startup_failure on every run
after the thin-caller migration (PR #811). Missing packages: read (for GHCR
digest lookups) and actions: read (for workflow-run status checks) were not
in the caller's permissions: block.

Fix (PR #817): declare every scope the reusable's jobs need at the
caller's top level:

permissions:
  contents: write # push squash promotion branch
  pull-requests: write # create / update / auto-merge the promotion PR
  issues: write # open / close failure-tracking issues
  packages: read # read image digests in release-gate checks
  actions: read # inspect workflow-run statuses in release-gate

Rule when writing thin callers: read the reusable workflow's job-level
permissions: blocks and make the caller's top-level permissions: a strict
superset of the union of all of them.

SHA-pinning a reusable that itself has nested SHA-pinned calls — inner SHA must still exist (2026-06-12)#

When you SHA-pin a reusable workflow (e.g.
projectbluefin/actions/.github/workflows/reusable-promote-squash.yml@<sha>),
GitHub validates the full call chain at startup — including any
uses: references inside the pinned reusable. If the pinned reusable
internally calls another workflow at a now-deleted SHA, the calling workflow
fails with:

failed to parse workflow: error parsing called workflow
--> "projectbluefin/actions/.github/workflows/reusable-release-gate.yml@<dead-sha>"
: failed to fetch workflow: workflow was not found.

This manifests as startup_failure on the outer caller — the error is not
visible without running the workflow and reading the dispatch HTTP response.

Cause in this session: the bluefin SHA for reusable-promote-squash.yml
(5f3cab) internally called reusable-release-gate.yml@5f8abb which had been
removed from the actions repo. The original dakota SHA (6c2278) internally
calls reusable-release-gate.yml@v1 (the managed tag), which remains valid.

Fix (PR #819): revert to the SHA whose nested calls use @v1 tags rather
than pinned SHAs for inner dependencies.

Rule: when picking a SHA to pin for a reusable workflow, verify that its
own nested uses: references are either @v1/managed-tags or still-live
SHAs. Prefer the version that uses managed tags internally — those age better.

Testing→main promotion pipeline — full cycle and failure modes (2026-06-12)#

How the cycle works (bluefin model)#

Renovate PR → testing branch (automerges when build CI passes)
    → push to testing → promote-testing-to-main fires
    → squash PR: auto/promote-testing-to-main → main
    → maintainer merges
    → execute-release fires (commit msg "ci: promote testing images to stable")
    → :testing retagged as :stable
    → push to main → sync-main-to-testing fires
    → main fast-forwarded into testing (testing == main again)
    → next Renovate cycle begins

Three invariants that must all hold#

1. baseBranchPatterns: ["testing"] in renovate.json5 — Renovate must target
   testing, not main. With baseBranchPatterns: ["main"], testing is a dead
   branch: nothing ever lands there, the promote workflow finds nothing to squash,
   and :stable never updates.

2. sync-main-to-testing.yml must exist — after each squash-merge promotion, the
   squash commit lands on main but not testing. Without this workflow, testing
   falls permanently behind main. The next promote run finds diverged trees (so
   sync_needed=true), but the squash produces nothing staged → git commit exits 1.

3. pr-triage.yml must exempt renovate/* PRs targeting testing — the triage
   workflow blocks all PRs not targeting main. Without an exemption, Renovate PRs
   to testing are immediately blocked and cannot automerge.

The empty-squash crash (known bug in reusable-promote-squash)#

When testing is behind main with no unique content:

• git merge --squash origin/testing says "Already up to date"
• Nothing is staged
• git commit exits 1 → job fails with misleading error

This is fixed by projectbluefin/actions#218 (adds git diff --cached --quiet guard
before git commit). In steady state (sync-main-to-testing present), this edge case
doesn't occur because testing == main after each sync, and the next promote run gets
sync_needed=false cleanly. The fix is defence-in-depth.

Root cause of 2026-06-11/12 breakage#

PR #741 changed baseBranchPatterns from ["testing"] to ["main"] to work around
the triage gate — but without also adding sync-main-to-testing.yml or exempting
Renovate from the gate. After promotion #797 (June 10), the cycle broke permanently:

• testing fell 20+ commits behind main (no sync workflow)
• Renovate stopped feeding testing (wrong base branch)
• Promote workflow crashed nightly (empty squash)
• :stable stopped updating

Fix: PR #822 (dakota) + PR #218 (actions).

gh pr merge --auto also fails when target branch has NO branch protection (2026-06-13)#

The --auto lesson above covers the bypass case (protection exists but bypass
not honoured). There is a second, distinct failure mode: if the target branch
has zero branch protection rules (no ruleset, no classic protection),
gh pr merge --auto fails immediately with:

GraphQL: Pull request Protected branch rules not configured for this branch
        (enablePullRequestAutoMerge)

testing has no branch protection by design. Any automerge workflow targeting
testing with --auto will always fail. The fix (applied in projectbluefin/actions
renovate-automerge.yml @v1) is to drop --auto entirely — CI success is
already guaranteed by the workflow_run trigger condition.

Two distinct --auto failure modes:

Diagnosis: check gh api repos/<owner>/<repo>/branches/<branch> --jq '.protected'.
If false, drop --auto. If true, check whether the token is in bypass_actors.

workflow_run always uses the DEFAULT BRANCH's workflow file (2026-06-13)#

When a workflow has on: workflow_run, GitHub runs it from the repository's
default branch — not from the branch that triggered the upstream workflow run.

Consequence for automerge fixes: if renovate-automerge.yml is fixed on a
feature branch or testing but the fix hasn't landed on main (the default
branch), every new workflow_run trigger still runs the old broken version from
main. The fix takes effect only when it merges to main.

Implication: fixes to workflow_run-triggered workflows that land on testing
(via a Renovate-style staging flow) are effectively inert until the promote PR
merges them to main.

Internal projectbluefin/ actions: use @v1 managed tag, not SHA pins (2026-06-13)#

SHA-pinning internal org actions (projectbluefin/actions) is
counter-productive and was the root cause of the June 13 automerge outage:

• The --auto bug was committed on June 7 at SHA fcd2a6bac15f
• Every consumer (dakota, bluefin, bluefin-lts, common) pinned different
  intermediate SHAs, all carrying the broken --auto
• Fixes require N separate Renovate bump PRs — one per consumer — each
  lagging by hours or days
• main and testing diverged to different SHAs, creating split-brain

AGENTS.md already states the policy:

projectbluefin/ refs (@v1, @main) are intentional managed tags and are exempted.

Use @v1 — it moves forward with every non-breaking fix and is maintained by
the org. @v1.0.0 and @v1.1.0 are static point-release tags if you need
a pinned version.

# ✗ — SHA pin, breaks propagation; 7 different SHAs across 10 files
uses: projectbluefin/actions/bootc-build/setup-runner@2a09e72e... # v1.1.0

# ✓ — managed tag, fixes propagate instantly
uses: projectbluefin/actions/bootc-build/setup-runner@v1

Enforcement: no-sha-pins-for-internal-actions pre-commit hook blocks any
future projectbluefin/.*@<sha> commits.

External actions (actions/checkout, taiki-e/install-action, etc.) remain
SHA-pinned — that policy is unchanged and correct.

build.yml push trigger must include testing for :testing images (2026-06-13)#

build.yml had push: branches: [main, next] — testing was missing.
publish.yml already listed testing in its workflow_run.branches filter
and had logic to publish :testing on testing-branch builds, but that path
was dead because build.yml never triggered on push to testing.

Result: :testing images were never updated by Renovate merges to testing.
The promote PR was always building from stale image content.

Fix (PR #830): add testing to build.yml's push trigger. The build job
runs on event_name != 'pull_request', so push-to-testing fires the full build.
BST artifact cache steps remain gated on merge_group || schedule || workflow_dispatch
(intentional quota management) — they skip for plain pushes, which is fine.

publish.yml: 4-job pipeline after speed-up refactor (2026-06-12)#

publish.yml was restructured to remove three major bottlenecks. New job graph:

setup → publish-image → boot-check → promote (critical path to :testing: ~46 min)
              ├──────→ smoke (observational, ~90 min, does NOT block promote)
              └──────→ publish-sbom (runs in parallel, does NOT block promote)

Job renames / splits:

• publish renamed to publish-image — exports OCI, pushes :$sha, signs. No SBOM.
• publish-sbom (new) — depends on publish-image, runs in parallel with promote.
  Contains: SBOM generation, artifact upload, oras attach, cosign sign SBOM.
• promote — depends on [setup, publish-image, boot-check] only. Neither SBOM nor smoke are in needs.

Critical: Do NOT add smoke to promote.needs. Smoke is observational (~90 min) and was previously
blocking promote despite the if: condition allowing pass or fail. Removing it from needs saves ~85 min
on the critical path to :testing. Fixed 2026-06-16 (#890).

skopeo copy in promote (P1):
The old podman pull → tag → push pattern transferred the full 8.5 GB image
round-trip for each re-tag. Replace with:

skopeo copy \
  --preserve-digests \
  --src-creds "$GH_ACTOR:$GH_TOKEN" \
  --dest-creds "$GH_ACTOR:$GH_TOKEN" \
  "docker://${IMAGE}:${BUILD_SHA}" \
  "docker://${IMAGE}:${TESTING_TAG}"

--preserve-digests is mandatory — it keeps the promoted tag pointing at
the same manifest digest that cosign signed. Omitting it causes GHCR to re-encode
layers and produce a new digest that breaks the signature chain.
skopeo is pre-installed on ubuntu-24.04 runners — no install step needed.

SBOM pip cache (P3):
buildstream-sbom is installed from a GitLab git URL every run (3–8 min with
retries). Cache the pip wheel at ~/.cache/pip keyed to the pinned commit SHA:

- uses: actions/cache@<SHA>
  with:
    path: ~/.cache/pip
    key: pip-sbom-<pinned-commit-sha>
    restore-keys: pip-sbom-

Mount into the bst2 container via -v "${HOME}/.cache/pip:/root/.cache/pip:rw"
in the just sbom podman run call. Update the cache key when the pin is bumped.

buildah replaces squash-all in just export (P6):
podman build --squash-all re-encoded the entire 8.5 GB image for a 2-line
sed edit to /usr/lib/os-release. Replace with:

CONTAINER=$(buildah from "$IMAGE_ID")
MOUNT=$(buildah mount "$CONTAINER")
sed -i "s/^VERSION_ID=.*/VERSION_ID=\"${DATE_TAG}\"/" "$MOUNT/usr/lib/os-release"
sed -i "s/^IMAGE_VERSION=.*/IMAGE_VERSION=\"${DATE_TAG}\"/" "$MOUNT/usr/lib/os-release"
buildah config --label "org.opencontainers.image.created=..." "$CONTAINER"
buildah commit --rm "$CONTAINER" "${FINAL_NAME}:${FINAL_TAG}"

buildah commit (no --squash) appends a tiny (~1 KB) delta layer. chunka's
BST path calls podman image mount which returns a merged overlayfs view
regardless of layer count — multi-layer input is transparent to chunkah.
buildah is pre-installed by setup-runner@v1 (resolute package).

digest re-derivation in publish-sbom:
publish-sbom needs the image digest for oras attach but GHA matrix job
outputs are fragile. Re-derive it with skopeo inspect instead:

DIGEST=$(skopeo inspect \
  --creds "$GH_ACTOR:$GH_TOKEN" \
  "docker://${IMAGE}:${BUILD_SHA}" \
  | python3 -c "import json,sys; print(json.load(sys.stdin)['Digest'])")

No inter-job output wiring needed. The image was pushed by publish-image and
is immediately available in GHCR before publish-sbom starts.

cache-warm cron: Mon–Fri (P4):
Changed from 0 6 * * 1,4 (Mon/Thu) to 0 6 * * 1-5 (Mon–Fri).
A junction ref bump on Tuesday left the CAS cold for 3 days, causing build.yml
to timeout at 360 min. Daily warming caps the cold window at 1 day.

Promotion PR: force-push dismisses approvals even when diff is unchanged (2026-06-13)#

When main advances (e.g. Renovate merges) while a promotion PR has a
maintainer approval, the promote workflow was rebuilding the squash branch
and force-pushing — even though the effective diff against main was
identical. GitHub dismisses approvals on any force-push regardless of
content. The first approver had to re-approve on every unrelated commit
landing on main, indefinitely.

Root cause: the rebuild step always ran git push --force without checking
whether the new squash content differed from the existing promotion branch.

Fix (actions#225): tree-identity check before force-pushing:

NEW_TREE=$(git write-tree)
EXISTING_TREE=$(git rev-parse "origin/${PROMOTION_BRANCH}^{tree}" 2>/dev/null || echo "")
if [ "$NEW_TREE" = "$EXISTING_TREE" ] && [ -n "$EXISTING_TREE" ]; then
  echo "promoted=skipped" # skip push — approvals preserved
else
  git commit && git push --force
  echo "promoted=true" # content changed — new approval required (correct)
fi

promoted=skipped passes all downstream != 'false' guards — PR body and
gate section still refresh. Only the push is skipped.

Rule: Never force-push a promotion branch when the squash tree is unchanged.
git write-tree before committing gives the tree hash of staged content without
creating a commit.

Promotion PR: force-push clears reviewRequests — maintainers team not notified (2026-06-13)#

After a force-push, GitHub clears all pending reviewer requests. reviewRequests
becomes []. The team doesn't know re-review is needed; the PR sits blocked
with no active requests.

Fix (actions#226): re-request the maintainers team after any force-push:

if [ "${{ steps.rebuild.outputs.promoted }}" = "true" ]; then
  gh pr edit "$PR_NUMBER" \
    --add-reviewer "${{ github.repository_owner }}/maintainers" 2>/dev/null
fi

Skip on promoted=skipped — approvals are preserved so no re-request is
needed. Re-requesting when nothing changed would spam reviewers with no new
content to review.

Both fixes are in reusable-promote-squash.yml@v1 and apply automatically
to bluefin, bluefin-lts, and dakota.

buildah in export recipe — do not use without confirming availability#

Context: f8b80d4 switched the just export recipe from podman build --squash-all to
buildah from + buildah mount + buildah commit to save ~35–50 min by avoiding full image re-encode.

Bug (dakota#841): This broke two things:

1. Boot failure on real hardware — the multi-layer buildah commit output differs from the
   single flat layer produced by --squash-all. chunka's composefs xattr injection expects to
   rechunk a flat image; multi-layer input produces a different composefs tree that fails to mount
   at boot.
2. Local/Argo builds broken — quay.io/podman/stable (used by just build and the Argo
   dakota-bst WorkflowTemplate) does not include buildah. GitHub Actions ubuntu-24.04 has
   buildah pre-installed, so GHCR builds succeeded while local/Argo builds errored with
   buildah: command not found (exit 127).

Fix: Reverted to podman build --squash-all in PR fixing #841.

Rule: Any just export change that introduces a tool dependency beyond podman must be
verified in both environments:

• quay.io/podman/stable:latest (Argo pipeline image)
• ubuntu-24.04 GitHub Actions runner
  If the tool is only available on ubuntu-24.04, the Justfile recipe must install it explicitly
  (e.g. dnf install -y buildah) or the approach must avoid it entirely.

chunka overlay dirs must land on BTRFS, not root FS (2026-06-13)#

Symptom: Chunkify image layers step fails with:

No space left on device

The GitHub Actions runner terminates mid-step with a System.IO.IOException in the
runner diagnostic log. The step is killed before sudo umount can run, leaving
orphaned overlay mounts (cleaned up when the ephemeral runner terminates).

Root cause: chunka@v1 (BST path) creates three overlay work dirs — UPPER,
WORK, MERGED — in /var/tmp. setup-runner mounts a BTRFS loopback over
/var/lib/containers using loopback-free: "1", leaving only ~1 GB free on the
root filesystem. inject-xattrs.py sets user.component xattrs on every path in
files/fakecap-manifest.tsv (700K+ entries). Each setxattr call on a regular
file in an overlayfs triggers a copy-up: the entire file is copied to UPPER.
Copy-ups from a full OS image easily exceed 1 GB, exhausting the root FS.

Fix (2026-06-13): Fixed centrally in projectbluefin/actions chunka@v1
(bootc-build/chunka/action.yml). At runtime, the action now picks the directory
with the most free space from [/var/lib/containers, /var/tmp]:

_OVERLAY_TMPDIR="/var/tmp"
for _candidate in /var/lib/containers /var/tmp; do
  if [[ -d "$_candidate" ]]; then
    _free=$(df --output=avail "$_candidate" 2>/dev/null | tail -1 || echo 0)
    _best=$(df --output=avail "$_OVERLAY_TMPDIR" 2>/dev/null | tail -1 || echo 0)
    if (( _free > _best )); then _OVERLAY_TMPDIR="$_candidate"; fi
  fi
done
UPPER=$(mktemp -d -p "$_OVERLAY_TMPDIR")
WORK=$(mktemp -d -p "$_OVERLAY_TMPDIR")
MERGED=$(mktemp -d -p "$_OVERLAY_TMPDIR")

On CI runners with setup-runner btrfs, /var/lib/containers wins (~49 GB).
On local dev machines /var/tmp is the fallback. The action also logs the chosen
dir and available space for future diagnosis.

The same fix was applied to the chunkify recipe in the dakota Justfile
(used by just build for local dev).

The fix is in @v1 — no dakota workflow changes required. All callers of
chunka@v1 (default and nvidia variants across all branches) get the fix
automatically.

Do not add a /var/tmp bind-mount workaround to individual workflows. The fix
belongs in the action, not scattered across consumers.

actions/cache does not create the cache directory on a cold miss — podman bind-mounts fail (2026-06-13)#

actions/cache only restores an existing archive; on a cache miss it does
nothing and leaves the target path absent. If a subsequent podman run uses
bind mounts such as -v "${HOME}/.cache/pip:/root/.cache/pip:rw" or
-v "${HOME}/.cache/buildstream:/root/.cache/buildstream:rw" and the host-side
directory does not exist, podman exits 125 (container failed to start)
before any command runs.

Error: statfs /home/runner/.cache/buildstream: no such file or directory
error: recipe `sbom` failed with exit code 125

Fix: mkdir -p every host cache directory in the Justfile recipe
immediately before the podman run, not in the workflow step. This makes the
fix unconditional regardless of where just sbom is invoked:

mkdir -p "${HOME}/.cache/buildstream" "${HOME}/.cache/pip"
podman run --rm ... \
  -v "${HOME}/.cache/buildstream:/root/.cache/buildstream:rw" \
  -v "${HOME}/.cache/pip:/root/.cache/pip:rw" ...

Rule: Any podman run -v HOST_PATH:... where HOST_PATH is a cache
directory that may not pre-exist must be preceded by mkdir -p HOST_PATH.
Never rely on actions/cache to guarantee the directory exists.

Boot-check gate: inline QEMU boot vs AT-SPI smoke (2026-06-13)#

The testsuite smoke suite runs AT-SPI / GNOME Settings accessibility
tests that take 80+ minutes in a VM and fail on timing sensitivity
in VMs, not on real image defects. Using it as a hard promote gate
blocks :testing on every merge without catching real regressions
(boot failures, composefs xattr breakage are caught by user reports,
not AT-SPI tests).

Fixed in PR #849 / closes #850:

publish.yml now has two separate jobs:

The promote job gates on boot-check.result == 'success'. Smoke
runs in parallel for signal and does NOT appear in promote.needs —
removing it from needs is what makes it truly non-blocking (an if:
condition alone is insufficient; the job still waits if the dep is in needs).

Rule: The per-merge gate should always be a deterministic boot
check (SSH reachable + GDM not-crashed). The full AT-SPI suite belongs
in the weekly pre-stable gate, not the per-merge pre-testing gate.

GDM headless caveat: QEMU runs with -display none (no display
device). GDM will be inactive in this environment — that is expected
behavior, not a regression. The health check uses
systemctl show gdm.service --property=ActiveState and fails only if
the state is failed (GDM crashed). inactive / activating are
both acceptable. Using systemctl is-active gdm.service was wrong:
it exits 3 on inactive, which triggers set -e and blocks every
promote run regardless of image health.

Observational reusable-workflow jobs must be split out of publish.yml (2026-06-16)#

continue-on-error is not supported on jobs that call a reusable workflow
via uses:. That means an observational suite like smoke still makes the
parent workflow red if it lives inside publish.yml, even when promote no
longer depends on it.

Symptom: publish pipeline lands :testing successfully, but
Publish Bluefin dakota still ends red because the observational smoke job
fails inside the same workflow run.

Fix: move observational reusable-workflow jobs into a separate workflow
triggered by workflow_run from the successful publish pipeline. Keep the
hard gate (boot-check) in publish.yml; keep the flaky/slow signal in the
follow-up workflow.

Rule: If a GitHub Actions job is both (a) observational and
(b) implemented as a reusable-workflow call, it does not belong in the
critical publish workflow.

OSTREE_PATH in boot-check kernel args must come from the BLS entry (2026-06-13)#

When constructing QEMU kernel args for an inline boot check, the
ostree= kernel argument requires the path in the format:

/ostree/boot.1/default/TREEHASH/N

where TREEHASH is the ostree commit SHA — a completely different
hash from the deploy directory name (/ostree/deploy/default/deploy/SHA.N).
Using the deploy directory path as the ostree= argument causes the initrd
to fail to switch-root and the VM hangs silently.

Fix: Read the exact path from the BLS (Boot Loader Specification)
entry on the boot partition (p2). The entry already contains the correct
ostree= value that the real bootloader would use:

sudo mkdir -p /mnt_boot
sudo mount "${LOOP}p2" /mnt_boot
OSTREE_PATH=$(sudo grep -rh '^options' /mnt_boot/loader/entries/*.conf 2>/dev/null \
  | grep -o 'ostree=[^ ]*' | head -1 | sed 's/ostree=//')
sudo umount /mnt_boot

Also: always detach the loopback device after unmounting the image
before handing disk.raw to QEMU. Export the loop device name as a step
output and run sudo losetup -d "${LOOP}" after umount. Leaving the loop
device attached while QEMU holds the file open is a resource leak.

Disk partition layout from bootc (systemd-boot, x86-64):

• p1 = BIOS boot (1 MiB, grub fallback — no filesystem)
• p2 = EFI System / xbootldr (512 MiB, FAT32, BLS entries at loader/entries/*.conf)
• p3 = Linux root (xfs, ostree deployments live here)

Mount p2 to read BLS entries (it is the EFI partition, not a separate /boot).
Mount p3 to find the ostree deployment directory.

testing branch divergence breaks Sync main → testing permanently (2026-06-14)#

reusable-sync-branches.yml uses git merge. When testing has commits
main doesn't (diverged), the merge exits 1 and every subsequent push to
main re-triggers the same failure — the pipeline is stuck until a human
manually resets testing.

How divergence happens: Renovate PRs land on testing (digest bumps) while
human PRs land on main touching the same files (publish.yml, Justfile).
The two branches accumulate incompatible histories on the same paths.

Emergency reset (API — no local clone needed):

MAIN_SHA=$(gh api repos/projectbluefin/dakota/branches/main --jq '.commit.sha')
gh api repos/projectbluefin/dakota/git/refs/heads/testing \
  -X PATCH --field sha="$MAIN_SHA" --field force=true

Systemic fix: projectbluefin/actions PR #237 adds divergence detection to
reusable-sync-branches.yml. When ahead > 0, force-reset instead of merge:

AHEAD=$(git rev-list --count "origin/main..origin/testing")
if [ "$AHEAD" -gt 0 ]; then
  git reset --hard origin/main && git push --force origin testing
else
  git merge origin/main ...
fi

Safe: all testing-only commits are Renovate digests that Renovate recreates automatically.

Diagnosis commands:

# Check branch status
gh api repos/projectbluefin/dakota/compare/testing...main \
  --jq '{ahead_by:.ahead_by, behind_by:.behind_by, status:.status}'
# List last sync run results
gh run list --repo projectbluefin/dakota \
  --workflow 'Sync main → testing' --limit 5 \
  --json conclusion,displayTitle --jq '.[] | "\(.conclusion) \(.displayTitle[:50])"'

pr-triage.yml approval said "auto-merge eligible" but never enabled it (2026-06-14)#

The Approved — clear label step removed pr/needs-review and posted
"Auto-merge is now eligible" but never called gh pr merge --auto.
PRs sat approved indefinitely.

Fixed in PR #858:

1. After approval: gh pr merge "$PR_URL" --auto --squash
2. After approval: gh pr update-branch "$PR_URL" (brings branch current so CI runs)
3. New pr-autoupdate.yml fires on every push to main, calls gh pr update-branch
   on all BEHIND PRs targeting main (skips Renovate/Mergeraptor bots)

Also required: validate must be in branch protection required status checks
so --auto waits for CI before merging, not just for review approval.

# Add validate as required check (branch protection API)
gh api repos/projectbluefin/dakota/branches/main/protection \
  --method PUT \
  --input - << 'JSON'
{
  "required_status_checks": {"strict": false, "checks": [{"context": "validate", "app_id": -1}]},
  "enforce_admins": false,
  "required_pull_request_reviews": {
    "dismiss_stale_reviews": true, "require_code_owner_reviews": true,
    "required_approving_review_count": 1
  },
  "restrictions": null
}
JSON

Rule: never post "auto-merge is eligible" in a comment without also calling
gh pr merge --auto. The comment is a lie otherwise.

bootc install to-disk --via-loopback fails with "No root filesystem specified" (2026-06-14)#

Newer bootc no longer defaults the root filesystem type. Without
[install.filesystem.root] type = "xfs" in the image's install config,
bootc install to-disk exits 1 with error: Installing to disk: No root filesystem specified.

Fix: add to files/bootc-install/00-defaults.toml:

[install.filesystem.root]
type = "xfs"

No behaviour change for users — xfs was always the implicit default. This makes it explicit.

bootc install to-disk correct approach for CI loop devices (2026-06-17)#

Use --via-loopback with the raw file path, and let the image's own bootc
install config provide the xfs rootfs + systemd bootloader defaults.

Working reference: projectbluefin/testsuite/.github/actions/gnome-e2e/action.yml
Source docs: https://github.com/bootc-dev/bootc/blob/main/docs/src/bootc-install.md

fallocate -l 30G disk.raw

sudo podman run --rm --privileged --pid=host \
  --security-opt label=type:unconfined_t \
  -v /dev:/dev \
  -v /var/lib/containers:/var/lib/containers \
  -v "$(pwd):/data" \
  "${IMAGE}" bootc install to-disk \
    --via-loopback /data/disk.raw \
    --wipe \
  || echo "bootc install exited $? (bootupd expected — continuing)"

# After container exits, attach with -P so the kernel scans the partition
# table and creates loop0p1/p2/p3 nodes on the host.
LOOP=$(sudo losetup -f --show -P disk.raw)
echo "BOOT_CHECK_LOOP=${LOOP}" >> "$GITHUB_ENV"
sudo udevadm settle --timeout=30 2>/dev/null || true

if ! sudo blkid "${LOOP}p3" &>/dev/null; then
  echo "ERROR: bootc install did not create a filesystem on ${LOOP}p3"
  sudo fdisk -l "${LOOP}" 2>&1 || true
  exit 1
fi

Dakota already ships the real install defaults in files/bootc-install/00-defaults.toml:

[install]
bootloader = "systemd"

[install.filesystem.root]
type = "xfs"

Why this shape matters: --via-loopback keeps bootc in control of the loop
lifecycle, avoiding host-side partition-node races. But the workflow should not
re-specify --generic-image or --filesystem xfs ad hoc — that drifted away
from the working testsuite flow and reintroduced Creating rootfs: No such file or directory (os error 2) in Dakota publish boot-checks on 2026-06-17.

Do NOT:

• Pre-create a host loop device and pass it as a block device path — bootc's
  internal sfdisk + mkfs races the host udevd, causing ENOENT on p3.
• Use --wipe with a pre-created host loop — wipes the partition table,
  removing the nodes you just created.
• Pre-partition with sfdisk before running bootc — bootc refuses with
  "Detected existing partitions".
• Add --generic-image or --filesystem xfs back into the CI boot-check
  command; Dakota already gets xfs/systemd from image install config.

Note: The boot-check gate never passed from PR #849 (2026-06-13) through
PR #895 (2026-06-16) due to iterating on the wrong approach. The first stable
shape was --via-loopback; the 2026-06-17 regression came from drifting away
from the working testsuite invocation, not from a need to go back to host-side
loop handling.

multi-user.target timing race in boot-check (2026-06-19)#

Symptom: boot-check fails immediately after SSH becomes reachable with
exit code 3 from systemctl is-active multi-user.target, even though the
image boots and SSH works.

Cause: systemctl is-active exits 3 for both inactive and activating.
SSH becomes reachable (sshd started) while systemd is still processing the rest
of multi-user.target's dependency graph. The check fires before the target
finishes activating.

Fix: Replace the single is-active call with a retry loop:

for _i in $(seq 1 6); do
  if "${SSH[@]}" sudo systemctl is-active multi-user.target 2>/dev/null; then
    break
  fi
  [[ ${_i} -lt 6 ]] || { echo "ERROR: multi-user.target not active after 60s"; exit 1; }
  echo " not yet active (attempt ${_i}/6), retrying in 10s…"
  sleep 10
done

Log trap: The GHA log interleaves the step's script preview with its output.
In a failed boot-check run, lines showing is-active gdm.service in the preview
column appear alongside inactive + exit code 3 in the output column — making
it look like GDM failed. The actual failing line is always
systemctl is-active multi-user.target immediately above. Verify by checking
which ==> echo precedes the inactive output, not which command appears in the
script preview.

Mergeraptor merges on next do not fire push events (2026-06-19)#

Symptom: Junction-bump PRs merge into next but build.yml never triggers.
The branch can go days without a build despite multiple commits landing.

Cause: Mergeraptor uses the GitHub API to merge PRs. Those merges do not
create a push event that triggers GitHub Actions workflows.

Fix: Add a scheduled dispatcher workflow (nightly-next-build.yml) that
calls gh workflow run build.yml --ref next once per night. Set it to 03:00 UTC
— after the 20:00 UTC junction tracker and its auto-merge window, and before US
daytime when main builds compete for the BST remote executor.

on:
  schedule:
    - cron: '0 3 * * *'
jobs:
  dispatch:
    runs-on: ubuntu-latest
    permissions:
      actions: write
    steps:
      - run: gh workflow run build.yml --repo "${{ github.repository }}" --ref next
        env:
          GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}

sync-main-to-testing does not need a GitHub App token (2026-06-19)#

reusable-sync-branches.yml declares GH_TOKEN as required: false and falls
back to github.token. The promote job in publish.yml already fast-forwards
the testing branch with GITHUB_TOKEN on every publish cycle — proof that no
App token is needed. Remove the generate-token job and both BLUEFINBOT_*
secret references entirely.

| "I'll just load the giant CI file." | That's how agents waste context. Route first, then go narrow. |
| "The workflow name tells me enough." | Wrong. Trigger type, branch filter, and reusable call semantics are usually the bug. |
| "Smoke failed, publish is broken." | Not necessarily. First check whether the failure is in the hard gate or in observational signal. |
| "I know GitHub Actions permissions from memory." | Good way to ship another startup_failure. Verify with Context7. |

Red Flags#

• Reading ci-reference.md before classifying the failure
• Editing reusable-workflow callers without checking top-level permissions:
• Treating AT-SPI smoke as the same class of signal as boot-check
• Changing publish/promotion behavior without mapping the branch and trigger path first
• Adding more prose to a router instead of splitting a focused skill

Verification#

After using this router, confirm:

• You identified the exact CI failure class before deep reading
• You loaded only the next relevant skill file
• You read the actual workflow being changed
• Any GitHub Actions or bootc syntax change was verified via Context7
• The lesson landed in a focused skill, not as more sprawl in the router

Promotion PR stuck at UNKNOWN mergeability (2026-06-19)#

Symptom: Promotion PR shows mergeable: UNKNOWN / mergeStateStatus: UNKNOWN
permanently. Web UI shows "Checking for the ability to merge automatically..." and
never resolves. gh pr merge --squash errors with "Head branch is out of date."

Root cause: sync-main-to-testing.yml fires on every push to main and its
cleanup-squash-branch job unconditionally deletes auto/promote-testing-to-main.
If any normal PR merges to main while a promotion PR is open, the squash branch
gets deleted. GitHub then permanently reports UNKNOWN mergeability for that PR.

Recovery:

gh pr close <promotion-pr> --repo projectbluefin/dakota \
  --comment "Closing — squash branch was deleted. Re-triggering."
gh workflow run promote-testing-to-main.yml --repo projectbluefin/dakota

Systemic fix: PR #931 guards the cleanup step — it skips deletion when an
open PR targets the squash branch. Also fixed in projectbluefin/actions:
reusable-promote-squash.yml now has contents: write and pull-requests: write
so gh pr merge --auto actually enables auto-merge on creation.

gh pr merge --auto requires contents: write in reusable workflows (2026-06-19)#

When a reusable workflow declares permissions: contents: read, GitHub restricts
it to read-only even if the caller grants write. gh pr merge --auto silently
fails — the PR gets no auto-merge flag and the maintainer sees "Enable automerge".

Fix: Add contents: write (and pull-requests: write) to the reusable
workflow's permissions block. Applied to:

• projectbluefin/actions/reusable-promote-squash.yml (promotion PRs)
• projectbluefin/dakota/.github/workflows/pr-triage.yml (regular PRs via #927)

Diagnostic: Check autoMergeRequest is non-null after the workflow runs:

gh pr view <n> --repo projectbluefin/dakota --json autoMergeRequest \
  --jq '.autoMergeRequest != null'

If false, the gh pr merge --auto call failed — check the workflow's declared
permissions first.

testing branch is independent — no fast-forward from main (2026-06-21)#

testing is an independent branch, identical to bluefin/bluefin-lts. It does NOT get
fast-forwarded from main. BST-changing merges to testing trigger their own build
and :testing publish. GHA-only merges (Renovate workflow pins) are filtered out.

Push trigger paths-ignore in build.yml:

push:
  branches: [main, next, testing]
  paths-ignore:
    - '.github/workflows/**' # workflows don't affect the image
    # .github/actions/** intentionally NOT ignored — local composite actions are used in build
    - 'docs/**'
    - '**.md'
    - 'AGENTS.md'

Flow:

BST PR → testing → build → :testing published
GHA-only PR → testing → filtered, no build
promote PR: testing → main → build → stable

The previous Fast-forward testing branch step in publish.yml was disabled (if: false)
in PR 1004. It caused a redundant 5h rebuild: main build → fast-forward testing → push
to testing → second full rebuild of identical content. Fix: PR 997 removed testing from
push triggers entirely (broke publishing); PR 1004 restored push trigger with
paths-ignore and removed the fast-forward instead.

sync-main-to-testing.yml is required — do not remove it#

After a testing → main promotion squash merge, the squash commit lands on main
but not in testing. sync-main-to-testing.yml merges main back into testing so
the next promotion PR is not blocked by a diverged history.

This is not the same as the removed publish.yml fast-forward step (which
caused double builds on every main push). The sync only fires on push to main
and handles a structural necessity of the squash-merge promotion model.

Rapid-fire PR merges cancel pending builds#

GitHub Actions concurrency with cancel-in-progress: false prevents in-progress
jobs from being cancelled, but pending (queued) jobs are replaced when a new
push arrives for the same concurrency group. Merging many PRs in quick succession
results in all but the last build being cancelled.

Symptom: All recent builds on a branch show cancelled status.

Fix: Trigger manually after the queue settles:

gh workflow run build.yml --ref main # or --ref testing

Always check for cancelled builds after batch-merging PRs.

PR triage gate — testing-first model#

The pr-triage.yml gate enforces branch targets. In the testing-first model:

• PRs targeting testing → allowed (all content PRs)
• PRs targeting next → allowed (GNOME master stream)
• PRs targeting anything else (stable, latest) → blocked

If the gate is only allowing renovate/* branches to target testing (old state),
update it to allow all branches targeting testing. See PR 1009.

Renovate must not manage projectbluefin/* — one exclusion rule covers all (2026-06-21)#

All projectbluefin/* actions (projectbluefin/actions, projectbluefin/testsuite,
projectbluefin/bonedigger) use org-managed tags (@v1, @main). Renovate must
not generate SHA-bump or pin-digest PRs for any of them.

Pattern that caused churn: pinDigests: true applied globally, then a group rule
for projectbluefin/actions that didn't set pinDigests: false. Renovate generated
paired PRs every actions release:

• "update projectbluefin/actions" — SHA bump
• "pin dependencies" — re-pins things that got unpinned

Correct renovate.json5:

{
  "matchDepNames": ["/^projectbluefin\\//"],
  "enabled": false,
  "pinDigests": false
}

One rule. Replaces all per-package exemptions (bonedigger, actions, etc.).

Renovate automerge label must be in the general rule (2026-06-21)#

The renovate-automerge.yml workflow uses the automerge label as its signal.
The general automerge rule must include labels or Renovate digest/patch PRs stall
with no auto-merge trigger:

{
  "matchUpdateTypes": ["digest", "pin", "patch", "minor"],
  "automerge": true,
  "automergeType": "pr",
  "automergeStrategy": "squash",
  "labels": ["chore/deps", "automerge"]
}

Without this, only PRs created by per-package group rules (that explicitly set labels)
get the automerge label. Everything else opens with pr/needs-review only and stalls.

update-iso-table.yml removed — [skip ci] churn (2026-06-21)#

update-iso-table.yml ran every 6 hours and committed docs to main with [skip ci].
Even with [skip ci], the commits touched main and caused noise. Removed.

If ISO table needs updating in future, do it on demand via workflow_dispatch or
move it to the dakota-iso repo where it naturally belongs.

testing branch must have required check for auto-merge to work (2026-06-21)#

gh pr merge --auto requires the target branch to have at least one required
status check or required review. testing had no protection — --auto silently
failed with a warning, leaving approved PRs stuck indefinitely.

Fix: add validate as a required check on testing via the API:

gh api repos/projectbluefin/dakota/branches/testing/protection \
  -X PUT --input - << 'PROTECTION'
{
  "required_status_checks": {"strict": false, "contexts": ["validate"]},
  "enforce_admins": false,
  "required_pull_request_reviews": null,
  "restrictions": null,
  "allow_force_pushes": true
}
PROTECTION

allow_force_pushes: true is required — the promote workflow force-pushes the squash branch.

pr-triage auto-merge must fall back to direct merge (2026-06-21)#

--auto can fail even on protected branches (race between approval and checks completing,
or protection not yet propagated). The correct pattern in on-pr-review:

if gh pr merge "$PR_URL" --auto --squash 2>/dev/null; then
  echo "auto-merge enabled"
else
  gh pr merge "$PR_URL" --squash 2>/dev/null \
    && echo "merged directly" \
    || echo "::warning::checks still running"
fi

Without the fallback, approved PRs with already-green CI sit permanently unmerged.

OCI-native daily promotion model — pipeline timing and deleted workflows (2026-06-23)#

Dakota migrated from a weekly squash-PR ceremony to a daily OCI-native promotion
flow (issue 1073). Key operational facts for CI debugging:

Deleted workflows (do not recreate): promote-testing-to-main.yml, pr-release-gate.yml, sync-main-to-testing.yml, cache-warm.yml.

Daily BST build windows (serialized for CAS):

03:00 UTC nightly-next-build → next branch (x86, 90-210 min)
13:00 UTC build.yml schedule → testing x86 default+nvidia serial (max-parallel:1, 90-390 min)
~18:00 publish.yml fires → :testing + :testing-nvidia published
~18:00 build-aarch64.yml fires (workflow_run from publish) → ARM default (~90-210 min)
~18:00 execute-release.yml fires (workflow_run from publish) → freshness check → promote
20:00 track-next-junctions → may trigger next junction build

cache-warm.yml is deleted. The daily 13:00 UTC build.yml schedule replaces it. CAS stays warm as long as the daily build runs. If the daily build is absent for >48 hours, expect cold-start non-determinism.

ARM build trigger changed. build-aarch64.yml now fires via workflow_run from publish.yml on testing — not a Tuesday cron. This ensures ARM starts only after x86 CAS writes complete, preventing write contention that was causing Cached elements after warm: 0.

SHA-based freshness check. execute-release.yml compares the :testing image digest to the current :stable digest. If they are equal, promotion is skipped (nothing new to ship). If different, the promote path runs: cosign verify → boot-check → skopeo copy → fast-forward main. The SHA used for cosign verify comes from github.event.workflow_run.head_sha, not a live :testing tag lookup.

testing is now the default GitHub branch. All PRs target testing. The old main-targeting PRs pattern is gone. main is a bookmark.