Factory Operations Skill#
Covers systems that keep the projectbluefin factory safe:
- Production gate - repository-specific human approval or automated verification before a build reaches
:stable - Promotion cadence - per-repo schedule, explicit enrollment, merge-queue selection, and E2E policy
- Factory health monitor - scheduled pipeline health monitoring with automatic issue creation
- Renovate auto-merge - automated dependency bump management
1. Production Gate (Track C-1)#
What it is#
Production authorization is repository-specific. Repositories using a GitHub
Environment can require distinct human approvers. Bluefin instead uses an
automated chain: exact-source-SHA E2E success, cosign verification, Tuesday UTC
release-window approval, and merge-queue enrollment after the gate succeeds.
Where it lives#
| Repo | Workflow | Gate |
|---|---|---|
projectbluefin/bluefin | promote-testing-to-main.yml | E2E + cosign + Tuesday window |
projectbluefin/dakota | weekly-testing-promotion.yml | production environment |
Workflow snippet#
jobs:
promote:
runs-on: ubuntu-latest
environment:
name: production
url: https://ghcr.io/projectbluefin/dakota:stable
steps:
- # ... SHA-lock + verify-e2e + skopeo copy ...
Manual GitHub UI setup (environment-gated repos only)#
For Dakota, after the workflow change is merged:
- Go to the repo ā Settings ā Environments ā New environment
- Name:
production - Set Required reviewers - list the maintainers.
- Set the required count to 2 (two distinct approvals).
- Restrict to the release branch.
Verification for environment-gated repos#
- Trigger the promotion workflow via
workflow_dispatch. - Confirm the job pauses with a yellow "Waiting for approval" status.
- One reviewer approves ā job stays paused.
- Second reviewer approves ā job runs.
- Author approving their own dispatch is blocked when approval is required.
What it does NOT prevent#
Repo admins can bypass Environment rules. All bypasses are permanently visible in:
gh api repos/<org>/<repo>/deployments- every deployment record- The Environment's deployment history page in GitHub UI
The protection is friction-ful for accidental/casual bypasses, not cryptographically airtight. This is the appropriate bar for a trusted team of 4.
2. Promotion Cadence and Merge Queue Contract#
Each consumer repo promotes :testing ā :stable (or :lts) via a thin caller to
reusable-promote-squash.yml@v1. The caller owns cadence through
enqueue_promotion; use_merge_queue selects the target branch's merge mechanism
and must not be overloaded as an on/off switch.
Per-repo schedule and inputs#
| Repo | Cron (UTC) | enqueue_promotion | use_merge_queue | run_e2e | Notes |
|---|---|---|---|---|---|
projectbluefin/bluefin | daily 0 4 * * * | Tuesday schedule or manual dispatch | true | true | push and non-Tuesday runs refresh the PR without enqueueing |
projectbluefin/dakota | Tuesday 0 4 * * 2 | schedule or manual dispatch | repository policy | repository policy | weekly release path only |
Release-window enrollment#
Callers that refresh a promotion PR outside the release window must pass an
explicit boolean:
enqueue_promotion: ${{ needs.release_window.outputs.should_enqueue == 'true' }}
use_merge_queue: true
The reusable workflow first builds the squash PR and runs the release gate. Its
enqueue job depends on both jobs and requires needs.gate.result == 'success'.
Cosign or E2E failure therefore cannot race with queue enrollment. Do not move
enrollment back into the PR-construction job.
enablePullRequestAutoMerge is blocked when the target branch has a merge queue
ruleset. use_merge_queue: true selects enqueuePullRequest; it does not decide
whether a release window is open.
Merge-blocking labels#
Both promotion and Renovate use .github/actions/merge-label-guard immediately
before each merge/enqueue write (including retries and promotion's validate
status). It reads every current label page and fails closed on API or malformed
data. Literal hold, bare do-not-merge, and do-not-merge/* block enrollment.
An earlier check is not sufficient after a release gate or check-rollup wait.
For a held PR, withdraw native auto-merge/queue enrollment only when its recorded
actor matches the authenticated token's actor; leave other actors' enrollment
untouched and report failures. No labels, reviews, or branch protections change.
auto_merge ā what the PR body promises#
auto_merge is the third, separate axis: whether anything ever merges the
promotion PR without a human. It is forwarded to render-pr-body and changes
only the merge notice in the body. It defaults to true, which renders today's
weekly auto-merge CAUTION block.
Set it to false in a repository where a person merges the PR from the GitHub
UI. A personal-account fork cannot use a merge queue, and gh pr merge --auto
with no merge method fails without a TTY, so nothing there enrolls the PR. Such a
caller pins both flags:
enqueue_promotion: false
auto_merge: false
Never derive the body's notice from enqueue_promotion. That flag is per run:
bluefin passes false on every push and every non-Tuesday run, and the body is
re-rendered on each of them, so the PR would claim a human merge all week and
lose the do-not-merge instruction that is actionable during exactly that
window.
The built-in true notice names the bluefin/dakota (Tuesday) and bluefin-lts
(Thursday) 04:00 UTC windows. A consumer with a different cadence passes
merge_note to render-pr-body; any non-blank value replaces the built-in
notice verbatim, whatever auto_merge is. reusable-promote-squash.yml
forwards its own optional merge_note input (default empty) to the render step,
so consumers set it on the reusable-workflow call.
E2E policy#
Set run_e2e: true when the consumer's post-build E2E workflow is the release
qualification signal. The gate looks up the completed run for the exact source
branch SHA and refuses enrollment unless it succeeded. e2e_image must be
non-empty to activate that lookup. Consumers that run an equivalent production
environment gate may leave this false, but must document the alternate trust
boundary.
3. Factory Health Monitor#
What it is#
A scheduled workflow (actions/.github/workflows/factory-health.yml) that checks the last 24 hours of
critical factory pipelines and opens an issue in projectbluefin/common when any monitored pipeline
falls below the success-rate threshold.
Schedule#
cron: '0 */6 * * *' - every 6 hours. Also triggerable via workflow_dispatch.
Monitored pipelines#
| Repo | Pipeline | Workflow queried |
|---|---|---|
projectbluefin/dakota | Build | Build Bluefin dakota |
projectbluefin/dakota | Promote | Publish Bluefin dakota |
projectbluefin/common | Build | Build |
projectbluefin/common | Unit Tests | Unit Tests |
Archived repositories are not active health/drift targets. The archived-only
package-cadence caller is retired; reusable-pkg-cadence.yml remains available
for explicit active consumers, independently of registry cleanup.
Alerting behavior#
- Success rate =
successful completed runs / completed production runs - Window = last 24 hours
- Threshold = 80%
- Minimum sample = 3 completed runs in the window (
MIN_RUNS) - Escalation floor = 2 back-to-back failures (
MIN_CONSECUTIVE_FAILURES) - Success rate measures the production pipeline, not pre-merge validation. The
historical rate recipe excluded onlyskipped/in_progress/cancelled/
action_requiredconclusions. That still counted pre-mergepull_requestand
merge_groupruns as pipeline outcomes ā but those validate in-flight work and
routinely fail transiently (a failed PR run or a merge-queue collision gets fixed
and re-run before the branch lands), deflating the real pipeline rate and firing
false alerts (e.g. projectbluefin/actions#503). The recipe therefore also drops
pull_requestandmerge_groupevents, counting only production pipeline events
such aspush,schedule,workflow_dispatch, andworkflow_run. - Open issues are deduplicated by repo + pipeline title prefix and alerting bot author
(only alerts opened by the workflow's alerting identity suppress new alerts, preventing human-filed
issues with matching titles from muting a pipeline) - Issues are filed in
projectbluefin/commonwith the labels that currently exist from:
priority/critical-urgent,kind/bug
Each pipeline lands in one of four statuses. Only alert files an issue:
| Status | Meaning |
|---|---|
healthy | rate ā„ threshold |
alert | rate < threshold on a trusted sample, or an under-sampled pipeline failing back-to-back |
low-sample | rate < threshold but fewer than MIN_RUNS completed runs in the window, and not failing back-to-back |
no-runs | nothing completed inside the window |
A rate needs a sample ā the 0/1 trap#
A threshold is meaningless on a sample of one. Nightly E2E runs once a day, so a 24h window
holds exactly one completed run and the only rates it can report are 100% and 0%. One flaky night
put the pipeline at 0/1 and filed a priority/p0 issue that a maintainer closed as "transient"
(actions#479,
actions#555). The same shape bites any
low-volume pipeline, and it got worse ā not better ā when #528
correctly narrowed the sample by excluding pre-merge runs. The no-runs branch only ever caught
n=0, never n=1.
Below MIN_RUNS the monitor reports low-sample and does not file an issue.
But a sample floor is a mute button, so it needs an escape hatch. A nightly that fails every
night would stay at n=1 forever and never alert. So low-sample escalates to alert once the
most recent MIN_CONSECUTIVE_FAILURES completed runs failed in a row. That streak is counted over
the whole fetched run history, not the window ā the window is exactly the thing that is too
small to tell a flake from an outage. A once-a-day pipeline that is genuinely down therefore alerts
on its second consecutive failure, roughly 24h later, instead of on the first flake.
Keep the two copies in lockstep. The shipped logic is inline bash in factory-health.yml;
scripts/monitor_pipeline.py mirrors it in Python for unit testing and is not invoked by the
workflow. Editing one without the other silently makes the tests describe code that never runs.
Coverage for both:
| Copy | Tests |
|---|---|
factory-health.yml (shipped) | tests/bats/test_factory_health_status.bats ā captures the run block verbatim |
scripts/monitor_pipeline.py (mirror) | tests/test_monitor_pipeline.py |
Bound the run fetch by the window, not by a run count#
gh run list must carry --created ">=${CUTOFF_ISO}". The monitor used to fetch each
pipeline's most recent 100 runs and then discard everything older than the 24h window
client-side. That makes the sample a function of run volume rather than of the window: once a
pipeline produces more runs than the fetch limit inside the window, the oldest in-window runs
are never fetched, and the rate is computed from a silently truncated sample. In the worst case
every fetched run predates the window and a healthy pipeline reports no-runs.
It was never close to firing ā measured across all monitored pipelines, a 100-run fetch spanned
9 to 93 days and yielded 0-38 in-window runs ā but the margin was accidental. It also shrinks as
the monitor discards more of what it fetches: any event the rate filter excludes still consumes
fetch slots on the way to being thrown away.
Rules:
CUTOFF_ISOandCUTOFF_EPOCHare both derived fromWINDOW_HOURS, so the server-side fetch
bound and the client-side cutoff cannot drift apart.- Keep the client-side cutoff filter.
--createdis a search qualifier applied by the API, not a
guarantee about the returned set; the rate must be computed over exactly the window the alert
claims. Verified equal on all 10 monitored pipelines over a 7-day window before relying on it. RUN_FETCH_LIMITstill caps the fetch, and saturating it (fetched >= RUN_FETCH_LIMIT) emits
a::warning::naming the pipeline. A truncated sample is reported, never passed off as a
complete window.- Emit that warning on stderr.
monitor_pipeline()writes its result JSON to stdout and the
caller captures it; a workflow command on stdout would corrupt the captured JSON. The runner
parses::warning::from both streams, so>&2costs nothing.
Alert lifecycle - alerts must close themselves#
Deduplication and closing are two halves of one mechanism. The dedupe check matches an open
issue by its title prefix (fix(factory): [<repo>] [<pipeline>] success rate dropped) and
skips opening a new one, so an alert that is never closed permanently mutes its pipeline -
every later breach of that repo + pipeline hits the dedupe branch and is dropped silently.
Each run therefore closes before it opens: scripts/factory_health_resolve.py maps the health
results onto the open issue list and returns the alerts whose pipeline is healthy again. The
workflow comments the recovered rate on each one and closes it with reason completed.
Rules that keep the pass safe:
- Only
healthyrecovers an alert.no-runsis an absence of evidence, not a fix - a pipeline
with no completed runs in the window keeps its alert open until a real run proves otherwise. alert_title_prefix()in the script must stay byte-identical totitle_prefixin the
workflow;tests/test_factory_health_resolve.pyasserts that by parsing the workflow. Change
the alert title format in one place only if you change it in both.- Every open alert matching the prefix is closed, not just the first, so duplicates from earlier
breaches do not linger. - Only alerts opened by the alerting account are closed. The title prefix is not proof the
workflow opened an issue, and anyone who can file an issue in the alert repo can copy it. The
workflow passes--authorasgh issue listreports it:app/<app-slug>from the app-token
step when routing is intended,app/github-actionson fallback. An empty author closes nothing. - The whole close pass is best-effort, and that includes resolving which alerts recovered. The
step runs underset -euo pipefail, so thefactory_health_resolve.pycall is wrapped in
if ! ...and falls back to[]on failure. Closing was added on top of a working alerting
pass; no failure in it may take that pass down.
Because the workflow now runs a repository script, the job checks out the repo and needs
contents: read in addition to actions: read and issues: write.
Authentication pattern#
Use the workflow github.token for read-only gh run list calls against the public factory repos.
Generate a GitHub App token scoped to projectbluefin/common before creating issues there. This keeps
cross-repo issue writes explicit while avoiding broader write scopes for routine monitoring.
Token generation is best-effort so a GitHub App installation permission mismatch cannot prevent the
health checks from running. If the cross-repo token is unavailable, use the workflow's repository-scoped
github.token to file alerts in projectbluefin/actions. This preserves monitoring and alerting while
keeping the fallback credential unable to write outside its source repository.
continue-on-error on a token step must be paired with a step that fails the job. This is the
general rule, not a factory-health detail: continue-on-error: true keeps the job green, so a
degraded run is indistinguishable from a healthy one in the checks UI. Report the degraded state as a
step output, then fail on it in a later step that runs after the useful work is done:
- name: Verify alert routing
if: always() && steps.monitor.outputs.alert_routing == 'fallback'
run: |
echo "::error title=...::..."
exit 1
A ::warning:: is not enough. Factory Health ran for months filing alerts into the wrong repository
while every scheduled run concluded success; nobody noticed until a reviewer read a raw run log
(projectbluefin/actions#556). A fallback repo may lack the supported
priority/critical-urgent and kind/bug labels, so alerts carry a misroute
banner and only apply labels already present in the destination.
Diagnosing "the common issue token is unavailable". Two different failures produce an empty token
and they have opposite remediations. Read the app-token step's own log before assuming either:
| Symptom in the app-token step | Cause | Fix |
|---|---|---|
The 'client-id' (or deprecated 'app-id') input must be set to a non-empty string. | credential not wired (see the vars.* gotcha below) | workflow change |
422 The permissions requested are not granted to this installation. | the App installation does not grant the requested permission on the target repo | App installation change ā no workflow change will fix it |
The 422 form is not trigger-dependent: it fails identically on schedule and on
workflow_dispatch, because installation permissions are a property of the App, not of the event.
Do not chase a schedule-versus-dispatch difference. The confirming cross-check is factory-drift.yml,
which mints a token from the same App with permission-contents: read and succeeds ā proving the
credentials and the installation are fine and only the requested permission is missing. Remediate by
granting the App that permission on the target repo and approving the new permission on the
installation.
MERGERAPTOR_APP_ID is a secrets.* value, not a vars.* value ā see the approved-secrets
table in docs/skills/supply-chain.md. Passing vars.MERGERAPTOR_APP_ID to
actions/create-github-app-token silently resolves to an empty string (repo/org variables and
secrets are separate namespaces) and the step fails with
The 'client-id' (or deprecated 'app-id') input must be set to a non-empty string. Always wire it
as client-id: ${{ secrets.MERGERAPTOR_APP_ID }} ā use client-id, not the deprecated app-id
input, for consistency across workflows.
Output#
The workflow always prints a markdown summary table to stdout and $GITHUB_STEP_SUMMARY, even when no
issues are opened. When alerts were misrouted, the summary also carries an "Alert routing degraded"
section naming the remediation, and the run concludes failure.
4. Renovate - Automated Dependency Maintenance#
What it does#
Renovate runs as the MergeRaptors GitHub App and opens PRs to bump pinned action SHAs and digests. Qualifying PRs auto-merge when CI passes ā no human review needed, even though main requires one approval and CODEOWNERS review. See "CI-gated review bypass" below for how that is safe.
Config#
Two files co-exist:
.github/renovate.json5- base org config (inherited fromprojectbluefin/renovate-config)renovate.json- repo-level overrides, including thepackageRulesautomerge block
Renovate's automerge: true does not merge the PR itself here; it sets GitHub's
autoMergeRequest on the PR, which the auto-merge workflow reads as the
eligibility signal. renovate.json is therefore the single source of truth for
what may merge unattended ā change policy there, not in the workflow.
What auto-merges: whatever renovate.json marks auto-mergeable (SHA digest bumps and pin updates), once every check on the PR has completed successfully. These carry no behavior change.
What never auto-merges: major version bumps, anything Renovate did not mark auto-mergeable, drafts, conflicting PRs, and any PR with a failing, pending, cancelled, or empty check rollup. Those still require a human approval and merge after confirming every required check is green.
CI-gated review bypass#
main keeps required_approving_review_count: 1 and require_code_owner_reviews: true for everyone. The MergeRaptor GitHub App is the only entry in the branch-protection review-bypass allowance ā no users, no teams:
gh api repos/projectbluefin/actions/branches/main/protection \
--jq '.required_pull_request_reviews.bypass_pull_request_allowances'
Expected: apps contains only mergeraptor; users and teams are empty.
The bypass is only reachable through automation:
| Piece | Responsibility |
|---|---|
.github/workflows/renovate-automerge.yml | Local workflow_run caller. Fires on every CI workflow completion, passes base_branch: main, require_auto_merge: true, and the MergeRaptor app credentials. |
.github/workflows/reusable-renovate-automerge.yml | Mints the app installation token, qualifies the PR, validates the full check rollup, and merges (respecting a merge queue if one exists). |
These gates must all pass before a merge happens:
- Author is
mergeraptororrenovate(in any spelling ā see the gotcha below). - Renovate enabled auto-merge on the PR (
autoMergeRequest != null), i.e.renovate.jsonsays it qualifies. - Every check is green and nothing is still running. Nonterminal check states ā
PENDING,QUEUED,IN_PROGRESS,WAITING,REQUESTED,EXPECTED(the same setgh pr checksplaces in its ownpendingbucket) ā are awaited up tocheck_timeout_seconds, never treated as failures.SKIPPEDandNEUTRALare non-blocking, matching GitHub's own merge semantics. Only terminal failure conclusions (FAILURE,CANCELLED,TIMED_OUT,ACTION_REQUIRED,STARTUP_FAILURE,STALE,ERROR) or an unrecognized state block outright. - Fresh labels permit merge. The shared merge-label guard runs after the wait
and again before a queue-strategy retry, refusing holds and unreadable policy.
Gotcha ā an absent check is not a passing check. gh pr checks reports only the check-runs that currently exist. A workflow that is queued but has not yet registered a check-run is simply missing from the rollup, not PENDING; and re-running a workflow removes its check-runs entirely while they re-queue. A naive "nothing is pending, so we're done" gate will happily merge with most checks never having run. Since this workflow is triggered by one CI workflow completing while siblings may still be queued, that race is the normal case, not an edge case. Cross-check in-flight runs for the same commit:
gh run list --repo "$GITHUB_REPOSITORY" --commit "$HEAD_SHA" --limit 100 \
--json status --jq '[.[] | select(.status != "completed")] | length'
Treat a non-zero count, an empty rollup, and an unreadable run list as "keep waiting" ā all three must fail closed. Only an all-SUCCESS rollup with zero in-flight runs may merge.
Gate 3 is why it is safe for any CI workflow to trigger the caller: an
early-finishing workflow cannot merge ahead of its still-running siblings. A
skip is not a failure ā the next successful workflow_run event retries the
whole evaluation.
Gotcha ā bot author login differs between REST and GraphQL. For a GitHub App, GraphQL's author.login returns the bare slug mergeraptor, while REST and gh pr list return app/mergeraptor, and a legacy bot user returns renovate[bot]. Any author matcher must normalise all three spellings or it will silently match nothing. Verify a matcher against live data before trusting it:
gh api graphql -f query='{repository(owner:"projectbluefin",name:"actions"){
pullRequests(first:20,states:OPEN,baseRefName:"main"){
nodes{number author{login} autoMergeRequest{enabledAt}}}}}'
Gotcha ā gh pr merge --auto cannot be used here. GitHub's auto-merge queue does not honour bypass_pull_request_allowances; only a direct merge does. --auto also errors with "Protected branch rules not configured" on unprotected base branches. The workflow therefore always does a direct merge, with the check-rollup gate standing in for what --auto would have waited on.
Gotcha ā a merge queue rejects any explicit merge strategy, but only as a warning. When the base branch has a merge queue, gh pr merge --squash prints ! The merge strategy for main is set by the merge queue and still exits 0, having enqueued the PR. Never infer merge-queue behaviour from the exit status ā match the stderr text. Two strings matter, and they do not share a substring:
Message (stderr, !-prefixed) | Meaning |
|---|---|
The merge strategy for <branch> is set by the merge queue | strategy flag ignored; PR was enqueued |
Pull request <repo>#<n> is already queued to merge | a previous attempt enqueued it ā success, not failure |
The reusable workflow exposes merge_method (squash by default, queue to omit the flag entirely), treats both messages as success, and retries without the flag if a future gh version hard-fails instead of warning.
Never infer the merge outcome from stderr text or the exit code. A plain gh pr merge with no strategy flag enqueues the PR while printing nothing at all and exiting 0 ā indistinguishable from a real merge by output alone. Conversely, gh's success line goes to stderr and includes the PR title, so a PR titled "ā¦update merge-queue action" would trip a naive grep "merge queue". Ask GitHub for the actual state instead:
gh pr view "$PR_NUMBER" --repo "$GITHUB_REPOSITORY" --json state --jq .state
# MERGED = landed; OPEN = enqueued, the queue lands it ~60s later
Gotcha ā gh pr merge --match-head-commit never fails with an HTTP 409. gh sends the pin as expectedHeadOid on the GraphQL mergePullRequest mutation, not to the REST merge endpoint. A head that moved since qualification is refused as a GraphQL error on HTTP 200, printed as GraphQL: <message> (mergePullRequest), so grep "409" on stderr does not see it. Decide "head moved" by reading the PR's current head after a failed merge, and treat it as a skip, not a failure:
gh pr view "$PR_NUMBER" --repo "$GITHUB_REPOSITORY" --json headRefOid --jq .headRefOid
# differs from the qualified SHA = moved; empty (query failed) = unknown, not moved
Apply the same check after every merge attempt, including the strategy-less merge-queue retry.
Gotcha ā the secrets context is unavailable in step-level if:. To conditionally mint an app token in a reusable workflow, mirror credential presence into job-level env first (HAS_APP_CREDS: ${{ secrets.app_id != '' && secrets.private_key != '' }}) and branch on env.HAS_APP_CREDS.
Consumer-validation exemption: Renovate PRs (author login ending in [bot] or starting with app/) are automatically exempt from the consumer PR + CI run evidence requirement, even when they touch action files. See docs/skills/consumer-validation.md.
Validation workflow#
.github/workflows/validate-renovate.yml runs renovate-config-validator --strict on PRs and pushes that touch either Renovate config file. Changes that fail validation are caught before merging.
Auto-merge repo setting#
The repository has allow_auto_merge: true enabled. Without this, GitHub ignores the automerge setting regardless of config.
Automated wiring assertion#
.github/workflows/renovate-automerge-wiring.yml runs scripts/renovate-automerge-wiring-check.sh on a daily schedule (and via workflow_dispatch). It asserts the declarative facts that make a real mergeraptor PR mergeable ā allow_auto_merge is true, and the MergeRaptor app is the sole review-bypass actor on main ā and reports any live mergeraptor/renovate PR with auto-merge enabled. The end-to-end merge itself needs a real mergeraptor PR (created out-of-band by Renovate) and cannot be forced from CI, so this check is the part of issue #403 that is automatable: it fails loudly if the wiring drifts, and its passing is the precondition evidence that the merge step in renovate-automerge.yml can land a qualifying PR. Covered by tests/bats/test_renovate_automerge_wiring_check.bats.
Gotcha ā allow_auto_merge and a ruleset's bypass_actors are omitted (JSON null/missing key), not reported false/[], for a caller without push/administration access to the repo (projectbluefin/actions#514). The default github.token on a schedule/workflow_dispatch run never has that access, so the job mints a MergeRaptor app token (create-github-app-token with MERGERAPTOR_APP_ID/MERGERAPTOR_PRIVATE_KEY) and uses it as GH_TOKEN for the check ā the same credential already trusted for the bypass path itself. The script additionally tells "field omitted because the token can't see it" apart from "field present and false/empty": it reports the former as FAIL: ⦠is unreadable rather than misreporting the repo setting as disabled.
Relationship to @v1#
Renovate keeps SHA pins current for third-party actions in this repo. Consumers donāt see the updates until a maintainer advances the @v1 tag. See the @v1 runbook in AGENTS.md for the exact commands.
Troubleshooting#
| Symptom | Cause | Fix |
|---|---|---|
| Renovate PR won't auto-merge | allow_auto_merge disabled on repo | gh api -X PATCH repos/projectbluefin/actions -f allow_auto_merge=true |
| Renovate PR sits green and unmerged | autoMergeRequest is null ā renovate.json doesn't mark this update type auto-mergeable (e.g. a major bump) | Expected. Review and merge by hand, or widen the packageRules automerge match |
| Auto-merge job logs "No qualifying Renovate/Mergeraptor PR" on a real Renovate PR | Author matcher missed a login spelling, or the PR is a draft/conflicting | Compare against the live GraphQL author.login ā GraphQL says mergeraptor, REST says app/mergeraptor |
Auto-merge job reaches the merge step and gh pr merge fails on review requirements | MergeRaptor missing from bypass_pull_request_allowances, or the job used github.token instead of the app token | Check the bypass list; confirm the caller passes app_id + private_key |
Wiring check FAILs with ⦠is unreadable (field omitted by the API) | GH_TOKEN lacks push/administration access to the repo ā the API silently omits allow_auto_merge/bypass_actors rather than returning false/[] | Confirm MERGERAPTOR_APP_ID/MERGERAPTOR_PRIVATE_KEY are set and the app token step ran; this is not evidence the repo setting is actually disabled |
| Wiring check's "Generate app token" step fails with a 422 before the script runs | The step requests permission-administration: read, and GitHub's installation-token endpoint refuses to mint a token for a permission the MergeRaptor installation does not hold | Grant the MergeRaptor App installation Administration: read on projectbluefin/actions. This is the mint-time form of the "unreadable" failure above ā same root cause, reported earlier |
| Renovate PR consumer-validation fails | Bot exemption not firing | Verify author login ends in [bot] or starts with app/ - check gh pr view NNN --json author |
| Renovate PR has merge conflict | Another bump landed first; branches diverged | Locally checkout the branch, git rebase origin/main, force-push |
| Two Renovate PRs update the same action | Both opened before either merged | Close the older/lower version one; merge the newer |
| Dependency Dashboard (issue #42) shows PRs as "Open" | Renovate dashboard is eventually consistent - PRs may already be merged | Confirm with gh pr view NNN --json mergedAt before acting; the dashboard self-corrects on next Renovate run |
| Renovate warns: "Fallback to renovate.json as preset is deprecated" | Config file named renovate.json instead of default.json | Rename: git mv renovate.json default.json - content stays identical |
Verification ā is auto-merge actually wired up?#
The bypass allowance and the workflow are independent halves. Configuring only one
leaves the system inert but looking correct. Check all five:
-
.github/workflows/renovate-automerge.ymlexists in this repo and its
workflow_run.workflows:list names every CI workflow that gatesmain. - The branch-protection bypass lists app
mergeraptorand nothing else
(command at the top of this section). -
MERGERAPTOR_APP_IDandMERGERAPTOR_PRIVATE_KEYare set as repo secrets ā
without them the job runs asgithub-actions[bot], which has no bypass and
cannot merge. -
gh api repos/projectbluefin/actions --jq .allow_auto_mergereturnstrue.
Run this with a token that has push/admin access to the repo ā GitHub
returnsnull(notfalse) for a lower-privileged token, which looks
identical to the setting being off and is exactly the trap that produced
issue #514. - A recent run of the "Renovate Auto-merge" workflow exists and its log ends in
either a merge or an explicit skip reason ā a workflow that never triggers is
the failure mode this checklist exists to catch:
bash gh run list --repo projectbluefin/actions --workflow renovate-automerge.yml --limit 5
5. Promotion PR Format (Design C)#
Every promotion PR in bluefin and dakota uses a consistent āDesign Cā body
generated by scripts/render_pr_body.py. The example below is bluefin's
testing ā stable flow; the branch pair is not baked into the template. The
render-pr-body action takes source_branch/target_branch and names the
caller's own workflow file (from GITHUB_WORKFLOW_REF), so finpilot's
main ā stable promotion reads as main ā stable and credits
promote-main-to-stable.yml.
The auto-merge CAUTION block below is printed only when the caller passes
auto_merge: 'true' (the default). A repository whose promotion is merged by a
human ā anything calling the reusable with enqueue_promotion: false ā should
pass auto_merge: 'false' and gets a "merged by a human" note instead, so the
body never advertises a merge window that will not fire.
Title format#
ci(promote): <primary-image> testing ā stable YYYY-MM-DD
Examples: ci(promote): bluefin testing ā stable 2026-06-11
Body structure#
## š¦ Bluefin testing ā stable Ā· 2026-06-11
> **12 days since the last stable release** Ā· [tag ā](release-url)
> Auto-maintained Ā· Updated ISO-timestamp Ā· [Run ā](run-url)
<!-- gate-section-start -->
### Release checklist
**ā
All checks passed**
| Check | Status | Details |
|---|---|---|
| Digest resolution | ā
passed | ... |
| Cosign signatures | ā
passed | ... |
| E2E | ā
passed | ... |
<!-- gate-section-end -->
### Variants being promoted
(variants table with digests when available)
### Changes in this promotion
(files changed, +/- lines against the current target tip, collapsible file list ā squash workflow only)
## Desktop Screenshot
> [!CAUTION]
> **Auto-merge scheduled for Tuesday 04:00 UTC (bluefin/dakota).**
> To block this release: add the `do-not-merge` label to this PR before that time.
> Remove the label when the issue is resolved -- the next weekly window will pick it up automatically.

The changes section describes the tree diff between the target tip and the
source tip ā exactly what the squash commit carries. It deliberately does not
count commits: on a squash-only target the source shares no history with what
was already promoted, so a commit count reports every commit since the fork
(hundreds) even when the promotion is a two-line digest bump (#691).
The gate checklist starts with ā³ placeholders written by the promote job,
then the gate job replaces only the <!-- gate-section-start/end --> block
with live ā
/ā results via scripts/render_gate_section.py.
Promotion PRs must carry the screenshot + caution block in the body, not a
separate GitHub comment. If the PR is labelled do-not-merge, the reusable
workflow skips auto-merge / merge-queue enrollment until the label is removed.
Scripts#
| Script | Called by | Purpose |
|---|---|---|
scripts/render_pr_body.py | promote job | Full PR body with ā³ gate placeholders |
scripts/render_gate_section.py | gate job | Targeted gate section replacement only |
Consumer repo branch targets#
| Repo | Workflow | Target branch for PRs |
|---|---|---|
projectbluefin/bluefin | reusable-promote-squash.yml | testing |
projectbluefin/dakota | reusable-promote.yml | main |
6. Promotion and sync-branches known patterns#
enqueuePullRequest vs enablePullRequestAutoMerge#
For repos with a merge queue enabled, enablePullRequestAutoMerge is blocked by GitHub.
Use the enqueuePullRequest GraphQL mutation instead:
gh api graphql \
-f query='mutation($id:ID!){enqueuePullRequest(input:{pullRequestId:$id}){mergeQueueEntry{id}}}' \
-f id="$(gh pr view <PR> --json id -q .id)"
reusable-promote-squash.yml uses this pattern when enabling auto-merge on promotion PRs.
E2E gate must use source_branch HEAD SHA ā not a hardcoded ref#
The promote-squash workflow queries the E2E gate against the source_branch HEAD SHA
(e.g. testing HEAD), not a hardcoded main or the caller's github.ref:
# Correct ā lock to the branch that the E2E workflows ran against
SHA=$(gh api repos/$REPO/git/ref/heads/$E2E_HEAD_BRANCH --jq '.object.sha')
Using a hardcoded branch or github.ref can match a more-recent commit that hasn't had
E2E run yet, silently allowing un-tested code through the promotion gate.
Force-push guard: skip when squash tree is unchanged#
Before force-pushing the squash branch to an existing promotion PR, compare the squash tree
to the PR's current HEAD. If they match, skip the force-push entirely ā force-pushing an
identical tree dismisses reviewers' approvals for no reason:
SQUASH_TREE=$(git rev-parse HEAD^{tree})
REMOTE_TREE=$(git ls-remote origin "refs/heads/$BRANCH" | cut -f1 | xargs git cat-file -p | grep tree | cut -d' ' -f2)
[ "$SQUASH_TREE" = "$REMOTE_TREE" ] && echo "no-op, skipping force-push"
Queue-entry guard: avoid branch rewrites when PR is merge-queued#
When a promotion PR is enrolled in a merge queue, GitHub locks the head branch and rejects
any force-push or branch mutation with GH006: Ref cannot be updated: A pull request using this branch as its head is in the merge queue and cannot be modified. Subsequent workflow runs (e.g. daily cron, push to testing, or PR review triggers) must not attempt to rebuild or mutate the branch while it is queued.
Before checkout or branch mutation, query GraphQL for an existing mergeQueueEntry on the
open promotion PR. If a queue entry exists, emit an informative notice, set promoted=false,
and exit 0 so the merge queue can progress undisturbed:
# shellcheck disable=SC2016 # GraphQL variables, not shell variables
PR_DATA=$(gh api graphql \
-f query='query($owner: String!, $repo: String!, $head: String!, $base: String!) {
repository(owner: $owner, name: $repo) {
pullRequests(headRefName: $head, baseRefName: $base, states: OPEN, first: 1) {
nodes {
id
number
url
mergeQueueEntry {
id
state
}
}
}
}
}' \
-F owner="$REPO_OWNER" \
-F repo="$REPO_NAME" \
-F head="$PROMOTION_BRANCH" \
-F base="$TARGET_BRANCH" \
--jq '.data.repository.pullRequests.nodes[0] // empty' 2>/dev/null) || PR_DATA=""
if [ -n "$PR_DATA" ]; then
QUEUE_ENTRY_ID=$(echo "$PR_DATA" | jq -r '.mergeQueueEntry.id // empty' 2>/dev/null || echo "")
if [ -n "$QUEUE_ENTRY_ID" ]; then
echo "::notice::Promotion PR #${PR_NUMBER} has active merge queue entry (${QUEUE_ENTRY_ID}) ā skipping branch mutation"
echo "promoted=false" >> "$GITHUB_OUTPUT"
exit 0
fi
fi
gh api failure output goes to stdout ā capture defensively#
gh api on a failed request (HTTP 404/500) prints the API error body to stdout
and the human-readable gh: ... message to stderr, then exits non-zero. A
capture like this is therefore NOT safe ā 2>/dev/null only drops the stderr
message, and the error JSON becomes the variable value:
# WRONG ā SHA becomes '{"message":"Not Found",...}' on a missing ref, because
# the error body is on stdout and '' only gets appended, never substituted
SHA=$(gh api repos/$REPO/git/ref/heads/$BRANCH --jq '.object.sha' 2>/dev/null || echo "")
Use the exit status to replace the value instead of || echo "":
# RIGHT ā SHA is empty on failure, populated on success
SHA=$(gh api repos/$REPO/git/ref/heads/$BRANCH --jq '.object.sha' 2>/dev/null) || SHA=""
Apply this pattern to every variable captured from a gh api call that can 404.
A leaked error JSON travels into downstream URLs and produces confusing failures
like unsupported protocol scheme "".
Branch-derived names must be defined once at the job level#
Any value derived from workflow inputs and used by multiple steps (squash branch
name, image ref, tag list) belongs in the job-level env: block as a single
expression:
env:
PROMOTION_BRANCH: ${{ format('auto/promote-{0}-to-{1}', inputs.source_branch, inputs.target_branch) }}
Duplicating the expression in each step's env: invites one step to be edited
and the others to drift. This bit the promote-squash workflow: rebuild/upsert
hardcoded the squash branch name while the validate-status step derived it from
the inputs, so any caller with a non-default branch model (maināstable) pushed
one branch and looked up another ā 404-ing before auto-merge could run. With the
name defined once, all steps stay in lockstep.
reusable-sync-branches: optional GH_TOKEN + force-reset for diverged branches#
reusable-sync-branches.yml merges source_branch into target_branch after a promotion.
Two patterns to know:
Protected branches: The workflow accepts an optional GH_TOKEN secret from the caller.
When provided, it uses a GitHub App token to bypass protected-branch push rules. Without it,
github.token is used ā which fails on protected branches.
Diverged target: If the target branch has commits not in source (e.g. direct CI fixes on
main while testing was being promoted), the workflow force-resets target to source
instead of attempting a merge. This is safe because:
mainonly receives CI fixes that don't need cherry-picking- A failed merge leaves the pipeline broken indefinitely
- Force-reset produces a clean, predictable state
How the Systems Work Together#
Factory health monitor runs every 6 hours
āāā¶ success rate < 80%
āāā fewer than 3 completed runs in the window
ā āāā 2+ back-to-back failures ā treated as an alert
ā āāā otherwise ā reported as low-sample, no issue
āāā no open alert issue ā opens issue in projectbluefin/common
āāā open alert issue exists ā logs and skips duplicate creation
Renovate detects stale SHA pin
āāā¶ Opens bump PR
āāā CI (actionlint) passes ā auto-merges
āāā CI fails ā stays open for human review
Batch of Renovate bumps land on main
āāā¶ Maintainer runs: git tag -f v1 origin/main && git push --force origin v1
āāā¶ All consumer repos pick up updated SHA pins on next workflow run
Renovate keeps pins fresh automatically; the factory health monitor surfaces failing pipelines quickly; and the production gate plus @v1 human authorization keep consumers safe.
Troubleshooting#
| Symptom | Cause | Fix |
|---|---|---|
| Environment gate never appears | production Environment not configured in GitHub UI | Follow the Manual GitHub UI setup steps above |
| Both reviewers approved but job didn't start | GitHub Environments cache can take ~30s to register approvals | Wait 30s and refresh the Actions run page |
When to Use#
Use this skill when:
- Setting up or auditing the machine-enforced 2-human production approval gate in consumer repositories.
- Modifying promotion schedules, cadence, or merge-queue integration across bluefin or dakota.
- Investigating factory health monitoring alerts or failure issue generation in
projectbluefin/common. - Debugging or configuring Renovate dependency updates and automated merge rules for first-party or third-party pins.
- Verifying the end-to-end promotion PR format (Design C) and checklist markers.
When NOT to Use#
Do not use this skill to:
- Modify individual composite action implementations (use
composite-actions.md). - Bypass the 2-human approval gate or force promotions directly to stable without verification.
- Manually edit generated promotion PRs while automation is active.
Core Process#
- Production gate enforcement: Configure GitHub Environment
productionwith 2 required maintainer reviewers; ensure promotional workflows declareenvironment: production. - Promotion workflow orchestration: Validate that weekly promotions lock the main HEAD SHA, execute full e2e testsuites, and post structured Design C promotion PRs.
- Merge-queue compliance: Follow the merge-queue contract (
use_merge_queue), ensuring queue entry guards prevent out-of-order race conditions. - Health monitoring: Maintain the 6-hour scheduled health check; confirm automated alerts fire when pipeline success rates drop below 80%.
- Renovate automation: Configure Renovate presets and package rules; ensure first-party references are ignored and third-party SHA bumps auto-merge upon passing CI.
Common Rationalizations#
| Rationalization | Reality |
|---|---|
| "Only one maintainer is available, so bypass the environment gate." | The 2-human rule is an intentional safety invariant preventing single-point compromise or accidental releases. |
| "Promotion can skip e2e tests because testing passed yesterday." | Builds drift constantly with upstream packages; promotional gates require explicit e2e verification of the exact SHA. |
| "A single failing run isn't worth investigating." | Repeated silent failures degrade pipeline health until mass breakages occur. |
| "Renovate auto-merge doesn't need verification if actionlint passed." | Auto-merge can fail silently if GitHub App credentials or branch protection rules are misconfigured. |
Red Flags#
- Production promotion jobs executing without an
environment: productionblock. - Self-approval or single-human approvals pushing images to the
:stabletag. - Disabling
run_e2eor promotion gates without explicit maintainer sign-off. - Renovate creating duplicate PRs to pin first-party
projectbluefin/actionsreferences. - Missing
MERGERAPTOR_APP_IDor private keys leading to silent auto-merge workflow failures. - A
gh run listin the health monitor bounded only by--limit, with the time window applied
afterwards injqā the sample then depends on run volume, not on the window, and truncation
is silent. - Factory alert issues open for a pipeline that has been green for days ā the pipeline is muted,
not healthy-and-noisy.
Verification#
- GitHub Environment
productionis configured with 2 required maintainer reviewers. - Weekly promotion workflows specify
environment: production. - Merge queue configuration matches each repository's promotion contract (
use_merge_queue). - Factory health monitor scheduled runs complete and create alert issues when failure thresholds are crossed.
- Factory health monitor closes an alert issue once its pipeline is back above the threshold, so the
dedupe check cannot mute that pipeline. - Renovate auto-merge workflow runs with valid GitHub App authentication and branch protection bypasses.
- Factory health
gh run listcalls pass--created ">=${CUTOFF_ISO}"and warn when the
RUN_FETCH_LIMITcap saturates (tests/test_factory_health_fetch.py).