Testing — Python Unit Tests#
Scope#
The unit-tests.yml workflow covers two test suites:
Python (pytest)#
| Directory | Scripts |
|---|---|
bootc-build/create-release/scripts/ | render_card.py, render_notes.py, sbom_diff.py |
bootc-build/chunka/ | inject-xattrs.py |
scripts/ | check-consumer-contract.py, monitor_pipeline.py, render_gate_section.py, render_pr_body.py |
All directories are registered in tests/conftest.py via sys.path.insert so test files can
import scripts directly without per-file path manipulation.
Path precedence rule for duplicate script names#
Some scripts/*.py modules are duplicated under .github/actions/* so composite actions can run
them via GITHUB_ACTION_PATH. If both directories are added to sys.path, insertion order decides
which copy gets imported.
- Add
scripts/before.github/actions/*when coverage should be attributed to shipped wrappers. - For regression tests that must target a specific file, prefer
importlib.util.spec_from_file_location
with an explicit path to avoid accidental shadowing fromsys.pathorder.
Coverage gate: --cov-fail-under=75
Consumer-contract guard: an empty live set still fails#
scripts/check-consumer-contract.py keeps external consumers (aurora, bazzite) from losing
required action inputs. get_live_inputs() returns an empty set when an action or reusable
workflow declares no inputs — that is a valid interface when the snapshot requires nothing, but
it is also the exact shape produced when the last required input is deleted.
check_contract() must compare the snapshot's required inputs against that live set and fail on
any miss. It must not return early (or skip) when the live set is empty: doing so turns a
removed required input into a silent pass. The guard keeps its missing-file diagnostic for files
that are gone, but for a file that exists it iterates the required inputs against whatever is
live — empty or not. An interface the snapshot requires nothing of still passes, because the
required list is then empty and the loop records no failures.
Regression coverage exercises all three "no inputs" shapes — absent inputs: key, inputs: {}
empty map, and inputs: with a null value — for both composite actions and reusable workflows,
plus a zero-required-input interface that must keep passing. See
tests/test_check_consumer_contract.py (TestEmptyInputGuard) and
tests/test_check_consumer_contract_cli.py.
Shell scripts (bats)#
| Script | Test file |
|---|---|
bootc-build/generate-tags/generate_tags.sh | tests/bats/test_generate_tags.bats |
actions/check-token-health/check_token_health.sh | tests/bats/test_check_token_health.bats |
scripts/resolve_digests.sh | tests/bats/test_resolve_digests.bats (10 tests) |
scripts/verify_signatures.sh | tests/bats/test_verify_signatures.bats (15 tests) |
scripts/release_gate.sh | tests/bats/test_release_gate.bats |
detect-changes image_flavors shell logic | tests/bats/test_detect_changes.bats (8 tests) |
push-image push/retry/alias shell logic | tests/bats/test_push_image.bats (16 tests) |
sign-and-publish keyless/key validation + SBOM attach/cache/path guards | tests/bats/test_sign_and_publish.bats (19 tests) |
setup-runner native-overlay setup & podman source selection | tests/bats/test_setup_runner.bats (16 tests) |
chunka config temp-file creation + fs.protected_regular drift guard | tests/bats/test_chunka.bats (4 tests) |
chunka chunkah pull retry loop (must fail closed, not fall through on the last attempt) | tests/bats/test_chunka_retry.bats (6 tests) |
create-release SBOM validation, previous-SBOM fetch, tag parsing, card-rendered gating, asset list | tests/bats/test_create_release.bats (27 tests), kept in sync with the action by tests/test_create_release_bats_sync.py |
factory-health.yml pipeline status classification (threshold, sample floor, failure streak) | tests/bats/test_factory_health_status.bats (12 tests), kept in sync with the workflow by tests/test_factory_health_classify_sync.py |
reusable-renovate-automerge.yml check-rollup classification + head-moved stop | tests/bats/test_renovate_automerge_checks.bats (12 tests), kept in sync with the workflow by tests/test_renovate_automerge_bats_sync.py |
reusable-renovate-automerge.yml PR-lookup / qualification matcher | tests/bats/test_renovate_automerge_find_pr.bats (10 tests), kept in sync with the workflow by tests/test_renovate_automerge_bats_sync.py |
renovate-automerge-wiring.yml wiring assertion (scripts/renovate-automerge-wiring-check.sh) | tests/bats/test_renovate_automerge_wiring_check.bats (10 tests) |
reusable-renovate-automerge.yml merge step: --match-head-commit pin, moved-head skip (409 or changed headRefOid), gh pr view --json state outcome reporting | tests/bats/test_renovate_automerge_merge.bats (19 tests), kept in sync with the workflow by tests/test_renovate_automerge_bats_sync.py |
| Shared merge-label policy and full Renovate/promotion merge paths | tests/test_merge_label_guard.py: holds, pagination, owned withdrawal, API failure, late-label races, retries, and unheld enrollment |
The Node suite runs in the node job in unit-tests.yml:
node --test tests/node/*.test.mjs
The bats suite runs in the bats job in unit-tests.yml. Run locally:
CI installs bats from .github/tools/bats/package-lock.json with
npm ci --ignore-scripts and adds its local binary directory to GITHUB_PATH.
Update the package manifest and regenerate the lock together when bumping bats.
bats tests/bats/test_resolve_digests.bats
bats tests/bats/test_verify_signatures.bats
# or all at once:
bats tests/bats/
Running tests locally#
cd ~/src/actions
python -m pytest tests/ -v --tb=short \
--ignore=tests/bats \
--cov=bootc-build/chunka \
--cov=bootc-build/create-release/scripts \
--cov=scripts \
--cov-report=term-missing \
--cov-fail-under=75
Raising the coverage threshold#
- Add tests in
tests/for the uncovered paths - Run locally and confirm the new total
- Update
--cov-fail-under=<new_value>in.github/workflows/unit-tests.yml - Set the threshold to the actual measured value — never to a round number that isn't yet reached
Do not set --cov-fail-under above the measured total — it will immediately fail CI on the
next run and block every PR.
Bats test patterns for shell scripts#
For nonempty signing-key validation, use an obviously test-only string, not PEM
private-key delimiters: even fake PEM trips the native credential scanner. Temp
repository tests should isolate inherited host hooks; production commits retain
the repository's hooks and native review/queue gates.
Mock external binaries via PATH injection#
Prepend a temp bin/ dir to PATH in setup() and write mock scripts there. This intercepts
skopeo, cosign, curl, etc. without system modification:
setup() {
TEST_TMP=$(mktemp -d)
export GITHUB_OUTPUT="${TEST_TMP}/github_output"
touch "$GITHUB_OUTPUT"
export MOCK_DIR="${TEST_TMP}/bin"
mkdir -p "$MOCK_DIR"
export PATH="${MOCK_DIR}:${PATH}"
}
make_skopeo_success() {
local digest="${1:-sha256:abc123}"
cat > "${MOCK_DIR}/skopeo" <<EOF
#!/usr/bin/env bash
echo "${digest}"
EOF
chmod +x "${MOCK_DIR}/skopeo"
}
Heredoc quoting rule for mocks with dynamic paths#
- Unquoted
<<EOF—${VAR}in the heredoc expands at write-time (when the test creates
the mock file). Use for file paths determined in the test (e.g.${CALL_COUNT_FILE}). - Quoted
<<'EOF'— everything is literal. Use for self-contained mocks where$1,$2,
etc. should be evaluated at run-time by the mock script.
Mixing these incorrectly is the most common source of flaky bats mocks.
Scripts that exit 0 on partial failure#
Some scripts (e.g. resolve_digests.sh) always exit 0 and use GITHUB_OUTPUT to communicate
pass/fail. Do not assert $status for the pass/fail signal — assert the output variable:
# Wrong — exits 0 even when skopeo fails
[ "$status" -ne 0 ]
# Correct
[ "$(get_output ok)" = "false" ]
This is by design: GitHub Actions steps exit-gate on non-zero, but release-gate scripts need
to continue past individual failures to collect all results before deciding.
Reading GITHUB_OUTPUT in bats#
Add a helper to extract named values:
get_output() {
local key="$1"
grep "^${key}=" "$GITHUB_OUTPUT" | head -1 | cut -d= -f2-
}
For multiline values written with <<EOF delimiters, read the raw file and use == glob
matching instead:
[[ "$(<"$GITHUB_OUTPUT")" == *"bluefin|sha256:deadbeef"* ]]
Testing shell logic embedded in action YAML#
Composite action steps often have non-trivial shell logic inline in action.yml with no
corresponding standalone script file. The canonical approach:
-
Capture the snippet verbatim in a bats variable — this serves as a change detector:
any edit to the action that alters testable behavior must update the test."Verbatim" is a claim the repo must be able to check. A copy guarded only by a comment
drifts silently: the bats file keeps passing against the old rules while the action or
workflow moves on. Fence the copy with begin/end markers and add a companion test that
parses the source and asserts the two agree —tests/test_factory_health_classify_sync.py
does this fortests/bats/test_factory_health_status.bats, comparing the dedented block
againstmonitor_pipeline()infactory-health.ymland cross-checking the thresholds the
batssetup()exports. Put that companion test in pytest, not bats, and make sure
unit-tests.ymllists the source file in itspathsfilter, so editing the source alone
is enough to run it.scripts/renovate-automerge-wiring-check.shis the same idea applied
to repository settings.
# In the bats file, at the top:
PUSH_LOGIC='
set -euo pipefail
...
# exact shell from action.yml run: block
'
- Run it with
bash -cin each test, setting env vars to simulate inputs:
@test "empty TAGS exits with error" {
export TAGS=""
run bash -c "$PUSH_LOGIC"
[ "$status" -ne 0 ]
[[ "$output" == *"At least one tag"* ]]
}
-
Mock external binaries via
PATHinjection (see "Mock external binaries" above). -
For best-effort commands (
... || true), assert they stay non-fatal even when the mocked
command fails, and add a companion test that verifies the command still executes with expected
arguments. This catches accidental removal of|| truewithout turning permissive steps into
silent no-ops.
When the action has sudo calls: add a pass-through sudo mock so tests run without
privilege. The mock just calls "$@", then mock the real tool (buildah, podman) separately.
Testing a retry loop that must fail closed (not fall through): a loop whose last iteration ends
in [[ "$cond" ]] && sleep N returns non-zero without triggering errexit, so execution runs on
past the loop as if it succeeded — the classic silent-failure bug. Running the extracted loop on
its own is not enough: when the loop is the final statement the script exits non-zero anyway,
coincidentally "passing" the buggy version. Append a sentinel line (e.g. echo PAST_LOOP) after the
loop and assert it does not appear when the loop is exhausted — that proves the loop stops
instead of falling through. See tests/bats/test_chunka_retry.bats.
Quoting pitfall: embedding single-quoted shell inside a double-quoted heredoc requires
escaping with '"'"'. Use '"'"' to produce a literal ' inside the shell snippet string:
# Produces: echo 'image_flavors=["main"]' >> "$GITHUB_OUTPUT"
SNIPPET='echo '"'"'image_flavors=["main"]'"'"' >> "$GITHUB_OUTPUT"'
Alternatively, assign the snippet via a POSIX $(cat <<'EOF'...EOF) block in setup()
if the quoting becomes unmanageable.
Adding a new script#
When a new Python script is added to any covered directory:
- Add a corresponding
tests/test_<scriptname>.py - Verify the new file appears in the coverage report (
--covpaths are directory-level) - Re-measure total coverage — if it dropped, add tests to compensate before raising a PR
Hyphenated filenames (e.g. inject-xattrs.py)#
Python cannot import inject-xattrs directly. Use importlib:
import importlib
inject_xattrs = importlib.import_module("inject-xattrs")
Register the script's directory in tests/conftest.py (preferred) rather than inline in the
test file — keeps the path boilerplate in one place for all current and future tests in that
directory.
Calling main() in tests#
For scripts that read sys.argv, patch argv rather than calling main with arguments:
def _call_main(module, *args):
orig = sys.argv
try:
sys.argv = ["script-name"] + list(args)
return module.main()
finally:
sys.argv = orig
For entrypoint coverage, also exercise the module as a script with runpy.run_path(..., run_name="__main__") and assert SystemExit for CLI-style exits. This covers the if __name__ == "__main__" guard without needing a full end-to-end invocation:
import runpy
import sys
orig = sys.argv
try:
sys.argv = ["script-name", "--help"]
with pytest.raises(SystemExit) as excinfo:
runpy.run_path("/path/to/script.py", run_name="__main__")
assert excinfo.value.code == 0
finally:
sys.argv = orig
Do not use __wrapped__ introspection — it only applies to functools.wraps-decorated functions
and silently has no effect on plain functions.
Hash-locked Python requirements#
The loose test requirements live in tests/requirements-test.in; the shared
workflow parser requirement lives in scripts/requirements.in. Regenerate their
hash-locked files for the CI interpreter after changing either input:
uv pip compile --generate-hashes --python-version 3.15 --python-platform linux tests/requirements-test.in -o tests/requirements-test.txt
uv pip compile --generate-hashes --python-version 3.15 --python-platform linux scripts/requirements.in -o scripts/requirements.txt
Install with pip install --require-hashes -r <lockfile>. The issue-policy
preview uses the lock from its checked-out first-party runtime, never caller data.
The preview uses Python 3.14 and requires wheels, avoiding unhashed isolated
build dependencies. For older runtimes without a lock, it pins the Linux x86-64
PyYAML wheel and its hash inline; update that fallback when changing the parser
or preview interpreter.
pytest-cov ships as a pytest plugin — no extra import needed in test files.
Required checks must be reachable#
A required status check on main must be able to report on every ref the
ruleset evaluates, or it does not gate — it deadlocks.
Two ways a path-filtered workflow breaks this:
pull_requestwith apaths:filter. A PR outside those paths never
schedules the workflow, the required context never reports, and the PR sits
atmergeStateStatus: BLOCKEDno matter how many approvals it has.
gh pr mergewill not even enqueue it — it silently arms auto-merge
(autoMergeRequestset,mergeQueueEntry: null) and waits forever.- No
merge_group:trigger. A merge queue using theALLGREENstrategy
waits for required checks on the merge-group ref. If no workflow listens
formerge_group, entries sit inAWAITING_CHECKSuntil the queue timeout
and never merge.
Filter the work, not the trigger. Let the workflow start on every PR and
skip expensive internals with a step-level if:; a job that runs and reports
success in seconds is what satisfies the ruleset.
Use native annotations when API-based reporters fail their final check-run
PATCH. Keep every required trigger, including push and merge_group:
reporter: github-annotations
fail_level: error
github-annotations writes runner logging commands instead of creating and
patching a second check through the Checks API. The native job still reports
the required Lint GitHub Actions context. Set fail_level: error explicitly:
the action passes its default fail_level: none to the CLI, which overrides
the deprecated fail_on_error fallback. Reviewdog scans without diff filtering for non-PR
events and sends no API requests on that path; its existing token input is
still required by the CLI.
The same reporter preserves PR diff filtering and needs only read access to
fetch that diff. Remove checks: write when no step writes to the Checks API;
do not broaden permissions to repair a reporter's final PATCH 404.
See Reviewdog's annotation reporter.
Diagnosing it: compare mergeStateStatus between an affected PR and one that
touches the filtered paths. Both SUCCESS on statusCheckRollup while only one
is CLEAN means a required context is absent, not failing.
gh pr view <n> --json reviewDecision,mergeStateStatus
gh api graphql -f query='{repository(owner:"projectbluefin",name:"actions"){
pullRequest(number:<n>){ mergeQueueEntry{state} autoMergeRequest{mergeMethod} }}}'
grep -rl merge_group .github/workflows/ # zero matches is the tell
When to Use#
Use this skill when changing Python scripts, shell logic in actions, unit-test workflows, test
coverage, or mocks for external commands.
When NOT to Use#
Do not use this skill as proof of end-to-end consumer or QEMU behavior; those require workflow
and environment checks described by their respective runbooks.
Core Process#
- Locate shipped entry point and its existing test or coverage path.
- Add focused regression coverage for changed branches and preserve exact embedded shell when
testing action YAML. - Mock external commands through a temporary PATH and isolate filesystem and environment state.
- Run smallest relevant pytest or bats selector, then repository gate when shared test
infrastructure or coverage changed.
Common Rationalizations#
- "The wrapper is covered." Confirm coverage targets the file CI actually executes, not a
duplicate module selected bysys.path. - "The command failed, so the test passed." Assert output variables when scripts intentionally
return zero after partial failures. - "A broad mock is simpler." Broad mocks hide argument and ordering regressions; assert calls.
Red Flags#
- Tests invoking real host
rpm,podman,buildah,skopeo, orcosign. - Coverage thresholds raised above the measured total.
- Shell snippets copied loosely instead of captured from the action definition.
- Tests depending on current working directory, host privileges, or persistent environment state.
Verification#
- Changed entry points and failure branches have focused tests.
- External tools and privileged calls are isolated with PATH mocks.
- Relevant pytest/bats command passes with configured coverage gate.
- Tests fail when guarded behavior is intentionally reverted.