Common CI/CD bugs and gotchas — a field reference

A PR's CI job passes every check. The deploy job runs afterward and rebuilds the image from the same commit — except a dependency's version range let a package resolve to a patch release published in the intervening minutes, or a build-time environment variable differs between the CI runner and the deploy runner. The artifact that was tested and the artifact that gets deployed are not the same bytes.

The fix: build exactly once, tag it immutably, and have every later stage (staging, production) promote that same artifact — never rebuild — the discipline covered in this domain's artifact-versioning lesson. If a deploy job contains a docker build step, that's the bug, not a detail.

A branch protection rule lists "build-and-test" as a required status check. Someone later renames the CI job (build-and-test → ci) as part of an unrelated cleanup. GitHub can't match the new job name to the old required-check name, so the rule silently stops blocking anything — every PR shows the new job running and passing, but the merge button was never actually gated on it, because the protection rule is still looking for a check name that no longer exists.

The fix: treat a CI job's name as a stable, load-bearing identifier — renaming it requires updating branch protection in the same change, and it's worth periodically auditing that every listed required check actually corresponds to a job that still runs, since this failure mode produces zero errors anywhere.

A GitHub Actions workflow's IAM trust policy uses "sub": "repo:alice/my-app:*" instead of scoping to ref:refs/heads/main. A pull request from any branch — including one from a fork, or an internal branch nobody's reviewing yet for a work-in-progress feature — can assume the same production-deploy role. Nothing about this looks broken until a PR workflow that shouldn't have deploy permissions turns out to have them.

The fix: scope the sub condition to the exact branch (or environment) that should be allowed to deploy, and use a separate, more restricted role for PR/test workflows — covered in depth in this domain's OIDC lesson.

A test that fails roughly 1 time in 20 due to a timing race gets re-run "just in case" often enough that it becomes routine. Months later, a genuine regression in the same test file fails for a real reason — and gets re-run and merged anyway, because the team's trained response to that test failing is "probably flaky, try again," not "investigate."

The fix: quarantine flaky tests explicitly (skip + tracked issue) rather than letting them sit in the required suite training everyone to distrust red CI — the difference between a missing test and a flaky one covered in this domain's testing-gates lesson.

A rollback reverts the application code to the previous deployed version, but a feature flag was flipped to 100% as part of the release (a separate action from the deploy). The old code, now running again after rollback, still checks the same flag — and since the flag state wasn't part of what got rolled back, the "old" code takes the new code path anyway, because the flag never turned off.

The fix: when a release involves a feature flag, treat the flag's state as part of what an incident response needs to check and potentially revert — rollback of code and rollback of feature visibility are two separate levers, and forgetting the second one is exactly why the deploy/release distinction from this domain's feature-flags lesson matters operationally, not just conceptually.

Every one of these bugs shares a shape: something that looks like a single lever (deploy, merge, rollback) is actually two or three separate mechanisms that can drift out of sync — the built artifact vs. the tested one, the branch protection rule's name vs. the actual job, the trust policy's stated scope vs. its intended scope, the test suite's presence vs. whether anyone still trusts it, the code version vs. the feature-flag state. The habit that catches all five: whenever "roll this back" or "this should have been blocked" doesn't produce the expected result, check for a second mechanism that quietly wasn't reverted or enforced alongside the first, rather than assuming the obvious lever is the only one that moved.