Common AWS bugs and gotchas — a field reference

A policy grants s3:GetObject on a bucket, but requests still fail with AccessDenied. The instinctive fix — add the permission again, double-check the ARN — doesn't help, because the actual cause is elsewhere: any explicit Deny, anywhere in the evaluation (an SCP at the organization level, a permissions boundary, a separate policy attached to the same role) always wins, regardless of how many Allow statements exist. IAM's evaluation logic is: if there's an explicit Deny anywhere that applies, deny wins; otherwise, if there's an explicit Allow, it's allowed; otherwise, implicit deny (nothing granted it, so no).

The fix: when a permission "should work" but doesn't, check for an SCP (this domain's Organizations lesson) or a permissions boundary (this domain's IAM lesson) before assuming the policy itself is wrong — AccessDenied with a seemingly-correct policy almost always means an explicit Deny exists somewhere else in the evaluation chain.

An engineer updates a security group's inbound rule to allow a new port, tests it, and the connection still fails exactly as before. Security groups themselves apply changes immediately — the actual cause is almost always something else in the path: a network ACL (the coarser, stateless firewall layer from the Cloud Computing domain's networking lesson) still blocking it, a route table missing the expected route, or — very commonly — the instance actually being tested is a different instance than the one the security group was attached to (a leftover old instance from before an auto-scaling replacement, or a typo in which instance's security group was actually edited).

The fix: verify which specific instance is actually being tested (aws ec2 describe-instances to confirm), then check NACLs and route tables before assuming the security group edit itself failed to apply — it almost never actually fails to apply, something else in the chain is still blocking.

A Multi-AZ RDS instance fails over as designed — the standby takes over, the endpoint DNS repoints — but the application doesn't recover cleanly, throwing connection errors for longer than the failover itself actually took. The root cause is usually connection pooling: application code holding open connections to the old primary's resolved IP address (cached at connection-establishment time) doesn't automatically know to reconnect using the new DNS resolution, even though the endpoint name didn't change.

The fix: ensure the database client library/connection pool has reasonable connection recycling (max connection lifetime, or retry-on-connection-error logic) rather than assuming a connection, once established, stays valid forever — Multi-AZ failover is exactly the kind of event that invalidates that assumption without any application-level signal beyond the connection simply breaking.

A bucket policy explicitly grants public read access, yet requests still get 403 Forbidden. The near-universal cause: Block Public Access settings (covered in this domain's S3 lesson) are still enabled, either at the bucket level or the account level — and Block Public Access overrides what the bucket policy says, by design, specifically to prevent exactly this kind of accidental-then-corrected public exposure from silently taking effect without a second deliberate step.

The fix: Block Public Access must be explicitly disabled (both at the bucket and, if set, the account level) in addition to writing the permissive bucket policy — the bucket policy alone is never sufficient on its own, which is a deliberate two-key safety mechanism, not a bug in the mechanism itself.

A CloudFormation update fails partway through, CloudFormation attempts an automatic rollback to the previous known-good state, and the rollback itself fails — leaving the stack in UPDATE_ROLLBACK_FAILED, a state where no further updates can be applied until it's explicitly resolved. This typically happens when a resource that partially succeeded during the failed update can't cleanly revert (e.g., a resource that was already deleted and can't come back, or a resource now in a state the rollback logic doesn't know how to reverse).

The fix: aws cloudformation continue-update-rollback, optionally with --resources-to-skip naming the specific stuck resource(s) — this tells CloudFormation to proceed with the rollback while explicitly acknowledging that resource can't be cleanly reverted, rather than the stack being permanently stuck. This is CloudFormation's version of the state-drift problem Terraform users hit differently (a state file that no longer matches reality) — same underlying category of issue, different tool-specific recovery path.

Every one of these bugs shares a shape: the obvious, single-layer explanation (the policy, the security group rule, the DNS endpoint, the bucket policy, the stack update) isn't actually wrong — there's a second, less-visible mechanism (an SCP, a NACL, a stale connection, Block Public Access, an unrevertable resource) that's the real cause. The habit that catches all five faster: when a change to the "obvious" layer doesn't produce the expected result, look for the second mechanism operating alongside it, rather than re-checking the first layer over and over assuming it must be misconfigured.