Cloud Computing's SQS lesson already covered the dead-letter-queue escape hatch for messages that keep failing. The part that lesson didn't go deep on is the retry itself — how long to wait between attempts, and why the naive version of retrying makes outages worse, not better.
3 min read
A poison message is one that fails every single time it's processed, no matter how many times it's retried — a malformed payload a consumer can't parse, a reference to an entity that was since deleted, a bug in the consumer's own code triggered by this specific input. Unlike a transient failure (a downstream API briefly down, a database connection blip, something that resolves if you just wait and try again), a poison message will keep failing until a human fixes either the message or the consumer — retrying it automatically forever just wastes resources and, worse, can block every message behind it if ordering matters for that key (lesson 4).
def consume(message):
while True:
try:
process(message)
return
except Exception:
retry_immediately() # no delay at allRetrying instantly and indefinitely is the retry strategy most likely to make an actual outage worse: if a downstream service is struggling under load and starts failing requests, instant retries from every consumer pile more load onto the already-struggling service, making recovery slower or impossible — a self-inflicted version of the thundering-herd problem this curriculum's caching domain covers from a different angle.
import random
def retry_delay(attempt):
base = min(2 ** attempt, 60) # exponential: 1s, 2s, 4s, 8s... capped at 60s
return base + random.uniform(0, 1) # jitter: spread out retries that would otherwise sync upExponential backoff waits progressively longer between each retry attempt, giving a struggling downstream system real room to recover instead of being hit immediately and repeatedly. Jitter — adding a small random amount to each delay — solves a second, subtler problem: without it, many consumers that all failed at the same moment (because the same downstream outage affected all of them at once) would all retry at exactly the same future moment too, recreating the exact thundering-herd spike backoff was meant to prevent, just delayed.
When a downstream dependency is known to be down — not just one request that happened to fail, but a sustained pattern of failures — a circuit breaker stops even attempting calls to it for a cooldown period, failing fast instead of waiting out a full retry-with-backoff cycle on every single message. This is a consumer-side optimization on top of backoff, not a replacement for it: backoff governs the spacing of individual retries; a circuit breaker governs whether to even bother attempting one right now, based on that dependency's recent track record.
Cloud Computing's lesson covered moving a message to a DLQ after N failed attempts, full stop. The fuller picture: N should be sized around how many retries, spaced by exponential backoff, a genuinely transient failure would plausibly need to resolve within — too few retries, and a message that would have succeeded on attempt 4 gets dead-lettered prematurely; too many, and a true poison message burns significant time and resources before anyone's alerted to it. A DLQ with active monitoring (an alert the moment its depth rises above zero) closes the loop — retries handle the transient case automatically, and a human gets pulled in specifically, and only, for the genuinely poisoned one.
Check your understanding
A quick comprehension check — not tracked, not graded, just for you.
1. What makes a message "poison" rather than just a transient failure?
2. Why can instant, unlimited retries make a real outage worse?
3. What specific problem does adding jitter to exponential backoff solve?
Message Queues & Event-Driven Architecture