Competing consumers, backpressure, and consumer lag

The competing consumers pattern is the straightforward fix for "one consumer can't keep up": run multiple consumer instances reading from the same queue (or, in Kafka, the same consumer group), each competing for the next available message. Throughput scales roughly linearly with consumer count, up to the point where something else becomes the bottleneck — the number of partitions in Kafka (lesson 2 established a partition can only be actively read by one consumer per group at a time), a downstream database's own write capacity, or a third-party API's rate limit.

Adding a 4th worker doesn't help once the shared database behind all three becomes the actual limiting factor — competing consumers only scales the part of the pipeline that's genuinely the bottleneck, and a real capacity plan has to identify which part that actually is before adding more workers.

Consumer lag is the gap between the latest message produced and the latest message a given consumer (or consumer group) has actually processed — in Kafka terms, the difference between the partition's latest offset and the consumer group's committed offset. A small, stable lag means consumers are keeping up in real time; a lag that's growing over time means production is outpacing consumption, and the gap will keep widening indefinitely unless something changes (more consumers, faster processing, or less incoming volume).

Monitoring consumer lag as an actual metric (not just watching for outright consumer crashes) is how a real system catches "processing is too slow for current volume" before it turns into hours of backlog and stale data — this is exactly the kind of signal Observability & Monitoring's SLO/alerting lessons are about choosing and acting on.

Backpressure is the general principle of a system's slower stage signaling "slow down" to a faster stage upstream, rather than the faster stage blindly producing work the slower stage can't absorb. A queue is, itself, a form of backpressure — it absorbs a burst without the producer needing to know anything changed — but a queue has limits too: if it grows faster than consumers can drain it indefinitely, something further upstream eventually needs to actually throttle, not just buffer — rejecting new requests at the API boundary, applying the rate-limiting techniques System Design already covered, rather than accepting unlimited additional work.

Adding consumers only helps up to the partition-count ceiling (Kafka) or until a shared downstream resource saturates — past that point, more consumers add contention without adding real throughput, and the actual fix is either increasing partition count (a disruptive, upfront decision per lesson 2) or addressing whatever the true bottleneck is directly. Diagnosing which of these is actually happening — partition ceiling, downstream bottleneck, or genuinely insufficient consumer count — is what separates a real fix from a guess.