Designing alerts that don't cause fatigue

An on-call engineer who gets paged for things that don't need immediate human action learns, correctly, that most pages aren't urgent — and starts responding slower, or dismissing pages reflexively, including the rare one that actually matters. This is alert fatigue, and it's usually a bigger practical threat to reliability than having too few alerts, because a team that's stopped trusting its alerts is functionally back to having no monitoring at all, just with more noise.

An alert should fire only when something requires human action now — not simply when something notable happened. "CPU is at 85%" might be completely normal for this service at this time of day and need no action at all; "error rate exceeded the SLO's error budget burn rate" ( lesson 5) requires an actual decision. A useful discipline: for every alert, explicitly answer "what should the on-call person actually do when this fires?" — if the honest answer is "nothing, it'll resolve itself" or "just note it for later," it shouldn't be a page, at most a non-urgent notification (a Slack message, a ticket) reviewed on a normal schedule instead of an interrupt demanding immediate attention.

Alerting directly on every possible internal cause (this specific database's CPU, that specific queue's depth, one particular downstream dependency's latency) produces a proliferation of alerts, many overlapping, most not actually indicating user impact by themselves. Alerting on user-facing symptoms instead — the RED metrics from lesson 3, the SLO burn rate from lesson 5 — catches the same underlying problems with far fewer, more meaningful alerts, and naturally avoids the worst-of-both-worlds case where five different cause-based alerts all fire for the exact same single underlying incident.

A naive SLO-based alert ("error budget will be exhausted in 30 days at the current rate") reacts far too slowly to a sudden, severe spike — by the time it fires, real damage has already accumulated. A more robust pattern checks the error rate over multiple time windows simultaneously — a short window (5 minutes) catches a sudden severe spike quickly, while a longer window (1 hour, 6 hours) catches a slower, more gradual degradation the short window alone would miss entirely (and might trigger too many false positives on its own, given normal short-term noise). Requiring elevated error rates across both a short and a long window before paging balances fast detection against false positives from brief, self-resolving blips.

Routing everything to the highest severity by default (when a lower one would genuinely do) is one of the most common, avoidable causes of alert fatigue — the fix isn't fewer alerts overall, it's correctly matching each alert's urgency to its actual, honest severity.

A well-designed alert links directly to a runbook — a short, concrete guide for that specific alert covering what it means, likely causes, and first diagnostic steps. This isn't the same as Software Engineering Fundamentals' postmortem practice (which happens after an incident, to learn from it) — a runbook exists during an active alert, to make the immediate response faster and less dependent on whoever happens to be on-call already knowing the system's history from memory.