A dashboard with forty panels showing everything a service could possibly emit isn't more informative than one showing the right six — it's actually less useful, because during an actual incident, nobody has time to scan forty panels looking for the one that matters.
3 min read
A dashboard built during a calm afternoon, by someone with time to add "just one more useful panel," tends to accumulate panels indefinitely. A dashboard opened during an active incident, by someone with a pager going off, needs the opposite: the smallest number of panels that answer "is this the problem, and roughly where" as fast as possible. These are different design goals, and optimizing for the first one actively hurts the second.
Lesson 3's RED metrics (for request-driven services) and USE metrics (for resources) are a genuinely strong default for a service's primary dashboard — request rate, error rate, latency percentiles, plus the utilization/saturation of its critical dependencies. A dashboard built from this framework directly answers lesson 1's investigation sequence (is something wrong → where) without needing to scroll past a dozen panels nobody currently cares about to find the two that do.
Lesson 2 already established why an average hides the tail of bad experiences a p99 reveals directly. A dashboard that only shows average latency is actively misleading during an incident — a real p99 spike can sit invisible underneath a barely-moved average, delaying exactly the detection a dashboard exists to enable. Every latency panel on a dashboard meant for incident response should default to showing p50, p95, and p99 together, not an average alone.
A single dashboard trying to serve every audience and every question at once tends to satisfy none of them well. A common, effective structure:
Jumping straight to a highly-detailed per-service dashboard without an overview first means guessing which service to even look at; a single overview trying to cram every service's full detail into one view becomes too dense to scan quickly. The two layers serve genuinely different moments in an investigation.
A latency or error-rate graph showing a step change right after a deploy is far faster to diagnose than the same graph without any marker showing when the deploy happened — correlating "when did this start" against "when did we last change something" is one of the single most common and effective diagnostic steps, and it only works quickly if deploys (and other significant changes: a feature flag flip, a config change) are visibly annotated directly on the relevant graphs, not tracked in some separate system a responder has to remember to check.
A metric that gets renamed, deprecated, or replaced leaves a dashboard panel showing stale or broken data — quietly, without an error, unless someone notices during an actual incident (the worst possible time to discover a dashboard doesn't work). Periodically reviewing dashboards for panels that no longer show meaningful data, the same way stale code gets refactored or deleted, keeps a dashboard trusted rather than accumulating quiet rot that only becomes visible under pressure.
Check your understanding
A quick comprehension check — not tracked, not graded, just for you.
1. Why can a dashboard with forty panels be less useful during an incident than one with six?
2. Why should every latency panel default to showing percentiles rather than just an average?
3. Why should deploys be annotated directly on dashboard graphs?
Observability & Monitoring