Metric types: counters, gauges, and histograms

Different questions about the same underlying event need genuinely different storage shapes. "How many requests have we served, total?" "How many database connections are open right now?" "What's the 95th percentile response time?" — each needs a different metric type, and using the wrong one for a given question either can't answer it at all or answers it wrong.

A counter tracks a cumulative total that only increases (until a process restart resets it to zero) — total requests served, total errors, total bytes sent:

A counter alone answers "how many, in total" — turning it into "how many per second right now" (the actually useful, alertable question) requires computing the rate of change over time, not reading the raw counter value directly. This is exactly what a query like Prometheus's rate(http_requests_total[5m]) does: the counter itself just accumulates, and the query layer derives the rate from it.

A gauge records a value that can move in either direction at any time — currently active connections, current queue depth, current memory usage:

Unlike a counter, a gauge's raw value is already meaningful on its own — "47 active connections right now" doesn't need a rate calculation to be useful, because the current value is the answer to the question being asked.

"Average response time: 120ms" sounds reassuring and can hide a real problem: if 95% of requests take 50ms and 5% take 2 seconds, the average (120ms) looks fine while a meaningful share of real users are having a genuinely bad experience. A histogram records values bucketed by range, preserving the actual distribution's shape instead of collapsing it into one number:

From bucketed counts like these, a query layer can compute percentiles — p50 (median), p95, p99 — which answer "how bad is the worst 5% (or 1%) of requests," a question a plain average can never answer honestly.

A service handling 10,000 requests where the average latency is excellent but p99 latency is terrible means roughly 100 real users had a bad experience — invisible in the average, glaring in the p99. Most production alerting is built on percentiles (p95, p99) specifically because they surface the tail of bad experiences that an average mathematically smooths away.

Adding labels to a metric (http_requests_total{user_id="42", ...}) lets you slice data more precisely, but each unique combination of label values creates a separate time series to store. A label like user_id on a system with millions of users creates millions of time series from one metric definition — a real, common way metrics systems become prohibitively expensive or simply stop working. Labels should describe a bounded, small set of values (status_code, region, method) — anything with effectively unbounded cardinality (a user ID, a raw URL with query parameters) belongs in a log or a trace attribute instead, not a metric label.