Docker & Containers' logging lesson covered how logs actually get shipped off a container. This lesson is the layer above that: which events actually deserve a log line at all, what level to give them, and — the question that trips up more systems than either — whether something should even be a log in the first place, instead of a metric.
3 min read
A log level (debug, info, warn, error, fatal) is a signal about urgency, not just a label:
DEBUG Detailed internal state, useful only while actively debugging
something specific — noisy, usually off in production.
INFO A normal, expected event worth recording (a request completed,
a job started) — not urgent, just informative history.
WARN Something unexpected happened, but the system recovered or
degraded gracefully — worth noticing, not worth waking anyone.
ERROR An operation failed and didn't recover on its own — worth
investigating, though not necessarily an emergency by itself.
FATAL The process itself can't continue — about to crash or shut down.
Logging routine, expected events at ERROR trains the same "alert
fatigue" response CI/CD's flaky-test lesson describes for tests: a team
that sees constant ERROR lines for things that aren't actually problems
stops trusting the word "error" in their own logs, and a genuine error
gets lost in the noise it's supposedly flagged as urgent. Logging real
failures at INFO has the opposite problem — a genuine issue never
surfaces in an error-filtered view at all, effectively hiding it. The
level has to match the actual urgency, consistently, or it stops
conveying real information.
Lesson 1 established that logs answer "what happened to this specific request" while metrics answer "is the system healthy overall." This has a direct, practical consequence for what to actually write: a log line for every request served produces an unmanageable volume at real traffic — the request-count question belongs to a counter metric instead, incremented on every request with no log line needed for the routine case. Logs are for events worth individually inspecting later (an error, an unusual state, a security-relevant action) — not a running tally of things that happened normally and don't need individual attention.
Wrong instinct: log every request at INFO
-> millions of log lines, most never read, real storage cost
Better: increment a request-count metric on every request,
log only the requests that actually failed or behaved
unusually
Even for events worth logging, logging every single occurrence isn't always necessary — a service handling 10,000 requests/second doesn't need every successful one logged individually to understand its behavior. Log sampling deliberately logs only a fraction (1 in 100, or 1 in 1000) of routine events, while still logging 100% of errors or unusual cases:
if response.status >= 400 or random.random() < 0.01:
log_request(response) # always log errors, sample 1% of everything elseThis keeps the interesting signal (every failure) fully intact while cutting the volume — and cost — of the routine, high-frequency case that doesn't need individual inspection.
Docker & Containers' logging lesson introduced correlation IDs mainly as something a log-shipping pipeline carries along. The discipline that actually makes them useful happens earlier: every log line an application writes for a given request needs to actually include that ID as a structured field, consistently, from the first line to the last — a correlation ID that only appears in some of a request's log lines (because a developer forgot to pass it into one function's logging calls) breaks the ability to reconstruct the full request later, silently and without any error telling you it happened.
Check your understanding
A quick comprehension check — not tracked, not graded, just for you.
1. What happens when a team logs routine, expected events at the ERROR level?
2. Why shouldn't every single successful request be logged individually at high traffic volume?
3. Why does log sampling still log 100% of errors while only sampling a fraction of routine events?
Observability & Monitoring