Container health checks: liveness vs readiness

A container can be running and still be useless — stuck in a deadlock, or up but not yet finished loading its config. 'Is the process alive' and 'is the process ready to serve traffic' are two different questions, and answering only one of them is how zombie containers keep receiving requests they can't handle.

Intermediate

4 min read

Two different questions, one easy-to-conflate answer

Orchestrators (Docker itself, ECS, Kubernetes) all ask some version of "is this container okay?" — but there are really two separate questions hiding inside that:

Liveness: is the process still running and functioning, or is it stuck/deadlocked/crashed in a way that only a restart fixes?
Readiness: is the process currently able to serve a request right now, even if it's fundamentally healthy? (Still loading a large in-memory cache on startup, temporarily overloaded, or its database connection just dropped.)

A container can be alive but not ready (booting up, warming a cache) — restarting it would be actively counterproductive, it just needs traffic held back for a few more seconds. A container can also be ready-looking but not actually alive — the process technically responds to a socket connection but is wedged in an infinite loop doing nothing useful. Answering only one question misses the other failure mode entirely.

`HEALTHCHECK` in Docker itself

Docker's own HEALTHCHECK instruction is a single check — closer to a liveness signal than the full liveness/readiness split, but still meaningfully better than nothing:

HEALTHCHECK --interval=30s --timeout=5s --start-period=15s --retries=3 \
  CMD curl -f http://localhost:3000/health || exit 1

--interval — how often to check.
--timeout — how long to wait for the check itself before considering it failed.
--start-period — a grace window after container start where failures don't count yet (so a slow-booting app isn't marked unhealthy before it's had a fair chance to come up).
--retries — consecutive failures required before the container is marked unhealthy.

docker ps then shows (healthy) or (unhealthy) next to the container, and Compose's condition: service_healthy (from the previous lesson) reads exactly this status.

Writing a `/health` endpoint that actually means something

The trap: a /health endpoint that always returns 200 regardless of the app's actual state answers "is the process running" — trivially true unless the process has already crashed — while telling you nothing about whether it can serve real requests.

// Weak — this "passes" even if the database is unreachable
app.get("/health", (req, res) => res.status(200).send("OK"));

// Meaningful — actually checks the dependency that matters
app.get("/health", async (req, res) => {
  try {
    await db.query("SELECT 1");
    res.status(200).json({ status: "ok" });
  } catch {
    res.status(503).json({ status: "database unreachable" });
  }
});

This is the same principle the load balancers and health checks lesson covers at the infrastructure layer — a process that's technically up but can't reach its dependencies should say so, so whatever's routing traffic (a load balancer, an orchestrator) stops sending it real requests instead of routing users into a dead end.

Splitting liveness and readiness explicitly

Orchestrators built for this (Kubernetes, covered in the next lesson) let you define the two checks separately, with different consequences:

Liveness probe fails → the orchestrator kills and restarts the container. Reserved for "this process is fundamentally broken and a restart is the only fix" — a deadlock, an unrecoverable internal error state.
Readiness probe fails → the orchestrator stops routing new traffic to this container, but does not restart it. Used for "temporarily can't serve requests" — still warming up, briefly overloaded, a dependency is down but might recover on its own.

Using the same check for both is the most common mistake: a liveness check that fires on "database is briefly slow" causes the orchestrator to restart a container that didn't need restarting — which doesn't fix a slow database, and now there's also a cold-starting container making things worse during an already-degraded moment.

Getting the timing wrong is its own failure mode

Two timing mistakes show up constantly in practice:

start-period (or the equivalent initial-delay setting) too short: a genuinely slow-booting app (loading a large model, warming a cache) gets marked unhealthy and restarted before it ever finished starting — a boot loop that never succeeds, entirely self-inflicted by an impatient health check.
Check interval too long relative to how fast things go wrong: a health check every 60 seconds means up to 60 seconds of real users hitting a container that's actually been broken since 2 seconds after the last check. The interval should be short enough that the blast radius of "broken but not yet detected" is acceptable for the traffic this service actually gets.

Container health checks: liveness vs readiness

Intermediate

4 min read

Two different questions, one easy-to-conflate answer

Orchestrators (Docker itself, ECS, Kubernetes) all ask some version of "is this container okay?" — but there are really two separate questions hiding inside that:

Liveness: is the process still running and functioning, or is it stuck/deadlocked/crashed in a way that only a restart fixes?
Readiness: is the process currently able to serve a request right now, even if it's fundamentally healthy? (Still loading a large in-memory cache on startup, temporarily overloaded, or its database connection just dropped.)

`HEALTHCHECK` in Docker itself

Docker's own HEALTHCHECK instruction is a single check — closer to a liveness signal than the full liveness/readiness split, but still meaningfully better than nothing:

HEALTHCHECK --interval=30s --timeout=5s --start-period=15s --retries=3 \
  CMD curl -f http://localhost:3000/health || exit 1

--interval — how often to check.
--timeout — how long to wait for the check itself before considering it failed.
--start-period — a grace window after container start where failures don't count yet (so a slow-booting app isn't marked unhealthy before it's had a fair chance to come up).
--retries — consecutive failures required before the container is marked unhealthy.

docker ps then shows (healthy) or (unhealthy) next to the container, and Compose's condition: service_healthy (from the previous lesson) reads exactly this status.

Writing a `/health` endpoint that actually means something

// Weak — this "passes" even if the database is unreachable
app.get("/health", (req, res) => res.status(200).send("OK"));

// Meaningful — actually checks the dependency that matters
app.get("/health", async (req, res) => {
  try {
    await db.query("SELECT 1");
    res.status(200).json({ status: "ok" });
  } catch {
    res.status(503).json({ status: "database unreachable" });
  }
});

Splitting liveness and readiness explicitly

Orchestrators built for this (Kubernetes, covered in the next lesson) let you define the two checks separately, with different consequences:

Liveness probe fails → the orchestrator kills and restarts the container. Reserved for "this process is fundamentally broken and a restart is the only fix" — a deadlock, an unrecoverable internal error state.
Readiness probe fails → the orchestrator stops routing new traffic to this container, but does not restart it. Used for "temporarily can't serve requests" — still warming up, briefly overloaded, a dependency is down but might recover on its own.

Getting the timing wrong is its own failure mode

Two timing mistakes show up constantly in practice:

start-period (or the equivalent initial-delay setting) too short: a genuinely slow-booting app (loading a large model, warming a cache) gets marked unhealthy and restarted before it ever finished starting — a boot loop that never succeeds, entirely self-inflicted by an impatient health check.
Check interval too long relative to how fast things go wrong: a health check every 60 seconds means up to 60 seconds of real users hitting a container that's actually been broken since 2 seconds after the last check. The interval should be short enough that the blast radius of "broken but not yet detected" is acceptable for the traffic this service actually gets.

Container health checks: liveness vs readiness

Two different questions, one easy-to-conflate answer

`HEALTHCHECK` in Docker itself

Writing a `/health` endpoint that actually means something

Splitting liveness and readiness explicitly

Getting the timing wrong is its own failure mode

Further reading

Container health checks: liveness vs readiness

Two different questions, one easy-to-conflate answer

`HEALTHCHECK` in Docker itself

Writing a `/health` endpoint that actually means something

Splitting liveness and readiness explicitly

Getting the timing wrong is its own failure mode

Further reading

Container health checks: liveness vs readiness

Two different questions, one easy-to-conflate answer

HEALTHCHECK in Docker itself

Writing a /health endpoint that actually means something

Splitting liveness and readiness explicitly

Getting the timing wrong is its own failure mode

Further reading

Container health checks: liveness vs readiness

Two different questions, one easy-to-conflate answer

HEALTHCHECK in Docker itself

Writing a /health endpoint that actually means something

Splitting liveness and readiness explicitly

Getting the timing wrong is its own failure mode

Further reading

`HEALTHCHECK` in Docker itself

Writing a `/health` endpoint that actually means something

`HEALTHCHECK` in Docker itself

Writing a `/health` endpoint that actually means something