Load balancers and health checks

You never run just one server for anything that matters — one server is one crash away from an outage. A load balancer is what turns a pile of identical instances into a single, resilient endpoint, and health checks are how it knows which of those instances are actually safe to send traffic to.

Beginner

3 min read

The problem: one server is a single point of failure

Run your app on one EC2 instance and give users its IP address, and the moment that instance crashes, gets redeployed, or simply gets overwhelmed, every user is down. Running multiple identical copies of your app fixes the capacity problem, but creates a new one: users need a single, stable address to hit, and something has to decide which of the N running copies handles each request.

That something is a load balancer: a piece of infrastructure that sits in front of your instances, accepts all incoming traffic, and distributes it across the healthy ones.

Layer 4 vs Layer 7

Load balancers operate at different levels of the network stack. A Layer 4 (network) load balancer routes based on IP and TCP/UDP port alone — fast, protocol-agnostic, but blind to what's inside the request. A Layer 7 (application) load balancer reads the actual HTTP request — path, headers, host — and can route /api/* to one target group and /static/* to another, terminate TLS, and inspect the response status. AWS's Application Load Balancer (ALB) is Layer 7; Network Load Balancer (NLB) is Layer 4, used when you need raw throughput or non-HTTP protocols.

Health checks: how bad instances get pulled from rotation

A load balancer only helps if it stops sending traffic to instances that can't serve it. Health checks are periodic requests the load balancer sends to each instance — typically an HTTP GET to a lightweight endpoint like /health — to decide if that instance stays in rotation.

The /health endpoint should be cheap and honest: return 200 only if the app can actually serve requests (database connection alive, dependencies reachable), not just "the process is running." A process that's up but can't reach its database should fail its health check — otherwise the load balancer keeps routing real users into a dead end.

An ALB target group health check, in Terraform:

resource "aws_lb_target_group" "app" {
  name     = "app-tg"
  port     = 3000
  protocol = "HTTP"
  vpc_id   = aws_vpc.main.id
 
  health_check {
    path                = "/health"
    interval            = 30
    timeout             = 5
    healthy_threshold   = 3
    unhealthy_threshold = 3
    matcher             = "200"
  }
}

healthy_threshold and unhealthy_threshold prevent flapping — a single dropped health check shouldn't yank an instance out of service; a single recovered check shouldn't immediately trust it back in.

Distribution algorithms

Once traffic reaches the load balancer, it still has to pick which healthy instance gets each request. Round robin cycles through instances evenly. Least outstanding requests sends new requests to whichever instance currently has the fewest in-flight — better when request duration varies a lot. Most managed load balancers default to something close to round robin and only need tuning under unusual traffic patterns.

The sticky-session gotcha

If your app keeps request state in server memory (a common mistake with session data), a user's second request landing on a different instance than their first breaks things — that instance never saw their session. Sticky sessions (session affinity) work around this by routing a user's requests to the same instance for a window of time, usually via a cookie the load balancer sets. It's a patch, not a fix: it reintroduces a single point of failure per user and complicates scaling. The actual fix is to keep servers stateless and put session data somewhere shared (Redis, a database) that any instance can read.