Auto-scaling: policies, metrics, and gotchas

Manually adding servers before a traffic spike and removing them after works right up until it doesn't — auto-scaling automates that decision based on real metrics. It also introduces its own failure modes: scale-out that arrives too late, and scale-in that guts capacity you still needed.

Intermediate

4 min read

What auto-scaling actually automates

Horizontal auto-scaling adjusts the number of running instances of your app in response to demand — adding instances when load rises (scale-out), removing them when it falls (scale-in). The alternative, manually watching dashboards and clicking "add instance," doesn't survive a 3am traffic spike or a slow Tuesday afternoon where you're overpaying for idle capacity.

An auto-scaling setup has three parts: a group of instances running the same configuration, a metric that reflects load, and a policy that maps metric values to instance counts.

Target tracking vs step scaling

Target tracking is the simplest and most common policy: pick a metric and a target value, and the scaler adjusts capacity to keep the metric near that target — conceptually similar to a thermostat.

resource "aws_appautoscaling_policy" "cpu" {
  name               = "keep-cpu-at-60"
  policy_type        = "TargetTrackingScaling"
  resource_id        = "service/my-cluster/my-service"
  scalable_dimension  = "ecs:service:DesiredCount"
  service_namespace  = "ecs"
 
  target_tracking_scaling_policy_configuration {
    target_value = 60.0
    predefined_metric_specification {
      predefined_metric_type = "ECSServiceAverageCPUUtilization"
    }
    scale_in_cooldown  = 300
    scale_out_cooldown = 60
  }
}

Step scaling is more manual but more precise: you define explicit thresholds and how many instances to add or remove at each ("if CPU > 80%, add 2; if CPU > 95%, add 5"). It's worth reaching for when a single smooth target doesn't capture your actual capacity cliff — e.g., a queue-depth metric where you want to react much more aggressively past a certain backlog size.

What metric should actually drive the decision

CPU utilization is the default because it's always available, but it's frequently the wrong signal. A memory-bound service can be melting down with low CPU. An API gateway's real bottleneck might be concurrent connections, not CPU. The metric should reflect the thing that actually limits your capacity:

Request-heavy web services: request count per target, or CPU if requests are compute-bound.
Queue consumers: queue depth or messages visible — scale workers based on backlog, not on the workers' own idle CPU.
Memory-bound workloads (caches, in-memory processing): memory utilization, not CPU.

Scaling on the wrong metric produces a system that looks "healthy" by its own dashboard while actually failing users — CPU sitting at 20% while the request queue backs up for a different reason entirely.

Cooldowns: preventing scaling from fighting itself

After a scaling action, most systems enforce a cooldown period — a window where further scaling decisions are paused, letting the new instances actually come online and start absorbing load before the next evaluation. Without a cooldown, a scaler can see load "still high" (because new instances haven't started serving yet), scale out again, overshoot badly, then scale in just as aggressively once the oversized fleet drops the metric — a feedback loop that never settles. Scale-out cooldowns are typically shorter (react fast to real load) than scale-in cooldowns (don't remove capacity the moment things look calm, in case it's a blip).

The gotchas: cold starts and scaling lag

Two failure modes matter in production that don't show up in a policy diagram:

Cold starts: a newly launched instance isn't instantly useful. It needs to boot, pull the container image, initialize the app, warm connection pools and caches — often tens of seconds. During a sharp traffic spike, the instances the scaler "added" aren't actually absorbing load yet, and the existing fleet is still overwhelmed in the meantime.
Scaling lag: metrics are typically averaged and evaluated on a delay (CloudWatch metrics commonly lag by 1–5 minutes). By the time the scaler reacts to "load is high," the spike may already be several minutes old — and by the time new capacity is actually online, the spike could be over, leaving you with excess capacity you now have to scale back down.

The practical mitigation for both is the same: don't cut scaling this close. Keep a baseline of extra capacity for predictable spiky traffic, use predictive scaling where available (forecasts based on historical patterns rather than reacting after the fact), and for genuinely latency-sensitive spiky workloads, consider serverless compute (which sidesteps instance boot time entirely) over instance-based auto-scaling.