Rollback strategies and automated rollback

Every deployment strategy in this domain eventually asks the same question: a new version is misbehaving in production — now what? Rollback speed usually matters more than rollback elegance, and the fastest rollback is one that was designed in before the deploy, not improvised during an incident.

Advanced

4 min read

Rollback vs roll-forward: two different recoveries

When production is broken, there are two ways out. Rollback reverts to the previous known-good version — fast, but it can't fix a problem that's already caused bad data (a buggy migration that corrupted rows won't be undone by reverting the app code). Roll-forward ships a fix on top of the broken version — slower (a new change has to be written, tested, and deployed under pressure), but it's the only option once bad data already exists, or when the "previous version" itself has a problem the new version was specifically fixing.

The practical default: rollback first if it's available and the damage is reversible, because "go back to the version that was working five minutes ago" is almost always faster than "carefully write and ship a fix while production is on fire." Roll-forward is the fallback for the cases rollback can't cover.

Why rollback speed depends on decisions made before the incident

A rollback is only fast if the previous version is still readily deployable — which is exactly what the earlier lessons in this domain set up:

Immutable, promoted artifacts (from the artifact-versioning lesson): the previous version's exact build still exists, tagged and pullable — "redeploy the last one" doesn't require rebuilding from an old commit under pressure, with all the drift risk that implies.
Blue-green's kept-warm idle environment: rollback is switching a router back to an environment that never stopped running — the fastest possible case, seconds not minutes.
The expand/contract migration pattern: because each schema change was independently backward-compatible with the immediately-preceding version, rolling back the application code doesn't also require rolling back the database — the old code still works against the current schema, since that was the whole point of the expand/contract discipline.

A team that skips these and rebuilds from source, doesn't keep a previous environment warm, and runs migrations that break backward compatibility, doesn't have a rollback plan — it has a rollback hope, discovered to not work at the worst possible time.

Automated rollback: letting metrics make the call

The canary lesson's auto_rollback_configuration is the general pattern: tie a deployment's progression (or reversal) directly to real metrics, not to a human watching a dashboard and deciding to act.

resource "aws_cloudwatch_metric_alarm" "error_rate" {
  alarm_name          = "deploy-error-rate-spike"
  comparison_operator = "GreaterThanThreshold"
  evaluation_periods   = 2
  metric_name          = "5xxErrorRate"
  namespace            = "AWS/ApplicationELB"
  period               = 60
  statistic            = "Average"
  threshold            = 5.0  # rollback if error rate exceeds 5%
}
 
resource "aws_codedeploy_deployment_group" "app" {
  auto_rollback_configuration {
    enabled = true
    events  = ["DEPLOYMENT_FAILURE", "DEPLOYMENT_STOP_ON_ALARM"]
  }
  alarm_configuration {
    alarms  = [aws_cloudwatch_metric_alarm.error_rate.alarm_name]
    enabled = true
  }
}

The advantage over a human-watched rollout: a metric threshold breach at 3am triggers rollback in seconds, before an on-call engineer has even opened a laptop. The trade-off is real too — a threshold set too sensitively rolls back on noise (a brief, self-resolving blip), and a threshold set too loosely lets real damage accumulate before it fires; getting this calibration right takes iteration and isn't free.

The gotcha: a rollback that isn't actually clean

"Just redeploy the old version" assumes the old version is genuinely compatible with everything else that's currently true in production — and that assumption breaks in a few predictable ways:

A migration that wasn't expand/contract-safe: rolling back the app without also reasoning about the schema state can put old code in front of a schema it was never written to handle.
A queue or API contract change: if the new version started writing messages in a new format and the rollback removes the code that reads that format, in-flight messages produced during the brief new-version window can now be unreadable by the old code.
Feature-flag state left on: if a flag was flipped on as part of the release (not just the deploy), rolling back the code without also reverting the flag state can leave old code trying to run a new code path it doesn't actually have.

The common thread: rollback is safe exactly to the extent that every change shipped alongside the deploy was actually independently reversible — which is the same backward-compatibility discipline this whole domain has been building toward, applied in reverse.

Rollback strategies and automated rollback

Advanced

4 min read

Rollback vs roll-forward: two different recoveries

Why rollback speed depends on decisions made before the incident

A rollback is only fast if the previous version is still readily deployable — which is exactly what the earlier lessons in this domain set up:

Immutable, promoted artifacts (from the artifact-versioning lesson): the previous version's exact build still exists, tagged and pullable — "redeploy the last one" doesn't require rebuilding from an old commit under pressure, with all the drift risk that implies.
Blue-green's kept-warm idle environment: rollback is switching a router back to an environment that never stopped running — the fastest possible case, seconds not minutes.
The expand/contract migration pattern: because each schema change was independently backward-compatible with the immediately-preceding version, rolling back the application code doesn't also require rolling back the database — the old code still works against the current schema, since that was the whole point of the expand/contract discipline.

Automated rollback: letting metrics make the call

resource "aws_cloudwatch_metric_alarm" "error_rate" {
  alarm_name          = "deploy-error-rate-spike"
  comparison_operator = "GreaterThanThreshold"
  evaluation_periods   = 2
  metric_name          = "5xxErrorRate"
  namespace            = "AWS/ApplicationELB"
  period               = 60
  statistic            = "Average"
  threshold            = 5.0  # rollback if error rate exceeds 5%
}
 
resource "aws_codedeploy_deployment_group" "app" {
  auto_rollback_configuration {
    enabled = true
    events  = ["DEPLOYMENT_FAILURE", "DEPLOYMENT_STOP_ON_ALARM"]
  }
  alarm_configuration {
    alarms  = [aws_cloudwatch_metric_alarm.error_rate.alarm_name]
    enabled = true
  }
}

The gotcha: a rollback that isn't actually clean

"Just redeploy the old version" assumes the old version is genuinely compatible with everything else that's currently true in production — and that assumption breaks in a few predictable ways:

A migration that wasn't expand/contract-safe: rolling back the app without also reasoning about the schema state can put old code in front of a schema it was never written to handle.
A queue or API contract change: if the new version started writing messages in a new format and the rollback removes the code that reads that format, in-flight messages produced during the brief new-version window can now be unreadable by the old code.
Feature-flag state left on: if a flag was flipped on as part of the release (not just the deploy), rolling back the code without also reverting the flag state can leave old code trying to run a new code path it doesn't actually have.

Rollback strategies and automated rollback

Rollback vs roll-forward: two different recoveries

Why rollback speed depends on decisions made before the incident

Automated rollback: letting metrics make the call

The gotcha: a rollback that isn't actually clean

Further reading

Rollback strategies and automated rollback

Rollback vs roll-forward: two different recoveries

Why rollback speed depends on decisions made before the incident

Automated rollback: letting metrics make the call

The gotcha: a rollback that isn't actually clean

Further reading