Zero-downtime deploys and database migrations

During any rolling deploy, old and new versions of your app run against the same database at the same time — sometimes for minutes. A schema migration that isn't compatible with both versions simultaneously will break production, even though each version works fine on its own.

Advanced

4 min read

The window every rolling deploy creates

A rolling deploy replaces instances one (or a few) at a time rather than all at once, specifically to avoid downtime. But that means for the duration of the rollout, some requests hit the old code and some hit the new code — against the exact same database.

If the new code's first action is a migration that drops a column the old code still reads, every request hitting an old instance between t1 and t3 breaks. The migration and the deploy are not one atomic event — they're two separate changes that must each be compatible with both versions of the app during the overlap.

The expand/contract pattern

The standard fix is to split any breaking schema change into multiple, individually backward-compatible deploys — commonly called expand/contract:

Expand: add the new schema element without touching the old one. Deploy. Both old and new app code work fine (old code ignores the new column; nothing depends on it yet).
Migrate: deploy app code that writes to both old and new columns, and backfill existing rows. Both versions still work.
Cut over: deploy app code that reads from and writes only the new column. Old code is now fully retired everywhere (previous rolling deploy completed).
Contract: only now, in a separate deploy, drop the old column.

Renaming a column, for example, is never a single migration — it's add new_column → dual-write → backfill → read from new_column → drop old_column, each a safe, independent step.

-- Step 1 (expand): safe with old code running unchanged
ALTER TABLE users ADD COLUMN email_normalized TEXT;
 
-- Step 4 (contract): only after every instance is on code that no longer reads `email`
ALTER TABLE users DROP COLUMN email;

Migration ordering relative to the deploy

Migrations should run before the new code that depends on them is live, and the migration itself must not break the currently-running old code. That means:

Adding a nullable column, or one with a default: always safe to run first — old code simply doesn't know it exists.
Adding a NOT NULL column with no default on a large table: can lock the table or fail on existing rows — add it nullable, backfill, then add the constraint in a later step.
Renaming or dropping anything: never safe to do directly — always goes through expand/contract.

Locking: the migration that silently takes the site down

Some migrations that are logically "safe" are still operationally dangerous on a large table. ALTER TABLE ... ADD COLUMN ... NOT NULL DEFAULT 'x' on Postgres pre-11 rewrote the entire table under an exclusive lock — on a large table, that's minutes of every query queuing behind the lock, which looks identical to an outage even though no "breaking" schema change happened. The fix is almost always: add the column nullable (metadata-only change, instant), backfill the default value in batches, then add a NOT NULL constraint using NOT VALID + a separate validation pass so the constraint check doesn't hold a long lock.

Application-level compatibility, not just schema-level

The same discipline applies to code, not just SQL. If version N of the app expects a field in an API response or a queue message that version N-1 doesn't produce yet, you have the same overlap problem in reverse. The rule of thumb: any change that isn't backward-compatible with the immediately-preceding deployed version needs to be split into an "add support for new thing, keep old thing working" deploy, followed later by a "remove old thing" deploy — never combined into one.