Why does the database break blue-green deployment in a monolith?

This checklist is for backend and platform engineers who run a monolith against a single shared database and want zero-downtime releases with a rollback path measured in seconds, not hours.

Blue-green deployment promises that you can stand up a new version (green) next to the current one (blue), shift traffic at the load balancer, and shift it back if something goes wrong. That works for stateless application servers because each environment has its own copy. The database is the exception. In most monolith setups, blue and green both point at the same database, so the schema is shared state that neither environment owns. The load balancer switch is the visible cutover. The real one happens when the schema changes.

Two application versions read and write the same tables during three distinct periods:

  • During cutover. Traffic shifts gradually or all at once, but both versions are live and connected for some window.
  • While in-flight requests overlap. Even after a hard switch, blue may still be finishing long requests, background jobs, or queued work that started before the switch. Those writes land in the same tables green is reading.
  • After a rollback. If you send traffic back to blue, blue now runs against whatever schema and data green left behind.

This last period is why instant rollback is conditional. Flipping the load balancer back takes seconds. That speed is useless if green dropped a column blue still selects, or wrote values blue can't parse. Rollback stays fast only if the schema still supports the old version.

                         +-------------------+
                         |   Load balancer   |
                         | (visible cutover) |
                         | rollback = seconds|
                         +---------+---------+
                                   |
              +--------------------+----------------+
              | before switch          after switch |
              v                                     v
   +---------------------+               +---------------------+
   |  BLUE  (version N)  |               | GREEN (version N+1) |
   | app servers         |               | app servers         |
   | in-flight requests  |               | new code, new reads |
   | background jobs     |               | writes new values   |
   +----------+----------+               +----------+----------+
              |                                     |
              | reads + writes       reads + writes |
              +--------------------+----------------+
                                   v
              +-----------------------------------------+
              |      SHARED DATABASE (one schema)       |
              | Real cutover = schema change, not the LB|
              |                                         |
              | 1. Cutover: blue and green both live    |
              | 2. Overlap: blue in-flight writes land  |
              |    in tables green is reading           |
              | 3. Rollback: blue runs on the schema    |
              |    and data green left behind           |
              +-----------------------------------------+

Blue and green are separate app environments behind one load balancer, but both read and write a single shared database, so the schema must support both versions during cutover, in-flight overlap, and after a rollback.

How does the expand/contract pattern keep a schema backward compatible?

Expand/contract, also called parallel change, makes that guarantee explicit. Every schema change is split into two migrations with opposite properties. The database then passes through an intermediate state that both the old and new application versions can use.

Expand migrations only add things: nullable columns, columns with defaults, new tables and new indexes. Blue ignores structures it doesn't know about. A SELECT that names its columns never sees the new one, and an INSERT that omits it still succeeds because the column accepts NULL or falls back to its default. Expand ships before or alongside the green code that depends on it.

Contract migrations remove or tighten: dropping columns, adding NOT NULL, dropping old tables. Each of these breaks blue. A dropped column fails blue's queries, and a new NOT NULL constraint rejects blue's inserts that leave the field empty. Contract can ship only after blue is fully retired and you have decided that rolling back to it is no longer an option.

Phase What it changes When it ships Rollback-safe?
Expand Adds nullable or defaulted columns, new tables, new indexes Before or with the new release Yes. The old code ignores the additions.
Contract Drops columns or tables, adds NOT NULL and other constraints After the old version is retired and the rollback window closes No. The old code depends on what was removed.

The rule follows from this: never put expand and contract in the same release. Between the two phases the schema is a superset of what each version needs, and that superset is what lets you flip traffic back.

Which schema changes are safe to ship during a blue-green cutover?

Classify every migration with one question: if this runs while blue is still serving traffic, does blue keep working? Blue was written against the old schema, so everything it reads must still exist and everything it writes must still be accepted.

Changes that are usually safe in one step:

  • Add a nullable column. Blue never selects it, and its inserts that omit the column still succeed.
  • Add a table. Nothing in blue references it.
  • Add an index. Query results don't change, but the build can lock the table. Check how your engine handles it.
Unsafe single-step change Why blue breaks Safe multi-step alternative
Rename a column or table Blue's queries use the old name Add the new name, dual-write, backfill, switch reads, drop the old name after blue is retired
Change a column type Blue writes values the new type rejects, or reads a format it can't parse Add a column with the new type, dual-write, backfill, switch reads, drop the old column later
Add NOT NULL without a default Blue's inserts omit the column and fail Add the column as nullable, ship code that always writes it, backfill, then add the constraint
Drop anything blue reads Blue's queries error Ship a release that stops reading it, then drop it in a later release
Tighten a constraint (unique, foreign key, check, length) Blue may write rows that violate it Ship code that complies, clean up existing rows, then add the constraint

Online DDL support and locking behavior vary by database engine, and sometimes by the specific operation. An index or constraint that builds without blocking writes on one engine may take an exclusive lock on another. Confirm each operation against your engine's documentation.

A non-blocking build still has a cost. Building an index or validating a constraint on a large table scans and writes a lot of data. That uses I/O and cache that blue needs for live traffic, so latency can rise for the version you are trying to protect. Run these builds during low traffic and watch database load while they run.

How do you rename a column across multiple releases without downtime?

…replica lag if you use replicas. A column rename is where these constraints show up most, because a single RENAME COLUMN breaks whichever version is still reading the old name. Split it across releases instead. The example renames full_name to display_name.

Release Step Rollback safe?
R1 Add display_name as nullable, no default Yes
R1 Write both columns, read full_name Yes
Between R1 and R2 Batched backfill Yes, because it is idempotent
R2 Read display_name, still write both Yes, to R1
R3 Stop writing full_name Only to R2
R4 Drop full_name No

The expand step is ALTER TABLE users ADD COLUMN display_name text NULL. A nullable column with no default avoids a table rewrite on most engines.

For dual writes, set both columns in the same statement: UPDATE users SET full_name = :v, display_name = :v WHERE id = :id. If the two writes are separate statements outside a transaction, a crash between them leaves the columns out of sync. If the second write goes to another store, use an outbox so it is retried.

The backfill copies one id range at a time, for example UPDATE users SET display_name = full_name WHERE id BETWEEN :lo AND :hi AND display_name IS NULL. Because of the IS NULL guard, a rerun does nothing and newer dual-written values are not overwritten. It also matters when you start. If blue is still running pre-R1 code, it updates full_name alone, and backfilled rows go stale. Start the backfill only after every instance runs R1. Before R2, run a query that counts rows where the two columns differ, and only switch reads when the count is zero.

-- R1 expand: nullable, no default, so most engines skip a table rewrite.
ALTER TABLE users ADD COLUMN display_name text NULL;

-- R1 dual write: one statement keeps both columns in sync.
-- If this write is ever split into two statements, wrap them in a transaction.
UPDATE users
SET full_name = :new_name, display_name = :new_name
WHERE id = :id;

-- Backfill: run only after every instance runs R1. Repeat for each id range
-- (for example 10000 ids at a time) and commit after each batch.
-- The IS NULL guard makes reruns no-ops and skips rows that dual writes
-- already set. Because the guard is in the UPDATE itself, it is checked
-- when the row is updated, not in an earlier separate read.
UPDATE users
SET display_name = full_name
WHERE id BETWEEN :lo AND :hi
  AND display_name IS NULL
  AND full_name IS NOT NULL;

-- Gate for R2: switch reads only when this returns 0.
-- Spelled out without IS DISTINCT FROM so it runs on more engines.
SELECT COUNT(*) AS drifted_rows
FROM users
WHERE (full_name IS NULL AND display_name IS NOT NULL)
   OR (full_name IS NOT NULL AND display_name IS NULL)
   OR full_name <> display_name;

-- R4 contract: run only after R3 is stable and rolling back to R2 is no longer needed.
-- ALTER TABLE users DROP COLUMN full_name;

Generic SQL for renaming full_name to display_name: add the column, dual-write in one statement, backfill in guarded batches that can be rerun safely, check for drift before switching reads, and drop the old column last.

What breaks beyond the schema: data values, jobs, caches and sessions?

A schema can be fully backward compatible while the data inside it is not. If green starts writing status = 'on_hold' and blue's code maps statuses with a switch that throws on unknown values, every blue instance that reads that row fails. That includes the instances you would roll back to. Ship the reading side first. One release teaches the code to tolerate unknown values by logging them and treating them as a safe default. A later release starts writing the new value.

The reverse failure is quieter. Many ORMs save every mapped attribute on update, and handwritten code often does UPDATE ... SET with all known columns. Blue doesn't know about green's new discount_code column, so it leaves that column alone. But if blue loads a row, edits one field, and writes back a full row shape from its own model, it can reset values it never meant to touch. This happens whenever blue serializes the whole record, for example through a JSON blob column or a cached object that gets written back. Updating only the changed columns limits each version to the fields it understands.

Shared state outside tables follows the same rules:

  • Queues and background jobs. Consumers from both versions pull from the same queue. Message payloads need the same additive, tolerant-reader approach as columns.
  • Caches and sessions. A session written by green may be read by blue after a rollback. Version the serialized format, or make the deserializer ignore unknown fields.
  • Scheduled tasks. Pin cron-style jobs to one color, or use a lease or lock. Otherwise both colors run them twice, or blue keeps running old logic after cutover.
import logging
from enum import Enum

log = logging.getLogger(__name__)

class Status(Enum):
    PENDING = "pending"
    SHIPPED = "shipped"
    UNKNOWN = "unknown"  # in-memory fallback only, never persisted

def parse_status(raw: str) -> Status:
    # Tolerant reader: a newer release may already write values like 'on_hold'.
    try:
        return Status(raw)
    except ValueError:
        log.warning("unknown order status %r, treating as UNKNOWN", raw)
        return Status.UNKNOWN

# Risky: writes back every column this version knows, using values loaded earlier.
# A stale read clobbers concurrent changes. If status was mapped to UNKNOWN,
# this also overwrites green's 'on_hold' with a value neither version wants.
def save_full_row(cur, order):
    cur.execute(
        "UPDATE orders SET status = %s, total = %s, note = %s WHERE id = %s",
        (order["status"].value, order["total"], order["note"], order["id"]),
    )

# Safer: touch only the column this code path changed, and only from a state
# this version understands. Rows in unknown states are left alone.
def mark_shipped(cur, order_id) -> bool:
    cur.execute(
        "UPDATE orders SET status = %s WHERE id = %s AND status = %s",
        (Status.SHIPPED.value, order_id, Status.PENDING.value),
    )
    return cur.rowcount == 1

A tolerant enum parser that maps unrecognized statuses to a non-persisted fallback, next to a full-row UPDATE that can clobber values and a guarded single-column UPDATE that changes only what this version owns (DB-API cursor with %s placeholders, as in psycopg).

How should you run migrations and plan rollback before cutover?

Run migrations as a separate pipeline step that finishes before any traffic shifts. Don't run them from green's startup. If green migrates on boot, every instance races to apply the change. A failed migration then blocks the deploy at the worst moment, and the schema changes underneath blue while blue is still serving users. Once the migration completes, run smoke tests against the migrated schema for both versions. Blue has to keep working there, because blue is your rollback target.

Test long-running DDL and backfills against production-sized data, not a small fixture. A change that runs instantly in staging can hold a table lock for minutes on a large table. Measure how long it takes and what it locks, then batch the backfill so it does not saturate I/O.

Define a rollback window during which blue can take traffic again. Keep contract migrations out of that window. Treat down migrations as a last resort, because they can drop columns or rows that green has already written. Rolling back application traffic should never require rolling back the schema.

Check Catches
Migration classified as expand or contract Contract changes shipped too early
Old version tested against new schema in CI Blue breaking after the migration
Backfill completion verified Green reading NULLs in new columns
Feature flags on new write paths Writes you cannot switch off
Error and query-latency monitoring on both versions Silent regressions on either color
Documented point after which rollback is no longer schema-safe Rollbacks that corrupt data

Each row is something a reviewer or a CI job can check before release. That means an unsafe change gets rejected in a pull request, where the fix is quick, instead of being found during an incident.

#!/usr/bin/env bash
# Pre-cutover pipeline stage: migrate, verify both colors, then shift traffic.
# Runs once from CI, never from green's startup, so instances don't race.
set -euo pipefail

DB_URL="postgres://deploy@db.internal:5432/shop"   # placeholder
BLUE_URL=""
GREEN_URL=""

# 1. Apply the expand-only migration. Fail fast instead of queueing behind
#    long-held locks; lock_timeout makes DDL give up rather than block traffic.
PGOPTIONS='-c lock_timeout=5s -c statement_timeout=15min' \
  psql "$DB_URL" -v ON_ERROR_STOP=1 -f migrations/0042_expand_add_customer_ref.sql

# 2. Confirm the batched backfill finished before green reads the new column.
remaining=$(psql "$DB_URL" -tAc \
  "SELECT count(*) FROM orders WHERE customer_ref IS NULL")
if [ "$remaining" -ne 0 ]; then
  echo "Backfill incomplete: $remaining rows still NULL" >&2
  exit 1
fi

# 3. Smoke-test BOTH versions against the migrated schema.
#    Blue must pass: it is the rollback target.
for url in "$BLUE_URL" "$GREEN_URL"; do
  ./bin/smoke-test --base-url "$url" || {
    echo "Smoke test failed for $url; not shifting traffic" >&2
    exit 1
  }
done

# 4. Only now move traffic. Rolling back later means pointing traffic at blue
#    again; the schema stays as-is. Contract migrations ship in a later release,
#    after the documented rollback window closes.
./bin/shift-traffic --to green

A CI pipeline stage that applies an expand migration with lock and statement timeouts, verifies the backfill completed, smoke-tests both blue and green against the new schema, and only then shifts traffic (the bin/ scripts are placeholders for your own tooling).

What does expand/contract cost, and when should you use an alternative?

Expand/contract has a real price. A change that would be one migration becomes three or four releases, and each one has to deploy and settle before the next can ship. While the change is in progress, the codebase carries temporary paths such as dual writes and fallback reads. The contract step is easy to skip because nothing visibly breaks when it is left undone, so dead columns pile up unless someone owns the cleanup. Running the previous release against the new schema in CI means you must keep that release buildable. Feature flags help with sequencing but fail in their own way: if blue and green read different flag state, the two versions write different data shapes at the same time.

Rollback also has a time limit. Once green writes data that blue cannot represent, rolling back stops being free.

Some changes should not be forced through blue-green at all. Splitting a table, changing a primary key type, or changing what existing values mean are all examples. Any of these can take more effort to stage safely than the downtime it avoids.

Alternative How it works Typical fit
Scheduled maintenance window Stop traffic, migrate, restart Key type changes, table splits, low-traffic systems
Roll-forward-only release Fix forward instead of rolling back Teams with fast, reliable deploys
Feature-flag-gated change Ship the code dark, enable it after the migration Behaviour changes with a clear on/off boundary

The extra effort pays off when you release often or have strict uptime commitments. If you release monthly and a short outage is acceptable, a maintenance window is usually cheaper and safer.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to blue green deployment database, this is where it helps:

Saves time

  • Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
  • One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
  • Reviewers see the exact diff and the list of affected services up front, so approval is a decision rather than an investigation.

Saves cost

  • Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
  • Less duplicated configuration to maintain: one shared definition inherited by many services.
  • Per-service approve/reject means a bad shared value can be refused where it would break something, before it is live.
  • OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.