Blue Green Deployment for a Monolith: Database Compatibility First

What is a blue-green deployment for a monolith, and why is it mostly a database problem?

A blue-green deployment runs two production environments side by side. One color (say, blue) serves all traffic while you deploy the new release to the other (green). When green passes its checks, you switch traffic to it. If something goes wrong, you switch back. Teams use it because the cutover is fast and deliberate, and rollback means repointing traffic instead of redeploying an old build under pressure.

The textbook diagram shows two complete stacks. For a monolith, the real picture usually has three parts: two application environments, one shared database and a router in front. The application tier is easy to duplicate because it is stateless or close to it. The primary database is not. Keeping a second copy fully synchronized while it takes writes would require bidirectional replication, conflict handling and a plan for writes that land during the switch. For most teams that costs more and carries more risk than the deploy it is meant to protect.

That leads to the core constraint. During the overlap window, and again if you roll back, old code and new code run against the same schema. Every schema change has to be one that both versions can read and write correctly.

The shared database also gives you no isolation. A migration that drops a column blue still queries will break blue immediately. A backfill that holds long locks will slow both colors at once. Switching the router back to blue does not undo either problem.

Copy                      +--------------------+
                      |       Users        |
                      +---------+----------+
                                |
                                v
                      +--------------------+
                      |       Router       |
                      | flip = cutover     |
                      | flip back = undo   |
                      +---+------------+---+
                          |            |
             live traffic |            | idle until cutover
                          v            v
          +-----------------+        +-----------------+
          |  BLUE  (v1)     |        |  GREEN  (v2)    |
          |  old app code   |        |  new app code   |
          |  stateless      |        |  stateless      |
          +--------+--------+        +--------+--------+
                   |                          |
                   |     reads + writes       |
                   +-----------+--------------+
                               |
                               v
               +-------------------------------+
               |   SHARED PRIMARY DATABASE     |
               |   one schema, no isolation    |
               |   must serve v1 AND v2        |
               |   bad migration hits both     |
               +-------------------------------+

In a monolith blue-green setup only the application tier is duplicated: the router flips traffic between blue and green, but both colors read and write one shared database whose schema must work for old and new code at the same time.

How do you keep schema changes compatible with both blue and green?

The discipline that makes a shared database survive two app versions is expand/contract, also called parallel change. Expand adds structures the old code ignores. Migrate ships code that writes to both old and new structures, then backfills existing rows. Contract removes the old structures in a later release, once no running version reads them.

Renaming fullname to display_name

• Release 1 (expand): ALTER TABLE users ADD COLUMN display_name as nullable. The app writes both columns and reads fullname. A batched UPDATE users SET display_name = fullname WHERE display_name IS NULL backfills old rows.
• Release 2 (migrate): The app reads display_name and still writes both. If you roll back to release 1, it still finds current data.
• Release 3 (contract): Stop writing fullname, then drop it.

Hazards outside the schema

• Serialized cache and queue payloads: Green may write cached objects or jobs that blue cannot deserialize. Version the keys or payloads.
• New enum or status values: Blue may crash or mishandle a status it has never seen.
• Message and API formats: Other systems that consume these formats need the same additive approach.

Copy-- Release 1 (expand): add the new column as nullable so blue's inserts still succeed.
ALTER TABLE users ADD COLUMN display_name VARCHAR(255) NULL;

-- App code in release 1 writes BOTH columns but still reads fullname.
INSERT INTO users (email, fullname, display_name)
VALUES ('ada@example.com', 'Ada Lovelace', 'Ada Lovelace');

UPDATE users SET fullname = 'Ada King', display_name = 'Ada King'
WHERE id = 42;

-- Backfill old rows in small key ranges to keep locks and transactions short.
-- Repeat with the next range (10001-20000, ...) until no rows remain.
UPDATE users
SET display_name = fullname
WHERE display_name IS NULL
  AND id BETWEEN 1 AND 10000;

-- Check that the backfill is complete before shipping release 2.
SELECT COUNT(*) FROM users
WHERE display_name IS NULL AND fullname IS NOT NULL;

-- Release 2 (migrate): app reads display_name and still writes both columns,
-- so a rollback to release 1 (which reads fullname) sees current data.
SELECT id, email, display_name FROM users WHERE id = 42;

-- Release 3 (contract): ship code that no longer writes fullname. After no
-- running version reads or writes the column, drop it in a later step.
ALTER TABLE users DROP COLUMN fullname;

Generic SQL for renaming fullname to display_name across three releases: add a nullable column, dual-write and backfill in batches, switch reads, then drop the old column once no running version uses it.

How should database migrations run relative to the app deploy?

Run the expand migration as its own pipeline step, after blue is confirmed healthy and before green receives any traffic. Do not run it from green's startup routine. If green migrates itself on boot, every replica races to apply the same DDL. A slow migration also delays health checks, and a failed migration leaves you with a half-started green and a schema in an unknown state. A separate step gives you one owner, one log, and a clear gate: green is not deployed until the migration has finished and blue still works against the new schema.

Lock behavior is the main hazard on large tables. Some DDL operations rewrite the table or hold a lock that blocks reads or writes for the whole operation. Others only update metadata and finish almost instantly. Which operations fall into which group depends on the engine, and often on the specific operation and its options. Check your database engine's documentation for each statement before you run it against a large production table. Set a lock timeout so the migration fails fast instead of queueing behind long transactions and blocking everything else behind it.

Keep data backfills out of the migration step. Run them as background jobs that work in small batches. For example, a job might select the next 1,000 rows where the new column is null, ordered by primary key. It updates those rows in a short transaction, commits, pauses briefly, and repeats until no rows remain. Short transactions keep locks brief and replication lag manageable. A job that can be resumed is safe to stop and restart.

The traffic switch can be undone in seconds. The database step cannot. That is why the migration must be expand-only and safe for blue to run against indefinitely.

Copy# Batched backfill, run as a background job after the expand migration.
# PostgreSQL syntax; adapt the SQL for your engine.
import os
import time

import psycopg2

BATCH_SIZE = 1000      # small batches keep each transaction short
PAUSE_SECONDS = 0.5    # gives replicas and other queries room to catch up

BATCH_SQL = """
    UPDATE users
    SET email_normalized = lower(email)
    WHERE id IN (
        SELECT id FROM users
        WHERE email_normalized IS NULL
          AND email IS NOT NULL
        ORDER BY id
        LIMIT %s
    )
"""

def backfill(dsn):
    conn = psycopg2.connect(dsn)
    try:
        with conn.cursor() as cur:
            # Fail fast instead of waiting behind long transactions.
            cur.execute("SET lock_timeout = '2s'")
        conn.commit()
        while True:
            with conn.cursor() as cur:
                cur.execute(BATCH_SQL, (BATCH_SIZE,))
                updated = cur.rowcount
            conn.commit()  # one short transaction per batch
            if updated == 0:
                break      # nothing left; the IS NULL filter makes reruns safe
            time.sleep(PAUSE_SECONDS)
    finally:
        conn.close()

if __name__ == "__main__":
    backfill(os.environ["DATABASE_URL"])  # e.g. postgresql://app:changeme@db.internal/app

A resumable background job that fills a new column in batches of 1,000 rows. It commits after each batch, pauses between batches, and sets a lock timeout so it fails fast rather than piling up behind long transactions.

How does traffic switching work, and which method should you use?

The switch itself is a routing change, and the method you pick decides how quickly and how precisely it takes effect.

DNS is the weakest option because you do not control the caches. Resolvers and clients hold records for at least the TTL, and some keep them longer. For a while, part of your traffic still reaches blue, and you cannot say how much. A rollback through DNS has the same delay.

With a reverse proxy, the switch is usually a single-line edit. In nginx, you point the upstream block at green's address and run nginx -s reload. Old worker processes finish their in-flight requests while new connections go to green. That draining is what makes the cutover clean. Without it, requests are cut off mid-response. Keep-alive connections are another catch: a client can keep sending requests to blue over an open connection until that connection closes.

Long-lived connections such as websockets, streams and long polls force a choice. You can cut them, and clients must reconnect. You can wait for them to end, which can delay the cutover indefinitely. A practical middle ground is a fixed drain timeout followed by a forced close, with clients built to reconnect.

Weighted or canary shifting limits the damage if green is bad. The cost is a longer window where both versions serve traffic at once, which makes the database compatibility rules even more important.

Copy# Main context: cap how long old workers may keep draining after a reload.
# Without this, a single open websocket can keep a blue-serving worker alive indefinitely.
worker_shutdown_timeout 60s;

events {}

http {
    # Short keep-alive so idle client connections close soon after the switch.
    keepalive_timeout 15s;

    # Forward websocket upgrade requests correctly.
    map $http_upgrade $connection_upgrade {
        default upgrade;
        ''      close;
    }

    upstream app {
        # Cutover: comment out blue, enable green, then run `nginx -s reload`.
        # server 10.0.1.10:8080;   # blue (previous release)
        server 10.0.2.10:8080;     # green (new release)

        # Canary alternative: keep both lines and use weights, e.g.
        # server 10.0.1.10:8080 weight=9;  # blue, ~90% of requests
        # server 10.0.2.10:8080 weight=1;  # green, ~10% of requests
    }

    server {
        listen 80;

        location / {
            proxy_pass
            proxy_http_version 1.1;
            proxy_set_header Upgrade $http_upgrade;
            proxy_set_header Connection $connection_upgrade;
            proxy_set_header Host $host;
        }
    }
}

A minimal nginx config that switches traffic from blue to green by changing the upstream and reloading, with a short keep-alive and a capped drain timeout so long-lived connections are force-closed rather than delaying the cutover indefinitely, plus a commented weighted variant for canary shifting.

What state breaks when both colors run at once?

The database is not the only shared surface. Anything a monolith keeps in process memory, or assumes it alone owns, can break when two versions run side by side.

Sessions and caches

If sessions live in server memory, switching traffic to green logs every user out. Sticky sessions hide this until the switch, then fail all at once. Move sessions to a shared store such as Redis or a database table. Make the session payload readable by both versions: add fields, never rename them, and tolerate unknown keys. In-memory caches are less dangerous, because each color holds its own copy. The risk is a shared external cache. If green writes entries in a new shape, blue may fail to deserialize them. Version your cache keys so each color reads only its own entries.

Jobs and consumers

Cron jobs, schedulers, and queue consumers start when the process boots, not when traffic arrives. Idle blue still runs them. This causes three failure modes:

• Duplicate processing. Both colors send the nightly invoice email.
• Old code choking on new data. Blue consumes a message or row that green wrote in a format blue cannot parse.
• Stalled work. You disable jobs on blue but forget to enable them on green, so nothing runs.

Gate singleton work behind one switch. You can use a feature flag that names the active color, or a leader lock (a database advisory lock or a lease row) that only one process can hold. Flip the switch as an explicit step in the cutover, and alert when no process holds the lock.

How do you implement a blue-green release step by step?

A runbook for one release

1. Deploy green alongside blue. Expand migrations are already applied, so both versions run against the same schema.
2. Check connection headroom. Both colors hold pools at once. Illustrative estimate, assumed inputs: 6 instances per color × 20 connections per pool = 120 per color, or 240 total. Add 10 for migrations and admin sessions and you need 250. If the database allows 200, green's warm-up will start failing connections. Shrink pools or put a connection pooler in front before you deploy.
3. Verify green privately. Readiness checks should run a real query and touch critical dependencies, not just return 200. Run smoke tests through a private route that public traffic cannot reach. Warm caches and connection pools so the first real users don't hit cold paths.
4. Switch traffic and watch the rollback criteria you defined in advance: error rate, latency percentiles, and key business events. Agree on thresholds before the cutover so nobody argues about them during it.
5. Keep blue warm for a fixed window. During that window, rollback is a traffic flip.
6. Decommission blue or keep it as the next release target.

The point of no return is the first step after which blue can no longer serve correctly. That step is usually the contract migration or an irreversible data transformation. Both remove what blue depends on, so run them only after the observation window closes.

A traffic flip also cannot undo side effects. Emails already sent, payments already captured, and calls already made to external APIs stay done. Recovering from those means compensating actions, not a rollback.

When should you avoid blue-green for a monolith, and what are the alternatives?

Blue-green fits teams that need a fast cutover and a rollback measured in seconds, and that can absorb the overhead. That overhead is real. Expand/contract turns one logical schema change into two or three releases. It adds temporary dual-write code that someone has to remember to delete. Often nobody does, and dead columns pile up in the schema. Two full environments also double application capacity during the overlap, and for longer if blue stays warm as a rollback target.

It still tends to save engineering time. Each cutover is a rehearsed, routine step rather than a one-off procedure. Migrations get written to be safe to repeat. Rollbacks become a traffic change instead of an emergency restore, so there are fewer late-night incidents spent reconstructing data.

Some changes are better handled with a maintenance window or a dedicated data migration project. Examples are type changes on very large tables, splitting or merging core entities, and rewriting foreign-key relationships. Forcing these through blue-green means months of dual writes and backfills. Scheduled downtime is often cheaper and easier to verify.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to blue green deployment monolith, this is where it helps:

Saves time

• Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
• One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
• Changing a shared variable is one edit and one review instead of editing every service's .env by hand.

Saves cost

• Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
• Less duplicated configuration to maintain: one shared definition inherited by many services.
• A wrong or stale value is caught at review time by the services it affects, not by a failed deploy.
• OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.