Blue-Green Deployment for a Monolith: Migrations and Sessions

What is a blue-green deployment for a monolith, and why does the naive model break?

Blue-green deployment runs two production environments side by side. Blue serves live traffic. Green receives the new release, gets smoke-tested, and then the load balancer switches traffic to it. Teams use this pattern for a monolith because a monolith ships as one large unit, and a rolling update leaves a long window where old and new nodes serve traffic together. Blue-green replaces that window with a near-instant cutover. If something goes wrong, you flip the router back, without redeploying the old build.

The textbook version describes two identical, independent environments. A monolith almost never has that. Both colors read and write the same database, because duplicating and resyncing production data on every release is impractical. Sessions are often held in process memory or pinned to a node. Background workers and schedulers pull from a shared queue.

Copy                 +----------------+
                 | Load balancer  |
                 +-------+--------+
                    /          \
          +--------+--+      +--+--------+
          | Blue (v1) |      | Green (v2)|
          +-----+-----+      +-----+-----+
                |   \          /   |
                |    \        /    |
   +------------+-----+------+-----+-------------+
   |  Database  |  Session store  |  Job queue   |
   +----------------------------------------------+
            shared state: owned by neither color

Because of this shared state, flipping the router does not isolate the two versions. A migration that green needs can break blue in the middle of the request it is serving. If sessions live in memory, they vanish at cutover. A job that v2 enqueues may be picked up by a v1 worker.

Blue-green for a monolith is therefore a compatibility contract more than a routing trick. During the overlap, and through any rollback, every piece of shared state must be readable and writable by both versions. The schema, the session format, the job payloads, and the scheduler ownership all fall under this rule.

Copy                           +-----------------+
                           |  Load balancer  |
                           +--------+--------+
                                    |
             +----------------------+----------------------+
             | live traffic             cutover / rollback |
             v                                             v
  +---------------------+                       +---------------------+
  | Blue app tier (v1)  |                       | Green app tier (v2) |
  | web, workers, cron  |                       | web, workers, cron  |
  +----------+----------+                       +----------+----------+
             |                                             |
             +----------------------+----------------------+
                                    | both versions, at the same time
                                    v
+-----------------------------------------------------------------------+
|                                                                       |
|  +-------------------+  +-------------------+  +-------------------+  |
|  | Database          |  | Session store     |  | Job queue         |  |
|  | one schema, both  |  | must be readable  |  | v2 jobs may hit   |  |
|  | versions query it |  | by v1 and v2      |  | a v1 worker       |  |
|  +-------------------+  +-------------------+  +-------------------+  |
|                                                                       |
|             SHARED STATE: owned by neither blue nor green             |
+-----------------------------------------------------------------------+

The load balancer flips traffic between the blue and green app tiers, but both tiers read and write the same database, session store and job queue, so each piece of shared state must work with v1 and v2 at the same time.

How do database migrations work in a blue-green deployment?

The schema is the hardest of these because it changes in place. The way through is expand/contract (parallel change). Additive changes ship first. Both colors run against the expanded schema. Destructive changes ship in a later release, after the old color is retired and rollback to it is no longer possible. No single release may require a schema that the previous release can't use.

Expand steps are generic SQL such as ALTER TABLE orders ADD COLUMN customer_ref text NULL. Contract steps look like ALTER TABLE orders DROP COLUMN customer_id. Constraints come last, for example ALTER TABLE orders ALTER COLUMN customer_ref SET NOT NULL. Apply a constraint only once no running version can write a null.

Dual-writing exists so that rollback is safe. If you roll back to the old color, it keeps writing the old column. The new column then goes stale, which is why the backfill must be safe to re-run. The cost is extra releases per change. That is the price of every deploy being reversible.

Copy-- PostgreSQL. Rename orders.customer_id -> orders.customer_ref via expand/contract.

-- RELEASE N (expand): additive only. Old color ignores the new column.
ALTER TABLE orders ADD COLUMN customer_ref text NULL;
-- New code dual-writes both columns. Old code writes only customer_id.

-- RELEASE N+1: backfill in batches. Safe to re-run.
-- IS DISTINCT FROM also repairs rows that went stale while an old color was live.
UPDATE orders
SET    customer_ref = customer_id::text
WHERE  id BETWEEN 1 AND 10000  -- advance the range per batch
AND    customer_ref IS DISTINCT FROM customer_id::text;

-- Add NOT NULL only once every running version writes customer_ref.
-- NOT VALID skips the full scan under a heavy lock. VALIDATE then scans
-- with a lighter lock. On PostgreSQL 12+, SET NOT NULL can use the
-- validated CHECK and skip its own scan.
ALTER TABLE orders ADD CONSTRAINT orders_customer_ref_nn
  CHECK (customer_ref IS NOT NULL) NOT VALID;
ALTER TABLE orders VALIDATE CONSTRAINT orders_customer_ref_nn;
ALTER TABLE orders ALTER COLUMN customer_ref SET NOT NULL;
ALTER TABLE orders DROP CONSTRAINT orders_customer_ref_nn;
-- Reads switch to customer_ref in this release. Writes still go to both columns.

-- RELEASE N+2 (contract): run only after the N color is retired and
-- rollback to it is off the table. Confirm nothing has drifted first.
SELECT count(*) FROM orders
WHERE  customer_ref IS DISTINCT FROM customer_id::text;  -- expect 0

ALTER TABLE orders DROP COLUMN customer_id;

A PostgreSQL rename of orders.customer_id to customer_ref, split across three releases: expand with a nullable column, run a re-runnable backfill and add the NOT NULL constraint with minimal locking, then drop the old column once the old color is retired.

How should you run migrations, backfills, and index builds without locking cutover?

Run migrations as their own pipeline stage, not on application startup. If every instance migrates on boot, several of them race for the same schema change. One wins, the others fail or block on locks, and a failed migration surfaces only after new instances are already starting. A separate, gated step runs exactly once. If it fails, the deploy stops before any traffic moves.

A typical stage outline:

1. Migrate: apply expand-only schema changes with a single runner holding a lock.
2. Verify: check that the schema matches what the new color expects.
3. Backfill: run as a resumable job, separate from the deploy.
4. Deploy: start the new color and run health checks.
5. Cut over: move traffic only after the earlier stages report success.

Backfills should update rows in small chunks, keyed by primary key ranges. Each chunk commits on its own and is followed by a short sleep. Select the next batch of IDs above the last processed ID, update only rows where the new column is still null, commit, record the checkpoint, and pause. Short transactions keep lock durations small. Checkpoints let a killed job resume. The null filter makes re-runs safe.

Build indexes online where the database supports it, such as PostgreSQL's concurrent index builds or MySQL's online DDL. A plain index build can block writes for its whole duration.

During all of this, watch primary CPU, I/O, and replication lag. Both colors share the same database, so a heavy backfill slows blue and green at once. Lag can also make reads on replicas return stale data. Throttle the job, or pause it, when these metrics rise.

Copy# Resumable, throttled backfill. Assumes PostgreSQL 10+ and psycopg2.
# Needs: CREATE TABLE backfill_checkpoint (job text PRIMARY KEY, last_id bigint);
# Reading lag columns in pg_stat_replication requires the pg_monitor role.
import time
import psycopg2

JOB = "orders_currency"
BATCH = 1000        # rows per transaction: keeps locks short
PAUSE_S = 0.2       # breathing room for blue and green traffic
MAX_LAG_S = 5       # back off if replicas fall this far behind

conn = psycopg2.connect("dbname=app user=backfill host=db-primary.internal")
cur = conn.cursor()

def replica_lag_seconds():
    cur.execute("SELECT COALESCE(MAX(EXTRACT(EPOCH FROM replay_lag)), 0) FROM pg_stat_replication")
    return float(cur.fetchone()[0])

# Resume from the last committed checkpoint if the job was killed.
cur.execute("SELECT last_id FROM backfill_checkpoint WHERE job = %s", (JOB,))
row = cur.fetchone()
last_id = row[0] if row else 0
conn.commit()

while True:
    while replica_lag_seconds() > MAX_LAG_S:   # shared primary: protect both colors
        conn.commit()
        time.sleep(5)
    cur.execute("SELECT id FROM orders WHERE id > %s ORDER BY id LIMIT %s", (last_id, BATCH))
    ids = [r[0] for r in cur.fetchall()]
    if not ids:
        break
    # The IS NULL filter makes re-runs and overlaps safe.
    cur.execute("UPDATE orders SET currency = 'USD' WHERE id = ANY(%s) AND currency IS NULL", (ids,))
    last_id = ids[-1]
    cur.execute("INSERT INTO backfill_checkpoint (job, last_id) VALUES (%s, %s) "
                "ON CONFLICT (job) DO UPDATE SET last_id = EXCLUDED.last_id", (JOB, last_id))
    conn.commit()   # chunk and checkpoint commit together
    time.sleep(PAUSE_S)

A resumable PostgreSQL backfill that updates rows in primary-key-ordered chunks, commits each chunk with its checkpoint, skips already-filled rows, and pauses when replica lag rises.

How do you handle sticky sessions during a blue-green cutover?

Load balancer affinity pins a client to one backend, usually through a cookie. After you flip traffic, users who still hold a blue affinity cookie keep landing on blue until the cookie expires or blue is removed from the pool. If sessions live in process memory, removing blue logs those users out, and any request that lands on green finds no session.

Move sessions to a shared store

Put session state in a store both colors read, such as Redis or a database table. Any instance can then serve any user, and affinity is no longer required for correctness. Version the serialized payload with a schema field. Each color should ignore fields it does not recognize and apply defaults for fields that are missing. This lets green write a session that blue can still read after a rollback. The serialization example below shows this pattern.

Choosing a cutover approach

A shared store adds a single point of failure. If it goes down, blue and green both fail at the same moment, and keeping two colors gives you no redundancy. Make the store highly available, and decide in advance whether the app should fail closed or fall back to anonymous access.

Copyimport json
from dataclasses import dataclass, field, fields

SCHEMA_VERSION = 2  # green writes v2; blue only knows v1 fields

@dataclass
class Session:
    user_id: str
    csrf_token: str
    # Added in v2. The default lets this code read v1 payloads written by blue.
    locale: str = "en-US"
    roles: list = field(default_factory=list)

def dumps(session: Session) -> str:
    payload = {"v": SCHEMA_VERSION}
    payload.update({f.name: getattr(session, f.name) for f in fields(Session)})
    return json.dumps(payload)

def loads(raw: str) -> Session:
    data = json.loads(raw)
    known = {f.name for f in fields(Session)}
    # Drop fields this color does not recognize (e.g. a v3 field after a rollback).
    kwargs = {k: v for k, v in data.items() if k in known}
    # Missing optional fields fall back to the dataclass defaults.
    # Missing required fields (user_id, csrf_token) raise TypeError: treat as no session.
    return Session(**kwargs)

# A v1 payload written by blue, read by green: locale and roles get defaults.
old = '{"v": 1, "user_id": "u-1042", "csrf_token": "placeholder-token"}'
print(loads(old))

# A future v3 payload: the unknown field is ignored instead of crashing.
newer = '{"v": 3, "user_id": "u-1042", "csrf_token": "placeholder-token", "theme": "dark"}'
print(loads(newer))

A versioned session payload that both colors can read: unknown fields are dropped and missing optional fields get defaults, so blue and green can share sessions through a common store and survive a rollback.

Draining, background jobs, and the cutover runbook

Background work causes the most trouble at cutover. HTTP traffic follows the load balancer, but schedulers and queue consumers run wherever a process is alive. If both colors run the nightly billing job, customers get charged twice. If neither runs it, nobody gets charged. Exactly one color must own background work at any moment, and the handoff must be explicit. Use one of these:

• Leader election through a lock in the database or coordination store, so only one instance runs schedules.
• A job-ownership flag that every worker checks before it starts work.
• Ordered stop/start: stop the old color's consumers, wait for in-flight messages to be acknowledged, then start the new ones.

Cutover runbook

1. Run smoke tests against the idle color using production config, credentials, and network paths.
2. Run health checks that perform a real database read and write and a session-store round trip. A plain liveness probe is not enough.
3. Review the go/no-go criteria:
   • Migrations applied and verified as backward compatible
   • Error rate and latency on smoke traffic match the baseline
   • Session store reachable from the new color
   • Rollback owner present and the rollback path tested
4. Flip traffic at the load balancer.
5. Transfer job ownership using one of the methods above.
6. Drain the old color within a fixed timeout, for example 60 seconds for requests and longer for WebSockets. Close long-lived connections when the timeout expires so that clients reconnect to the new color.
7. Keep the old color warm until the rollback window closes.

What is a safe rollback strategy for a monolith blue-green deployment?

Decide what "rollback" means before you flip traffic, and write it into the runbook. Flipping traffic back to blue is safe only while two things hold: the schema is still in its expanded state, and every row green has written can be read correctly by blue's code.

The second condition is easy to break. Suppose green writes a new status value, or stores a field in a new serialization format. Blue has never seen either one. After a rollback, blue may reject those rows, map them to a default, or overwrite them with its own format on the next update. All three silently corrupt data written during the cutover window. Writes that green made only to a new column are also lost from blue's point of view, because blue never reads that column. To avoid this, put new data formats behind a feature flag. Turn the flag on only after you have decided you will not roll back. Until then, green should write data that blue understands.

The contract step removes the old columns and code paths, so traffic rollback is no longer possible. From that point, "rollback" means rolling forward: ship a fix through the same pipeline.

When should you avoid blue-green for a monolith, and what are the alternatives?

Blue-green suits teams that release often, hold a strict uptime target, and can keep up backward-compatibility discipline on every schema change. It is a poor fit in three cases:

• Irreversible migrations. A destructive data transform or a third-party schema you don't control removes the old version's ability to run, so the main benefit is lost.
• Load-dependent bugs. Green handles no real traffic before cutover, so it will not surface contention or cache-stampede bugs that appear only at full production load.
• Very large environments. Some estates are too large or too stateful to duplicate.

The development cost is real. Expand/contract turns one change into several releases. Dual-write code has to be tracked, usually as tickets with owners, and then removed. Compute roughly doubles during the overlap window. Illustrative estimate: assume 20 app instances, a 2-hour overlap per deploy, and 30 deploys a month. That adds 1,200 instance-hours to a baseline of about 14,600, or roughly 8% more app-tier compute. The shared database is not duplicated.

In return you get fewer maintenance windows, fast and low-stress rollback, and schema releases that are decoupled from code releases.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to blue green deployment monolith, this is where it helps:

Saves time

• Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
• One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
• Changing a shared variable is one edit and one review instead of editing every service's .env by hand.

Saves cost

• Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
• Less duplicated configuration to maintain: one shared definition inherited by many services.
• A wrong or stale value is caught at review time by the services it affects, not by a failed deploy.
• OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.