Rollback Deployment Database Migration: Reverting a Monolith Safely

What a rollback really means once a migration has run

When a release ships with a migration, two things change: the application code and the database schema. Reverting the code is usually cheap: redeploy the previous artifact. Reverting the schema is a separate operation with its own risks. In practice, a rollback after a migration means running release N-1 against schema N. Whether that works depends on one question: does N-1 tolerate schema N? If N-1 reads a column that N dropped, or inserts rows without a value for a new NOT NULL column with no default, the rollback fails or corrupts data.

The usual instinct is to write a down-migration and run it alongside the code revert. That is a poor default for several reasons:

• It can destroy data. Rows and values written by release N after the deploy may live only in the new columns or tables. Dropping them discards real user data.
• It is rarely tested. Down-migrations run almost never and often fail against production data that does not match what the author assumed.
• It can lock large tables. Reversing a schema change on a large table can hold locks long enough to block writes, which turns a bad deploy into an outage.
• It extends the incident. You are now debugging a migration under pressure instead of restoring service.

This guide is for backend and platform engineers shipping a monolith that shares a single database across the codebase. It applies wherever code and schema deploy together and rollback by redeploy is your main recovery path.

Classifying migrations by rollback safety

Before running any migration, ask this: if the previous release's code runs against the new schema, does it still work? That question puts every schema change into one of three groups.

"Additive" describes the DDL. It does not guarantee that old code keeps working. Watch for hidden coupling:

• ORM behavior. Some ORMs select every column into a strict mapping or validate the schema at boot. With these, an extra column can stop the old release from starting.
• NOT NULL without a default. Old code inserts rows without the new column, so the database rejects every insert.
• Enum and check-constraint changes. Removing an allowed value makes old writes fail. Adding a value is also risky, as the next point shows.
• New data old code can't parse. If the new release writes a new status value or format, those rows remain after a rollback. Old code reads them and throws.

Locking and online DDL behavior differ by database and storage engine. An operation that runs online in one database can take a table lock in another. Check each change against the documentation for the database and version you actually run.

How expand-and-contract works: a column rename across releases

Take a rename of customers.full_name to display_name. Each step below ships as its own release, and each one leaves the previous release able to run against the current schema.

Release N: expand. Add display_name as a nullable column with no default, so the change stays cheap. No code reads or writes it yet. Rolling back to N‑1 is safe because the old code ignores the column.

Release N+1: dual-write. Every write path sets both columns in the same statement or transaction. Reads still use full_name. Rolling back to N is safe because full_name remains the source of truth.

Backfill (a job, not a release). Walk the table in primary-key ranges and copy full_name into display_name where it is still null. Keep batches small and sleep between them. This keeps each lock short and gives replicas time to catch up. The null check makes the job idempotent, so you can stop it and restart it at any point.

Release N+2: switch reads. Read display_name and keep writing both columns. Rolling back to N+1 is safe because full_name is still current.

Release N+3: stop writing the old column. full_name must be nullable by now so that inserts still succeed. Rolling back to N+2 is safe. Rolling back to N+1 or earlier is not, because those releases read a column that has gone stale.

Release N+4: contract. Drop full_name only after the team agrees that N+2 is no longer a rollback target. Before dropping it, confirm N+3 has no remaining references to the column, including ORM mappings and SELECT * queries.

Copyimport time

BATCH_SIZE = 1000       # rows per transaction; keeps each lock short
PAUSE_SECONDS = 0.5     # throttle so replicas can catch up between batches

# Release N+1 to N+3: every write path sets both columns in one statement.
# full_name stays the source of truth until reads switch in N+2.
def update_customer_name(conn, customer_id, name):
    with conn.cursor() as cur:
        cur.execute(
            "UPDATE customers SET full_name = %s, display_name = %s WHERE id = %s",
            (name, name, customer_id),
        )
    conn.commit()

# Backfill job (not a release). It walks primary-key ranges, so each
# batch touches a bounded slice of the table even where ids are sparse.
# The IS NULL check makes reruns safe: stop and restart at any point.
def backfill_display_name(conn, start_id=0):
    with conn.cursor() as cur:
        cur.execute("SELECT COALESCE(MAX(id), 0) FROM customers")
        max_id = cur.fetchone()[0]
    conn.commit()  # end the read transaction before batching

    lower = start_id
    while lower <= max_id:
        upper = lower + BATCH_SIZE
        with conn.cursor() as cur:
            cur.execute(
                "UPDATE customers SET display_name = full_name "
                "WHERE id >= %s AND id < %s AND display_name IS NULL",
                (lower, upper),
            )
        conn.commit()   # one short transaction per batch
        print(f"backfilled ids [{lower}, {upper})")  # log progress for restarts
        lower = upper
        time.sleep(PAUSE_SECONDS)

Python using a generic DB-API connection with %s placeholders (adjust for your driver) to show the dual-write path used from release N+1 to N+3 and an idempotent, throttled backfill that copies full_name into display_name one primary-key range at a time.

Implementing it: decoupled migrations, feature flags and CI guardrails

Run migrations as their own step

Run migrations as a separate pipeline stage or one-off job that finishes before any new application instance starts. Coupling them to app boot causes several instances to race for the migration lock, and a failed migration takes the release down with it. Separating them means every migration must work with two codebases: the release currently serving traffic and the one about to replace it. If it can't, it isn't ready to ship.

Define the rollback target

"Roll back" should mean "return to the previous release" only while that release is still compatible with the current schema. If three releases have shipped since a migration ran, write down which release is still a valid rollback target. Contract steps should only remove support for releases older than that target.

Gate new writes with flags

Put new write paths, such as dual-writing display_name, behind a feature flag. If the new path corrupts data, you can turn writes off without redeploying. Test the off position in CI and in staging. An off branch that has never run is not a fallback.

Guardrails in CI

• A review rule or migration lint that rejects drops, renames, and type narrowing in the same release as the code that stops using the column.
• A job that applies schema N and boots release N-1 against it, then runs its smoke and integration suites.
• Checks that flag lock-heavy operations, such as table rewrites or non-concurrent index builds, on large tables.

The N-1 job catches boot failures and broken queries. It will not catch semantic drift, such as old code ignoring a column that new code now relies on.

Copy#!/usr/bin/env bash
# CI guardrails: lint new migrations, then boot release N-1 against schema N.
# ./scripts/*.sh are this project's own wrappers; swap in your migration tool and test runner.
set -euo pipefail

PREV_TAG=$(git describe --tags --abbrev=0 HEAD^)   # release N-1, still the rollback target
export DATABASE_URL="postgres://ci:ci-password@localhost:5432/app_ci"

# 1. Lint: reject destructive or lock-heavy DDL in migrations added since N-1.
#    An intended contract step gets an explicit review override, not a silent pass.
NEW_MIGRATIONS=$(git diff --name-only --diff-filter=A "$PREV_TAG"...HEAD -- db/migrations/)
if [ -n "$NEW_MIGRATIONS" ]; then
  if grep -nEi 'DROP (COLUMN|TABLE)|RENAME (COLUMN|TO)|ALTER COLUMN .* TYPE' $NEW_MIGRATIONS; then
    echo "Destructive change found: ship it in a later contract release." >&2
    exit 1
  fi
  # Index builds without CONCURRENTLY block writes on large tables.
  if grep -nEi 'CREATE (UNIQUE )?INDEX' $NEW_MIGRATIONS | grep -viE 'CONCURRENTLY'; then
    echo "Non-concurrent index build found." >&2
    exit 1
  fi
fi

# 2. Apply schema N from the current commit, as the real migration step would.
./scripts/migrate.sh

# 3. Check out release N-1 separately and run it against schema N.
git worktree add /tmp/release-prev "$PREV_TAG"
cd /tmp/release-prev
./scripts/start-app.sh &              # boot failures surface here
APP_PID=$!
trap 'kill "$APP_PID" 2>/dev/null || true' EXIT

# Wait for the app to report healthy, then run N-1's own suites.
for _ in $(seq 1 30); do
  curl --fail --silent  >/dev/null && break
  sleep 2
done
curl --fail --silent  >/dev/null
./scripts/smoke-test.sh && ./scripts/integration-test.sh
# Catches boot failures and broken queries, not semantic drift between N-1 and N.

A CI script that blocks destructive and non-concurrent DDL in new migrations, then applies schema N and runs release N-1's smoke and integration suites against it.

Rollback decision runbook for an incident

Work through these steps in order before anyone triggers a deploy.

1. Confirm what actually ran. Read the migration history table in production rather than the release notes, because a partially failed deploy may have applied some migrations and not others. Tag each applied migration with its class: additive, expand, contract, or destructive.
2. Inspect writes from the bad window. Query rows created or updated since the deploy. Look for new enum values, changed JSON shapes, or nulls in columns the old code treats as required. If the previous release throws on even one of those rows, a rollback becomes a second outage.
3. Choose the lever.

Roll forward is faster when the cause is understood, the fix is small, and the pipeline completes in minutes. A code rollback is safer when builds and tests are slow, because redeploying a known-good artifact skips both. It is also safer when nobody on the call can say with confidence what the correct fix is.

A down-migration is justified only when the migration itself is the cause of the incident and reversing it loses no data. Examples are an added index that changed query plans for the worse, or a new constraint that is rejecting valid writes. Never run a down-migration to undo a destructive change. Treat that as a restore from backup, which is a separate procedure.

Alternatives and when to avoid full backward compatibility

Some changes cannot be made backward compatible at a reasonable cost. Splitting one table into several, changing a primary key's type, or replacing a denormalized model can mean many releases of dual-writing code. For these, choose one of three deliberate alternatives.

Maintenance window. Stop traffic, run the migration, deploy, and verify before reopening. Rollback is simple because nothing wrote to the new schema while you checked it. You can still revert code and schema together. The cost is planned downtime, and that only works if the migration fits inside the window.

Roll-forward-only release. Accept that this release cannot be reverted. Announce it as such, ship it alone with no unrelated changes, and keep the team ready to hotfix. This is honest about the risk rather than pretending a down-migration will save you.

Shadow table. Build the new structure beside the old one, backfill it, and keep both in sync. Then switch reads to the new table. The old table stays intact as the fallback until you trust the new one.

Backups and point-in-time recovery are the disaster path, not the rollback path. A restore discards every write since the restore point, including orders, payments and sign-ups. Whether that loss is acceptable is a business decision, not a deploy decision.

Impact on development cost and engineering time

Expand-and-contract turns a one-deploy rename into three or four releases: add the column, dual-write and backfill, switch reads, then drop the old column. Each step has to reach every instance before the next one starts, so lead time grows from hours to days.

Until the work finishes, the codebase carries dual-write code and redundant columns. Authors have to reason about two code versions sharing one schema. Running migrations ahead of code can also hide defects that only show up once the new code exercises the schema. Backfills and dual-writes add write load. If one write path is missed, such as a batch job or an admin script, the old and new columns drift apart without any error. Feature flags add state that someone must later remove. Contract steps that never ship pile up as schema debt.

The payoff comes during incidents. Rollback becomes an ordinary redeploy instead of an emergency down-migration or a restore.

Illustrative estimate (assumed numbers, not a benchmark). Net hours per year = (avoided hard rollbacks × hours saved each) − (risky migrations × extra hours each)

• Assume 30 risky migrations at 4 extra hours each: 120 hours of cost.
• Assume 3 incidents where a code revert would otherwise have needed a down-migration or restore, each saving 40 responder-hours: 120 hours saved.

That is roughly break-even in engineering time. The real argument is the shorter outage and the reduced risk of data loss, which this formula does not count.

To keep contract steps from stalling:

• Open the contract ticket when the expand change merges, and give it an owner and a due date.
• Record the deprecated column and its flag in a tracked list.
• Fail CI when an item on that list passes its due date.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to rollback deployment database migration, this is where it helps:

Saves time

• Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
• One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
• Changing a shared variable is one edit and one review instead of editing every service's .env by hand.

Saves cost

• Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
• Less duplicated configuration to maintain: one shared definition inherited by many services.
• A wrong or stale value is caught at review time by the services it affects, not by a failed deploy.
• OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.