What is a pre-deploy migration step, and why shouldn't migrations run at app startup?

A pre-deploy migration step is a dedicated stage in your pipeline that applies schema changes to the database, verifies they succeeded, and only then allows the new application version to roll out. It is gated: if the migration fails, the deploy stops, and the running version keeps serving traffic against an unchanged schema.

The reason to separate it comes from how Kubernetes rolling updates work. During a rollout, old and new pods run side by side and share one database. For some window, which can last minutes on large deployments, the schema must satisfy both versions at once. A migration that renames a column or adds a NOT NULL constraint can break the old pods mid-rollout, even when the new code is correct.

Running migrations in each pod's entrypoint seems simpler, but it fails in three ways:

  • Concurrent execution. Ten replicas starting together means ten processes racing to apply the same change. Lock tables in migration tools reduce this risk, but every pod still blocks waiting for the lock.
  • Startup delays. A slow migration holds the container in startup. Readiness or liveness probes can time out, causing restarts that interrupt the migration partway through.
  • Unclear ownership. When a migration fails, the error is buried in one pod's logs among many crash-looping replicas, and no single pipeline step clearly reports the failure.

This approach is for teams running rolling deploys against a shared database. It is worth the effort once you have more than one replica, or once migrations take longer than a few seconds.

  CI/CD PIPELINE (each stage gates the next)
  +---------+     +------------------+     +-----------+     +------------+
  |  Build  |---->| Migrate (1 job)  |---->|  Verify   |---->|  Rollout   |
  |  image  |     | apply schema     |     |  schema   |     |  new pods  |
  +---------+     +------------------+     +-----------+     +------------+
                      |       |                  |                 |
                      |       | fail             | fail            |
                      |       v                  v                 |
                      |   +--------------------------+             |
                      |   | STOP: deploy halts       |             |
                      |   | old version keeps serving|             |
                      |   +--------------------------+             |
                      |                                            |
                      | one writer                                 |
                      v                                            v
  +--------------------------------+     +----------------------------------+
  |  Shared database (one schema)  |     |  Rolling update window           |
  |  must satisfy BOTH old (v1)    |<----|  +---------+        +---------+  |
  |  and new (v2) app versions     |<----|  | old pod |        | new pod |  |
  |  queries from both at once     |     |  |   v1    |        |   v2    |  |
  +--------------------------------+     |  +---------+        +---------+  |
                                         +----------------------------------+

A single gated migration job changes the schema before rollout, and a failure stops the deploy. During the rolling update that follows, old and new pods query the same database, so the schema must work for both versions.

Who should use a gated pre-deploy migration step?

Use it if most of the following describe your setup:

  • You run more than one replica with a rolling update strategy. This is the case where old and new pods share the database during rollout. It is also where migrate-on-startup races between pods.
  • Several services or teams deploy against the same database. A single, logged migration step gives one place to see what changed and when.
  • Your tables are large enough that some DDL takes noticeable time or locks. Isolating that work in a Job with its own timeouts stops it from tangling with pod startup and probe behaviour.
  • You need an audit trail or approvals for schema changes. A pipeline stage or Helm hook can be reviewed, approved and logged in a way that pod entrypoints cannot.
  • You deploy frequently. The more often you ship, the more often a migration failure in a startup path turns into a crash loop in production.

The people who interact with it most are application developers who write migrations and classify them, reviewers who check expand/contract sequencing, and platform or database owners who maintain the Job template, the DDL credentials and the pipeline gate.

If you run a single instance, use the Recreate strategy with acceptable downtime, or are at an early stage where the schema changes daily, the last section explains why a lighter approach is usually enough.

How does the gated migration step work end to end?

The mechanism has a few moving parts. Each one is covered in detail in the sections below.

  1. Before merge. The developer writes the migration as a new versioned file and declares its class: additive, backfill, locking or destructive. A CI check scans the file and fails if the declared class does not match statements such as DROP or ALTER COLUMN ... TYPE. Breaking changes are split into expand/contract releases.
  2. Build. The pipeline builds an image that contains the migration files and the migration tool. This is either a dedicated migrations image or the application image with a command override.
  3. Migrate. The pipeline creates a Kubernetes Job, either directly or as a Helm pre-install,pre-upgrade hook. The Job connects with DDL-capable credentials from a separate Secret. It first takes a lock, either the tool's lock table or a database advisory lock, so only one run proceeds. It then compares the migrations recorded as applied in the database against the files in the image and applies each pending one in order. After each one succeeds, it records the version. Session-level lock and statement timeouts make it fail fast rather than stall production queries.
  4. Gate. The pipeline blocks on the Job's complete or failed condition, or Helm waits on the hook. A failure or a timeout stops the deploy. Because the old application version is compatible with the expanded schema, it keeps serving.
  5. Verify. A short step confirms the database reports the expected schema version, the latest applied migration, before anything else proceeds.
  6. Rollout. Kubernetes performs the rolling update. Old and new pods coexist against a schema that supports both. In a later release, the contract step removes what is no longer referenced.

The key property is ordering. The schema always moves first, in one place, under one writer. The code then follows, and it never depends on a schema state that has not been applied and verified.

How does the expand/contract pattern keep old and new pods compatible?

During a rolling deploy, old and new pods serve traffic against the same database at the same time. Any schema change must work for both versions until the last old pod is gone. Expand/contract, also called parallel change, does this by never making a breaking change in one step.

Worked example: renaming users.fullname to users.display_name

Release 1: expand. Add display_name as a nullable column with no default that forces a table rewrite. Deploy code that writes to both columns and still reads from fullname. Old pods ignore the new column, so nothing breaks. After all pods run this version, backfill display_name in small batches so you avoid long locks.

Release 2: switch reads. Read from display_name and keep writing to both columns. Rolling back to Release 1 is still safe because fullname stays current.

Release 3: contract. Stop writing fullname. Then, in this release's migration or a later one, drop it. The column is only removed once no running code references it.

What it costs

One logical rename becomes three deploys, and each must fully roll out before the next begins. The contract step is easy to forget. When it is skipped, dead columns, dual-write code and stale indexes stay around indefinitely, so track it as a ticket with an owner.

Renames are the simple case. Type changes need a new column and conversion logic. Adding a NOT NULL constraint or foreign key needs a backfill first, and often a constraint-validation approach that avoids a long lock on a large table.

-- PostgreSQL. Each block ships as a separate release's migration.

-- Release 1: expand. A nullable column with no default is a
-- metadata-only change, so there is no table rewrite.
ALTER TABLE users ADD COLUMN display_name text;

-- Backfill only after every pod writes to both columns.
-- Run this repeatedly from a job until it updates 0 rows.
-- Small batches keep each row lock short.
UPDATE users
SET display_name = fullname
WHERE id IN (
  SELECT id FROM users
  WHERE display_name IS NULL AND fullname IS NOT NULL
  ORDER BY id
  LIMIT 1000
);

-- Release 2: switch reads. Application code changes only.
-- No schema change, so rolling back to Release 1 stays safe.

-- Release 3: contract. Run only after no deployed code reads
-- or writes fullname, including any image you might roll back to.
ALTER TABLE users DROP COLUMN fullname;

-- Optional, if display_name must become required.
-- NOT VALID skips the full-table scan when the constraint is added.
-- VALIDATE then checks existing rows under a weaker lock.
ALTER TABLE users
  ADD CONSTRAINT users_display_name_not_null
  CHECK (display_name IS NOT NULL) NOT VALID;
ALTER TABLE users VALIDATE CONSTRAINT users_display_name_not_null;

PostgreSQL migrations for the three-release rename of users.fullname to users.display_name, covering the batched backfill and an optional NOT NULL check that is added without a long lock.

How should you classify each migration before merge?

Assign each migration a class before it merges. The class determines how it ships, who must approve it, and what backup must exist, so ambiguous changes get the scrutiny they need while the change is still a diff.

Class Typical changes Rollout path Required review Backup requirement
Additive and safe New table, nullable column, index built without blocking writes Pre-deploy Job in the same release Standard code review Routine scheduled backups
Needs a backfill Populating a new column, preparing for NOT NULL Schema change pre-deploy, batched backfill as a separate job, constraint in a later release Reviewer who knows the table's size and write rate Confirm a recent restorable backup exists
Locks or rewrites a large table Column type change, index build without an online option Separate from app deploys, either in a window or with an online schema-change approach Database or platform owner, tested on a production-sized copy Fresh backup or snapshot immediately before
Destructive Dropping a column or table, renames Only in the contract phase, after no running code reads the object Two approvers, including the data owner Verified fresh backup with a tested restore

Enforce this mechanically rather than relying on memory. Add a required class field to the pull request template. Add a CI check that scans migration files for statements such as DROP, ALTER COLUMN ... TYPE, or new constraints, and fails unless the declared class matches. This takes a reviewer a few minutes. It replaces finding the same problem when a table lock stalls every request during a deploy, when the fix means an incident call, a rollback decision, and a postmortem.

How do you run migrations as a Kubernetes Job or Helm hook?

Define the migration as a Job that runs the migration tool's own command, either from a dedicated migrations image or from the application image with an explicit command override. Set restartPolicy: Never so each retry is a fresh pod whose logs survive for inspection. Set backoffLimit low, such as 1 or 2, because a migration that failed halfway rarely succeeds on blind retry. Set activeDeadlineSeconds to the longest runtime you will tolerate, so the Job fails cleanly instead of holding locks indefinitely.

In a plain pipeline, apply the Job and block on kubectl wait --for=condition=complete job/<name> --timeout=<duration>. A non-zero exit fails the stage, and the rollout step never runs. Also check for condition=failed so a failed Job stops the pipeline right away rather than waiting for the timeout.

With Helm, add the annotation helm.sh/hook: pre-install,pre-upgrade to the same Job. Use helm.sh/hook-weight to order it relative to other hooks, and helm.sh/hook-delete-policy: before-hook-creation,hook-succeeded to keep failed pods available for debugging. Hook resources sit outside the release's normal resource tracking, so helm uninstall does not remove them unless a delete policy does. If the hook fails, Helm marks the release failed and does not apply the new manifests. helm rollback does not undo a hook Job. Whatever DDL already ran stays in the database.

Run the Job under its own ServiceAccount and mount a separate Secret that holds DDL-capable credentials. The application's runtime credentials should never be able to alter the schema.

apiVersion: batch/v1
kind: Job
metadata:
  name: shop-db-migrate
  annotations:
    # Run before Helm applies the release manifests on install and upgrade
    "helm.sh/hook": pre-install,pre-upgrade
    # Lower weights run first; leave room for earlier setup hooks
    "helm.sh/hook-weight": "-5"
    # Delete the old Job before a new one is created, and delete it on success.
    # A failed Job is kept so its pod logs can be inspected.
    "helm.sh/hook-delete-policy": before-hook-creation,hook-succeeded
spec:
  # A half-applied migration rarely succeeds on blind retry
  backoffLimit: 1
  # Fail the Job, and the hook, instead of holding locks indefinitely
  activeDeadlineSeconds: 600
  template:
    spec:
      # Each retry is a fresh pod, so earlier pods and their logs remain
      restartPolicy: Never
      # Scoped identity for migrations only. It must exist before this hook runs,
      # either created outside the chart or by a hook with a lower weight.
      serviceAccountName: shop-migrator
      containers:
        - name: migrate
          image: registry.example.com/shop/migrations:1.4.2
          # Run the migration tool directly, not the app server
          command: ["/app/migrate", "up"]
          envFrom:
            # DDL-capable credentials. The app's runtime Secret cannot alter the schema.
            - secretRef:
                name: shop-db-ddl-credentials

A migration Job configured as a Helm pre-install/pre-upgrade hook, with a low backoffLimit, a runtime deadline, restartPolicy Never, a dedicated ServiceAccount, a separate DDL Secret, and hook weight and delete-policy annotations.

How do you prevent concurrent runs, long locks, and partial application?

  • Serialize runs. Rely on the migration tool's lock table, or take a database advisory lock (for example, pg_advisory_lock in PostgreSQL or GET_LOCK in MySQL) before applying anything. Two Jobs can overlap after a retry, a re-triggered pipeline, or parallel releases, and only the database can arbitrate between them reliably.
  • Make migrations idempotent. Use guards such as IF NOT EXISTS so a re-run after a partial failure succeeds instead of erroring on objects that already exist.
  • Set timeouts on the migration session. A short lock timeout makes DDL fail fast instead of queuing behind a long transaction. While it waits, it also blocks every query queued behind it. A statement timeout caps runaway operations. A failed migration that you can retry is cheaper than a stalled one that blocks production traffic.
  • Build indexes online. Use CREATE INDEX CONCURRENTLY in PostgreSQL or your engine's online DDL option. In PostgreSQL, a concurrent build cannot run inside a transaction block, so it needs its own non-transactional migration.
  • Keep backfills out of schema migrations. Run large data updates as separate, batched, resumable jobs. One giant UPDATE holds locks and generates load for the entire run.
  • Plan for mid-migration kills. MySQL commits implicitly around most DDL statements. A Job killed halfway through can leave some statements applied and the version record unwritten. Keep migrations to one DDL statement each, and set the Job's activeDeadlineSeconds below the pipeline or Helm timeout. Kubernetes then terminates the Job first and reports the failure, instead of the pipeline giving up while the Job keeps running unobserved.
  • Track progress per target. In multi-tenant or multi-database setups, record the migration state of each database. After a failure, resume from the targets that are not yet done rather than re-running against every database.
-- Run with: psql "$DATABASE_URL" -f 0042_add_status.sql   (no -1 / --single-transaction)
\set ON_ERROR_STOP on

-- Session settings: fail fast instead of queuing behind long transactions.
SET lock_timeout = '5s';        -- give up if a table lock is not granted quickly
SET statement_timeout = '60s';  -- cap ordinary DDL

-- Serialize runs: a session-level advisory lock is held until this session ends,
-- so an overlapping Job (retry, re-triggered pipeline) exits instead of racing.
DO $$
BEGIN
  IF NOT pg_try_advisory_lock(727001) THEN
    RAISE EXCEPTION 'another migration run holds advisory lock 727001';
  END IF;
END $$;

-- Idempotent, metadata-only change: safe to re-run after a partial failure.
ALTER TABLE orders ADD COLUMN IF NOT EXISTS status text;

-- Online index build. CONCURRENTLY cannot run inside a transaction block,
-- which is why this file is not wrapped in BEGIN/COMMIT.
-- The build can take a while, so lift the statement cap for this step only.
SET statement_timeout = 0;
-- Caveat: if a concurrent build is killed, it leaves an INVALID index that
-- IF NOT EXISTS will silently skip. Check pg_index.indisvalid and DROP it
-- before retrying.
CREATE INDEX CONCURRENTLY IF NOT EXISTS orders_status_idx
  ON orders (status);

-- Backfilling orders.status belongs in a separate batched job, not here.

SELECT pg_advisory_unlock(727001);

A PostgreSQL migration run through psql that takes an advisory lock to block concurrent runs, sets lock and statement timeouts, uses idempotent IF NOT EXISTS guards, and builds an index concurrently outside a transaction block.

What does rollback actually mean for schema changes?

Rolling back the application is cheap: redeploy the previous image and the old pods come back within minutes. Rolling back the schema is a different operation. A migration that dropped a column, merged two tables, or rewrote values in place has destroyed information, and no down-migration can recreate it. Reversing such a step leaves the database structurally correct but holding empty or wrong data.

  • Default to roll-forward. When a migration causes problems, ship a new migration that fixes it. Because expand/contract keeps the previous release compatible with the current schema, you can usually revert the app and leave the schema where it is.
  • Treat down-migrations with suspicion. They rarely run, so they rot: they reference columns that later changes renamed, or they assume data shapes that no longer exist. If you keep them, run them in CI against a realistic schema. Apply up, then down, then up again, and assert the result.
  • Gate destructive steps on a verified backup. Before any drop or in-place transform, require a backup or snapshot you have confirmed is restorable, and record its identifier in the release.
  • Make schema state visible.
    • Log every applied migration version from the Job.
    • Store the schema version alongside the release version so you can tell which code ran against which schema.
    • Alert on migration Jobs that stay Pending or keep running past their expected duration. A stuck Job often means a lock wait, and it is blocking the deploy behind it.

How can a gated migration step save engineering time?

The step adds some pipeline time to each deploy. It saves engineering time by moving failures to places where they are cheap to find and fix.

  • Problems surface at review instead of during an incident. Classification and the CI check that scans migration files catch locking or destructive changes while they are still a diff. Fixing a migration before merge takes minutes of a reviewer's attention. Diagnosing a table lock that stalls production traffic pulls several people into an incident.
  • One place to look when something fails. A failed Job is one pod with preserved logs and a pipeline stage that turned red. Compare that with searching through the logs of many crash-looping replicas to find which one ran the migration and where it stopped.
  • Failed deploys are non-events. Because the gate stops the rollout and the old version keeps serving against a compatible schema, a failed migration does not need an emergency rollback. The team can fix forward on its own schedule.
  • Less time debugging startup behaviour. Removing migrations from the entrypoint eliminates a class of problems where probes time out, pods restart mid-migration, and migration failures look like application crashes.
  • Less environment drift. The same Job runs the same way in staging and production, so differences between environments are visible as different applied versions rather than discovered through unexplained failures.
  • Cheaper recovery from partial runs. Idempotent migrations, advisory locks and per-target progress tracking mean a retry resumes cleanly. Nobody has to inspect the database by hand to work out what was applied.
  • No cost on deploys without schema changes. With no-op detection, deploys that carry no new migrations skip the Job entirely, so the time cost only falls on the deploys that need the protection. The next section works through an illustrative estimate of this trade-off.

What are the alternatives, and when should you avoid a gated migration step?

Helm hook Separate pipeline stage Migrate-on-startup or manual
Control Ordering tied to the release Approvals, retries and ordering live in CI Pods race each other, or a person decides
Visibility Job logs plus release status Stage status in the pipeline UI Scattered pod logs, or none
Maintenance cost Hook annotations and delete policies Extra stage and cluster or database credentials in CI Lowest
Failure behaviour Failed hook fails the release Stage fails and the deploy never starts CrashLoopBackOff, or drift between environments

A separate stage gives you the clearest gate, but it opens a window where the new schema is live and the old code is still serving. That window can stretch to hours if the deploy stage waits for approval or is retried. Expand/contract makes this safe. Without it, the window becomes an outage.

Gating every deploy has two costs. Scheduling the Job, pulling the image and connecting to the database add latency, and each step is another place the deploy can fail. Most deploys contain no schema change. If you compare the pending migrations against the applied ones and skip the Job when nothing is new, you pay that cost only when it buys you something.

Illustrative estimate, with assumed inputs: each gated run adds 60 seconds, and the team ships 20 deploys a week. Gating every deploy costs about 17 hours of pipeline time a year. If only 2 of those 20 deploys carry migrations, no-op detection cuts this to under 2 hours. That is less time than one incident takes to recover from, if a half-applied startup migration costs three engineers an afternoon.

The full approach is overkill in three cases:

  • Single-instance apps.
  • Recreate-strategy deploys where brief downtime is acceptable.
  • Early-stage projects whose schema changes daily.

In all three, a migration on startup behind a lock is enough.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to database migrations ci cd, this is where it helps:

Saves time

  • Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
  • One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
  • Reviewers see the exact diff and the list of affected services up front, so approval is a decision rather than an investigation.

Saves cost

  • Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
  • Less duplicated configuration to maintain: one shared definition inherited by many services.
  • Per-service approve/reject means a bad shared value can be refused where it would break something, before it is live.
  • OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.