How do init containers, Jobs and Helm hooks run Kubernetes database migrations?
All three mechanisms run your migration command in a container. They differ in how many times it runs and what it is ordered against.
- Init containers belong to the pod spec. Every pod runs its init containers in order, and each must exit successfully before the app containers start. A Deployment with ten replicas runs the migration ten times, possibly concurrently, and the migration repeats whenever a pod is rescheduled. Ordering is per pod: the schema change is guaranteed only relative to that pod's own app container.
- Kubernetes Jobs are separate objects that run pods until a specified number complete successfully. A migration Job runs once per Job object, with retries. It has no built-in relationship to your Deployment, so you have to order it against the rollout yourself.
- Helm hooks are ordinary resources, usually Jobs, annotated with a lifecycle point such as
pre-installorpre-upgrade. Helm creates the hook, waits for a hook Job to complete, and only then applies the rest of the release. Helm supplies the ordering that a plain Job lacks.
Rolling update timeline (old = v1, new = v2):
| Step | Cluster state | Init container | Plain Job | pre-upgrade hook |
|---|---|---|---|---|
| 1 | Only v1 pods serving | — | Runs whenever applied | Runs here, before any v2 pod exists |
| 2 | First v2 pod created | Runs in this pod | — | — |
| 3 | v1 and v2 pods serve together | Runs in each new v2 pod | — | — |
| 4 | Only v2 pods serving | Runs again on any reschedule | — | — |
In every column, v1 pods are still serving traffic after the schema has changed.
STEP 1 STEP 2 STEP 3 STEP 4
┌──────────────┬──────────────┬──────────────┬──────────────┐
Cluster state │ v1 v1 v1 │ v1 v1 v2 │ v1 v2 v2 │ v2 v2 v2 │
├──────────────┼──────────────┼──────────────┼──────────────┤
pre-upgrade │ MIGRATE │ │ │ │
hook (Job) │ before any v2│ │ │ │
│ Helm waits, │ │ │ │
│ then applies │ │ │ │
│ Deployment │ │ │ │
├──────────────┴──────────────┴──────────────┴──────────────┤
Plain Job │ MIGRATE once (with retries), whenever the Job is applied │
│ ◀── could land in any step: no built-in ordering ──▶ │
├──────────────┬──────────────┬──────────────┬──────────────┤
Init │ │ MIGRATE │ MIGRATE │ MIGRATE │
container │ │ in 1st v2 pod│ in each v2 │ on any pod │
│ │ │ (concurrent) │ reschedule │
└──────────────┴──────────────┴──────────────┴──────────────┘
time ─────────────────────────────────────────────────────────────────────▶
In every row, v1 pods are still serving traffic after the schema changes.
Where a pre-upgrade hook, a plain Job and an init container each run the migration during a v1-to-v2 rolling update.
How does a migration run work end to end?
Whichever mechanism triggers it, a single migration run follows the same sequence inside the container and inside the cluster:
- The image carries the tool and the migrations. The container image includes the migration tool and the versioned migration files for this release. Usually that is the application image or a sibling image built from the same commit, so the schema changes and the code that expects them are released together.
- The tool connects and takes a lock. It reads connection details from environment variables or a mounted Secret, connects to the database, and takes the tool's own lock so that no other migrator applies changes at the same time.
- It compares versions. The tool reads its history table in the target database, compares the recorded versions with the migration files in the image, and builds a list of pending migrations.
- It applies and records each migration. For each pending migration, the tool runs the statements, then writes a row to the history table. Where the database and the statement allow it, both happen in one transaction.
- It exits with a status code. Exit code 0 means every pending migration was applied, or there was nothing to do. A non-zero code means something failed.
- Kubernetes turns the exit code into state. For an init container, a 0 lets the kubelet start the app containers, and a non-zero triggers a restart with backoff. For a Job, a successful pod counts toward completion and the Job gets the
Completecondition. Failures create new pods untilbackoffLimitis reached, after which the Job gets theFailedcondition. - Something waits on that state before rolling out the app. For an init container, the pod itself waits. For a plain Job, your pipeline or GitOps tool waits, for example with
kubectl wait. For a Helm hook, Helm waits. Only then are new application pods created against the migrated schema.
Steps 1 to 5 are identical across all three approaches. The differences come from steps 6 and 7: how often the run is triggered, how failure is reported, and what is ordered after it.
Why are init containers the wrong default for schema migrations?
That table shows the core problem: an init container is scoped to a pod, while a schema migration is scoped to a database. Kubernetes has no concept of "run this once per rollout" at the pod level. With three replicas and a rolling update that surges new pods, several init containers can start the same migration at the same moment. Whether that is safe depends entirely on the migration tool's locking, such as a database advisory lock or a lock row in its history table. If the tool doesn't lock, or the lock is released before the work commits, you get duplicate DDL or a half-applied schema.
Long migrations cause a second problem. While an init container runs, the pod sits in Init and never becomes Ready. A ten-minute index build stalls the rollout and can exceed the Deployment's progressDeadlineSeconds, which marks the rollout as failed even though the migration may still be running. If the init container fails, the kubelet restarts it with exponential backoff (Init:CrashLoopBackOff). That retries a possibly partial migration on a delay you don't control.
The cost also recurs. Every scale-up, node drain, eviction or reschedule reruns the init container. Each run opens a database connection just to confirm there is nothing to do. During an autoscaling burst, those connections arrive all at once.
Finally, credentials that can alter the schema must be referenced in the pod spec of every replica. That puts them on every node running the app, for the app's whole lifetime.
Init containers are acceptable in three cases: single-replica deployments, local or dev environments, or trivial, fast migrations run by a tool whose locking you have verified.
apiVersion: apps/v1
kind: Deployment
metadata:
name: shop-api
spec:
replicas: 3 # each new pod runs the migration, not once per rollout
progressDeadlineSeconds: 600 # a long migration keeps pods in Init and can trip this
strategy:
type: RollingUpdate
rollingUpdate:
maxSurge: 1 # surged pods can start while others are still migrating
maxUnavailable: 0
selector:
matchLabels:
app: shop-api
template:
metadata:
labels:
app: shop-api
spec:
initContainers:
- name: migrate
image: registry.example.com/shop-api:1.4.2
# Safe only if the tool takes a DB-level lock held until its work commits.
# Also reruns on every scale-up, drain, eviction and reschedule.
command: ["./bin/migrate", "up"]
env:
- name: DATABASE_URL
valueFrom:
secretKeyRef:
name: shop-db-migrator # DDL-capable credentials on every replica's node
key: url
containers:
- name: api
image: registry.example.com/shop-api:1.4.2
ports:
- containerPort: 8080
A three-replica Deployment that runs migrations in an init container, annotated with where concurrent runs, rollout deadlines, repeated reruns and schema-altering credentials cause problems.
How do you implement migrations as a dedicated Kubernetes Job?
A dedicated Job moves the migration out of the application's pod lifecycle and gives it its own contract with the cluster. The manifest should set a few fields deliberately:
restartPolicy: Neverso each failed attempt leaves a pod behind whose logs you can read, instead of a container restarting in place.backoffLimitset low, such as 1 or 2. A migration that failed twice is unlikely to succeed on a third blind retry.activeDeadlineSecondsto cap total runtime. A migration stuck on a table lock should fail visibly rather than hang the pipeline.- Resource requests and limits sized for the migration tool, not copied from the app container.
- A separate Secret holding a database role that can alter schema. The application's role then needs only read and write access to data, so a compromised app pod cannot drop tables.
Because Job specs are largely immutable, give each Job a release-specific name, for example by suffixing the image tag. The Job then runs once per release, and you have one object to inspect for success or failure and one place to read logs.
In CI/CD, apply the Job, then block on kubectl wait --for=condition=complete job/<name> --timeout=<duration>. Set the timeout slightly above activeDeadlineSeconds. If the Job fails, the wait does not return early. It times out, the step exits non-zero, and the pipeline stops before you apply the Deployment.
The trade-off is that the cluster does not enforce this ordering. Only the pipeline does. If someone applies the Deployment manifest directly, or a tool syncs it independently, new pods can start against the old schema.
#!/usr/bin/env bash
set -euo pipefail
TAG="${IMAGE_TAG:?IMAGE_TAG must be set}" # e.g. 1.42.0
JOB="db-migrate-${TAG//./-}" # release-specific name; Job specs are largely immutable
kubectl apply -f - <<EOF
apiVersion: batch/v1
kind: Job
metadata:
name: ${JOB}
spec:
backoffLimit: 1 # one retry; a blind third attempt rarely helps
activeDeadlineSeconds: 600 # a migration stuck on a lock fails instead of hanging
template:
spec:
restartPolicy: Never # each failed attempt leaves a pod whose logs you can read
containers:
- name: migrate
image: registry.example.com/shop/migrate:${TAG}
envFrom:
- secretRef:
name: db-migrator-credentials # schema-altering role, separate from the app's Secret
resources: # sized for the migration tool, not the app
requests: {cpu: 100m, memory: 128Mi}
limits: {cpu: 500m, memory: 256Mi}
EOF
kubectl wait --for=condition=complete "job/${JOB}" --timeout=660s || {
kubectl logs "job/${JOB}" || true
exit 1
}
kubectl set image deployment/shop app="registry.example.com/shop/app:${TAG}"
kubectl rollout status deployment/shop --timeout=300s
A pipeline step applies a release-named migration Job with its own limits, retry policy, deadline and narrower credential Secret, then uses kubectl wait so the Deployment rolls out only after the migration completes.
How do Helm hooks sequence a migration Job, and where do they leak?
Helm hooks address that gap by moving the ordering into the release itself. You annotate the migration Job with helm.sh/hook: pre-install,pre-upgrade, and Helm renders and applies it before any regular manifests. For Job hooks, Helm waits until the Job completes. If it fails, Helm marks the release as failed and does not apply the new Deployment, so pods running incompatible code never start. helm.sh/hook-weight orders multiple hooks within the same phase, lowest first. helm.sh/hook-delete-policy controls cleanup and accepts before-hook-creation, hook-succeeded and hook-failed.
Where hooks leak
Hook resources sit outside the release's normal manifest tracking. Helm does not reconcile, diff or garbage-collect them like regular release objects. As a result, cleanup depends entirely on the delete policy:
- Without a delete policy, a failed Job lingers, and the next upgrade fails on a name conflict.
- With
hook-failed, the Job and its pods are removed, often before anyone has read the logs.
before-hook-creation with a log shipper is usually the safer combination, because the failed Job stays inspectable until the next attempt replaces it.
Ordering also catches people out. A pre-install hook runs before the chart's Secrets and ConfigMaps exist. If the migration needs database credentials, those objects must also be hooks with a lower weight.
Timeouts are a third gap. If a migration outlasts Helm's --timeout, Helm reports failure, but the Job keeps running and altering the schema.
Finally, helm rollback restores manifests, not data. It does not undo any schema change the migration already applied.
apiVersion: v1
kind: Secret
metadata:
name: {{ .Release.Name }}-migrate-db
annotations:
"helm.sh/hook": pre-install,pre-upgrade
"helm.sh/hook-weight": "-5" # created before the Job (weight 0)
"helm.sh/hook-delete-policy": before-hook-creation
stringData:
DATABASE_URL: postgres://app:changeme@db.example.internal:5432/app
---
apiVersion: batch/v1
kind: Job
metadata:
name: {{ .Release.Name }}-migrate
annotations:
"helm.sh/hook": pre-install,pre-upgrade
"helm.sh/hook-weight": "0"
# Keep a failed Job for inspection; replace it on the next attempt.
"helm.sh/hook-delete-policy": before-hook-creation
spec:
backoffLimit: 0 # a migration should not blindly retry
activeDeadlineSeconds: 540 # stop the Job before helm --timeout 10m expires
template:
spec:
restartPolicy: Never
containers:
- name: migrate
image: registry.example.com/app:{{ .Chart.AppVersion }}
command: ["./migrate", "up"]
envFrom:
- secretRef:
name: {{ .Release.Name }}-migrate-db
A migration Job and its credentials Secret run as pre-install/pre-upgrade hooks, ordered by weight, with before-hook-creation cleanup and a Job deadline shorter than Helm's timeout so the Job stops before Helm reports failure.
Why is expand/contract required no matter which mechanism you choose?
Every mechanism in this article shares one constraint: the database is updated once, but the application is updated pod by pod. During a rolling update, pods running version N and version N+1 serve traffic against the same schema at the same time, sometimes for minutes. After a helm rollback, version N comes back and runs against a schema that version N+1 has already migrated. A migration is safe only if the schema it produces works for both the previous and the next application version.
A column rename across three releases
Suppose you want to rename users.fullname to users.display_name. A single ALTER TABLE ... RENAME COLUMN breaks every old pod the moment it commits. Expand/contract spreads the change across releases instead:
- Expand (release N+1): Add
display_nameas a nullable column. The application writes to both columns and reads fromdisplay_name, falling back tofullnamewhen the new column is empty. Old pods ignore the new column and keep working. - Migrate (release N+1 or N+2): Backfill
display_namein batches. This can run as a separate Job so a long backfill never blocks a deploy. - Contract (release N+3): Once no running or rollback-eligible version reads
fullname, drop it.
Destructive operations such as drops, renames, and type narrowing always ship at least one release after the code stops depending on the old shape.
This costs extra releases, and it saves engineering time. A broken rollout becomes an ordinary rollback rather than an incident that needs a hand-written schema fix while error rates climb.
How do you make migrations idempotent and safe to retry?
Every mainstream migration tool records applied versions in a table inside the target database, and most take a lock before running so that two migrators cannot apply the same version at once. Use the tool's mechanism instead of inventing your own. The version table makes a clean re-run a no-op, and the lock covers the case where a retried pod starts while the previous one is still connected.
The version table only helps if a migration either completes or leaves no trace. Kubernetes will happily re-run a half-finished migration. A Job with a non-zero backoffLimit creates a new pod after a failure. With restartPolicy: OnFailure, the kubelet restarts the container in place. A node drain or OOM kill can stop the process between two statements. In each case the tool sees the version as unapplied and starts again from the top.
On PostgreSQL, wrapping each migration in a transaction rolls back the partial work, so a retry starts clean. Some statements cannot run inside a transaction, such as CREATE INDEX CONCURRENTLY. On MySQL, DDL statements commit implicitly. In both cases a failure can leave a column or index in place while the version row is never written. The retry then fails on "already exists" and keeps failing until someone intervenes by hand.
Guard every non-transactional step so it can run twice safely, using IF NOT EXISTS / IF EXISTS where the database supports them. Keep each migration to one logical change, and isolate non-transactional statements in their own migration file. That way a failure leaves at most one step to reason about.
What changes when you deploy with GitOps tools instead of the Helm CLI?
That way a failure leaves at most one change half-applied. The deployment tool still has to run that migration at the right moment, and this is where Helm CLI behavior stops being a safe assumption.
Helm hooks are a feature of the Helm release lifecycle, not of the chart's manifests. Many GitOps controllers do not run that lifecycle. They render the chart to plain YAML, much like helm template, and reconcile the output with their own engine. A hook annotation then means whatever the tool decides it means. Some tools translate pre-install and pre-upgrade into their own pre-sync phase. Others drop the annotation and apply the migration Job alongside the Deployment, which removes the ordering guarantee the hook was supposed to provide.
Two further problems are easy to miss:
- Delete policies may be ignored. Completed Jobs can stay in the cluster. A Job's pod template is immutable, so the next sync can fail when it tries to update the old Job with a new image.
- Install versus upgrade may not exist. Without a Helm release record, the tool cannot tell a first install from an upgrade. Hooks keyed to only one of those events can fire on every sync or never fire.
Before relying on hooks, render your chart through the actual tool and check how each annotation is handled. The more portable approach is a plain Job, ordered with the tool's native sequencing so that it must succeed before the Deployment is applied. Give the Job a unique name per release, for example derived from the image tag, and set ttlSecondsAfterFinished so that old Jobs get cleaned up.
Who should use each approach?
A dedicated, pipeline-gated Job suits most teams running a multi-replica service against a shared relational database. It works with the Helm CLI, plain manifests and GitOps controllers alike. It is the right fit if you want one run per release, a single object to inspect when something fails, and schema-altering credentials kept out of application pods. Teams with an existing CI/CD system that can block on kubectl wait, or a GitOps tool with native sequencing, can adopt it with little extra work.
A Helm hook suits teams that deploy with the helm CLI directly, from a workstation or a pipeline, and want the ordering to travel with the chart rather than live in pipeline scripts. It fits best when the people operating the release understand hook semantics: hooks are not tracked like normal release resources, delete policies control what survives for debugging, and helm rollback leaves the schema as it is.
Init containers suit single-replica services, local and development clusters, and short-lived preview environments where each environment has its own throwaway database. In those settings there is only one pod to run the migration, concurrency is not a concern, and the simplicity is worth more than strict sequencing.
Whatever the trigger, the team needs migrations written in expand/contract style and a migration tool with a version table and a lock. Without those, none of the three mechanisms makes a rolling deploy safe.
When should each approach be avoided?
Avoid init containers for any Deployment with more than one replica, or one that autoscales. Avoid them for migrations that take longer than a few seconds, because the pod sits in Init and the rollout can hit progressDeadlineSeconds. Avoid them when your security model requires that application pods never hold DDL-capable credentials. Also avoid them when the migration tool's locking has not been verified to hold until the work commits.
Avoid Helm hooks when a GitOps controller renders your chart instead of the Helm CLI running it, unless you have confirmed how that tool treats each hook annotation and delete policy. Avoid them when a migration can outlast Helm's --timeout, since Helm will report failure while the Job keeps changing the schema. Avoid them when your team relies on helm rollback as a complete undo.
Avoid a pipeline-gated Job as the only safeguard when people or tools can apply the Deployment outside the pipeline, because nothing in the cluster enforces the ordering. In that case, move the ordering into your GitOps tool's sequencing or restrict who can apply the Deployment.
Avoid running the migration as part of the deploy at all in a few cases:
- Long data backfills. Large batch updates can run for hours. Ship them as a separate Job, decoupled from any release, as described in the expand/contract section.
- Schemas owned elsewhere. If another team or service owns the database schema, the application's deploy should not alter it.
- Changes that cannot be made backward-compatible. Some changes need a maintenance window or a coordinated cutover. Plan those explicitly instead of letting a rolling update apply them automatically.
Which migration approach should your team use?
| Criterion | Init container | Standalone Job (CI/CD-sequenced) | Helm hook Job |
|---|---|---|---|
| Concurrency safety | Every replica races; needs a DB lock | One pod per run if parallelism: 1 |
One pod per run if parallelism: 1 |
| Sequencing | Per pod, so old and new pods overlap | Explicit pipeline gate | Implicit, tied to helm install/upgrade |
| Observability | Mixed into app pod restarts | Own logs, status, exit code | Own logs, deleted by default hook policy |
| Credential exposure | DDL rights in every app pod | Isolated ServiceAccount and Secret | Isolated ServiceAccount and Secret |
| Rollback | Old image meets new schema | Schema stays forward; app rolls back | helm rollback skips upgrade hooks; schema stays |
| Ops and dev cost | Lowest upfront, highest incident cost | Pipeline work once | Low if already on Helm |
| Suits | Single replica, dev clusters | Any size, Helm or GitOps | Helm CLI users who accept hook semantics |
The deciding factor is whether the schema change has its own lifecycle, separate from any pod.
Init containers couple the schema change to pod scheduling. Each autoscaling event, node drain or crash loop runs the migrator again. Every application pod also carries credentials that can alter tables.
Helm hooks move the migration into a Job. That Job still lives outside the release's tracked resources, so a failed hook can leave a half-applied upgrade and no Job left to inspect. GitOps controllers also interpret hooks in their own way.
Recommendation: run migrations by default as a decoupled Job with a unique name per release, gated explicitly by your pipeline or sync waves. Use a Helm hook only as the trigger for that same Job, and only if your team understands that hooks are not revision-managed and that rollback will not undo the schema change.
How OneEnv can help
Disclosure: OneEnv is the product behind this blog.
OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.
Applied to kubernetes database migrations, this is where it helps:
Saves time
- Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
- One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
- Reviewers see the exact diff and the list of affected services up front, so approval is a decision rather than an investigation.
Saves cost
- Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
- Less duplicated configuration to maintain: one shared definition inherited by many services.
- Per-service approve/reject means a bad shared value can be refused where it would break something, before it is live.
- OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.
These are qualitative benefits, not measured results — how much you save depends on how many services share the value.
Want to try it? Start at oneenv.io.