What does helm rollback actually restore, and how does it work?

A Helm rollback re-applies the Kubernetes manifests rendered for an earlier revision. It does not restore system state. Helm has no record of what your application wrote to a database, a PersistentVolume or an external queue, so none of that changes. Rollback also leaves alone anything outside the release's stored manifest, such as CRDs installed from a chart's crds/ directory and resources created by operators at runtime.

By default, Helm stores each revision as a Secret in the release namespace, named sh.helm.release.v1.<release>.v<N>. That Secret contains the rendered manifest, the values used and the chart metadata. When you roll back to revision 2, Helm reads revision 2's Secret, patches the live objects to match its manifest, and writes the result as a new revision. History is never deleted or rewound. In helm history, the earlier revisions show as superseded, and the newest one shows deployed with a description like "Rollback to 2".

helm history                 release Secrets (ns: app)       live cluster objects
REV  STATUS      DESCRIPTION
1    superseded  Install      sh.helm.release.v1.web.v1
2    superseded  Upgrade      sh.helm.release.v1.web.v2 ─┐
3    superseded  Upgrade      sh.helm.release.v1.web.v3  │ manifest copied
4    deployed    Rollback     sh.helm.release.v1.web.v4 <┘ ──► Deployment, Service,
                 to 2                                          ConfigMap (match rev 2)

Every rollback adds a revision, so history grows faster during an incident. The --history-max setting on helm upgrade prunes the oldest Secrets once the cap is reached. If the last known-good revision falls off the end, there is nothing left to roll back to. Check how many revisions you keep before you need them.

 Release Secrets (ns: app)                        Live cluster objects
 ┌────────────────────────────────┐
 │ sh.helm.release.v1.web.v1      │
 │  REV 1  superseded  Install    │
 ├────────────────────────────────┤
 │ sh.helm.release.v1.web.v2      │──┐
 │  REV 2  superseded  Upgrade    │  │ manifest
 ├────────────────────────────────┤  │ copied
 │ sh.helm.release.v1.web.v3      │  │
 │  REV 3  superseded  Upgrade    │  │
 ├────────────────────────────────┤  │
 │ sh.helm.release.v1.web.v4      │◄─┘            ┌──────────────────────┐
 │  REV 4  deployed    Rollback   │               │ Deployment           │
 │                     to 2       │─── patch ────►│ Service              │
 └────────────────────────────────┘               │ ConfigMap            │
   ▲                                              │ (= rev 2 manifest)   │
   │                                              └──────────────────────┘
   └─ --history-max prunes oldest Secrets first;
      if v2 is pruned, there is nothing to roll back to

 ┌─ NOT restored by rollback ───────────────────────────────────┐
 │ PV data, database rows, external queues/APIs                 │
 │ CRDs from chart crds/, operator-created runtime resources    │
 └──────────────────────────────────────────────────────────────┘

Rolling back to revision 2 copies its stored manifest into a new revision 4 Secret and patches the live objects to match, while revisions 1-3 stay in history until --history-max prunes the oldest ones.

How do you triage a failed Helm release before rolling back?

Run this before you touch anything. It is meant for the backend or platform engineer on call. The first goal is to find out what failed, not to restore service in the next thirty seconds.

Start with helm status <release> and helm history <release>. History shows each revision's state. deployed is the revision Helm considers live. failed means the upgrade was applied, or partly applied, and then errored. pending-upgrade or pending-rollback means a Helm process stopped before it recorded a result. A pending state changes the procedure, so note it now.

Next, look at the cluster:

  • kubectl get events --sort-by=.metadata.creationTimestamp in the release namespace
  • kubectl get pods to see pod status
  • kubectl logs on failing pods, adding --previous for containers in a crash loop

Capture all of this to files immediately. A rollback scales down the new ReplicaSet. An --atomic upgrade does the same automatically. Hook pods can be deleted by their delete policy, and events expire. Once that happens, the logs that explain the failure are gone.

Map the failure to a first move

Failure type Typical symptoms Rollback as first move?
Application failure CrashLoopBackOff, failing readiness probes, errors in logs Yes, if the previous revision is known-good and no schema change shipped
Chart or templating error Upgrade fails at render or validation, and often no new revision is recorded No. Nothing changed, so fix the chart.
Hook failure Failed Job in events, release marked failed Only after checking what the hook already did, such as a migration
--wait timeout Release failed while pods are still pulling or starting Maybe not. Confirm the workload is actually unhealthy first.
#!/usr/bin/env bash
# Snapshot evidence for a failed Helm release BEFORE any rollback.
# A rollback or --atomic cleanup can remove the pods, hook Jobs and logs you need.
set -uo pipefail

RELEASE="${1:?usage: $0 <release> <namespace>}"
NS="${2:?usage: $0 <release> <namespace>}"
OUT="triage-${RELEASE}-$(date -u +%Y%m%dT%H%M%SZ)"
mkdir -p "$OUT"

# Helm's view: current status and per-revision state (deployed, failed, pending-*)
helm status "$RELEASE" -n "$NS" > "$OUT/helm-status.txt" 2>&1
helm history "$RELEASE" -n "$NS" --max 10 > "$OUT/helm-history.txt" 2>&1

# What the latest revision applied, for diffing against the last good one later
helm get values "$RELEASE" -n "$NS" --all > "$OUT/values.yaml" 2>&1
helm get manifest "$RELEASE" -n "$NS" > "$OUT/manifest.yaml" 2>&1

# Cluster's view: events expire, so save them first
kubectl get events -n "$NS" --sort-by=.metadata.creationTimestamp > "$OUT/events.txt" 2>&1
kubectl get pods,jobs,replicasets -n "$NS" -o wide > "$OUT/workloads.txt" 2>&1
kubectl describe pods -n "$NS" > "$OUT/pods-describe.txt" 2>&1

# Logs from every pod in the namespace (includes hook pods that a delete policy may remove).
# --previous captures the last crashed container in a CrashLoopBackOff.
for pod in $(kubectl get pods -n "$NS" -o name); do
  name="${pod#pod/}"
  kubectl logs "$pod" -n "$NS" --all-containers > "$OUT/logs-$name.txt" 2>&1
  kubectl logs "$pod" -n "$NS" --all-containers --previous \
    > "$OUT/logs-$name-previous.txt" 2>/dev/null || true
done

echo "Evidence saved to $OUT. Read helm-history.txt for pending-* states before rolling back."

A triage script that saves Helm revision state, rendered values and manifests, namespace events, pod descriptions, and current and previous container logs to a timestamped directory before a rollback can remove them.

How do you choose the right revision and run helm rollback safely?

Start with helm history <release> and read it top to bottom. Running helm rollback <release> with no revision targets the previous revision, which is the wrong choice after stacked bad upgrades. If someone retried a broken upgrade twice during the incident, "previous" is just another copy of the failure. Pick the last revision whose status is deployed or superseded and that you know served traffic correctly.

Confirm what you are restoring

Before running anything, inspect the target:

  • helm get values <release> --revision N shows the user-supplied values for that revision. Add --all to see the computed values after chart defaults are merged.
  • helm get manifest <release> --revision N shows the exact rendered Kubernetes objects Helm will apply.

These commands matter because the old values return exactly as written. That includes Secret or ConfigMap names that may have been rotated or deleted since, replica counts sized for last month's load, resource limits that someone has since raised, and image tags that may have been pruned from the registry. Diffing this manifest against the current one shows you the full change before it reaches the cluster.

Run it with readiness checks

Run helm rollback <release> N --wait --timeout 10m, adjusting the timeout to how long your pods normally take to become ready. Without --wait, Helm reports success as soon as the API server accepts the objects, not when the workload is healthy. The rollback is recorded as a new revision number, so the history stays auditable.

Rollback also overwrites manual kubectl hotfixes made during the incident on any field the chart manages, such as edited env vars, patched limits or scaled replicas. Before you roll back, either write those changes down or move them into values.

#!/usr/bin/env bash
set -euo pipefail

RELEASE=checkout-api
NS=payments
TARGET=7   # last revision known to have served traffic correctly

# 1. Read the full history; do not trust "previous" after stacked bad upgrades
helm history "$RELEASE" -n "$NS"

# 2. Inspect what the target revision will restore
helm get values "$RELEASE" -n "$NS" --revision "$TARGET"        # user-supplied values
helm get values "$RELEASE" -n "$NS" --revision "$TARGET" --all  # merged with chart defaults

# 3. Diff the target manifest against what is live now
#    Look for stale Secret/ConfigMap names, old replica counts, limits, pruned image tags
diff <(helm get manifest "$RELEASE" -n "$NS" --revision "$TARGET") \
     <(helm get manifest "$RELEASE" -n "$NS") || true

# 4. Save any manual kubectl hotfixes before rollback overwrites them
kubectl get deployment "$RELEASE" -n "$NS" -o yaml > "hotfix-snapshot-$(date +%Y%m%d%H%M).yaml"

# 5. Roll back to an explicit revision and wait for readiness
#    Set --timeout to how long your pods normally take to become ready
helm rollback "$RELEASE" "$TARGET" -n "$NS" --wait --timeout 10m

# 6. Confirm the rollback was recorded as a new revision
helm history "$RELEASE" -n "$NS" --max 3

A pre-rollback routine that picks an explicit revision, inspects and diffs its values and manifest, snapshots live hotfixes, then rolls back with readiness checks.

How do you recover a release stuck in pending-upgrade or pending-rollback?

When a CI job is killed mid-upgrade, Helm never writes a final status. The newest revision stays pending-upgrade, and every later command fails with "another operation (install/upgrade/rollback) is in progress". Helm treats the pending state as a lock, so retrying the same upgrade won't clear it.

The usual fix is to run helm history, find the last revision marked deployed, and roll back to that number explicitly. The rollback creates a new revision with a final status, which supersedes the stuck one. It also re-applies a manifest you already know worked.

If the rollback itself hangs in pending-rollback, the last resort is editing Helm's storage directly. In Helm 3, each revision is a Secret labelled owner=helm and name=<release>. Export all of the release's Secrets with kubectl get secret -l ... -o yaml before deleting or relabelling the pending one. Changing these Secrets alters only Helm's record, not the live objects. Afterwards, Helm's view of the release can disagree with what is actually running in the cluster.

Hooks need checking before any retry:

  • Which hooks run. A rollback runs pre-rollback and post-rollback hooks, not the upgrade hooks.
  • Leftover hook Jobs. A hook Job that failed under a hook-succeeded-only delete policy stays in the cluster. The next attempt then fails with "already exists". Delete the Job after reading its logs.
  • When --no-hooks fits. Use it when a broken hook is the only thing blocking recovery.
  • When --no-hooks is dangerous. Avoid it when the hook does stateful work the rollback depends on, such as reverting a schema migration. Skipping that hook leaves old code running against a new schema.
# Placeholder release and namespace; adjust for your environment.
RELEASE=myapp
NS=prod

# 1. Confirm the stuck state and find the last revision whose status is 'deployed'.
helm status "$RELEASE" -n "$NS"
helm history "$RELEASE" -n "$NS" --max 10

# 2. Clear any failed hook Job left behind, but read its logs first.
kubectl get jobs -n "$NS"
kubectl logs job/myapp-db-migrate -n "$NS"
kubectl delete job myapp-db-migrate -n "$NS"

# 3. Usual fix: roll back explicitly to the known-good revision (here, 41).
#    This runs pre-rollback/post-rollback hooks, not upgrade hooks.
helm rollback "$RELEASE" 41 -n "$NS" --wait --timeout 10m

# Use --no-hooks only if a broken hook is the sole blocker and it does no
# stateful work, such as reverting a schema migration, that the rollback needs:
# helm rollback "$RELEASE" 41 -n "$NS" --no-hooks --wait --timeout 10m

# 4. Last resort, only if the rollback itself sticks in pending-rollback.
#    Back up every revision Secret for this release before touching any of them.
kubectl get secret -n "$NS" -l owner=helm,name="$RELEASE" -o yaml \
  > "helm-${RELEASE}-secrets-backup.yaml"

# List revisions with their stored status to identify the pending one.
kubectl get secret -n "$NS" -l owner=helm,name="$RELEASE" \
  -L version,status

# Delete only the pending revision's Secret (here, v43). This changes Helm's
# record only. Live objects are untouched and may no longer match Helm's view.
kubectl delete secret "sh.helm.release.v1.${RELEASE}.v43" -n "$NS"

# 5. Verify that Helm's view and the cluster agree again.
helm history "$RELEASE" -n "$NS" --max 5
kubectl get pods -n "$NS"

Recovery sequence for a release stuck in a pending state: inspect the history, clear leftover hook Jobs, roll back explicitly to the last deployed revision, and back up the release Secrets with kubectl before deleting a pending revision as a last resort.

When should you avoid helm rollback and fix forward instead?

A rollback re-applies the manifests stored with an earlier revision. Helm doesn't track anything outside those manifests, so state that changed elsewhere stays changed. Before you roll back, list every change the failed release made outside the Deployment spec.

Resource type What helm rollback does to it Reversal or forward-fix plan
CRDs from the chart's crds/ directory Nothing. Helm installs these on first install only and never upgrades or deletes them. Apply CRD changes separately. If the old controller can't read the new schema, fix forward.
PVCs and their data Restores the PVC object spec at most. StatefulSet volumeClaimTemplates PVCs aren't Helm-managed at all. Data on disk is untouched. Restore from a snapshot or backup taken before the upgrade.
Schema migrations in hooks or init jobs Doesn't reverse them. A pre-upgrade migration stays applied. Ship a tested down-migration as a pre-rollback hook, or fix forward.
Queues, DNS records, operator-created cloud resources Not tracked. Operators may reconcile toward the old spec, but deleted or mutated external state isn't restored. Reverse each one manually with a runbook, or leave it and fix forward.

Migrations are the most dangerous row. Suppose the failed release dropped a column, renamed a table or rewrote data in a new format. The previous application code will then run against a schema it was never written for. That can mean write errors or corrupted rows, which is often worse than the original failure because it turns an outage into data damage.

In that situation, keep the new schema and fix forward. Ship a patched chart with a corrected image, a feature flag turned off or a config fix. Roll back only when every stateful change in the table above is either reversible or compatible with the old code.

Is helm upgrade --atomic a replacement for a rollback checklist?

A pipeline that rolls back on its own sounds like it makes the checklist unnecessary. It covers some failures, but it brings trade-offs of its own.

helm upgrade --atomic implies waiting for resources to become ready. If the upgrade fails or the timeout expires, Helm rolls back automatically. --cleanup-on-fail deletes resources that the failed upgrade created. Flag names and defaults can change between major versions, so confirm them with helm upgrade --help for the Helm version you run.

There are two costs:

  • Lost evidence. The rollback removes the failing pods before anyone looks at them. Your pipeline should capture events, pod logs and helm status output before Helm reverts.
  • Duplicated work. If the timeout is too short, Helm can roll back a rollout that was slow but correct. During an incident, that doubles the restarts.
Approach Best for Main risk
Manual rollback Reversible failures that need human judgement Slow response
--atomic upgrade Stateless services in automated pipelines Hides failures; can fight slow rollouts
Fix-forward Irreversible stateful changes Needs a quick, safe patch

Setting --timeout from real rollout data

Illustrative estimate, using these assumed inputs:

  • 6 replicas, replaced one at a time because of the rolling-update settings and the PodDisruptionBudget
  • 90 s to pull the image
  • 45 s for the readiness probe to pass

Each pod takes 135 s, and six sequential replacements take about 810 s. A 30% margin brings that to about 18 minutes. Use timings measured from your own past rollouts in place of these assumptions.

Copy-ready Helm rollback checklist: verify, clean up and save engineering time

Keep this list in the runbook next to the timeout estimate, and fill it in during the incident rather than afterwards.

Before you touch anything

  • Capture helm history, helm status and helm get manifest for the failed revision.
  • Save kubectl describe output, recent events and logs from failing pods. A rollback replaces those pods, and their evidence goes with them.
  • Announce in the incident channel that one person owns the release. Two engineers running upgrades at once is a common cause of stuck releases.

Choosing the revision

  • Pick the last revision whose status was deployed and that actually served traffic.
  • Diff its values and manifest against the failed revision with helm get values --revision and helm get manifest --revision.
  • Confirm that no schema migration or CRD change prevents going back.

Executing the rollback

  • Set --wait and a --timeout based on your measured rollout time.
  • Watch events while the rollback runs. Do not start a second rollback until the first one has finished or failed.

Verifying and cleaning up

  • helm status reports deployed on the new revision number.
  • kubectl rollout status succeeds for every Deployment, StatefulSet and DaemonSet in the chart.
  • Running image tags and ConfigMap/Secret contents match the target revision.
  • Health checks or smoke tests pass against real endpoints.
  • Record the failed revision number and the saved evidence for the postmortem.

Why writing it down saves time

Each step blocks a specific kind of rework. Capturing evidence first means nobody has to reproduce the failure in staging. Checking the diff before rolling back prevents a second rollout that fails for the same reason. Verifying beyond Helm's exit code catches a rollback that reports success while still running stale config. That is the kind of false confidence a rollback-only plan creates.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to helm rollback failed release, this is where it helps:

Saves time

  • Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
  • One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
  • Reviewers see the exact diff and the list of affected services up front, so approval is a decision rather than an investigation.

Saves cost

  • Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
  • Less duplicated configuration to maintain: one shared definition inherited by many services.
  • Per-service approve/reject means a bad shared value can be refused where it would break something, before it is live.
  • OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.