Helm vs Kustomize for Backend Services: Templating Trade-offs

How does it work: Helm vs Kustomize

Helm is a template engine with a release manager attached. A chart holds Go templates; helm install or helm upgrade merges values files and --set overrides, renders the templates into Kubernetes manifests, applies them, and records the result as a named release. Each upgrade creates a new revision stored in the cluster (by default as Secrets in the release namespace), which is what makes helm rollback possible.

Kustomize has no template language. It starts from complete, valid YAML in a base directory, and each overlay declares patches and transformers (name prefixes, labels, image tags, replica counts) to apply on top. kustomize build or kubectl apply -k produces plain manifests and keeps no release state. History lives in Git, and a GitOps controller or kubectl apply reconciles the result.

In short, Helm works by substitution and logic over text, then tracks what it applied. Kustomize works by transforming parsed objects and leaves tracking to Git and whatever applies the output.

These models fail in different ways. Helm renders text, so a wrong indent width on toYaml .Values.resources | nindent 10 can produce YAML that does not parse. It can also produce YAML that parses but has the wrong structure. Kustomize works on parsed objects, so its errors are about targeting. A patch that names a missing resource fails the build. A strategic-merge patch with a misspelled container name is worse: it adds a second container instead of raising an error.

Neither tool is strictly more capable. The better choice depends on who consumes the configuration: other teams installing your service with their own values, or your own team promoting one service through known environments.

Copy             HELM                                 KUSTOMIZE

+-------------+  +-------------+    +-------------+  +-------------+
| values.yaml |  | templates/  |    | base/       |  | overlays/   |
| + --set     |  | (Go tmpl)   |    | valid YAML  |  | prod/ ...   |
+------+------+  +------+------+    +------+------+  +------+------+
       |                |                  |                |
       +-------+--------+                  +-------+--------+
               |                                   |
               v                                   v
     +-------------------+              +---------------------+
     | helm template /   |              | kustomize build /   |
     | install / upgrade |              | kubectl apply -k    |
     | (renders TEXT)    |              | (patches OBJECTS)   |
     +---------+---------+              +----------+----------+
               |  fails on: indent,                |  fails on: patch
               |  nindent/toYaml                   |  target mismatch
               v                                   v
     +-------------------+              +---------------------+
     | rendered manifests|              | built manifests     |
     +---------+---------+              +----------+----------+
               |                                   |
               v                                   v
     +-------------------+              +---------------------+
     | release (named)   |              | kubectl apply or    |
     | rev 1, 2, 3 ...   |              | GitOps controller   |
     | Secrets in ns     |              | (no release state)  |
     +-------------------+              +---------------------+
        helm rollback                     history lives in Git

Helm renders values and Go templates as text into manifests tracked as a release with in-cluster revisions. Kustomize patches valid YAML objects from a base and overlays into plain manifests, which kubectl apply or a GitOps controller reconciles from Git.

How do you run one service across dev, staging and prod with each tool?

Take a typical backend API where only three things change per environment: replica count, CPU and memory limits, and the image tag. Everything else (ports, probes, environment variable names, service account) stays the same.

Helm layout

The chart holds Chart.yaml, a templates/ directory with deployment.yaml, and a default values.yaml. Next to it sit values-dev.yaml, values-staging.yaml and values-prod.yaml. The Deployment template reads .Values.replicaCount, .Values.image.tag and .Values.resources, and each environment file overrides only those keys. You deploy with helm upgrade --install and pass -f values-prod.yaml. Helm merges that file over the defaults. Each environment file stays a few lines long because the shape of the Deployment lives in one template.

Kustomize layout

A base/ directory holds a complete, valid deployment.yaml and a kustomization.yaml that lists it. Under overlays/, each of dev/, staging/ and prod/ has its own kustomization.yaml. That file references the base, uses the images field to set newTag, and uses the replicas field to set the count. For resource limits, a small strategic merge patch under patches matches the Deployment and container by name and sets the new values. You render with kustomize build overlays/prod or apply with kubectl apply -k.

In both layouts, each environment differs from the shared definition by roughly ten lines. That is a useful check. If one environment file grows much larger than the others, configuration is drifting.

Keep Kustomize overlays shallow. A chain like base → region → environment → cluster means a single replica count can be set or overridden at four levels. Tracing where a final value came from then requires reading every layer in order, so prefer one overlay level per deployable target.

CopyreplicaCount: 4
image:
  tag: "1.8.2"
resources:
  limits:
    cpu: "1"
    memory: 1Gi
---
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - ../../base            # plain, valid Deployment lives here
images:
  - name: registry.example.com/orders-api   # must match image name in base
    newTag: "1.8.2"
replicas:
  - name: orders-api      # Deployment metadata.name
    count: 4
patches:
  - path: resources-patch.yaml   # strategic merge patch for limits
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: orders-api        # matches the base Deployment
spec:
  template:
    spec:
      containers:
        - name: api       # merged with the base container by name
          resources:
            limits:
              cpu: "1"
              memory: 1Gi

The same prod change (4 replicas, image tag 1.8.2, 1 CPU and 1Gi memory limits) as a Helm values file and as a Kustomize overlay with the images and replicas fields plus a strategic merge patch.

How should you review and validate rendered Kubernetes manifests in CI?

Validation in CI starts from a structural difference. A Kustomize base is a directory of ordinary Kubernetes manifests, so kubectl, schema validators and editors can parse it directly. A Helm template is not valid YAML until it has been rendered. A file full of {{ }} directives tells a validator nothing about the Deployment it will produce. For both tools, validate the rendered output, not the source.

Render, validate, then diff

The pipeline has three steps:

1. Render. Run helm template with the same values files each environment uses. For Kustomize, run kubectl kustomize or kustomize build against each overlay. Write the output to files.
2. Validate. Check the rendered files against Kubernetes schemas, including CRD schemas if you use them. kubectl apply --dry-run=server against a non-production cluster also catches admission and API-version problems.
3. Diff. Render the base branch and the pull request the same way, then post the difference on the PR.

For Helm, this step is mandatory. A misplaced nindent or an extra space inside a conditional block can produce YAML that parses fine but is wrong. An env entry might land under the wrong container, or a label might silently disappear. These mistakes surface only at render or apply time, and helm lint does not catch all of them.

The rendered diff is also where most of the time savings come from. A reviewer looking at a values or template change has to work out its effect across every environment in their head. A rendered diff shows the exact objects that will change, such as "prod replicas 3 → 6" or "staging loses a volume mount." Reviews get shorter, and errors are caught before apply, not during a rollout.

Copy#!/usr/bin/env bash
set -euo pipefail

ENVS=(staging prod)
BASE_REF=origin/main
OUT=$(mktemp -d)

render() {
  local src=$1 dest=$2
  mkdir -p "$dest"
  for env in "${ENVS[@]}"; do
    # Same values files the real deploy uses, so the output matches production intent.
    helm template api "$src/charts/api" \
      -f "$src/charts/api/values.yaml" \
      -f "$src/charts/api/values-$env.yaml" \
      --namespace "$env" > "$dest/helm-api-$env.yaml"
    kubectl kustomize "$src/kustomize/worker/overlays/$env" > "$dest/kustomize-worker-$env.yaml"
  done
}

render . "$OUT/pr"
git worktree add --detach "$OUT/base-src" "$BASE_REF"
render "$OUT/base-src" "$OUT/base"

for f in "$OUT"/pr/*.yaml; do
  kubectl apply --dry-run=server -f "$f" > /dev/null
done

diff -ru "$OUT/base" "$OUT/pr" > rendered.diff || true
echo "Rendered diff written to rendered.diff (post it as a PR comment)."

git worktree remove --force "$OUT/base-src"

A CI script that renders a Helm chart and Kustomize overlays for both the base branch and the pull request, validates the PR output with a server-side dry run, and writes a unified diff of the rendered manifests for reviewers.

How can Helm or Kustomize save engineering time?

Either tool saves time in the same basic way: it removes copy-paste across environments and makes the per-environment difference small and explicit. Where the time goes depends on which work each tool takes off your hands.

• Smaller, safer changes. With one shared definition and short per-environment files, a change to a probe or a port is made once instead of once per environment. That removes the most common drift bug, where staging gets a fix and prod does not.
• Faster reviews. Combined with the rendered diff from CI, reviewers check the actual object changes rather than mentally evaluating templates or overlay stacks. Kustomize patches are already plain YAML, so even the source diff is readable without rendering.
• Faster recovery. Helm's helm rollback returns a release to a previous revision without hunting for the old commit. With Kustomize and GitOps, git revert plus a re-sync does the same job and leaves an audit trail in the repository.
• Reuse instead of authoring. Consuming a maintained upstream chart for an ingress controller or cert-manager avoids writing and maintaining those manifests yourself. A Kustomize patch on top handles the few fields the chart does not expose.
• Less onboarding for in-house services. An engineer who can read a Deployment can change a Kustomize overlay without learning Go templating, values precedence or whitespace control.

The savings reverse when the tool doesn't fit. A chart for a single team-owned service adds template tests and a values interface nobody else uses. Kustomize for a service with many optional components produces near-duplicate overlays that must be kept in sync by hand.

As an illustrative estimate with assumed inputs: suppose a team runs one service in three environments and makes an assumed four configuration changes a week that apply to all environments. With copied raw manifests, each change touches three files and needs three separate checks. With a shared base or template, it touches one file plus a rendered diff that shows all three results at once. The saving per change is small, but it compounds with the number of environments and services, and the bigger benefit is avoiding the incident where one copy was missed.

When is Helm chart templating worth the complexity?

Helm earns its complexity when the manifests need logic rather than substitution. Typical cases are an optional HorizontalPodAutoscaler, Ingress, or ServiceMonitor gated on a single flag; a range over a list of workers or cron jobs that would otherwise be near-identical copies; and named helper templates for labels and selectors shared across charts. The strongest case is publishing a versioned, parameterized chart that other teams or third parties install with only a values file. They depend on a stable interface, not your YAML.

The release lifecycle is the other payoff. helm install, helm upgrade, and helm rollback operate on a release whose revision history is stored in the cluster, so you can see what was deployed and return to an earlier revision. Hooks let a pre-upgrade Job run database migrations before new pods start.

When to avoid it

• values.yaml becomes a second API. Every exposed knob is something you must document, version, and keep backward compatible.
• Templates get hard to read. Nested conditionals and nindent helpers turn small changes into whitespace debugging.
• Release state can drift from git. The release history in the cluster, not the repository, records what was applied.
• Upgrades can get stuck. An interrupted upgrade can leave the release in a pending or failed state that blocks further upgrades until you intervene.
• Hooks are less portable. Tools that render the chart and apply the output themselves may skip hooks or handle them differently.
• GitOps controllers overlap. A GitOps controller already handles sync, history, and rollback, so Helm's lifecycle features duplicate it.

Copy{{- /* templates/backend.yaml: helpers like orders.fullname live in _helpers.tpl */}}
{{- range .Values.workers }}
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: {{ include "orders.fullname" $ }}-{{ .name }}
  labels:
    {{- include "orders.labels" $ | nindent 4 }}
spec:
  replicas: {{ .replicas | default 1 }}
  selector:
    matchLabels:
      app.kubernetes.io/instance: {{ $.Release.Name }}
      app.kubernetes.io/component: {{ .name }}
  template:
    metadata:
      labels:
        app.kubernetes.io/instance: {{ $.Release.Name }}
        app.kubernetes.io/component: {{ .name }}
    spec:
      containers:
        - name: {{ .name }}
          image: "{{ $.Values.image.repository }}:{{ $.Values.image.tag }}"
          args: {{ toJson .args }}
{{- end }}
{{- if .Values.autoscaling.enabled }}
---
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: {{ include "orders.fullname" . }}-api
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: {{ include "orders.fullname" . }}-api
  minReplicas: {{ .Values.autoscaling.minReplicas }}
  maxReplicas: {{ .Values.autoscaling.maxReplicas }}
{{- end }}

A Helm template that loops over a list of workers to emit one Deployment each, uses shared named helpers for names and labels, and renders an HPA only when autoscaling is enabled.

When are Kustomize overlays enough, and where do they fall short?

For a service your team owns end to end, overlays on plain manifests cover most real differences between environments. Image tag bumps, replica counts and resource requests map directly onto Kustomize's built-in images and replicas fields or onto a short patch. Common labels and annotations are applied across every resource in one place. ConfigMap and Secret generators append a content hash to the object name and rewrite references to it, so a config change produces a new name, alters the pod template and triggers a rollout. Without the hash, pods keep running with stale config. One-off differences, such as a debug sidecar in dev or a stricter PodDisruptionBudget in prod, fit a strategic merge patch or a JSON 6902 patch. The base stays valid YAML that anyone can read.

Kustomize has no release concept. It only renders manifests. Something else applies them, either kubectl apply or a GitOps controller, and rollback means git revert followed by a re-sync. For most teams that is simpler than tracking a separate release history.

Where overlays fall short

• No logic. Kustomize has no conditionals, loops or computed values. Optional resources and per-tenant variations end up as near-duplicate overlays, a sprawl of components or a wrapper script that generates the overlays.
• Patch target mismatches. A patch whose selector matches nothing can apply to zero resources without any error. A renamed resource or a wrong kind can mean an overlay does less than you think.
• Leftover generated objects. Each config change leaves behind the previous hash-suffixed ConfigMap unless your tooling prunes it. Consumers outside the kustomization that reference a fixed name won't see the new one. You can turn off the suffix with disableNameSuffixHash, but then config changes no longer trigger a rollout.

Can you combine Helm and Kustomize?

Yes, and many teams do. The usual split is by who owns the YAML. Third-party components such as ingress controllers, cert-manager or a database operator arrive as Helm charts maintained upstream, so you consume them as charts. In-house backend services stay as plain Kubernetes manifests with Kustomize overlays, where every field is visible in review.

When an upstream chart doesn't expose a value you need, like a toleration, an extra label or a sidecar, you have two options. You can run helm template, commit or pipe the output, and apply Kustomize patches on top. Or you can let Kustomize inflate the chart through its helmCharts field, which requires kustomize build --enable-helm and a Helm binary on the build host. Inflation keeps everything in one build, but Kustomize does not run Helm's release lifecycle. Hooks, rollback history and lookup-based logic either don't behave as they would under helm install or are absent entirely. Some teams prefer to vendor the rendered or pulled chart into the repo so builds don't depend on a chart repository being reachable.

The combination has real costs:

• An extra build step, which CI and local tooling must reproduce exactly.
• Two mental models: Go templates and values on one side, strategic merge and JSON patches on the other.
• Harder debugging, because the YAML that reaches the cluster matches neither the chart source nor your patch files.
• Fragile patches. When an upstream release renames a container, restructures a values key or moves a field, patches can stop applying or silently target nothing. Diff the rendered output on every chart upgrade.

For a single small service with no environment variation, plain kubectl apply on raw manifests is still a legitimate choice.

What are the alternatives to Helm and Kustomize?

Helm and Kustomize are the two most common ways to manage Kubernetes manifests, but they are not the only ones. The main alternatives trade along the same axis: how much logic you allow in configuration versus how plain the output stays.

• Plain manifests with kubectl apply. One directory of YAML per environment, applied directly. There is nothing to learn beyond Kubernetes itself, and the files are exactly what reaches the cluster. It works for one service with little variation. It stops working once several environments must be kept in sync by hand, because every shared change has to be copied.
• Text substitution with shell tools. Running manifests through envsubst or sed to fill in an image tag or namespace before applying them. It is quick to set up and needs no extra tooling, but it has the same text-level failure modes as Helm templates with none of the structure: no defaults, no validation of missing variables unless you add it, and no release tracking.
• Generating manifests from a general-purpose programming language. Code builds Kubernetes objects using a client library or plain data structures, then serializes them to YAML. You get real loops, functions, types and unit tests. The cost is that configuration becomes a program: reviewers must read code to know what will be deployed, and the rendered-diff step in CI becomes essential rather than optional.
• Purpose-built configuration languages. Some languages are designed specifically for generating structured data such as JSON or YAML, with features like inheritance, functions or type constraints. They sit between Kustomize and a general-purpose language in expressiveness, but each team member has to learn another language and its tooling.
• Kubernetes operators. For software with a complex lifecycle, such as databases that need ordered upgrades, backups or failover, a controller that reconciles a custom resource replaces static manifests altogether. This is usually something you consume rather than write for an ordinary stateless backend service, because building and maintaining an operator costs far more than a chart or overlay.

For a typical stateless backend service, these alternatives rarely beat the two main options. Plain manifests are fine at very small scale. Generated configuration makes sense when a platform team needs to produce many similar services from one definition and is willing to own that generator as software.

Should your backend team choose Helm or Kustomize?

Start with who consumes the config. If the only consumer is the team that owns the service, a chart's values.yaml becomes an API with one user. You still pay to design it, document it and keep it backward compatible. Kustomize avoids that cost because overlays patch real manifests and need no interface.

Development cost works in opposite directions. Duplicated overlay YAML is cheap to read but drifts when someone updates one environment and forgets the others. Template logic removes the duplication, but each conditional is code that needs rendering tests across value combinations. Onboarding follows the same pattern. Any engineer who can read a Deployment can follow a patch. Charts also require Go templating, whitespace control and the precedence rules for values.

Checklist:

• Team-owned backend service deployed from git: use Kustomize.
• Service shipped to other teams or customers as a versioned artifact: use Helm, and treat values.yaml as a public contract.
• Third-party software you install: consume the upstream chart, then use Kustomize to adjust its rendered output rather than forking the chart.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to Helm vs Kustomize, this is where it helps:

Saves time

• Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
• One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
• Reviewers see the exact diff and the list of affected services up front, so approval is a decision rather than an investigation.

Saves cost

• Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
• Less duplicated configuration to maintain: one shared definition inherited by many services.
• Per-service approve/reject means a bad shared value can be refused where it would break something, before it is live.
• OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.