What does each Helm chart testing layer actually catch?

Each layer works on a different artifact, so each catches a different kind of mistake.

helm lint checks the chart: Chart.yaml metadata, file layout, and whether the templates render into valid YAML. Suppose a template has an unclosed {{ if }} block, or calls required on image.repository and the default values leave it empty. Rendering fails, and lint reports it. Lint does not check rendered fields against Kubernetes API schemas, so it passes a container port spelled contianerPort.

kubeconform validates rendered manifests against Kubernetes JSON schemas. With -strict, it rejects properties the schema does not define, so contianerPort fails. This matters because without strict validation the misspelled field can be silently dropped, and the container's port is never declared. kubeconform has no cluster and no registry access, so it accepts image: myapp:1.2.3-typo.

chart-testing (ct) works across a chart repository. ct lint finds charts that changed against the target branch, requires a version bump in each one, and runs linting on them. ct install deploys each changed chart into a real cluster, waits for its resources to become ready, and runs helm test hooks. A nonexistent image tag leaves pods stuck pulling the image. A readiness probe that hits the wrong path keeps pods unready. In both cases the install times out, and only this layer can see that.

Failure helm lint kubeconform (strict) ct lint ct install
Template syntax error ✅ — ✅ ✅
Missing required value ✅ — ✅ ✅
Misspelled field (contianerPort) ❌ ✅ ❌ Depends on API server field validation
Chart changed without version bump ❌ ❌ ✅ ❌
Bad image tag ❌ ❌ ❌ ✅
Failing readiness probe or helm test ❌ ❌ ❌ ✅
#!/usr/bin/env bash
# Run the three Helm chart testing gates in order, cheapest first.
set -euo pipefail

CHART=charts/myapp
CI_VALUES="$CHART/ci/default-values.yaml"

# Gate 1: helm lint
# Checks Chart.yaml and file layout, and renders the templates.
# Catches an unclosed {{ if }} block or an empty `required` value.
helm lint "$CHART" --values "$CI_VALUES"

# Gate 2: kubeconform
# Render offline, then validate every manifest against the Kubernetes JSON schemas.
# -strict rejects properties the schema does not define, such as
# `contianerPort`. It does not check whether an image tag exists.
helm template myapp "$CHART" --values "$CI_VALUES" \
  | kubeconform -strict -summary -kubernetes-version 1.29.0

# Gate 3a: ct lint
# Finds charts changed relative to main, requires a version bump in
# each changed chart, and lints them.
ct lint --target-branch main

# Gate 3b: ct install
# Needs a reachable cluster (for example, an ephemeral one created
# earlier in the job). Installs each changed chart, waits for its
# resources to become ready, and runs its `helm test` hooks.
# A bad image tag or a failing readiness probe makes the install
# time out here and nowhere earlier.
ct install --target-branch main

A CI script that runs helm lint, kubeconform in strict mode, and chart-testing against a chart, from cheapest to most expensive, with comments on which failure each gate catches.

Why do you need all three tools, and who should run them?

The table shows almost no overlap, and that comes from what each tool inspects. helm lint checks the chart: its metadata, its template syntax, and whether it renders. kubeconform checks the rendered manifests against Kubernetes schemas, but knows nothing about your repository. chart-testing checks your git history and a live cluster, which is the only way to find problems that appear at runtime. If you drop one layer, you lose a category of error, not a few redundant checks.

Run the layers in order of increasing cost. Linting and schema validation take seconds and need no cluster. ct install needs a throwaway cluster, image pulls and readiness waits, so it takes minutes. If cheap checks run first, a misspelled field fails the build before anyone provisions a cluster.

As an illustrative estimate, assume a typo that lint would catch in about 10 seconds instead surfaces during a staging deploy. Someone notices about 20 minutes later, reads the events, fixes the chart and redeploys. If that happens a few times a month across a team, the cost adds up to hours, and much of it is spent switching context away from other work.

Who should adopt the full stack:

  • Platform teams with chart monorepos. ct only tests the charts that changed, which keeps CI fast as the repository grows.
  • Anyone publishing charts. Consumers combine values you never tested, so missing version bumps and schema drift break their deployments, not yours.
  • Backend teams shipping internal charts used by several services or environments.

For a single internal chart with one values file, helm lint plus kubeconform is usually enough. Your normal deploy pipeline already acts as the install test.

How do you run helm lint and kubeconform usefully?

Run helm lint --strict once per values file, not once per chart. --strict turns warnings into failures, so lint problems block the build instead of scrolling past in the logs. Pass each file with -f. Keep at least two in the repo, for example under a ci/ directory:

  • A minimal file sets only what the chart requires. It catches templates that assume an optional value exists and fail with nil-pointer errors when it doesn't.
  • A fully-featured file turns on every toggle: ingress, autoscaling, extra volumes, service monitors.

Rendering with defaults alone gives false confidence. Most charts ship optional blocks disabled, so a broken if .Values.ingress.enabled branch is never rendered and lint passes. The bug then surfaces when a team first enables that feature in production.

Lint has a limit. It renders templates and checks that the chart structure and YAML are valid, but it does not compare resources against the Kubernetes API schema. A containerPort typed as a string, a misspelled field, or an apiVersion your cluster no longer serves will all lint cleanly.

kubeconform covers that gap. For each values file, run helm template with the same -f and pipe the output into kubeconform -strict -summary -kubernetes-version <your cluster version> -. The trailing - reads manifests from stdin. -strict rejects fields that aren't in the schema. Pinning the version means validation runs against the API you actually deploy to, not whatever kubeconform defaults to.

Finally, decide how to handle missing schemas instead of leaving it to the defaults. By default, kubeconform reports a resource with no schema as an error. -ignore-missing-schemas silently skips it, which hides problems in every custom resource. The next section covers supplying schemas so you rarely need that flag.

#!/usr/bin/env bash
# Lint and schema-validate a chart against every CI values file.
set -euo pipefail

CHART=./charts/web-api
# Match the cluster you deploy to, not kubeconform's default.
K8S_VERSION=1.29.0

# ci/minimal-values.yaml: only required values.
# ci/full-values.yaml: ingress, autoscaling, extra volumes, service monitors all on.
for values in "$CHART"/ci/*-values.yaml; do
  echo "==> $values"

  # --strict turns lint warnings into a failed build.
  helm lint "$CHART" --strict -f "$values"

  # Lint does not check resources against the Kubernetes API schema,
  # so render the same values and validate the output.
  # pipefail makes a failed helm template stop the script.
  # Missing schemas: no -ignore-missing-schemas flag on purpose.
  # A resource without a schema fails the build until you supply one.
  helm template ci-release "$CHART" -f "$values" \
    | kubeconform \
        -strict \
        -summary \
        -kubernetes-version "$K8S_VERSION" \
        -
done

A CI script that runs helm lint --strict and then pipes helm template into kubeconform for each values file, pinning the Kubernetes version and letting resources with missing schemas fail the build.

How should kubeconform handle CRDs and custom resources?

By default, kubeconform only has schemas for built-in Kubernetes kinds. A Certificate, ServiceMonitor or any resource defined by your own operator has no schema to check against. That leaves two honest options: give kubeconform schemas for those kinds, or exclude them explicitly.

Supplying generated schemas

kubeconform accepts -schema-location more than once. You can keep the default location for core kinds and add a second, templated location pointing at JSON schemas converted from your CRDs. The kubeconform repository includes an openapi2jsonschema script for this conversion. You get real field validation: a typo in spec.issuerRef fails the build instead of failing at apply time.

The cost is drift. Schemas are generated from whichever CRD files you fed the script. If clusters run a newer or older operator release, CI validates against a contract the cluster doesn't enforce. Regenerate schemas whenever you bump an operator's version, ideally in the same change.

Skipping kinds deliberately

-skip takes a list of kinds and excludes them by name. Unlike -ignore-missing-schemas, it is explicit. Anything not on the list still fails if its schema is missing, so a new, unvalidated kind cannot slip through unnoticed. The cost is an accepted, documented gap in coverage.

Approach Catches field errors in custom resources Maintenance Main risk
Generated schemas Yes Regenerate when operators change Schemas drift from installed CRDs
Explicit -skip No Update the list when kinds are added Known blind spot
-ignore-missing-schemas No None Unknown blind spot

Running with several -kubernetes-version values catches deprecated API versions before an upgrade. Each version is a full extra pass, though, and failures repeat across passes. Limit it to the versions your clusters actually run.

#!/usr/bin/env bash
# Validate rendered chart output, including custom resources, against several Kubernetes versions.
set -euo pipefail

CHART=./charts/my-app
SCHEMA_DIR=./ci/schemas

# 1. Convert the CRDs pinned for the operator releases you run into JSON schemas.
#    Regenerate in the same change that bumps an operator version, so CI and
#    clusters agree on the contract.
mkdir -p "$SCHEMA_DIR"
(
  cd "$SCHEMA_DIR"
  FILENAME_FORMAT='{kind}_{version}' \
    python3 ../../scripts/openapi2jsonschema.py ../crds/*.yaml
)

# 2. Render once, then validate against each target Kubernetes version.
#    Every version is a full extra pass, so keep the list to the versions you
#    actually run or are about to upgrade to.
helm template my-app "$CHART" -f "$CHART/ci/values.yaml" > rendered.yaml

for k8s_version in 1.29.0 1.30.0; do
  echo "== Kubernetes $k8s_version =="
  # 'default' covers built-in kinds; the second location covers your CRDs.
  # -skip lists kinds with no schema on purpose, as a documented gap.
  # Any kind missing a schema that is not on the list still fails the build,
  # which -ignore-missing-schemas would hide.
  kubeconform \
    -strict \
    -summary \
    -kubernetes-version "$k8s_version" \
    -schema-location default \
    -schema-location "$SCHEMA_DIR/{{ .ResourceKind }}_{{ .ResourceAPIVersion }}.json" \
    -skip ExternalSecret,VaultAuth \
    rendered.yaml
done

A CI script that validates rendered chart manifests with kubeconform using the default schemas for built-in kinds, JSON schemas generated from pinned CRDs, an explicit -skip list for kinds that are deliberately not validated, and one pass per target Kubernetes version.

How does chart-testing (ct) lint and install charts in a monorepo?

Chart-testing works from git, not the filesystem. A ct.yaml at the repository root sets target-branch (usually main), chart-dirs (for example charts), and the remote to compare against. With check-version-increment: true, ct fails any changed chart whose version in Chart.yaml was not bumped, so a merged change cannot overwrite a published chart version. validate-maintainers: true checks the listed maintainers, and validate-yaml plus a lint-conf file run yamllint on chart files, catching duplicate keys and indentation that helm lint accepts.

Because ct diffs the PR head against the target branch to decide which charts changed, it needs full history. A shallow checkout makes the merge base unreachable, and ct either errors or misdetects changes. Set the checkout's fetch depth to zero and make sure the target branch is fetched.

For ct install, create a kind cluster as a job step. ct installs each changed chart into a temporary namespace. If the chart has a ci/ directory, ct runs a separate install for every *-values.yaml file in it, so ci/ingress-values.yaml and ci/ha-values.yaml each test a real configuration. After each install, ct runs helm test and then deletes the release and its namespace. Delete the kind cluster in an always-run step so it is removed even when the job fails.

helm test only proves something if the chart ships a test hook. A Pod annotated helm.sh/hook: test that sends a request to the release's Service name and port fails the install when the Service has no ready endpoints or the app returns an error. That turns "pods were created" into "the Service answers."

# ct.yaml (repository root)
remote: origin
target-branch: main
chart-dirs:
  - charts
check-version-increment: true   # changed chart must bump Chart.yaml version
validate-maintainers: true      # maintainers in Chart.yaml are checked
validate-yaml: true             # run yamllint on Chart.yaml and values files
lint-conf: .github/lintconf.yaml  # yamllint rules (duplicate keys, indentation)
---
# charts/web/templates/tests/test-connection.yaml
# Runs during `helm test`, which ct install invokes after each install.
apiVersion: v1
kind: Pod
metadata:
  name: "{{ include "web.fullname" . }}-test-connection"
  annotations:
    "helm.sh/hook": test
    "helm.sh/hook-delete-policy": before-hook-creation,hook-succeeded
spec:
  restartPolicy: Never
  containers:
    - name: wget
      image: busybox:1.36
      # Request the release's Service by DNS name and port.
      # A non-zero exit (no ready endpoints, timeout, HTTP error) fails the test.
      command: ["wget"]
      args:
        - "-q"
        - "-T"
        - "10"
        - "-O-"
        - " include "web.fullname" . }}:{{ .Values.service.port }}/"

A ct.yaml that sets the target branch, chart directories, version-bump enforcement, maintainer checks and yamllint, followed by a helm test hook Pod that fails the ct install run unless the deployed Service answers HTTP requests.

How should you structure the CI pipeline?

Run the stages in this order, each one starting only after the previous one passes:

  1. Pinned tool setup. Install exact versions of helm, ct, kubeconform and kind, plus a specific kind node image.
  2. ct lint. Detect the charts that changed against the target branch and enforce a version bump on each.
  3. helm lint against every values file a chart ships, not just the defaults.
  4. kubeconform on helm template output for each values file.
  5. kind cluster creation.
  6. ct install, which runs the helm test hooks.
  7. Teardown, set to run even when earlier stages fail.

Stages 2–4 need no cluster and finish in seconds. Gate stage 5 behind them so that a typo never costs you a cluster boot.

Why pin every tool

  • Lint behaviour drifts. Lint rules and ct's change detection can differ between releases.
  • Schema versions must match the cluster. The Kubernetes version kubeconform validates against should be the same one the kind node image runs. Otherwise static checks and the install stage disagree about which APIs exist.
  • Unpinned versions cause unrelated failures. If you install "latest", a pipeline can go red on a day when no chart changed.

Bump pinned versions in their own pull request so a tooling change is never mixed with a chart change.

Illustrative estimate (not a measurement)

Assume the static stages take 1 minute and the cluster stages take 6 minutes. Assume 30% of runs fail a static check.

  • Without gating: 100 runs × 7 min = 700 minutes.
  • With gating: 30 × 1 + 70 × 7 = 520 minutes.

That saves roughly a quarter of runner time. The larger gain is that failing runs report back in 1 minute instead of 7.

When should you avoid this setup, and what are the alternatives?

Passing every check here does not prove a chart deploys. Schema validation never reaches admission webhooks, RBAC, resource quotas, image pulls or controller behaviour, so a manifest can be valid and still be rejected or crash-loop in production.

The kind install step also has costs. Every run pays for cluster startup, and failures come from the environment rather than the chart: image pull rate limits, readiness timeouts tuned for a laptop, and missing load balancers or storage classes. Charts with hard external dependencies such as a database, cloud IAM or a secrets manager need stubs to install at all. Those stubs are test code that someone must keep in step with the real dependency.

Version-bump enforcement creates friction for README-only changes. Turning it off removes that friction, but you also lose the guarantee that a published chart version never changes.

If a chart rarely changes or has one consumer, the effort to build and maintain the pipeline may cost more than the bugs it catches. The alternatives below can replace parts of this setup or sit alongside it:

Approach What it catches What it costs
Template unit tests Conditional logic and value-to-field mapping Assertions must be updated when templates change
Policy-as-code on rendered manifests Organisational rules such as limits, labels and forbidden settings You must write and maintain the policies
Server-side dry run against a real cluster Admission webhooks, CRD schemas and some RBAC Needs cluster credentials in CI
Long-lived staging cluster Real dependencies, controllers and image pulls Ongoing infrastructure cost and drift from production

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to helm chart testing ci, this is where it helps:

Saves time

  • Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
  • One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
  • Reviewers see the exact diff and the list of affected services up front, so approval is a decision rather than an investigation.

Saves cost

  • Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
  • Less duplicated configuration to maintain: one shared definition inherited by many services.
  • Per-service approve/reject means a bad shared value can be refused where it would break something, before it is live.
  • OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.