Kubernetes ConfigMap and Secret: Per-Service Config That Rolls Out

What ConfigMaps and Secrets hold, and why config lives outside the image

A ConfigMap is a Kubernetes object that stores non-sensitive configuration as key-value pairs: feature flags, upstream URLs, log levels, or a whole config file stored as a single value. A Secret has the same shape but is meant for sensitive values such as database passwords, API tokens, TLS keys, and SSH keys. Kubernetes handles Secrets somewhat differently, a topic covered later. Both objects are namespaced and are read by pods at runtime.

Keeping configuration outside the image is what makes one image usable everywhere. You build an artifact once, then promote it through staging and production with different inputs. Rotating a credential or changing a timeout is then a change to an object, not a rebuild. The Kubernetes documentation calls this out directly: you can update secrets and application configuration "without rebuilding your container images, and without exposing secrets in your stack configuration" (Kubernetes overview). It also keeps credentials out of image layers, which tend to get copied into registries and caches.

This guide treats each ConfigMap and Secret as a per-service, versioned input to a Deployment, not as shared mutable state. If several services read from one object, editing it changes all of them at once, with no rollout and no clean rollback.

Each object is limited to roughly 1 MiB. Large files, ML models, and binary assets belong in object storage or a volume.

This guide is for backend and platform engineers who run many services in each cluster and need config changes to be predictable and reviewable.

Copy# Per-service, versioned config for one service (orders-api).
# Bumping the name suffix (v3 -> v4) and updating the Deployment's references
# triggers a normal rollout, and rollback restores the old reference.
apiVersion: v1
kind: ConfigMap
metadata:
  name: orders-api-config-v3
  namespace: shop
data:
  LOG_LEVEL: "info"
  INVENTORY_URL: ""
  REQUEST_TIMEOUT_MS: "2500"
---
# Sensitive values for the same service. stringData accepts plain text;
# the API server stores it base64-encoded under .data.
apiVersion: v1
kind: Secret
metadata:
  name: orders-api-secrets-v3
  namespace: shop
type: Opaque
stringData:
  DB_PASSWORD: "replace-me-placeholder"
  PAYMENTS_API_TOKEN: "tok_placeholder_123"
---
# The Deployment, abridged to the container spec. The image is built once
# and promoted unchanged; only these referenced objects differ per environment.
# spec.template.spec.containers:
- name: orders-api
  image: registry.example.com/orders-api:1.8.0
  envFrom:
    - configMapRef:
        name: orders-api-config-v3
    - secretRef:
        name: orders-api-secrets-v3

A ConfigMap and a Secret owned by a single service, with versioned names, consumed by an unchanged image through envFrom. The Deployment part is abridged to its container spec and is not a standalone manifest.

How config reaches a pod: env injection versus volume mounts

Kubernetes gives you two delivery paths, and they behave very differently once a value changes.

Environment variables

env with valueFrom.configMapKeyRef or secretKeyRef maps one key to one variable. envFrom imports every key in the object, with an optional prefix to avoid collisions. Keys that are not valid environment variable names are skipped and reported as an event rather than failing the pod, so a typo can silently drop a setting. Marking a reference optional: true lets the pod start when the object or key is missing. Without it, the container stays in a waiting state. The kubelet resolves values once, when the container starts. Editing the ConfigMap afterward changes nothing in running processes, because the environment is fixed when the process is created.

Volume mounts

Mounting a ConfigMap or Secret as a volume turns each key into a file. The kubelet swaps updated files in atomically through a symlink. Changes appear eventually, after the kubelet's sync period plus its cache delay, not instantly. Mounts that use subPath never receive updates. Even when files do change, the app only sees the new values if it re-reads or watches the files.

API server → kubelet (watch/cache) → container ├─ env: resolved once at container start └─ /etc/config/<key>: refreshed on kubelet sync (not with subPath)

CopyapiVersion: v1
kind: Pod
metadata:
  name: orders-api
spec:
  containers:
    - name: app
      image: registry.example.com/orders-api:1.4.2
      env:
        # One key -> one variable. Resolved once at container start.
        - name: LOG_LEVEL
          valueFrom:
            configMapKeyRef:
              name: orders-config
              key: log_level
        - name: DB_PASSWORD
          valueFrom:
            secretKeyRef:
              name: orders-db
              key: password
              optional: true   # pod starts even if Secret/key is missing
      # Import every key; prefix avoids collisions. Invalid names are skipped.
      envFrom:
        - configMapRef:
            name: orders-config
          prefix: ORDERS_
      volumeMounts:
        # Whole-volume mount: kubelet refreshes files eventually.
        - name: config
          mountPath: /etc/config
          readOnly: true
        # subPath mount: this file NEVER receives updates.
        - name: config
          mountPath: /etc/app/feature-flags.json
          subPath: feature-flags.json
  volumes:
    - name: config
      configMap:
        name: orders-config   # each key becomes /etc/config/<key>

A single pod spec that reads the same ConfigMap through env, envFrom, a whole-volume mount and a subPath mount, with comments noting which of these paths pick up later edits.

Secrets are not encrypted by default: what actually protects them

A Secret's data field is base64, which is an encoding and not encryption. Anyone who can read the object can decode it in one step, so the encoding only exists to carry binary values through YAML and JSON. What actually protects the value is everything around it.

RBAC scoped by namespace and name. Grant get on Secrets only to the service accounts and people that need them, and use resourceNames to narrow a Role to specific Secrets. Be careful with list and watch. They return full Secret contents, not just names, so a broad "read-only" role across a namespace exposes every credential in it.

Pod creation is credential access. Anyone who can create a pod in a namespace can mount any Secret in that namespace and print it, whatever their direct Secret permissions say. Treat create rights on pods, Deployments, and Jobs as equivalent to reading the Secrets they could reference. This is the main reason to give each service its own namespace.

Encryption at rest. By default, Secrets are stored in etcd unencrypted, so etcd backups and snapshots contain them in the clear. To fix this, configure an encryption provider on the API server with --encryption-provider-config, preferably one backed by an external key management service so the key does not sit next to the data.

Version control. Never commit plain Secret manifests. Git history keeps the value even after the file is deleted. Two common options:

• Commit encrypted or sealed manifests that only an in-cluster controller can decrypt.
• Keep values in an external secret store and sync them into the cluster at runtime.

Even with all of this in place, injected values can still escape through the application itself. Crash dumps, debug endpoints that print the environment, and inherited child process environments can all expose them. This is another reason to prefer mounted files for high-value credentials.

Copy# 1) Role: the payments service account may read exactly one Secret.
#    No list/watch, which would return every Secret's contents in the namespace.
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: read-payments-db-secret
  namespace: payments
rules:
  - apiGroups: [""]
    resources: ["secrets"]
    resourceNames: ["payments-db-credentials"]
    verbs: ["get"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: payments-api-reads-db-secret
  namespace: payments
subjects:
  - kind: ServiceAccount
    name: payments-api
    namespace: payments
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: read-payments-db-secret
---
# 2) Encryption at rest. This is NOT applied with kubectl: it is a file on the
#    control plane passed to kube-apiserver via --encryption-provider-config.
apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
  - resources: ["secrets"]
    providers:
      # First provider encrypts new writes; the key lives in an external KMS.
      - kms:
          apiVersion: v2
          name: example-kms-plugin
          endpoint: unix:///var/run/kmsplugin/socket.sock
          timeout: 3s
      # identity lets the server still read Secrets written before encryption
      # was enabled. Rewrite existing Secrets so they get encrypted too.
      - identity: {}

A namespace-scoped Role that grants get on a single named Secret to one service account, plus an API server EncryptionConfiguration that encrypts Secrets in etcd with a KMS-backed key.

Rolling out on config change: hash annotations, rollout restart, and immutable versions

A Deployment only creates a new ReplicaSet when its pod template changes. Editing a ConfigMap or Secret that the template references does not change the template, so nothing rolls out. Environment variables keep their old values until the container restarts. Mounted files are eventually refreshed by the kubelet, except when mounted with subPath, but most applications read config once at startup and never notice the new file.

Three ways to force the rollout

Config hash annotation. Your templating tool or CI computes a checksum of the rendered ConfigMap and writes it into a pod template annotation. When the data changes, the annotation changes, the template changes, and a normal rolling update follows. This depends on tooling: if someone runs kubectl edit on the ConfigMap directly, the hash stays the same and the change is silently never applied.

kubectl rollout restart deployment/<name>. This stamps a restartedAt annotation on the template and triggers a rollout. It needs no tooling, but it is a manual step. ConfigMaps keep no revision history, so the annotation tells you when pods restarted, not which config they picked up.

Immutable, versioned objects. You create app-config-v2 with immutable: true and change the Deployment to reference it. The template change triggers the rollout, and rolling back the Deployment also restores the previous config. Two costs come with this. Old versions pile up unless something deletes them. During an incident, you also cannot hot-patch a value. You have to create a new object and update the reference.

Copy# Immutable, versioned ConfigMap: never edited in place.
# To change config, create app-config-v3 and update the reference below.
apiVersion: v1
kind: ConfigMap
metadata:
  name: app-config-v2
  labels:
    app: checkout
    config-version: "v2"   # label makes old versions easy to find and delete
immutable: true            # API server rejects any later change to data
data:
  LOG_LEVEL: "info"
  PAYMENT_TIMEOUT_MS: "3000"
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: checkout
spec:
  replicas: 3
  selector:
    matchLabels:
      app: checkout
  template:
    metadata:
      labels:
        app: checkout
    spec:
      containers:
        - name: checkout
          image: registry.example.com/checkout:1.4.0
          envFrom:
            # Changing this name from v1 to v2 changes the pod template,
            # so the Deployment creates a new ReplicaSet and rolls out.
            # Rolling back the Deployment points pods at v1 again,
            # as long as app-config-v1 has not been deleted.
            - configMapRef:
                name: app-config-v2

An immutable, versioned ConfigMap and a Deployment that references it by name, so switching to a new version changes the pod template and triggers a rolling update.

A per-service layout that scales across teams

Those trade-offs only stay manageable if the objects themselves are easy to find, attribute, and change without side effects. That depends on layout more than on any single rollout technique.

One ConfigMap and one Secret per service

Give each service exactly one ConfigMap and one Secret, created in the service's own namespace and named after the service, for example orders-api-config and orders-api-secrets. Because the name is predictable, anyone debugging a pod knows what to look for without reading the Deployment. Because the objects are namespaced, RBAC can grant a team access to its own Secrets and nothing else.

Label every object with app.kubernetes.io/name, app.kubernetes.io/part-of, and an owner label such as team: payments. Then kubectl get configmaps -l team=payments across namespaces answers "what does this team own?" When a value looks wrong, the owner label tells you who to ask.

Why shared ConfigMaps fail

The most common failure is several services reading one shared ConfigMap that someone edits in place. Environment variables are only read when a container starts. Mounted files update with a delay, and only on pods that use them as volumes. After an edit, some pods run the new values and others run the old ones until they happen to restart. No new ReplicaSet was created, so there is no revision to roll back to and no record of what the previous values were.

Share config only for deliberate platform-wide values, such as a cluster domain or a tracing endpoint. Version those values too, with names like platform-config-v3, and let each service opt in by changing its reference.

The example below applies this pattern. It shows a versioned, immutable ConfigMap with ownership labels, and the Deployment that references it by its versioned name.

When to use this approach, when to avoid it, and the alternatives

Where native objects fit

ConfigMaps and Secrets work best for small, per-service values that change at deploy time: endpoints, log levels, timeouts, and a few credentials. Each object is capped at 1 MiB. Changing one should mean a rollout, which is the workflow the previous sections built.

Where they strain

• Large or binary files. Models, certificate bundles and datasets hit the size cap and fill the API server's storage. Put them in object storage or a volume.
• High-churn values. Anything that changes several times an hour causes repeated rollouts. A hash-annotated rollout is a poor fit for something that is effectively live data.
• Strict secret requirements. Audited access, automatic rotation and short-lived credentials need a system designed for those jobs.

Avoid marking references optional unless the default is genuinely safe. A missing required key stops the pod with CreateContainerConfigError, and you see it during the rollout. An optional key that is missing gives you a pod that starts, passes readiness, and then misbehaves at runtime, usually much later and in a place that is harder to trace.

Alternatives

Verifying, debugging, and the impact on cost and engineering time

A debugging workflow that holds up

Start with kubectl describe pod <pod>. The Events list shows whether the kubelet could resolve every referenced ConfigMap and Secret. CreateContainerConfigError almost always means a non-optional reference points to an object or key that doesn't exist in the pod's namespace. Common causes are a typo, a missing key, or a manifest applied before its config. Next, run kubectl get configmap <name> -o yaml to compare the stored data with what you expect.

To see what the process actually received, run kubectl exec <pod> -- env for injected variables and kubectl exec <pod> -- cat <path> for mounted files. Do this because env values are fixed at container start. Mounted files may update later, but not for subPath mounts.

To confirm a config-driven rollout, run kubectl rollout status deployment/<name>. Then check kubectl rollout history deployment/<name> to see a new revision whose template carries the changed hash annotation.

kubectl rollout undo restores the previous pod template, not the previous ConfigMap contents. Rollback only restores old config if you use versioned, immutable object names.

Cost and time

The savings come from three places:

• A config change skips the image build, scan, and push.
• Rollback becomes a single command.
• Pods stop silently running stale config.

The costs are a hash step in CI and garbage collection of old versioned objects.

Illustrative estimate (assumed numbers, not a benchmark):

saved hours/month = changes × rebuild minutes ÷ 60

With 40 config changes and an assumed 12-minute rebuild pipeline per change:

40 × 12 ÷ 60 = 8 hours of pipeline time per month.

Subtract the time you spend maintaining the hash step and the cleanup jobs.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to kubernetes configmap secret, this is where it helps:

Saves time

• Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
• One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
• Changing a shared variable is one edit and one review instead of editing every service's .env by hand.

Saves cost

• Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
• Less duplicated configuration to maintain: one shared definition inherited by many services.
• A wrong or stale value is caught at review time by the services it affects, not by a failed deploy.
• OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.