External Secrets Operator with AWS Secrets Manager: A Practical Guide

What External Secrets Operator does and what it does not

External Secrets Operator (ESO) is a sync mechanism. It reads values from an external store such as AWS Secrets Manager and writes them into ordinary Kubernetes Secrets. AWS stays the authoritative source. The in-cluster Secret is a disposable cache: delete it and the operator recreates it on the next reconcile, and edit it by hand and the operator overwrites it.

Teams adopt it for three reasons. First, one source of truth serves many clusters and non-Kubernetes consumers. Second, access to the value is governed and logged in AWS through IAM policies and CloudTrail, not scattered across cluster configs. Third, Git holds only references to secrets, never the values, so GitOps repositories stay safe to share.

CopyAWS Secrets Manager  (authoritative)
        ▲
        │  GetSecretValue, authorized via workload identity
        │  (IAM role bound to the operator's ServiceAccount)
        │
External Secrets Operator ── polls every refreshInterval
        │
        │  create / update
        ▼
Kubernetes Secret  (cache, stored in etcd)
        │
        │  env vars or volume mount
        ▼
Pods  (values held in process memory)

What ESO does not do is make secrets safer once they are inside the cluster. A Kubernetes Secret is base64-encoded, and base64 is encoding, not encryption. The values sit in etcd and in the memory of every pod that mounts them. Anyone with get, list or watch on Secrets in that namespace can read them, and list returns full values, not just names.

Because of this, enable etcd encryption at rest, preferably backed by a KMS key. Restrict Secret read verbs in RBAC to the workloads and operators that actually need them.

Copy+--------------------------------------+
|  AWS Secrets Manager (authoritative) |
|  IAM policies; CloudTrail logs reads |
+--------------------------------------+
                   ^
                   |  GetSecretValue
                   |  authorized via workload identity:
                   |  IAM role bound to operator ServiceAccount
                   |
+--------------------------------------+
|  External Secrets Operator           |
|  reconciles every refreshInterval    |
+--------------------------------------+
                   |
                   |  create / update
                   |  (recreates if deleted, overwrites edits)
                   v
+--------------------------------------+  <-- RBAC: get/list/watch
|  Kubernetes Secret (disposable cache)|      decides who reads values
|  base64-encoded, stored in etcd      |  <-- enable etcd encryption
+--------------------------------------+      at rest (KMS-backed)
                   |
                   |  env vars or volume mount
                   v
+--------------------------------------+
|  Pods (values in process memory)     |
+--------------------------------------+

  Git holds ExternalSecret references only, never secret values

External Secrets Operator uses workload identity to read from AWS Secrets Manager on each refreshInterval and writes a Kubernetes Secret that remains readable in etcd and pod memory, so RBAC and etcd encryption at rest still govern in-cluster exposure.

SecretStore, ClusterSecretStore and ExternalSecret: how the pieces fit

External Secrets Operator works with three custom resources. A store describes how to reach a provider such as AWS Secrets Manager: the region and the identity used to authenticate. An ExternalSecret says which remote secrets to fetch and how to shape them. Each data entry maps a remote key, and optionally one property inside a JSON secret, to a key in a target Kubernetes Secret. The operator creates that Secret and keeps it reconciled.

The two store types differ mainly in who can use them:

The ExternalSecret only names a store. The store's IAM identity decides what can actually be read.

Suppose a ClusterSecretStore uses a role that can call secretsmanager:GetSecretValue on every secret in the account. Any user who can create an ExternalSecret in any namespace can point it at prod/payments/db and get the value back as a Secret they can read. That user never needs AWS access.

To close this path:

• Scope the role to a resource prefix.
• Restrict the store to specific namespaces.
• Treat create on ExternalSecret as a privileged verb in RBAC.
• Use admission policy to reject references to stores a namespace should not use.

Authenticating to AWS without static credentials

Static access keys look like the quick fix, but they don't solve the problem. If you put an access key in a bootstrap Kubernetes Secret, you have a long-lived credential in etcd that someone has to rotate by hand. Anyone who can read that namespace's Secrets can use it. That is the exposure the operator was supposed to remove.

Bind an IAM role to the ServiceAccount

On EKS, IRSA lets a pod exchange its projected ServiceAccount token for short-lived AWS credentials. It needs three pieces:

• An IAM OIDC provider registered for the cluster's issuer URL.
• A role trust policy that allows sts:AssumeRoleWithWebIdentity from that provider. It should include conditions that pin <issuer>:sub to system:serviceaccount:<namespace>:<name> and <issuer>:aud to sts.amazonaws.com. Without the sub condition, any ServiceAccount in the cluster can assume the role.
• The eks.amazonaws.com/role-arn annotation on the ServiceAccount that the SecretStore references.

EKS Pod Identity and other platforms' workload identity follow the same model: trust a specific workload identity, not a stored secret.

Scope the policy by name

Grant secretsmanager:GetSecretValue and secretsmanager:DescribeSecret only on matching ARNs, such as arn:aws:secretsmanager:<region>:<account>:secret:team-payments/*. AWS appends a random suffix to secret ARNs, so you need the trailing wildcard even for a single secret.

If the secrets use customer-managed KMS keys, also grant kms:Decrypt on those key ARNs. You can restrict it further with kms:ViaService set to Secrets Manager.

A <team>/<env>/<service> naming convention is what makes these rules practical. One prefix maps to one role and one store.

When sync fails with AccessDenied

Trust-policy mistakes rarely produce a specific error. They usually show up as a generic access-denied on the ExternalSecret. Check these in order:

• Does the sub condition exactly match the namespace and ServiceAccount name?
• Is the issuer URL correct, and does the OIDC provider exist in the role's account?
• Is the annotation present on the ServiceAccount that the store actually references?
• Does the policy's resource pattern match the secret's name and region?
• Does the KMS key policy allow the role?

Copy{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "ReadTeamPaymentsSecrets",
      "Effect": "Allow",
      "Action": [
        "secretsmanager:GetSecretValue",
        "secretsmanager:DescribeSecret"
      ],
      "Resource": "arn:aws:secretsmanager:us-east-1:111122223333:secret:team-payments/*"
    },
    {
      "Sid": "DecryptViaSecretsManagerOnly",
      "Effect": "Allow",
      "Action": "kms:Decrypt",
      "Resource": "arn:aws:kms:us-east-1:111122223333:key/1234abcd-12ab-34cd-56ef-1234567890ab",
      "Condition": {
        "StringEquals": {
          "kms:ViaService": "secretsmanager.us-east-1.amazonaws.com"
        }
      }
    }
  ]
}

A least-privilege permissions policy for the team-payments role: read-only access to secrets under the team-payments/ prefix (the wildcard also covers the random ARN suffix) and kms:Decrypt on one customer-managed key, only when called through Secrets Manager.

Step-by-step implementation: from AWS secret to Kubernetes Secret

Install and create the source secret

Install the operator with Helm from the project's chart repository into its own namespace, and let the chart install the CRDs. Before writing any manifests, run kubectl api-resources | grep external-secrets and kubectl explain externalsecret.spec to see which API version and fields your installation serves. Examples online often target a different release, and field names have changed between versions.

Next, create the secret in AWS Secrets Manager with aws secretsmanager create-secret. Pass --secret-string as a JSON object, for example with username and password keys. A single JSON secret keeps related values together, so they are rotated as one unit.

Define the store and the ExternalSecret

Create a namespaced SecretStore that sets the AWS provider to the SecretsManager service and region. Its JWT auth block should reference the annotated ServiceAccount from the previous section. A namespaced store keeps that IAM role available only to the namespace that owns it.

The ExternalSecret maps each remoteRef.key and remoteRef.property pair to a key in the target Secret. Select individual properties rather than the whole blob, so pods only receive the values they need.

Verify

kubectl describe externalsecret should show a Ready condition with a synced reason. If it does not, the events there usually name the IAM or path error. Then inspect the generated Secret's keys and decode one value to confirm it matches the source.

Ownership policies

Set creationPolicy and deletionPolicy explicitly. With the Owner creation policy, deleting the ExternalSecret garbage-collects the Secret, and any pod that mounts it fails on its next restart. If you point the ExternalSecret at an existing, hand-managed Secret, sync overwrites its data. Use Merge or Orphan deliberately when you adopt existing Secrets, and decide in advance what should happen to the Secret when the source is removed.

Copy# Confirm the served version first: kubectl api-resources | grep external-secrets
# Older releases serve external-secrets.io/v1beta1 instead of v1.
apiVersion: external-secrets.io/v1
kind: SecretStore
metadata:
  name: aws-secrets
  namespace: payments            # store is usable only from this namespace
spec:
  provider:
    aws:
      service: SecretsManager
      region: eu-west-1
      auth:
        jwt:
          serviceAccountRef:
            name: payments-eso-sa  # ServiceAccount annotated with the IAM role
---
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: payments-db
  namespace: payments
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: aws-secrets
    kind: SecretStore
  target:
    name: payments-db-credentials
    creationPolicy: Owner        # Secret is deleted if this ExternalSecret is deleted
    deletionPolicy: Retain       # keep the last synced values if the remote secret disappears
  data:
    - secretKey: DB_USER         # key in the generated Kubernetes Secret
      remoteRef:
        key: prod/payments/db    # name of the AWS secret
        property: username       # single JSON field, not the whole blob
    - secretKey: DB_PASSWORD
      remoteRef:
        key: prod/payments/db
        property: password

A namespaced SecretStore using the IRSA-annotated ServiceAccount and an ExternalSecret that maps individual JSON properties into a Secret, with creationPolicy and deletionPolicy set explicitly.

Refresh intervals and secret rotation without outages

refreshInterval sets how often the operator re-reads the provider. It is the maximum lag between a change in AWS Secrets Manager and the Kubernetes Secret. Each refresh also costs at least one API call per ExternalSecret.

Illustrative estimate: assume 400 ExternalSecrets across a cluster. At a 1m interval, that is about 400 calls per minute, or 576,000 per day. At 1h, it falls to 9,600 per day. Short intervals buy freshness you rarely need. They also raise cost and the risk of throttling, which shows up as failed syncs during exactly the busy periods you care about.

An updated Secret does not restart anything. Environment variables are read once, at container start. Mounted volumes are updated by the kubelet after a delay, and only help if the application rereads the file. subPath mounts are never updated.

For database passwords, never invalidate the old credential at the moment you publish the new one. Pods pick up the new value at different times, so both credentials must work for a while. Use one of two patterns:

• Dual users: alternate between two users.
• Grace period: keep the old password valid on the database side.

Revoke the old credential only after every pod has rolled over.

Copy# ExternalSecret: hourly polling keeps API volume low.
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: orders-db
spec:
  refreshInterval: 1h            # max lag between AWS change and K8s Secret
  secretStoreRef:
    name: aws-secrets-manager
    kind: ClusterSecretStore
  target:
    name: orders-db-credentials  # Kubernetes Secret ESO writes
  data:
    - secretKey: password
      remoteRef:
        key: prod/orders/db      # placeholder Secrets Manager name
        property: password
---
# Deployment fragment: two ways to pick up a new value.
apiVersion: apps/v1
kind: Deployment
metadata:
  name: orders-api
spec:
  template:
    metadata:
      annotations:
        # Pipeline bumps this when the secret changes; any pod-template
        # change triggers a normal rolling update.
        example.com/db-secret-version: "2026-10-27-01"
    spec:
      containers:
        - name: app
          image: registry.example.com/orders-api:1.4.2
          volumeMounts:
            - name: db-creds
              mountPath: /etc/db   # whole-dir mount, NOT subPath, so kubelet updates it
              readOnly: true
      volumes:
        - name: db-creds
          secret:
            secretName: orders-db-credentials

An ExternalSecret with an hourly refreshInterval, plus a Deployment that mounts the synced Secret as a directory without subPath, so apps that reread the file see updates, and uses a pipeline-bumped pod-template annotation to force a rolling restart when needed.

Running it in production: observability, failure modes and multi-tenancy

Once the operator manages credentials for real workloads, it is critical infrastructure.

Observability

Each ExternalSecret reports a Ready condition in its status, with a reason and message that usually name the cause: an AccessDenied from IAM, a missing key, or a store that is not ready. Kubernetes events on the object carry the same information. The operator also exposes Prometheus metrics for sync calls and condition status. Alert when any ExternalSecret stays not-ready longer than a few refresh intervals, and when sync errors climb across many objects at once. The second pattern usually points to a broken store or IAM role rather than one bad key.

Failure modes

When AWS Secrets Manager is unreachable or credentials expire, the last synced Kubernetes Secret normally stays in place, so running pods keep working. That hides the outage. New ExternalSecrets never produce a Secret, new namespaces deploy pods stuck on missing volumes, and rotations stop reaching the cluster. Without alerts, the first symptom is a database rejecting a revoked password days later.

Multi-tenancy

Give each namespace or team its own SecretStore backed by its own IAM role, scoped to a path prefix in Secrets Manager. Treat any ClusterSecretStore as privileged, and use admission policies to control which namespaces may reference it. Keep ExternalSecret manifests in Git and deploy them through GitOps. They hold references, not values, so review and audit stay simple.

Budget time for operator upgrades and CRD API version migrations, which require updating stored manifests before older API versions are removed.

Who should use it, when to avoid it, and alternatives

The operator pays off for platform teams running several clusters, and for organizations that already keep credentials in AWS Secrets Manager. In both cases one secret has many consumers. Each manual copy into a cluster is a chance for values to drift and a step someone has to remember during rotation. Keeping the source of truth in AWS also keeps CloudTrail auditing and IAM access policy in one place, so they are not split across clusters.

Skip it for a single small cluster with a handful of secrets. There, running and upgrading another controller can cost more than updating a few Secrets by hand. Also avoid it when policy forbids secret values from ever reaching etcd. The operator's output is a normal Kubernetes Secret, so that requirement rules it out.

The operator reduces engineering time in two places: you stop copying secrets by hand, and rotation becomes routine work instead of a coordinated change. Against that, plan for the initial setup and for debugging workload identity. IAM trust policies and service account annotations are where most first deployments stall. You also take on the ongoing cost of operating one more critical controller.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to external secrets operator, this is where it helps:

Saves time

• Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
• One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
• Reviewers see the exact diff and the list of affected services up front, so approval is a decision rather than an investigation.

Saves cost

• Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
• Less duplicated configuration to maintain: one shared definition inherited by many services.
• Per-service approve/reject means a bad shared value can be refused where it would break something, before it is live.
• OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.