What is the External Secrets Operator and how does it sync AWS Secrets Manager?

The External Secrets Operator (ESO) is a Kubernetes controller, not a secret store. It holds nothing on its own. AWS Secrets Manager remains the source of truth, and ESO runs a reconciliation loop that copies values from it into native Kubernetes Secrets.

The loop has three parts:

  • ExternalSecret resources: The controller watches these. Each one declares which remote secret (or which JSON keys inside it) to fetch, and which Kubernetes Secret to write.
  • SecretStore or ClusterSecretStore: The ExternalSecret references one of these. It tells ESO how to reach AWS: the region, the service, and the identity to authenticate as.
  • Refresh schedule: On every refresh interval, ESO fetches the current value and updates the target Secret if it has changed.

Your pods never call AWS. They read an ordinary Secret through environment variables or a mounted volume, the same way they would read one created with kubectl. Application code needs no AWS SDK, no IAM logic and no retry handling for Secrets Manager outages.

That design has a cost. The synced Secret is a full copy of the value, held in etcd. It is only as protected as your etcd encryption at rest and your RBAC rules on get, list and watch for Secrets. If someone can read Secrets in a namespace, they can read the AWS-managed value.

Teams adopt ESO for three reasons:

  • One authoritative place to change a credential.
  • No application changes.
  • No static AWS keys or encoded secrets committed to Git.

It fits backend and platform engineers running workloads on EKS, or on other clusters, whose organizations have already standardized on Secrets Manager.

                      +----------------------------+
                      |    AWS Secrets Manager     |
                      |     (source of truth)      |
                      +-------------+--------------+
                                    ^
                                    |  fetch on every refresh interval
 ===================================|========== Kubernetes cluster ==========
                                    |
 +----------------+         +-------+--------+         +--------------------+
 | ExternalSecret |<--------| ESO controller |-------->| SecretStore or     |
 | remote key,    | watches | reconciliation |  reads  | ClusterSecretStore |
 | JSON keys,     |         | loop           |         | region, service,   |
 | target Secret  |         |                |         | auth identity      |
 +----------------+         +-------+--------+         +--------------------+
                                    |  writes Secret; updates it if changed
                                    v
                         +----------------------+        +------------------+
                         | Kubernetes Secret    |------->| Pod              |
                         | (full copy in etcd)  | env or | no AWS calls     |
                         |                      | volume | reads Secret     |
                         +----------------------+        +------------------+

 The copy is protected only by etcd encryption at rest and by RBAC
 on Secrets (get, list, watch).

ESO's controller watches ExternalSecret resources, uses a SecretStore or ClusterSecretStore to fetch values from AWS Secrets Manager on each refresh interval, and writes a native Kubernetes Secret that pods read without ever calling AWS.

How does ESO work, step by step?

Each reconciliation of an ExternalSecret follows the same sequence:

  1. Admission. When you apply an ExternalSecret or store, ESO's validating webhook checks the resource against the CRD schema and rejects malformed specs before they reach the controller.
  2. Store resolution. The controller reads the ExternalSecret's secretStoreRef and loads the referenced SecretStore (same namespace) or ClusterSecretStore (cluster-scoped, subject to its conditions). If the store is not ready, the ExternalSecret inherits that failure.
  3. Authentication. Using the store's auth block, ESO obtains AWS credentials. With IRSA, it uses a token for the referenced service account to assume the annotated IAM role through STS, which returns short-lived credentials. With Pod Identity, the credentials come from the EKS Pod Identity association.
  4. Fetch. ESO calls GetSecretValue for each remoteRef.key. For JSON secrets it extracts the requested property, or every top-level key when dataFrom.extract is used.
  5. Render. If target.template is set, ESO renders the fetched values into the final keys with Go templating.
  6. Write. ESO creates or updates the target Kubernetes Secret according to creationPolicy. With Owner, it sets an owner reference so the Secret is garbage-collected with the ExternalSecret.
  7. Status and requeue. ESO sets the Ready condition and emits events describing success or failure, then schedules the next reconciliation after refreshInterval.

The kubelet and your application handle everything after step 6. ESO's job ends once the Kubernetes Secret matches the value in AWS.

Who should use the External Secrets Operator?

ESO suits teams that meet most of these conditions:

  • Secrets Manager is already the system of record. Security or platform teams create, rotate and audit credentials in AWS, and Kubernetes workloads need to consume them.
  • Several services, namespaces or clusters consume the same credentials. One rotation in AWS should reach all of them without anyone running kubectl create secret by hand.
  • Deployments are managed with GitOps or reviewed manifests. ExternalSecret resources contain only references and templates, so they can live in Git and go through code review without exposing values.
  • Applications already read configuration from env vars or files. ESO adds nothing to application code, so it fits existing services, including third-party images you cannot modify.
  • Kubernetes Secrets in etcd are acceptable under your threat model, because etcd encryption at rest and Secret RBAC are already in place.

In practice, the main users are platform engineers, who install the controller, define stores and IAM roles, and set guardrails, and application engineers, who write ExternalSecret manifests for their own services. Teams that cannot accept an etcd copy, need per-pod audit records, or run a single small cluster with a few static credentials should read the last section before adopting it.

How can ESO save engineering time?

ESO removes repeated, manual work in a few specific places:

  • No per-service secret-loading code. Without ESO, each service either embeds an AWS SDK client with caching, retries and IAM handling, or relies on a person or pipeline to copy values into Kubernetes. ESO does this once, in one controller.
  • Rotation happens in one place. Changing a value in Secrets Manager propagates to every cluster and namespace that references it on the next refresh. Nobody has to find and update each copy.
  • Onboarding is one manifest. Giving a new service a credential means writing a short ExternalSecret and, at most, extending an IAM policy prefix. No pipeline changes and no out-of-band kubectl steps.
  • Reviewable, reproducible environments. Because manifests contain references rather than values, rebuilding a cluster or creating a new environment means reapplying Git, not reconstructing secrets by hand.
  • Faster diagnosis. Sync state appears as Ready conditions and events on Kubernetes resources, which is quicker to inspect than tracing a failed CI step or a missing manual update.

As an illustrative estimate with assumed inputs: suppose 15 credentials are rotated quarterly across 3 clusters, and a manual update (fetching the value, updating the Secret, verifying the rollout) takes an assumed 10 minutes per credential per cluster. That is 45 updates, or about 450 minutes of hands-on work each quarter, plus the risk of missing a cluster. With ESO, the same rotation is a change in Secrets Manager, and the remaining work is the reload step described in the rotation section below. Your own numbers depend on how many credentials you have, how often they rotate, and how automated your current process already is. Weigh the savings against the overhead of running ESO: upgrading the controller and CRDs, maintaining IAM roles, and alerting on sync failures.

How do you install ESO and authenticate to AWS without static keys?

Install the chart from the project's official Helm repository into its own namespace, conventionally external-secrets, with --create-namespace. A dedicated namespace keeps the controller's service account, RBAC, and webhook isolated from application workloads and makes it easy to scope who can modify them.

ESO's API has moved between versions, and snippets online often target an older one. Before applying any manifest, confirm what your cluster actually serves: kubectl get crds | grep external-secrets.io, then kubectl get crd secretstores.external-secrets.io -o jsonpath='{.spec.versions[?(@.served==true)].name}'. Repeat this for clustersecretstores and externalsecrets. Then check the controller with kubectl logs -n external-secrets deploy/external-secrets for webhook certificate or RBAC errors.

Authenticate with workload identity

On EKS, use IAM Roles for Service Accounts (IRSA) or EKS Pod Identity so the controller gets short-lived credentials instead of access keys stored in a Kubernetes Secret. With IRSA, annotate the service account the store references with eks.amazonaws.com/role-arn, and restrict the role's trust policy to that exact namespace and service account name. With Pod Identity, create an association between that service account and the role. Either way, nothing long-lived needs rotating or protecting.

Least-privilege policy

Grant only secretsmanager:GetSecretValue and secretsmanager:DescribeSecret, scoped to specific secret ARNs or a name prefix such as prod/payments/*. A trailing wildcard is needed in either case because Secrets Manager appends a random suffix to every ARN. If secrets use a customer-managed KMS key, also allow kms:Decrypt on that key's ARN. Without it, reads fail with an access error even though the Secrets Manager permissions are correct.

cat > eso-secretsmanager-read.json <<'EOF'
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "ReadPaymentsSecretsOnly",
      "Effect": "Allow",
      "Action": [
        "secretsmanager:GetSecretValue",
        "secretsmanager:DescribeSecret"
      ],
      "Resource": "arn:aws:secretsmanager:us-east-1:111122223333:secret:prod/payments/*"
    },
    {
      "Sid": "DecryptWithCustomerManagedKeyViaSecretsManager",
      "Effect": "Allow",
      "Action": "kms:Decrypt",
      "Resource": "arn:aws:kms:us-east-1:111122223333:key/1234abcd-12ab-34cd-56ef-1234567890ab",
      "Condition": {
        "StringEquals": {
          "kms:ViaService": "secretsmanager.us-east-1.amazonaws.com"
        }
      }
    }
  ]
}
EOF

aws iam create-policy \
  --policy-name eso-prod-payments-read \
  --policy-document file://eso-secretsmanager-read.json

aws iam attach-role-policy \
  --role-name eso-prod-payments \
  --policy-arn arn:aws:iam::111122223333:policy/eso-prod-payments-read

Creates and attaches an IAM policy that allows only GetSecretValue and DescribeSecret on the prod/payments/ prefix. It also allows kms:Decrypt on one customer-managed key, but only when the call comes through Secrets Manager.

Should you use a SecretStore or a ClusterSecretStore?

A SecretStore is namespaced, so an ExternalSecret can only reference a store in its own namespace. A ClusterSecretStore is cluster-scoped, and any namespace can reference it unless you restrict it. The choice matters less for convenience than for the IAM role behind the store. ESO fetches whatever an ExternalSecret asks for using that role's permissions. It does not judge whether the requesting namespace should see a given secret. A broadly scoped role behind an unrestricted ClusterSecretStore means anyone who can create an ExternalSecret in any namespace can read every secret that role can reach.

Aspect SecretStore ClusterSecretStore
Scope One namespace Whole cluster, optionally limited by conditions
IAM boundary One role per team or namespace One role shared across referencing namespaces
Management overhead A store and role per namespace One store, one role
Blast radius Limited to that team's secret paths Everything the role can read, in every allowed namespace
Good fit Application and team secrets Platform-owned secrets such as shared registry credentials

Default to per-team IAM roles with namespaced stores. Scope each role's policy to a path prefix such as payments/* so the IAM policy enforces the namespace boundary. Use a ClusterSecretStore only for secrets the platform team owns. Restrict it with spec.conditions (a namespaceSelector or an explicit namespaces list) so only approved namespaces can reference it.

The manifest below defines a namespaced SecretStore for a payments namespace. It authenticates through a service account annotated with that team's role.

apiVersion: v1
kind: ServiceAccount
metadata:
  name: payments-eso
  namespace: payments
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::111122223333:role/eso-payments-reader
---
apiVersion: external-secrets.io/v1
kind: SecretStore
metadata:
  name: aws-secrets-manager
  namespace: payments
spec:
  provider:
    aws:
      service: SecretsManager
      region: us-east-1
      auth:
        jwt:
          serviceAccountRef:
            # Resolved in the store's own namespace (payments)
            name: payments-eso

A namespaced SecretStore for the payments namespace that authenticates to AWS Secrets Manager through an IRSA-annotated service account, whose IAM role is limited to the payments/ secret prefix.*

How do you write an ExternalSecret and consume it in a Deployment?

An ExternalSecret references the store through secretStoreRef and declares what to copy. Use data for individual fields: each entry sets a secretKey (the key in the Kubernetes Secret) and a remoteRef with the AWS secret name as key and, for JSON secrets, a property that selects one field. Use dataFrom with extract to pull every top-level JSON key at once. This is convenient, but any new key added in AWS appears in the pod without review. When an application expects a single value such as a connection string, target.template renders one with Go templating, for example postgres://{{ .username }}:{{ .password }}@{{ .host }}/app. This keeps the components in AWS and the assembly in Git.

Set refreshInterval for each secret based on how quickly changes must arrive. As an illustrative estimate with assumed inputs, 300 ExternalSecrets at 1m make about 432,000 GetSecretValue calls per day. At 1h they make about 7,200. Secrets Manager bills per API call and enforces request-rate quotas, so a short interval should be reserved for credentials that actually rotate.

target.creationPolicy defaults to Owner: ESO creates the Secret and adds an owner reference, so deleting the ExternalSecret also deletes the Secret. If another tool already manages that Secret, use Merge so ESO only adds keys to it, or use Orphan so the Secret stays after ESO stops managing it. deletionPolicy (Retain, Delete, Merge) controls what happens when the secret in AWS disappears.

Consume the Secret through envFrom/secretKeyRef or a volume. Environment variables are read only when the container starts. Mounted volumes pick up changes after the kubelet's sync delay. subPath mounts never update.

apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: app-db
spec:
  refreshInterval: 1h          # this credential does not rotate, so hourly polling is enough
  secretStoreRef:
    name: aws-secrets-manager
    kind: SecretStore
  target:
    name: app-db
    creationPolicy: Owner      # switch to Merge if another tool already owns this Secret
    template:                  # only the templated key is written to the Secret
      data:
        DATABASE_URL: "postgres://{{ .username }}:{{ .password }}@{{ .host }}/app"
  data:                        # pick individual fields from the JSON secret prod/app/db
    - {secretKey: username, remoteRef: {key: prod/app/db, property: username}}
    - {secretKey: password, remoteRef: {key: prod/app/db, property: password}}
    - {secretKey: host, remoteRef: {key: prod/app/db, property: host}}
---
apiVersion: apps/v1
kind: Deployment
metadata: {name: app}
spec:
  selector: {matchLabels: {app: app}}
  template:
    metadata: {labels: {app: app}}
    spec:
      containers:
        - name: app
          image: registry.example.com/app:1.4.2
          env:                 # read once at container start; a restart is needed to see changes
            - name: DATABASE_URL
              valueFrom: {secretKeyRef: {name: app-db, key: DATABASE_URL}}
          volumeMounts:        # file refreshes after the kubelet sync delay; do not use subPath
            - {name: db-secret, mountPath: /etc/app/db, readOnly: true}
      volumes:
        - name: db-secret
          secret: {secretName: app-db}

An ExternalSecret maps JSON fields from one Secrets Manager entry into a templated DATABASE_URL with an hourly refresh, and a Deployment consumes the result both as an env var fixed at container start and as a mounted file that updates after the kubelet sync delay.

How does secret rotation actually reach running pods?

Rotation reaches a pod in three stages, and each adds latency. ESO polls Secrets Manager on refreshInterval and updates the Kubernetes Secret. The kubelet then propagates the change to mounted volumes on its own sync cycle. Finally, the application has to notice. Worst case, the end-to-end delay is roughly refreshInterval + kubelet sync delay + application reload time. As an illustrative estimate, a 1h refreshInterval, a kubelet delay of about a minute, and a 30-second rolling restart mean pods can run on the old value for over an hour after rotation. Volumes mounted with subPath never receive updates at all.

Reload method How it triggers Catches ESO-driven changes? Cost
Reloader-style controller Watches the Secret and patches the Deployment to force a rollout Yes Full rollout per rotation; one more controller to run
Checksum annotation Hash of Secret content in the pod template, computed at deploy time Only when your pipeline re-renders and re-applies No extra components, but blind to out-of-band rotation
App re-reads mounted file File watch or re-read on auth failure Yes, with no restart Code changes. Watch the directory, because the kubelet swaps a symlink rather than rewriting the file

None of these makes the change atomic across replicas. For a while, some pods will hold the new credential and others the old one. Rotate so that both stay valid through that overlap window. For databases, alternate between two users rather than changing one user's password in place. Secrets Manager keeps the prior version under the AWSPREVIOUS staging label. Revoke it only after the overlap window, which should exceed your worst-case propagation time. Revoking it immediately turns rotation into an outage.

How do you debug SecretSyncedError and other sync failures?

Read the status before the logs

Start with kubectl describe externalsecret <name>. The Ready condition and the Events list usually give the reason directly. SecretSyncedError means ESO reached the provider but could not build the target Secret. A store that isn't ready points to authentication or configuration on the SecretStore. Run kubectl describe on the store as well, because an ExternalSecret inherits its store's failures.

Common provider errors map to a short list of causes:

  • AccessDenied: The role lacks secretsmanager:GetSecretValue on that ARN. Other causes are a customer-managed KMS key that doesn't grant kms:Decrypt, or a trust policy that doesn't match the service account's namespace and name.
  • ResourceNotFound: The key is misspelled, the store points at the wrong region, or the name belongs to another account and needs a full ARN.

To check the target Secret's keys, use kubectl describe secret, which lists key names and byte sizes without values. Avoid -o yaml and jsonpath on .data in shared terminals and CI logs. Base64 is not protection.

Hardening

  • Encryption at rest: Enable KMS envelope encryption for etcd, because synced values become ordinary Secrets.
  • RBAC: Restrict get and list on Secrets. list returns contents too.
  • Alerting: Alert on the controller's sync-error metrics and on Ready=False. Failures are otherwise silent.
  • Multiple replicas: Enable leader election when running more than one controller replica, so replicas don't reconcile the same resources at the same time.
  • GitOps: Exclude generated Secrets from drift detection and pruning so your GitOps tool doesn't delete or overwrite them.

An ESO or AWS outage stalls updates and rotation. Existing Secrets stay in place, so running pods keep working, and new pods that mount those Secrets still start.

When should you avoid ESO, and what are the alternatives?

ESO is a poor fit in three situations:

  • Secrets must never be stored in etcd. ESO's output is a native Kubernetes Secret. Anyone with get on Secrets in that namespace, or access to etcd backups, can read the value.
  • Auditors require per-pod access records. AWS sees one caller: the operator's IAM role. CloudTrail records that ESO read the secret, not which workload consumed it.
  • Small setups. With one cluster and a few static credentials, the controller, its CRDs and its IAM role add more to upgrade, monitor and debug than they save.

The CSI secrets-store driver with the AWS provider mounts values directly into the pod's filesystem using the pod's own service account. That avoids the etcd copy and gives each workload its own audit trail. The cost is that every pod start depends on AWS being reachable. If you enable the driver's optional sync-to-Secret feature, the etcd copy comes back.

Approach Development cost Time saved by central rotation and manifests Operational overhead Main failure mode
ESO Low High Controller, CRDs, one IAM role Stale values when sync fails
CSI driver Low to medium Medium Node DaemonSet, per-pod IAM Pods fail to start during an AWS outage
Direct SDK calls High, repeated in each service Low Minimal infrastructure Each service handles caching and retries differently
Encrypted secrets in Git Low Low, because rotation requires a commit Key management A leaked decryption key exposes the full history

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to external secrets operator aws, this is where it helps:

Saves time

  • Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
  • One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
  • Reviewers see the exact diff and the list of affected services up front, so approval is a decision rather than an investigation.

Saves cost

  • Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
  • Less duplicated configuration to maintain: one shared definition inherited by many services.
  • Per-service approve/reject means a bad shared value can be refused where it would break something, before it is live.
  • OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.