What is the External Secrets Operator and how does it sync AWS Secrets Manager?
The External Secrets Operator (ESO) is a Kubernetes controller, not a secret store. It holds nothing on its own. AWS Secrets Manager remains the source of truth, and ESO runs a reconciliation loop that copies values from it into native Kubernetes Secrets.
The loop has three parts:
- ExternalSecret resources: The controller watches these. Each one declares which remote secret (or which JSON keys inside it) to fetch, and which Kubernetes Secret to write.
- SecretStore or ClusterSecretStore: The ExternalSecret references one of these. It tells ESO how to reach AWS: the region, the service, and the identity to authenticate as.
- Refresh schedule: On every refresh interval, ESO fetches the current value and updates the target Secret if it has changed.
Your pods never call AWS. They read an ordinary Secret through environment variables or a mounted volume, the same way they would read one created with kubectl. Application code needs no AWS SDK, no IAM logic and no retry handling for Secrets Manager outages.
That design has a cost. The synced Secret is a full copy of the value, held in etcd. It is only as protected as your etcd encryption at rest and your RBAC rules on get, list and watch for Secrets. If someone can read Secrets in a namespace, they can read the AWS-managed value.
Teams adopt ESO for three reasons:
- One authoritative place to change a credential.
- No application changes.
- No static AWS keys or encoded secrets committed to Git.
It fits backend and platform engineers running workloads on EKS, or on other clusters, whose organizations have already standardized on Secrets Manager.
+----------------------------+
| AWS Secrets Manager |
| (source of truth) |
+-------------+--------------+
^
| fetch on every refresh interval
===================================|========== Kubernetes cluster ==========
|
+----------------+ +-------+--------+ +--------------------+
| ExternalSecret |<--------| ESO controller |-------->| SecretStore or |
| remote key, | watches | reconciliation | reads | ClusterSecretStore |
| JSON keys, | | loop | | region, service, |
| target Secret | | | | auth identity |
+----------------+ +-------+--------+ +--------------------+
| writes Secret; updates it if changed
v
+----------------------+ +------------------+
| Kubernetes Secret |------->| Pod |
| (full copy in etcd) | env or | no AWS calls |
| | volume | reads Secret |
+----------------------+ +------------------+
The copy is protected only by etcd encryption at rest and by RBAC
on Secrets (get, list, watch).
ESO's controller watches ExternalSecret resources, uses a SecretStore or ClusterSecretStore to fetch values from AWS Secrets Manager on each refresh interval, and writes a native Kubernetes Secret that pods read without ever calling AWS.
How does ESO work, step by step?
Each reconciliation of an ExternalSecret follows the same sequence:
- Admission. When you apply an ExternalSecret or store, ESO's validating webhook checks the resource against the CRD schema and rejects malformed specs before they reach the controller.
- Store resolution. The controller reads the ExternalSecret's
secretStoreRefand loads the referenced SecretStore (same namespace) or ClusterSecretStore (cluster-scoped, subject to itsconditions). If the store is not ready, the ExternalSecret inherits that failure. - Authentication. Using the store's
authblock, ESO obtains AWS credentials. With IRSA, it uses a token for the referenced service account to assume the annotated IAM role through STS, which returns short-lived credentials. With Pod Identity, the credentials come from the EKS Pod Identity association. - Fetch. ESO calls
GetSecretValuefor eachremoteRef.key. For JSON secrets it extracts the requestedproperty, or every top-level key whendataFrom.extractis used. - Render. If
target.templateis set, ESO renders the fetched values into the final keys with Go templating. - Write. ESO creates or updates the target Kubernetes Secret according to
creationPolicy. WithOwner, it sets an owner reference so the Secret is garbage-collected with the ExternalSecret. - Status and requeue. ESO sets the
Readycondition and emits events describing success or failure, then schedules the next reconciliation afterrefreshInterval.
The kubelet and your application handle everything after step 6. ESO's job ends once the Kubernetes Secret matches the value in AWS.
Who should use the External Secrets Operator?
ESO suits teams that meet most of these conditions:
- Secrets Manager is already the system of record. Security or platform teams create, rotate and audit credentials in AWS, and Kubernetes workloads need to consume them.
- Several services, namespaces or clusters consume the same credentials. One rotation in AWS should reach all of them without anyone running
kubectl create secretby hand. - Deployments are managed with GitOps or reviewed manifests. ExternalSecret resources contain only references and templates, so they can live in Git and go through code review without exposing values.
- Applications already read configuration from env vars or files. ESO adds nothing to application code, so it fits existing services, including third-party images you cannot modify.
- Kubernetes Secrets in etcd are acceptable under your threat model, because etcd encryption at rest and Secret RBAC are already in place.
In practice, the main users are platform engineers, who install the controller, define stores and IAM roles, and set guardrails, and application engineers, who write ExternalSecret manifests for their own services. Teams that cannot accept an etcd copy, need per-pod audit records, or run a single small cluster with a few static credentials should read the last section before adopting it.
How can ESO save engineering time?
ESO removes repeated, manual work in a few specific places:
- No per-service secret-loading code. Without ESO, each service either embeds an AWS SDK client with caching, retries and IAM handling, or relies on a person or pipeline to copy values into Kubernetes. ESO does this once, in one controller.
- Rotation happens in one place. Changing a value in Secrets Manager propagates to every cluster and namespace that references it on the next refresh. Nobody has to find and update each copy.
- Onboarding is one manifest. Giving a new service a credential means writing a short ExternalSecret and, at most, extending an IAM policy prefix. No pipeline changes and no out-of-band
kubectlsteps. - Reviewable, reproducible environments. Because manifests contain references rather than values, rebuilding a cluster or creating a new environment means reapplying Git, not reconstructing secrets by hand.
- Faster diagnosis. Sync state appears as
Readyconditions and events on Kubernetes resources, which is quicker to inspect than tracing a failed CI step or a missing manual update.
As an illustrative estimate with assumed inputs: suppose 15 credentials are rotated quarterly across 3 clusters, and a manual update (fetching the value, updating the Secret, verifying the rollout) takes an assumed 10 minutes per credential per cluster. That is 45 updates, or about 450 minutes of hands-on work each quarter, plus the risk of missing a cluster. With ESO, the same rotation is a change in Secrets Manager, and the remaining work is the reload step described in the rotation section below. Your own numbers depend on how many credentials you have, how often they rotate, and how automated your current process already is. Weigh the savings against the overhead of running ESO: upgrading the controller and CRDs, maintaining IAM roles, and alerting on sync failures.
How do you install ESO and authenticate to AWS without static keys?
Install the chart from the project's official Helm repository into its own namespace, conventionally external-secrets, with --create-namespace. A dedicated namespace keeps the controller's service account, RBAC, and webhook isolated from application workloads and makes it easy to scope who can modify them.
ESO's API has moved between versions, and snippets online often target an older one. Before applying any manifest, confirm what your cluster actually serves: kubectl get crds | grep external-secrets.io, then kubectl get crd secretstores.external-secrets.io -o jsonpath='{.spec.versions[?(@.served==true)].name}'. Repeat this for clustersecretstores and externalsecrets. Then check the controller with kubectl logs -n external-secrets deploy/external-secrets for webhook certificate or RBAC errors.
Authenticate with workload identity
On EKS, use IAM Roles for Service Accounts (IRSA) or EKS Pod Identity so the controller gets short-lived credentials instead of access keys stored in a Kubernetes Secret. With IRSA, annotate the service account the store references with eks.amazonaws.com/role-arn, and restrict the role's trust policy to that exact namespace and service account name. With Pod Identity, create an association between that service account and the role. Either way, nothing long-lived needs rotating or protecting.
Least-privilege policy
Grant only secretsmanager:GetSecretValue and secretsmanager:DescribeSecret, scoped to specific secret ARNs or a name prefix such as prod/payments/*. A trailing wildcard is needed in either case because Secrets Manager appends a random suffix to every ARN. If secrets use a customer-managed KMS key, also allow kms:Decrypt on that key's ARN. Without it, reads fail with an access error even though the Secrets Manager permissions are correct.
cat > eso-secretsmanager-read.json <<'EOF'
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "ReadPaymentsSecretsOnly",
"Effect": "Allow",
"Action": [
"secretsmanager:GetSecretValue",
"secretsmanager:DescribeSecret"
],
"Resource": "arn:aws:secretsmanager:us-east-1:111122223333:secret:prod/payments/*"
},
{
"Sid": "DecryptWithCustomerManagedKeyViaSecretsManager",
"Effect": "Allow",
"Action": "kms:Decrypt",
"Resource": "arn:aws:kms:us-east-1:111122223333:key/1234abcd-12ab-34cd-56ef-1234567890ab",
"Condition": {
"StringEquals": {
"kms:ViaService": "secretsmanager.us-east-1.amazonaws.com"
}
}
}
]
}
EOF
aws iam create-policy \
--policy-name eso-prod-payments-read \
--policy-document file://eso-secretsmanager-read.json
aws iam attach-role-policy \
--role-name eso-prod-payments \
--policy-arn arn:aws:iam::111122223333:policy/eso-prod-payments-read
Creates and attaches an IAM policy that allows only GetSecretValue and DescribeSecret on the prod/payments/ prefix. It also allows kms:Decrypt on one customer-managed key, but only when the call comes through Secrets Manager.
Should you use a SecretStore or a ClusterSecretStore?
A SecretStore is namespaced, so an ExternalSecret can only reference a store in its own namespace. A ClusterSecretStore is cluster-scoped, and any namespace can reference it unless you restrict it. The choice matters less for convenience than for the IAM role behind the store. ESO fetches whatever an ExternalSecret asks for using that role's permissions. It does not judge whether the requesting namespace should see a given secret. A broadly scoped role behind an unrestricted ClusterSecretStore means anyone who can create an ExternalSecret in any namespace can read every secret that role can reach.
| Aspect | SecretStore | ClusterSecretStore |
|---|---|---|
| Scope | One namespace | Whole cluster, optionally limited by conditions |
| IAM boundary | One role per team or namespace | One role shared across referencing namespaces |
| Management overhead | A store and role per namespace | One store, one role |
| Blast radius | Limited to that team's secret paths | Everything the role can read, in every allowed namespace |
| Good fit | Application and team secrets | Platform-owned secrets such as shared registry credentials |
Default to per-team IAM roles with namespaced stores. Scope each role's policy to a path prefix such as payments/* so the IAM policy enforces the namespace boundary. Use a ClusterSecretStore only for secrets the platform team owns. Restrict it with spec.conditions (a namespaceSelector or an explicit namespaces list) so only approved namespaces can reference it.
The manifest below defines a namespaced SecretStore for a payments namespace. It authenticates through a service account annotated with that team's role.
apiVersion: v1
kind: ServiceAccount
metadata:
name: payments-eso
namespace: payments
annotations:
eks.amazonaws.com/role-arn: arn:aws:iam::111122223333:role/eso-payments-reader
---
apiVersion: external-secrets.io/v1
kind: SecretStore
metadata:
name: aws-secrets-manager
namespace: payments
spec:
provider:
aws:
service: SecretsManager
region: us-east-1
auth:
jwt:
serviceAccountRef:
# Resolved in the store's own namespace (payments)
name: payments-eso
A namespaced SecretStore for the payments namespace that authenticates to AWS Secrets Manager through an IRSA-annotated service account, whose IAM role is limited to the payments/ secret prefix.*
How do you write an ExternalSecret and consume it in a Deployment?
An ExternalSecret references the store through secretStoreRef and declares what to copy. Use data for individual fields: each entry sets a secretKey (the key in the Kubernetes Secret) and a remoteRef with the AWS secret name as key and, for JSON secrets, a property that selects one field. Use dataFrom with extract to pull every top-level JSON key at once. This is convenient, but any new key added in AWS appears in the pod without review. When an application expects a single value such as a connection string, target.template renders one with Go templating, for example postgres://{{ .username }}:{{ .password }}@{{ .host }}/app. This keeps the components in AWS and the assembly in Git.
Set refreshInterval for each secret based on how quickly changes must arrive. As an illustrative estimate with assumed inputs, 300 ExternalSecrets at 1m make about 432,000 GetSecretValue calls per day. At 1h they make about 7,200. Secrets Manager bills per API call and enforces request-rate quotas, so a short interval should be reserved for credentials that actually rotate.
target.creationPolicy defaults to Owner: ESO creates the Secret and adds an owner reference, so deleting the ExternalSecret also deletes the Secret. If another tool already manages that Secret, use Merge so ESO only adds keys to it, or use Orphan so the Secret stays after ESO stops managing it. deletionPolicy (Retain, Delete, Merge) controls what happens when the secret in AWS disappears.
Consume the Secret through envFrom/secretKeyRef or a volume. Environment variables are read only when the container starts. Mounted volumes pick up changes after the kubelet's sync delay. subPath mounts never update.
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
name: app-db
spec:
refreshInterval: 1h # this credential does not rotate, so hourly polling is enough
secretStoreRef:
name: aws-secrets-manager
kind: SecretStore
target:
name: app-db
creationPolicy: Owner # switch to Merge if another tool already owns this Secret
template: # only the templated key is written to the Secret
data:
DATABASE_URL: "postgres://{{ .username }}:{{ .password }}@{{ .host }}/app"
data: # pick individual fields from the JSON secret prod/app/db
- {secretKey: username, remoteRef: {key: prod/app/db, property: username}}
- {secretKey: password, remoteRef: {key: prod/app/db, property: password}}
- {secretKey: host, remoteRef: {key: prod/app/db, property: host}}
---
apiVersion: apps/v1
kind: Deployment
metadata: {name: app}
spec:
selector: {matchLabels: {app: app}}
template:
metadata: {labels: {app: app}}
spec:
containers:
- name: app
image: registry.example.com/app:1.4.2
env: # read once at container start; a restart is needed to see changes
- name: DATABASE_URL
valueFrom: {secretKeyRef: {name: app-db, key: DATABASE_URL}}
volumeMounts: # file refreshes after the kubelet sync delay; do not use subPath
- {name: db-secret, mountPath: /etc/app/db, readOnly: true}
volumes:
- name: db-secret
secret: {secretName: app-db}
An ExternalSecret maps JSON fields from one Secrets Manager entry into a templated DATABASE_URL with an hourly refresh, and a Deployment consumes the result both as an env var fixed at container start and as a mounted file that updates after the kubelet sync delay.
How does secret rotation actually reach running pods?
Rotation reaches a pod in three stages, and each adds latency. ESO polls Secrets Manager on refreshInterval and updates the Kubernetes Secret. The kubelet then propagates the change to mounted volumes on its own sync cycle. Finally, the application has to notice. Worst case, the end-to-end delay is roughly refreshInterval + kubelet sync delay + application reload time. As an illustrative estimate, a 1h refreshInterval, a kubelet delay of about a minute, and a 30-second rolling restart mean pods can run on the old value for over an hour after rotation. Volumes mounted with subPath never receive updates at all.
| Reload method | How it triggers | Catches ESO-driven changes? | Cost |
|---|---|---|---|
| Reloader-style controller | Watches the Secret and patches the Deployment to force a rollout | Yes | Full rollout per rotation; one more controller to run |
| Checksum annotation | Hash of Secret content in the pod template, computed at deploy time | Only when your pipeline re-renders and re-applies | No extra components, but blind to out-of-band rotation |
| App re-reads mounted file | File watch or re-read on auth failure | Yes, with no restart | Code changes. Watch the directory, because the kubelet swaps a symlink rather than rewriting the file |
None of these makes the change atomic across replicas. For a while, some pods will hold the new credential and others the old one. Rotate so that both stay valid through that overlap window. For databases, alternate between two users rather than changing one user's password in place. Secrets Manager keeps the prior version under the AWSPREVIOUS staging label. Revoke it only after the overlap window, which should exceed your worst-case propagation time. Revoking it immediately turns rotation into an outage.
How do you debug SecretSyncedError and other sync failures?
Read the status before the logs
Start with kubectl describe externalsecret <name>. The Ready condition and the Events list usually give the reason directly. SecretSyncedError means ESO reached the provider but could not build the target Secret. A store that isn't ready points to authentication or configuration on the SecretStore. Run kubectl describe on the store as well, because an ExternalSecret inherits its store's failures.
Common provider errors map to a short list of causes:
AccessDenied: The role lackssecretsmanager:GetSecretValueon that ARN. Other causes are a customer-managed KMS key that doesn't grantkms:Decrypt, or a trust policy that doesn't match the service account's namespace and name.ResourceNotFound: The key is misspelled, the store points at the wrong region, or the name belongs to another account and needs a full ARN.
To check the target Secret's keys, use kubectl describe secret, which lists key names and byte sizes without values. Avoid -o yaml and jsonpath on .data in shared terminals and CI logs. Base64 is not protection.
Hardening
- Encryption at rest: Enable KMS envelope encryption for etcd, because synced values become ordinary Secrets.
- RBAC: Restrict
getandliston Secrets.listreturns contents too. - Alerting: Alert on the controller's sync-error metrics and on
Ready=False. Failures are otherwise silent. - Multiple replicas: Enable leader election when running more than one controller replica, so replicas don't reconcile the same resources at the same time.
- GitOps: Exclude generated Secrets from drift detection and pruning so your GitOps tool doesn't delete or overwrite them.
An ESO or AWS outage stalls updates and rotation. Existing Secrets stay in place, so running pods keep working, and new pods that mount those Secrets still start.
When should you avoid ESO, and what are the alternatives?
ESO is a poor fit in three situations:
- Secrets must never be stored in etcd. ESO's output is a native Kubernetes Secret. Anyone with
geton Secrets in that namespace, or access to etcd backups, can read the value. - Auditors require per-pod access records. AWS sees one caller: the operator's IAM role. CloudTrail records that ESO read the secret, not which workload consumed it.
- Small setups. With one cluster and a few static credentials, the controller, its CRDs and its IAM role add more to upgrade, monitor and debug than they save.
The CSI secrets-store driver with the AWS provider mounts values directly into the pod's filesystem using the pod's own service account. That avoids the etcd copy and gives each workload its own audit trail. The cost is that every pod start depends on AWS being reachable. If you enable the driver's optional sync-to-Secret feature, the etcd copy comes back.
| Approach | Development cost | Time saved by central rotation and manifests | Operational overhead | Main failure mode |
|---|---|---|---|---|
| ESO | Low | High | Controller, CRDs, one IAM role | Stale values when sync fails |
| CSI driver | Low to medium | Medium | Node DaemonSet, per-pod IAM | Pods fail to start during an AWS outage |
| Direct SDK calls | High, repeated in each service | Low | Minimal infrastructure | Each service handles caching and retries differently |
| Encrypted secrets in Git | Low | Low, because rotation requires a commit | Key management | A leaked decryption key exposes the full history |
How OneEnv can help
Disclosure: OneEnv is the product behind this blog.
OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.
Applied to external secrets operator aws, this is where it helps:
Saves time
- Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
- One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
- Reviewers see the exact diff and the list of affected services up front, so approval is a decision rather than an investigation.
Saves cost
- Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
- Less duplicated configuration to maintain: one shared definition inherited by many services.
- Per-service approve/reject means a bad shared value can be refused where it would break something, before it is live.
- OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.
These are qualitative benefits, not measured results — how much you save depends on how many services share the value.
Want to try it? Start at oneenv.io.