What is the External Secrets Operator and how does it work?
The External Secrets Operator (ESO) is a Kubernetes controller that copies values from an external secret store, such as AWS Secrets Manager or HashiCorp Vault, into ordinary Kubernetes Secrets. The sync runs in one direction only. The external system remains the source of truth, and ESO does not store, version, or rotate anything itself. It does not replace your vault. It delivers the vault's contents to the cluster.
The three core objects
- SecretStore / ClusterSecretStore defines how to reach a provider: endpoint, region, auth method. A
SecretStoreis namespaced and can only be referenced from its own namespace. AClusterSecretStoreis cluster-scoped and usable from any namespace. That makes it convenient for platform teams, but anyone allowed to create an ExternalSecret can use it to read whatever the store's credentials can reach. - ExternalSecret declares what to fetch: which remote keys or properties, how to map them to keys in the target Secret, and how often to refresh.
- The generated Secret is a standard
Secretobject. Pods consume it through env vars or volume mounts with no code changes.
The reconcile loop
The controller watches ExternalSecrets. On each reconcile it resolves the referenced store, authenticates, fetches the values, and creates or updates the target Secret. It repeats this on the configured refresh interval.
Why teams adopt it
Secrets stop appearing in Git, Helm values, and CI variables, and each one has a single authoritative copy. ESO fits platform teams running many services whose apps already expect native Secrets.
The trade-off is that the value still lands in etcd. Its protection there depends on encryption at rest and RBAC on Secrets.
+------------------------------------------+
| External store (source of truth) |
| AWS Secrets Manager / HashiCorp Vault |
+------------------------------------------+
| one-way read by the controller
+-- Kubernetes cluster -------------|----------------------------------+
| | |
| +----------------------+ | +----------------------+ |
| | SecretStore or | | | ExternalSecret | |
| | ClusterSecretStore | | | storeRef, remote | |
| | provider, region, | | | keys -> Secret keys | |
| | auth method | | | refreshInterval | |
| +----------------------+ | +----------------------+ |
| | | | |
| v v v |
| +----------------------------------------------------------------+ |
| | ESO controller (reconcile loop, per ExternalSecret): | |
| | resolve store -> authenticate -> fetch values -> create/update | |
| | target Secret; repeat every refresh interval | |
| +----------------------------------------------------------------+ |
| | writes |
| v |
| +----------------------+ +----------------------+ |
| | Secret (native) | ---------------> | Pod | |
| | stored in etcd | | env vars or volume | |
| | (encryption at rest, | | mounts, no code | |
| | RBAC on Secrets) | | changes | |
| +----------------------+ +----------------------+ |
+----------------------------------------------------------------------+
The ESO controller reads the SecretStore or ClusterSecretStore and the ExternalSecret, fetches values one way from the external source of truth, and writes an ordinary Secret to etcd, where pods consume it without code changes.
Installing the operator with Helm and verifying it is healthy
Install ESO into its own namespace, conventionally external-secrets. A dedicated namespace limits who can read the controller's service account token. That token matters because the controller will hold credentials capable of reading every secret you sync. A separate namespace also lets you apply tighter RBAC and network policies to the operator without affecting application workloads.
Add the project's chart repository with helm repo add, then run helm install with --namespace external-secrets --create-namespace. Chart values change between releases, including those that control whether the chart installs CRDs, resource requests, and which components are enabled. Read the values reference in the project documentation for the release you are installing instead of copying a pinned version or values file from a blog post. Pin the chart version in your own GitOps repository once you have chosen it, so upgrades are deliberate.
Before configuring any provider, confirm two things.
CRDs are registered
Run kubectl get crds and look for entries in the external-secrets.io group, such as externalsecrets, secretstores and clustersecretstores. If they are missing, any manifest referencing those kinds will be rejected. That usually means CRD installation was disabled in your values.
Controller pods are ready
Run kubectl get pods -n external-secrets. With default values you should see three components running and ready: the main controller, the webhook, and the cert controller. The cert controller manages the webhook's TLS certificate. The webhook validates ESO resources on admission. Until it is ready, creating a SecretStore can fail with a webhook connection error that looks like a provider problem but is not one. Wait for all three to report ready before moving on.
# Get the chart repository URL from the External Secrets Operator docs.
# It is a placeholder here so nothing is copied blindly.
ESO_CHART_REPO="<chart-repo-url-from-project-docs>"
# Register the chart repository and refresh the local index.
helm repo add external-secrets "$ESO_CHART_REPO"
helm repo update
# Install into a dedicated namespace so the controller's service account
# and its credentials are isolated from application workloads.
# Check the values reference for your chosen release first, especially the
# value that controls CRD installation. Then pin the chart version in your
# GitOps repo, for example by adding: --version <chosen-chart-version>
helm install external-secrets external-secrets/external-secrets \
--namespace external-secrets \
--create-namespace
# Check 1: the CRDs are registered.
# Expect externalsecrets, secretstores and clustersecretstores in the
# external-secrets.io group. If they are missing, CRD installation was
# probably disabled in your values.
kubectl get crds | grep 'external-secrets.io'
# Check 2: the controller, webhook and cert controller pods are ready.
# Every pod should show READY n/n and STATUS Running. Wait for the webhook
# before you create a SecretStore, or admission can fail with a
# webhook connection error.
kubectl get pods -n external-secrets
Installs External Secrets Operator into its own namespace with Helm, then checks that the external-secrets.io CRDs are registered and the controller, webhook and cert controller pods are ready.
How do you sync secrets from AWS Secrets Manager without static keys?
Credentials should come from the cluster's workload identity mechanism, not from an access key stored in a Kubernetes Secret. On EKS with IRSA, you annotate a service account with eks.amazonaws.com/role-arn. The IAM role's trust policy accepts only that service account's OIDC subject. The SecretStore references the account through auth.jwt.serviceAccountRef, and ESO exchanges its projected token for short-lived AWS credentials. Nothing long-lived exists to leak or rotate.
Scope the role's policy to secretsmanager:GetSecretValue and secretsmanager:DescribeSecret on a name prefix such as arn:aws:secretsmanager:<region>:<account>:secret:prod/payments/*. Secrets Manager appends a random suffix to every ARN, so a trailing wildcard is needed even for a single secret. Set region explicitly on the SecretStore. Otherwise ESO falls back to whatever the environment provides, and lookups against the wrong region fail as "secret not found."
Use an ExternalSecret with data and remoteRef.property to pull one key from a JSON secret. This fits when an app needs only a password. Use dataFrom with extract to expand every JSON key into the target Secret.
refreshInterval controls how often ESO re-reads each secret. Short intervals pick up rotations faster. They also increase calls that are billed per request and count against account-level throttling limits shared with every other client.
Illustrative estimate (all inputs assumed):
- 200
ExternalSecrets - One
GetSecretValuecall per refresh - 30-day month
At 1h, that is 200 × 24 × 30 = 144,000 calls per month. At 5m, it is 1,728,000, twelve times the volume. Start at an hour and shorten the interval only for secrets that rotate on a schedule.
# SecretStore: IRSA-backed auth via an annotated service account, region pinned
apiVersion: external-secrets.io/v1
kind: SecretStore
metadata:
name: aws-payments
namespace: payments
spec:
provider:
aws:
service: SecretsManager
region: us-east-1 # explicit; avoids silent wrong-region lookups
auth:
jwt:
serviceAccountRef:
name: eso-payments # carries eks.amazonaws.com/role-arn
---
# Pull a single key from a JSON secret
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata: {name: payments-db, namespace: payments}
spec:
refreshInterval: 1h # deliberate: each refresh is a billed API call
secretStoreRef: {name: aws-payments, kind: SecretStore}
target: {name: payments-db}
data:
- secretKey: password
remoteRef: {key: prod/payments/db, property: password}
---
# Expand every JSON key of a secret into the target Secret
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata: {name: payments-api, namespace: payments}
spec:
refreshInterval: 1h
secretStoreRef: {name: aws-payments, kind: SecretStore}
target: {name: payments-api}
dataFrom:
- extract: {key: prod/payments/api}
A SecretStore using IRSA with an explicit region, plus ExternalSecrets that pull one JSON key and extract a whole secret on a deliberate 1h refresh.
How do you connect External Secrets Operator to HashiCorp Vault?
Vault's Kubernetes auth method lets ESO authenticate with a service account token instead of a stored Vault token. Vault validates the token against the cluster's TokenReview API, so Vault needs network access to the Kubernetes API server and permission to call it.
On the Vault side, create a role that sets bound_service_account_names and bound_service_account_namespaces to exactly the service account and namespace your SecretStore uses. Attach a policy that grants only read on the paths that workload needs. If the role is broader than that, any pod that can run as a matching service account can read those secrets.
In the SecretStore, set provider.vault.server, the auth mountPath and role, and two fields that are easy to get wrong:
pathis the KV mount name, such assecret.versionis the KV engine version,v1orv2.
KV v2 stores data under a data/ segment, so the API path for app/db is secret/data/app/db. With version: v2, ESO inserts that segment for you. Your ExternalSecret's remoteRef.key should be app/db, and remoteRef.property selects a field such as password. Most "secret not found" errors come from one of these mistakes:
- declaring
v1against a v2 mount - writing
data/into the key yourself - granting the policy on
secret/app/*instead ofsecret/data/app/*
Dynamic secrets are a poor fit. ESO copies values into a Kubernetes Secret on an interval. It does not renew leases and does not revoke old credentials when it fetches new ones. Each refresh can create a new credential while the old lease stays valid until its TTL expires, and pods are never told that a credential has changed. Vault Agent and CSI-style injection are built around the lease lifecycle. Use one of those for short-lived credentials, and use ESO for static KV data.
# Vault side (for reference): role payments-eso is bound to SA payments-eso in
# namespace payments; its policy grants read on secret/data/app/db only
# (KV v2 policies must include the data/ segment).
apiVersion: external-secrets.io/v1
kind: SecretStore
metadata:
name: vault-backend
namespace: payments
spec:
provider:
vault:
server: ""
path: "secret" # KV mount name, not a full path
version: "v2" # must match the mount; ESO adds data/ for v2
auth:
kubernetes:
mountPath: "kubernetes" # where the Kubernetes auth method is enabled
role: "payments-eso" # Vault role bound to this SA and namespace
serviceAccountRef:
name: "payments-eso"
---
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
name: db-credentials
namespace: payments
spec:
refreshInterval: 1h # static copy on an interval; no lease renewal
secretStoreRef:
name: vault-backend
kind: SecretStore
target:
name: db-credentials # Kubernetes Secret that ESO creates
data:
- secretKey: password
remoteRef:
key: app/db # not secret/data/app/db; ESO builds that path
property: password # field inside the KV entry
A SecretStore that authenticates to a KV v2 mount with Vault's Kubernetes auth method, and an ExternalSecret that copies one field from secret/data/app/db into a Kubernetes Secret.
How do rotated secrets reach running pods?
ESO for static credentials that rotate on a schedule. For those, the target.template block builds derived values inside the operator. It renders fetched fields such as username, password and host into a single DATABASE_URL or a complete config file. The application then reads one finished value and never concatenates fields itself. When any source field changes, the rendered output changes with it.
Kubernetes does not restart pods when a Secret changes. How the new value reaches a running pod depends on how the pod consumes it:
| Consumption method | Picks up rotation? | What you need |
|---|---|---|
| Environment variable | No, values are fixed at container start | A rollout |
| Volume mount | Yes, the kubelet updates the file after a delay | The app must reread the file |
subPath volume mount |
No | A rollout |
To force a rollout, put a checksum of the Secret's contents in a pod template annotation, so a changed value produces a new ReplicaSet. Alternatively, run a reloader-style controller that watches Secrets and restarts the workloads that reference them.
Pods roll gradually, so old and new pods run side by side for a while. Rotation is only safe if the provider accepts both the old and the new credential during that window, or if you revoke the old credential only after the rollout finishes.
Lifecycle policies also matter. With creationPolicy: Owner, the default, deleting the ExternalSecret also deletes the Secret, and pods that depend on it fail on their next start. Orphan keeps the Secret but leaves it unmanaged. deletionPolicy controls what happens when the provider-side secret disappears. Retain, the default, keeps the last value, which can be stale or revoked. Delete removes the Secret, which can cause an outage.
How do you debug and monitor External Secrets Operator sync failures?
Start with the status conditions, because the operator records every failed reconcile there. Running kubectl get externalsecret -A shows a Ready column and a status reason for each resource. kubectl describe externalsecret <name> adds the condition message and recent events, which usually contain the provider's own error text. If the message points at authentication, run kubectl describe on the referenced SecretStore or ClusterSecretStore. A store that is not Ready breaks every ExternalSecret that uses it, so check the store before chasing individual secrets.
| Symptom in condition or event | Likely cause | Where to fix |
|---|---|---|
AccessDenied or Vault 403 permission denied |
IAM role or Vault policy lacks read on that path, or the service account binding changed | IAM trust and permissions policy, Vault role and policy |
ResourceNotFound or Vault 404 |
Wrong secret name, property key, or KV mount and version | remoteRef.key and property, store path settings |
| Not found, but the secret exists in the console | Store points at a different region | region on the SecretStore |
ThrottlingException or rate-limit errors |
Many ExternalSecrets with short refresh intervals | Longer refreshInterval, fewer duplicate fetches |
The dangerous failure is the quiet one. When auth breaks, the Kubernetes Secrets that already exist stay in place, so running pods look healthy. Rotations stop arriving, and you only notice when the old credential is revoked. New namespaces and new deployments have no Secret at all and fail at startup.
Don't let a crashing pod be your detector. Scrape the operator's Prometheus metrics and alert on sync errors and on ExternalSecrets whose ready condition stays false. A failing store then shows up as one clear alert minutes after the change that broke it. Without that alert, someone has to work backwards from a CreateContainerConfigError during a release, often days later and with less context.
When should you avoid External Secrets Operator, and what are the alternatives?
ESO's trust boundary is the store, not the ExternalSecret. Anyone who can create or edit an ExternalSecret in a namespace can point it at any path the store's credentials reach and then read the result as an ordinary Secret. A ClusterSecretStore backed by one broad IAM role or Vault policy therefore exposes everything to every namespace allowed to reference it.
Prefer per-namespace SecretStores, each with its own IAM role or Vault role limited to that team's path prefix. In RBAC, treat write access to ExternalSecrets as equivalent to read access on the backend. Reserve ClusterSecretStores for shared values, and restrict which namespaces may use them. The cost is more stores, roles and policies to keep consistent. Provision them automatically alongside each namespace, or teams will start reusing broad roles to save time.
Cases where ESO is the wrong choice
- Policy forbids secrets in etcd. ESO always writes a Kubernetes Secret.
- Most credentials are dynamic. Short-lived leases get copied into static Secrets that outlive the lease.
- Nobody watches sync status. Failures then stay silent.
| Approach | Stored in etcd | App changes | Rotation | Operational cost |
|---|---|---|---|---|
| ESO | Yes | None | Periodic refresh; pod must reread | One controller |
| Secrets Store CSI driver | Only if sync is enabled | Read files | Optional remount polling | Driver plus provider per node |
| Vault Agent injection | No | Read files | Agent re-renders templates | Sidecar per pod |
| Git/CI | Yes | None | Manual commit or pipeline run | Low tooling, high manual effort |
ESO adds one component to operate. In return, it removes per-app secret-fetching code and the manual copying of values between consoles and clusters.
How OneEnv can help
Disclosure: OneEnv is the product behind this blog.
OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.
Applied to external secrets operator, this is where it helps:
Saves time
- Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
- One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
- Reviewers see the exact diff and the list of affected services up front, so approval is a decision rather than an investigation.
Saves cost
- Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
- Less duplicated configuration to maintain: one shared definition inherited by many services.
- Per-service approve/reject means a bad shared value can be refused where it would break something, before it is live.
- OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.
These are qualitative benefits, not measured results — how much you save depends on how many services share the value.
Want to try it? Start at oneenv.io.