External Secrets Operator AWS Tutorial: Secrets Manager on Kubernetes

What External Secrets Operator does and why native Kubernetes Secrets fall short

A Kubernetes Secret is a convenience object, not a security boundary. Its values are base64-encoded, which is an encoding rather than encryption. Anyone who can get Secrets in a namespace can read them, and anyone who can create pods there can mount them. Whether the data is encrypted in etcd depends on how the cluster's encryption at rest is configured, and that setting is owned by whoever runs the control plane, not by the application team.

The larger gap is lifecycle. A Secret has no notion of where its value came from or how it should change. Values get applied from a laptop, a CI variable, or a sealed file in Git, and after a few environments nobody can say which copy is authoritative. Rotation becomes a manual hunt across clusters.

External Secrets Operator (ESO) is a controller that closes that gap. It reads values from an external provider, here AWS Secrets Manager, and writes them into ordinary Kubernetes Secrets. Pods consume those Secrets exactly as before, through environment variables or volume mounts, so application manifests and code do not change.

This tutorial treats AWS as the single source of truth and the in-cluster Secret as a cache. That framing matters because three things interact. IAM scoping decides what each cluster or namespace may read. Refresh behavior decides how quickly a change propagates. Workload reload decides when a running process actually uses the new value. If you design any one of these in isolation, rotation breaks.

ESO fits best for platform and backend teams running several services or environments on EKS, where secrets already live in AWS and consistency across clusters matters more than per-app customization.

How it works: SecretStore, ClusterSecretStore, ExternalSecret and the target Secret

Four objects make up the model, and each has a different owner. A SecretStore describes how to reach a provider: which AWS region, which service (Secrets Manager or Parameter Store), and which identity to authenticate with. A ClusterSecretStore contains the same information but is cluster-scoped, so any namespace can reference it. An ExternalSecret lives in an application namespace. It names a store, lists which remote keys to read, and declares the target Secret that the operator should create and keep in sync. Pods consume that target Secret the same way they consume any other Secret, so workloads never call AWS themselves.

CopyAWS Secrets Manager
        │  GetSecretValue (on each refresh)
        ▼
External Secrets Operator ◄── reads ExternalSecret + SecretStore
        │  create / update
        ▼
Kubernetes Secret (target)
        │  env vars or volume mount
        ▼
       Pod

This split matters for availability. The operator is on the critical path for new and changed secrets. If it is down, a new ExternalSecret produces nothing, and a rotated value in AWS never reaches the cluster. Secrets that already exist are ordinary API objects, so they persist, and pods keep starting with the last synced values. An operator outage therefore causes stale data, not an immediate service failure.

Copy┌──────────────────────────────┐
│     AWS Secrets Manager      │
└──────────────┬───────────────┘
               │ GetSecretValue (each refresh)
               ▼
┌──────────────────────────────┐   reads   ┌─────────────────────────────┐
│  External Secrets Operator   │◄──────────┤ SecretStore (one namespace) │
│  syncs on refresh interval   │           │  or ClusterSecretStore      │
│                              │           │ (whole cluster): region,    │
│                              │           │ service, IAM identity       │
│                              │           └─────────────────────────────┘
│                              │   reads   ┌─────────────────────────────┐
│                              │◄──────────┤ ExternalSecret (app ns):    │
│                              │           │ store ref, remote keys,     │
│                              │           │ target Secret name          │
└──────────────┬───────────────┘           └─────────────────────────────┘
               │ create / update (critical path: stops if operator down)
               ▼
┌──────────────────────────────┐
│  Kubernetes Secret (target)  │
│  ordinary API object: keeps  │
│  last synced values if the   │
│  operator is down            │
└──────────────┬───────────────┘
               │ env vars or volume mount
               ▼
┌──────────────────────────────┐
│ Pod (never calls AWS itself) │
└──────────────────────────────┘

The operator reads an ExternalSecret and its SecretStore or ClusterSecretStore, fetches values from AWS Secrets Manager, and writes a target Secret that pods consume and that persists with last-synced values if the operator goes down.

ESO setup on EKS: Helm install and keyless authentication with IAM

Install the operator with its official Helm chart into a dedicated namespace such as external-secrets. Pin the chart version you tested in staging rather than tracking the latest release, because CRD changes between releases can break existing manifests. Install values and CRD handling change over time, so take them from the project's current documentation, not from older blog posts.

Keyless authentication

The operator needs AWS credentials, and they should not be long-lived. Storing an access key pair in a Kubernetes Secret recreates the problem ESO is meant to solve: a static credential that anyone with read access to that namespace can copy, that never expires, and that you must rotate by hand.

On EKS, use IAM Roles for Service Accounts (IRSA), or EKS Pod Identity where it is available. With IRSA, annotate a service account with eks.amazonaws.com/role-arn and point the SecretStore's auth block at that service account. With Pod Identity, create an association between the role and the service account. In both cases AWS issues short-lived credentials that are scoped to that identity.

Least-privilege policy

Grant secretsmanager:GetSecretValue only on specific ARNs or a prefix such as prod/payments/*. Add secretsmanager:DescribeSecret only if your setup needs it. If a secret is encrypted with a customer-managed KMS key, also allow kms:Decrypt on that key.

Create one role per namespace or team. This contains the damage if a single SecretStore is misconfigured.

Clusters outside EKS

On other clusters, configure OIDC federation between the cluster's service account issuer and IAM. If you fall back to static credentials, the cluster is back to holding a long-lived key, and that weakens the whole security model.

Copy# Least-privilege policy for one team's SecretStore (prod/payments).
# Placeholder account ID, region and KMS key ID; replace with your own.
cat > payments-eso-policy.json <<'EOF'
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "ReadPaymentsSecretsOnly",
      "Effect": "Allow",
      "Action": [
        "secretsmanager:GetSecretValue",
        "secretsmanager:DescribeSecret"
      ],
      "Resource": "arn:aws:secretsmanager:eu-west-1:111122223333:secret:prod/payments/*"
    },
    {
      "Sid": "DecryptWithCustomerManagedKeyViaSecretsManager",
      "Effect": "Allow",
      "Action": "kms:Decrypt",
      "Resource": "arn:aws:kms:eu-west-1:111122223333:key/1234abcd-12ab-34cd-56ef-1234567890ab",
      "Condition": {
        "StringEquals": {
          "kms:ViaService": "secretsmanager.eu-west-1.amazonaws.com"
        }
      }
    }
  ]
}
EOF

# Create the policy. Attach it to the payments team's role only, which IRSA or
# Pod Identity binds to the service account that the SecretStore references.
aws iam create-policy \
  --policy-name eso-payments-read \
  --policy-document file://payments-eso-policy.json

This example creates an IAM policy for one team that allows reading only the Secrets Manager secrets under prod/payments/ and decrypting with that team's customer-managed KMS key when the request comes through Secrets Manager. Drop DescribeSecret if your setup does not need it.*

Writing ExternalSecrets: key extraction, templating and ownership policies

The simplest mapping is dataFrom with extract. Point it at a Secrets Manager secret that stores a JSON object, and each top-level key becomes a key in the Kubernetes Secret. This works well when the AWS secret was designed for one consumer. When it wasn't, use data entries with remoteRef.key and remoteRef.property to pull only the fields the workload needs. Extracting only those fields keeps unrelated credentials out of the namespace and makes the manifest show exactly what the pod depends on.

Applications often expect a single DATABASE_URL or a rendered config file rather than separate fields. target.template handles this. It uses Go templates to combine extracted values into new keys, so you can build postgres://{{ .username }}:{{ .password }}@... without a sidecar or init script. Keep templates small, because a typo renders a broken value that still syncs successfully.

creationPolicy decides who owns the target Secret. Owner, the default, sets an owner reference, so deleting the ExternalSecret garbage-collects the Secret. Orphan leaves the Secret in place. Merge writes into an existing Secret.

deletionPolicy covers the AWS side. With Retain, the default, a deleted or renamed AWS secret causes sync errors while pods keep running on the last values, which may be stale. With Delete, the target Secret is removed, and pods that restart afterward fail to start.

Run each of these paths in staging before production: delete the ExternalSecret, delete the AWS secret, and rename the AWS secret. Then confirm the cluster behaves the way you expect.

CopyapiVersion: external-secrets.io/v1   # use v1beta1 on older ESO releases
kind: ExternalSecret
metadata:
  name: orders-db
  namespace: orders
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: aws-secrets-manager    # ClusterSecretStore configured for Secrets Manager
    kind: ClusterSecretStore
  target:
    name: orders-db              # Kubernetes Secret that ESO creates
    creationPolicy: Owner        # default: Secret is garbage-collected with this ExternalSecret
    deletionPolicy: Retain       # default: keep last values if the AWS secret disappears
    template:
      engineVersion: v2
      data:
        # Build one value the app expects from the extracted fields.
        # A typo here still syncs, so check the rendered Secret.
        DATABASE_URL: "postgres://{{ .username }}:{{ .password }}@{{ .host }}:5432/orders"
  # Pull only the fields this workload needs from the JSON secret,
  # not every credential stored in it.
  data:
    - secretKey: username
      remoteRef:
        key: prod/orders/db      # Secrets Manager secret name
        property: username       # top-level key in the JSON value
    - secretKey: password
      remoteRef:
        key: prod/orders/db
        property: password
    - secretKey: host
      remoteRef:
        key: prod/orders/db
        property: host
  # Alternative for a single-consumer secret: map every top-level JSON key.
  # dataFrom:
  #   - extract:
  #       key: prod/orders/db

An ExternalSecret that extracts selected JSON fields from a Secrets Manager secret, renders them into a DATABASE_URL with a template, and sets creationPolicy and deletionPolicy explicitly, with the whole-secret dataFrom mapping shown as a commented alternative.

Secret rotation end to end: refreshInterval, overlap windows and pod reloads

Rotation happens in AWS Secrets Manager, not in the cluster. ESO only notices the change on its next sync, so refreshInterval is the upper bound on how stale the Kubernetes Secret can be. Pods are a separate step. Environment variables are fixed when the container starts. Mounted Secret volumes are updated by the kubelet only after its sync delay, and subPath mounts never update.

The common outage comes from invalidating the old credential before every pod has the new one. Keep both credentials valid for an overlap window of at least refreshInterval plus the worst-case reload time, and then add margin for failed syncs.

Illustrative estimate. This example makes three assumptions: 200 ExternalSecrets, one GetSecretValue call per refresh, and $0.05 per 10,000 API calls (check current pricing). With a 1-minute interval, that comes to about 8.6M calls and about $43 per month. With a 1-hour interval, it comes to about 144K calls and under $1 per month. In large clusters, short intervals also push request rates toward Secrets Manager throttling limits. Throttled calls cause sync failures, and those failures affect every tenant that shares the same IAM role.

Copy# Rotation happens in AWS Secrets Manager; ESO re-reads it on this schedule.
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: orders-db
  namespace: orders
spec:
  refreshInterval: 15m        # worst-case staleness of the Kubernetes Secret
  secretStoreRef:
    name: aws-secrets-manager
    kind: ClusterSecretStore
  target:
    name: orders-db           # Kubernetes Secret that ESO creates and updates
    creationPolicy: Owner
  dataFrom:
    - extract:
        key: prod/orders/db   # JSON secret, e.g. {"username":...,"password":...}
---
# Mount the Secret as a volume without subPath so the kubelet can update files.
# The app must re-read /etc/db-creds on change or on a timer. Env vars would stay
# fixed until the pod restarts.
apiVersion: apps/v1
kind: Deployment
metadata:
  name: orders-api
  namespace: orders
spec:
  selector:
    matchLabels: { app: orders-api }
  template:
    metadata:
      labels: { app: orders-api }
    spec:
      containers:
        - name: api
          image: registry.example.com/orders-api:1.4.2
          volumeMounts:
            - { name: db-creds, mountPath: /etc/db-creds, readOnly: true }
      volumes:
        - name: db-creds
          secret: { secretName: orders-db }

This example pairs an ExternalSecret that syncs every 15 minutes with a Deployment that mounts the Secret as a volume without subPath, so an application that re-reads its files picks up rotated credentials without a restart.

Observability, multi-tenant layout and a pre-production checklist

That shared blast radius is why visibility comes first.

Reading failures

kubectl describe externalsecret <name> shows a Ready condition and recent events. Most failures look like one of these:

• AccessDeniedException: the IAM role lacks secretsmanager:GetSecretValue on that ARN.
• ResourceNotFoundException: the key name or region is wrong.
• A KMS access error: the secret uses a customer-managed key, and the role cannot call kms:Decrypt.

When a sync fails, the target Secret keeps its last good value. That makes failures quiet, so you need alerts rather than waiting for someone to notice.

Metrics and availability

Enable the operator's Prometheus metrics. Alert on two signals: sync errors rising, or any ExternalSecret staying not-ready for longer than a couple of refresh intervals.

Run two or more replicas with leader election. Only one replica reconciles at a time, but a standby shortens failover. Set CPU and memory requests so node pressure does not evict the operator.

Multi-tenant layout

Name secrets as env/team/app/purpose. This lets IAM policies grant access by path prefix instead of listing individual secrets.

• Per-namespace SecretStores give each team its own IAM role and keep failures local to that team.
• A shared ClusterSecretStore is simpler to run, but it centralizes privilege.

If you use a ClusterSecretStore, add a validating admission policy that limits which stores each namespace may reference. Keep ExternalSecret manifests in Git. They contain references, not values, so code review covers access changes.

Pre-production checklist

• IAM can read only the intended secret paths.
• A rotation drill ends with pods running on the new credential.
• A simulated permission denial or outage triggers alerts and leaves existing Secrets in place.
• etcd encryption at rest and RBAC restrictions on Secrets are confirmed.

When to avoid ESO, alternatives, and impact on cost and engineering time

ESO changes where secrets originate, not where they end up. Every ExternalSecret still produces an ordinary Kubernetes Secret stored in etcd. Anyone who can get secrets in a namespace, or exec into a pod that mounts one, can read the value. etcd encryption at rest, tight RBAC and exec restrictions remain necessary.

ESO adds its own operating cost. You run an extra controller and its CRDs, and you have to follow its upgrade cycle. Some upgrades move CRDs to a new API version, so you must migrate stored manifests before the old version is removed. A small cluster with a few static credentials is often better served by native Secrets created from CI.

ESO saves engineering time in several ways:

• Secrets Manager becomes the single source of truth.
• Manifests can be reviewed in Git without exposing values.
• Engineers stop copying the same secret between namespaces and clusters.
• Rotation follows the same path for every service.

Against those gains, weigh Secrets Manager API charges and controller maintenance. API calls scale roughly as ExternalSecrets × (refreshes per day), so long refresh intervals and shared ClusterSecretStores keep the call count down.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to external secrets operator aws, this is where it helps:

Saves time

• Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
• One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
• Reviewers see the exact diff and the list of affected services up front, so approval is a decision rather than an investigation.

Saves cost

• Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
• Less duplicated configuration to maintain: one shared definition inherited by many services.
• Per-service approve/reject means a bad shared value can be refused where it would break something, before it is live.
• OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.