Why aren't native Kubernetes Secrets enough for GitOps?
A Kubernetes Secret looks like it should be safe to commit, but it isn't. The values under data are base64-encoded, which is an encoding for carrying arbitrary bytes in YAML, not a security control. Anyone who can read the manifest can recover the plaintext with base64 -d. The stringData field doesn't even do that much; it holds the value in clear text.
That is a problem for GitOps, where the repository is supposed to describe the entire desired state of the cluster. Once a Secret manifest is pushed, the value lives in every clone, every fork, every CI checkout, and the full commit history. Deleting the file later does not remove it from history, so the only safe response to a leaked commit is to rotate the credential.
Encryption at rest for etcd is often mistaken for a fix. It is a separate cluster-level setting, configured on the API server, that encrypts Secret objects before they are written to etcd. It protects against someone reading etcd's storage or backups directly. It does nothing for the YAML in your repository. It also does not restrict API access, because anyone with RBAC permission to get a Secret still receives the decrypted value.
So the real design question is this: where should the source of truth for secret values live? One option is to keep it in Git in encrypted form. The other is to keep it in an external secret store and let Git hold only references to it. Sealed Secrets takes the first approach, and External Secrets Operator takes the second.
kubectl create secret generic db-credentials \
--from-literal=password='example-not-real' \
--dry-run=client -o yaml > db-credentials.yaml
cat db-credentials.yaml
echo 'ZXhhbXBsZS1ub3QtcmVhbA==' | base64 -d
kubectl get secret db-credentials \
-o jsonpath='{.data.password}' | base64 -d
A Secret manifest generated with kubectl stores its value as base64. Anyone who can read the file or its Git history can decode it, and anyone with RBAC read access can decode the value from the live cluster even when etcd encryption at rest is enabled.
How does it work? The two models in brief
Both tools work the same way at the last step. A controller running in the cluster creates an ordinary Kubernetes Secret, and pods consume it through environment variables or volume mounts exactly as before. Your Deployments do not need to know which tool produced the Secret. The difference lies in where the controller gets the value from.
- Sealed Secrets: You encrypt the value on your machine or in CI with a public key that belongs to the cluster's controller. You commit the ciphertext as a
SealedSecretresource. When it is applied, the controller decrypts it with its private key and writes the Secret. The value travels through Git, encrypted. - External Secrets Operator: You commit an
ExternalSecretresource that names a store and a path, not a value. The operator authenticates to that store, reads the value, writes the Secret, and repeats the read on a schedule. The value never enters Git.
Put simply, Sealed Secrets is an encryption tool for Git, and ESO is a synchronization tool between an external store and the cluster. The next two sections walk through each in detail.
How does Sealed Secrets work?
Sealed Secrets has two parts: a controller running in the cluster and a CLI, kubeseal, running on engineers' machines or in CI. On first start, the controller generates a key pair and stores the private key as a Secret in its own namespace. The public key can be shared freely. Anyone with it can encrypt, but only the controller can decrypt.
The workflow
- Write an ordinary Secret manifest locally, usually with
kubectl create secret generic ... --dry-run=client -o yaml, so nothing reaches the API server. - Pipe it through
kubeseal, which encrypts each value with the controller's public key and outputs aSealedSecretcustom resource. - Commit that file and delete the plaintext.
- When the resource is applied, the controller decrypts it and creates a regular Secret with the same name. Pods consume that Secret as they would any other.
The SealedSecret manifest looks like a Secret. Its spec.encryptedData holds ciphertext per key, and an optional spec.template sets the metadata, labels and type for the generated Secret.
Scopes
By default the scope is strict: the Secret's name and namespace are mixed into the encryption, so the ciphertext only decrypts under that exact name and namespace. This prevents someone who can create resources in their own namespace from copying a sealed value out of another namespace's repo and having the controller decrypt it for them. Namespace-wide scope allows renaming within the same namespace. Cluster-wide scope allows any name in any namespace. You choose the scope when sealing, and the looser scopes trade that protection for portability.
For a small team, the appeal is that there is no secret store to provision, secure, pay for or keep available. The only new component is one controller.
kubectl create secret generic db-credentials \
--namespace payments \
--from-literal=username=app_user \
--from-literal=password='placeholder-not-a-real-password' \
--dry-run=client -o yaml > db-credentials.secret.yaml
kubeseal --format yaml < db-credentials.secret.yaml > db-credentials.sealed.yaml
rm db-credentials.secret.yaml
git add db-credentials.sealed.yaml
kubectl apply -f db-credentials.sealed.yaml
A minimal kubeseal workflow that seals a locally generated Secret with the default strict scope, commits only the ciphertext, and shows the resulting SealedSecret manifest.
How does External Secrets Operator work with Vault and cloud secret managers?
External Secrets Operator (ESO) takes the opposite approach: Git holds references to secrets, and the values stay in a system built to store them, such as HashiCorp Vault or a cloud provider's secret manager. ESO is a controller that reads those references, fetches the values, and writes them into ordinary Kubernetes Secrets. Workloads consume the result the same way they consume any other Secret.
Two resource types divide the work:
- SecretStore (namespaced) or ClusterSecretStore (cluster-wide) defines how to reach and authenticate to a provider.
- ExternalSecret declares which remote keys to fetch and the name of the target Secret.
The operator reconciles each ExternalSecret on its refreshInterval, so changes in the backend reach the cluster without a commit.
Vault with Kubernetes auth
Store a Vault token in the cluster and you have a long-lived credential that still needs its own rotation. Kubernetes auth removes it. The SecretStore's provider.vault block sets server, the KV mount in path, and version: v2. Under auth.kubernetes it sets a mountPath, a Vault role, and a serviceAccountRef. ESO presents that service account's token, and Vault verifies it against the cluster before issuing a short-lived Vault token. The Vault role limits which service accounts and namespaces can authenticate, and the policy attached to it limits which paths they can read.
In the ExternalSecret, each data entry maps a remoteRef.key (the Vault path) and an optional remoteRef.property (one field inside it) to a secretKey in the generated Secret. dataFrom with extract copies every field at a path.
Check the apiVersion against the CRDs installed in your cluster, because it has changed across ESO releases.
apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
name: vault-backend
namespace: payments
spec:
provider:
vault:
server: ""
path: "secret" # KV mount name
version: "v2" # KV v2 engine
auth:
kubernetes:
mountPath: "kubernetes" # Vault auth mount
role: "payments-eso" # Vault role bound to this SA/namespace
serviceAccountRef:
name: "payments-eso" # no static Vault token in the cluster
---
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
name: payments-db
namespace: payments
spec:
refreshInterval: 1h # re-sync from Vault without a commit
secretStoreRef:
name: vault-backend
kind: SecretStore
target:
name: payments-db-credentials # native Secret ESO creates
data:
- secretKey: DB_PASSWORD # key in the generated Secret
remoteRef:
key: payments/db # Vault path under the mount
property: password # single field at that path
dataFrom:
- extract:
key: payments/config # copy every field at this path
A Vault SecretStore that authenticates with a Kubernetes service account, and an ExternalSecret that maps one Vault field to a Secret key and extracts all fields from a second path.
External secrets vs sealed secrets: how GitOps and rotation differ
The two tools differ in what Git holds, and that difference determines how rotation works.
Sealed Secrets
Git: SealedSecret (ciphertext) --> controller decrypts with
in-cluster private key --> Secret --> Pod
External Secrets Operator
Git: ExternalSecret (references) --> operator authenticates and
pulls values from Vault / cloud store --> Secret --> Pod
With Sealed Secrets, Git is the source of truth for the value itself, only encrypted. Rotating a credential means generating the new value, re-sealing it against the cluster's public key, and committing the new ciphertext. Every rotation goes through a commit, which gives you an audit trail. It also means rotation is a manual developer task, and an automated rotation in an upstream system does nothing until someone re-seals.
With ESO, Git stores only the reference: which store, which path, which key. You rotate by updating the value in the external store. The operator picks it up on its next sync, governed by the refreshInterval on the ExternalSecret, and overwrites the Kubernetes Secret. No commit is needed. As a result, Git history no longer shows when a value changed, and that record lives in the provider's audit log instead.
Neither model restarts your workloads. Environment variables are read once at container start, so pods need a rollout or a reloader mechanism that watches the Secret. Volume-mounted Secrets are refreshed by the kubelet after a delay, but only if the application rereads the file.
Short refresh intervals shorten propagation time, but each ExternalSecret then calls the provider more often. Across hundreds of resources, that can hit API rate limits or raise per-request costs.
SEALED SECRETS
+--------------------+ +---------------------+ +--------+ +-----+
| Git: SealedSecret |--->| controller decrypts |--->| Secret |--->| Pod |
| (ciphertext) | | w/ in-cluster key | +--------+ +-----+
+--------------------+ +---------------------+
Rotate: new value -> re-seal with public key -> commit new ciphertext
EXTERNAL SECRETS OPERATOR
+--------------------+ +---------------------+ +--------+ +-----+
| Git: ExternalSecret|--->| operator syncs every|--->| Secret |--->| Pod |
| (references only) | | refreshInterval | +--------+ +-----+
+--------------------+ +---------------------+
| ^
auth/pull| |values
v |
+---------------------+
| Vault / cloud store |
+---------------------+
Rotate: update value in store -> next sync overwrites Secret (no commit)
Short refreshInterval -> more provider API calls (rate limits, cost)
BOTH MODELS
Env vars are read once at container start -> rollout or reloader needed
Sealed Secrets keeps encrypted values in Git and rotates via re-seal and commit, while ESO keeps only references in Git and rotates via the external store on each refresh, and in both cases pods using environment variables still need a restart.
What are the risks and trade-offs of each approach?
Sealed Secrets: the key is the single point of failure
The controller's private key decrypts every SealedSecret in the cluster. If you lose it without a backup, every sealed secret becomes unrecoverable, and you must reseal all of them from the original plaintext. Rebuilding a cluster therefore means restoring that key before the controller starts.
Ciphertext also stays in Git history permanently. If the key leaks, generating a new key does not help, because an attacker can decrypt old commits with the leaked one. You have to rotate every underlying credential.
There is no central record of who read which secret. Multi-cluster setups force a choice. Per-cluster keys mean resealing each secret for every cluster. A shared key means one leak exposes every cluster.
ESO: a runtime dependency
ESO depends on the external store and its authentication path being available. Existing Secrets usually remain if the store goes down, but fresh clusters and new namespaces cannot get their Secrets until it recovers. You also take on provider rate limits, managed-service pricing, or the work of running Vault yourself. Git no longer describes the full state, so drift questions must be answered from the store.
| Risk | Sealed Secrets | ESO |
|---|---|---|
| Main failure mode | Lost or leaked controller key | Store or auth outage |
| Read audit | None central | Store's audit log |
| Source of truth | Git | Git plus store |
| Ongoing cost | Key custody and resealing | Store operations and API usage |
Both tools produce ordinary Kubernetes Secrets, so RBAC on Secrets and encryption at rest in etcd still matter. Development cost differs too. Sealed Secrets is quick to adopt, but resealing slows changes. ESO takes more effort up front to set up the store, its policies, and authentication.
Who should use Sealed Secrets, and who should use ESO?
Sealed Secrets fits teams that run GitOps on one or a few clusters, have no secret store today, and want every change, including secret changes, to go through pull requests. Platform teams at early-stage companies, internal tooling clusters, and homelab or edge clusters without reliable access to a central store fall into this group. It also fits when the people who change secrets are the same engineers who own the repository, because resealing is a developer task.
ESO fits organizations where secrets already live in Vault or a cloud secret manager, where a security team owns those secrets separately from the application teams, or where credentials are rotated automatically by the store or by an upstream system such as a database's credential rotation. It also suits fleets of many clusters that need the same values, and environments where auditors expect a log of secret reads.
Neither is a good fit for teams that do not use GitOps and create Secrets out-of-band from CI, or for teams whose policy forbids secret material in etcd. Those cases are covered by the alternatives in the next section.
When should you use Sealed Secrets, ESO, or avoid both?
The deciding question is usually where a secret's source of truth already lives, and who will keep it running.
| Situation | Better fit | Why |
|---|---|---|
| Small team, one or two clusters, no external store | Sealed Secrets | Git stays the single source of truth, and the only new component is the in-cluster controller. |
| Vault or a cloud secret manager already in use | ESO | Reuses existing access policies and rotation instead of copying values into Git. |
| Auditors need to know who read a secret | ESO | The store can log reads. Sealed Secrets cannot: the controller decrypts it, and anyone who can read the resulting Secret leaves no trace. |
| Same secrets needed across many clusters | ESO | One value serves every cluster. Sealed Secrets needs a reseal per cluster key, or a shared key that widens the blast radius. |
| No reliable store, or no one to operate it | Sealed Secrets, or one of the alternatives below | ESO cannot sync when its store is down or misconfigured, so an unowned store becomes a single point of failure. |
If neither fits, three alternatives are common:
- Native Secrets with etcd encryption at rest and tight RBAC. This is enough when secrets are created out-of-band, for example by CI, and never need to live in Git.
- SOPS-style file encryption inside your GitOps tooling. Values stay in Git, but encryption keys come from a KMS or age keys rather than one cluster's controller, so a single file can target several clusters.
- Secrets Store CSI Driver. This mounts values from an external store directly into pods without creating a Kubernetes Secret. That suits teams whose policy forbids secret material in etcd.
How can it save engineering time?
Both tools save time in the same basic way: they remove the manual, out-of-band step of creating Secrets by hand in each cluster. When secrets are declared in Git, a new cluster or namespace gets its Secrets from the same reconciliation loop that applies everything else. No one has to remember which kubectl create secret commands were run where, and no one has to rebuild that list during an incident.
Beyond that shared benefit, each tool saves time in different places.
Sealed Secrets saves setup and operations time. There is no store to provision, no auth method to configure, and no store availability to monitor. A small team can have encrypted secrets in Git in an afternoon. The time cost shows up later, on every rotation and every additional cluster, because each requires a reseal and a commit.
ESO saves recurring time on rotation and fan-out. Once the store, policies and auth are in place, rotating a value is a single change in the store, and every cluster that references it picks it up on its next refresh. Adding a cluster means applying the same ExternalSecret manifests, not resealing values against a new key. Teams that already run a store also avoid maintaining a second copy of each value in Git.
Illustrative estimate
The following numbers are assumed inputs chosen for the example, not measurements. Substitute your own.
- Assumed: 40 secrets, each deployed to 3 clusters with per-cluster sealing keys.
- Assumed: each secret is rotated 4 times a year.
- Assumed: one reseal, commit, review and merge for one secret in one cluster takes 15 minutes of engineer time.
- Assumed: one rotation in the external store takes 5 minutes per secret, regardless of cluster count.
With Sealed Secrets: 40 × 4 × 3 × 15 minutes = 7,200 minutes, or 120 hours a year. With ESO: 40 × 4 × 5 minutes = 800 minutes, or about 13 hours a year.
This illustrative estimate leaves out ESO's one-time cost of setting up the store, its policies and Kubernetes auth, plus ongoing store operations if you run Vault yourself. With 5 secrets on 1 cluster rotated once a year, the same assumed rates give 75 minutes a year for Sealed Secrets, and ESO's setup effort would likely never pay back. The calculation shows the shape of the trade-off: Sealed Secrets' time cost grows with secrets × rotations × clusters, while ESO's is mostly fixed up front.
Neither tool removes the time spent restarting workloads that read secrets as environment variables. A reloader mechanism or a rollout step in your pipeline is what saves that time, and it applies to both.
How do you use both together or migrate between them?
The two tools coexist without conflict, and a hybrid setup is often the stable end state. Keep Sealed Secrets for the bootstrap credential that ESO needs to reach the store when Kubernetes auth is not an option, such as a cluster outside the cloud provider or a Vault without the Kubernetes auth method. Move every application secret to ESO. This resolves the chicken-and-egg problem: the one credential that cannot come from the store is encrypted in Git, and everything else comes from the store.
For example, a SealedSecret decrypts to a Secret named vault-approle holding a secret-id key. A Vault SecretStore then references that Secret through its AppRole auth secretRef, and ExternalSecrets in that namespace authenticate with it.
Migration path
- Load existing values into the external store. Decrypt them with cluster access or recover them from the source of truth, then write them under the paths you intend to use.
- Add ExternalSecrets that target the same Secret names. Workloads keep referencing the same names, so no Deployment changes are needed.
- Verify and restart. Compare the synced data with the current Secret. Then restart the pods that read the values as environment variables, because those pods do not pick up the change on their own.
- Remove the SealedSecrets. Check ownership references first, because deleting a SealedSecret can garbage-collect the Secret it created. Keep the controller's private key backed up until no old manifest in Git needs decrypting.
You can run this one namespace at a time, with a rollback available at each step. That avoids freezing deploys for a single big-bang cutover.
How OneEnv can help
Disclosure: OneEnv is the product behind this blog.
OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.
Applied to external secrets vs sealed secrets, this is where it helps:
Saves time
- Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
- One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
- Reviewers see the exact diff and the list of affected services up front, so approval is a decision rather than an investigation.
Saves cost
- Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
- Less duplicated configuration to maintain: one shared definition inherited by many services.
- Per-service approve/reject means a bad shared value can be refused where it would break something, before it is live.
- OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.
These are qualitative benefits, not measured results — how much you save depends on how many services share the value.
Want to try it? Start at oneenv.io.