Kubernetes Secrets Management with External Secrets Operator

A native Kubernetes Secret is not encrypted. It is base64-encoded, which is an encoding scheme any human can reverse in one shell command. That single misunderstanding is the root of most kubernetes secrets management incidents I see: teams treat a Secret object as a vault, commit it to Git “because it looks scrambled,” and grant broad read access because “it’s already protected.” It is not. The real engineering problem is not hiding a value inside the cluster — it is safely syncing a value from an external source of truth into the cluster, with rotation, least privilege, and a clear blast radius when something breaks.

This post is an architecture decision record for that problem. The recommended answer for most teams is the External Secrets Operator, but the decision has real trade-offs against the Secrets Store CSI Driver, Sealed Secrets, SOPS, and doing nothing.

What this covers: why native Secrets are insufficient, the External Secrets Operator reconcile architecture, a decision matrix against four alternatives, rotation and propagation lag, GitOps and multi-tenancy, and the failure modes that bite in production.

Context and Background

Kubernetes ships a first-class Secret resource, and for a long time teams assumed that using it was secrets management. It is not. A Secret is a key-value object stored in etcd. By default it is stored as plaintext in etcd — the base64 in the manifest is a transport encoding for binary safety, not a security control. Anyone with get secrets RBAC in the namespace, or read access to an etcd backup, sees the value.

Kubernetes added encryption at rest for etcd via an EncryptionConfiguration, and managed providers (EKS, GKE, AKS) now wire this to a cloud KMS key. That closes the “someone stole an etcd snapshot” gap, but it does nothing about the harder questions: where is the source of truth, who rotates the value, how does the cluster get the new version, and how do you stop one compromised workload from reading every other team’s credentials.

The market answered with a source-of-truth pattern. Put secrets in a dedicated secret manager — HashiCorp Vault, AWS Secrets Manager, Google Secret Manager, Azure Key Vault — and project them into the cluster on demand. The two dominant projections are the External Secrets Operator, which writes a native Secret, and the Secrets Store CSI Driver, which mounts values as files and skips the Secret object entirely. The rest of this ADR compares them and the escape-hatch tools around them. If you are building toward zero-trust workload identity, this pairs directly with SPIFFE/SPIRE workload identity, which supplies the machine identity these operators authenticate with.

Reference Architecture: The External Secrets Operator

The External Secrets Operator (ESO) is a Kubernetes controller that reconciles secret material from an external provider into native Secret objects. As of mid-2026 the project ships a stable external-secrets.io/v1 API after resuming regular releases through 2025; older tutorials on v1beta1 still work but you should target v1 for anything new. It is a controller, not a sidecar or an admission webhook, which matters for how you reason about its blast radius.

Direct answer: ESO watches ExternalSecret custom resources, authenticates to a provider defined by a SecretStore or ClusterSecretStore, fetches the requested keys on a refreshInterval, applies an optional Go template, and writes the result into a standard Kubernetes Secret that your pods consume through the normal env or volumeMount mechanisms. Your application never learns that a secret manager exists.

Figure 1: The ESO reconcile loop. The controller reads a SecretStore for auth and endpoint, reads an ExternalSecret for the desired keys, fetches from the provider, and writes a native Secret the pod mounts.

Figure 1 shows the four moving parts and the single output. The controller is the only component that talks to the provider; the workload only ever sees a plain Secret. This indirection is the whole point: your deployment manifests stay provider-agnostic, and you can swap Vault for AWS Secrets Manager by changing one SecretStore, not a thousand pod specs.

SecretStore and ClusterSecretStore define trust boundaries

A SecretStore answers two questions: which external system, and how do I authenticate to it. It is namespaced — it lives in one namespace and only ExternalSecret resources in that same namespace may reference it. This is the primary multi-tenancy control. A ClusterSecretStore is the cluster-scoped twin: any namespace can reference it. That convenience is also its danger, because a single over-broad ClusterSecretStore pointed at a wildcard Vault path becomes a cluster-wide read primitive. Treat ClusterSecretStore as privileged infrastructure and scope its provider-side credential to the narrowest path prefix you can.

Authentication is where the design gets serious. ESO never wants a static provider credential sitting in the cluster, because that would just move the bootstrap-secret problem one level up. Instead it uses the platform’s native workload identity: IRSA (IAM Roles for Service Accounts) on EKS, Workload Identity on GKE, Managed Identity / Workload Identity Federation on AKS, and Kubernetes-auth or JWT/OIDC on Vault. The controller presents a projected ServiceAccount token; the provider validates it against the cluster’s OIDC issuer and returns short-lived credentials. There is no long-lived secret to rotate on the ESO side at all.

ExternalSecret, templating, and PushSecret

The ExternalSecret is the per-workload request. It names a target Secret to create, a refreshInterval, and either an explicit list of data keys (each mapping a remoteRef in the provider to a key in the output Secret) or a dataFrom block that pulls a whole document. A target.template block lets you reshape the output — assemble a JDBC URL from a host and password, render a full config file, set specific type fields like kubernetes.io/dockerconfigjson for image-pull secrets. Templating is what lets ESO feed applications that expect a particular file format rather than discrete environment variables.

PushSecret reverses the arrow. It takes a Secret that originates in the cluster — say, a credential minted by a database operator — and writes it out to the provider, with modes like IfNotExists so you do not clobber an externally managed value. Push is the less common direction but it closes the loop for workloads that generate their own secrets and need them backed up to the source of truth.

Provider authentication in depth

The authentication story deserves more than a sentence, because it is where most ESO deployments either become genuinely secure or quietly reintroduce the bootstrap-secret problem they were meant to solve. The failure pattern is predictable: a team installs ESO, points a SecretStore at AWS Secrets Manager, and — to move fast — hands it a static IAM access key stored in a Kubernetes Secret. Now the most privileged credential in the system, the one that can read every other secret, is itself a plaintext Secret in etcd. That is a regression, not a solution.

The correct pattern is federated identity. On EKS, IRSA associates a Kubernetes ServiceAccount with an IAM role through the cluster’s OIDC provider; ESO’s pod receives a projected token, AWS STS exchanges it for temporary role credentials, and no static key exists anywhere. GKE Workload Identity and AKS Workload Identity Federation follow the same shape against their respective IAM systems. For Vault, the Kubernetes auth method validates the ServiceAccount token against the cluster’s token reviewer and issues a short-lived Vault token bound to a policy. In every case the provider-side credential is ephemeral and scoped, and the cluster stores nothing long-lived. When you design the SecretStore, the auth block is the security boundary — spend your review time there, not on the remoteRef paths.

Namespace scoping and multi-tenancy

Multi-tenancy is not a feature you turn on; it is a discipline you enforce through where stores live and how narrow their provider credentials are. The clean model is one SecretStore per tenant namespace, each authenticating as a distinct workload identity whose provider policy only grants read on that tenant’s path prefix — tenant-a/* and nothing else. An ExternalSecret in namespace A physically cannot reference the store in namespace B, and even if it could, the credential would not resolve foreign paths. That is defense in depth: the Kubernetes boundary and the provider boundary both have to fail before a leak crosses tenants.

The ClusterSecretStore breaks this model by design, and that is exactly why it needs governance. Because any namespace can reference it, a single cluster-scoped store with a broad Vault policy becomes a shared read primitive that erases tenant isolation. There are legitimate uses — a genuinely shared platform secret like an internal registry pull credential — but the credential behind a ClusterSecretStore should be scoped to only those shared paths, never a wildcard. Enforce with policy: forbid tenant namespaces from creating ExternalSecret resources that reference a ClusterSecretStore at all, so the shared store is reachable only from platform-owned namespaces.

Deeper Analysis: ESO vs the CSI Driver and the Decision Matrix

The sharpest architectural fork is ESO versus the Secrets Store CSI Driver, because they disagree on one fundamental question: should a Kubernetes Secret object exist at all?

Figure 2: ESO writes a durable Secret into etcd that any RBAC-holder can read; the CSI Driver mounts values into a per-pod tmpfs volume that vanishes when the pod dies, so no Secret object exists.

Figure 2 makes the divergence concrete. With ESO, the provider value lands in etcd as a Secret and is subject to normal RBAC, can be shared by many pods, survives pod restarts, and is visible to anything that can get secrets. With the Secrets Store CSI Driver, the node’s CSI plugin fetches the value at pod-mount time and writes it to a tmpfs volume scoped to that one pod. When the pod dies, the material is gone. No Secret object exists, so there is nothing for a broad get secrets to leak — at the cost of a mount-time dependency on the provider and application manifests that must be changed to use the CSI volume.

Both models have a rotation story, and neither gives you rotation for free. The CSI Driver can refresh mounted files with --enable-secret-rotation and a --rotation-poll-interval. ESO refreshes on the ExternalSecret‘s refreshInterval. In both cases a long-running process that read the secret into memory at startup will not notice, which is the propagation problem covered in the next section.

The decision matrix

The table below scores the five realistic options. This is an opinion grounded in the mechanics, not a benchmark; weight it against your own threat model.

Dimension	External Secrets Operator	Secrets Store CSI Driver	Sealed Secrets	SOPS	Native Secret only
Source of truth	External manager	External manager	Git (encrypted)	Git (encrypted)	etcd
Creates a K8s Secret	Yes	No (file mount)	Yes	Yes	Yes
Rotation without redeploy	Yes, on interval	Yes, on poll	No, re-seal and commit	No, re-encrypt and commit	No
GitOps-native	Yes, CRs in Git	Yes, CRs in Git	Yes, ciphertext in Git	Yes, ciphertext in Git	No, plaintext risk
Provider outage impact	Serves last synced value	Pod cannot mount / start	None (self-contained)	None (self-contained)	None
Secret in etcd at rest	Yes	No	Yes (decrypted)	Yes (decrypted)	Yes
Multi-provider	Broad	Broad	N/A	N/A	N/A
Operational weight	One controller	DaemonSet + provider	One controller	Client-side tooling	None

The pattern is clear. If your organization already runs Vault or a cloud secret manager as the canonical store and you want minimal application change plus rotation, ESO is the default. If your threat model forbids a Secret object living in etcd at all — for example a workload handling cardholder data where you want to minimize where the value rests — the CSI Driver’s no-Secret model is worth its extra coupling; that regime overlaps heavily with the controls in card tokenization and PCI-DSS Vault architecture. Sealed Secrets and SOPS are the right answer when you have no external manager and want an encrypt-in-Git flow, but neither rotates without a human committing a new ciphertext, which disqualifies them for high-churn credentials.

GitOps compatibility

ESO is unusually GitOps-friendly precisely because the ExternalSecret and SecretStore custom resources contain no secret material. They are pure references — “fetch key prod/db/password from this store.” You commit them to Git, Argo CD or Flux applies them, and the actual value never touches your repository. This is the clean separation GitOps wants: declarative desired state in Git, sensitive material resolved at runtime by a controller. Sealed Secrets and SOPS also live in Git, but they put ciphertext there, which means key rotation and re-encryption become Git operations rather than provider operations.

Trade-offs, Gotchas, and What Goes Wrong

The honest limits of ESO cluster around propagation lag, provider coupling, and RBAC.

Figure 4: The four dominant failure classes — provider outage, bad rotation, refresh storms, and over-broad stores — and how two of them widen into a tenant-isolation blast radius.

Propagation lag and stale env vars. ESO updating a Secret does not update a running pod’s environment variables — env vars are set once at container start. If you inject secrets as env, a rotated credential is invisible until the pod restarts. Mounted secret files do get updated by the kubelet, but only after a sync delay that can be a minute or more, and only if the app re-reads the file. The common fix is a reloader controller that watches the Secret and triggers a rolling restart, or an application that watches its own mounted files. Plan for this explicitly; it is the single most common “rotation did nothing” surprise.

Provider outage. ESO fails safe by default: if the provider is unreachable, the last-synced Secret stays in place and workloads keep running on the old value. That is usually what you want. But it also means a silent failure — the Secret looks fine while drifting further from the source of truth. Alert on SecretSyncError and reconcile-age metrics, not just on pod health.

Refresh storms. Thousands of ExternalSecret resources with a short refreshInterval become thousands of provider API calls per cycle. AWS Secrets Manager and Vault both rate-limit, and a synchronized fleet will earn HTTP 429s and exponential backoff. Stagger intervals, lengthen them for stable secrets, and prefer dataFrom to pull one document over many single-key fetches.

Secret sprawl and RBAC leaks. Because ESO writes real Secret objects, every synced credential is now readable by anything with get secrets in that namespace, including a compromised sidecar or an overly broad developer role. An over-scoped ClusterSecretStore compounds this by letting one namespace pull another tenant’s material. Scope stores per namespace, keep provider credentials least-privilege at the path level, and enforce the boundary with policy — a Kyverno or OPA Gatekeeper policy that forbids ClusterSecretStore references from tenant namespaces turns an architectural intention into an enforced rule.

Orphaned and drifted Secrets. Deleting an ExternalSecret can, depending on the deletionPolicy, leave the managed Secret behind — useful when you do not want a config change to yank a live credential, dangerous when you assumed cleanup happened. The inverse also bites: if someone edits the target Secret by hand, ESO overwrites the drift on the next reconcile, silently discarding an emergency manual patch. Decide the deletionPolicy deliberately and treat the managed Secret as read-only to humans, because the controller is the source of truth for its contents and will win every conflict.

The CSI Driver is not a free lunch. The “no Secret in etcd” property is real, but it moves cost rather than removing it. Because the driver fetches at mount time, every pod scheduling event becomes a provider call, so a large scale-up or a node failure that reschedules many pods can produce a burst of provider traffic exactly when the system is already stressed. And the model only holds if you never enable the driver’s optional Kubernetes-Secret sync — turning that on to get env-var support quietly recreates the Secret object you adopted the CSI Driver to avoid.

Rotation and Propagation: A Closer Look

Rotation is where secrets management earns or loses its keep, so it deserves its own walk-through.

Figure 3: A rotation event propagates from the provider through the controller’s refresh interval into the Secret, but a running pod’s env vars stay stale until a reloader or watch triggers a reload.

Figure 3 traces a single rotation. An operator or automated job writes a new version to the provider. Nothing happens in the cluster yet — ESO only fetches on its interval. When the interval elapses, the controller pulls the current version, updates the Secret’s data, and here the path forks. A pod consuming the secret as files may pick up the change after the kubelet’s sync; a pod consuming it as env vars will not, ever, until it restarts. A reloader that hashes the Secret and rolls the deployment closes the gap deterministically.

The capacity math is worth doing before you scale. Total provider calls per minute is roughly the number of ExternalSecret resources divided by the average refreshInterval in minutes. Ten thousand ExternalSecrets on a one-hour interval is about 167 calls per minute steady-state — comfortable. The same ten thousand on a one-minute interval is 10,000 calls per minute, which will hit provider quotas and throttle. These are illustrative figures to show the shape of the relationship, not a benchmark of any specific provider; measure your own limits. The lesson holds regardless: refresh interval is a capacity knob, not just a freshness knob, and most secrets tolerate a long interval because rotation events are rare and can be pushed rather than polled.

Operational and capacity considerations

ESO is a single controller (with leader election) watching every ExternalSecret in the cluster, so its resource envelope scales with the number of resources and the aggressiveness of their intervals, not with the number of pods. In small clusters it is comfortable at tens of millibars of CPU and a few hundred mebibytes of memory; the figure that actually stresses it is the reconcile rate — total resources multiplied by refresh frequency — because each reconcile is a provider round-trip plus an API-server write. Two levers keep it healthy at scale: lengthen intervals for stable secrets, and shard providers so no single SecretStore credential fans out to tens of thousands of paths. If you run ESO across many clusters, run one instance per cluster rather than a central instance reaching in, because cross-cluster network paths to the API server become both a latency tax and a wider blast radius.

Watch the etcd and API-server side too. Every synced secret is a Secret object, and a large fleet of small ExternalSecrets means a large population of Secret objects, each consuming etcd space and appearing in every list secrets a controller performs. This is the secret-sprawl cost in operational terms: it is not just a security surface, it is API-server load. Prefer consolidating related keys into one templated Secret over minting dozens of single-key Secrets, both for RBAC cleanliness and for etcd hygiene.

Provenance and supply-chain integrity of the operator itself also matter here — you are granting a controller cluster-wide read access to your most sensitive material, so pin and verify its images the same way you would any privileged component, along the lines of SLSA and Sigstore supply-chain controls.

Practical Recommendations

Default to the External Secrets Operator when you already run a real secret manager and want rotation with minimal application change. Reach for the Secrets Store CSI Driver when your policy forbids a Secret object in etcd and you can afford the mount-time provider dependency. Use Sealed Secrets or SOPS only when you have no external manager and accept manual rotation.

Regardless of tool, adopt these controls:

Scope stores tightly. Prefer namespaced SecretStore; treat ClusterSecretStore as privileged and enforce who may reference it with policy-as-code.
Use workload identity, not static credentials. IRSA, GKE/AKS Workload Identity, or Vault Kubernetes-auth — never a long-lived provider key stored in the cluster.
Plan propagation explicitly. Decide up front whether rotated secrets reach pods via file mounts, a reloader, or a native restart, and test it.
Tune refresh interval for capacity. Long intervals for stable secrets; short only where freshness is genuinely required; stagger to avoid synchronized storms.
Monitor sync health. Alert on SecretSyncError and reconcile age, not just pod status, so a silent provider outage does not hide behind healthy pods.
Least-privilege the provider credential at the path/prefix level so a leaked store credential exposes the smallest possible set of secrets.

Frequently Asked Questions

Is a Kubernetes Secret encrypted?

Not by default. A native Secret is base64-encoded, which is an encoding, not encryption — anyone can decode it instantly. The value is stored in etcd, and unless you configure encryption at rest with an EncryptionConfiguration backed by a KMS key, it sits in etcd as plaintext. Even with encryption at rest, RBAC still governs who can read the Secret through the API, so encryption at rest is necessary but not sufficient for real kubernetes secrets management.

External Secrets Operator vs Secrets Store CSI Driver — which should I use?

Use ESO when you want a native Secret object, minimal application changes, and interval-based rotation; it fails safe by serving the last value during a provider outage. Use the CSI Driver when your policy forbids a Secret object in etcd, since it mounts values into a per-pod tmpfs volume and creates no Secret. The trade-off is that the CSI Driver couples pod startup to provider availability and requires editing your workload manifests to add the CSI volume.

Does the External Secrets Operator support HashiCorp Vault?

Yes. Vault is one of the most widely used ESO providers. You define a SecretStore or ClusterSecretStore with the Vault provider, choose an auth method — Kubernetes auth, JWT/OIDC, AppRole, or a token — and reference paths in either KV v1 or KV v2 engines. Kubernetes auth is the recommended path because it uses the pod’s ServiceAccount token instead of a static credential, so there is no long-lived Vault token to store or rotate inside the cluster.

How does secret rotation actually reach my pods?

The operator updates the Secret in etcd on its refresh interval, but that update does not automatically reach a running pod. Environment variables are fixed at container start and never change until restart. Mounted secret files are updated by the kubelet after a sync delay, but only if your application re-reads them. The reliable pattern is a reloader controller that watches the Secret and triggers a rolling restart, or an application that natively watches its mounted files.

Can I store ExternalSecret resources safely in Git?

Yes, and this is a core advantage. An ExternalSecret and its SecretStore contain only references — which key to fetch from which store — not the secret value itself. You can commit them to Git and let Argo CD or Flux apply them without ever putting sensitive material in your repository. This is cleaner than Sealed Secrets or SOPS, which store ciphertext in Git and turn key rotation into a re-encryption-and-commit workflow.

What happens if the secret provider goes down?

ESO fails safe: the last successfully synced Secret remains in place and your workloads keep running on it. The risk is that the failure is silent — the Secret looks healthy while drifting from the source of truth — so you must alert on SecretSyncError events and reconcile-age metrics. The Secrets Store CSI Driver behaves differently: because it fetches at mount time, a provider outage can prevent new pods from starting, which is a harder failure during a scale-up or node replacement.

Kubernetes Secrets Management with External Secrets Operator (2026)