Secrets accumulate across engineering organisations the way a house accumulates junk drawers. A database password lives in a .env file on three laptops. An API key sits in a CI variable that was set two years ago by someone who left the company. A TLS certificate is hardcoded in a deployment script because “we’ll fix it later.” Each one of these is a credential with without an owner, an expiry date, or an audit trail, which means each one is a breach waiting for its moment.
The instinct is to solve this with better storage: encrypt the secrets, put them in a vault, lock down access. Storage matters, but it addresses only the surface problem. The deeper failure is architectural. Secrets sprawl happens because teams treat credentials as static configuration rather than as dynamic resources with a lifecycle that demands active management.
The Sprawl Is Structural
Secrets end up everywhere because the path of least resistance scatters them. A developer needs a database credential for local testing, so they copy it from the wiki. CI needs an API token, so someone pastes it into a pipeline variable. A Helm chart needs a TLS cert, so it gets committed to a private repo because “the repo is private anyway.” Each shortcut is individually rational and collectively builds an attack surface that’s difficult to audit.
What makes this an architectural problem rather than a discipline problem is that the system itself provides no feedback loop. A secret that was provisioned eighteen months ago and never rotated looks identical to one provisioned yesterday. There’s no health check, staleness indicator, or ownership metadata. The system treats a two-year-old database password the same as a fresh one, and without policy enforcement, so does the team.
Secrets as a Control Plane
Treating secrets management as a control plane problem rather than a storage problem reframes the entire discipline. A control plane provides centralized policy with distributed consumption, and that’s precisely what secrets need.
In a control plane model, secrets have three properties that static configuration lacks: lifecycle management (creation, rotation, revocation), policy enforcement (who can access what, under what conditions), and audit completeness (every access logged, every grant traceable). The consuming services don’t need to know where the secret is stored or how it’s managed; they request access through a well-defined interface and receive credentials that are scoped, time-limited, and logged.
If you can describe your secrets management without using the words rotation, revocation, or audit, you’re describing a password manager rather than a control plane.
This framing connects directly to how I think about key management in crypto wallets. The lifecycle of a signing key and the lifecycle of a database credential solve the same fundamental problem at different scales: controlling access to a sensitive resource through time, not just at the point of initial provisioning.
Vault Patterns That Actually Work
HashiCorp Vault (or its equivalents) gets interesting when you move past the basics of “store a secret, retrieve a secret” and into the patterns that make the control plane model real.
Dynamic secrets are the most powerful pattern. Vault generates short-lived database credentials on demand, each unique to the requesting service, each with a TTL measured in minutes or hours. A service requests database access, Vault creates a credential that lives for one hour, and the credential is automatically revoked when the lease expires. If that credential leaks, the blast radius is constrained by time, since the credential was already close to expiry.
Lease-based access extends this idea to every secret type. Every credential comes with an explicit expiry, and the consuming service must renew its lease or lose access. This inverts the default from “access until explicitly revoked” to “access until explicitly renewed,” which is a safer posture.
Vault’s power comes with significant operational overhead; smaller teams may find the setup, unsealing procedures, and ongoing maintenance burden substantial, so weighing that cost against managed alternatives like AWS Secrets Manager is worth doing early.
Policy-as-code makes the access rules auditable and version-controlled. Vault policies define which authentication backends can access which secret paths, under what conditions, and those policies live in source control alongside the infrastructure they protect. When an auditor asks “who can access the production database credentials,” you point them at a pull request history rather than manually surveying permissions across multiple systems.
The Kubernetes Secrets Landscape
Kubernetes has its own gravitational pull on secrets management, and the native approach is deceptively inadequate. kubectl create secret stores values as base64-encoded data, which is encoding, not encryption. Anyone with read access to the namespace can decode every secret in it. The etcd backing store can be configured with encryption at rest, albeit the default configuration leaves secrets unencrypted.
The ecosystem has produced several layers of improvement, each addressing a different gap:
Sealed Secrets solve the GitOps problem: how do you commit secrets to a repository safely? SealedSecret resources are encrypted with a cluster-specific public key, so they can live in source control. The controller decrypts them into native Kubernetes secrets at deploy time. This is good for the storage-and-delivery problem, albeit it doesn’t address lifecycle management.
External Secrets Operator bridges Kubernetes and external secret stores like Vault, AWS Secrets Manager, or GCP Secret Manager. It syncs secrets from the external store into Kubernetes, which means the external store remains the source of truth whilst Kubernetes pods consume secrets through the native API. Rotation in the external store propagates automatically.
CSI Secrets Store Driver skips the Kubernetes Secret object altogether, mounting secrets directly into pods as volumes from the external store. The secret never exists as a Kubernetes resource, which reduces the attack surface. The trade-off is added complexity in pod configuration and a dependency on the CSI driver’s availability.
Rotation as the Real Test
The litmus test for whether you have secrets management or just secret storage: can you rotate a credential without downtime?
If rotating a database password requires updating twelve configuration files across four services, redeploying each one, and crossing your fingers that you found every reference, you have a storage system with extra steps. Genuine secrets management means the consuming services don’t hold the credential directly; they hold a reference to the credential, or they request fresh credentials through the control plane on each connection. When the underlying credential rotates, the consumers either pick up the new value automatically or their next lease renewal hands them the updated version.
Audit and Access Logging
Storage encryption protects against a narrow threat model: someone gaining read access to the backing store. The broader questions, such as who accessed a given credential, whether that access was expected, and whether the accessor’s permissions made sense for their role at the time, are harder to answer and far more operationally relevant.
A control plane approach makes these questions answerable because every secret access passes through a single, auditable interface. Vault’s audit backend, for instance, captures the full spectrum: authentications, secret reads, policy evaluations, lease grants, and revocations. When a security team needs to investigate whether a credential was compromised, they have a complete timeline of access in one place, down to the authenticated identity, the policy that authorized it, and the exact timestamp.
The audit trail also surfaces operational insights. If a service is reading the same secret hundreds of times per minute, something is misconfigured; healthy consumers cache credentials and refresh on lease boundaries. If a credential was last accessed six months ago, it’s a candidate for revocation, since dead credentials are attack surface with no corresponding business value.
The Key Management Parallel
Secrets management and key ceremony practices converge on the same core problem. Crypto key ceremonies, where organizations generate and distribute signing keys through formal, witnessed procedures, enforce the same discipline that a secrets control plane automates: ensuring that sensitive material is created, distributed, and eventually retired through a process that’s auditable, repeatable, and resistant to single points of compromise.
A key ceremony’s formality (witnessed procedures, dual control, mandatory logging) provides a blueprint for what automated secrets management should enforce at machine scale across thousands of credentials per hour.
The scale differs enormously. A key ceremony might happen once a quarter for a handful of critical signing keys, whilst a secrets control plane processes thousands of credential requests per hour. The principles are identical: dual control (no single person can provision or access alone), separation of duties (the person who creates the credential is distinct from the person who consumes it), and mandatory audit (every action on a sensitive resource is logged and reviewable).
Most engineering organizations have internalized these principles for production access, requiring code reviews, approval gates, and audit logs for changes to production infrastructure. Secrets management applies the same rigour to the credentials that grant that access in the first place, and the credentials that grant access to everything else.