Secrets Management: Vaults, Rotation and Workload Identity
Secrets management explained: why secrets don't belong in code, vaults and KMS, dynamic short-lived credentials, rotation, workload identity and auditing.
Secrets management is how a system stores, distributes, rotates and audits credentials such as database passwords, API keys, signing keys and certificates. The goals are that secrets never live in source code or images, each workload gets only the secrets it needs, credentials are short-lived or rotated regularly, and every access is logged. Modern platforms go further and replace long-lived secrets with workload identity.
Where secrets go wrong
- Committed to source control: a key in a repository is effectively public, even after deletion, because history and forks keep it.
- Baked into images or config files, where anyone with read access to the registry or server can read them.
- Shared and long-lived: one database password used by ten services for three years can't be rotated without breaking all of them.
- Over-privileged: an analytics job holding an admin credential turns a small bug into a breach.
- Leaked through logs and error messages when requests or environment variables are dumped.
A secrets architecture
| Component | Role |
|---|---|
| Secrets service (vault) | Stores secrets encrypted, enforces access policies per workload, issues and revokes credentials, logs every access |
| Key management service (KMS) or HSM | Holds root encryption keys that never leave it; the vault uses it for envelope encryption. See encryption |
| Workload identity | A platform-issued, verifiable identity for each service (a Kubernetes service account token, a cloud instance role) used to authenticate to the vault without a stored password |
| Delivery mechanism | Inject secrets at runtime as files or environment variables, or have the app fetch them through an SDK |
Static vs dynamic secrets
| Static secrets | Dynamic secrets | |
|---|---|---|
| What | A fixed value stored and handed out (an API key from a vendor) | Credentials generated on request with a short lease (a database user valid for one hour) |
| Rotation | Scheduled; needs coordination with every consumer | Automatic: they simply expire |
| Blast radius if leaked | Until someone notices and rotates | Until the lease expires, and revocable per workload |
| Attribution | Shared credentials hide who did what | Each workload has its own credentials |
Rotation without downtime
- Create the new credential while the old one still works (most systems allow two active keys).
- Distribute the new credential; applications reload it without restarting, by watching the secret file or refetching on a schedule.
- Verify that traffic is using the new credential.
- Revoke the old one.
If rotating a secret is scary, it won't be done during an incident, which is exactly when it's needed. Rotate routinely and automatically so the process is tested.
Operating principles
- Least privilege: policies scoped per service and environment; production secrets unreadable from staging.
- No humans on the hot path: people get time-limited, approved, audited break-glass access, not standing credentials.
- Scan for leaks in repositories, CI logs and container images, and treat any detected secret as compromised.
- High availability: the secrets service is a critical dependency; replicate it, cache leases in clients, and plan for it being briefly unreachable.
- Audit and alert on unusual access, such as a service reading secrets it never used before.
Where it’s used
Pods authenticate with service account identity and receive short-lived database credentials as mounted files.
Deployment jobs use federated, short-lived cloud credentials instead of stored access keys.
An internal certificate authority issues short-lived TLS certificates for service-to-service mTLS, rotated automatically.
In an interview
- Keep secrets out of code and images; inject at runtime.
- Use workload identity to authenticate and dynamic, short-lived credentials where possible.
- Describe zero-downtime rotation with two active credentials.
- Enforce least privilege, audit access and make the vault highly available.
Further reading
- OWASP Secrets Management Cheat SheetOWASP Foundation
- SPIFFE: Secure Production Identity Framework for EveryoneSPIFFE (CNCF)