Secrets Management

Secrets management is the practice of securely storing, distributing, accessing, and rotating sensitive credentials — database passwords, API keys, TLS certificates, encryption keys — in distributed systems. Hardcoding secrets in source code or configuration files is a critical vulnerability: leaked repositories expose production credentials within minutes. Modern secrets management systems like HashiCorp Vault, AWS Secrets Manager, and Azure Key Vault provide centralized, encrypted storage with fine-grained access control, automatic rotation, and comprehensive audit logging for every secret access.

Why This Matters

A single leaked database password can lead to a complete data breach. In distributed systems, the attack surface multiplies: dozens of services each need credentials for databases, APIs, message queues, and other services. Without centralized secrets management, secrets end up in Git repositories, Docker images, CI/CD configs, and environment variables — all insecure. HashiCorp Vault pioneered the modern approach: a centralized encrypted store with identity-based access control, dynamic secrets that are created on demand and automatically revoked, and comprehensive audit logs. The zero-trust principle applies: no service inherently trusts another, and every secret access is authenticated, authorized, and logged. Automatic rotation eliminates the risk of long-lived credentials being compromised.

The Building Blocks

• Encrypted Storage: Centralized secret store with encryption at rest (AES-256-GCM), protecting secrets even if the storage backend is compromised
• Access Control: Identity-based policies defining which services, roles, or users can read which secrets, enforcing least-privilege access across the organization
• Dynamic Secrets: On-demand generation of short-lived credentials (database users, cloud tokens) that are automatically revoked after a TTL, eliminating long-lived static secrets
• Automatic Rotation: Scheduled credential rotation that updates both the secret store and the downstream system (database password change) without manual intervention or downtime
• Audit Logging: Immutable records of every secret access — who accessed what secret, when, and from where — enabling security investigations and compliance reporting

Under the Hood

Secrets management systems operate on the principle of encrypt, authenticate, authorize, and audit. At the core, a secrets engine encrypts secrets with a master key that is itself protected — Vault uses Shamir's Secret Sharing to split the master key into shares requiring a quorum to unseal. Secrets are stored in an encrypted backend (Consul, etcd, DynamoDB) where even database administrators cannot read plaintext values.

Authentication determines identity. Services prove who they are through platform-native methods — Kubernetes service account tokens, AWS IAM roles, TLS certificates, or OIDC tokens. Once authenticated, authorization policies (written in HCL or JSON) determine which secrets paths the identity can access. A payment service might read secrets/data/database/payments but not secrets/data/database/analytics.

Dynamic secrets are the most powerful feature. Instead of storing a static database password, Vault connects to the database with admin credentials, creates a temporary user with minimal permissions and a short TTL (e.g., 1 hour), and returns those credentials to the requesting service. When the TTL expires, Vault automatically drops the user. If credentials are compromised, the exposure window is limited to the TTL. For PKI, Vault acts as a certificate authority, issuing short-lived TLS certificates on demand. This eliminates the operational burden of certificate management and reduces the blast radius of any single credential compromise.

How Companies Actually Do This

HashiCorp Created Vault, the industry-standard secrets management platform, used by thousands of organizations to manage dynamic secrets, encryption as a service, and PKI across cloud and on-premise infrastructure

Netflix Built their own secrets management system that integrates with AWS IAM and provides per-application encrypted secret envelopes, ensuring no service can access another service's credentials

Spotify Uses Vault with Kubernetes authentication to inject secrets into pods at startup, with automatic rotation for database credentials and API keys across their microservices fleet

Common Pitfalls

1. Storing secrets in Git repositories, Docker images, or CI/CD configs — automated scanners find these within minutes of exposure; use .gitignore and pre-commit hooks as safety nets
2. Using long-lived static credentials that never rotate — if compromised, attackers have indefinite access; dynamic secrets with short TTLs limit the blast radius
3. Making the secrets management system a single point of failure — if Vault goes down and services cannot retrieve secrets, the entire platform fails; cache secrets with encrypted local fallback

Interview Questions Worth Practicing

1. How do dynamic secrets reduce the blast radius of a credential compromise compared to static secrets?
2. How would you design secrets management for a Kubernetes-based microservices platform with 50 services?
3. What is Shamir's Secret Sharing and how does Vault use it to protect the master encryption key?

The Tradeoffs

• Security vs. Availability: Strict access controls and short TTLs maximize security but create operational risk if the secrets manager itself becomes unavailable
• Dynamic vs. Static: Dynamic secrets eliminate long-lived credential risk but require the secrets manager to be always-available and connected to every credential backend
• Centralized vs. Distributed: Centralized management provides unified audit and policy but creates a critical dependency; distributed approaches are resilient but harder to audit

How to Explain This in an Interview

Here is how I would explain Secrets Management in a system design interview:

Secrets management is the secure lifecycle handling of credentials, keys, and certificates in distributed systems. I would use HashiCorp Vault with platform-native authentication — Kubernetes service accounts, AWS IAM roles — so services prove their identity without static tokens. The most impactful feature is dynamic secrets: instead of storing a database password, Vault creates a temporary user with a 1-hour TTL and auto-revokes it. This limits the blast radius of any compromise. Access policies enforce least privilege — each service reads only its own secrets. Automatic rotation updates credentials on schedule without downtime. Every access is audit-logged. The main tradeoff is that the secrets manager becomes a critical dependency, so it needs high availability and local caching for resilience.

Related Topics

• Service Mesh
• Service Discovery
• Microservices
• Disaster Recovery

Original DosiBridge notes for Secrets Management

This section is written specifically for this page, not copied from a general system-design checklist. The useful way to study Secrets Management is to connect the concept to a production service where growth exposes the first weak assumption in the original design. That framing changes the design discussion from definitions to operating choices.

What makes this topic distinct

• Membership: For Secrets Management, decide the membership before choosing tools; otherwise the design looks clean but fails when secrets changes.
• Clock Drift: For Secrets Management, decide the clock drift before choosing tools; otherwise the design looks clean but fails when management changes.
• Retry Semantics: For Secrets Management, decide the retry semantics before choosing tools; otherwise the design looks clean but fails when secrets changes.
• Partition Behavior: For Secrets Management, decide the partition behavior before choosing tools; otherwise the design looks clean but fails when management changes.
• Reconciliation: For Secrets Management, decide the reconciliation before choosing tools; otherwise the design looks clean but fails when secrets changes.

Production review question

If a team adopts Secrets Management, what observable signal proves it helped instead of only making the architecture diagram more fashionable?

Practical metric to track

Track secrets-management health as a named dashboard or review item: combine one user-facing metric, one saturation metric, and one correctness metric. For Secrets Management, this keeps the discussion grounded in behavior that can be measured after deployment instead of generic advice that sounds good in an interview but never gets verified.