Secrets Management

Every system holds secrets: database passwords, API keys, TLS private keys, signing keys, OAuth client secrets. A secret is any credential that grants access, and the central truth of this page is simple: a secret’s danger is proportional to how widely it’s spread and how long it lives. Good secrets management shrinks both.

The cardinal sin: secrets in code

Hard-coding a secret — or committing it to git — is the most common serious security mistake there is.

# DON'T
const DB_PASSWORD = "hunter2";          // now in every clone, every CI log, forever in git history

Git never forgets

Committing a secret and then deleting it in a later commit does not remove it — it lives in the repository history and in every clone and fork. A leaked key must be treated as compromised and rotated immediately, not just deleted. Bots scan public (and breached private) repos for credentials within minutes. Use pre-commit secret scanners to stop it at the source.

The first step: configuration, not code

The baseline practice is to pull secrets out of the codebase and inject them at runtime — classically via environment variables or mounted config files, supplied by the deployment platform, not the repo.

code  ──reads──►  process env  ◄──injected at deploy by──  the platform / secret store

This is a real improvement (the secret isn’t in source control) but it’s not the finish line: env vars can leak into logs, crash dumps, error trackers, and child processes. They’re fine for low-sensitivity config; high-value secrets want a dedicated store.

Secret managers / vaults

A secrets manager — HashiCorp Vault, AWS Secrets Manager, GCP Secret Manager, etc. — is a purpose-built service that stores secrets encrypted, controls who can read each one, audits every access, and can hand out secrets dynamically.

   app  ──(authenticates with its identity)──►  Vault
        ◄──(returns the DB password, logs the access)──

What this buys over env vars: access control per secret, an audit trail (who read what, when), central rotation, and encryption at rest. The cost: another critical service to run and a dependency in your startup path.

Two ideas that shrink the blast radius

Rotation

Secrets should expire and be replaced on a schedule (and immediately on any suspected leak). Rotation limits the window a stolen credential is useful. The reason rotation is often neglected is that it’s painful when secrets are hard-coded or copy-pasted everywhere — which is itself an argument for centralizing them, since a vault can rotate in one place.

Short-lived, dynamic credentials

The strongest pattern: don’t store a long-lived password at all. The vault generates a credential on demand with a short TTL (say, an hour), tied to the requesting service’s identity.

Static secret:   one password, lives for years, shared widely      → big blast radius
Dynamic secret:  unique, expires in 1h, issued per-service per-use  → tiny blast radius

A leaked one-hour credential is nearly worthless by the time anyone finds it. This pairs naturally with workload identity (the cloud platform vouches for which service is asking), removing the bootstrapping problem of “how does the app authenticate to the vault?”

Least privilege + blast-radius thinking

Two questions to ask of every secret: Who can read it? (grant the minimum — see Authorization) and What’s the damage if it leaks? Scope each secret narrowly (per-service, per-environment) so one compromise doesn’t unlock everything. A single god-credential shared across all services is the worst case.

It actually happened — the Codecov bash-uploader breach (2021)

Codecov’s popular CI script — the “bash uploader” that thousands of projects piped straight into their build pipelines — was quietly modified by an attacker who had gained access through a mistake in how Codecov built a Docker image. For roughly two months the tampered script did one extra thing: it scraped the build environment’s environment variables and shipped them to a server the attacker controlled. Those env vars were exactly what this page warns they often are — a junk drawer of CI secrets: cloud keys, tokens, signing credentials. The breach is a precise illustration of two ideas here: env vars are not a strong vault (they leak into logs, child processes, and, here, a compromised script), and any secret that might have been exposed must be treated as compromised and rotated — which countless downstream teams then had to scramble to do.

The thread

What does this buy us, and what does it cost? Moving from hard-coded → env vars → a vault → short-lived dynamic credentials each step shrinks the blast radius of a leak at the cost of more infrastructure and operational discipline. The mindset that matters: assume any single secret will eventually leak, and design so that when it does, the damage is small, detectable (audited), and quickly revocable (rotated).

The architect’s lens

Secrets management is blast-radius engineering for credentials. The five questions:

• Why does it exist? Because every system holds credentials (DB passwords, API keys, signing keys), and a secret’s danger is proportional to how widely it’s spread and how long it lives — good management shrinks both.
• What problem does it solve? Keeping a leaked credential from being catastrophic. The progression hard-coded → env vars → a vault → short-lived dynamic credentials each shrinks the blast radius, adding per-secret access control, an audit trail, central rotation, and finally credentials that expire in an hour and are nearly worthless by the time anyone finds them.
• What are the trade-offs? A vault buys access control, auditing, and one-place rotation — but costs another critical service to run and a dependency in your startup path. Dynamic credentials are strongest but require workload identity so the app can bootstrap its own authentication to the vault.
• When should I avoid it (over-engineering)? Env vars are fine for low-sensitivity config; you don’t need a vault for everything. But never hard-code secrets or commit them to git, and never share one god-credential across all services — scope each per-service, per-environment.
• What breaks if I remove it? Secrets end up in code, CI logs, and forever in git history where bots find them in minutes. Env vars alone leak into logs, crash dumps, and child processes — exactly the Codecov 2021 breach, where a tampered CI script scraped the environment’s secret junk drawer, forcing countless teams to scramble-rotate.

Check your understanding

1. Why is deleting a committed secret in a later commit insufficient, and what’s the correct response to a leak?
2. What does moving secrets to environment variables fix, and what does it still not protect against?
3. List three things a secrets manager gives you over plain env vars.
4. How do short-lived dynamic credentials shrink the blast radius compared with a static password?
5. Apply “least privilege” and “blast radius” to deciding the scope of a database credential.

Show answers

1. Because git never forgets — a committed secret lives in the repository history and in every clone and fork, so deleting it in a later commit doesn’t remove it. The correct response is to treat the leaked key as compromised and rotate it immediately, not just delete it — bots scan repos for credentials within minutes.
2. Moving secrets to environment variables / mounted config gets them out of source control (injected at deploy, not committed). It does not protect against env vars leaking into logs, crash dumps, error trackers, and child processes — fine for low-sensitivity config, but high-value secrets want a dedicated store.
3. Access control per secret (who can read each one), an audit trail (who read what, when), and central rotation — plus encryption at rest and dynamic issuance. The cost is another critical service to run and a dependency in your startup path.
4. A static password lives for years and is shared widely → a big blast radius if it leaks. A dynamic credential is generated on demand with a short TTL (say an hour), unique and tied to the requesting service’s identity, so a leaked one-hour credential is nearly worthless by the time anyone finds it — a tiny blast radius.
5. Ask two questions of the credential: Who can read it? — grant the minimum (least privilege), and What’s the damage if it leaks? — scope it narrowly (per-service, per-environment, minimal table/permission rights) so one compromise doesn’t unlock everything. A single god-credential shared across all services is the worst case; assume the secret will leak and design so the damage is small, audited, and quickly rotated.