Secrets and configuration management

Database passwords, API keys, encryption keys must be stored securely — not in code, not in .env files checked into git, but in encrypted secret stores that only the right compute platform can access. The gotcha: accidentally committing a secret means rotating the value, auditing logs, and hoping no one got there first.

Advanced

5 min read

The escalating failures

Database password in code:
  WRONG ← anyone who reads the repo can access production

Database password in .env (checked into git):
  STILL WRONG ← git history preserves it forever, even after "deletion"

Database password in local .env (gitignored):
  BETTER, but...
  → multiple developers have copies
  → laptop theft = database access
  → no rotation/access audit trail

Database password in AWS Secrets Manager:
  RIGHT ← encrypted, rotated on schedule, audited, only accessible from
          specific resources (Lambda, ECS tasks) with the right IAM role

The secret management problem at scale

You have: a database password, a third-party API key, an encryption key, a signing key, a webhook HMAC secret. Each needs:

Encryption at rest: stored encrypted, unreadable without the master key.
Access control: only the applications that need it can read it (via IAM roles, not shared credentials).
Rotation: old values expire; new values are issued; applications don't notice.
Audit logging: who accessed this secret, when, from which IP.
Revocation: if a secret is leaked, instantly invalidate it without deploying new code.

A single .env file in someone's laptop meets zero of these requirements.

AWS Secrets Manager: the common pattern

AWS Secrets Manager is a managed service for storing and rotating secrets. Internally, it encrypts everything with KMS (Key Management Service), logs all access, and automates rotation.

Store a secret:

aws secretsmanager create-secret \
  --name prod/database/password \
  --secret-string "mySecurePassword123"

Retrieve it from a Lambda or ECS task (only the IAM role grants permission):

import boto3
 
secrets = boto3.client('secretsmanager')
response = secrets.get_secret_value(SecretId='prod/database/password')
password = response['SecretString']

Lambda and ECS tasks have an IAM role attached that grants secretsmanager:GetSecretValue on specific secret ARNs — no hardcoded credentials needed. The infrastructure itself proves identity.

Secrets in Terraform

Terraform needs secrets to configure resources (database root password, API keys for providers). Never put actual secrets in .tf files — use terraform.tfvars (which is gitignored) or input variables:

variable "database_password" {
  type      = string
  sensitive = true  # don't print this in terraform output
  description = "Root password for the database"
}
 
resource "aws_db_instance" "main" {
  password = var.database_password
  # ...
}

Pass the value via environment variable or command-line:

export TF_VAR_database_password="mySecurePassword123"
terraform apply

Or from a file:

terraform apply -var-file=prod-secrets.tfvars

prod-secrets.tfvars is gitignored, stored on the machine running Terraform (or passed in CI/CD via GitHub Secrets / encrypted CI variables).

The real gotcha: secrets committed to git

If a secret is ever committed to git (even briefly, even in history):

Rotate immediately — the secret is in git history forever, visible to anyone with repo access.
Audit logs — check if the secret was used from unexpected places.
Force-push only if no one else has cloned yet — if others have cloned the history, they have the old secret too.
Use a tool like git-filter-repo to scrub history — removes commits containing the secret from the entire history.

Better yet: prevent this via tooling:

# .pre-commit-config.yaml (or similar)
- repo: https://github.com/trufflesecurity/trufflehog
  rev: v3.0.0
  hooks:
    - id: trufflehog
      name: TruffleHog (detect secrets)
      args: ['filesystem', '--json']
      stages: [commit]

This runs before each commit and fails if secrets (API keys, passwords, private keys) are detected.

Environment-specific secrets

Secrets differ across environments. Use namespaced secret names:

dev/database/password — for development deployments
staging/database/password — for staging
prod/database/password — for production

Your code reads an environment variable to determine which secret to fetch:

env = os.getenv('ENVIRONMENT', 'dev')
secret_name = f'{env}/database/password'
password = secrets_client.get_secret_value(SecretId=secret_name)

This keeps the code identical across environments; the ENVIRONMENT variable (which is safe and harmless to log) determines which secrets are used.

Rotation: keeping old secrets working while new ones are issued

When a secret expires, databases usually need a grace period where both old and new passwords work. AWS Secrets Manager supports this:

# Configure automatic rotation
secrets.rotate_secret(
    SecretId='prod/database/password',
    RotationLambdaARN='arn:aws:lambda:...',
    RotationRules={
        'AutomaticallyAfterDays': 30
    }
)

AWS calls a Lambda on schedule, which updates the database with a new password, stores the new password in Secrets Manager, and tells the database to retire the old one. Applications reading the secret always get the latest value.

Machine-to-machine secrets: service accounts

Services calling other services (Lambda calling an internal API, a cron job calling a billing endpoint) need to authenticate. Options:

Bad: embed a static API key in code. Better: store the API key in Secrets Manager, retrieve it like any other secret. Best: use short-lived tokens issued by the system itself.

For internal APIs, use your existing JWT infrastructure (same private key you use for user auth) but with a synthetic "service account" subject:

token = jwt.encode(
    {'sub': 'service:billing-worker', 'exp': datetime.utcnow() + timedelta(hours=1)},
    secret_key,
    algorithm='HS256'
)
 
# Calling service includes the token
response = requests.post(
    'https://internal.example.com/charge',
    headers={'Authorization': f'Bearer {token}'}
)

The receiving service validates the JWT (same way it validates user JWTs) and checks that the sub claim is a known service account. No separate API key to manage; the token expires automatically; audit logs show the exact service that called.

Secrets and configuration management

Advanced

5 min read

The escalating failures

Database password in code:
  WRONG ← anyone who reads the repo can access production

Database password in .env (checked into git):
  STILL WRONG ← git history preserves it forever, even after "deletion"

Database password in local .env (gitignored):
  BETTER, but...
  → multiple developers have copies
  → laptop theft = database access
  → no rotation/access audit trail

Database password in AWS Secrets Manager:
  RIGHT ← encrypted, rotated on schedule, audited, only accessible from
          specific resources (Lambda, ECS tasks) with the right IAM role

The secret management problem at scale

You have: a database password, a third-party API key, an encryption key, a signing key, a webhook HMAC secret. Each needs:

Encryption at rest: stored encrypted, unreadable without the master key.
Access control: only the applications that need it can read it (via IAM roles, not shared credentials).
Rotation: old values expire; new values are issued; applications don't notice.
Audit logging: who accessed this secret, when, from which IP.
Revocation: if a secret is leaked, instantly invalidate it without deploying new code.

A single .env file in someone's laptop meets zero of these requirements.

AWS Secrets Manager: the common pattern

AWS Secrets Manager is a managed service for storing and rotating secrets. Internally, it encrypts everything with KMS (Key Management Service), logs all access, and automates rotation.

Store a secret:

aws secretsmanager create-secret \
  --name prod/database/password \
  --secret-string "mySecurePassword123"

Retrieve it from a Lambda or ECS task (only the IAM role grants permission):

import boto3
 
secrets = boto3.client('secretsmanager')
response = secrets.get_secret_value(SecretId='prod/database/password')
password = response['SecretString']

Lambda and ECS tasks have an IAM role attached that grants secretsmanager:GetSecretValue on specific secret ARNs — no hardcoded credentials needed. The infrastructure itself proves identity.

Secrets in Terraform

variable "database_password" {
  type      = string
  sensitive = true  # don't print this in terraform output
  description = "Root password for the database"
}
 
resource "aws_db_instance" "main" {
  password = var.database_password
  # ...
}

Pass the value via environment variable or command-line:

export TF_VAR_database_password="mySecurePassword123"
terraform apply

Or from a file:

terraform apply -var-file=prod-secrets.tfvars

prod-secrets.tfvars is gitignored, stored on the machine running Terraform (or passed in CI/CD via GitHub Secrets / encrypted CI variables).

The real gotcha: secrets committed to git

If a secret is ever committed to git (even briefly, even in history):

Rotate immediately — the secret is in git history forever, visible to anyone with repo access.
Audit logs — check if the secret was used from unexpected places.
Force-push only if no one else has cloned yet — if others have cloned the history, they have the old secret too.
Use a tool like git-filter-repo to scrub history — removes commits containing the secret from the entire history.

Better yet: prevent this via tooling:

# .pre-commit-config.yaml (or similar)
- repo: https://github.com/trufflesecurity/trufflehog
  rev: v3.0.0
  hooks:
    - id: trufflehog
      name: TruffleHog (detect secrets)
      args: ['filesystem', '--json']
      stages: [commit]

This runs before each commit and fails if secrets (API keys, passwords, private keys) are detected.

Environment-specific secrets

Secrets differ across environments. Use namespaced secret names:

dev/database/password — for development deployments
staging/database/password — for staging
prod/database/password — for production

Your code reads an environment variable to determine which secret to fetch:

env = os.getenv('ENVIRONMENT', 'dev')
secret_name = f'{env}/database/password'
password = secrets_client.get_secret_value(SecretId=secret_name)

This keeps the code identical across environments; the ENVIRONMENT variable (which is safe and harmless to log) determines which secrets are used.

Rotation: keeping old secrets working while new ones are issued

When a secret expires, databases usually need a grace period where both old and new passwords work. AWS Secrets Manager supports this:

# Configure automatic rotation
secrets.rotate_secret(
    SecretId='prod/database/password',
    RotationLambdaARN='arn:aws:lambda:...',
    RotationRules={
        'AutomaticallyAfterDays': 30
    }
)

Machine-to-machine secrets: service accounts

Services calling other services (Lambda calling an internal API, a cron job calling a billing endpoint) need to authenticate. Options:

Bad: embed a static API key in code. Better: store the API key in Secrets Manager, retrieve it like any other secret. Best: use short-lived tokens issued by the system itself.

For internal APIs, use your existing JWT infrastructure (same private key you use for user auth) but with a synthetic "service account" subject:

token = jwt.encode(
    {'sub': 'service:billing-worker', 'exp': datetime.utcnow() + timedelta(hours=1)},
    secret_key,
    algorithm='HS256'
)
 
# Calling service includes the token
response = requests.post(
    'https://internal.example.com/charge',
    headers={'Authorization': f'Bearer {token}'}
)

Secrets and configuration management

The escalating failures

The secret management problem at scale

AWS Secrets Manager: the common pattern

Secrets in Terraform

The real gotcha: secrets committed to git

Environment-specific secrets

Rotation: keeping old secrets working while new ones are issued

Machine-to-machine secrets: service accounts

Further reading

Secrets and configuration management

The escalating failures

The secret management problem at scale

AWS Secrets Manager: the common pattern

Secrets in Terraform

The real gotcha: secrets committed to git

Environment-specific secrets

Rotation: keeping old secrets working while new ones are issued

Machine-to-machine secrets: service accounts

Further reading