Secrets rotation

Zero-downtime pattern: new cred → update store → rolling restart → revoke old after soak.

Reviewed: ·Tested on: Kubernetes 1.29, Terraform 1.8, Ubuntu 22.04

Full runbook: rotate without downtime →
🐘

Database credentials

Rotate app DB user password without dropping connections.

Dual-user pattern: create new user → update app secret → rolling restart → revoke old user.

  1. Create new DB user with same grants (or ALTER USER with new password on replica first)
  2. Update K8s Secret / ExternalSecret / Vault path with new password
  3. kubectl rollout restart deploy/<app> -n <ns> — one replica at a time if PDB allows
  4. Verify app health and connection count
  5. Revoke or drop old credentials after soak period (24h)
pg-rotate-user →k8s-secret-update-rollout →GuideConfig
☁️

IRSA / workload IAM role

Rotate or replace IAM role used by Kubernetes service account.

Attach new role to SA annotation → rolling restart pods → detach old role after all pods migrated.

  1. Create new IAM role with least-privilege policy (copy from old, tighten if possible)
  2. Update trust policy for OIDC provider + service account subject
  3. Patch ServiceAccount: eks.amazonaws.com/role-arn: arn:aws:iam::ACCOUNT:role/NEW
  4. Rollout restart workloads using that SA
  5. Validate AWS API calls from pod (aws sts get-caller-identity)
  6. Delete old role after confirmation
irsa-verify-pod →k8s-rollout-restart-sa →GuideConfig
🔒

TLS certificates (ingress / LB)

Renew public or internal certs before expiry.

cert-manager auto-renew; manual: issue new cert → update secret → ingress picks up without pod restart.

  1. Check expiry: openssl or cert-manager Certificate status
  2. For cert-manager: fix ClusterIssuer / DNS-01 or HTTP-01 challenge if renew failed
  3. For manual/nginx: certbot renew or import new cert to K8s tls secret
  4. Verify browser + curl -v; monitor cert-manager Certificate Ready=True
openssl-cert-check →k8s-cert-expiry →GuideConfig
🔑

API keys & static tokens

Third-party API keys, registry tokens, webhook secrets.

Issue new key in provider UI → update secret in Vault/ESO → rolling restart → revoke old key after soak.

  1. Generate new key in provider (GitHub PAT, Stripe, SendGrid, Docker registry)
  2. Store in Vault or ExternalSecret — never commit
  3. Update K8s Secret reference or reload via Reloader if annotated
  4. Rollout restart affected deployments
  5. Revoke old key in provider after 24–48h
vault-kv-update →k8s-secret-update-rollout →GuideConfig
🗄️

Vault unseal / root token

Operational rotation of unseal keys and root — use Shamir rekey procedure.

Use vault operator rekey during maintenance window; HA Vault keeps serving during rekey if done correctly.

  1. Schedule maintenance; ensure HA peers healthy
  2. vault operator rekey init — threshold unchanged or updated per policy
  3. Distribute new unseal shares to separate custodians
  4. Revoke old root token; generate new root via recovery if using auto-unseal
vault-unseal →Guide
Go-live checklist →Vault guide →External Secrets template →