Scenario runbooks
Incident-style flows — symptom to fix with links to guides, configs, and cheat sheets.
Reviewed: ·Tested on: Kubernetes 1.29, Terraform 1.8, Ubuntu 22.04
Fix 502 behind Nginx + K8s Ingress
Upstream unreachable — trace from browser to pod through ingress and service.
🔒Terraform state lock stuck
plan/apply blocked by DynamoDB or remote backend lock — release safely.
⎈Helm release failed or pending
Upgrade stuck in pending-install or pending-upgrade — diagnose and recover.
🔄CI pipeline failed at deploy step
Build passed but deploy to K8s/registry failed — common fixes across CI platforms.
💥Pod CrashLoopBackOff
Container starts then exits — check logs and exit code.
📦ImagePullBackOff
Cluster cannot pull container image.
🔐Ingress TLS cert expired or missing
Browser SSL error or cert-manager not ready.
🖥️Node NotReady / disk pressure
Scheduler cannot place pods; node conditions unhealthy.
🔑CI pipeline auth failed (registry / cloud)
401 on docker push or AWS/kubectl denied in CI job.
📐Terraform drift / unexpected plan changes
Plan wants to recreate resources or state differs from reality.
🔒Vault is sealed
Secrets unavailable after restart until unseal.
⏳Pod running but not Ready
readinessProbe failing — service has no endpoints.
🔌Connection refused upstream
Nginx/ingress cannot reach backend service.
☁️AWS credentials expired or invalid
CLI or CI fails STS or API calls.
⎈Helm release stuck pending
Upgrade cannot complete; revision hung.
🔑Rotate secrets without downtime
DB creds, IRSA, TLS, API keys — dual-credential pattern and rolling restarts.
💾Database restore drill (quarterly)
Prove backups work — restore to staging before prod depends on it.
☸️Velero restore failed or incomplete
Backup exists but restore hangs, missing PVCs, or wrong namespace.
💥Pod OOMKilled — out of memory
Container killed by kernel OOM — fix limits, leaks, or node pressure.
🔄Argo CD sync failed or OutOfSync
GitOps app stuck — compare diff, fix RBAC, hooks, or resource conflicts.
🔍DNS not resolving in cluster or from pod
Service name fails, external DNS wrong, or CoreDNS unhealthy.
🔴Redis connection refused or timeout
App cannot reach Redis — network, auth, maxclients, or wrong host.
📈High CPU throttling or latency spike
Pods hit CPU limits, HPA lagging, or node CPU saturated.