Network debug

Follow layers in order — don't skip to pods when DNS is wrong.

Reviewed: ·Tested on: Kubernetes 1.29, Terraform 1.8, Ubuntu 22.04

Start from symptom

🚫Connection refused→ layer 5
Timeout / hang→ layer 3
Could not resolve host→ layer 1
🌐502 Bad Gateway→ layer 4
🔒SSL / certificate error→ layer 4

Layer-by-layer checklist

1🌍

DNS

Does the hostname resolve to the right IP?

  • dig +short app.example.com matches LB or ingress IP
  • TTL not caching stale record after cutover
  • CNAME chain ends at expected target
2🧱

Firewall / security group

Is traffic allowed on the port from client to LB/node?

  • AWS SG: inbound 443 from 0.0.0.0/0 or corporate CIDR to LB
  • NACL not blocking return traffic
  • On-prem: ufw/iptables allow 443
3⚖️

Load balancer

Does the LB have healthy targets?

  • Target group health checks passing
  • Listener rules forward to correct target group
  • Idle timeout >= app response time
4🌐

Ingress / reverse proxy

Does ingress route to the right service:port?

  • kubectl describe ingress — backend service name and port number
  • Ingress controller logs — upstream errors
  • TLS secret attached if HTTPS
5🔗

Service & endpoints

Do Service endpoints exist and match ready pods?

  • kubectl get endpoints — not empty
  • Service port matches container targetPort
  • selector labels match pod labels
6☸️

Pod / NetworkPolicy

Is the pod listening and allowed to receive traffic?

  • kubectl exec — curl localhost:PORT/health
  • Readiness probe passing
  • NetworkPolicy allows ingress from ingress namespace