Kubernetes Workload Failures: CrashLoopBackOff, ImagePullBackOff, Pending and OOMKilled

Every pod failure investigation starts the same way: get the pod's own account of events before touching nodes or networking.

kubectl get pod <name> -n <ns> -o wide
kubectl describe pod <name> -n <ns>      # Events section is the gold
kubectl logs <name> -n <ns> --previous    # the crashed attempt, not the waiting one

describe shows scheduling and pull events; logs --previous shows why the process died. Ninety percent of cases end here.

The container ran and exited (or was killed) repeatedly, and kubelet is now backing off restarts. The exit code in describe tells you which family of problem you have:

kubectl logs <pod> -n <ns> --previous --tail=100
kubectl get events -n <ns> --sort-by=.lastTimestamp | tail
kubectl get pod <pod> -n <ns> -o jsonpath='{.status.containerStatuses[0].lastState}'

Typical culprits in the logs: connection refused to a database that is not ready (add an initContainer or startup probe), ConfigMap "x" not found, or a missing secret key that crashes the app at boot.

The event text distinguishes the four causes instantly:

# on the node where the pod is scheduled
crictl pull <image>          # containerd
systemctl status containerd
curl -sI https://registry.example.com/v2/

If crictl pull fails with a TLS error but curl works, the runtime's trust store is the problem, not the network.

The Events section will say exactly why. The common ones:

• 0/5 nodes are available: 3 node(s) had taint {...}, 2 node(s) didn't match Pod's node affinity — taints/tolerations or affinity rules exclude every node.
• Insufficient cpu / Insufficient memory — requests exceed what is schedulable. Compare with kubectl describe node → Allocated resources.
• pod has unbound immediate PersistentVolumeClaims — the PVC is unbound; the StorageClass may not exist or the provisioner is failing. Check kubectl get pvc and the provisioner's logs.

A node with 4 vCPU does not offer 4 vCPU to pods; kubelet and system daemons reserve part of it. If requests across the namespace total near the allocatable value, the next pod goes Pending even though top shows the node idle — scheduling uses requests, not usage.

kubectl describe node <node> | sed -n '/Allocated resources/,/Events/p'

Exit Code 137 with Reason: OOMKilled in lastState means the cgroup limit was hit — the container exceeded its own resources.limits.memory. A node-level OOM (kernel oom-killer under system pressure) shows in dmesg on the node and may kill any process, not just the limited one.

kubectl get pod <pod> -o jsonpath='{.status.containerStatuses[*].lastState.terminated}'
# on the node:
dmesg -T | grep -i 'out of memory'

• Check real usage over a week: kubectl top pod now, but use metrics-server history or Prometheus container_memory_working_set_bytes for the p99.
• Set requests near observed p50 and limits near p99 + headroom. Working-set spikes during GC or large batch reads are the usual killers.
• Java workloads: the JVM heap is not the whole story — set -XX:MaxRAMPercentage and leave 25–30% for metaspace, threads and off-heap.

A container restarting with Exit Code 137 but no OOMKilled reason is usually the kubelet enforcing a failing liveness probe. Check describe for Liveness probe failed: HTTP probe failed with statuscode 503. Fixes: give slow-starting apps a startupProbe with generous failureThreshold, point liveness at a cheap endpoint (not one that checks the database), and never let liveness and readiness share an expensive handler.

• Ship logs to a collector — kubectl logs --previous is lost after the next restart.
• Set requests and limits on every workload; unbounded pods are the root of both Pending and node OOM drama.
• Use startupProbe for anything that takes over 10 seconds to boot.
• Tag images immutably (SHA or semver), never latest in production — ImagePullBackOff investigations double when tags move.
• Alert on kube_pod_status_phase{phase="Pending"} older than 5 minutes and on kube_pod_container_status_restarts_total rate.