Kubernetes CrashLoopBackOff in Production: A Debugging Playbook That Actually Works

The status that lies to you

CrashLoopBackOff is the most alarming-looking status in Kubernetes and one of the least informative. It doesn’t tell you what broke. It tells you that Kubernetes tried to start your container, the container exited, and Kubernetes is now waiting an exponentially increasing backoff period before trying again. That’s it. The actual cause could be a one-line typo in an env var or a genuine infrastructure failure, and the status looks identical either way.

I’ve been called into enough of these at 2am to have a strong opinion: almost every CrashLoopBackOff in production traces back to one of five boring causes. The skill isn’t cleverness, it’s discipline about the order you check things in, and resisting the urge to kubectl delete pod before you’ve looked at anything.

Stop restarting, start reading

The single most common mistake I see on-call engineers make is treating a deterministic failure like a flake. If a pod crashes, restarts, and crashes again in the same way, restarting it a third time will not help. You’re not un-jamming a stuck process, you’re destroying the evidence of what went wrong, because the previous container’s logs and its exit reason get overwritten by the next crash.

Before you touch anything, capture the state:

kubectl get pod <pod-name> -n <namespace> -o wide
kubectl describe pod <pod-name> -n <namespace>
kubectl logs <pod-name> -n <namespace> --previous
kubectl logs <pod-name> -n <namespace> --previous --all-containers

describe gives you the event stream: image pull status, scheduling decisions, and — critically — the last termination reason and exit code. logs --previous gives you what the container printed right before it died, which is gone forever once it crashes again. If you only remember one command from this article, make it --previous.

Read the exit code before you theorize

The exit code in kubectl describe pod output (under Last State: Terminated) narrows the search enormously:

• Exit code 0 with restarts happening anyway usually means the main process finished and exited cleanly, but the container’s entrypoint was supposed to keep running. Common with misconfigured CMD/ENTRYPOINT, or a script that runs once instead of a long-running server.
• Exit code 1 is a generic application error. Go straight to --previous logs.
• Exit code 137 means SIGKILL. In Kubernetes this is almost always an OOMKill. Confirm with:

kubectl get pod <pod-name> -n <namespace> -o jsonpath='{.status.containerStatuses[0].lastState.terminated.reason}'

If it prints OOMKilled, your container is exceeding its memory limit. Check actual usage against the limit:

kubectl top pod <pod-name> -n <namespace>

then either raise the limit (if the workload genuinely needs more) or find the memory leak (if usage climbs over time rather than sitting at a stable higher baseline). Don’t just double the limit and move on without checking which one it is — I’ve seen teams 4x a memory limit to “fix” a leak that came back a week later at the new, larger ceiling.

• Exit code 143 means SIGTERM, a graceful shutdown request. This is often a probe or a rolling deploy interrupting the pod, not a crash at all. Check whether a deployment or HPA is actively churning pods.

The liveness probe trap

This is the cause I see missed most often because it doesn’t look like a probe problem, it looks like the application is broken. Here’s the pattern: your app has a slow startup (JIT warmup, loading a large model, connecting to a database with retries, walking a big cache), and the liveness probe’s initialDelaySeconds or timeoutSeconds is too tight for that startup time. Kubernetes calls the probe, gets no response in time, decides the container is unhealthy, and kills it. The container was never broken, it just hadn’t finished starting yet. It restarts, starts the same slow warmup, gets killed at the same point, forever.

The tell is in describe pod: a Killing event followed by Unhealthy warnings referencing Liveness probe failed, right before the crash. Check the actual probe config against how long startup really takes:

kubectl get pod <pod-name> -n <namespace> -o jsonpath='{.spec.containers[0].livenessProbe}'

If your app takes 40 seconds to become ready and your probe has initialDelaySeconds: 10, you’ve found it. The fix, since Kubernetes 1.16+, is a startup probe, which gets its own generous timeout and only hands off to the liveness probe once the container has actually finished starting:

startupProbe:
  httpGet:
    path: /healthz
    port: 8080
  failureThreshold: 30
  periodSeconds: 5
livenessProbe:
  httpGet:
    path: /healthz
    port: 8080
  periodSeconds: 10
  timeoutSeconds: 3

This is exactly the shape of bug that got worse after a node-group upgrade in a CrashLoopBackOff production recovery I did: a smaller instance type in the new node group made startup slower under CPU throttling, which pushed a probe that had “always been fine” past its timeout. The on-call team had been restarting pods for two hours before I got the call, because the symptom looked like an application regression from the same deploy that happened to land around the same time. It wasn’t. It was a probe that had been marginal for months and finally tipped over.

Config and secret mismatches

The other repeat offender is a ConfigMap or Secret that changed shape. A Helm chart bump renames a key, an env var referencing valueFrom.configMapKeyRef points at a key that no longer exists, and the container crashes on startup trying to read a config value that isn’t there. This one is quick to confirm:

kubectl get configmap <name> -n <namespace> -o yaml
kubectl get pod <pod-name> -n <namespace> -o jsonpath='{.spec.containers[0].envFrom}'

Diff what the pod spec expects against what the ConfigMap actually contains. If you use Helm, helm diff upgrade before applying catches this class of bug before it ships, which is worth setting up in CI if you’re not already.

ImagePullBackOff and Pending look similar, aren’t

Two adjacent statuses get lumped in with CrashLoopBackOff conversationally but have entirely different causes. ImagePullBackOff means the kubelet can’t pull the image, check the tag exists, registry auth, and network egress from the node. Pending means the pod hasn’t been scheduled at all, check describe for FailedScheduling events, which usually point at insufficient CPU/memory on any node, a taint with no matching toleration, or a PersistentVolumeClaim that can’t bind. Neither of these is a crash loop, and treating them like one wastes time looking at logs that don’t exist yet because the container never started.

A checklist for the next 2am page

1. kubectl describe pod — read every event, note the exit code and termination reason.
2. kubectl logs --previous — read what the container actually said.
3. If exit 137: check top pod and the memory limit, is it a leak or a genuinely undersized limit?
4. If probe-related events precede the kill: check initialDelaySeconds/timeoutSeconds against real startup time, consider a startupProbe.
5. If it crashes instantly on every attempt: diff the ConfigMap/Secret against what the app expects, check for a recent chart or manifest change.
6. Only after you know the cause, decide the fix. Don’t roll back blind and don’t bump resource limits blind either, both can mask the real bug until it comes back worse.
7. Write down what you found. The five-minute post-mortem note is what turns a fire into a guardrail.

None of this requires exotic tooling. It requires not skipping steps under pressure, which is exactly what’s hard to do at 2am with a VP watching the incident channel. If you’re mid-outage right now and want a second pair of eyes, or you want the guardrails put in afterward so it can’t recur, that’s infrastructure debugging and incident response work I do regularly, including live on a call.