Recovered a production cluster from a CrashLoopBackOff outage
A node upgrade left an entire production namespace in CrashLoopBackOff. Mitigated in under an hour, root-caused to a probe and config-map mismatch, and fixed so it can't recur.
The problem
After a routine EKS node-group upgrade, an entire production namespace fell into CrashLoopBackOff. The on-call team had been restarting pods for two hours with no improvement and traffic was failing.
The approach
- ▸Stopped the restart loop and read the actual signal, which meant `kubectl describe` events and `--previous` logs rather than cycling pods.
- ▸Traced the crash to a liveness probe timing out during a slow startup that the new, smaller node type made worse, on top of a config-map key that a recent chart bump had renamed.
- ▸Restored service by correcting the probe thresholds and pinning the config-map values, then watched it hold under real traffic.
- ▸Wrote up a post-mortem with the timeline and added guardrails. Startup probes, resource requests sized to the node type, and a chart-diff check in CI.
The result
- ✓Production mitigated in under an hour after engagement
- ✓Root cause identified and documented, not just patched
- ✓Guardrails added so the same failure can't recur
- ✓On-call team left with a repeatable debugging playbook
stack: Kubernetes · EKS · Helm · Prometheus
Why it stayed broken for two hours
The team was treating a deterministic failure like a flake. They kept deleting and recreating pods that would crash the same way every time. More restarts were never going to help. The previous container’s logs and the pod events pointed straight at a probe timeout, made worse by a smaller node and a renamed config key. The symptom looked terrifying; the cause was mundane, which is usually how these go.
Done by Harshit Luthra, an independent infrastructure and AI engineering consultant. Bring me a similar problem →
Questions about this work
What actually causes CrashLoopBackOff after a node upgrade?+
Here it was two things stacking. A liveness probe timed out during a slow startup that a smaller new node type made worse, and a recent chart bump had renamed a config-map key. The fix was correcting probe thresholds and pinning the config-map values. CrashLoopBackOff is almost never "Kubernetes is broken", it's a mundane cause behind a scary status.
How fast can you mitigate a live production outage?+
This one was mitigated in under an hour from the start of the engagement, after the on-call team had spent two hours restarting pods. For active outages I jump on within hours, drive to mitigation first, then root cause.
Why didn't restarting the pods fix it?+
Because it was a deterministic failure, not a flake. The pods crashed the same way every restart. The previous container's logs and pod events pointed straight at the probe timeout and the renamed key, which is where the real signal was the whole time.