How to Debug Kubernetes CrashLoopBackOff and OOMKilled Pods: A Step-by-Step Troubleshooting Guide (2026)
A Kubernetes CrashLoopBackOff tells you almost nothing on its own. It is not an error, a pod phase, or a root cause. It is the kubelet saying “this container keeps dying, so I am waiting before I try again.” The real evidence sits one field away, in lastState, and in a handful of places most runbooks never mention: the cgroup’s memory.events counters, the probe events, and the logs of the previous container instance, which disappear when you delete the pod. In 2026 the ground has also shifted. cgroup v2 is the default on current distributions, the kubelet kills the whole container on an out-of-memory event, the crash-loop backoff cap is now tunable per node, and container-level restart rules are beta. Old advice such as “just raise the limit” is now actively misleading.
What this covers: an eight-step triage method, an exit-code decision tree, runnable reproduction manifests, a triage shell script, how OOMKilled differs from probe kills and evictions, cgroup v2 memory internals, JVM, Node.js, Python and Go heap settings, and the 1.33–1.37 changes to restart behaviour.
Context and Background
Every container in a pod has a state: Waiting, Running or Terminated. When a container exits and the pod’s restartPolicy allows a restart, the kubelet restarts it in place on the same node. The Kubernetes pod lifecycle documentation describes the sequence precisely. The first crash gets an immediate restart. After that the kubelet applies an exponential backoff of 10 seconds, 20 seconds, 40 seconds and so on, capped at 300 seconds. If a container then runs for 10 minutes without problems, the kubelet resets the backoff timer for that container. CrashLoopBackOff is simply the reason shown on the Waiting state while that backoff timer is running.
That definition has a practical consequence. The pod’s phase stays Running during a Kubernetes CrashLoopBackOff, because at least one container is in the process of restarting. Dashboards that alert only on phase miss a Kubernetes CrashLoopBackOff entirely. You must alert on restartCount growth or on the waiting reason.
The restart machinery has changed more in the last two years than in the previous five. Kubernetes 1.33 added the alpha ReduceDefaultCrashLoopBackOffDecay feature gate, which drops the initial delay to 1 second and the cap to 60 seconds cluster-wide. The KubeletCrashLoopBackOffMax gate, alpha since 1.32, became beta and enabled by default in 1.35. It lets an administrator set crashLoopBackOff.maxContainerRestartPeriod in the kubelet configuration to any value between 1s and 300s. Container-level restartPolicy and restartPolicyRules (the ContainerRestartRules gate) reached beta in 1.35, and the RestartAllContainers action reached beta in 1.36. All of these are confirmed in the current feature gate reference.
Memory failures also look different now. Since Kubernetes 1.28 the kubelet sets the cgroup v2 memory.oom.group flag on container cgroups, so an out-of-memory event kills every process in the container together. Kubernetes 1.32 added the singleProcessOOMKill kubelet option to restore the older per-process behaviour. If you have already read our deep dive on CPU throttling, CFS quota and cgroup v2 limits, this post is the memory and restart half of the same story.
How to Debug a Kubernetes CrashLoopBackOff, Step by Step
To debug a Kubernetes CrashLoopBackOff, read the container’s lastState.terminated reason and exit code first, then the previous instance’s logs with kubectl logs --previous, then the pod events. The exit code tells you which class of failure you have: application error, missing binary, OOM kill, probe kill or segfault. Reproduce with kubectl debug only after that.
Here is the full procedure. Each step is cheap, and each one rules out a class of causes before you spend time on the next.
- Confirm the waiting reason. Make sure it is
CrashLoopBackOffand notImagePullBackOff,CreateContainerConfigErroror anInit:variant. - Read
lastState.terminated. Record thereason,exitCode,startedAtandfinishedAtfor the failing container. - Pull the previous instance’s logs with
kubectl logs <pod> -c <container> --previous. - Read events in time order. Look for
BackOff,Unhealthy,Killing,OOMKillingorEvicted. - Decode the exit code. Map 1, 2, 126, 127, 137, 139 and 143 to a failure class.
- For exit code 137, separate the three killers. Distinguish the cgroup OOM killer, a probe-driven SIGKILL, and node-pressure eviction.
- Check probes and init containers. Compare the liveness timeouts with real startup time and inspect init container status.
- Reproduce interactively with kubectl debug. Use a pod copy with the entrypoint overridden, or a node debug session to read cgroup files.

Figure 1: Decision tree from waiting reason and exit code to the likely root cause.
Figure 1 starts from the symptom everyone sees, a pod that is not Ready with a climbing restart count. The first branch is the waiting reason, which separates image problems and init failures from a true Kubernetes CrashLoopBackOff. Inside that loop, the exit code in lastState is the main branching variable. Exit 137 needs a second look at the reason field, because the same signal can come from the kernel or from the kubelet.
The evidence is in lastState, not state
During backoff the current state is Waiting with reason CrashLoopBackOff, which carries no diagnostic information. The termination record of the last attempt lives in lastState.terminated. It holds the exitCode, a reason such as Error, OOMKilled or Completed, an optional message, and the start and finish timestamps. The difference between startedAt and finishedAt is one of the most underused signals in pod restart troubleshooting.
A container that dies within one second almost always has a startup problem: a missing file, a bad flag, a refused connection the code treats as fatal. One that dies after exactly initialDelaySeconds + failureThreshold × periodSeconds is almost always being killed by a liveness probe. One that dies after a variable, load-dependent time and shows OOMKilled is running into its memory limit. Three timestamps classify most cases before you open a log.
The kubectl describe pod output prints the same fields under Last State. The official debugging walkthrough shows exactly this shape for a container running false: State: Waiting, Reason: CrashLoopBackOff and Last State: Terminated, Reason: Error, Exit Code: 1.
Logs from the dead instance, not the waiting one
kubectl logs <pod> without flags tries to read the current container instance. During backoff that instance has not started yet or has only just started, so you see nothing or only the opening lines. The --previous flag reads the log of the prior terminated instance, which is the one that actually crashed. This is the documented path for crash logs.
There are two limits. First, the kubelet keeps only a small number of dead container instances for log retrieval, so --previous reaches back one restart, not ten. Second, deleting the pod deletes those logs. Never “fix” a Kubernetes CrashLoopBackOff by deleting the pod before you have copied the previous logs, because a new pod from the ReplicaSet starts with a clean history.
When the container writes nothing useful to stdout, set terminationMessagePolicy: FallbackToLogsOnError. The termination message documentation states that Kubernetes then uses the last chunk of container log output, at most 2,048 bytes or 80 lines, whichever is smaller, as the termination message when the container exits with an error. That text appears in lastState.terminated.message and survives log rotation, which makes it valuable for crashes that happen before your logging pipeline ships anything.
Events give you the killer’s identity
Events are where the kubelet explains its own actions. A BackOff event with the message “Back-off restarting failed container” confirms a Kubernetes CrashLoopBackOff. An Unhealthy event followed by “failed liveness probe, will be restarted” confirms that the kubelet, not the application, ended the process. An Evicted status with “The node was low on resource: memory” means node-pressure eviction, which is a different mechanism with a different fix.
Events expire, typically after about an hour by default on the API server, so collect them early. Sort them by timestamp, because the default order is not chronological. The triage script later in this guide does both.
Walk-through: Exit Codes, Repro Manifests and a Triage Script
Exit codes are the main branching variable for any Kubernetes CrashLoopBackOff. They follow POSIX shell conventions, and the container runtime passes them through. Values from 1 to 125 are the program’s own choice. Values 126 and 127 come from the shell or exec layer. Values above 128 usually mean “killed by signal N”, where the code is 128 + N. That arithmetic underpins every exit code 137 diagnosis.
| Exit code | Meaning | Typical Kubernetes cause | First check |
|---|---|---|---|
| 0 | Clean exit | A one-shot process in a Deployment, which restarts on any exit | Workload type; use a Job |
| 1 | Generic application error | Unhandled exception, failed config parse, refused dependency | logs --previous |
| 2 | Misuse of shell builtin or CLI usage error | Bad flags or arguments in command/args |
Rendered args in the spec |
| 126 | Command found but not executable | Missing execute bit, wrong architecture, noexec mount | Image build and securityContext |
| 127 | Command not found | Wrong entrypoint path, shell missing in distroless image | command vs image contents |
| 137 | 128 + 9, SIGKILL | Cgroup OOM kill, or kill after a probe-driven grace period | lastState.reason and events |
| 139 | 128 + 11, SIGSEGV | Native crash in C extension, JNI, musl vs glibc mismatch | Core dump, native libs |
| 143 | 128 + 15, SIGTERM | Graceful stop after probe failure, eviction or rollout | Events for Killing |
An application can call exit(137) itself, and a wrapper script can turn any signal into exit 1. Some runtimes also report a separate StartError reason when the entrypoint cannot be started at all. Treat the code as a hypothesis and confirm it with the reason and the events.
Reproduction manifests you can run
The fastest way to learn each Kubernetes CrashLoopBackOff signature is to produce it deliberately in a scratch namespace. The manifests below use public images. The memory test uses the same polinux/stress image as the official Kubernetes memory-limit tutorial, and the rest use busybox.
# crashloop-lab.yaml - kubectl create ns crashlab && kubectl apply -n crashlab -f crashloop-lab.yaml
apiVersion: v1
kind: Pod
metadata:
name: exit1-config
spec:
containers:
- name: app
image: busybox:1.36
command: ["sh", "-c", "echo 'FATAL: DATABASE_URL not set' >&2; exit 1"]
terminationMessagePolicy: FallbackToLogsOnError
---
apiVersion: v1
kind: Pod
metadata:
name: exit127-entrypoint
spec:
containers:
- name: app
image: busybox:1.36
command: ["sh", "-c", "/app/server --port 8080"]
---
apiVersion: v1
kind: Pod
metadata:
name: oom-137
spec:
containers:
- name: app
image: polinux/stress
resources:
requests: {memory: "50Mi"}
limits: {memory: "100Mi"}
command: ["stress"]
args: ["--vm", "1", "--vm-bytes", "250M", "--vm-hang", "1"]
---
apiVersion: v1
kind: Pod
metadata:
name: liveness-kill
spec:
containers:
- name: app
image: busybox:1.36
command: ["sh", "-c", "sleep 3600"]
livenessProbe:
exec:
command: ["cat", "/tmp/healthy"]
initialDelaySeconds: 5
periodSeconds: 5
failureThreshold: 3
---
apiVersion: v1
kind: Pod
metadata:
name: init-fail
spec:
initContainers:
- name: wait-for-db
image: busybox:1.36
command: ["sh", "-c", "nslookup db-does-not-exist.crashlab.svc.cluster.local && exit 0 || exit 1"]
containers:
- name: app
image: busybox:1.36
command: ["sh", "-c", "sleep 3600"]
Watch them with kubectl get pods -n crashlab -w. Within a few minutes you will see exit1-config and exit127-entrypoint cycle through Error and CrashLoopBackOff, oom-137 cycle through OOMKilled, liveness-kill fail its probe about 20 seconds after each start and then wait out the default 30-second termination grace period before being killed, and init-fail sit in Init:Error and Init:CrashLoopBackOff. The main container of init-fail never starts, which is the point: an init crash loop blocks the whole pod.
The liveness case is instructive. PID 1 in this container, the shell or the sleep it runs, has no SIGTERM handler, and the kernel does not apply default signal actions to PID 1 in a namespace, so it ignores SIGTERM; so after the termination grace period the runtime sends SIGKILL. The record then shows exit code 137 with reason Error, not OOMKilled. That pair is the signature of a probe kill. A well-behaved application that handles SIGTERM would instead show 143.
A triage script that collects everything in one pass
The script below gathers the Kubernetes CrashLoopBackOff evidence from steps 1 to 5 for one pod and writes it to a timestamped folder. It uses only kubectl and standard shell tools. Run it before anyone deletes or rolls the pod.
#!/usr/bin/env bash
# crashtriage.sh <namespace> <pod> - snapshot evidence for a crash-looping pod
set -euo pipefail
NS="${1:?namespace}"; POD="${2:?pod}"
OUT="triage-${POD}-$(date -u +%Y%m%dT%H%M%SZ)"; mkdir -p "$OUT"
echo "== container status summary ==" | tee "$OUT/summary.txt"
kubectl get pod "$POD" -n "$NS" -o jsonpath='{range .status.initContainerStatuses[*]}init/{.name}{"\t"}restarts={.restartCount}{"\t"}waiting={.state.waiting.reason}{"\t"}last={.lastState.terminated.reason}/{.lastState.terminated.exitCode}{"\t"}{.lastState.terminated.startedAt}->{.lastState.terminated.finishedAt}{"\n"}{end}{range .status.containerStatuses[*]}{.name}{"\t"}restarts={.restartCount}{"\t"}waiting={.state.waiting.reason}{"\t"}last={.lastState.terminated.reason}/{.lastState.terminated.exitCode}{"\t"}{.lastState.terminated.startedAt}->{.lastState.terminated.finishedAt}{"\n"}{end}' \
| tee -a "$OUT/summary.txt"
echo "== termination messages ==" >> "$OUT/summary.txt"
kubectl get pod "$POD" -n "$NS" -o jsonpath='{range .status.containerStatuses[*]}{.name}: {.lastState.terminated.message}{"\n"}{end}' >> "$OUT/summary.txt"
kubectl get pod "$POD" -n "$NS" -o yaml > "$OUT/pod.yaml"
kubectl describe pod "$POD" -n "$NS" > "$OUT/describe.txt"
kubectl get events -n "$NS" --field-selector "involvedObject.name=$POD" \
--sort-by=.lastTimestamp > "$OUT/events.txt" || true
for c in $(kubectl get pod "$POD" -n "$NS" -o jsonpath='{.spec.initContainers[*].name} {.spec.containers[*].name}'); do
kubectl logs "$POD" -n "$NS" -c "$c" --previous --timestamps > "$OUT/logs-$c-previous.txt" 2>&1 || true
kubectl logs "$POD" -n "$NS" -c "$c" --timestamps --tail=500 > "$OUT/logs-$c-current.txt" 2>&1 || true
done
NODE=$(kubectl get pod "$POD" -n "$NS" -o jsonpath='{.spec.nodeName}')
echo "node=$NODE" >> "$OUT/summary.txt"
kubectl describe node "$NODE" | sed -n '/Conditions:/,/Addresses:/p' > "$OUT/node-conditions.txt" || true
kubectl top pod "$POD" -n "$NS" --containers > "$OUT/top.txt" 2>&1 || echo "metrics API unavailable" > "$OUT/top.txt"
grep -E "OOMKilled|Evicted|liveness|startup probe|BackOff|Killing" "$OUT/events.txt" "$OUT/describe.txt" | sort -u > "$OUT/signals.txt" || true
echo "Evidence saved to $OUT"; cat "$OUT/summary.txt"
The summary shows container name, restart count, current waiting reason, last terminated reason and exit code, and the start-to-finish interval. The signals.txt file greps for the event phrases that identify the killer. The script deliberately captures node conditions too, because a MemoryPressure=True condition changes the diagnosis from “this container is too big” to “this node is overcommitted.”
What the backoff timeline costs you

Figure 2: The default restart backoff ladder, the 300-second cap, and the per-node override.
Figure 2 shows the default ladder. The first restart is immediate. The next waits are 10, 20, 40, 80 and 160 seconds, and then every restart waits the 300-second cap. A clean run of 10 minutes resets the ladder. The dotted edge shows that maxContainerRestartPeriod, set per node, lowers the cap.
The arithmetic matters when a Kubernetes CrashLoopBackOff is part of an incident. Using the documented defaults, a container that crashes instantly accumulates 310 seconds of waiting across its first six restarts, then settles at about one attempt every five minutes, or roughly 12 per hour. This is our own derivation from the documented schedule, not a measured benchmark. It means a fix you push at minute 20 of an incident can take up to five minutes to be picked up by an existing pod. Deleting the pod, or running kubectl rollout restart on the Deployment, skips the wait because a new pod starts with a fresh backoff counter.
If you lower the node cap to 60 seconds, the same container settles at about 60 attempts per hour. That is faster recovery from transient dependency outages, at the cost of five times as many image-layer checks, container creations and log lines per crashing container. The documentation is explicit that if you set the cap below the 10-second initial delay, the initial delay becomes the cap. With the alpha ReduceDefaultCrashLoopBackOffDecay gate enabled, the defaults become 1 second initial and 60 seconds maximum, and per-node configuration still takes precedence.
OOMKilled in Kubernetes: cgroup v2, memory.events and Runtime Heaps
An OOMKilled Kubernetes container is the most common cause of exit code 137, and the most commonly misdiagnosed. To fix it properly you need to know which component made the decision, what it measured, and how the language runtime inside the container sizes itself.

Figure 3: Three paths that end a container under memory pressure, and the signal each leaves behind.
Figure 3 separates three paths. The first path is the container exceeding its own memory.max, which triggers the kernel OOM killer inside that cgroup and produces OOMKilled. The second path is the node’s available memory falling below the kubelet’s eviction threshold, which produces an Evicted pod. The third path is the node running out of memory before the kubelet reacts, in which case the global OOM killer picks a victim using oom_score_adj.
What memory.max actually counts
On cgroup v2, the container’s resources.limits.memory becomes the cgroup’s memory.max. The kernel cgroup v2 documentation defines the sequence. When usage is about to exceed memory.max, the kernel first attempts direct reclaim. If reclaim fails, the cgroup goes into OOM state and the OOM killer acts within that cgroup.
The charge is not just your heap. It includes anonymous memory for all processes, kernel memory attributed to the cgroup, and page cache. Clean page cache is reclaimable, so it rarely causes an OOM kill by itself. Two things that look like disk are not reclaimable this way, though. Files written to an emptyDir with medium: Memory live in tmpfs and count against the container’s memory limit. Dirty pages waiting for writeback can also push a write-heavy container over the edge when storage is slow.
The kernel exposes counters in memory.events. The max field counts how many times usage was about to exceed the boundary. The oom field counts how many times an allocation was about to fail at the limit. The oom_kill field counts processes killed by any OOM killer, and oom_group_kill counts group OOM events. A high max count with zero oom_kill tells you the container lives at its limit and survives by reclaiming cache, which usually shows up as latency rather than crashes.
Group OOM kill and why reason can be missing
Since Kubernetes 1.28 the kubelet sets memory.oom.group to 1 on cgroup v2 container cgroups. The kernel treats the cgroup as an indivisible workload, so all its tasks are killed together or not at all. For a single-process container this changes nothing. For a container that runs a supervisor with worker processes, such as gunicorn, uWSGI, PHP-FPM, a Node cluster master, or a shell wrapper plus app, it changes a lot.
Under the old per-process behaviour, the OOM killer usually picked the largest worker. The master survived and forked a replacement, so the container never exited, restartCount stayed at zero, and the only evidence was a kernel log line and a dropped request. Under group kill, the whole container dies, lastState.reason reads OOMKilled, and the pod enters a Kubernetes CrashLoopBackOff if it happens repeatedly. That is louder but more honest.
The singleProcessOOMKill kubelet option, added in 1.32, prevents the flag from being set. The kubelet configuration reference says the default on cgroup v2 Linux is false. Teams that turn it on regain the old “worker dies quietly” behaviour. If you inherit a cluster where OOM kills seem to have vanished, check this setting before celebrating.
Reading the counters on a live node
Kubernetes does not surface memory.events through the API, so you read it on the node. A kubectl debug node session with the sysadmin profile gives you a privileged pod with the host filesystem at /host:
kubectl debug node/<node-name> -it --profile=sysadmin --image=busybox:1.36
# inside the debug pod
chroot /host
POD_UID=$(crictl inspectp $(crictl pods --name <pod-name> -q) | grep -m1 '"uid"' | cut -d'"' -f4)
find /sys/fs/cgroup/kubepods.slice -type d -name "*${POD_UID//-/_}*"
cat /sys/fs/cgroup/kubepods.slice/<qos-slice>/<pod-slice>/memory.events
cat /sys/fs/cgroup/kubepods.slice/<qos-slice>/<pod-slice>/memory.max
journalctl -k --since "1 hour ago" | grep -i -E "oom|killed process"
The paths above are typical for nodes that use the systemd cgroup driver, where pod cgroups sit under kubepods.slice and Burstable or BestEffort sub-slices, and pod UIDs have dashes replaced by underscores. Verify the layout on your own distribution. Read the pod-level cgroup’s memory.events as well as the container’s. Each container restart creates a new container cgroup with fresh counters, while the pod cgroup persists for the pod’s lifetime, so its oom_kill total tells you how many kills happened since the pod was scheduled.
Language runtimes and the limit they think they have
Most “mysterious” OOM kills behind a Kubernetes CrashLoopBackOff are a mismatch between what the runtime believes its budget is and what memory.max allows.
JVM. Modern HotSpot JVMs detect container limits through UseContainerSupport. The default -XX:MaxRAMPercentage is 25 per cent, per the Java command reference. With a 1 GiB limit the maximum heap is about 256 MiB, which surprises teams who raised the limit expecting the heap to follow fully. The opposite failure is more dangerous. Setting MaxRAMPercentage=90 leaves only about 100 MiB for metaspace, code cache, thread stacks, direct buffers and GC structures, and the cgroup kills the process with no Java stack trace. A Java OutOfMemoryError and an OOMKilled container are different events. The former is the JVM refusing an allocation inside its heap. The latter is the kernel killing the process from outside. The illustrative budget below shows why heap percentages between roughly 50 and 75 are a common starting point.
| Component, 1 GiB limit (illustrative) | Approximate share |
|---|---|
| Java heap at MaxRAMPercentage=60 | ~614 MiB |
| Metaspace and code cache | 100–200 MiB depending on framework |
| Thread stacks, e.g. 200 threads at 1 MiB reserved | up to ~200 MiB, often less is committed |
| Direct and NIO buffers, GC overhead | tens of MiB, workload dependent |
These figures are an illustrative budget, not measurements. Measure your own with -XX:NativeMemoryTracking=summary and jcmd <pid> VM.native_memory summary.
Node.js. According to a Red Hat analysis of Node.js 20 in containers, Node reads the cgroup memory.max and sizes the default old-generation heap at about half the container’s memory, levelling off around 2 GiB for containers of 4 GiB and larger. A 1 GiB container gets roughly a 520 MiB heap. Native buffers, the Buffer pool and worker threads live outside that heap. A Node process that throws “JavaScript heap out of memory” has hit its own heap ceiling. One that disappears with exit 137 has hit the cgroup. Set --max-old-space-size explicitly to keep the boundary deterministic across image upgrades.
Python. CPython has no heap ceiling. It allocates until the kernel refuses, so a leaking Python service or a pandas job that loads a too-large file simply grows its resident set until the cgroup OOM killer ends it. Multi-worker servers multiply this: four gunicorn workers each loading a 300 MiB model need well over 1.2 GiB, because copy-on-write sharing after fork erodes as reference counts are written. Preload models before forking, cap workers explicitly, and use --max-requests to recycle workers that drift.
Go. Since Go 1.19 the runtime honours a soft memory limit through GOMEMLIMIT, described in the Go GC guide. Setting it to roughly 80 to 90 per cent of the container limit makes the garbage collector work harder before the kernel intervenes. It is a soft limit, so a genuine leak will still end in an OOM kill.
Why kubectl top lies to you during an OOM
kubectl top reads the Metrics API, which reports the container’s working set at the scrape interval of metrics-server. An allocation spike that rises from 400 MiB to the 1 GiB limit within a couple of seconds and gets killed is often invisible between two samples. The graph shows a flat 400 MiB line and then a gap. Use kubectl top to spot steady growth and leaks. Use memory.events, the kernel log, and container-runtime OOM events to diagnose sudden kills. Note also that the working set excludes inactive file cache, so kubectl top can legitimately read lower than the cgroup’s memory.current.
Probes, Init Containers and Other Restart Triggers

Figure 4: How a liveness probe failure becomes a container kill and a restart.
Figure 4 traces the most common self-inflicted Kubernetes CrashLoopBackOff. The kubelet runs the liveness probe. After the configured number of consecutive failures, three by default, it asks the runtime to stop the container. The runtime sends SIGTERM, waits for the grace period, then sends SIGKILL if the process is still alive. The container’s exit is recorded as 143 or 137 with reason Error. The kubelet then restarts it under the normal backoff ladder.
Liveness probes that kill slow starters
Probe defaults are aggressive, and they cause more Kubernetes CrashLoopBackOff incidents than most teams expect: periodSeconds 10, timeoutSeconds 1 and failureThreshold 3. A liveness probe with no initialDelaySeconds gives an application roughly 30 seconds to answer within a one-second timeout. A JVM service that warms a cache for 45 seconds will be killed every time it nearly finishes starting. That is a textbook Kubernetes CrashLoopBackOff with clean application logs and no exception.
The documented fix is a startup probe. The probe documentation recommends a startup probe with the same check and a failureThreshold × periodSeconds long enough to cover the worst-case startup time. In its example, 30 × 10 seconds gives the application up to five minutes. Liveness and readiness probes do not run until the startup probe succeeds. If the startup probe never succeeds, the container is killed and restarted under the pod’s restart policy.
A second classic is a liveness probe that checks a dependency. If /healthz queries the database and the database has a 30-second failover, every replica fails liveness at once, the kubelet restarts every replica, and the restarts add connection storms to the failover. Liveness should answer one question: “is this process wedged beyond self-recovery?” Dependency health belongs in readiness, which removes the pod from Service endpoints without killing it.
A third trap is the one-second timeout under CPU throttling. A container with a tight CPU limit that is throttled at the moment of the probe can miss a one-second deadline even though the application is healthy. The fix is a sensible timeout and CPU sizing, which our CFS quota and CPU throttling guide covers in depth.
Init containers block the whole pod
Regular init containers run sequentially, and each must succeed before the next starts. A failing init container restarts under the pod’s restartPolicy and the same backoff ladder. The pod status shows Init:Error or Init:CrashLoopBackOff, and the main containers never start, so kubectl logs <pod> without -c often shows nothing useful. Always pass -c <init-container-name> --previous.
Native sidecars are init containers with restartPolicy: Always. They ignore the pod-level restart policy and restart independently. A sidecar in Kubernetes CrashLoopBackOff, such as a log shipper, or mesh proxy can hold a pod in a degraded state even when the main application is healthy, and the pod’s displayed status may not make that obvious. Check initContainerStatuses, not only containerStatuses. Our explainer on native sidecar containers covers the ordering rules.
Exit code 0 is also a crash loop
A Deployment’s pod template only allows restartPolicy: Always. A container whose process finishes successfully, a migration script or a CLI that prints help and exits, gets restarted anyway and enters the backoff ladder with lastState.reason: Completed. The fix is not in the container; it is the workload type. Run one-shot work as a Job, or as an init container if it must precede the app.
Using kubectl debug when the container will not stay up
kubectl exec needs a running container, and a container in Kubernetes CrashLoopBackOff is running for seconds at a time. Ephemeral containers with --target share the process namespace of a running target, which is equally unhelpful when the target keeps dying. The documented answer is a pod copy with the command overridden:
# copy the pod, replace the crashing container's command with a shell
kubectl debug <pod> -n <ns> -it --copy-to=<pod>-debug --container=<container> -- sh
# copy the pod, keep the original image, add a tools container that shares processes
kubectl debug <pod> -n <ns> -it --copy-to=<pod>-debug --image=busybox:1.36 --share-processes --profile=general
# inside the copy: run the real entrypoint by hand and watch it fail
env | sort; ls -l /app; cat /proc/self/cgroup; /app/server --port 8080; echo "exit=$?"
With --copy-to, you get the same image, environment, volumes and service account, but without the restart loop. You can run the entrypoint by hand, see the exact error, and inspect mounted ConfigMaps and Secrets. Choose the debug profile deliberately. The documentation lists legacy, general, baseline, restricted, netadmin and sysadmin. It notes that legacy is the default when --profile is omitted and is planned for deprecation, and it recommends general. Namespaces enforcing Pod Security Standards need baseline or restricted. Custom profiles through --custom have been stable since 1.32.
A pod copy is not managed by the ReplicaSet and may not match Service selectors if you change labels, so delete it when you finish. An ephemeral container added to a running pod cannot be removed afterwards; it stays in the pod spec until the pod is deleted. Adding ephemeral containers also requires pods/ephemeralcontainers permission, which is deliberately separate from pods patch rights.
Restart Rules and Backoff Tuning in Kubernetes 1.35–1.37
Container restart rules, beta and enabled by default since 1.35, let a container declare its own restartPolicy and a list of restartPolicyRules. The only supported condition is exitCodes with an operator and values. The first matching rule wins, and otherwise the container’s own policy applies. In practice this lets you separate retryable from non-retryable exits. A worker can exit 42 to mean “transient, restart me” and exit 1 to mean “configuration is wrong, stop,” under a pod policy of Never. That turns a meaningless Kubernetes CrashLoopBackOff into a clean Failed pod that your controller or Job logic can act on.
The RestartAllContainers action, beta since 1.36, goes further. A matching exit terminates every container in the pod and restarts the pod in place, re-running init containers, while keeping the pod UID, IP, sandbox and volumes. The documentation is clear that this fast path does not honour terminationGracePeriodSeconds or run preStop hooks. It targets batch and ML workloads where rescheduling is expensive. It also depends on the NodeDeclaredFeatures gate, so check your nodes before relying on it.
Kubernetes CrashLoopBackOff tuning is an operator decision, not an application one. Lowering maxContainerRestartPeriod on a node pool that runs short-lived edge or CI workloads shortens recovery after transient failures. Leaving it at 300 seconds on general pools protects the node and the registry from a fleet of hot-looping containers. Remember that the setting is per node, so two replicas of the same Deployment can recover at different speeds if node pools differ.
Memory QoS, which graduated to beta in 1.37, uses cgroup v2 memory controls to protect and throttle memory before the hard limit. It complements, rather than replaces, correct limits. Our Kubernetes 1.37 Memory QoS guide explains the tiers.
Trade-offs, Gotchas, and What Goes Wrong
Raising the memory limit hides leaks. Doubling a limit after an OOMKilled event often turns a crash every hour into a crash every two hours, and you lose the signal. Before raising the limit, plot memory over several restart cycles. A sawtooth that climbs to the limit at a steady rate is a leak. A flat line with a sudden spike is a burst, such as a large request body or a batch load, and wants a code-level cap.
Removing limits moves the problem to the node. A container with no memory limit cannot be OOMKilled by its own cgroup, but it can push the node into eviction or a global OOM. The kubelet’s default hard eviction threshold on Linux is memory.available<100Mi. The node-pressure eviction documentation notes that hard evictions use a zero-second grace period. If the node runs out before the kubelet reacts, the kernel picks a victim by oom_score_adj: -997 for Guaranteed pods, 1000 for BestEffort, and a request-proportional value for Burstable pods. Your unbounded pod may survive while a neighbour dies.
Evicted pods do not crash-loop. An evicted pod moves to Failed with reason Evicted and is not restarted in place. Its controller creates a replacement elsewhere. If you see many Evicted pods and no OOMKilled, the fix is node capacity or requests, not limits. The kubelet does not use QoS class to order evictions; it uses whether usage exceeds requests, then priority, then usage relative to requests.
Probe kills masquerade as OOM. Exit 137 with reason Error after a fixed interval is a probe kill, not memory. Raising memory will not help, and it costs money.
Restarting is not recovering. A container with a corrupted on-disk state in a persistent volume will crash-loop forever, because each restart sees the same state. Backoff only spaces the failures out. A Kubernetes CrashLoopBackOff caused by data needs data repair, not restarts.
The --previous window is one restart deep. If the first crash had a different cause from the current one, for example an OOM followed by a corrupted lockfile, you will only see the latest. Ship logs off-node for anything you need to keep.
Readiness gating hides crash loops from users and from you. A Deployment with healthy old replicas and a new ReplicaSet stuck in Kubernetes CrashLoopBackOff looks fine from outside. It stalls quietly until progressDeadlineSeconds marks the rollout as failed. Alert on rollout status, not only on endpoint health.
Practical Recommendations
Treat a Kubernetes CrashLoopBackOff as a symptom with a fixed diagnostic order, and make the evidence easy to collect before anyone touches the pod. The biggest improvement most teams can make is cultural: nobody deletes a pod in Kubernetes CrashLoopBackOff until lastState, --previous logs and events are saved. The triage script above makes that a ten-second habit.
Next, make each failure class self-identifying. Enable FallbackToLogsOnError on every container, so the last lines of output land in the pod status. Give processes a real init, such as tini or dumb-init, or handle SIGTERM in the application, so a graceful stop shows 143 and a hard kill stands out as 137. Use distinct exit codes for “configuration invalid” and “dependency unavailable,” and, on 1.35 or later clusters, map them with restartPolicyRules.
Finally, size memory from the runtime outward. Set the runtime’s own ceiling explicitly, through MaxRAMPercentage, --max-old-space-size or GOMEMLIMIT, so the language runtime fails first with a stack trace and the cgroup is only the backstop. For broader shutdown hygiene, pair this with our guide to graceful shutdown, PreStop hooks and PodDisruptionBudgets.
A short checklist for every service:
- Alert on
restartCountincrease and on waiting reasonCrashLoopBackOff, not on pod phase. - Add a startup probe for anything that takes more than about 20 seconds to start.
- Keep liveness checks local to the process; move dependency checks to readiness.
- Set
terminationMessagePolicy: FallbackToLogsOnError. - Set runtime heap limits to roughly 50–75 per cent of the container limit for JVM and Node, and set
GOMEMLIMITfor Go. - Record
memory.eventsand kernel OOM logs in node-level monitoring. - Run one-shot work as Jobs or init containers, never as Deployment containers.
- Check
singleProcessOOMKillandmaxContainerRestartPeriodin your kubelet configuration.
Frequently Asked Questions
What does CrashLoopBackOff mean in Kubernetes?
CrashLoopBackOff is the reason shown on a container’s Waiting state while the kubelet is delaying a restart after repeated crashes. It is not itself an error. The kubelet restarts the first crash immediately, then waits 10, 20, 40 seconds and so on, capped at five minutes, and resets the timer after 10 minutes of clean running. The pod’s phase usually stays Running. To find the cause, read the container’s lastState.terminated reason and exit code, then the previous instance’s logs and the pod’s events.
What does exit code 137 mean in a Kubernetes pod?
Exit code 137 is 128 plus 9, meaning the process was killed with SIGKILL. In Kubernetes there are two common sources. If lastState.reason is OOMKilled, the kernel’s OOM killer ended the container because it exceeded its memory limit. If the reason is Error and events show a failed liveness probe, the kubelet stopped the container and the runtime sent SIGKILL after the grace period expired. Node-pressure eviction usually appears differently, as a Failed pod with reason Evicted. Always confirm with the reason field and events.
How do I see logs from a crashed container?
Run kubectl logs with the pod name, the container flag and –previous. That reads the log of the last terminated instance rather than the one currently waiting or starting. Only the most recent dead instance is kept, and deleting the pod removes those logs, so copy them first. For init containers, pass the init container’s name explicitly. For crashes that happen before logs are written, set terminationMessagePolicy to FallbackToLogsOnError so the final lines appear in the pod status as the termination message.
Why is my pod OOMKilled when kubectl top shows low memory?
kubectl top reports the working set sampled at the metrics-server scrape interval. A fast allocation spike can hit the limit and be killed between two samples, so the graph never shows it. The working set also excludes some file cache, while the cgroup limit counts more. Check the cgroup’s memory.events oom_kill counter on the node, the kernel log, and the lastState reason. Also check for tmpfs emptyDir volumes, which count against the container’s memory limit, and runtime heap settings that exceed the limit.
How can I debug a container that crashes too fast to exec into?
Use kubectl debug with –copy-to to create a copy of the pod and override the crashing container’s command with a shell. The copy has the same image, environment and volumes but does not run the failing entrypoint, so you can start it by hand and watch the error. Alternatively, add a tools container with –share-processes. Pick the general profile, or baseline or restricted in Pod Security namespaces. Delete the copy afterwards, because no controller manages it.
Can I change the CrashLoopBackOff delay?
Yes, with limits. Since Kubernetes 1.35 the KubeletCrashLoopBackOffMax feature is beta and enabled by default. Administrators can set crashLoopBackOff.maxContainerRestartPeriod in the kubelet configuration to any value from 1s to 300s, per node. The separate alpha gate ReduceDefaultCrashLoopBackOffDecay changes cluster defaults to a 1-second start and 60-second cap. There is no per-pod field for the delay. Container restart rules, beta since 1.35, let you choose which exit codes restart at all.
Further Reading
- Kubernetes CPU throttling, CFS quota and cgroup v2 limits: why tight CPU limits cause probe timeouts and latency.
- Graceful shutdown with PreStop hooks, SIGTERM handling and PodDisruptionBudgets: getting exit 143 instead of 137 on every stop.
- Kubernetes 1.37 Memory QoS and pod-level resource managers: memory throttling tiers that act before the OOM killer.
- In-place pod resize for vertical scaling without restarts: changing memory limits without triggering a restart.
- Kubernetes documentation, Pod Lifecycle: container restarts and backoff.
- Linux kernel documentation, Control Group v2: memory interface files.
By Riju — about
