CrashLoopBackOff Triage

Summary

• CrashLoopBackOff is not an error. It is kubelet telling you a container keeps exiting and restarts are now being delayed by exponential backoff.
• Insight: The state name tells you nothing about the cause. The evidence lives in the previous container's logs and the pod's lastState.terminated block, not in the current one.
• Key Concepts: restart backoff, exit code, lastState.terminated, --previous logs, init containers, probe-induced restarts.
• When to Use: Any pod stuck restarting, whether after a deploy, a config change, or a secret rotation.
• Limitations: Backoff triage finds why the process exited. If the process exits because a dependency is down, you have found a symptom, not the root cause.

Recipe

Work in a fixed order. Do not skip to kubectl logs - it lies to you when the container has already restarted.

1. Get the pod's state and exit code: kubectl describe pod.
2. Read the previous container's logs with --previous.
3. Read the pod's events for scheduling, mount, and probe failures.
4. Classify by exit code, then fix the matching cause.

kubectl get pod -n prod -l app=checkout
kubectl describe pod -n prod checkout-7d9c8b6f4-x2klm
kubectl logs -n prod checkout-7d9c8b6f4-x2klm --previous
kubectl get events -n prod --sort-by=.lastTimestamp | tail -30

If the pod has several containers, name the one that is failing.

kubectl logs -n prod checkout-7d9c8b6f4-x2klm -c checkout --previous --tail=200

Working Example

A deploy goes out and the rollout stalls. Start with state, not logs.

$ kubectl get pods -n prod -l app=checkout
NAME                        READY   STATUS             RESTARTS      AGE
checkout-7d9c8b6f4-x2klm    0/1     CrashLoopBackOff   5 (48s ago)   4m12s

RESTARTS 5 with a recent timestamp means the backoff is active. Kubelet doubles the delay per restart - 10s, 20s, 40s, up to a 5 minute cap - and resets it once the container stays up for 10 minutes.

Now pull the structured evidence.

$ kubectl describe pod -n prod checkout-7d9c8b6f4-x2klm
...
    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       Error
      Exit Code:    1
      Started:      Mon, 16 Feb 2026 09:14:02 +0000
      Finished:     Mon, 16 Feb 2026 09:14:03 +0000
    Ready:          False
    Restart Count:  5

Exit code 1 and a one-second lifetime means the app started and rejected its own configuration. Read the previous container's output.

$ kubectl logs -n prod checkout-7d9c8b6f4-x2klm --previous
{"level":"fatal","msg":"config: STRIPE_API_KEY is required","ts":"2026-02-16T09:14:03Z"}

That is the answer. The env var comes from a Secret key that does not exist.

$ kubectl describe pod -n prod checkout-7d9c8b6f4-x2klm | grep -A3 Environment
    Environment:
      STRIPE_API_KEY:  <set to the key 'stripe_api_key' in secret 'checkout-secrets'>  Optional: false
 
$ kubectl get secret -n prod checkout-secrets -o jsonpath='{.data}' | tr ',' '\n'
{"stripe_key":"c2tfbGl2ZV8..."}

The Secret has stripe_key; the Deployment asks for stripe_api_key. Fix the reference and roll.

env:
  - name: STRIPE_API_KEY
    valueFrom:
      secretKeyRef:
        name: checkout-secrets
        key: stripe_key

kubectl rollout restart deployment/checkout -n prod
kubectl rollout status deployment/checkout -n prod --timeout=120s

Note the failure mode: a missing key with optional: false blocks the container from starting at all and shows CreateContainerConfigError, not CrashLoopBackOff. Here the key existed at deploy time and the app rejected an empty value, so the container ran and exited.

Deep Dive

Classify by exit code first

The exit code narrows the search space before you read a single log line.

• 1 - generic application error. The app started and chose to die. Read its logs.
• 2 - shell or CLI misuse. Usually a bad command/args or a missing flag.
• 125-127 - container runtime level. 127 is "executable not found", the classic wrong command or missing binary in a distroless image.
• 137 - SIGKILL. Check Reason: OOMKilled in lastState. If the reason is Error instead, something killed it externally.
• 139 - SIGSEGV, a segfault in the process.
• 143 - SIGTERM. The container was asked to stop and did. Often a probe-driven restart or eviction, not a crash.

kubectl get pod -o jsonpath gets you there without scrolling describe.

kubectl get pod -n prod checkout-7d9c8b6f4-x2klm \
  -o jsonpath='{.status.containerStatuses[*].lastState.terminated.exitCode}{"\n"}'

Probes cause "crashes" that are not crashes

A failing livenessProbe makes kubelet kill and restart the container. The app is fine; the probe is wrong.

Tell them apart by exit code and events. Probe kills produce a SIGTERM (143) and a visible event.

$ kubectl describe pod -n prod api-5f7b9c4d8-qq2wn | grep -i unhealthy
  Warning  Unhealthy  2m (x9 over 4m)  kubelet  Liveness probe failed: Get "http://10.4.2.19:8080/healthz": context deadline exceeded

The usual cause is a liveness probe with no initialDelaySeconds (or no startupProbe) against an app that needs 30 seconds to warm up. Kubelet kills it mid-boot, forever.

Use a startupProbe for slow starts rather than inflating the liveness delay.

startupProbe:
  httpGet: { path: /healthz, port: 8080 }
  failureThreshold: 30
  periodSeconds: 5
livenessProbe:
  httpGet: { path: /healthz, port: 8080 }
  periodSeconds: 10

That gives 150 seconds to start, then a tight 10 second liveness cadence afterwards.

Init containers fail silently in kubectl logs

If the pod shows Init:CrashLoopBackOff, the app container has never run. Logs default to the app container and return nothing useful.

kubectl logs -n prod checkout-7d9c8b6f4-x2klm -c wait-for-db --previous

When there are no logs at all

Some crashes leave nothing - a segfault, a distroless image with a bad entrypoint, or a config parse that dies before the logger initialises.

kubectl debug gives you a shell alongside the broken pod without changing it.

kubectl debug -n prod checkout-7d9c8b6f4-x2klm -it \
  --image=busybox:1.36 --target=checkout -- sh

The ephemeral container joins the pod's namespaces, so you can inspect the same filesystem view, mounts, and network. --target shares the process namespace with that container.

To test a fixed command without touching the live Deployment, copy the pod and override the entrypoint.

kubectl debug -n prod checkout-7d9c8b6f4-x2klm --copy-to=checkout-dbg \
  --container=checkout -- sleep 3600
kubectl exec -n prod -it checkout-dbg -- sh

On the node itself, crictl sees what kubelet sees.

crictl ps -a --name checkout
crictl logs --previous <container-id>

Gotchas

kubectl logs without --previous returns nothing or the wrong run. The current container may be in backoff and have no output yet. Always use --previous on a restarting pod.

Confusing CrashLoopBackOff with CreateContainerConfigError. The latter means the container never started - a missing Secret, ConfigMap, or key. There are no logs to read because there was no process.

Assuming an image rollback fixes it. If the cause is a rotated secret or a dead dependency, the previous image will crash too. Check whether the old ReplicaSet's pods are also failing before blaming the build.

Ignoring RESTARTS age. RESTARTS 400 (3d ago) is a pod that recovered days ago. Only a recent restart timestamp is an active incident.

Editing the live Deployment to add sleep infinity. That destroys the crash evidence and the rollout history. Use kubectl debug --copy-to instead.

Alternatives

Rollback first, diagnose after. If the crash started with a deploy, kubectl rollout undo deployment/checkout -n prod restores service in seconds. Take the exit code and previous logs first, then roll back - the evidence disappears when the ReplicaSet scales to zero.

Scale to zero and reproduce locally. Best for complex startup bugs where the crash is deterministic and not environment-dependent. Run the same image with docker run and the same env file.

Fix forward. Appropriate when the cause is a one-line config error and you have a fast pipeline. Riskier under time pressure than a rollback.

FAQs

Why does the pod restart slower over time? Kubelet applies exponential backoff to restarts - 10s, 20s, 40s, doubling to a 5 minute cap. The timer resets after the container runs successfully for 10 minutes.

Does restartPolicy: Never avoid CrashLoopBackOff? Yes, but only because the pod goes to Failed and stays there. For Deployments the policy is forced to Always, so this is only relevant for Jobs and bare pods.

Exit code 137 always means OOM? No. 137 is SIGKILL (128 + 9). Check lastState.terminated.reason - OOMKilled confirms memory; Error means something else killed the process.

Can I get logs from a pod that was deleted? Not with kubectl. This is why cluster log shipping matters - once the pod object is gone, kubelet's log files go with it.

Why does describe show no events? Events default to a 1 hour TTL. A pod crashing for a day may have no events left, though the restart count and lastState persist.

Related

• How to Think During a Cluster Incident
• Incident Basics
• OOMKilled & CPU Throttling
• ImagePullBackOff & Registry Outages
• Incident Response Best Practices

Stack versions: This page was written for Kubernetes 1.36.2, Docker Engine 29.6.1 (BuildKit default), containerd (CRI runtime on nodes), Helm 3, Compose v2, Argo CD (latest - verify at build), and Gateway API (GA - verify controller support at build).