Incident Basics

This page is a hands-on intro to running the first minutes of a Kubernetes incident: how to scope the blast radius, read pod states, capture evidence, and mitigate fast without breaking more.

Prerequisites

• kubectl matching your cluster minor (client skew is supported within one minor of the API server).
• Cluster on a supported minor: Kubernetes 1.36.2 here (1.34 to 1.36 supported; 1.33 is EOL).
• crictl on nodes for container-runtime triage (containerd via CRI runs the pods).
• Read access to logs and events, plus rollback rights for your Deployments.

# Confirm your context and versions before you touch a live cluster
kubectl config current-context
kubectl version

Basic Examples

1. Scope the incident

Decide in seconds whether this is one workload or the whole cluster.

kubectl get pods -A --field-selector=status.phase!=Running
kubectl get nodes

• Non-Running pods in one namespace point to an app problem.
• Non-Running pods spread across every namespace point to shared infrastructure.
• A NotReady node explains every pod scheduled onto it.
• Start wide, then narrow - never start by debugging a single pod.

2. Read the pod reason as a routing table

The STATUS column tells you which runbook to open.

kubectl get pods -n prod -o wide

• CrashLoopBackOff: the container starts then exits - go to logs and exit codes.
• OOMKilled: the kernel killed it for memory - go to limits and usage.
• ImagePullBackOff / ErrImagePull: registry or pull-secret problem.
• Pending: it cannot be scheduled - go to capacity, requests, taints.

3. Describe the pod for events

describe surfaces the recent events that explain the state.

kubectl describe pod <pod> -n prod

• Read the Events: section at the bottom first.
• Last State: Terminated shows the exit code and reason.
• Back-off restarting failed container confirms a crash loop.
• FailedScheduling names the exact scheduling constraint that failed.

4. Get the crash logs, including the previous container

A crashed container's logs live in the previous instance, not the current one.

kubectl logs <pod> -n prod --previous
kubectl logs <pod> -n prod -c <container> --tail=100

• --previous retrieves logs from the last terminated container.
• Without it you often see an empty or restarting container.
• Name the container with -c for multi-container pods.
• Pipe to a file to preserve evidence before the pod churns.

5. Check recent changes

Most incidents follow a change - a deploy, config edit, or scaling event.

kubectl rollout history deployment/api -n prod
kubectl get events -n prod --sort-by=.lastTimestamp | tail -30

• A recent rollout that correlates with the failure is your prime suspect.
• Events are namespaced and sorted here by time.
• Look for Scaled, Killing, Unhealthy, and BackOff reasons.
• Correlate the timestamp of the change with the start of the symptoms.

6. Mitigate with a rollback

If a bad deploy caused it, reverting is usually the fastest fix.

kubectl rollout undo deployment/api -n prod
kubectl rollout status deployment/api -n prod

• undo returns to the previous ReplicaSet revision.
• rollout status blocks until the healthy revision is available.
• Add --to-revision=<n> to target a specific known-good revision.
• Root-cause later, on a healthy system, with no user pain.

7. Capture evidence before mutating

Deleting a pod for a fresh start also deletes its logs and state.

kubectl describe pod <pod> -n prod > /tmp/incident-pod.txt
kubectl logs <pod> -n prod --previous > /tmp/incident-logs.txt

• Save describe and --previous logs before any restart.
• These files become the postmortem timeline.
• Snapshot node state too if a node is involved.
• Evidence gone is root cause gone.

Intermediate Examples

8. Attach an ephemeral debug container

When the image has no shell, kubectl debug gives you tools without rebuilding.

kubectl debug -it <pod> -n prod --image=busybox:1.36 --target=app -- sh

• debug injects an ephemeral container sharing the pod's namespaces.
• --target=app shares the process namespace of the app container.
• Ideal for distroless or scratch images with no shell.
• The ephemeral container is not restarted and leaves the pod spec unchanged.

9. Cordon and drain a bad node

Stop new pods landing on a suspect node, then move existing ones off.

kubectl cordon node-7
kubectl drain node-7 --ignore-daemonsets --delete-emptydir-data

• cordon marks the node unschedulable but leaves running pods.
• drain evicts pods so replacements schedule elsewhere.
• --ignore-daemonsets is required because DaemonSet pods do not drain.
• Uncordon later with kubectl uncordon node-7 once healthy.

10. Inspect the runtime on a node with crictl

When kubectl cannot see a container, drop to the CRI layer.

$ sudo crictl ps -a
$ sudo crictl logs <container-id>
$ sudo journalctl -u containerd -n 100 --no-pager

• crictl talks directly to containerd via the CRI socket.
• Use it when the kubelet or API path is unhealthy.
• journalctl -u containerd and -u kubelet show node-agent errors.
• Docker Engine is not the in-cluster runtime - do not look for docker ps on nodes.

Stack versions: This page was written for Kubernetes 1.36.2, Docker Engine 29.6.1 (BuildKit default), containerd (CRI runtime on nodes), Helm 3, Compose v2, Argo CD (latest - verify at build), and Gateway API (GA - verify controller support at build).