Incident Response Best Practices
Practices for running Kubernetes incidents that end quickly and teach you something. Grouped by the phase of the incident they apply to.
How to Use This List
Read groups A and B before your next incident, not during one. Preparation is the only part you cannot do under pressure.
Groups C through E are the live response. They assume you already have the runbooks from group A.
Treat this as a review checklist for your existing runbooks. Anything you cannot find in yours is a gap.
A - Prepare Before the Page
Keep kubectl debug examples in the runbook, verbatim. Nobody recalls --target or --copy-to syntax at 3am. Paste the exact commands with a real namespace and pod name so responders only edit the identifiers.
# ephemeral container alongside a broken pod
kubectl debug -n <ns> <pod> -it --image=busybox:1.36 --target=<container> -- sh
# a copy with the entrypoint replaced, so the original keeps failing for evidence
kubectl debug -n <ns> <pod> --copy-to=<pod>-dbg --container=<container> -- sleep 3600
# a privileged pod on a NotReady node, host fs at /host
kubectl debug node/<node> -it --image=busybox:1.36Write runbooks per symptom, not per service. Responders arrive with a symptom - pods restarting, latency up, node gone. A runbook indexed by service is unfindable when you do not yet know which service is at fault.
Pre-authorize the mitigations. Decide now who may roll back, scale up, or disable a feature flag without approval. An escalation path that requires waking a director is a 20 minute outage extension.
Verify your access before you need it. A responder who discovers their kubeconfig is expired during a SEV1 has just become a second incident. Test read access to prod on a schedule.
Ship logs off the cluster. Kubelet's logs vanish with the pod. If your only log source is kubectl logs, every deleted pod erases its own postmortem.
Alert on symptoms, page on impact. Alert on the CFS throttling ratio and the memory working-set ratio; page only when users are affected. Paging on a resource metric trains people to ignore pages.
B - Instrument for the Incident You Will Have
Set memory requests equal to memory limits for latency-sensitive workloads. It buys Guaranteed QoS and makes eviction order predictable. Memory does not burst usefully - the gap only buys you an eviction at the worst moment.
Use a startupProbe for slow-starting apps. Inflating initialDelaySeconds on a liveness probe is the most common self-inflicted CrashLoopBackOff. A startup probe separates "still booting" from "hung".
Pin images by digest and use imagePullPolicy: IfNotPresent. A :latest tag silently forces Always, which converts every registry blip into an outage. Digests are content-addressed, so caching is safe.
Audit PodDisruptionBudgets against real replica counts. A PDB that can never be satisfied blocks every drain, and you will discover it while a node is dying. Check minAvailable against the Deployment's replicas.
Set GOMAXPROCS or MaxRAMPercentage from the limit, not the node. Runtimes that autodetect the node's 64 cores while holding a 500m quota throttle themselves into the ground.
Keep a restartPolicy-aware view of "recovered". RESTARTS 400 (3d ago) is history. Only a recent restart timestamp is an incident. Make your dashboards show restart age, not just count.
C - Triage in a Fixed Order
Establish blast radius before cause. One pod, one node, one namespace, or the cluster - that answer changes every subsequent step. Draining during a cluster-wide event turns a partial outage into a total one.
Read state before logs. kubectl describe pod gives you the exit code and lastState.terminated reason. kubectl logs without --previous on a restarting pod returns the wrong run, or nothing.
Classify by exit code before reading a line of output. 137 with Reason: OOMKilled is memory. 127 is a missing executable. 143 is a SIGTERM, usually a probe kill rather than a crash.
Trust the event message verbatim. A 401 and a 403 on an image pull have different fixes - credential versus registry ACL. Paraphrasing the error is how people fix the wrong thing.
Check whether the workload is actually down. Pods keep serving for the ~300 seconds before NoExecute eviction fires. A NotReady node is not automatically a user-facing outage.
Change one variable at a time. Under adrenaline the instinct is to bump the limit, restart the pod, and roll back at once. You will restore service and never know which one did it.
D - Mitigate Without Destroying Evidence
Capture evidence before you replace anything. Once the node is terminated or the ReplicaSet scales to zero, the answer is gone. Thirty seconds of describe and logs --previous saves the postmortem.
Never edit the live Deployment to add sleep infinity. It destroys the crash loop you were diagnosing and pollutes the rollout history. kubectl debug --copy-to gets you the same shell against an untouched original.
Prefer rollback to fix-forward under time pressure. kubectl rollout undo is seconds and well-tested. A one-line config fix through CI is 10 minutes and a fresh chance to be wrong.
Cordon before you drain. Cordon stops the bleeding without moving anything, which gives you room to think. Drain is the irreversible half.
Stop the churn during a registry outage. Pin the HPA, scale down the cluster autoscaler, and suspend GitOps sync. Running pods are fine; every new pod is the problem.
Relax a blocking PDB deliberately, never with --force. --force deletes unmanaged pods with nothing to recreate them. Editing the PDB is a reversible, visible decision.
E - Close the Loop
Separate mitigation from root cause in the timeline. "Error rate returned to zero at 09:42" and "we understood why at 14:10" are different events. Conflating them hides how long you were guessing.
Record which variable you changed and why. The postmortem question is never "what fixed it" - it is "did we raise the limit because we had evidence, or because we were out of ideas?"
Distinguish a leak from an undersized limit in writing. An OOM kill 20 seconds after start is sizing. One 3.5 hours in is a leak. Raising the limit on a leak converts a 4 hour restart cycle into a 12 hour one and closes the ticket falsely.
Feed every incident back into the runbook. If a responder had to invent a command, that command belongs in the runbook before the incident closes.
Keep postmortems blameless and specific. "Human error" is never a root cause. The system let a wrong secret key reach prod without failing the deploy - that is the finding.
Track repeat incidents as one problem. The third OOM this quarter is not three incidents. It is one unfixed leak with three pages attached.
When You Are Done
Your runbooks should let a responder who has never seen the service reach a correct mitigation. Test that claim with someone from another team.
Every practice above that you cannot demonstrate in your cluster today is a backlog item, not a belief.
FAQs
How much process is right for a small team? Match ceremony to severity. A SEV3 needs a ticket and a fix. A SEV1 needs an incident commander separate from the person in the terminal.
Should we always roll back first? If the symptom started with a deploy, yes - after capturing the exit code and previous logs. Rollback is fast and reversible; diagnosis is neither.
Who decides severity? The first responder, immediately, and they round up. Severity is a decision that sets tempo, not a measurement to get right.
How do we stop alert fatigue? Page on user impact, alert on resource symptoms. Anything that pages without a required human action is a bug in the alert.
Is kubectl debug safe in production?
The ephemeral container form is - it adds a container without restarting the pod, and cannot be removed once added. Gate it with RBAC and expect it in your audit log.
What belongs in a runbook versus a dashboard? The dashboard answers "is it happening". The runbook answers "what do I type next". Keep the commands in the runbook.
Related
- How to Think During a Cluster Incident
- Incident Basics
- CrashLoopBackOff Triage
- OOMKilled & CPU Throttling
- ImagePullBackOff & Registry Outages
- Node NotReady & CNI Failures
Stack versions: This page was written for Kubernetes 1.36.2, Docker Engine 29.6.1 (BuildKit default), containerd (CRI runtime on nodes), Helm 3, Compose v2, Argo CD (latest - verify at build), and Gateway API (GA - verify controller support at build).