How to Think During a Cluster Incident

Summary

A cluster incident is any degradation of user-facing service or platform capacity that needs a coordinated response, not just a single failing pod.
Insight: The first 15 minutes decide the outcome. Structured triage beats clever guessing, because guessing under adrenaline changes multiple variables at once and destroys your evidence.
Key Concepts: severity (SEV), incident commander (IC), blast radius, mitigation vs root cause, observe-orient-decide-act, change correlation.
When to Use: The moment alerts fire or users report errors and the cause is not obvious within a minute or two.
Limitations/Trade-offs: A heavy process slows a one-line fix. Match ceremony to severity - a SEV3 does not need a bridge call.
Related Topics: severity classification, kubectl triage order, rollback strategy, blameless postmortems.

Foundations

An incident is a mindset shift, not a Kubernetes state. Your job changes from building to restoring service and protecting evidence.

The core mental model is simple. Stabilize first, diagnose second, root-cause later.

Users care about mitigation - the error rate dropping. They do not care why yet. Restore service, then investigate calmly.

Severity sets the tempo. A useful default scale:

SEV1: major outage or data loss, all-hands, exec-visible.
SEV2: significant degradation, one user-facing feature down or heavy error rate.
SEV3: minor or contained impact, single tenant, or a workaround exists.

Severity is a decision, not a measurement. When unsure, round up and downgrade later once impact is clear.

Every incident needs one incident commander. The IC does not fix the problem - they coordinate, decide, and own communication.

Separating the IC from the hands-on responder prevents the classic failure where the person deepest in a terminal also tries to update stakeholders and misses both.

Mechanics & Interactions

The reasoning loop is observe, orient, decide, act. Run it deliberately.

Observe: gather signals before touching anything. What is the actual user symptom, and what changed recently?

Orient: form a hypothesis that a single cause could explain the signals. Resist the urge to fix the first anomaly you see.

Decide: pick the smallest action that tests the hypothesis or reduces impact.

Act: make one change, then re-observe. One variable at a time keeps the signal readable.

Change correlation is your highest-yield first question. Most incidents follow a deploy, a config change, a scaling event, or a certificate or credential expiry.

# What rolled out recently, and can we revert it?
kubectl rollout history deployment/api -n prod
kubectl get events -n prod --sort-by=.lastTimestamp | tail -30

A first-15-minutes triage order that works under pressure:

# 1. Scope: is this one workload or the whole cluster?
kubectl get pods -A --field-selector=status.phase!=Running
kubectl get nodes
 
# 2. Focus: the affected workload's pods and their reasons
kubectl get pods -n prod -l app=api -o wide
 
# 3. Evidence: why is a specific pod unhealthy?
kubectl describe pod <pod> -n prod
kubectl logs <pod> -n prod --previous

Read kubectl get pods reasons as a routing table. CrashLoopBackOff sends you to logs and exit codes. OOMKilled sends you to memory limits. ImagePullBackOff sends you to the registry and pull secrets. Pending sends you to scheduling and node capacity.

Mitigation options are usually faster than a fix. Roll back the last deploy, scale replicas, cordon a bad node, or fail over a dependency.

# Fastest mitigation for a bad rollout: revert to the known-good revision
kubectl rollout undo deployment/api -n prod

Communication runs in parallel, not after. A short status every 15 to 30 minutes - impact, current action, next update time - prevents a swarm of side-channel questions that distract responders.

Advanced Considerations & Applications

Blast radius shapes your ordering. A single failing Deployment is contained. A failing node, DNS, ingress controller, or the API server itself is a platform incident that can mask hundreds of downstream symptoms.

When many unrelated workloads fail at once, suspect shared infrastructure - CoreDNS, the CNI, the control plane, storage, or a cloud-provider outage - before you debug any single app.

Preserve evidence before you destroy it. Deleting a crashing pod for a fresh start also deletes the logs and state that explain the failure.

# Capture state before mutating anything
kubectl describe pod <pod> -n prod > /tmp/incident-pod.txt
kubectl logs <pod> -n prod --previous > /tmp/incident-logs.txt

Know when to stop diagnosing and just mitigate. If a rollback restores service, take it, then root-cause with a healthy system and no user pain.

The postmortem is part of the incident, not an afterthought. Keep a timestamped log of every action as you go - it becomes the timeline and prevents arguments about what actually happened.

Blameless framing is what makes people report the real sequence. The goal is resilient systems, not a responsible person.

Common Misconceptions

"The fastest fix is to start changing things." Uncoordinated changes multiply variables and hide the cause. One deliberate change beats five hopeful ones.

"Root cause comes first." Mitigation comes first. You root-cause after users are safe.

"Restarting the pod is diagnosis." A restart that masks a crash without capturing --previous logs throws away the only evidence you had.

"Only huge outages need an incident commander." Even a SEV3 benefits from one owner. The IC role scales down cheaply and prevents duplicated, conflicting actions.

"If it is broken, it must be Kubernetes." Often it is a recent app deploy, an expired credential, or an upstream dependency. Kubernetes is frequently the messenger, not the cause.

FAQs

What is the single most useful first command? kubectl get pods -A filtered to non-Running, plus kubectl get nodes. It tells you scope - one app or the whole cluster - in seconds.

Should I roll back before I understand the cause? If a recent deploy correlates with the failure and rollback restores service, yes. Understanding can wait until users are unblocked.

Who declares severity? Whoever notices first makes a provisional call. The incident commander confirms or adjusts it. Round up when unsure.

How often should I send updates? Every 15 to 30 minutes for a live SEV1 or SEV2, and always state when the next update will come.

When do I page more people? When your hypotheses are exhausted, when the blast radius is platform-wide, or when the fix needs an owner you do not have - do not wait to be a hero.

Do small teams need this? Yes. The roles collapse onto fewer people, but the observe-decide-act discipline and one-change-at-a-time rule matter even more when you are short-handed.

Stack versions: This page was written for Kubernetes 1.36.2, Docker Engine 29.6.1 (BuildKit default), containerd (CRI runtime on nodes), Helm 3, Compose v2, Argo CD (latest - verify at build), and Gateway API (GA - verify controller support at build).

How to Think During a Cluster Incident

Summary

A cluster incident is any degradation of user-facing service or platform capacity that needs a coordinated response, not just a single failing pod.
Insight: The first 15 minutes decide the outcome. Structured triage beats clever guessing, because guessing under adrenaline changes multiple variables at once and destroys your evidence.
Key Concepts: severity (SEV), incident commander (IC), blast radius, mitigation vs root cause, observe-orient-decide-act, change correlation.
When to Use: The moment alerts fire or users report errors and the cause is not obvious within a minute or two.
Limitations/Trade-offs: A heavy process slows a one-line fix. Match ceremony to severity - a SEV3 does not need a bridge call.
Related Topics: severity classification, kubectl triage order, rollback strategy, blameless postmortems.

Foundations

An incident is a mindset shift, not a Kubernetes state. Your job changes from building to restoring service and protecting evidence.

The core mental model is simple. Stabilize first, diagnose second, root-cause later.

Users care about mitigation - the error rate dropping. They do not care why yet. Restore service, then investigate calmly.

Severity sets the tempo. A useful default scale:

SEV1: major outage or data loss, all-hands, exec-visible.
SEV2: significant degradation, one user-facing feature down or heavy error rate.
SEV3: minor or contained impact, single tenant, or a workaround exists.

Severity is a decision, not a measurement. When unsure, round up and downgrade later once impact is clear.

Every incident needs one incident commander. The IC does not fix the problem - they coordinate, decide, and own communication.

Separating the IC from the hands-on responder prevents the classic failure where the person deepest in a terminal also tries to update stakeholders and misses both.

Mechanics & Interactions

The reasoning loop is observe, orient, decide, act. Run it deliberately.

Observe: gather signals before touching anything. What is the actual user symptom, and what changed recently?

Orient: form a hypothesis that a single cause could explain the signals. Resist the urge to fix the first anomaly you see.

Decide: pick the smallest action that tests the hypothesis or reduces impact.

Act: make one change, then re-observe. One variable at a time keeps the signal readable.

Change correlation is your highest-yield first question. Most incidents follow a deploy, a config change, a scaling event, or a certificate or credential expiry.

# What rolled out recently, and can we revert it?
kubectl rollout history deployment/api -n prod
kubectl get events -n prod --sort-by=.lastTimestamp | tail -30

A first-15-minutes triage order that works under pressure:

# 1. Scope: is this one workload or the whole cluster?
kubectl get pods -A --field-selector=status.phase!=Running
kubectl get nodes
 
# 2. Focus: the affected workload's pods and their reasons
kubectl get pods -n prod -l app=api -o wide
 
# 3. Evidence: why is a specific pod unhealthy?
kubectl describe pod <pod> -n prod
kubectl logs <pod> -n prod --previous

Mitigation options are usually faster than a fix. Roll back the last deploy, scale replicas, cordon a bad node, or fail over a dependency.

# Fastest mitigation for a bad rollout: revert to the known-good revision
kubectl rollout undo deployment/api -n prod

Communication runs in parallel, not after. A short status every 15 to 30 minutes - impact, current action, next update time - prevents a swarm of side-channel questions that distract responders.

Advanced Considerations & Applications

When many unrelated workloads fail at once, suspect shared infrastructure - CoreDNS, the CNI, the control plane, storage, or a cloud-provider outage - before you debug any single app.

Preserve evidence before you destroy it. Deleting a crashing pod for a fresh start also deletes the logs and state that explain the failure.

# Capture state before mutating anything
kubectl describe pod <pod> -n prod > /tmp/incident-pod.txt
kubectl logs <pod> -n prod --previous > /tmp/incident-logs.txt

Know when to stop diagnosing and just mitigate. If a rollback restores service, take it, then root-cause with a healthy system and no user pain.

The postmortem is part of the incident, not an afterthought. Keep a timestamped log of every action as you go - it becomes the timeline and prevents arguments about what actually happened.

Blameless framing is what makes people report the real sequence. The goal is resilient systems, not a responsible person.

Common Misconceptions

"The fastest fix is to start changing things." Uncoordinated changes multiply variables and hide the cause. One deliberate change beats five hopeful ones.

"Root cause comes first." Mitigation comes first. You root-cause after users are safe.

"Restarting the pod is diagnosis." A restart that masks a crash without capturing --previous logs throws away the only evidence you had.

"Only huge outages need an incident commander." Even a SEV3 benefits from one owner. The IC role scales down cheaply and prevents duplicated, conflicting actions.

"If it is broken, it must be Kubernetes." Often it is a recent app deploy, an expired credential, or an upstream dependency. Kubernetes is frequently the messenger, not the cause.

FAQs

What is the single most useful first command? kubectl get pods -A filtered to non-Running, plus kubectl get nodes. It tells you scope - one app or the whole cluster - in seconds.

Should I roll back before I understand the cause? If a recent deploy correlates with the failure and rollback restores service, yes. Understanding can wait until users are unblocked.

Who declares severity? Whoever notices first makes a provisional call. The incident commander confirms or adjusts it. Round up when unsure.

How often should I send updates? Every 15 to 30 minutes for a live SEV1 or SEV2, and always state when the next update will come.

When do I page more people? When your hypotheses are exhausted, when the blast radius is platform-wide, or when the fix needs an owner you do not have - do not wait to be a hero.

Do small teams need this? Yes. The roles collapse onto fewer people, but the observe-decide-act discipline and one-change-at-a-time rule matter even more when you are short-handed.

Stack versions: This page was written for Kubernetes 1.36.2, Docker Engine 29.6.1 (BuildKit default), containerd (CRI runtime on nodes), Helm 3, Compose v2, Argo CD (latest - verify at build), and Gateway API (GA - verify controller support at build).

How to Think During a Cluster Incident

Search Documentation

How to Think During a Cluster Incident