Upgrade Best Practices

A field-tested checklist for moving a production cluster between versions without downtime. Work top to bottom: prepare, upgrade the control plane, then roll the nodes, verifying at each step.

How to Use This List

Treat the groups as phases in order, not a menu.

Group A happens days before the change window; groups B through D happen during it.

Each bullet leads with the practice in bold, then the reason. Adopt them as gates, and do not advance a phase until the previous one is green.

A - Prepare Before the Window

• Read the changelog and API removals. Every minor release lists removed and deprecated APIs. Audit your manifests and Helm charts against that list first.
• Rehearse on a non-prod cluster that mirrors prod. Match the same Kubernetes minor, CNI, CSI drivers, ingress, and Pod Security posture, then run the exact upgrade steps there.
• Scan for deprecated APIs in your Git manifests. Run a tool like pluto over rendered manifests and check the apiserver_requested_deprecated_apis metric on live traffic.
• Confirm add-on compatibility. CoreDNS, the CNI, CSI drivers, and metrics-server each have a supported version matrix - line them up with the target minor before you start.
• Back up cluster state. Snapshot etcd (or confirm your managed provider does) and export critical resources so you can recover if something goes wrong.

B - Upgrade the Control Plane First

• Always move the control plane before any node. The API server must be the highest version in the cluster; nodes may lag but never lead.
• Step through one minor at a time. Go 1.34 to 1.35 to 1.36 - the control plane refuses to skip minors, so do not try.
• Respect the version skew policy. The kubelet may trail the API server by up to three minors; controller-manager and scheduler by only one.
• Wait for control-plane health before proceeding. Confirm all API server, scheduler, and controller-manager instances are healthy before touching nodes.
• Let managed providers own the control plane. On EKS, GKE, and AKS the provider replaces control-plane components - your job is to trigger the bump and then rotate nodes.

C - Roll the Nodes Safely

• Replace nodes, do not patch them in place. Roll the pool onto a new image so every node is identical and free of drift.
• Provision surge capacity. Set max-surge (or maxUnavailable: 0 with surge) so new nodes join Ready before old ones drain - no capacity dip.
• Keep nodes close to the control plane. The three-minor skew window is a scheduling limit, not a target; rotate nodes soon after the control-plane bump.
• Match the node image to the control-plane minor. Use the OS image published for your Kubernetes version - never an image built for a newer minor.
• Tame the autoscaler during the roll. Pause aggressive Karpenter or Cluster Autoscaler consolidation so it does not terminate your surge nodes mid-upgrade.

D - Protect Workloads During Drains

• Set a PodDisruptionBudget for every critical workload. Use minAvailable or maxUnavailable so drains cannot evict all replicas at once.
• Avoid impossible budgets. A maxUnavailable: 0 PDB blocks every eviction and hangs the drain - keep budgets realistic.
• Handle SIGTERM and size the grace period. Catch SIGTERM, fail readiness, drain connections, and set terminationGracePeriodSeconds to cover your slowest request.
• Run at least two replicas. A PDB cannot protect a single-replica app during a drain; scale to two or more first.
• Pass --ignore-daemonsets on manual drains. DaemonSet pods are node-bound and cannot be evicted, so drain requires you to acknowledge them.

E - Verify and Keep an Exit

• Verify versions on every node. Confirm kubectl get nodes -o wide shows the expected kubelet version and all nodes Ready.
• Smoke-test the workload. Check ingress, DNS, and a few critical request paths before declaring the upgrade done.
• Keep a rollback path for nodes. Prefer blue/green node groups on risky changes so you can shift back to the old pool instantly.
• Watch metrics through the window. Track error rate, latency, and pod restarts during and after the roll, not just at the end.
• Record what you did. Note the versions, add-on bumps, and any surprises so the next upgrade is faster and safer.

When You Are Done

Every node reports the target kubelet version and Ready, and no workload dropped below its PodDisruptionBudget during the roll.

Deprecated APIs are gone from both manifests and live traffic, and all add-ons are on compatible versions.

You have a written record and, for risky changes, a proven rollback path still available.

FAQs

What is the single most important rule? Upgrade the control plane before any node, and never let a node run a newer version than the API server.

Why rehearse on a separate cluster? Because add-on and API incompatibilities surface there safely, before they can break production.

Can I skip minor versions to save time? No for the control plane. Step through each minor; only nodes may jump because they are replaced.

How do I avoid a capacity dip? Provision surge so new nodes join before old ones drain, and set realistic PodDisruptionBudgets.

How do I know the upgrade succeeded? All nodes Ready at the target version, no PDB breaches, clean smoke tests, and stable error and latency metrics.

What is the safest option for a risky upgrade? Blue/green node groups - keep the old pool until the new one is proven so rollback is instant.

Related

• How a Living Cluster Gets Upgraded
• Upgrades Basics
• Node Image Updates
• Cordon & Drain
• Surge Node Pools

Stack versions: This page was written for Kubernetes 1.36.2, Docker Engine 29.6.1 (BuildKit default), containerd (CRI runtime on nodes), Helm 3, Compose v2, Argo CD (latest - verify at build), and Gateway API (GA - verify controller support at build).