How a Living Cluster Gets Upgraded
Summary
- A Kubernetes upgrade is not a single event but a coordinated sequence that moves the control plane first, then the nodes, one minor version at a time.
- Insight: A running cluster is never fully "on" one version - it is a distributed system that tolerates a bounded amount of version skew between its components by design.
- Key Concepts: the version skew policy, control-plane-first ordering, immutable node replacement, cordon/drain, and PodDisruptionBudgets.
- When to Use: every time you move between Kubernetes minor versions (for example 1.35 to 1.36), or rotate node OS images for CVEs.
- Limitations/Trade-offs: you cannot skip minor versions on the control plane, and nodes must never run ahead of the API server.
- Related Topics: node image rotation, cordon and drain mechanics, surge capacity during rollouts.
Foundations
A Kubernetes cluster is a living system that serves traffic while you change it underneath.
You almost never take it down to upgrade. Instead you replace components in an order that keeps the whole thing serving.
The mental model: the control plane is the brain, the nodes are the muscle, and they are allowed to disagree about their version within strict limits.
Those limits are the version skew policy. It is what makes a rolling, zero-downtime upgrade possible.
The second pillar is the node-replacement model. Modern nodes are cattle, not pets - you do not patch them in place, you drain and replace them.
Mechanics & Interactions
The skew policy sets which components may lag which. The API server is always the highest version in the cluster.
kube-apiserver is the reference point. Every other component compares its version against it.
The kubelet may be up to three minor versions older than the API server, and must never be newer. So a 1.36 API server tolerates 1.33, 1.34, 1.35, and 1.36 kubelets.
kube-proxy on a node must match that node's kubelet minor version, and stay within the same three-version window of the API server.
kube-controller-manager, kube-scheduler, and cloud-controller-manager may be at most one minor older than the API server, and never newer.
kubectl is supported within one minor version of the API server in either direction.
This asymmetry is deliberate: nodes are allowed to lag far behind so you can upgrade the control plane fully before touching a single node.
Because the API server is the reference, you upgrade it first. Then the other control-plane components. Only then the nodes.
You also cannot jump minors on the control plane. Going from 1.34 to 1.36 means stepping through 1.35 first, letting each version settle.
Node replacement is the other half. To retire an old node you cordon it, mark it unschedulable, then drain it to evict its pods gracefully.
Draining respects PodDisruptionBudgets (PDBs), which cap how many replicas of a workload can be down at once. The eviction API blocks if a drain would violate a budget.
Once drained, the old node is deleted and a new node running the target version joins the pool. Repeat across the pool and the fleet rolls forward.
Advanced Considerations & Applications
On managed platforms (EKS, GKE, AKS) the provider upgrades the control plane for you when you bump the cluster version.
You still own the node upgrade. The managed control plane will happily run ahead of your old nodes - within the skew window - until you rotate them.
That skew window is a scheduling window, not a comfort zone. Run nodes as close to the control plane as your maintenance cadence allows.
Deprecated APIs are the sharpest edge. A minor bump can remove a beta API your manifests still reference, so check the removal list before you upgrade.
Add-ons matter as much as core. CNI, CoreDNS, CSI drivers, ingress controllers, and metrics-server each have their own supported version matrix against Kubernetes.
The safest pattern is to rehearse on a non-production cluster that mirrors production's add-ons and Pod Security posture, then promote the same steps.
For node rollouts, provisioning surge capacity first - new nodes join before old ones leave - avoids a capacity dip during the drain.
Autoscalers like Karpenter or the Cluster Autoscaler interact with drains, so pause or tune aggressive consolidation while a controlled upgrade is in flight.
Common Misconceptions
"An upgrade takes the cluster offline." No. Control-plane components and nodes are replaced incrementally while workloads keep serving.
"I can jump from 1.34 straight to 1.36." Not for the control plane. You must pass through each intervening minor version.
"Nodes can be newer than the control plane if I am in a hurry." Never. The kubelet must not exceed the API server version - upgrade the control plane first, always.
"Draining a node just deletes its pods." Draining uses the eviction API, which honors PodDisruptionBudgets and graceful termination, so it is safe rather than abrupt.
"Patch releases need the same care as minors." Patch upgrades within a minor (1.36.1 to 1.36.2) carry no API removals and are far lower risk, though you still roll them out progressively.
FAQs
How many minor versions can a node lag the control plane? Up to three. A 1.36 API server supports kubelets down to 1.33.
Which component do I upgrade first?
The kube-apiserver, then the rest of the control plane, then the nodes.
Can I skip a minor version? No for the control plane - step through each minor. Nodes can skip because they are replaced, not patched.
Who upgrades the control plane on EKS or GKE? The provider, when you bump the cluster version. You still rotate the nodes yourself.
What stops a drain from taking down my app? PodDisruptionBudgets. The eviction API refuses evictions that would breach the budget.
Do I need to check my manifests before upgrading? Yes. Removed and deprecated APIs are the most common upgrade break - audit them first.
Related
Stack versions: This page was written for Kubernetes 1.36.2, Docker Engine 29.6.1 (BuildKit default), containerd (CRI runtime on nodes), Helm 3, Compose v2, Argo CD (latest - verify at build), and Gateway API (GA - verify controller support at build).