Node Image Updates
Summary
- A node image update replaces the operating system image on your worker nodes - kernel, container runtime, and kubelet - by rolling in fresh nodes rather than patching in place.
- Insight: Node images are immutable artifacts. You do not
apt upgradea running node; you build or select a new image and roll the pool onto it. - Key Concepts: node OS images (AMI, Bottlerocket, OS SKU), launch templates, rolling node-group updates, cordon/drain, and CVE-driven rotation.
- When to Use: kernel and runtime CVEs, kubelet patch releases, base-image hardening, and routine node hygiene between Kubernetes upgrades.
- Limitations: rolling a pool costs capacity churn and evictions, so it must respect PodDisruptionBudgets and node headroom.
Recipe
Point the node group at a new image reference and let the platform roll it.
- Identify the target image (a newer AMI ID, Bottlerocket version, or AKS node image).
- Update the node group or launch template to that image.
- Trigger a rolling update so new nodes join and old nodes drain.
- Verify kernel, containerd, and kubelet versions on the fresh nodes.
Working Example
Rotate an EKS managed node group to the latest patched AMI for its Kubernetes version:
# Roll the node group to the newest AMI for its current K8s version
aws eks update-nodegroup-version \
--cluster-name prod \
--nodegroup-name workers \
--forceThe managed node group cordons, drains, and replaces nodes one batch at a time.
For self-managed nodes on an Auto Scaling group, point the launch template at a new AMI and roll:
aws ec2 create-launch-template-version \
--launch-template-name workers-lt \
--source-version 1 \
--launch-template-data '{"ImageId":"ami-0abc123newpatched"}'
aws autoscaling start-instance-refresh \
--auto-scaling-group-name workers-asg \
--preferences '{"MinHealthyPercentage":90,"InstanceWarmup":120}'MinHealthyPercentage: 90 keeps 90 percent of capacity serving while nodes cycle.
On Bottlerocket, updates are handled in-cluster by the update operator:
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: brupop-agent
namespace: brupop-bottlerocket-aws
spec:
template:
spec:
containers:
- name: brupop
image: bottlerocket-update-operator:latestThe operator cordons, drains, applies the new image, and reboots each node in turn.
Deep Dive
Why images are immutable
Modern node images bundle a pinned kernel, containerd, and kubelet into one artifact.
Replacing the whole node guarantees every node in the pool is identical, which removes configuration drift.
In-place patching reintroduces drift and leaves you unsure what is actually running where.
What a node image contains
The OS image ships the Linux kernel, systemd, containerd as the CRI runtime, and the kubelet.
Docker Engine is not on the node - containerd runs your pods via the CRI. Docker stays on the build and developer side.
Patching a runtime or kernel CVE therefore means selecting a newer image, not upgrading a package on a live host.
Image updates versus Kubernetes upgrades
A node image update can keep the same Kubernetes minor and only refresh the OS, kernel, and kubelet patch level.
A Kubernetes minor upgrade is a superset: it also moves the kubelet minor version, and must follow a control-plane-first order.
Keep the two mentally separate even when one triggers the other.
Controlling the roll
Every managed platform exposes surge and health knobs: max unavailable, max surge, or min healthy percentage.
These bound how many nodes drain at once so the workload keeps its capacity during the rotation.
The eviction phase honors PodDisruptionBudgets, so a poorly set PDB can stall the whole roll.
Gotchas
DaemonSet pods block a naive drain. Always pass --ignore-daemonsets; the platform handles this for you on managed rolls.
No surge headroom means an outage. If the pool is at capacity with no surge, draining a node has nowhere to reschedule its pods.
Stuck PDBs freeze the rotation. A budget of maxUnavailable: 0 makes eviction impossible - the roll hangs until you fix the budget.
Image and Kubernetes version must be compatible. Do not pick an AMI built for 1.37 on a 1.36 cluster; use the image published for your control-plane minor.
Local emptyDir data is lost on replacement. Drains with --delete-emptydir-data discard node-local scratch data - do not store anything durable there.
Alternatives
In-place OS patching (kured + unattended-upgrades). Applies package updates to running nodes and reboots them one at a time. Simpler tooling but reintroduces drift and does not update kubelet or containerd cleanly.
Full node-group replacement (blue/green). Stand up a new node group on the new image, shift workloads, then delete the old group. Highest safety and instant rollback, at the cost of double capacity during the cutover.
Bottlerocket update operator. Purpose-built in-cluster rotation for the Bottlerocket OS. Excellent when you standardize on that image, but tied to that OS.
Pick rolling node-group updates for routine hygiene, and blue/green when the change is risky or you need a clean rollback.
FAQs
Do I need a Kubernetes upgrade to patch a kernel CVE? No. A node image update can refresh the kernel and runtime while staying on the same Kubernetes minor.
Does the node still run Docker? No. Nodes run containerd via the CRI. Docker is for build and local dev, not the in-cluster runtime.
How do I avoid a capacity dip during the roll? Set a max-surge or min-healthy-percentage so new nodes join before old ones drain.
What happens to my pods when a node is replaced? They are evicted with graceful termination, subject to PodDisruptionBudgets, and rescheduled onto other nodes.
How often should I rotate node images? On CVE disclosure at minimum, plus a regular cadence (monthly is common) to stay near the latest patched image.
Related
- How a Living Cluster Gets Upgraded
- Upgrades Basics
- Cordon & Drain
- Surge Node Pools
- Upgrade Best Practices
Stack versions: This page was written for Kubernetes 1.36.2, Docker Engine 29.6.1 (BuildKit default), containerd (CRI runtime on nodes), Helm 3, Compose v2, Argo CD (latest - verify at build), and Gateway API (GA - verify controller support at build).