Change Windows

Summary

• A change window is an agreed period during which the platform team may perform disruptive cluster maintenance: control plane upgrades, node pool rotations, CNI or CSI changes, and add-on rollouts.
• Insight: change windows are not a substitute for safe change. They exist to bound the blast radius of changes that are inherently disruptive, and to make the disruption predictable to tenants.
• Key Concepts: maintenance window, freeze window, node drain, PodDisruptionBudget, surge upgrade, canary cluster, change advisory.
• When to Use: any multi-tenant cluster where tenants have their own release cadence and the platform's changes can evict their pods.
• Limitations: windows do not stop security patches, cloud provider maintenance, or spot reclamation. Treat them as policy, not as a technical guarantee.

Recipe

Publish the window as a calendar contract, then encode it in the tooling that actually performs the change.

1. Define two window types: a recurring maintenance window when disruptive change is allowed, and ad-hoc freeze windows when it is not.
2. Require every tenant workload to declare a PodDisruptionBudget so drains are survivable.
3. Configure the node upgrade mechanism to respect the window.
4. Gate GitOps sync of platform components on the window.
5. Announce the window and the intended change 48 hours ahead in the tenant channel.

Working Example

Start with the tenant-side contract. A drain cannot be safe unless the workload tells Kubernetes what it can tolerate.

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: checkout-api
  namespace: checkout
spec:
  minAvailable: 80%
  selector:
    matchLabels:
      app: checkout-api

A managed node group can then be told when it may recycle nodes. On GKE the window is declared on the cluster itself.

gcloud container clusters update prod-euw1 \
  --region europe-west1 \
  --maintenance-window-start 2026-07-19T01:00:00Z \
  --maintenance-window-end 2026-07-19T05:00:00Z \
  --maintenance-window-recurrence 'FREQ=WEEKLY;BYDAY=SA'

A freeze is expressed the same way, as an exclusion covering a launch period.

gcloud container clusters update prod-euw1 \
  --region europe-west1 \
  --add-maintenance-exclusion-name black-friday \
  --add-maintenance-exclusion-start 2026-11-24T00:00:00Z \
  --add-maintenance-exclusion-end 2026-12-02T00:00:00Z \
  --add-maintenance-exclusion-scope no_upgrades

For self-managed clusters, the equivalent is a drain performed by your own automation.

kubectl drain node-euw1-3 \
  --ignore-daemonsets \
  --delete-emptydir-data \
  --timeout=600s

The platform's own add-ons live in Git, so the window belongs in the sync policy too. Argo CD supports sync windows on an AppProject.

apiVersion: argoproj.io/v1alpha1
kind: AppProject
metadata:
  name: platform
  namespace: argocd
spec:
  sourceRepos:
    - 'https://github.com/acme/platform-addons.git'
  destinations:
    - namespace: '*'
      server: 'https://kubernetes.default.svc'
  syncWindows:
    - kind: allow
      schedule: '0 1 * * 6'
      duration: 4h
      applications:
        - '*'
      manualSync: true
    - kind: deny
      schedule: '0 0 24 11 *'
      duration: 192h
      applications:
        - '*'

The allow window opens Saturday at 01:00 for four hours. The deny window covers the launch freeze and wins over any overlapping allow.

Note manualSync: true. It lets an on-call engineer push an emergency fix during a deny window without editing the project, which is the escape hatch every freeze needs.

Deep Dive

What actually disrupts tenants

Control plane upgrades are usually non-disruptive to running pods. The API server goes away briefly, which breaks controllers and kubectl, but kubelets keep containers running.

Node upgrades are the disruptive part. Every node replacement is a drain, and a drain is an eviction of every tenant pod on it.

Add-on changes sit in between. A CNI upgrade can drop packets during a DaemonSet rollout, and a CoreDNS change can produce resolution failures that look like application bugs.

Windows versus surge

A surge upgrade adds new nodes before removing old ones, so capacity never dips. It reduces the need for a window but does not eliminate it, because the drain still evicts pods.

If your workloads are all stateless with healthy PDBs and fast readiness, you can widen the window toward "any time". Stateful sets, long-running jobs, and singleton leaders are what keep windows narrow.

Freeze windows and the error budget

A freeze is a blunt instrument. The better version is conditional: freeze automatically when the platform error budget is exhausted, and allow change freely when it is healthy.

That inverts the politics. Instead of arguing about whether Friday is safe, the budget decides.

Emergency changes

Every window policy needs a documented bypass for CVEs and active incidents. Write down who can invoke it and what record it leaves.

A bypass that requires a manager's approval at 03:00 is a bypass that will be skipped rather than used.

Gotchas

PDBs that block drains forever. minAvailable: 100% or a PDB on a single-replica Deployment makes kubectl drain hang until the timeout. Audit for PDBs that can never be satisfied.

Windows in local time. Cron schedules in Argo CD sync windows and cloud maintenance policies default to UTC unless a timezone is set. A window written for "1 AM" can land in the middle of a business day for a different region.

Spot and preemptible nodes ignore your window. Reclamation happens whenever the provider wants. Do not run window-sensitive workloads on spot capacity and then blame the window policy.

Cloud provider forced upgrades. Managed control planes are auto-upgraded when a minor goes out of support. Kubernetes 1.33 is EOL, so a cluster left there will be moved for you, window or not.

Freezes that never end. A freeze extended "just one more week" repeatedly produces a giant, high-risk batch of changes. Cap freeze duration and let the backlog be visible.

Alternatives

Continuous change with no window. Viable when every workload has a PDB, surge upgrades are on, and node rotation is routine. This is where mature platforms end up.

Canary clusters. Run the change on a non-production or low-traffic cluster first for a fixed soak period, then promote. This buys more safety than a window does and works well combined with one.

Blue/green node pools. Stand up a new pool at the new version, shift workloads with taints and labels, delete the old pool. More expensive, but the rollback is instant.

Per-tenant opt-in windows. Let each namespace declare its own tolerance and schedule drains accordingly. Powerful, but the scheduling complexity grows fast.

FAQs

Do control plane upgrades need a window? Usually not for running pods, but they do interrupt the API server. Anything that reconciles or scales during the upgrade will stall, so a window is still courteous.

Can I enforce a window technically rather than by policy? Partly. Cloud maintenance policies and Argo CD sync windows are real enforcement. Human-run kubectl is not, unless you remove direct cluster access.

How long should a window be? Long enough to finish the change and roll it back. If your node pool takes three hours to rotate, a two-hour window guarantees a half-finished cluster.

Should security patches respect the freeze? No. Define a severity threshold above which patching proceeds regardless, and make that part of the published policy.

What if a tenant has no PodDisruptionBudget? Their pods are evicted immediately on drain with no protection. Enforce PDB presence with an admission policy rather than a wiki page.

Related

• Platform SLOs
• Error Budgets for Platform
• Platform Delivery Basics
• Treating the Platform as a Product
• Platform Delivery Best Practices

Stack versions: This page was written for Kubernetes 1.36.2, Docker Engine 29.6.1 (BuildKit default), containerd (CRI runtime on nodes), Helm 3, Compose v2, Argo CD (latest - verify at build), and Gateway API (GA - verify controller support at build).