Platform Delivery Best Practices
Practices for running a Kubernetes platform as a product with published SLOs, honest error budgets, and a change cadence tenants can plan around.
How to Use This List
Read the group that matches your current pain rather than working top to bottom.
Each practice states what to do and why it matters. Most are cheap to adopt individually.
If you are starting from nothing, do group A first. SLOs without measurement are opinions, and every later group depends on the numbers being real.
A - Measure Before You Promise
Instrument the three SLIs that tenants actually feel. API server availability, DNS resolution latency, and scheduling latency map directly to "can I deploy", "can my pods talk", and "do my pods come back". Everything else is diagnostics.
Probe the API server from outside the cluster. A Prometheus that discovers targets through the API server cannot record the API server being down. Add a blackbox prober against /livez so the outage is visible in the data.
Baseline for two weeks before setting a target. Measure your real p99 DNS latency and 5xx rate first, then set the SLO slightly tighter than observed. Targets invented in a planning meeting are missed within a month.
Start at 99.9%, not 99.99%. Three nines is roughly 43 minutes of budget over 30 days and is achievable on a managed control plane. Four nines exhausts permanently, and a permanently red budget is ignored.
Compute SLIs per cluster, never aggregated. Averaging a dev cluster into production hides real burn and produces a number that is true of nothing. Report each cluster separately.
Separate scheduling latency from pod startup time. scheduler_pod_scheduling_duration_seconds measures the scheduler's decision, not image pulls or init containers. If you promise "pods start fast", measure creation-to-Ready as its own SLI.
B - Make the Budget Govern Change
Publish the remaining budget as a single 0-to-1 number. A tenant asking whether to launch Friday needs one figure, not a dashboard of histograms. Record it as a rule so every consumer reads the same value.
Alert on multiwindow burn rate, not instantaneous error rate. Pair a short and a long window so both must agree before firing. A two-minute blip should never page; a sustained 14.4x burn always should.
Agree the freeze threshold before you need it. Decide in advance that below 25% remaining, discretionary platform change stops. Choosing the threshold during the argument is how budgets lose authority.
Wire the freeze to the pipeline, not to a person's conscience. Have the exhaustion alert patch a deny sync window onto the Argo CD AppProject. A freeze that depends on everyone remembering is not a control.
Always leave a documented bypass. Keep manualSync: true on deny windows and name who may use it. A bypass requiring a director's approval at 03:00 gets circumvented rather than used.
Never exclude "not our fault" outages from the budget. A cloud provider control plane failure still broke your tenants' deploys. Budgets that track blame instead of experience stop predicting anything.
C - Bound the Blast Radius of Cluster Change
Require a PodDisruptionBudget on every tenant workload. Without one, a node drain evicts pods with no protection at all. Enforce presence with an admission policy rather than documentation.
Audit for PDBs that can never be satisfied. minAvailable: 100%, or any PDB on a single-replica Deployment, makes kubectl drain hang until timeout. These are found during the upgrade unless you look for them first.
Turn on surge upgrades so capacity never dips. Adding new nodes before draining old ones removes the capacity risk, though the eviction itself still happens. This is what eventually lets you widen the change window.
Treat add-on rollouts as disruptive too. A CNI upgrade drops packets during the DaemonSet roll, and a CoreDNS change surfaces as application bugs. These deserve the same window and canary treatment as a node rotation.
Set change windows in an explicit timezone. Argo CD sync windows and cloud maintenance policies default to UTC. A window written for "1 AM" can land mid-business-day in another region.
Never rely on a window for spot capacity. Preemption happens whenever the provider decides, regardless of your policy. Do not run window-sensitive workloads on spot and then blame the schedule.
Cap freeze duration and let the backlog be visible. A freeze extended "one more week" repeatedly produces one enormous, high-risk batch of changes. The extension is the risk, not the change.
D - Rehearse Failure Deliberately
Run a full cluster failover game day at least annually. Fail the primary cluster over to the secondary with the people who would actually be on call. A DR plan never executed is a document, not a capability.
Time the recovery and treat the number as the real RTO. The measured duration is your actual recovery objective, whatever the runbook claims. Publish it, including when it disappoints.
Rotate who runs the game day. If the same principal engineer always drives it, you have tested that engineer, not the platform. Recovery must survive them being on holiday.
Practice a node pool rotation under production load. Most drain failures only appear when real traffic and real PDBs are involved. A quiet-cluster rehearsal proves very little.
Feed game day findings into the budget policy. If the exercise burns budget, that is a real signal about your resilience. Record it rather than exempting the exercise.
Rehearse the rollback, not just the upgrade. Teams practise the change and improvise the reversal at 03:00. The rollback path is the one worth having muscle memory for.
E - Run the Platform as a Product
Publish SLOs and budgets where tenants can see them. A platform SLO visible only in the platform team's Grafana cannot settle a launch-week argument. Give it the treatment an external service's status page gets.
Announce disruptive change 48 hours ahead in the tenant channel. Predictability is most of what tenants want from a platform. Surprise maintenance costs more trust than the outage itself.
Track DORA metrics on platform change alongside app change. Change failure rate and time to restore should improve as the budget policy matures. If deployment frequency falls permanently, the SLO is mispriced.
Patch security regardless of the freeze. Define the severity threshold above which patching always proceeds and write it into the published policy. Otherwise every CVE reopens the debate.
Retire EOL versions on a schedule you control. Kubernetes 1.33 is EOL, and a managed control plane left there will be force-upgraded on the provider's timetable. Choose your window before the provider chooses one for you.
Review the budget weekly, even without automation. A standing look at the remaining fraction builds the trust that makes the automated freeze acceptable later. Start the ritual before you build the machinery.
When You Are Done
You should be able to answer "is it safe to change the cluster this week" with a number rather than a discussion.
Tenants should be able to see that number without asking you.
A drain should be routine, a failover should be rehearsed, and a freeze should have both a trigger and an exit.
FAQs
How many platform SLOs should we run? Three to five. Beyond that nobody remembers them and none of them drive a decision.
Do we need error budgets before change windows? No, and windows are easier to start with. But conditional freezes driven by a budget are strictly better than a fixed calendar, so treat windows as the stepping stone.
Is a yearly game day really enough? It is the floor, not the goal. Quarterly is better if your platform changes fast, but an annual exercise that actually happens beats a quarterly one that keeps slipping.
What if leadership overrides the freeze? Then you do not have a budget policy. Escalate once with the data, and if it is overridden routinely, stop maintaining the theatre.
Should tenants have their own SLOs too? Yes. Yours measures the API server, DNS, and scheduling their services depend on; theirs measures their service. Both are needed to locate a problem.
Related
- Platform SLOs
- Error Budgets for Platform
- Change Windows
- Platform Delivery Basics
- Treating the Platform as a Product
Stack versions: This page was written for Kubernetes 1.36.2, Docker Engine 29.6.1 (BuildKit default), containerd (CRI runtime on nodes), Helm 3, Compose v2, Argo CD (latest - verify at build), and Gateway API (GA - verify controller support at build).