Cost Optimization Best Practices

How to Use This List

These are the practices that move a Kubernetes bill, roughly in the order they pay off.

Work top to bottom. Measurement comes first because every later decision depends on trusting the numbers.

Each group ends up somewhere concrete: a changed manifest, a NodePool, or a report someone owns.

Do not attempt all of them in one quarter. Two groups done properly beat six done partially.

A - Measure Before You Cut

Install allocation before you optimize. Deploy OpenCost or Kubecost and let it collect at least two weeks. Optimizing without per-namespace attribution is guessing with extra steps.

Enforce cost labels at admission. Require team, env, and cost-center on every namespace. Spend attributed to __unallocated__ is spend nobody will fix.

Track efficiency, not just dollars. Requested-vs-used ratio per namespace is the metric that survives traffic growth. Dollars alone punish the teams that are succeeding.

Reconcile against the billing export monthly. Allocation is a model. If it drifts far from the invoice, find out why before anyone makes a decision on it.

Separate idle from workload cost. Idle is the platform team's number; workload cost is the app team's number. Merging them means neither team knows what to fix.

Give every report an owner. A dashboard with no named human and no recurring meeting produces zero savings, reliably.

Set requests from measured P95 usage, not from a guess. Over-requesting is the single largest hidden cost in most clusters, because the scheduler reserves what you asked for whether you use it or not.

Run VPA in Off mode to get recommendations. It produces target values from real history without touching running pods. Read them, then set requests yourself.

Set memory limits equal to memory requests. Memory is incompressible, so a limit above the request just makes OOM kills less predictable and Guaranteed QoS unreachable.

Be careful with CPU limits. CPU is compressible and throttling is a real latency source. Many teams set CPU requests and skip CPU limits entirely on latency-sensitive services.

Reject pods with no requests. A request-less pod is invisible to bin-packing and to allocation. Enforce a LimitRange default or an admission policy.

Re-check requests after every major release. Requests set for last year's traffic shape are just as wrong as requests that were never measured.

C - Make Nodes Fit the Pods

Adopt Karpenter and let it choose the shape. Provisioning from actual pod requests beats picking the nearest pre-defined node group, and consolidation keeps the fit as workloads change.

Give NodePools a wide instance menu. Multiple families, generations, and sizes mean better prices and better spot availability. A narrow menu recreates the rigidity you left node groups to escape.

Enable WhenEmptyOrUnderutilized consolidation. Empty-only consolidation removes dead nodes but never fixes a fleet of half-full ones.

Set consolidateAfter longer than your rollout. Otherwise every deploy triggers a churn cycle you will end up blaming on Karpenter.

Use disruption budgets with a schedule. Freeze voluntary node churn during peak hours, and let it run freely overnight.

Expire nodes on a clock. expireAfter on the node template gives you a maximum node age and rolls AMI updates through without a bespoke rotation job.

D - Default Stateless Workloads to Spot

Taint spot nodes and make workloads opt in. Nothing should land on interruptible capacity by accident, least of all your control-plane add-ons.

Diversify across instance types and zones. Reclamation happens per pool. One instance type in one zone means one event takes your whole tier.

Run interruption handling. Karpenter does it natively with an interruption queue; the AWS Node Termination Handler is the standalone equivalent. Without it, nodes disappear mid-request.

Keep pod grace periods inside the notice window. A 5-minute terminationGracePeriodSeconds cannot drain inside a 2-minute warning, so the work dies anyway.

Handle SIGTERM in the application. Stop accepting work, finish or requeue what is in flight, exit. A drained pod that dropped its request did not actually succeed.

Keep an on-demand floor. Spot pools run dry, always during the surge that created the demand. Size on-demand to your baseline and let spot absorb the rest.

Never put singleton stateful pods on spot. Zonal volumes cannot follow a pod to a new zone, and repeated failovers cost more in incidents than the compute saves.

E - Cut the Costs That Are Not Compute

Consolidate load balancers behind one ingress or Gateway. A Service of type LoadBalancer per app means a cloud load balancer per app, each with its own hourly charge.

Keep chatty traffic in-zone. Cross-AZ egress is billed per gigabyte and hides inside service-to-service calls. Topology-aware routing helps where latency allows.

Right-size and clean up volumes. Over-provisioned PVCs bill for the full size, and PVCs from deleted namespaces frequently outlive them.

Set log and metric retention deliberately. Observability spend routinely rivals compute. Sample aggressively and drop high-cardinality labels nobody queries.

Garbage-collect registry images. Untagged layers accumulate forever, and per-gigabyte storage plus egress on pulls is a real line item.

Scale non-production to zero off-hours. Dev and staging clusters running 168 hours a week for 40 hours of use are the easiest saving on this list.

F - Buy the Baseline, Rent the Peak

Commit to your floor. Savings Plans, Reserved Instances, or CUDs on the capacity you provably always run is free money with no reliability cost.

Right-size before you commit. Committing to an over-provisioned baseline locks in the waste for one to three years.

Use ARM where the workload allows. Graviton-class instances are meaningfully cheaper per core, and a multi-arch image build makes the move mostly mechanical.

Review commitments quarterly. Traffic shape and instance families both move. A commitment set and forgotten drifts into a bad deal.

When You Are Done

You should be able to name your three most expensive namespaces and say why each costs what it does.

Your requested-vs-used ratio should be within roughly 30% for production namespaces, not the 3x that clusters drift toward.

Node count should track workload demand rather than sit at a high-water mark set during last quarter's incident.

Stateless workers should be on spot by default, and interruptions should show up in the drain logs rather than in a postmortem.

Most importantly, cost should be a number a team sees weekly and can influence directly, rather than a surprise the platform team absorbs at month end.

FAQs

What single change saves the most? Right-sizing requests, in nearly every cluster. Over-requesting inflates node count, and node count is the bill.

Should I do spot or right-sizing first? Right-sizing. Spot makes waste 70% cheaper; right-sizing removes it. Do both, in that order.

Is Karpenter worth it on a small cluster? Below roughly 20 nodes the savings are modest, but the operational win of deleting node groups often justifies it anyway.

How do I get teams to care about cost? Showback with a named owner and a weekly number. Chargeback only after the numbers have gone unchallenged for a quarter.

Do CPU limits save money? No. Requests reserve capacity and drive cost; limits only cap a container's ceiling and can throttle it. Set requests carefully, use CPU limits sparingly.

How often should requests be revisited? Quarterly, plus after any major release or traffic-shape change. Treat VPA recommendations as the input to a human decision.

Stack versions: This page was written for Kubernetes 1.36.2, Docker Engine 29.6.1 (BuildKit default), containerd (CRI runtime on nodes), Helm 3, Compose v2, Argo CD (latest - verify at build), and Gateway API (GA - verify controller support at build).

Cost Optimization Best Practices

How to Use This List

These are the practices that move a Kubernetes bill, roughly in the order they pay off.

Work top to bottom. Measurement comes first because every later decision depends on trusting the numbers.

Each group ends up somewhere concrete: a changed manifest, a NodePool, or a report someone owns.

Do not attempt all of them in one quarter. Two groups done properly beat six done partially.

A - Measure Before You Cut

Install allocation before you optimize. Deploy OpenCost or Kubecost and let it collect at least two weeks. Optimizing without per-namespace attribution is guessing with extra steps.

Enforce cost labels at admission. Require team, env, and cost-center on every namespace. Spend attributed to __unallocated__ is spend nobody will fix.

Track efficiency, not just dollars. Requested-vs-used ratio per namespace is the metric that survives traffic growth. Dollars alone punish the teams that are succeeding.

Reconcile against the billing export monthly. Allocation is a model. If it drifts far from the invoice, find out why before anyone makes a decision on it.

Separate idle from workload cost. Idle is the platform team's number; workload cost is the app team's number. Merging them means neither team knows what to fix.

Give every report an owner. A dashboard with no named human and no recurring meeting produces zero savings, reliably.

B - Right-Size Requests First

Run VPA in Off mode to get recommendations. It produces target values from real history without touching running pods. Read them, then set requests yourself.

Set memory limits equal to memory requests. Memory is incompressible, so a limit above the request just makes OOM kills less predictable and Guaranteed QoS unreachable.

Be careful with CPU limits. CPU is compressible and throttling is a real latency source. Many teams set CPU requests and skip CPU limits entirely on latency-sensitive services.

Reject pods with no requests. A request-less pod is invisible to bin-packing and to allocation. Enforce a LimitRange default or an admission policy.

Re-check requests after every major release. Requests set for last year's traffic shape are just as wrong as requests that were never measured.

C - Make Nodes Fit the Pods

Adopt Karpenter and let it choose the shape. Provisioning from actual pod requests beats picking the nearest pre-defined node group, and consolidation keeps the fit as workloads change.

Enable WhenEmptyOrUnderutilized consolidation. Empty-only consolidation removes dead nodes but never fixes a fleet of half-full ones.

Set consolidateAfter longer than your rollout. Otherwise every deploy triggers a churn cycle you will end up blaming on Karpenter.

Use disruption budgets with a schedule. Freeze voluntary node churn during peak hours, and let it run freely overnight.

Expire nodes on a clock. expireAfter on the node template gives you a maximum node age and rolls AMI updates through without a bespoke rotation job.

D - Default Stateless Workloads to Spot

Taint spot nodes and make workloads opt in. Nothing should land on interruptible capacity by accident, least of all your control-plane add-ons.

Diversify across instance types and zones. Reclamation happens per pool. One instance type in one zone means one event takes your whole tier.

Run interruption handling. Karpenter does it natively with an interruption queue; the AWS Node Termination Handler is the standalone equivalent. Without it, nodes disappear mid-request.

Keep pod grace periods inside the notice window. A 5-minute terminationGracePeriodSeconds cannot drain inside a 2-minute warning, so the work dies anyway.

Handle SIGTERM in the application. Stop accepting work, finish or requeue what is in flight, exit. A drained pod that dropped its request did not actually succeed.

Keep an on-demand floor. Spot pools run dry, always during the surge that created the demand. Size on-demand to your baseline and let spot absorb the rest.

Never put singleton stateful pods on spot. Zonal volumes cannot follow a pod to a new zone, and repeated failovers cost more in incidents than the compute saves.

E - Cut the Costs That Are Not Compute

Consolidate load balancers behind one ingress or Gateway. A Service of type LoadBalancer per app means a cloud load balancer per app, each with its own hourly charge.

Keep chatty traffic in-zone. Cross-AZ egress is billed per gigabyte and hides inside service-to-service calls. Topology-aware routing helps where latency allows.

Right-size and clean up volumes. Over-provisioned PVCs bill for the full size, and PVCs from deleted namespaces frequently outlive them.

Set log and metric retention deliberately. Observability spend routinely rivals compute. Sample aggressively and drop high-cardinality labels nobody queries.

Garbage-collect registry images. Untagged layers accumulate forever, and per-gigabyte storage plus egress on pulls is a real line item.

Scale non-production to zero off-hours. Dev and staging clusters running 168 hours a week for 40 hours of use are the easiest saving on this list.

F - Buy the Baseline, Rent the Peak

Commit to your floor. Savings Plans, Reserved Instances, or CUDs on the capacity you provably always run is free money with no reliability cost.

Right-size before you commit. Committing to an over-provisioned baseline locks in the waste for one to three years.

Use ARM where the workload allows. Graviton-class instances are meaningfully cheaper per core, and a multi-arch image build makes the move mostly mechanical.

Review commitments quarterly. Traffic shape and instance families both move. A commitment set and forgotten drifts into a bad deal.

When You Are Done

You should be able to name your three most expensive namespaces and say why each costs what it does.

Your requested-vs-used ratio should be within roughly 30% for production namespaces, not the 3x that clusters drift toward.

Node count should track workload demand rather than sit at a high-water mark set during last quarter's incident.

Stateless workers should be on spot by default, and interruptions should show up in the drain logs rather than in a postmortem.

Most importantly, cost should be a number a team sees weekly and can influence directly, rather than a surprise the platform team absorbs at month end.

FAQs

What single change saves the most? Right-sizing requests, in nearly every cluster. Over-requesting inflates node count, and node count is the bill.

Should I do spot or right-sizing first? Right-sizing. Spot makes waste 70% cheaper; right-sizing removes it. Do both, in that order.

Is Karpenter worth it on a small cluster? Below roughly 20 nodes the savings are modest, but the operational win of deleting node groups often justifies it anyway.

How do I get teams to care about cost? Showback with a named owner and a weekly number. Chargeback only after the numbers have gone unchallenged for a quarter.

Do CPU limits save money? No. Requests reserve capacity and drive cost; limits only cap a container's ceiling and can throttle it. Set requests carefully, use CPU limits sparingly.

How often should requests be revisited? Quarterly, plus after any major release or traffic-shape change. Treat VPA recommendations as the input to a human decision.

Stack versions: This page was written for Kubernetes 1.36.2, Docker Engine 29.6.1 (BuildKit default), containerd (CRI runtime on nodes), Helm 3, Compose v2, Argo CD (latest - verify at build), and Gateway API (GA - verify controller support at build).

Cost Optimization Best Practices

Search Documentation

Cost Optimization Best Practices