Scheduling Best Practices

A practical, opinionated set of scheduling rules for production Kubernetes 1.36 clusters.

The theme throughout: prefer soft, expressive constraints over hard pins, reserve nodes with taints rather than manual placement, and always leave the scheduler room to make a good decision.

How to Use This List

Read top to bottom once, then adopt the groups that match your workloads.

Each item is a practice with a short rationale.

Start with resource requests (group A) because nothing else works well without them.

Treat the hard-versus-soft distinction (group B) as the single most important habit for availability.

A - Resource Requests and Fit

Always set CPU and memory requests. The scheduler packs by requests, so a Pod without them is treated as near-zero and packs badly. Requests are the foundation of every placement decision.
Set memory limit equal to memory request for critical Pods. This yields the Guaranteed QoS class and makes the Pod the last to be evicted under node memory pressure.
Do not over-request. Padding requests to be safe strands capacity and forces needless scale-out. Right-size from real usage using metrics or the Vertical Pod Autoscaler in recommend mode.
Remember limits do not affect scheduling. Only requests decide the fit; limits govern runtime throttling and OOM kills. Do not rely on limits to influence placement.

B - Affinity: Prefer Soft, Constrain Only When Needed

Default to preferred (soft) rules for availability. preferredDuringSchedulingIgnoredDuringExecution guides placement without ever forcing a Pod to Pending. Reserve required rules for genuine constraints like CPU architecture or licensed hardware.
Prefer topology spread over pod anti-affinity for HA. Topology spread expresses maxSkew directly and scales far better than anti-affinity's all-or-nothing per-domain logic. Use anti-affinity only for strict never-co-locate needs.
Avoid required hostname anti-affinity for large replica counts. It caps replicas at the node count and strands the rest. Soft anti-affinity or topology spread avoids the trap.
Keep node affinity minimal. Every extra required term shrinks the feasible set and risks unschedulable Pods. Match on stable, well-known labels only.

C - Spreading and High Availability

Spread across zones first, then nodes. Add topology spread constraints on topology.kubernetes.io/zone and kubernetes.io/hostname so one failure domain cannot take the whole service down.
Use ScheduleAnyway unless strict balance is mandatory. DoNotSchedule can strand Pods when a domain loses capacity; the soft variant favors spread without blocking. Reach for DoNotSchedule only where balance is a hard requirement.
Scope spread with matchLabelKeys during rollouts. Adding pod-template-hash to matchLabelKeys spreads each revision independently so a rolling update does not create false skew.
Pair PodDisruptionBudgets with your spread. Spreading protects against node loss; a PDB protects against voluntary disruptions like drains and upgrades. You need both.

D - Node Reservation with Taints

Reserve special nodes with taints, not manual pinning. Taint GPU, spot, ARM, and system pools so they repel by default, and add tolerations plus affinity to the Pods allowed there. Never hardcode nodeName.
Use the taint plus toleration plus affinity trio. A toleration only permits; it does not attract. Add nodeSelector or node affinity so the workload is actually pulled onto the reserved pool.
Scope tolerations narrowly. Avoid operator: Exists with an empty key, which tolerates every taint including not-ready and unreachable and can pin Pods to dead nodes.
Let built-in taints do their job. Do not blanket-tolerate node.kubernetes.io/not-ready or unreachable; the default tolerationSeconds of 300 exists so Pods leave failed nodes.

E - Priority, Preemption, and Rebalancing

Define a small, meaningful set of PriorityClasses. If everything is critical, preemption has no victims to choose and priority becomes meaningless. Keep application values below system-cluster-critical.
Mark batch and best-effort work with preemptionPolicy: Never. These Pods keep sensible queue ordering but never evict others, which limits churn.
Prefer autoscaling over preemption for capacity problems. Cluster Autoscaler or Karpenter adds nodes instead of evicting Pods; use preemption only for genuine priority contention, not for a cluster that is simply too small.
Run the descheduler to correct drift. The scheduler decides once and never rebalances. Schedule the descheduler with strategies like LowNodeUtilization and RemovePodsViolatingTopologySpreadConstraint, guarded by PDBs.

F - Operability and Safety

Diagnose Pending with describe first. kubectl describe pod lists the per-node failure reasons; do not guess at capacity. The events name the exact Filter plugin that rejected each node.
Never use nodeName in production. It skips filtering and scoring entirely, so the Pod can land where it does not fit and get evicted. Keep it for debugging only.
Set default constraints cluster-wide where possible. A scheduler configuration profile can apply baseline zone and hostname spreading so teams get HA without per-workload YAML.
Test scheduling under failure. Cordon a node or simulate a zone loss and confirm Pods reschedule within your budget. Constraints that look right on paper can strand Pods in practice.

When You Are Done

You should have requests on every Pod, soft affinity as the default, topology spread for HA, taints reserving special pools, a lean set of priority classes, and the descheduler correcting drift.

Verify by cordoning a node and watching Pods reschedule cleanly.

If any workload goes Pending during that test, a required rule is too strict.

FAQs

What is the single most impactful practice? Setting accurate resource requests. Every other scheduling decision depends on them being correct.

Anti-affinity or topology spread? Topology spread for general HA; it scales better and expresses skew directly. Reserve anti-affinity for strict never-co-locate rules.

Should I use DoNotSchedule or ScheduleAnyway? Default to ScheduleAnyway so Pods never strand on imbalance; use DoNotSchedule only where perfect balance is a hard requirement.

When do I need the descheduler? When placement drifts over time from scale events or utilization skew; the core scheduler never rebalances running Pods.

How do I reserve GPU nodes cleanly? Taint the nodes, add a matching toleration to GPU Pods, and add node affinity or nodeSelector so those Pods are attracted to the pool.

Stack versions: This page was written for Kubernetes 1.36.2, Docker Engine 29.6.1 (BuildKit default), containerd (CRI runtime on nodes), Helm 3, Compose v2, Argo CD (latest - verify at build), and Gateway API (GA - verify controller support at build).

Scheduling Best Practices

A practical, opinionated set of scheduling rules for production Kubernetes 1.36 clusters.

The theme throughout: prefer soft, expressive constraints over hard pins, reserve nodes with taints rather than manual placement, and always leave the scheduler room to make a good decision.

How to Use This List

Read top to bottom once, then adopt the groups that match your workloads.

Each item is a practice with a short rationale.

Start with resource requests (group A) because nothing else works well without them.

Treat the hard-versus-soft distinction (group B) as the single most important habit for availability.

A - Resource Requests and Fit

Always set CPU and memory requests. The scheduler packs by requests, so a Pod without them is treated as near-zero and packs badly. Requests are the foundation of every placement decision.
Set memory limit equal to memory request for critical Pods. This yields the Guaranteed QoS class and makes the Pod the last to be evicted under node memory pressure.
Do not over-request. Padding requests to be safe strands capacity and forces needless scale-out. Right-size from real usage using metrics or the Vertical Pod Autoscaler in recommend mode.
Remember limits do not affect scheduling. Only requests decide the fit; limits govern runtime throttling and OOM kills. Do not rely on limits to influence placement.

B - Affinity: Prefer Soft, Constrain Only When Needed

Default to preferred (soft) rules for availability. preferredDuringSchedulingIgnoredDuringExecution guides placement without ever forcing a Pod to Pending. Reserve required rules for genuine constraints like CPU architecture or licensed hardware.
Prefer topology spread over pod anti-affinity for HA. Topology spread expresses maxSkew directly and scales far better than anti-affinity's all-or-nothing per-domain logic. Use anti-affinity only for strict never-co-locate needs.
Avoid required hostname anti-affinity for large replica counts. It caps replicas at the node count and strands the rest. Soft anti-affinity or topology spread avoids the trap.
Keep node affinity minimal. Every extra required term shrinks the feasible set and risks unschedulable Pods. Match on stable, well-known labels only.

C - Spreading and High Availability

Spread across zones first, then nodes. Add topology spread constraints on topology.kubernetes.io/zone and kubernetes.io/hostname so one failure domain cannot take the whole service down.
Use ScheduleAnyway unless strict balance is mandatory. DoNotSchedule can strand Pods when a domain loses capacity; the soft variant favors spread without blocking. Reach for DoNotSchedule only where balance is a hard requirement.
Scope spread with matchLabelKeys during rollouts. Adding pod-template-hash to matchLabelKeys spreads each revision independently so a rolling update does not create false skew.
Pair PodDisruptionBudgets with your spread. Spreading protects against node loss; a PDB protects against voluntary disruptions like drains and upgrades. You need both.

D - Node Reservation with Taints

Reserve special nodes with taints, not manual pinning. Taint GPU, spot, ARM, and system pools so they repel by default, and add tolerations plus affinity to the Pods allowed there. Never hardcode nodeName.
Use the taint plus toleration plus affinity trio. A toleration only permits; it does not attract. Add nodeSelector or node affinity so the workload is actually pulled onto the reserved pool.
Scope tolerations narrowly. Avoid operator: Exists with an empty key, which tolerates every taint including not-ready and unreachable and can pin Pods to dead nodes.
Let built-in taints do their job. Do not blanket-tolerate node.kubernetes.io/not-ready or unreachable; the default tolerationSeconds of 300 exists so Pods leave failed nodes.

E - Priority, Preemption, and Rebalancing

Define a small, meaningful set of PriorityClasses. If everything is critical, preemption has no victims to choose and priority becomes meaningless. Keep application values below system-cluster-critical.
Mark batch and best-effort work with preemptionPolicy: Never. These Pods keep sensible queue ordering but never evict others, which limits churn.
Prefer autoscaling over preemption for capacity problems. Cluster Autoscaler or Karpenter adds nodes instead of evicting Pods; use preemption only for genuine priority contention, not for a cluster that is simply too small.
Run the descheduler to correct drift. The scheduler decides once and never rebalances. Schedule the descheduler with strategies like LowNodeUtilization and RemovePodsViolatingTopologySpreadConstraint, guarded by PDBs.

F - Operability and Safety

Diagnose Pending with describe first. kubectl describe pod lists the per-node failure reasons; do not guess at capacity. The events name the exact Filter plugin that rejected each node.
Never use nodeName in production. It skips filtering and scoring entirely, so the Pod can land where it does not fit and get evicted. Keep it for debugging only.
Set default constraints cluster-wide where possible. A scheduler configuration profile can apply baseline zone and hostname spreading so teams get HA without per-workload YAML.
Test scheduling under failure. Cordon a node or simulate a zone loss and confirm Pods reschedule within your budget. Constraints that look right on paper can strand Pods in practice.

When You Are Done

You should have requests on every Pod, soft affinity as the default, topology spread for HA, taints reserving special pools, a lean set of priority classes, and the descheduler correcting drift.

Verify by cordoning a node and watching Pods reschedule cleanly.

If any workload goes Pending during that test, a required rule is too strict.

FAQs

What is the single most impactful practice? Setting accurate resource requests. Every other scheduling decision depends on them being correct.

Anti-affinity or topology spread? Topology spread for general HA; it scales better and expresses skew directly. Reserve anti-affinity for strict never-co-locate rules.

Should I use DoNotSchedule or ScheduleAnyway? Default to ScheduleAnyway so Pods never strand on imbalance; use DoNotSchedule only where perfect balance is a hard requirement.

When do I need the descheduler? When placement drifts over time from scale events or utilization skew; the core scheduler never rebalances running Pods.

How do I reserve GPU nodes cleanly? Taint the nodes, add a matching toleration to GPU Pods, and add node affinity or nodeSelector so those Pods are attracted to the pool.

Stack versions: This page was written for Kubernetes 1.36.2, Docker Engine 29.6.1 (BuildKit default), containerd (CRI runtime on nodes), Helm 3, Compose v2, Argo CD (latest - verify at build), and Gateway API (GA - verify controller support at build).

Scheduling Best Practices

Search Documentation

Scheduling Best Practices