How the Scheduler Decides
Summary
- Definition: The scheduler decides by running each pending Pod through a filter phase that eliminates infeasible nodes and a score phase that ranks the survivors.
- Insight: Filtering is boolean (a node either can or cannot run the Pod); scoring is a weighted numeric competition among the nodes that survived.
- Key Concepts: the scheduling framework, PreFilter/Filter/PreScore/Score/Reserve/Permit/Bind extension points, plugins, and
NodeResourcesFit. - When to Use: understanding this pipeline is how you read
FailedSchedulingevents and predict where a Pod will land. - Limitations/Trade-offs: scoring is heuristic and samples nodes at scale, so it optimizes for speed over a globally optimal packing.
- Related Topics: node affinity, taints and tolerations, topology spread, and priority/preemption.
Foundations
The scheduler is built on the scheduling framework, a pluggable pipeline with named extension points.
Each phase of scheduling one Pod is a sequence of plugin calls at those extension points.
The two phases that matter most are Filter and Score.
Filter answers a yes/no question per node: can this node run this Pod right now.
Score answers a how-good question per feasible node: on a scale that plugins contribute to, how desirable is this node.
The mental model is a funnel.
All nodes enter, filtering narrows them to the feasible set, scoring ranks that set, and the top-ranked node wins the binding.
If the feasible set is empty, the Pod stays Pending and the failure reasons from the Filter plugins are surfaced in the Pod's events.
Mechanics & Interactions
A Pod's journey through the framework has clear stages.
Sort orders the scheduling queue, primarily by Pod priority, so higher-priority Pods are attempted first.
PreFilter computes and caches facts about the Pod once (for example, its total resource requests) before per-node work begins.
Filter runs per node. Core filter plugins include NodeResourcesFit (enough allocatable CPU, memory, and other resources for the Pod's requests), NodeAffinity (required node affinity and nodeSelector), TaintToleration (the Pod tolerates the node's NoSchedule and NoExecute taints), NodeName, NodePorts (no host-port conflict), NodeUnschedulable (the node is not cordoned), and VolumeBinding (requested PersistentVolumes can attach here).
PostFilter runs only when no node passed Filter. This is where preemption lives: the scheduler looks for lower-priority Pods to evict to make the pending Pod fit.
PreScore and Score run over the feasible nodes. Scoring plugins include NodeResourcesFit (bin-pack or spread depending on strategy), InterPodAffinity, PodTopologySpread, TaintToleration (preferring nodes with fewer tolerated PreferNoSchedule taints), and ImageLocality (prefer nodes that already cache the container image).
Each scoring plugin returns 0 to 100 per node, results are normalized, multiplied by a plugin weight, and summed.
Reserve and Permit update internal state and can delay binding (used by features like gang scheduling).
Bind issues the API call setting spec.nodeName.
A crucial interaction: NodeResourcesFit uses requests, so a node crowded by actual usage but light on reserved requests still passes filtering and can win scoring.
Advanced Considerations & Applications
At scale, the scheduler does not score every feasible node.
percentageOfNodesToScore lets it stop once it has found enough feasible nodes, trading a little placement quality for large throughput gains in clusters with thousands of nodes.
The default NodeResourcesFit scoring strategy is LeastAllocated, which spreads Pods toward emptier nodes.
Switching to MostAllocated (via a scheduler config profile) packs Pods tightly, which suits cost-driven autoscaling where you want to drain and remove underused nodes.
The scheduler config also supports multiple profiles, each a named set of plugins and weights.
A Pod selects a profile through spec.schedulerName, so one kube-scheduler process can serve several placement policies.
For placement logic Kubernetes does not ship, you write a custom plugin at the relevant extension point or run a second scheduler binary entirely.
Reading failures is the day-to-day application of all this.
kubectl describe pod reports lines like 3 Insufficient memory, 2 node(s) had untolerated taint {gpu: true}, or 1 node(s) didn't match pod affinity rules, and each maps directly to the Filter plugin that rejected those nodes.
Common Misconceptions
"Scoring decides feasibility." No. Filtering decides feasibility. Scoring only ranks nodes that already passed filtering.
"The highest-resource node always wins." With the default LeastAllocated strategy the emptier node tends to win, not the biggest. And many other plugins contribute to the final score.
"A Pending Pod means the cluster is full." Often it means constraints (taints, affinity, topology) eliminated otherwise-capacious nodes. Read the events to see which plugin rejected them.
"Preemption happens during normal scheduling." Preemption runs in PostFilter, only after every node failed the Filter phase for a Pod that is preemption-eligible.
"Scoring is deterministic to a single node." Ties are broken randomly among top-scoring nodes, so identical Pods can land on different nodes.
FAQs
How do I see exactly why a Pod is Pending? Run kubectl describe pod <name> and read the FailedScheduling event; it lists the count and reason per rejected node.
Can I change the scoring behavior? Yes, through a scheduler configuration file that sets plugin weights, the NodeResourcesFit strategy, and profiles. It is a cluster-operator-level change.
What is the difference between Filter and Score plugins? Filter plugins return feasible or not (boolean); Score plugins return a number that ranks feasible nodes.
Does image locality really move placement? It nudges it. ImageLocality adds score to nodes already caching the image, which can speed startup, but it is one weighted signal among many.
Why did two identical replicas land on the same node? Nothing spread them. Add topology spread or pod anti-affinity; scoring alone does not guarantee separation.
Related
- What Scheduling Is and Why It's Hard
- Scheduling Basics
- nodeSelector, Node Affinity & Anti-Affinity
- Topology Spread Constraints
- Pod Priority, Preemption & the Descheduler
Stack versions: This page was written for Kubernetes 1.36.2, Docker Engine 29.6.1 (BuildKit default), containerd (CRI runtime on nodes), Helm 3, Compose v2, Argo CD (latest - verify at build), and Gateway API (GA - verify controller support at build).