Taints & Tolerations

Summary

Definition: A taint marks a node as repellent; a toleration on a Pod lets that Pod ignore a matching taint. Together they reserve nodes for opted-in workloads.
Insight: Taints repel by default; tolerations only permit, they do not attract. To pull a Pod onto a reserved node you still need affinity or nodeSelector.
Key Concepts: the three effects NoSchedule, PreferNoSchedule, NoExecute; the key, value, operator, and tolerationSeconds fields.
When to Use: dedicate GPU, spot, ARM, or control-plane nodes; drain nodes gracefully; and let the node lifecycle controller evict Pods from broken nodes.
Limitations: a toleration is permission, not placement; over-tolerating (operator: Exists with no key) makes a Pod tolerate everything.

Recipe

Taint the node to repel everything by default.

Add a matching toleration to the specific Pods allowed there.

Add node affinity or nodeSelector so those Pods are actually attracted to the reserved pool.

That three-part pattern (taint plus toleration plus affinity) is how you cleanly dedicate a node pool.

Working Example

Reserve a GPU pool: taint the nodes, then a training Pod tolerates the taint and targets the pool.

# Taint every GPU node so nothing lands there by default
kubectl taint nodes gpu-node-1 nvidia.com/gpu=present:NoSchedule
kubectl label node gpu-node-1 node.acme.io/pool=gpu

apiVersion: apps/v1
kind: Deployment
metadata:
  name: trainer
spec:
  replicas: 2
  selector:
    matchLabels:
      app: trainer
  template:
    metadata:
      labels:
        app: trainer
    spec:
      tolerations:
        - key: "nvidia.com/gpu"
          operator: "Equal"
          value: "present"
          effect: "NoSchedule"
      nodeSelector:
        node.acme.io/pool: gpu
      containers:
        - name: trainer
          image: ghcr.io/acme/trainer:0.9.0
          resources:
            limits:
              nvidia.com/gpu: 1

General workloads without that toleration are now kept off the GPU nodes automatically.

kubectl describe node gpu-node-1 | grep Taints

Deep Dive

The three effects

NoSchedule blocks new Pods that lack a matching toleration but leaves already-running Pods in place.

PreferNoSchedule is a soft version: the scheduler tries to avoid the node but will use it if nothing else fits.

NoExecute blocks new Pods and also evicts already-running Pods that do not tolerate the taint.

Matching semantics

A toleration matches a taint when the key and effect match and either operator: Equal with an equal value, or operator: Exists (which ignores value).

operator: Exists with no key at all tolerates every taint, which is powerful and easy to misuse.

An empty effect in a toleration matches all effects for that key.

tolerationSeconds and NoExecute

For a NoExecute taint, a toleration may set tolerationSeconds, the grace period the Pod is allowed to stay before eviction.

This is exactly how node problems work: the node controller adds node.kubernetes.io/not-ready:NoExecute and node.kubernetes.io/unreachable:NoExecute, and Pods are evicted after their tolerationSeconds (default 300).

Built-in taints

Control-plane nodes carry node-role.kubernetes.io/control-plane:NoSchedule so ordinary workloads stay off them.

The node lifecycle controller also applies taints like node.kubernetes.io/disk-pressure and node.kubernetes.io/memory-pressure to steer scheduling away from stressed nodes.

Other automatic taints include node.kubernetes.io/pid-pressure, node.kubernetes.io/network-unavailable, and node.kubernetes.io/unschedulable (set when you cordon a node).

Drain uses taints under the hood

kubectl drain first cordons the node (adding the unschedulable taint) and then evicts its Pods, respecting PodDisruptionBudgets.

kubectl cordon alone just marks the node unschedulable so no new Pods land there, without evicting the current ones.

This is the standard flow before a node upgrade or decommission.

DaemonSet tolerations

The DaemonSet controller adds tolerations for the node-condition taints (not-ready, unreachable, disk-pressure, memory-pressure, pid-pressure, unschedulable) so system agents keep running even on troubled nodes.

That is why a NoExecute taint often does not remove DaemonSet Pods as you might expect; add your own custom taint if you truly need them gone.

Gotchas

A toleration alone does not place a Pod on the tainted node; without affinity the scheduler may still pick an untainted node. Add nodeSelector or node affinity to attract it.

operator: Exists with an empty key tolerates all taints, including not-ready and unreachable, which can keep a Pod pinned to a dead node. Scope tolerations narrowly.

Tainting a node with NoExecute evicts running Pods immediately (subject to tolerationSeconds); do this deliberately, not casually.

DaemonSets get broad tolerations by default so they can run everywhere, so a taint may not remove a DaemonSet Pod as expected.

Removing a taint uses the trailing-dash syntax: kubectl taint nodes gpu-node-1 nvidia.com/gpu=present:NoSchedule-.

Alternatives

Node affinity and nodeSelector attract Pods to nodes but do not repel others; use taints when the default must be "keep out."

Topology spread constraints distribute Pods evenly but do not reserve capacity; combine them with taints for dedicated, balanced pools.

Separate clusters provide the hardest isolation when tainting within one cluster is not strong enough for compliance boundaries.

Pick taints when a node class must reject all workloads except those explicitly allowed.

FAQs

What is the difference between a taint and a toleration? The taint is on the node and repels Pods; the toleration is on the Pod and grants an exception.

Does a toleration force a Pod onto a tainted node? No. It only permits scheduling there. Add node affinity or nodeSelector to attract the Pod.

How do I evict Pods from a node right now? Add a NoExecute taint (or kubectl drain), which evicts Pods that do not tolerate it.

Why do my DaemonSet Pods ignore my taint? DaemonSets carry default tolerations for many taints so system agents run everywhere. Check the DaemonSet's tolerations.

Can one node have multiple taints? Yes. A Pod must tolerate every taint on the node to be schedulable there.

Stack versions: This page was written for Kubernetes 1.36.2, Docker Engine 29.6.1 (BuildKit default), containerd (CRI runtime on nodes), Helm 3, Compose v2, Argo CD (latest - verify at build), and Gateway API (GA - verify controller support at build).

Taints & Tolerations

Summary

Definition: A taint marks a node as repellent; a toleration on a Pod lets that Pod ignore a matching taint. Together they reserve nodes for opted-in workloads.
Insight: Taints repel by default; tolerations only permit, they do not attract. To pull a Pod onto a reserved node you still need affinity or nodeSelector.
Key Concepts: the three effects NoSchedule, PreferNoSchedule, NoExecute; the key, value, operator, and tolerationSeconds fields.
When to Use: dedicate GPU, spot, ARM, or control-plane nodes; drain nodes gracefully; and let the node lifecycle controller evict Pods from broken nodes.
Limitations: a toleration is permission, not placement; over-tolerating (operator: Exists with no key) makes a Pod tolerate everything.

Recipe

Taint the node to repel everything by default.

Add a matching toleration to the specific Pods allowed there.

Add node affinity or nodeSelector so those Pods are actually attracted to the reserved pool.

That three-part pattern (taint plus toleration plus affinity) is how you cleanly dedicate a node pool.

Working Example

Reserve a GPU pool: taint the nodes, then a training Pod tolerates the taint and targets the pool.

# Taint every GPU node so nothing lands there by default
kubectl taint nodes gpu-node-1 nvidia.com/gpu=present:NoSchedule
kubectl label node gpu-node-1 node.acme.io/pool=gpu

apiVersion: apps/v1
kind: Deployment
metadata:
  name: trainer
spec:
  replicas: 2
  selector:
    matchLabels:
      app: trainer
  template:
    metadata:
      labels:
        app: trainer
    spec:
      tolerations:
        - key: "nvidia.com/gpu"
          operator: "Equal"
          value: "present"
          effect: "NoSchedule"
      nodeSelector:
        node.acme.io/pool: gpu
      containers:
        - name: trainer
          image: ghcr.io/acme/trainer:0.9.0
          resources:
            limits:
              nvidia.com/gpu: 1

General workloads without that toleration are now kept off the GPU nodes automatically.

kubectl describe node gpu-node-1 | grep Taints

Deep Dive

The three effects

NoSchedule blocks new Pods that lack a matching toleration but leaves already-running Pods in place.

PreferNoSchedule is a soft version: the scheduler tries to avoid the node but will use it if nothing else fits.

NoExecute blocks new Pods and also evicts already-running Pods that do not tolerate the taint.

Matching semantics

A toleration matches a taint when the key and effect match and either operator: Equal with an equal value, or operator: Exists (which ignores value).

operator: Exists with no key at all tolerates every taint, which is powerful and easy to misuse.

An empty effect in a toleration matches all effects for that key.

tolerationSeconds and NoExecute

For a NoExecute taint, a toleration may set tolerationSeconds, the grace period the Pod is allowed to stay before eviction.

Built-in taints

Control-plane nodes carry node-role.kubernetes.io/control-plane:NoSchedule so ordinary workloads stay off them.

The node lifecycle controller also applies taints like node.kubernetes.io/disk-pressure and node.kubernetes.io/memory-pressure to steer scheduling away from stressed nodes.

Other automatic taints include node.kubernetes.io/pid-pressure, node.kubernetes.io/network-unavailable, and node.kubernetes.io/unschedulable (set when you cordon a node).

Drain uses taints under the hood

kubectl drain first cordons the node (adding the unschedulable taint) and then evicts its Pods, respecting PodDisruptionBudgets.

kubectl cordon alone just marks the node unschedulable so no new Pods land there, without evicting the current ones.

This is the standard flow before a node upgrade or decommission.

DaemonSet tolerations

That is why a NoExecute taint often does not remove DaemonSet Pods as you might expect; add your own custom taint if you truly need them gone.

Gotchas

A toleration alone does not place a Pod on the tainted node; without affinity the scheduler may still pick an untainted node. Add nodeSelector or node affinity to attract it.

operator: Exists with an empty key tolerates all taints, including not-ready and unreachable, which can keep a Pod pinned to a dead node. Scope tolerations narrowly.

Tainting a node with NoExecute evicts running Pods immediately (subject to tolerationSeconds); do this deliberately, not casually.

DaemonSets get broad tolerations by default so they can run everywhere, so a taint may not remove a DaemonSet Pod as expected.

Removing a taint uses the trailing-dash syntax: kubectl taint nodes gpu-node-1 nvidia.com/gpu=present:NoSchedule-.

Alternatives

Node affinity and nodeSelector attract Pods to nodes but do not repel others; use taints when the default must be "keep out."

Topology spread constraints distribute Pods evenly but do not reserve capacity; combine them with taints for dedicated, balanced pools.

Separate clusters provide the hardest isolation when tainting within one cluster is not strong enough for compliance boundaries.

Pick taints when a node class must reject all workloads except those explicitly allowed.

FAQs

What is the difference between a taint and a toleration? The taint is on the node and repels Pods; the toleration is on the Pod and grants an exception.

Does a toleration force a Pod onto a tainted node? No. It only permits scheduling there. Add node affinity or nodeSelector to attract the Pod.

How do I evict Pods from a node right now? Add a NoExecute taint (or kubectl drain), which evicts Pods that do not tolerate it.

Why do my DaemonSet Pods ignore my taint? DaemonSets carry default tolerations for many taints so system agents run everywhere. Check the DaemonSet's tolerations.

Can one node have multiple taints? Yes. A Pod must tolerate every taint on the node to be schedulable there.

Stack versions: This page was written for Kubernetes 1.36.2, Docker Engine 29.6.1 (BuildKit default), containerd (CRI runtime on nodes), Helm 3, Compose v2, Argo CD (latest - verify at build), and Gateway API (GA - verify controller support at build).

Taints & Tolerations

Buscar en la documentación

Taints & Tolerations