Control Plane Internals
Summary
- The control plane is the set of processes that store cluster state and drive it toward your desired configuration.
- Insight: The API server is the only component that touches etcd; everything else coordinates through the API.
- Key Concepts: kube-apiserver, etcd, kube-scheduler, kube-controller-manager, cloud-controller-manager, kubelet, admission control.
- When to Use: When debugging why an object was accepted but nothing happened, or when sizing and hardening a cluster.
- Limitations/Trade-offs: More components means more failure surfaces, but also graceful, independent degradation.
- Related Topics: The reconciliation loop, etcd and Raft, watches and informers.
Foundations
The control plane is a small number of long-running programs plus one database.
kube-apiserver is the front door and the only writer to etcd. It handles authentication, authorization, admission, validation, and persistence.
etcd is the consistent key-value store that holds every object. It is the single source of truth.
kube-scheduler assigns unscheduled Pods to nodes. That is its entire job.
kube-controller-manager runs the built-in controllers that turn high-level objects into low-level ones and keep them converged.
cloud-controller-manager holds provider-specific logic so the core stays cloud-neutral.
On each node, the kubelet and a network dataplane execute what the control plane decided.
The defining rule: components never call each other. They read and write API objects, and other components react.
Mechanics & Interactions
Trace a single kubectl apply -f deploy.yaml end to end.
The request hits the API server over TLS. The API server authenticates the caller, then checks RBAC to authorize the verb on the resource.
Next come admission controllers. Mutating webhooks may edit the object (inject sidecars, defaults); validating webhooks may reject it. Policy engines run here too.
The API server validates the final object against its schema, then writes it to etcd. etcd commits it through Raft consensus and returns success.
Only now is the request complete. Your Deployment exists as data. No Pods run yet.
The deployment controller, watching Deployments, sees the new object and creates a ReplicaSet. The replicaset controller sees that and creates Pod objects with no node assigned.
The scheduler, watching for Pods with an empty spec.nodeName, filters feasible nodes, scores them, and binds each Pod to a node.
The kubelet on the chosen node, watching for Pods bound to itself, pulls images and calls containerd over the CRI to start containers. The kubelet reports Pod status back to the API server.
# Follow the handoff yourself
kubectl apply -f deploy.yaml
kubectl get pods -o wide --watch
kubectl get events --sort-by=.lastTimestampEvery arrow in that chain is a watch-and-react, never a direct call.
Advanced Considerations & Applications
The control plane is designed to degrade independently.
If the scheduler is down, running Pods keep running; only new Pods stay Pending. Placement stops, workloads do not.
If the controller-manager is down, existing Pods survive, but scaling, self-healing, and rollout progress stall until it returns.
If the API server is unreachable, kubelets keep the current Pods alive from their last known spec. You lose the ability to change anything, not the workloads themselves.
If etcd loses quorum, the API server goes read-mostly or errors on writes. This is the outage that actually threatens the cluster.
For high availability, run multiple API servers behind a load balancer, an odd-numbered etcd cluster (3 or 5), and leader-elected controller-manager and scheduler instances.
Leader election matters: the controller-manager and scheduler run active-passive. Only the elected leader acts, so you never get two schedulers double-binding a Pod.
Secure the control plane aggressively. etcd holds Secrets, so enable encryption at rest, restrict etcd to the API server, and lock down the kubelet API and RBAC.
Common Misconceptions
"The API server does the scheduling." No. The API server only stores and serves objects. A separate scheduler assigns nodes.
"Controllers write to etcd." Controllers only use the API. The API server is the sole etcd client.
"A successful apply means Pods are running." It means the object was persisted. Running Pods are the eventual result of several downstream loops.
"Losing the control plane stops my apps." Existing Pods keep running during a control-plane outage; you just cannot make changes.
"Admission control is optional plumbing." Admission is where defaults, sidecars, and policy enforcement happen. It is often the reason an object looks different from what you submitted.
FAQs
What is the actual order of a request? Authentication, then authorization (RBAC), then mutating admission, then validation, then validating admission, then persistence to etcd.
Why is the scheduler separate from the controller-manager? Separation of concerns and independent scaling and failure. Placement is a distinct problem from reconciliation.
Can I run the control plane on worker nodes? You can co-locate, but production clusters isolate control-plane nodes and taint them so user workloads stay off.
What does the cloud-controller-manager do? It handles provider integration - provisioning load balancers, managing node lifecycle, and configuring routes - separately from the cloud-neutral core.
How many etcd and API server replicas do I need? Use an odd number of etcd members (3 or 5) for quorum and at least two API servers behind a load balancer for availability.
Related
- A Map of the Kubernetes Machinery
- The Reconciliation Loop Explained
- etcd, Raft & Cluster State
- Watches, Informers & the List-Watch Pattern
- Why Kubernetes Is a Distributed System
Stack versions: This page was written for Kubernetes 1.36.2, Docker Engine 29.6.1 (BuildKit default), containerd (CRI runtime on nodes), Helm 3, Compose v2, Argo CD (latest - verify at build), and Gateway API (GA - verify controller support at build).