How Kubernetes Keeps N Copies Running
Summary
- Kubernetes keeps N copies of your app running by continuously comparing desired state (what you declared) against observed state (what exists) and acting to close the gap.
- Insight: you never "start" or "restart" Pods by hand in production. You declare intent, and a control loop makes reality match it.
- Key Concepts: the reconciliation loop, the Deployment -> ReplicaSet -> Pod ownership chain, labels and selectors, and owner references.
- When to Use: this model underpins every stateless workload you run on Kubernetes, so understanding it is prerequisite to rolling updates, rollbacks, and autoscaling.
- Limitations/Trade-offs: the loop guarantees eventual convergence, not instant or ordered behavior; a Pod that keeps crashing will be recreated forever, not fixed.
- Related Topics: Deployments, ReplicaSets, rolling updates, and self-healing.
Foundations
A Pod is the smallest deployable unit in Kubernetes: one or more containers sharing a network namespace and storage.
A bare Pod is fragile. If its node dies, the Pod dies with it and nothing brings it back.
That fragility is the whole reason controllers exist.
A ReplicaSet is a controller whose single job is to keep a specified number of identical Pods running.
You tell it replicas: 3 and a Pod template, and it ensures exactly three matching Pods exist at all times.
A Deployment sits one level higher. It manages ReplicaSets so you can update the Pod template safely over time.
The chain is: Deployment owns ReplicaSet, ReplicaSet owns Pods.
You almost always create a Deployment, not a ReplicaSet or Pod directly.
Mechanics & Interactions
The engine behind all of this is the reconciliation loop, sometimes called a control loop.
Each controller watches the API server for objects it cares about, compares desired state to actual state, and takes action to reduce the difference.
Desired state lives in the object's spec. Observed state lives in its status.
For a ReplicaSet, the desired state is spec.replicas: 3. The observed state is how many matching Pods actually exist right now.
If observed is less than desired, the ReplicaSet creates Pods. If observed is greater, it deletes the newest extras.
How does a ReplicaSet know which Pods are "its" Pods? Through labels and selectors.
apiVersion: apps/v1
kind: ReplicaSet
metadata:
name: web
spec:
replicas: 3
selector:
matchLabels:
app: web
template:
metadata:
labels:
app: web
spec:
containers:
- name: web
image: registry.example.com/web@sha256:abc123The selector says "count every Pod with label app: web." The template.metadata.labels must match that selector, or the API server rejects the object.
This label-based ownership is loose by design. A ReplicaSet does not hold a list of Pod names; it queries by label every reconcile.
To make ownership precise, Kubernetes also stamps each managed Pod with an owner reference pointing back to the ReplicaSet, and the ReplicaSet carries one pointing to the Deployment.
Owner references drive garbage collection: delete the Deployment, and cascading deletion removes its ReplicaSets and their Pods.
The loop is level-triggered, not edge-triggered.
That means the controller acts on the current state of the world, not on a stream of events it must not miss.
If a controller misses an event or restarts, it simply re-observes reality and reconciles again. This is why Kubernetes is resilient to dropped messages and crashes.
Advanced Considerations & Applications
Self-healing falls out of this design for free.
Kill a Pod and the ReplicaSet notices the count dropped, then creates a replacement to restore desired state.
Drain a node and its Pods are evicted, then rescheduled elsewhere by the same convergence process.
The scheduler and the ReplicaSet are separate actors. The ReplicaSet creates a Pod object; the scheduler later assigns it to a node. Each is its own reconciliation loop.
Replicas is not availability. Three replicas on one node all die when that node fails.
You get real availability by spreading Pods across nodes and zones using anti-affinity or topology spread constraints, plus a PodDisruptionBudget to bound voluntary disruptions.
The loop will fight you. If you manually delete a Pod that a ReplicaSet owns, it comes back within seconds.
To actually remove Pods, change the desired state - scale the Deployment to zero or delete it - rather than deleting Pods directly.
Crash loops are not healed, only retried. A ReplicaSet keeps the right number of Pod objects alive, but if the container inside keeps exiting, the kubelet restarts it per the Pod's restartPolicy and backoff.
The count stays correct while the app stays broken, which is why readiness probes and monitoring matter more than replica count alone.
At scale, this same model powers rolling updates: a Deployment creates a new ReplicaSet and shifts replicas from old to new, letting two ReplicaSets coexist during the transition.
Common Misconceptions
"A Deployment runs my Pods." It does not run anything directly. It manages ReplicaSets, and those manage Pods. The kubelet on each node actually runs the containers via containerd.
"Scaling to zero deletes the Deployment." Setting replicas: 0 removes the Pods but keeps the Deployment and ReplicaSet objects, so you can scale back up instantly.
"If I delete a Pod, it stays gone." Any Pod owned by a ReplicaSet is recreated to restore the replica count. Only changing desired state removes it for good.
"More replicas means higher availability." Only if those replicas are spread across failure domains. Ten replicas on one node give you zero fault tolerance for node failure.
"The controller reacts to events." It reconciles against observed state. Events are just hints to reconcile sooner; correctness never depends on receiving them.
FAQs
What happens if the controller manager is down? Existing Pods keep running because the kubelet is independent, but no new reconciliation happens, so a failed Pod will not be replaced until the controller manager recovers.
Can I create a ReplicaSet directly? Yes, but you lose rolling updates and rollback. Use a Deployment unless you have a specific reason not to.
How fast is self-healing? Detection is near-immediate through the watch mechanism, but total recovery includes scheduling, image pull, and startup, so it ranges from seconds to a minute or more.
Why did my Pod come back after I deleted it? Because a ReplicaSet owns it and reconciled the missing replica. Change the desired count instead of deleting Pods.
Does the Deployment talk to nodes? No. It only writes ReplicaSet objects to the API server. The scheduler places Pods and the kubelet runs them.
Related
Stack versions: This page was written for Kubernetes 1.36.2, Docker Engine 29.6.1 (BuildKit default), containerd (CRI runtime on nodes), Helm 3, Compose v2, Argo CD (latest - verify at build), and Gateway API (GA - verify controller support at build).