etcd, Raft & Cluster State
Summary
- etcd is the consistent, replicated key-value store that holds all Kubernetes cluster state.
- Insight: etcd health is cluster health - if etcd loses quorum, the control plane cannot accept writes.
- Key Concepts: Raft consensus, quorum, leader/follower, linearizable reads, compaction, defragmentation.
- When to Use: When sizing control-plane HA, planning backups, or diagnosing slow or failing API writes.
- Limitations/Trade-offs: Strong consistency costs write latency and bounds cluster size by etcd throughput.
- Related Topics: Control plane internals, why Kubernetes is a distributed system, the object model.
Foundations
etcd stores every object in your cluster as a key-value pair.
A Pod, a Secret, a ConfigMap, a Node - all of it lives in etcd under a hierarchical key like /registry/pods/default/web.
Only the API server reads and writes etcd. No other component connects to it directly.
etcd is not just a database; it is a replicated state machine. Several etcd members hold identical copies of the data.
Keeping those copies identical, even when members fail or messages are lost, is the hard problem. etcd solves it with the Raft consensus algorithm.
Because etcd is the only source of truth, its availability defines the cluster's ability to change. Lose etcd and you lose the ability to schedule, scale, or heal anything.
Mechanics & Interactions
Raft elects one leader among the etcd members; the rest are followers.
All writes go through the leader. The leader appends the change to its log and replicates it to followers.
A write is committed only when a majority of members have persisted it. That majority is the quorum.
Quorum is (n/2) + 1. A 3-member cluster needs 2; a 5-member cluster needs 3.
This is why cluster size is always odd. Three members tolerate one failure; five tolerate two. A fourth member adds no extra fault tolerance over three but adds latency.
If the leader fails, followers notice the missing heartbeat, start an election, and a new leader is chosen once it has a majority of votes.
While no leader has quorum, etcd cannot commit writes. The API server then rejects mutations - the cluster is effectively frozen for changes.
Reads in Kubernetes default to linearizable: they route through consensus so you never read stale data. This is why a kubectl get right after an apply reflects the write.
# Inspect etcd health (run on a control-plane node)
ETCDCTL_API=3 etcdctl endpoint health --cluster
ETCDCTL_API=3 etcdctl endpoint status --write-out=tableAdvanced Considerations & Applications
etcd performance sets a hard ceiling on cluster scale.
Every write is an fsync to disk on a majority of members, so etcd is extremely sensitive to disk latency. Use fast local SSDs and never share the disk.
Network latency between etcd members matters just as much. Stretching etcd across regions with high round-trip times cripples write throughput. Keep members close.
etcd keeps a history of every key revision. Compaction discards old revisions; without it the store grows until it hits its space quota and goes read-only.
Compaction frees logical space but not disk space. Defragmentation reclaims the physical file, and it briefly blocks the member, so do it one member at a time.
Back up etcd, not just your manifests. A snapshot of etcd is a snapshot of the entire cluster.
# Take a point-in-time snapshot of all cluster state
ETCDCTL_API=3 etcdctl snapshot save /backup/etcd-$(date +%F).dbBecause etcd holds Secrets in plaintext by default, enable encryption at rest at the API server so Secrets are encrypted before they reach etcd.
The resourceVersion you see on objects comes from etcd's revision counter. It is the backbone of watches and optimistic concurrency.
Common Misconceptions
"More etcd members means more reliability." Beyond the odd number you need, extra members add consensus latency without improving fault tolerance. Three or five is standard.
"etcd is just a database I can query." It is a consensus system with strict availability semantics, reachable only through the API server, not a general app datastore.
"Compaction reclaims disk space." Compaction removes old revisions logically; defragmentation is what shrinks the physical file.
"A control-plane outage corrupts my data." etcd's consensus ensures committed writes survive member failures. Losing quorum blocks new writes; it does not lose committed ones.
"Secrets are encrypted in etcd by default." They are base64-encoded, not encrypted. You must enable encryption at rest to protect them.
FAQs
What happens if etcd loses quorum? The cluster stops accepting writes. Running Pods keep running, but you cannot schedule, scale, or change anything until quorum returns.
Why must etcd cluster size be odd? Quorum is a strict majority. An odd count maximizes fault tolerance per member; adding an even member does not raise the failures tolerated.
Why is etcd so sensitive to disk speed? Every committed write is fsynced on a majority of members. Slow disks directly raise write latency and can trigger leader elections.
How do I back up a cluster?
Take an etcd snapshot with etcdctl snapshot save. Restoring it recreates the entire cluster state.
Does reading from Kubernetes hit etcd every time? Linearizable reads go through consensus, but the API server also serves watch caches, so not every read reaches disk.
Related
- Control Plane Internals
- Why Kubernetes Is a Distributed System
- The Object Model: Spec, Status & resourceVersion
- Watches, Informers & the List-Watch Pattern
- Kubernetes Internals Best Practices
Stack versions: This page was written for Kubernetes 1.36.2, Docker Engine 29.6.1 (BuildKit default), containerd (CRI runtime on nodes), Helm 3, Compose v2, Argo CD (latest - verify at build), and Gateway API (GA - verify controller support at build).