Kubernetes Internals Best Practices

These practices translate how Kubernetes actually works - reconciliation loops, a single etcd source of truth, and an unreliable network - into concrete design habits.

They apply whether you are shipping an app onto a cluster or building controllers and operators on top of the API.

How to Use This List

Read the groups top to bottom; they move from application design to controllers to cluster operations.

Treat each bold item as a rule of thumb with a short rationale, not an absolute law.

If you build operators, groups B and C are essential; if you only deploy workloads, groups A and D matter most.

A - Design for Eventual Consistency

Assume convergence takes time. Never assume a change takes effect instantly; the system reconciles over seconds, and any instant may be mid-flight.
Tolerate transient bad states. Expect brief windows where a Service routes to a terminating Pod, and handle them with readiness probes and retries.
Declare intent, not steps. Express what you want in spec and let controllers figure out how, instead of scripting imperative changes.
Make app startup and shutdown graceful. Honor SIGTERM, drain in-flight work, and use preStop hooks so rolling updates stay clean during convergence.
Do not couple to internal timing. Avoid logic that depends on how fast a controller acts; timing varies with load and backoff.

B - Build Idempotent, Level-Triggered Controllers

Reconcile from current state, not from the event. Read the object fresh each pass and decide from reality, so a missed or duplicate event is harmless.
Make every reconcile idempotent. Check existence before creating and converge toward the goal, because reconcile can run many times for one change.
Enqueue keys, not objects. Put object keys on a work queue so rapid changes deduplicate into a single reconcile of current state.
Return errors to get backoff. Let the work queue requeue with exponential backoff instead of blocking or sleeping inside reconcile.
Rely on periodic resync. Lean on resync to catch drift and missed watch events rather than assuming the event stream is complete.

C - Respect the Object Model

Never conflate spec and status. Write only spec as a client and only status as the owning controller, and use the status subresource where available.
Handle 409 conflicts by retrying. Treat optimistic-concurrency conflicts as normal - re-read and reconcile again rather than failing.
Use ownerReferences for cleanup. Wire child objects to their parent so garbage collection cascades instead of leaving orphans.
Use finalizers for external cleanup. Guard deletion of external resources with a finalizer, and always remove it once cleanup completes to avoid stuck deletes.
Prefer Server-Side Apply for shared objects. Use field-level ownership when several actors edit one object, instead of fragile read-modify-write.

D - Read from Caches, Write Sparingly

Watch, do not poll. Use informers and list-watch so the expensive list happens once and changes stream in, sparing the API server and etcd.
Share informers across controllers. Maintain one watch per resource and fan out to handlers instead of every controller opening its own.
Expect the watch cache to lag. Treat informer reads as eventually consistent and relist on a 410 Gone rather than assuming perfect freshness.
Keep writes minimal and meaningful. Avoid needless status updates in a tight loop; every write hits etcd and every etcd write is an fsync on a quorum.
Never talk to etcd directly. Go through the API server so admission, validation, and authorization always apply.

E - Operate the Control Plane for Failure

Run etcd with an odd quorum. Use 3 or 5 members on fast, dedicated SSDs so the cluster tolerates node loss without losing write capacity.
Back up etcd, not just manifests. Snapshot etcd regularly, because a snapshot is the entire cluster state and your real recovery point.
Protect Secrets at rest. Enable encryption at rest so Secrets are not stored as plaintext in etcd, and restrict etcd access to the API server.
Spread control-plane components across failure domains. Place etcd members and API servers so one zone outage cannot take quorum.
Keep clocks in sync. Run NTP on every node, since TLS validity, token expiry, and leader election all depend on accurate time.

When You Are Done

You should be able to explain, for any workload you run, why it will converge and how it behaves during a node or control-plane failure.

Your controllers should survive being killed mid-reconcile, replayed events, and update conflicts without corrupting state.

If you can force-delete a node's Pods, restart a controller, and lose an etcd member without data loss, your design respects how Kubernetes really works.

Why design for eventual consistency instead of expecting instant changes? Because Kubernetes converges through independent loops with no cross-object transaction. Given stable intent it reaches the goal, but not atomically or instantly.

What is the single most important controller rule? Idempotency. Reconcile can run repeatedly for one change, so acting on current state and checking before creating keeps you correct.

Why should I never write to etcd directly? The API server enforces authentication, authorization, admission, and validation. Bypassing it skips all safety and can corrupt cluster state.

How do I avoid stuck deletions? Ensure every finalizer has a healthy controller that removes it after cleanup. A missing or failing controller leaves objects stuck with a deletionTimestamp.

Why back up etcd specifically? etcd holds all cluster state. A YAML repo rebuilds intent, but only an etcd snapshot captures the full live state, including status and generated objects.

Stack versions: This page was written for Kubernetes 1.36.2, Docker Engine 29.6.1 (BuildKit default), containerd (CRI runtime on nodes), Helm 3, Compose v2, Argo CD (latest - verify at build), and Gateway API (GA - verify controller support at build).

They apply whether you are shipping an app onto a cluster or building controllers and operators on top of the API.

How to Use This List

Read the groups top to bottom; they move from application design to controllers to cluster operations.

Treat each bold item as a rule of thumb with a short rationale, not an absolute law.

If you build operators, groups B and C are essential; if you only deploy workloads, groups A and D matter most.

A - Design for Eventual Consistency

Assume convergence takes time. Never assume a change takes effect instantly; the system reconciles over seconds, and any instant may be mid-flight.
Tolerate transient bad states. Expect brief windows where a Service routes to a terminating Pod, and handle them with readiness probes and retries.
Declare intent, not steps. Express what you want in spec and let controllers figure out how, instead of scripting imperative changes.
Make app startup and shutdown graceful. Honor SIGTERM, drain in-flight work, and use preStop hooks so rolling updates stay clean during convergence.
Do not couple to internal timing. Avoid logic that depends on how fast a controller acts; timing varies with load and backoff.

B - Build Idempotent, Level-Triggered Controllers

Reconcile from current state, not from the event. Read the object fresh each pass and decide from reality, so a missed or duplicate event is harmless.
Make every reconcile idempotent. Check existence before creating and converge toward the goal, because reconcile can run many times for one change.
Enqueue keys, not objects. Put object keys on a work queue so rapid changes deduplicate into a single reconcile of current state.
Return errors to get backoff. Let the work queue requeue with exponential backoff instead of blocking or sleeping inside reconcile.
Rely on periodic resync. Lean on resync to catch drift and missed watch events rather than assuming the event stream is complete.

C - Respect the Object Model

Never conflate spec and status. Write only spec as a client and only status as the owning controller, and use the status subresource where available.
Handle 409 conflicts by retrying. Treat optimistic-concurrency conflicts as normal - re-read and reconcile again rather than failing.
Use ownerReferences for cleanup. Wire child objects to their parent so garbage collection cascades instead of leaving orphans.
Use finalizers for external cleanup. Guard deletion of external resources with a finalizer, and always remove it once cleanup completes to avoid stuck deletes.
Prefer Server-Side Apply for shared objects. Use field-level ownership when several actors edit one object, instead of fragile read-modify-write.

D - Read from Caches, Write Sparingly

Watch, do not poll. Use informers and list-watch so the expensive list happens once and changes stream in, sparing the API server and etcd.
Share informers across controllers. Maintain one watch per resource and fan out to handlers instead of every controller opening its own.
Expect the watch cache to lag. Treat informer reads as eventually consistent and relist on a 410 Gone rather than assuming perfect freshness.
Keep writes minimal and meaningful. Avoid needless status updates in a tight loop; every write hits etcd and every etcd write is an fsync on a quorum.
Never talk to etcd directly. Go through the API server so admission, validation, and authorization always apply.

E - Operate the Control Plane for Failure

Run etcd with an odd quorum. Use 3 or 5 members on fast, dedicated SSDs so the cluster tolerates node loss without losing write capacity.
Back up etcd, not just manifests. Snapshot etcd regularly, because a snapshot is the entire cluster state and your real recovery point.
Protect Secrets at rest. Enable encryption at rest so Secrets are not stored as plaintext in etcd, and restrict etcd access to the API server.
Spread control-plane components across failure domains. Place etcd members and API servers so one zone outage cannot take quorum.
Keep clocks in sync. Run NTP on every node, since TLS validity, token expiry, and leader election all depend on accurate time.

When You Are Done

You should be able to explain, for any workload you run, why it will converge and how it behaves during a node or control-plane failure.

Your controllers should survive being killed mid-reconcile, replayed events, and update conflicts without corrupting state.

If you can force-delete a node's Pods, restart a controller, and lose an etcd member without data loss, your design respects how Kubernetes really works.

FAQs

What is the single most important controller rule? Idempotency. Reconcile can run repeatedly for one change, so acting on current state and checking before creating keeps you correct.

Why should I never write to etcd directly? The API server enforces authentication, authorization, admission, and validation. Bypassing it skips all safety and can corrupt cluster state.

How do I avoid stuck deletions? Ensure every finalizer has a healthy controller that removes it after cleanup. A missing or failing controller leaves objects stuck with a deletionTimestamp.

Why back up etcd specifically? etcd holds all cluster state. A YAML repo rebuilds intent, but only an etcd snapshot captures the full live state, including status and generated objects.

Stack versions: This page was written for Kubernetes 1.36.2, Docker Engine 29.6.1 (BuildKit default), containerd (CRI runtime on nodes), Helm 3, Compose v2, Argo CD (latest - verify at build), and Gateway API (GA - verify controller support at build).

Kubernetes Internals Best Practices

Search Documentation

Kubernetes Internals Best Practices