Kubernetes Internals Basics

This page is a hands-on tour of the core loop that runs Kubernetes: you declare desired state, and controllers work to make reality match it.

Each example shows one internal mechanism you can observe with kubectl on any cluster.

Prerequisites

• A running cluster (kind, minikube, or a cloud cluster) on Kubernetes 1.36.2.
• kubectl matching your cluster minor version.
• Read access to the default namespace and permission to create Deployments.

# Quick local cluster for these examples
kind create cluster --name internals
kubectl version --short

Basic Examples

1. Declare desired state

You describe what you want, not the steps to get there.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: web
spec:
  replicas: 3
  selector:
    matchLabels: { app: web }
  template:
    metadata:
      labels: { app: web }
    spec:
      containers:
        - name: web
          image: nginx:1.27

• spec is your desired state; you never write "start three containers".
• Controllers read this and create the objects needed to satisfy it.
• Applying the same file twice is safe - it is a declaration, not a command.
• The API server stores this object in etcd.

2. Watch controllers create children

Apply the Deployment and watch objects appear that you never wrote.

kubectl apply -f web.yaml
kubectl get deploy,rs,pods -l app=web

• The deployment controller created a ReplicaSet.
• The replicaset controller created three Pods.
• You declared one object; controllers produced the rest.
• This chain is the reconciliation loop in action.

3. See desired vs observed state

Every object carries both what you want and what is true now.

kubectl get deploy web -o jsonpath='{.spec.replicas} desired / {.status.availableReplicas} available{"\n"}'

• spec holds desired state; status holds observed state.
• Controllers drive status toward spec continuously.
• When they match, the controller has nothing to do.

4. Self-healing by deletion

Delete a Pod and watch Kubernetes replace it.

kubectl delete pod -l app=web --field-selector 'status.phase=Running' --wait=false
kubectl get pods -l app=web --watch

• Deleting a Pod makes observed state drift from desired.
• The replicaset controller notices the count is low and creates a replacement.
• Nothing "restarts" the old Pod; a new one is created to satisfy the count.

5. Scale by editing intent

Change the desired number and let the system converge.

kubectl scale deploy/web --replicas=5
kubectl get rs -l app=web

• You edited spec.replicas; you did not start any Pods yourself.
• The controller computed the difference and created two more Pods.
• Scaling down works the same way in reverse.

6. Everything is an API object

The whole cluster is queryable objects, not hidden state.

kubectl get pod -l app=web -o yaml | head -n 20
kubectl api-resources | head

• Pods, Nodes, Services, and Events are all objects with the same shape.
• metadata, spec, and status are the universal envelope.
• If you can name it, you can get it, watch it, and version it.

7. Events tell the story

Events are the audit trail of the loop.

kubectl get events --sort-by=.lastTimestamp | tail

• Scheduling, pulling, and starting each emit an Event.
• Events are how you see which component acted and when.
• They expire after about an hour, so read them promptly.

Intermediate Examples

8. Optimistic concurrency in action

Two writers cannot silently clobber each other.

kubectl get deploy web -o jsonpath='{.metadata.resourceVersion}{"\n"}'

• Every object has a resourceVersion that changes on each write.
• An update that carries a stale version is rejected with a conflict.
• Controllers simply re-read and retry, which is why they are safe to run continuously.

9. Owner references and cascading delete

Children know their parent, so deletes cascade.

kubectl get rs -l app=web -o jsonpath='{.items[0].metadata.ownerReferences[0].kind}{"\n"}'
kubectl delete deploy web
kubectl get rs,pods -l app=web

• The ReplicaSet has an ownerReference pointing at the Deployment.
• Deleting the Deployment garbage-collects its ReplicaSet and Pods.
• Ownership, not a script, drives cleanup.

10. Watch instead of poll

Clients stream changes rather than asking repeatedly.

kubectl get pods -l app=web --watch --output-watch-events

• --watch opens a long-lived stream of ADDED, MODIFIED, and DELETED events.
• This is the same list-watch mechanism controllers use internally.
• It is how the whole system stays in sync without constant polling.
• Controllers read from a cache kept fresh by this stream, so they rarely poll the API server.

11. Introspect any resource

The API describes itself, so you never have to guess field names.

kubectl explain deployment.spec.strategy
kubectl explain pod.spec.containers.resources

• kubectl explain reads the live schema the API server publishes.
• Every field, type, and default is discoverable without leaving the terminal.
• This is the same schema the API server uses to validate your objects.
• Custom resources you install show up here too, since they register their own schema.

12. Desired state survives restarts

Intent lives in etcd, so the cluster rebuilds reality after failures.

kubectl get deploy web -o yaml | grep -A2 'strategy:'

• Your Deployment's spec is durable state stored through the API server in etcd.
• If a node reboots, controllers re-drive observed state back toward that stored intent.
• Nothing you declared is lost when a Pod, node, or even a controller restarts.
• This durability of intent is why declarative configuration is the whole model.

Stack versions: This page was written for Kubernetes 1.36.2, Docker Engine 29.6.1 (BuildKit default), containerd (CRI runtime on nodes), Helm 3, Compose v2, Argo CD (latest - verify at build), and Gateway API (GA - verify controller support at build).