API Objects & Reconciliation

Summary

• Every Kubernetes resource is an API object with a spec (desired state) and usually a status (observed state), stored in etcd behind the API server.
• Insight: Controllers run an endless loop comparing spec to status and taking action to close the gap - that loop is the whole system in one sentence.
• Key Concepts: spec vs status, controllers and control loops, level-triggered reconciliation, owner references, and generation/observedGeneration.
• When to Use: understanding this is what turns Kubernetes from a mystery into a predictable system you can debug.
• Limitations: reconciliation is eventually consistent, so changes are not instant and failures can be quiet inside a controller.
• Learn to read status and events to see reconciliation working or failing.

Recipe

Submit desired state and let controllers converge the cluster.

kubectl apply -f deployment.yaml
kubectl get deploy web -o yaml | grep -A5 status:
kubectl get events --sort-by=.lastTimestamp

• Write intent into spec.
• Controllers read spec, act, and write results into status.
• Read status and events to confirm convergence.

You never edit status yourself. It is owned by controllers and reflects reality.

Working Example

A Deployment shows the full chain of controllers reconciling toward desired state.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: web
spec:
  replicas: 3
  selector:
    matchLabels:
      app: web
  template:
    metadata:
      labels:
        app: web
    spec:
      containers:
        - name: web
          image: ghcr.io/acme/web:1.4.2

Apply it and watch the layers of reconciliation.

kubectl apply -f deployment.yaml
kubectl get deploy,rs,pods -l app=web

The Deployment controller creates a ReplicaSet. The ReplicaSet controller creates three Pods. The scheduler binds each Pod to a node. Each kubelet starts the container via containerd.

Now break desired vs observed on purpose.

kubectl delete pod -l app=web --wait=false
kubectl get pods -l app=web -w

The ReplicaSet controller sees two Pods where three are desired and immediately creates a replacement. Nobody told it to; it is just closing the gap. That is reconciliation.

Deep Dive

spec and status

spec is your desired state and is the only part you write. status is the controller's report of observed reality.

kubectl explain deployment.spec
kubectl explain deployment.status

When they match, the object is converged. When they differ, a controller is either working on it or stuck, and events tell you which.

The control loop

A controller runs a loop: observe the current state via watches, compare it to spec, and act to reduce the difference.

This is level-triggered, not edge-triggered. The controller reacts to the current state, not to a stream of change events, so a missed event does not break it - the next resync sees the true state and corrects it.

That property is why Kubernetes self-heals. Delete a Pod, lose a node, or restart a controller, and reconciliation still converges on the desired count.

Owner references and garbage collection

Objects created by a controller carry an ownerReferences field pointing at their parent.

kubectl get rs -l app=web -o jsonpath='{.items[0].metadata.ownerReferences}'

Delete the Deployment and its ReplicaSets and Pods are garbage-collected because they are owned by it. This is cascade deletion, driven by owner references.

generation and observedGeneration

The API server bumps metadata.generation each time you change spec. A controller records the generation it last acted on in status.observedGeneration.

When observedGeneration equals generation and status conditions are healthy, the controller has caught up with your latest change. This is how kubectl rollout status knows a rollout is done.

Custom resources extend the model

CRDs let you register new object kinds, and a custom controller reconciles them exactly like built-in ones. This is the operator pattern, and it is how cert-manager, databases, and meshes ship as native Kubernetes APIs.

Gotchas

Reconciliation is eventually consistent. apply returns immediately, but convergence takes time and can silently stall in a controller.

A selector that does not match the Pod template is rejected, and a selector that overlaps another controller's Pods causes them to fight over the same Pods.

Editing status by hand does nothing useful; the owning controller overwrites it on the next loop.

A wedged rollout often shows observedGeneration lagging generation. Check kubectl describe conditions and Pod events for the real cause (bad image, failing probe, insufficient resources).

Deleting a parent cascades to children by default. Use --cascade=orphan only when you deliberately want to keep the children.

Do not confuse "applied" with "healthy." Always read status and events, not just the exit code of apply.

Alternatives

Bare Pods skip controllers entirely and are not reconciled, so a lost Pod stays lost. Use them only for one-shot debugging.

StatefulSets and DaemonSets are alternative controllers for stateful apps and per-node workloads; they reconcile with different guarantees (stable identity, one Pod per node).

Jobs and CronJobs reconcile toward completion rather than a steady replica count, for batch work.

Operators (CRDs + controllers) extend reconciliation to your own domain objects when the built-in workload types are not enough.

FAQs

What is the difference between spec and status? spec is desired state you write; status is observed state a controller writes. Reconciliation drives status toward spec.

Why did my Pod come back after I deleted it? A controller (ReplicaSet via a Deployment) reconciled the gap. Delete the owning object or scale to zero instead.

What does level-triggered mean? Controllers act on the current state, not on individual change events, so missed events self-correct on the next resync.

How do I know a rollout finished? When status.observedGeneration matches metadata.generation and conditions are healthy; kubectl rollout status checks this for you.

What are owner references for? They link children to parents so cascade deletion and garbage collection work correctly.

Can I add my own reconciled objects? Yes, with a CRD plus a controller (the operator pattern). Kubernetes treats them like native objects.

Related

• What Kubernetes Is and Why It Exists
• Declarative vs Imperative
• Labels & Selectors
• kubectl Essentials
• Kubernetes Fundamentals Best Practices

Stack versions: This page was written for Kubernetes 1.36.2, Docker Engine 29.6.1 (BuildKit default), containerd (CRI runtime on nodes), Helm 3, Compose v2, Argo CD (latest - verify at build), and Gateway API (GA - verify controller support at build).