The Reconciliation Loop Explained

Summary

A reconciliation loop repeatedly compares desired state against observed state and takes action to close the gap.
Insight: Controllers are level-triggered, not edge-triggered - they act on current state, not on the event that woke them.
Key Concepts: desired state (spec), observed state (status), level-triggered control, idempotency, eventual consistency.
When to Use: Understand this before writing an operator or debugging why a resource "isn't updating instantly".
Limitations/Trade-offs: Convergence is eventual, not transactional; there is no atomic multi-object commit.
Related Topics: The object model, watches and informers, the control plane.

Foundations

A controller runs one function over and over: reconcile.

Reconcile answers a single question. Given the desired state of this object, does the world match it, and if not, what one step moves it closer?

Desired state lives in an object's spec. Observed state lives in its status and in the real resources the controller manages.

The loop never assumes it knows the current situation from memory. Each pass it reads the truth fresh and decides from scratch.

This is the difference between a control system and a script. A script runs a sequence once. A controller drives toward a goal forever.

Borrow the analogy of a thermostat. It does not remember that it turned the heat on. It reads the temperature, compares it to the target, and acts.

Mechanics & Interactions

The loop is level-triggered rather than edge-triggered.

Edge-triggered means reacting to the transition - "a Pod was deleted". Level-triggered means reacting to the state - "there are two Pods but I want three".

Kubernetes chooses level-triggered because it is robust to missed events. If a controller crashes and misses a notification, the next reconcile still sees the true count and fixes it.

A watch event is only a hint that something changed. It tells the controller to look, not what to do.

That is why reconcile must be idempotent. Running it twice on the same state must produce the same result as running it once.

A typical reconcile looks like this:

func (r *Reconciler) Reconcile(ctx, req) (Result, error) {
    obj := r.get(req)          // read desired state fresh
    actual := r.observe(obj)   // read the real world
    if actual == obj.Spec {    // already converged
        return Done, nil
    }
    r.makeOneStep(obj, actual) // move one step closer, idempotently
    return Requeue, nil        // ask to be called again
}

When reconcile cannot finish - a dependency is missing, an API call failed - it returns an error or requeue. The controller backs off and tries again later.

This retry-with-backoff is baked into every controller. Transient failure is the normal case, not an exception.

Controllers also run periodic resyncs. Even with no events at all, they re-examine everything on an interval to catch drift the watch stream missed.

Advanced Considerations & Applications

Reconciliation is why Kubernetes self-heals but never promises instant results.

There is no distributed transaction across objects. Creating a Deployment does not atomically create its Pods; a chain of independent loops does that, one hop at a time.

So the correct mental model is eventual consistency. Given a stable desired state and enough time, the system converges. At any instant it may be mid-flight.

This shapes how you write operators. Never assume ordering, never assume a child exists yet, and always tolerate being called when nothing needs doing.

It also shapes how you write apps. A Service may briefly route to a Pod that is terminating; readiness probes and graceful shutdown exist precisely because convergence is not instantaneous.

Requeue strategy matters in production. Tight requeues waste API server capacity; long ones slow convergence. Controllers use exponential backoff with a cap to balance both.

Status is a report, not a command. Writing status records what the controller observed; it never causes an action by itself.

Common Misconceptions

"The controller reacts to the delete event." It reacts to the resulting state. The event only prompts a fresh read; the decision comes from comparing spec to reality.

"Applying a change updates the cluster atomically." The write to etcd is atomic for that one object, but the downstream effects unfold across many independent loops over time.

"If reconcile runs twice, I get duplicate resources." Only if it is written badly. Correct reconcilers are idempotent and check existence before creating.

"No events means the controller is idle and blind." Periodic resync means controllers re-check state even without events, catching drift and missed notifications.

"Status drives behavior." Status is output. Desired state in spec drives behavior; status merely records what happened.

Why level-triggered instead of edge-triggered? Because it survives missed events and crashes. Reading current state each pass means a dropped notification only delays convergence, never breaks it.

Why must reconcile be idempotent? Because it can be invoked many times for the same state - by events, resyncs, or retries. Idempotency makes those repeats harmless.

How fast does the cluster converge? Usually seconds, but there is no guarantee. Load, backoff, and dependency readiness all affect timing.

Can two controllers fight over one object? They can if they both write the same field. Ownership conventions and separate concerns keep controllers from thrashing.

What triggers a reconcile with no changes? The periodic resync. It re-lists everything on an interval so drift and missed events are eventually corrected.

Stack versions: This page was written for Kubernetes 1.36.2, Docker Engine 29.6.1 (BuildKit default), containerd (CRI runtime on nodes), Helm 3, Compose v2, Argo CD (latest - verify at build), and Gateway API (GA - verify controller support at build).

The Reconciliation Loop Explained

Summary

A reconciliation loop repeatedly compares desired state against observed state and takes action to close the gap.
Insight: Controllers are level-triggered, not edge-triggered - they act on current state, not on the event that woke them.
Key Concepts: desired state (spec), observed state (status), level-triggered control, idempotency, eventual consistency.
When to Use: Understand this before writing an operator or debugging why a resource "isn't updating instantly".
Limitations/Trade-offs: Convergence is eventual, not transactional; there is no atomic multi-object commit.
Related Topics: The object model, watches and informers, the control plane.

Foundations

A controller runs one function over and over: reconcile.

Reconcile answers a single question. Given the desired state of this object, does the world match it, and if not, what one step moves it closer?

Desired state lives in an object's spec. Observed state lives in its status and in the real resources the controller manages.

The loop never assumes it knows the current situation from memory. Each pass it reads the truth fresh and decides from scratch.

This is the difference between a control system and a script. A script runs a sequence once. A controller drives toward a goal forever.

Borrow the analogy of a thermostat. It does not remember that it turned the heat on. It reads the temperature, compares it to the target, and acts.

Mechanics & Interactions

The loop is level-triggered rather than edge-triggered.

Edge-triggered means reacting to the transition - "a Pod was deleted". Level-triggered means reacting to the state - "there are two Pods but I want three".

Kubernetes chooses level-triggered because it is robust to missed events. If a controller crashes and misses a notification, the next reconcile still sees the true count and fixes it.

A watch event is only a hint that something changed. It tells the controller to look, not what to do.

That is why reconcile must be idempotent. Running it twice on the same state must produce the same result as running it once.

A typical reconcile looks like this:

func (r *Reconciler) Reconcile(ctx, req) (Result, error) {
    obj := r.get(req)          // read desired state fresh
    actual := r.observe(obj)   // read the real world
    if actual == obj.Spec {    // already converged
        return Done, nil
    }
    r.makeOneStep(obj, actual) // move one step closer, idempotently
    return Requeue, nil        // ask to be called again
}

When reconcile cannot finish - a dependency is missing, an API call failed - it returns an error or requeue. The controller backs off and tries again later.

This retry-with-backoff is baked into every controller. Transient failure is the normal case, not an exception.

Controllers also run periodic resyncs. Even with no events at all, they re-examine everything on an interval to catch drift the watch stream missed.

Advanced Considerations & Applications

Reconciliation is why Kubernetes self-heals but never promises instant results.

There is no distributed transaction across objects. Creating a Deployment does not atomically create its Pods; a chain of independent loops does that, one hop at a time.

So the correct mental model is eventual consistency. Given a stable desired state and enough time, the system converges. At any instant it may be mid-flight.

This shapes how you write operators. Never assume ordering, never assume a child exists yet, and always tolerate being called when nothing needs doing.

It also shapes how you write apps. A Service may briefly route to a Pod that is terminating; readiness probes and graceful shutdown exist precisely because convergence is not instantaneous.

Requeue strategy matters in production. Tight requeues waste API server capacity; long ones slow convergence. Controllers use exponential backoff with a cap to balance both.

Status is a report, not a command. Writing status records what the controller observed; it never causes an action by itself.

Common Misconceptions

"The controller reacts to the delete event." It reacts to the resulting state. The event only prompts a fresh read; the decision comes from comparing spec to reality.

"Applying a change updates the cluster atomically." The write to etcd is atomic for that one object, but the downstream effects unfold across many independent loops over time.

"If reconcile runs twice, I get duplicate resources." Only if it is written badly. Correct reconcilers are idempotent and check existence before creating.

"No events means the controller is idle and blind." Periodic resync means controllers re-check state even without events, catching drift and missed notifications.

"Status drives behavior." Status is output. Desired state in spec drives behavior; status merely records what happened.

FAQs

Why must reconcile be idempotent? Because it can be invoked many times for the same state - by events, resyncs, or retries. Idempotency makes those repeats harmless.

How fast does the cluster converge? Usually seconds, but there is no guarantee. Load, backoff, and dependency readiness all affect timing.

Can two controllers fight over one object? They can if they both write the same field. Ownership conventions and separate concerns keep controllers from thrashing.

What triggers a reconcile with no changes? The periodic resync. It re-lists everything on an interval so drift and missed events are eventually corrected.

Stack versions: This page was written for Kubernetes 1.36.2, Docker Engine 29.6.1 (BuildKit default), containerd (CRI runtime on nodes), Helm 3, Compose v2, Argo CD (latest - verify at build), and Gateway API (GA - verify controller support at build).

The Reconciliation Loop Explained

Search Documentation

The Reconciliation Loop Explained