CRDs & Operators Best Practices

This is a working checklist for building and running CRDs and operators at tech-lead depth. It assumes you have read the section's explainers and want concrete rules to apply.

How to Use This List

Treat each group as a review gate. Before you ship a CRD or adopt an operator, walk the relevant groups and confirm you have not skipped a practice.

The two recurring themes are: prefer proven operators over homegrown ones, and treat every CRD you own as a versioned public API you cannot casually break.

A - Decide Before You Build

Adopt before you build. Reach for a mature, actively maintained operator first; write your own only when the domain logic is genuinely yours and no existing operator fits.

Match the mechanism to the need. Use plain manifests for static workloads, Helm for packaging, and an operator only for stateful day-2 automation that templates cannot express.

Justify the operator's complexity. An operator is long-lived, privileged software with on-call cost; adopt one only when it removes real operational toil.

Target a realistic capability level. Aim for reliable install and upgrade first; pursue full auto-pilot lifecycle only when the investment is warranted.

B - Design the CRD as an API

Treat the CRD as a public contract. Once a version is served, users depend on it; plan for compatibility from day one rather than after the first breaking change.

Write strict structural schemas. Fully type every field, mark required fields, and avoid x-kubernetes-preserve-unknown-fields so typos and drift are rejected at admission.

Validate in the schema. Use enum, minimum/maximum, pattern, and CEL x-kubernetes-validations for cross-field rules before falling back to a validating webhook.

Split spec and status. Enable the status subresource so controllers report state without racing user edits to spec, and add the scale subresource when an HPA should target the resource.

Add printer columns. Surface the fields operators care about, plus an Age column, so kubectl get is useful without -o yaml.

Namespace-scope by default. Prefer scope: Namespaced unless the resource is genuinely cluster-wide, to keep tenancy and RBAC clean.

C - Version and Evolve Safely

Evolve additively. Add optional fields with defaults rather than changing or removing existing ones; a tightened constraint can invalidate stored objects.

Introduce new versions deliberately. Add the new version as served-but-not-stored, migrate stored objects, promote it to storage, then stop serving the old one only after consumers move.

Keep conversion lossless. A conversion webhook must round-trip through the storage version; preserve removed fields (for example in an annotation) during deprecation.

Run conversion webhooks highly available. Every read and write of the resource routes through the webhook, so downtime or an expired certificate breaks the whole type.

Clean up storedVersions. After migrating away from a version, ensure no objects are stored in it before removing it, or the API server will refuse.

D - Build the Controller Correctly

Make Reconcile idempotent. The same object is reconciled many times, on resync, restart, and owned-object events; create-if-missing and update-if-different, never assume first run.

Reconcile level-triggered. Recompute desired state from the current object each call rather than reacting to specific events, so the loop self-heals after missed messages.

Handle conflicts and stale reads. Cache reads are eventually consistent; retry on Conflict and prefer server-side apply or get-modify-retry over blind overwrites.

Use owner references. Set controller references on created objects so garbage collection and owned-event reconciliation work correctly.

Requeue instead of blocking. Return an error or RequeueAfter to retry with backoff rather than sleeping inside the reconcile loop.

Use finalizers for external cleanup. When deletion must tear down external resources, add a finalizer, clean up on deletionTimestamp, then remove it.

E - Operate Safely in Production

Scope RBAC tightly. Grant the controller only the verbs and resources it needs, and limit watched namespaces when the operator supports it, to shrink blast radius.

Enable leader election. Run multiple replicas for availability but ensure only one actively reconciles, preventing conflicting actions.

Pin versions and gate upgrades. Install operators via a pinned Helm chart or OLM through GitOps, and test upgrades with representative custom resources in staging first.

Monitor the controller itself. Alert on reconcile errors, queue depth, and latency; if the operator stops reconciling, workloads silently stop self-healing.

Set requests, limits, and a PDB. Treat the operator as a control-plane component: give it resource bounds, probes, and a PodDisruptionBudget.

Plan CRD deletion carefully. Deleting a CRD cascades to all its custom resources; know what happens to managed workloads before uninstalling an operator.

When You Are Done

You should be able to point to a strict schema, a documented versioning and conversion plan, an idempotent reconciler, and scoped RBAC with monitoring for every operator you run.

If any of those is missing, the gap is your next task before this reaches production.

FAQs

What is the single most important CRD practice? Treating the CRD as a versioned public API, so schema and version changes are planned rather than breaking.

Should we ever write our own operator? Yes, when the operational logic is specific to your domain and no mature operator fits; otherwise adopt an existing one.

How do I keep an operator's blast radius small? Scope RBAC and watched namespaces, run it in a dedicated namespace, enable leader election, and pin versions.

Do I need a conversion webhook? Only when serving multiple versions with differing schemas; if all served versions share a schema, strategy: None is enough.

Why must Reconcile be idempotent? Because resyncs, restarts, and owned-object events cause many reconciles per logical change, and non-idempotent logic corrupts state.

How should operators be installed? Declaratively and pinned, via Helm or OLM managed through GitOps, so installs and upgrades are reviewed and reproducible.

Stack versions: This page was written for Kubernetes 1.36.2, Docker Engine 29.6.1 (BuildKit default), containerd (CRI runtime on nodes), Helm 3, Compose v2, Argo CD (latest - verify at build), and Gateway API (GA - verify controller support at build).

CRDs & Operators Best Practices

This is a working checklist for building and running CRDs and operators at tech-lead depth. It assumes you have read the section's explainers and want concrete rules to apply.

How to Use This List

Treat each group as a review gate. Before you ship a CRD or adopt an operator, walk the relevant groups and confirm you have not skipped a practice.

The two recurring themes are: prefer proven operators over homegrown ones, and treat every CRD you own as a versioned public API you cannot casually break.

A - Decide Before You Build

Adopt before you build. Reach for a mature, actively maintained operator first; write your own only when the domain logic is genuinely yours and no existing operator fits.

Match the mechanism to the need. Use plain manifests for static workloads, Helm for packaging, and an operator only for stateful day-2 automation that templates cannot express.

Justify the operator's complexity. An operator is long-lived, privileged software with on-call cost; adopt one only when it removes real operational toil.

Target a realistic capability level. Aim for reliable install and upgrade first; pursue full auto-pilot lifecycle only when the investment is warranted.

B - Design the CRD as an API

Treat the CRD as a public contract. Once a version is served, users depend on it; plan for compatibility from day one rather than after the first breaking change.

Write strict structural schemas. Fully type every field, mark required fields, and avoid x-kubernetes-preserve-unknown-fields so typos and drift are rejected at admission.

Validate in the schema. Use enum, minimum/maximum, pattern, and CEL x-kubernetes-validations for cross-field rules before falling back to a validating webhook.

Split spec and status. Enable the status subresource so controllers report state without racing user edits to spec, and add the scale subresource when an HPA should target the resource.

Add printer columns. Surface the fields operators care about, plus an Age column, so kubectl get is useful without -o yaml.

Namespace-scope by default. Prefer scope: Namespaced unless the resource is genuinely cluster-wide, to keep tenancy and RBAC clean.

C - Version and Evolve Safely

Evolve additively. Add optional fields with defaults rather than changing or removing existing ones; a tightened constraint can invalidate stored objects.

Introduce new versions deliberately. Add the new version as served-but-not-stored, migrate stored objects, promote it to storage, then stop serving the old one only after consumers move.

Keep conversion lossless. A conversion webhook must round-trip through the storage version; preserve removed fields (for example in an annotation) during deprecation.

Run conversion webhooks highly available. Every read and write of the resource routes through the webhook, so downtime or an expired certificate breaks the whole type.

Clean up storedVersions. After migrating away from a version, ensure no objects are stored in it before removing it, or the API server will refuse.

D - Build the Controller Correctly

Make Reconcile idempotent. The same object is reconciled many times, on resync, restart, and owned-object events; create-if-missing and update-if-different, never assume first run.

Reconcile level-triggered. Recompute desired state from the current object each call rather than reacting to specific events, so the loop self-heals after missed messages.

Handle conflicts and stale reads. Cache reads are eventually consistent; retry on Conflict and prefer server-side apply or get-modify-retry over blind overwrites.

Use owner references. Set controller references on created objects so garbage collection and owned-event reconciliation work correctly.

Requeue instead of blocking. Return an error or RequeueAfter to retry with backoff rather than sleeping inside the reconcile loop.

Use finalizers for external cleanup. When deletion must tear down external resources, add a finalizer, clean up on deletionTimestamp, then remove it.

E - Operate Safely in Production

Scope RBAC tightly. Grant the controller only the verbs and resources it needs, and limit watched namespaces when the operator supports it, to shrink blast radius.

Enable leader election. Run multiple replicas for availability but ensure only one actively reconciles, preventing conflicting actions.

Pin versions and gate upgrades. Install operators via a pinned Helm chart or OLM through GitOps, and test upgrades with representative custom resources in staging first.

Monitor the controller itself. Alert on reconcile errors, queue depth, and latency; if the operator stops reconciling, workloads silently stop self-healing.

Set requests, limits, and a PDB. Treat the operator as a control-plane component: give it resource bounds, probes, and a PodDisruptionBudget.

Plan CRD deletion carefully. Deleting a CRD cascades to all its custom resources; know what happens to managed workloads before uninstalling an operator.

When You Are Done

You should be able to point to a strict schema, a documented versioning and conversion plan, an idempotent reconciler, and scoped RBAC with monitoring for every operator you run.

If any of those is missing, the gap is your next task before this reaches production.

FAQs

What is the single most important CRD practice? Treating the CRD as a versioned public API, so schema and version changes are planned rather than breaking.

Should we ever write our own operator? Yes, when the operational logic is specific to your domain and no mature operator fits; otherwise adopt an existing one.

How do I keep an operator's blast radius small? Scope RBAC and watched namespaces, run it in a dedicated namespace, enable leader election, and pin versions.

Do I need a conversion webhook? Only when serving multiple versions with differing schemas; if all served versions share a schema, strategy: None is enough.

Why must Reconcile be idempotent? Because resyncs, restarts, and owned-object events cause many reconciles per logical change, and non-idempotent logic corrupts state.

How should operators be installed? Declaratively and pinned, via Helm or OLM managed through GitOps, so installs and upgrades are reviewed and reproducible.

Stack versions: This page was written for Kubernetes 1.36.2, Docker Engine 29.6.1 (BuildKit default), containerd (CRI runtime on nodes), Helm 3, Compose v2, Argo CD (latest - verify at build), and Gateway API (GA - verify controller support at build).

CRDs & Operators Best Practices

Search Documentation

CRDs & Operators Best Practices