Using Operators in Production

Summary

• Running third-party operators well is mostly about evaluation, scoping, and lifecycle, not writing code: you adopt someone else's controller and own its blast radius.
• Insight: An operator is privileged, always-on software in your cluster. Treat installing one like adding a new control-plane component, not like deploying an app.
• Key Concepts: operator evaluation, RBAC scope, blast radius, upgrade path, CRD ownership, leader election, watched namespaces.
• When to Use: When a mature operator (Prometheus, cert-manager, a database operator) automates real toil better than you could safely script.
• Limitations: Operators own CRDs cluster-wide, so uninstalling or version-skewing them can break every resource they manage.

Recipe

Prefer a proven, actively maintained operator over building your own.

Read its RBAC and CRDs before installing, and scope its watched namespaces if it supports that.

Pin the version, install through GitOps, and test upgrades in a non-production cluster first.

Monitor the operator itself, not just the workloads it manages.

Working Example

Install cert-manager pinned to a specific version via Helm, then verify it is healthy before relying on it.

helm repo add jetstack https://charts.jetstack.io
helm repo update
 
helm install cert-manager jetstack/cert-manager \
  --namespace cert-manager --create-namespace \
  --version v1.17.0 \
  --set crds.enabled=true \
  --set global.leaderElection.namespace=cert-manager
 
kubectl -n cert-manager rollout status deploy/cert-manager
kubectl get crds | grep cert-manager.io

Once installed, you interact with the operator through its CRDs, not its internals.

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-prod
spec:
  acme:
    server: https://acme-v02.api.letsencrypt.org/directory
    email: platform@example.com
    privateKeySecretRef:
      name: letsencrypt-prod-account-key
    solvers:
      - http01:
          ingress:
            ingressClassName: nginx

A Certificate (or an annotated Ingress) then triggers the operator to obtain and renew TLS certificates automatically, which is the day-2 toil it removes.

Deep Dive

Evaluating an operator

Judge maturity by release cadence, issue responsiveness, security posture, and how it handles upgrades, not by feature count.

Check the CNCF Operator Capability Level it targets. Level 1-2 (install and configure) is enough for many; stateful systems benefit from higher levels but add complexity.

Read the CRDs it ships. They become part of your cluster's public API, and other teams will build on them, so their stability is your concern.

Inspect the RBAC in its install manifests. An operator that requests cluster-admin-equivalent permissions is a large blast radius you are accepting.

Lifecycle and installation

Install via a versioned, declarative path: a pinned Helm chart or an Operator Lifecycle Manager (OLM) subscription managed through GitOps like Argo CD.

Separate CRD installation from the operator Deployment where the chart allows, because CRDs are cluster-scoped and often need different change control than the workload.

Prefer namespaced or explicitly watched-namespace deployments when the operator supports them, so its authority is bounded to where it is needed.

Enable leader election so multiple replicas can run for availability while only one acts, avoiding duplicate reconciliation.

Upgrades and CRD compatibility

The riskiest moment is upgrading the operator, because a new version may ship new CRD versions or changed reconcile behavior.

Read the changelog for CRD schema changes and conversion requirements, and apply CRD updates deliberately rather than letting an automatic sync surprise you.

Test the upgrade in a staging cluster with representative custom resources. Watch for objects failing conversion or the operator re-reconciling everything at once.

Never skip across many minor versions blindly; operators, like Kubernetes itself, generally support stepwise upgrades.

Blast radius and isolation

An operator holds standing permissions and runs continuously, so a bug or a bad reconcile can affect every resource of its type at once.

Limit exposure by scoping RBAC and watched namespaces, running critical operators in dedicated namespaces, and setting resource requests and limits on the operator Pod.

Add a PodDisruptionBudget and sensible probes so the operator stays available, and alert on its reconcile-error and queue-depth metrics.

Have a rollback plan. Because uninstalling an operator can orphan or delete its managed resources, know in advance what happens to workloads if you remove it.

Gotchas

Deleting a CRD deletes its objects. Removing an operator's CRD cascades to every custom resource of that kind. Back up and understand ownership before uninstalling.

Unbounded RBAC. Some operators default to cluster-wide permissions. Scope them down or accept the wider blast radius knowingly.

Auto-upgrades via GitOps. An unpinned chart or latest tag can upgrade the operator unexpectedly and break conversion. Pin versions and gate upgrades.

Two operators fighting. Installing overlapping operators (or the same one twice without leader election) causes conflicting reconciliation. Ensure single ownership per resource type.

No monitoring of the operator itself. If the operator crashes, managed workloads silently stop being reconciled. Alert on the controller's own health and metrics.

Alternatives

Helm chart without an operator. For stateless or simple stateful apps, a chart plus manifests avoids adding a long-lived controller; see Operators vs Helm vs Plain Manifests.

Managed cloud service. For databases, a managed offering removes both the operator and the operational burden, at the cost of portability.

Build your own operator. Only when no mature operator fits and the domain logic is genuinely yours; see The Operator Pattern Explained.

FAQs

Should I build or adopt an operator? Adopt a proven one whenever possible. Building means owning security, upgrades, and on-call for a control-plane component.

How do I limit an operator's blast radius? Scope its RBAC and watched namespaces, run it in a dedicated namespace, and pin versions with gated upgrades.

What happens to my resources if I uninstall an operator? It depends on the operator, but deleting its CRDs deletes their custom resources. Verify behavior before uninstalling.

How do I install operators declaratively? Use a pinned Helm chart or OLM subscription managed through GitOps so installs and upgrades are reviewed and reproducible.

Do operators need leader election? Yes if you run multiple replicas for availability, so only one instance actively reconciles at a time.

Why monitor the operator separately? Because if it stops reconciling, your workloads keep running but drift and self-healing quietly stop, which is easy to miss.

Related

• The Operator Pattern Explained
• Operators vs Helm vs Plain Manifests
• CRD Design & Versioning
• How Controllers Work (controller-runtime)
• CRDs & Operators Best Practices

Stack versions: This page was written for Kubernetes 1.36.2, Docker Engine 29.6.1 (BuildKit default), containerd (CRI runtime on nodes), Helm 3, Compose v2, Argo CD (latest - verify at build), and Gateway API (GA - verify controller support at build).