cert-manager

Summary

• Definition: cert-manager is a Kubernetes controller that issues and renews X.509 certificates from ACME CAs (Let's Encrypt), private CAs, or Vault, and stores them in Secrets.
• Insight: you never run certbot again - you declare a Certificate (or annotate an Ingress) and a control loop keeps the Secret valid forever.
• Key Concepts: Issuer/ClusterIssuer, Certificate, CertificateRequest, Order/Challenge (ACME), and HTTP-01 vs DNS-01 solvers.
• When to Use: any cluster terminating TLS at an Ingress or Gateway, mTLS between services, or webhook serving certs.
• Limitations: ACME rate limits are real, DNS-01 needs cloud credentials, and a wildcard certificate cannot be issued over HTTP-01.

Recipe

Install the controller with its CRDs, create a ClusterIssuer, then let Ingress annotations do the rest.

helm repo add jetstack https://charts.jetstack.io
helm upgrade --install cert-manager jetstack/cert-manager \
  --namespace cert-manager --create-namespace \
  --set crds.enabled=true

• crds.enabled=true installs the CRDs with the chart, so Helm manages their lifecycle.
• The release runs three Deployments: the controller, the webhook, and cainjector.
• Wait for the webhook to be Ready before applying any Issuer, or admission will reject it.

Working Example

A production ClusterIssuer using Let's Encrypt with an HTTP-01 solver, plus a certificate wired to an Ingress.

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-prod
spec:
  acme:
    server: https://acme-v02.api.letsencrypt.org/directory
    email: platform@example.com
    privateKeySecretRef:
      name: letsencrypt-prod-account-key
    solvers:
      - http01:
          ingress:
            ingressClassName: nginx

The account key Secret is created by cert-manager on first registration, not by you.

Now annotate the Ingress and cert-manager creates the Certificate for you.

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: checkout
  namespace: prod
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt-prod
spec:
  ingressClassName: nginx
  tls:
    - hosts:
        - checkout.example.com
      secretName: checkout-tls
  rules:
    - host: checkout.example.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: checkout
                port:
                  number: 80

• The cert-manager.io/cluster-issuer annotation triggers the ingress-shim, which synthesizes a Certificate.
• spec.tls[].secretName is where the signed key pair lands once the Order completes.
• The hosts in spec.tls become the certificate's DNS SANs.

Watch it progress from the terminal.

kubectl -n prod get certificate checkout-tls
kubectl -n prod describe certificate checkout-tls
kubectl -n prod get order,challenge

Deep Dive

The resource chain

A Certificate is the desired state, and it spawns a CertificateRequest holding the CSR.

For ACME issuers that request spawns an Order, which spawns one Challenge per DNS name.

When every Challenge is valid, the CA signs, and the controller writes tls.crt and tls.key into the target Secret.

Issuer vs ClusterIssuer

An Issuer is namespaced and only usable by Certificates in the same namespace.

A ClusterIssuer is cluster-scoped and referenced by name from anywhere, which is what most platforms standardize on.

The issuerRef.kind field selects between them, and it defaults to Issuer - forgetting to set kind: ClusterIssuer is a classic mistake.

HTTP-01 vs DNS-01

HTTP-01 proves control by serving a token at /.well-known/acme-challenge/<token> over port 80, so the host must be publicly reachable.

cert-manager creates a temporary Pod, Service, and Ingress for the challenge, then deletes them.

DNS-01 proves control by writing a _acme-challenge TXT record, which works for private clusters and is the only path to wildcards.

solvers:
  - dns01:
      route53:
        region: us-east-1
    selector:
      dnsZones:
        - example.com

• The selector restricts a solver to specific zones, letting one issuer mix HTTP-01 and DNS-01.
• On EKS, prefer IRSA over static keys; the pod's service account assumes a role with route53:ChangeResourceRecordSets.

Explicit Certificates and Gateway API

Outside Ingress you write the Certificate yourself, which gives you control over key type, duration, and renewal window.

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: checkout-tls
  namespace: prod
spec:
  secretName: checkout-tls
  duration: 2160h      # 90d
  renewBefore: 360h    # 15d
  privateKey:
    algorithm: ECDSA
    size: 256
  dnsNames:
    - checkout.example.com
  issuerRef:
    name: letsencrypt-prod
    kind: ClusterIssuer

• renewBefore must be less than duration, or the controller rejects the resource.
• Let's Encrypt caps duration at 90 days regardless of what you ask for.
• For Gateway API, enable the gateway-shim (--enable-gateway-api) and annotate the Gateway instead of an Ingress.

Gotchas

Staging first. Let's Encrypt production allows 50 certificates per registered domain per week, and a broken solver burns through that fast. Develop against https://acme-staging-v02.api.letsencrypt.org/directory, then flip the server URL.

HTTP-01 needs port 80 open. Teams redirect all HTTP to HTTPS at the load balancer and then wonder why every Challenge is stuck pending. Exempt the /.well-known/acme-challenge/ path.

Deleting the Secret does not break you. cert-manager notices the missing Secret and reissues, but that counts against rate limits. Do not "fix" TLS by deleting Secrets in a loop.

The webhook is a hard dependency. If the cert-manager webhook is unavailable, applying any cert-manager CRD fails admission. This bites during cluster bootstrap and during node drains that evict all webhook replicas.

CRDs and helm uninstall. Removing the release can remove the CRDs, which garbage-collects every Certificate in the cluster. Check crds.keep behavior before uninstalling anything in production.

Copying Secrets across namespaces. A Secret is namespaced; a Certificate in prod cannot serve an Ingress in staging. Issue one per namespace, or use a sync tool like reflector.

Alternatives

Cloud-managed certificates (ACM on AWS, Google-managed certs) terminate TLS at the cloud load balancer and renew automatically. Pick these when your traffic never needs in-cluster TLS termination and you accept the vendor coupling.

Manual Secrets from a corporate CA work when a security team issues certs out of band. It is simple and auditable, but renewal becomes a calendar reminder, which is how outages happen.

Service mesh mTLS (Istio, Linkerd) issues workload identities automatically for pod-to-pod traffic. It solves a different problem than public edge TLS, and many clusters run both.

Vault PKI via cert-manager replaces ACME with an internal CA while keeping the same Certificate CRD. Choose it for private-only workloads that must chain to an enterprise root.

FAQs

Does cert-manager restart my pods when a certificate renews? No. It rewrites the Secret, and the Ingress controller picks up the change. Applications mounting the Secret as a volume see the update but may need a reload hook.

Why is my Challenge stuck in pending? Almost always reachability. Run kubectl describe challenge and read the Reason - it reports the actual HTTP or DNS probe failure.

Can I use one wildcard certificate for everything? Yes, with DNS-01. It reduces rate-limit pressure but concentrates blast radius, since one leaked key covers every subdomain.

What happens if the ACME account key Secret is lost? cert-manager registers a new account on the next Order. Existing certificates keep working until renewal.

Do I need cainjector? Yes, if you use CA-injected webhooks or issue certs for admission webhooks. It patches CA bundles into webhook configurations and CRDs.

How do I check expiry without a browser? kubectl -n prod get secret checkout-tls -o jsonpath='{.data.tls\.crt}' | base64 -d | openssl x509 -noout -dates.

Related

• Platform Tools Basics
• The Platform Engineer's Toolkit, Conceptually
• external-dns
• k9s & stern
• Platform Tools Best Practices

Stack versions: This page was written for Kubernetes 1.36.2, Docker Engine 29.6.1 (BuildKit default), containerd (CRI runtime on nodes), Helm 3, Compose v2, Argo CD (latest - verify at build), and Gateway API (GA - verify controller support at build).