Immutable Infrastructure Principle

Summary

• Immutable infrastructure means you never modify a running instance; you build a new versioned artifact and replace the old one.
• Insight: Containers make immutability the path of least resistance - the image is the unit of change, and rollback is just deploying the previous digest.
• Key Concepts: build-once artifacts, no in-place mutation, config and secrets injected at runtime, externalized state, digest-pinned rollouts.
• When to Use: Nearly all production container workloads, where reproducibility, auditability, and fast rollback matter.
• Limitations: Requires disciplined externalization of state and config; stateful data must live outside the ephemeral container.

Recipe

Bake the artifact once, deploy it many times, and never touch a running container.

Build an image tagged and digest-pinned, inject configuration and secrets at runtime rather than baking or editing them in place, keep the container filesystem read-only, and store durable data in volumes or managed services.

To change anything - code, a library, a config value - you build a new version and roll it out, and to undo it you roll back to the prior digest.

Working Example

Start with an image that is hard to mutate at runtime.

FROM gcr.io/distroless/static:nonroot
COPY --chown=nonroot:nonroot app /app
USER nonroot
ENTRYPOINT ["/app"]

Distroless has no shell or package manager, so there is nothing to SSH in and patch - you are pushed toward rebuilds.

Deploy it read-only with config injected, not baked.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: api
spec:
  replicas: 3
  selector:
    matchLabels: { app: api }
  template:
    metadata:
      labels: { app: api }
    spec:
      containers:
        - name: api
          image: registry.example.com/team/api@sha256:9b2c...e41
          envFrom:
            - configMapRef:
                name: api-config
            - secretRef:
                name: api-secrets
          securityContext:
            readOnlyRootFilesystem: true
            runAsNonRoot: true
            allowPrivilegeEscalation: false
          volumeMounts:
            - name: tmp
              mountPath: /tmp
      volumes:
        - name: tmp
          emptyDir: {}

The rootfs is read-only, config and secrets come from ConfigMap and Secret, and the only writable path is a scratch emptyDir.

Shipping a change is a new digest and a rolling update.

kubectl set image deployment/api api=registry.example.com/team/api@sha256:7f10...c92
kubectl rollout status deployment/api
kubectl rollout undo deployment/api   # instant rollback to prior digest

Deep Dive

Build once, promote the same artifact

The same image digest should flow from CI through staging to production unchanged.

Rebuilding per environment reintroduces drift and defeats "what you tested is what you ship."

Inject the environment-specific parts (URLs, flags, credentials) at deploy time so one artifact serves every stage.

Config and secrets at runtime

Immutability does not mean hardcoding config into the image - that would force a rebuild for every value change.

Mount configuration from ConfigMaps and secrets from Secret objects (or an external manager like Vault or a CSI secrets driver).

The artifact stays constant; only its runtime environment varies.

State lives outside the container

The container filesystem is disposable, so anything that must survive a restart lives elsewhere.

Use PersistentVolumeClaims for data that must persist on a node's storage, and managed databases or object storage for real durability.

Treat every container as if it could be killed and rescheduled at any moment, because it can be.

Rollback is a deploy, not a repair

Because each release is an immutable, versioned artifact, rolling back is deploying the previous known-good digest.

There is no "un-patching" and no snowflake state to reconstruct - the previous version is bit-for-bit what it was.

This is what makes GitOps tools like Argo CD and Flux powerful: the desired state is a pinned image plus manifests in Git, and the controller reconciles the cluster to it.

Gotchas

Using the latest tag breaks immutability because the same name resolves to different content over time; pin by digest or an immutable semantic tag.

Writing app data to the container filesystem loses it on restart and blocks readOnlyRootFilesystem; redirect writes to a volume or emptyDir.

Baking secrets or environment config into the image forces rebuilds and leaks secrets into layers; inject them at runtime instead.

SSHing into a pod to hotfix creates a snowflake that vanishes on reschedule and hides the real change from Git; fix in the image and redeploy.

Assuming rollback is safe when a database migration has already run can corrupt data; design migrations to be backward compatible with the previous image.

Alternatives

Mutable, configuration-managed infrastructure (Ansible, Chef, Puppet converging live hosts) still suits long-lived VMs and appliances that cannot be rebuilt cheaply.

Blue-green deployments run two full immutable environments and switch traffic, giving instant rollback at the cost of double capacity during the switch.

Canary rollouts shift a fraction of traffic to a new immutable version before full promotion, trading rollout speed for safety.

In-place kernel or node patching is sometimes unavoidable for the host layer, but managed node pools that replace nodes with new images keep even that immutable.

Choose based on how expensive a rebuild is and how fast you need rollback.

FAQs

Does immutable mean nothing can be written at runtime? No - the artifact is immutable, but you still write to mounted volumes and scratch space; only the baked image and OS should not change in place.

Where do config and secrets go then? Injected at runtime via ConfigMaps, Secrets, or an external secrets manager, so the image stays constant across environments.

How do I roll back? Redeploy the previous image digest; because releases are versioned artifacts, the prior state is exactly reproducible.

Is SSHing into a pod ever OK? For read-only debugging, yes; for fixing state that must persist, no - make the change in the image and redeploy.

How does this relate to GitOps? GitOps stores the desired immutable state (pinned images plus manifests) in Git and reconciles the cluster to it automatically.

What about stateful workloads? Keep the compute immutable and push durable state to PVCs or managed data services so the container remains disposable.

Related

• What a Container Really Is
• Container Basics
• Containers vs Virtual Machines
• OCI Image & Runtime Specs
• Container Fundamentals Best Practices

Stack versions: This page was written for Kubernetes 1.36.2, Docker Engine 29.6.1 (BuildKit default), containerd (CRI runtime on nodes), Helm 3, Compose v2, Argo CD (latest - verify at build), and Gateway API (GA - verify controller support at build).