:latest in Production

Summary

Using the :latest tag in production lets identical manifests run different code as the tag is repointed over time.
Insight: :latest is not a version. It is a mutable pointer that whoever pushed last controls, so your running code becomes non-deterministic.
Key Concepts: mutable tag, image digest, imagePullPolicy, drift, immutable deploy.
When to Use: Never for production workloads. Pin an immutable tag or a digest instead.
Limitations: Even a pinned semantic tag can be force-pushed. Only a digest is truly immutable.

Recipe

Stop referencing :latest and pin what actually runs.

Tag every build with an immutable identifier, ideally the digest.

Set imagePullPolicy explicitly so nodes do not re-pull behind your back.

Enforce the rule at admission so a :latest reference cannot reach the cluster.

Working Example

Here is the incident shape. A Deployment references :latest.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: checkout
spec:
  replicas: 6
  selector:
    matchLabels: { app: checkout }
  template:
    metadata:
      labels: { app: checkout }
    spec:
      containers:
        - name: checkout
          image: registry.example.com/checkout:latest
          imagePullPolicy: Always

The manifest never changes, so GitOps reports "in sync" and everyone believes production is stable.

Meanwhile CI pushes a new checkout:latest on every merge to main.

Nothing redeploys, because the Deployment spec is byte-for-byte identical.

Then a node autoscaler adds capacity at 2 a.m. and schedules two new pods.

With imagePullPolicy: Always, those two pods pull the current :latest, which is now three commits ahead of the six pods already running.

You now serve two code versions behind one Service, and a schema change in the new build corrupts sessions on a fraction of requests.

The fix is to pin the digest and make pulls deterministic.

containers:
  - name: checkout
    image: registry.example.com/checkout@sha256:9b2c...e41a
    imagePullPolicy: IfNotPresent

Roll out by updating the digest in the manifest, so every replica converges on exactly one build.

# Resolve the digest for the tag you tested and pin it
docker buildx imagetools inspect registry.example.com/checkout:v1.8.3 \
  --format '{{.Manifest.Digest}}'

Deep Dive

Why the tag drifts

A tag in an OCI registry is a label pointing at a manifest digest.

Pushing a new image with the same tag moves the label. The old digest still exists but the tag now resolves elsewhere.

So checkout:latest today and checkout:latest next week can be different content with identical YAML.

How imagePullPolicy interacts

imagePullPolicy: Always re-pulls on every pod start, so new pods track the moving tag.

IfNotPresent uses the node's cached image if present, so pods on warm nodes and cold nodes can diverge.

Kubernetes defaults the policy to Always when the tag is :latest, which is exactly the drift-maximizing combination.

Only a digest reference removes ambiguity: the same digest always resolves to the same bytes regardless of policy.

Rollback and provenance

With :latest you cannot cleanly roll back, because "the previous latest" is not addressable by tag.

With digests, kubectl rollout undo restores the exact prior image, and your Git history records which digest ran when.

Digests also anchor supply-chain checks: sign the digest with cosign and verify it at admission.

Gotchas

The Deployment looks healthy because readiness passes on both versions - the defect is data behavior, not a crash.

kubectl get pods shows all pods Running, so restart-count alerts never fire.

kubectl describe pod reveals the truth: compare the Image ID (a digest) across pods, not the Image field (the tag).

kubectl get pods -l app=checkout \
  -o jsonpath='{range .items[*]}{.status.containerStatuses[0].imageID}{"\n"}{end}' | sort -u

If that command prints more than one line, you are running mixed versions.

A subtler trap: mutating :latest while pods are stable means a later eviction or node reboot silently upgrades a subset.

Alternatives

Pin an immutable semantic tag like v1.8.3 when your CI guarantees tags are never force-pushed.

This is readable in manifests and adequate when the registry enforces tag immutability.

Pin a digest when you need cryptographic certainty and supply-chain verification - the strongest option.

Combine both for humans and machines: checkout:v1.8.3@sha256:... records intent and locks content.

Enforce the choice with admission policy. A Kyverno or Validating Admission Policy rule can reject any image ending in :latest or lacking a digest.

# Kyverno: block the :latest tag cluster-wide
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: disallow-latest-tag
spec:
  validationFailureAction: Enforce
  rules:
    - name: require-non-latest
      match:
        any:
          - resources: { kinds: ["Pod"] }
      validate:
        message: "Images must not use the :latest tag."
        pattern:
          spec:
            containers:
              - image: "!*:latest"

FAQs

Is :latest ever acceptable? In local development and throwaway demos, yes. In any environment where drift causes an incident, no.

Why not just set imagePullPolicy: Never? That does not fix drift, it only pins to whatever each node happened to cache, which can differ per node.

Digest strings are ugly - is there a lighter option? Immutable semantic tags are a fine middle ground if your registry enforces immutability and CI never overwrites tags.

How do I detect drift already in production? Compare imageID across pods of a Deployment. More than one unique digest means mixed versions.

Does GitOps prevent this? Only if the manifest pins a digest. GitOps happily keeps a :latest manifest "in sync" while the underlying image moves.

Stack versions: This page was written for Kubernetes 1.36.2, Docker Engine 29.6.1 (BuildKit default), containerd (CRI runtime on nodes), Helm 3, Compose v2, Argo CD (latest - verify at build), and Gateway API (GA - verify controller support at build).

:latest in Production

Summary

Using the :latest tag in production lets identical manifests run different code as the tag is repointed over time.
Insight: :latest is not a version. It is a mutable pointer that whoever pushed last controls, so your running code becomes non-deterministic.
Key Concepts: mutable tag, image digest, imagePullPolicy, drift, immutable deploy.
When to Use: Never for production workloads. Pin an immutable tag or a digest instead.
Limitations: Even a pinned semantic tag can be force-pushed. Only a digest is truly immutable.

Recipe

Stop referencing :latest and pin what actually runs.

Tag every build with an immutable identifier, ideally the digest.

Set imagePullPolicy explicitly so nodes do not re-pull behind your back.

Enforce the rule at admission so a :latest reference cannot reach the cluster.

Working Example

Here is the incident shape. A Deployment references :latest.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: checkout
spec:
  replicas: 6
  selector:
    matchLabels: { app: checkout }
  template:
    metadata:
      labels: { app: checkout }
    spec:
      containers:
        - name: checkout
          image: registry.example.com/checkout:latest
          imagePullPolicy: Always

The manifest never changes, so GitOps reports "in sync" and everyone believes production is stable.

Meanwhile CI pushes a new checkout:latest on every merge to main.

Nothing redeploys, because the Deployment spec is byte-for-byte identical.

Then a node autoscaler adds capacity at 2 a.m. and schedules two new pods.

With imagePullPolicy: Always, those two pods pull the current :latest, which is now three commits ahead of the six pods already running.

You now serve two code versions behind one Service, and a schema change in the new build corrupts sessions on a fraction of requests.

The fix is to pin the digest and make pulls deterministic.

containers:
  - name: checkout
    image: registry.example.com/checkout@sha256:9b2c...e41a
    imagePullPolicy: IfNotPresent

Roll out by updating the digest in the manifest, so every replica converges on exactly one build.

# Resolve the digest for the tag you tested and pin it
docker buildx imagetools inspect registry.example.com/checkout:v1.8.3 \
  --format '{{.Manifest.Digest}}'

Deep Dive

Why the tag drifts

A tag in an OCI registry is a label pointing at a manifest digest.

Pushing a new image with the same tag moves the label. The old digest still exists but the tag now resolves elsewhere.

So checkout:latest today and checkout:latest next week can be different content with identical YAML.

How imagePullPolicy interacts

imagePullPolicy: Always re-pulls on every pod start, so new pods track the moving tag.

IfNotPresent uses the node's cached image if present, so pods on warm nodes and cold nodes can diverge.

Kubernetes defaults the policy to Always when the tag is :latest, which is exactly the drift-maximizing combination.

Only a digest reference removes ambiguity: the same digest always resolves to the same bytes regardless of policy.

Rollback and provenance

With :latest you cannot cleanly roll back, because "the previous latest" is not addressable by tag.

With digests, kubectl rollout undo restores the exact prior image, and your Git history records which digest ran when.

Digests also anchor supply-chain checks: sign the digest with cosign and verify it at admission.

Gotchas

The Deployment looks healthy because readiness passes on both versions - the defect is data behavior, not a crash.

kubectl get pods shows all pods Running, so restart-count alerts never fire.

kubectl describe pod reveals the truth: compare the Image ID (a digest) across pods, not the Image field (the tag).

kubectl get pods -l app=checkout \
  -o jsonpath='{range .items[*]}{.status.containerStatuses[0].imageID}{"\n"}{end}' | sort -u

If that command prints more than one line, you are running mixed versions.

A subtler trap: mutating :latest while pods are stable means a later eviction or node reboot silently upgrades a subset.

Alternatives

Pin an immutable semantic tag like v1.8.3 when your CI guarantees tags are never force-pushed.

This is readable in manifests and adequate when the registry enforces tag immutability.

Pin a digest when you need cryptographic certainty and supply-chain verification - the strongest option.

Combine both for humans and machines: checkout:v1.8.3@sha256:... records intent and locks content.

Enforce the choice with admission policy. A Kyverno or Validating Admission Policy rule can reject any image ending in :latest or lacking a digest.

# Kyverno: block the :latest tag cluster-wide
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: disallow-latest-tag
spec:
  validationFailureAction: Enforce
  rules:
    - name: require-non-latest
      match:
        any:
          - resources: { kinds: ["Pod"] }
      validate:
        message: "Images must not use the :latest tag."
        pattern:
          spec:
            containers:
              - image: "!*:latest"

FAQs

Is :latest ever acceptable? In local development and throwaway demos, yes. In any environment where drift causes an incident, no.

Why not just set imagePullPolicy: Never? That does not fix drift, it only pins to whatever each node happened to cache, which can differ per node.

Digest strings are ugly - is there a lighter option? Immutable semantic tags are a fine middle ground if your registry enforces immutability and CI never overwrites tags.

How do I detect drift already in production? Compare imageID across pods of a Deployment. More than one unique digest means mixed versions.

Does GitOps prevent this? Only if the manifest pins a digest. GitOps happily keeps a :latest manifest "in sync" while the underlying image moves.

Stack versions: This page was written for Kubernetes 1.36.2, Docker Engine 29.6.1 (BuildKit default), containerd (CRI runtime on nodes), Helm 3, Compose v2, Argo CD (latest - verify at build), and Gateway API (GA - verify controller support at build).

:latest in Production

Search Documentation

:latest in Production