Tagging & Immutability
Summary
- Tagging is how you name image builds; immutability is the guarantee that a name never repoints to different bytes.
- Insight: Tags are mutable pointers, so treat them as convenient labels and treat the digest as the real, immutable identity.
- Key Concepts: Semver tags for humans, git SHA tags for traceability, digests for deployment, and registry immutable-tag rules for enforcement.
- When to Use: Every image you build; the discipline matters most for anything that reaches staging or production.
- Limitations: Immutability prevents overwriting, so you must also plan retention and garbage collection to control storage growth.
Recipe
Give each build multiple tags, then deploy by digest.
- Build once and tag with the git SHA for traceability.
- Add a semver tag when you cut a release.
- Push all tags; capture the digest from the push output.
- Reference the digest (not the tag) in your Kubernetes manifests.
- Enforce immutable tags in the registry so releases cannot be overwritten.
Working Example
This CI snippet tags one build with a short SHA and a semver release, then pushes both.
REGISTRY=harbor.example.com/team-a/api
SHA=$(git rev-parse --short HEAD)
VERSION=1.4.0
docker build \
-t "$REGISTRY:sha-$SHA" \
-t "$REGISTRY:$VERSION" \
.
docker push "$REGISTRY:sha-$SHA"
docker push "$REGISTRY:$VERSION"
# Resolve the pushed tag to its immutable digest
DIGEST=$(docker buildx imagetools inspect "$REGISTRY:$VERSION" \
--format '{{.Manifest.Digest}}')
echo "$REGISTRY@$DIGEST"Deploy that digest, not the tag:
apiVersion: apps/v1
kind: Deployment
metadata:
name: api
spec:
replicas: 3
selector:
matchLabels:
app: api
template:
metadata:
labels:
app: api
spec:
containers:
- name: api
image: harbor.example.com/team-a/api@sha256:9b2c...e1
imagePullPolicy: IfNotPresentBecause the manifest names a digest, every replica and every future rollout pulls identical bytes.
Deep Dive
Why :latest breaks production
:latest is just a default tag with no special meaning to the registry.
It is mutable, so a later push can repoint it, and two nodes pulling "the same" tag can end up with different content.
It also defeats rollbacks, because you have no stable name for the previous known-good build.
The three tags every build should carry
A short git SHA tag ties the image back to an exact commit for debugging and audit.
A semver tag (1.4.0) communicates intent and compatibility to humans and dependency tooling.
The digest is the deployment identity and the anchor for signatures and SBOMs.
imagePullPolicy and tags interact
With a fixed tag, IfNotPresent avoids re-pulling if the node already cached that tag.
That caching is exactly why a mutable tag is dangerous: a node may keep stale bytes while another node pulls new ones.
Deploying by digest sidesteps the whole problem, because the digest is the content.
Enforcing immutability at the registry
Client discipline is not enough, so enforce it server-side.
Harbor supports immutable tag rules, ECR offers image tag immutability per repository, and ACR and Artifact Registry have equivalent settings.
Once enabled, a push that would move an existing tag is rejected rather than silently overwriting it.
Retention alongside immutability
Immutable tags mean old images accumulate, since you can no longer reclaim a name by overwriting it.
Add retention or lifecycle policies to keep a bounded number of recent images per repository and expire the rest.
Keep enough history to roll back several releases, but not so much that storage and scan cost balloon.
Exclude tags that are still referenced by running workloads from deletion, so garbage collection never removes an in-use digest.
Gotchas
Reusing a release tag after a hotfix silently changes what "1.4.0" means; cut 1.4.1 instead.
Enabling immutable tags without a retention policy causes storage to grow without bound, so add lifecycle rules.
A moving-window tag like sha-$SHA is unique per commit, but a branch tag like main is mutable and should never be deployed.
Multi-arch images use an index whose digest differs from any single-platform manifest digest, so pin the index digest for portability across amd64 and arm64 nodes.
imagePullPolicy: Always re-pulls on every pod start, which adds latency and registry load; prefer digest pins with IfNotPresent.
Alternatives
Deploy-by-tag with a strict "never overwrite" convention is simpler but relies on human discipline and registry enforcement to be safe.
Deploy-by-digest is the most robust and is what GitOps tools and admission signing policies expect, at the cost of a resolve step in CI.
Tools like Kustomize can pin digests for you via images: transformers, and Argo CD Image Updater can write resolved digests back to git, automating the pinning while keeping GitOps as the source of truth.
For local development, a mutable tag such as dev is fine, because reproducibility matters less than iteration speed there.
FAQs
Should I ever use :latest?
Only for throwaway local work; never for anything deployed, because it cannot be rolled back reliably.
Do I lose readability by deploying digests?
Keep human-readable semver and SHA tags on the image too; the digest is only the deploy reference, and tooling can show you the associated tags.
How do I roll back with digests?
Redeploy the previous known-good digest, which you should record per release in git or your release notes.
Does immutability stop me from deleting images?
No. Immutability prevents overwriting a tag; you still delete images via retention or lifecycle policies to reclaim space.
What digest do I pin for multi-arch images?
Pin the image index digest, and containerd selects the correct per-architecture manifest for each node at pull time.
Related
- Registries Basics
- Why the Software Supply Chain Matters
- Registry Authentication
- cosign & Image Signing
- Supply Chain Best Practices
Stack versions: This page was written for Kubernetes 1.36.2, Docker Engine 29.6.1 (BuildKit default), containerd (CRI runtime on nodes), Helm 3, Compose v2, Argo CD (latest - verify at build), and Gateway API (GA - verify controller support at build).