From Commit to Running Pod
Summary
- Definition: Delivery to Kubernetes is a chain that turns a git commit into a scheduled Pod through build, scan, sign, manifest update, and cluster sync.
- Insight: The pipeline never talks to production directly. It publishes an artifact and a desired state, and the cluster converges toward that state on its own.
- Key Concepts: immutable image, digest pinning, desired state, reconciliation, provenance (scan + signature), pull-based sync.
- When to Use: Any team running more than one service or one environment on Kubernetes benefits from this separation between CI (build artifacts) and CD (converge state).
- Limitations/Trade-offs: More moving parts than
kubectl apply, an added GitOps controller to operate, and a learning curve around drift and reconciliation. - Related Topics: deploy pipeline mechanics, image updater bots, progressive delivery.
Foundations
A commit is source. A running Pod is a scheduled process on a node. Everything between them exists to make that transition safe, repeatable, and auditable.
The mental model splits into two halves. CI produces a versioned artifact. CD makes the cluster run it.
CI is push-based and ephemeral. A runner checks out the commit, builds an image, tests it, scans it, signs it, and pushes it to a registry.
CD is pull-based and continuous. A controller inside the cluster watches a git repository of manifests and reconciles the live state to match.
The artifact that connects the two halves is the container image, addressed by an immutable digest such as sha256:.... Tags move; digests do not.
Kubernetes itself is a reconciliation engine. You declare desired state in objects like Deployments, and controllers work to make actual state match.
GitOps extends that same loop outward. Git becomes the source of desired state, and an in-cluster agent (Argo CD or Flux) is the controller that reconciles the cluster to git.
Mechanics & Interactions
Trace one commit through the chain.
Build. A CI job builds the image with BuildKit (the default backend in Docker Engine 29.x). The build is reproducible from the Dockerfile and pinned base images.
docker build -t registry.example.com/api:git-$GIT_SHA .
docker push registry.example.com/api:git-$GIT_SHAScan. A scanner such as Trivy or Grype inspects the image for known CVEs in OS packages and language dependencies. A policy gate fails the build on unacceptable findings.
trivy image --exit-code 1 --severity HIGH,CRITICAL \
registry.example.com/api:git-$GIT_SHASign. Sigstore cosign signs the image by digest, producing provenance that admission controllers can later verify.
cosign sign registry.example.com/api@sha256:<digest>Update the manifest. CI does not run kubectl. Instead it writes the new image reference into a git-tracked manifest, usually a Kustomize overlay or a Helm values file, and commits that change.
kustomize edit set image \
api=registry.example.com/api@sha256:<digest>
git commit -am "api: deploy $GIT_SHA" && git pushSync. The in-cluster GitOps controller notices the git change and applies it. Argo CD compares the rendered manifests to live objects and reconciles the difference.
Schedule. Once the Deployment's Pod template changes, the Deployment controller creates a new ReplicaSet and performs a rolling update. The scheduler places new Pods on nodes.
Run. On each node, the kubelet asks the CRI runtime (containerd) to pull the image by digest and start the container. Docker Engine is not the in-cluster runtime; dockershim was removed years ago.
The kubelet then runs the container's probes. Readiness gates traffic; liveness restarts a wedged process; startup protects slow boots.
At that point the commit is a running, ready Pod serving traffic behind a Service.
Advanced Considerations & Applications
Digest over tag. Pinning the deployed image by digest makes the rollout deterministic. A moving tag like :latest can silently change what runs and breaks rollback.
Environments as directories. With Kustomize overlays or Helm value files per environment, promotion is a git change from staging values to production values, fully reviewable.
Rollback is a git revert. Because desired state lives in git, reverting the deploy commit reconciles the cluster back to the previous digest. No special tooling and no snowflake state.
Provenance at admission. A policy controller (for example Kyverno or Sigstore's policy controller) can require a valid cosign signature before a Pod is admitted, closing the gap between "scanned in CI" and "trusted at runtime".
Drift detection. GitOps controllers continuously compare live state to git. A manual kubectl edit gets flagged and, if self-heal is on, reverted, which keeps production honest.
Progressive rollout. For risky changes, a rollout controller like Argo Rollouts replaces the plain Deployment and gates promotion on live metrics. That is the subject of the progressive delivery page.
Common Misconceptions
"CI deploys to the cluster." In a GitOps flow CI only builds and commits. An in-cluster controller pulls and applies, so no CI credential ever touches the API server.
"The tag is the version." Tags are mutable pointers. The digest is the identity, and reproducible delivery pins the digest.
"Docker runs my Pods." Docker builds images in CI, but nodes run containers through containerd via the CRI. The two use the same OCI image format, which is why images built by Docker run fine.
"Rolling updates are instant." A rollout is gradual by design, bounded by maxSurge, maxUnavailable, and readiness probes. Traffic shifts only as new Pods pass readiness.
"GitOps means no pipeline." You still need CI to build, test, scan, and sign. GitOps replaces the deploy step, not the build.
FAQs
Where does the pipeline stop and the cluster start? At the git commit to the manifest repo. CI ends by pushing that commit; the cluster's controller takes over from there.
Do I need Argo CD or Flux to do this? No, but a pull-based controller gives you drift detection, self-heal, and a clean audit trail that push-based kubectl apply cannot.
How is a bad deploy rolled back? Revert the deploy commit in git. The controller reconciles the previous digest back into the cluster.
Why sign images if I already scan them? Scanning checks content; signing proves origin. Admission control verifies the signature so only images your pipeline produced can run.
What actually pulls the image onto the node? The kubelet instructs containerd (the CRI runtime) to pull by digest and start the container.
Is this overkill for one service? The full chain scales down cleanly. Even a single service benefits from digest pinning, scanning, and git-tracked manifests.
Related
Stack versions: This page was written for Kubernetes 1.36.2, Docker Engine 29.6.1 (BuildKit default), containerd (CRI runtime on nodes), Helm 3, Compose v2, Argo CD (latest - verify at build), and Gateway API (GA - verify controller support at build).